SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We use a set of standard benchmark datasets and baseline methods to compare the performance of H-GRAM on meta-learning and graph analysis tasks. The HNN models do not scale to the large datasets used in the meta-learning task, and hence, we limit our tests to Euclidean baselines. To compare against HNN models, we rely on standard node classification and link prediction on small datasets. Also, we do not consider other learning paradigms, such as self-supervised learning because they require an exhaustive set of nodes and labels and do not handle disjoint problem settings.
For the task of meta-learning, we utilize the experimental setup from earlier approaches [18]; two synthetic datasets to understand if H-GRAM is able to capture local graph information and five real-world datasets to evaluate our model's performance in a few-shot setting.
- Synthetic Cycle [18] contains multiple graphs with cycle as the basis with different topologies (House, Star, Diamond, and Fan) attached to the nodes on the cycle. The classes of the node are defined by their topology.
- Synthetic BA [13] uses Barabási-Albert (BA) graph as the basis with different topologies planted within it. The nodes are labeled using spectral clustering over the Graphlet Distribution Vector [27] of each node.
- ogbn-arxiv [17] is a large citation graph of papers, where titles are the node features and subject areas are the labels.
- Tissue-PPI [16 43] contains multiple protein-protein interaction (PPI) networks collected from different tissues, gene signatures and ontology functions as features and labels, respectively.
- FirstMM-DB [25] is a standard 3D point cloud link prediction dataset.
- Fold-PPI [43] is a set of tissue PPI networks, where the node features and labels are the conjoint triad protein descriptor [30] and protein structures, respectively.
- Tree-of-Life [44] is a large collection of PPI networks, originating from different species.
4.2 Baseline Methods
For comparison with HNNs, we utilize the standard benchmark citation graphs of Cora [29], Pubmed [24], and Citeseer [29]. For the baselines, we select the following methods to understand H-GRAM's performance compared to state-of-the-art models in the tasks of meta-learning and standard graph processing.
- Meta-Graph [3], developed for few-shot link prediction over multiple graphs, utilizes VGAE [20] model with additional graph encoding signals.
Meta-GNN [40] is a MAML developed over simple graph convolution (SGC) network [36].
- FS-GIN [37] runs Graph Isomorphism Network (GIN) on the entire graph and then uses the few-shot labelled nodes to propagate loss and learn.
- FS-SGC [36] is the same as FS-GIN but uses SGC instead of GIN as the GNN network.
- ProtoNet [31] learn a metric space over label prototypes to generalize over unseen classes.
- MAML [12] is a Model-Agnostic Meta Learning (MAML) method that learns on multiple tasks to adapt the gradients faster on unseen tasks.
- HMLP [13], HGCN [4], and HAT [15] are the hyperbolic variants of Euclidean multi-layer perceptron (MLP), Graph Convolution Network (GCN), and Attention (AT) networks that use hyperbolic gyrovector operations instead of the vector space model.
It should be noted that not all the baselines can be applied to both node classification and link prediction. Hence, we compare our model against the baselines only on the applicable scenarios.
We adopt the following standard problem setting which is widely studied in the literature [13]. The details of the datasets used in our experiments are provided in Table 1. In the case of synthetic datasets, we use a 2-way setup for disjoint label problems, and for the shared label problems the cycle graph and Barabási-Albert (BA) graph have 17 and 10 labels, respectively. The evaluation of our model uses 5 and 10 gradient update steps in meta-training and meta-testing, respectively. In the case of real-world datasets, we use 3-shot and 16-shot setup for node classification and link prediction, respectively. For real-world disjoint labels problem, we use the 3-way classification setting. The evaluation of our model uses 20 and 10 gradient update steps in meta-training and meta-testing, respectively. In the case of Tissue-PPI dataset, we perform each 2-way protein function task three times and average it over 10 iterations for the final result. In the case of link prediction task, we need to ensure the distinct nature of support and query set in all meta-training tasks. For this, we hold out a fixed set comprised of  $30\%$  and  $70\%$  of the edges as a preprocessing step for every graph for the support and query set, respectively.
5.1 RQ1: Performance of Meta-Learning
To analyze the meta-learning capability of H-GRAM, we compare it against previous approaches in this area on a standard evaluation setup. We consider two experimental setups inline with previous evaluation in the literature [18]; (i) with synthetic datasets to analyze performance on different problem setups without altering the graph topology, and (ii) with real-world datasets to analyze per
Table 1: Basic statistics of the datasets used in our experiments. The columns present the dataset task (node classification or link prediction), number of graphs  $|\mathcal{G}^{\cup}|$ , nodes  $|V|$ , edges  $|E|$ , node features  $|X|$ , and labels  $|Y|$ . Node, Link, and N/L indicates whether the datasets are used for node classification, link prediction, and both, respectively.
Dataset	Task	G^U	V	E	X	Y
Synth. Cycle	Node	10	11,476	19,687	-	17
Synth. BA	Node	10	2,000	7,647	-	10
ogbn-arxiv	Node	1	169,343	1,166,243	128	40
Tissue-PPI	Node	24	51,194	1,350,412	50	10
FirstMM-DB	Link	41	56,468	126,024	5	2
Fold-PPI	Node	144	274,606	3,666,563	512	29
Tree-of-Life	Link	1,840	1,450,633	8,762,166	-	2
Cora	N/L	1	2,708	5,429	1,433	7
Pubmed	N/L	1	19,717	44,338	500	3
CiteSeer	N/L	1	3,312	4,732	3,703	6
Table 2: Performance of H-GRAM and the baselines on synthetic and real-world datasets. The top three rows define the task, problem setup (Single Graph (SG), Multiple Graphs (MG), Shared Labels (SL) or Disjoint Labels (DL)) and dataset. The problems with disjoint labels use a 2-way meta-learning setup, and in the case of shared labels, the cycle (S. Cy) and BA (S. BA) graph have 17 and 10 labels, respectively. In our evaluation, we use 5 and 10 gradient update steps in metatraining and meta-testing, respectively. The columns present the average multi-class classification accuracy and  $95\%$  confidence interval over five-folds. Note that the baselines are only defined for certain tasks, “-” implies that the baseline is not defined for the task and setup. Meta-Graph is only defined for link prediction. The confidence intervals for the results are provided in Appendix D.
Synthetic Datasets	Real-world (Node Classification)	Real-world (Link Prediction)
Task Setup Dataset	\( \langle SG,DL\rangle \)S. Cy	\( \langle MG,SL\rangle \)S. BA	\( \langle MG,SL\rangle \)S. Cy	\( \langle MG,SL\rangle \)S. BA	\( \langle SG,DL\rangle \)ogbn-arxiv	\( \langle MG,SL\rangle \)Tissue-PPI	\( \langle MG,DL\rangle \)Fold-PPI	\( \langle MG,SL\rangle \)FirstMM-DB	\( \langle MG,SL\rangle \)Tree-of-Life
Meta-Graph	-	-	-	-	-	-	-	.719	.705
Meta-GNN	.720	.694	-	-	-	.273	-	-	-
FS-GIN	.684	.749	-	-	-	.336	-	-	-
FS-SGC	.574	.715	-	-	-	.347	-	-	-
ProtoNet	.821	.858	.282	.657	.749	.866	.372	.546	.382
MAML	.842	.848	.511	.726	.653	.844	.389	.745	.482
G-META	.872	.867	.542	.734	.767	.867	.451	.768	.561
H-GRAM	.883	.873	.555	.746	.779	.888	.472	.786	.584
formance for practical application. Based on the problem setup, the datasets are partitioned into node-centric subgraphs with corresponding root node's label as the subgraph's ground truth label. The subgraphs are subsequently batched into tasks which are further divided into support set and query set for meta-learning. The evaluation metric for both the tasks of node classification and link prediction is accuracy  $\mathcal{A} = |Y = \hat{Y}| / |Y|$ . For robust comparison, the metrics are computed over five folds of validation splits in a 2-shot setting for node classification and 32-shot setting for link prediction. Table 2 presents the five-fold average and  $95\%$  confidence interval of our experiments on synthetic and real-world datasets, respectively. From the results, we observe that H-GRAM consistently outperforms the baseline methods on a diverse set of datasets and meta-learning problem setups. For the disjoint labels setting, H-GRAM outperforms the best baseline in both the cases of single and multiple graphs. In the case of synthetic graphs, we observe that subgraph methods of H-GRAM and G-Meta outperform the entire graph encoding based approaches showing that subgraph methods are able to limit the over-smoothing problem [6] and improve performance. Also, we observe that meta-learning methods (ProtoNet and MAML) are unreliable in their results producing good results for some tasks and worse for others, whereas H-GRAM is consistently better across the board. Hence, we conclude that using label prototypes to learn inductive biases and transferring them using MAML meta updates is a more robust technique. We note that H-GRAM, unlike previous HNN models, is able to handle graphs with edges and nodes in the order of millions, as evident by the performance on large real-world datasets including ogbn-arxiv, Tissue-PPI, Fold-PPI, and Tree-of-Life. Our experiments clearly demonstrate the significant performance of H-GRAM in a wide-range of applications and prove the effectiveness of meta-learning in HNN models.
Table 3: Comparison with HNN models on standard benchmarks. We compare the Single Graph, Shared Labels (SG,SL) setup of the H-GRAM model to the baselines. The columns report the average multi-class classification accuracy and  $95\%$  confidence interval over five-folds on the tasks of node classification (Node) and link prediction (Link) in the standard citation graphs.
Dataset Task	Cora	Pubmed	Citeseer
Node	Link	Node	Link	Node	Link
HMLP	.754±.029	.765±.047	.657±.045	.848±.038	.879±.078	.877±.090
HAT	.796±.036	.792±.038	.681±.034	.908±.038	.939±.034	.922±.036
HGCN	.779±.026	.789±.030	.696±.029	.914±.031	.950±.032	.928±.030
H-GRAM	.827±.037	.790±.026	.716±.029	.896±.025	.924±.033	.936±.030
5.2 RQ2: Comparison with HNN models
The current HNN formulations of HMLP, HAT, and HGCN do not scale to large datasets, and hence, we were not able to compare them against H-GRAM on large-scale datasets. However, it is necessary to compare the standard HNN formulations with H-GRAM to understand the importance of subgraph encoders and meta-learning.
Figure 3: Time taken (per epoch) by H-GRAM compared to other HNNs with varying number of nodes  $|\mathcal{V}| = \{10^i\}_{i=1}^7$  in Syn. BA graph. H-GRAM(multi) is the multi-GPU version of H-GRAM.
Thus, we utilize the single graph and shared labels setup of H-GRAM to evaluate its performance on citation networks of Cora, Pubmed, and Citeseer for both tasks of node classification and link prediction. We also compare the time taken by our model and other HNNs on varying number of nodes  $\left(|\mathcal{V}| = \{10^i\}_{i=1}^7\right)$  in the Synthetic BA graph. For this experiment, we also consider a multi-GPU version of H-GRAM that parallelizes the HNN update computations and accumulates them for meta update. From our results, presented in Table 3 we observe that H-GRAM is, on average, able to outperform the best baseline. This shows that our formulation of HNN models using meta-learning over node-centric subgraphs is more effective than the traditional models, while also being scalable over large datasets. The scalability allows for multi-GPU training and translates the performance gains of HNNs to larger datasets. In the results provided in Figure 3 we observe that the time taken by the models is inline with their parameter complexity (HMLP≤HGCN≤HAT≤H-GRAM). However, the traditional HNNs are not able to scale beyond  $|\mathcal{V}| = 10^4$ ,
whereas, H-GRAM is able to accommodate large graphs. Another point to note is that H-GRAM(multi) is able to parallelize well over multiple GPUs with its time taken showing stability after  $10^{4}$  nodes (which is the size of nodes that a single GPU can accommodate).
5.3 RQ3:Challenging Few-shot Settings
To understand the effect of different few-shot learning scenarios, we vary the number of few-shots  $M$  and hops  $K$  in the neighborhood. For the experiment on few-shot classification, we consider the problems of node classification on Fold-PPI dataset and link prediction on FirstMM-DB dataset. We vary  $M = 1,2,3$  and  $M = 16,32,64$  for node classification and link prediction, respectively, and calculate the corresponding accuracy and  $95\%$  confidence interval of H-GRAM. The results for this experiment are presented in Figures 4a and 4b for node classification and link prediction, respectively. To determine the effect of hops in the neighborhood, we vary  $K = 1,2,3$  for the same problem setting and compute the corresponding performance of our model. The results for varying neighborhood sizes are reported in Figure 4c. In the results on varying the number of few-shots, we observe a linear trend in both the tasks of node classification and link prediction, i.e., a linear increase in H-GRAM's accuracy with an increase in the number of shots in meta-testing. Thus, we conclude that H-GRAM, like other generic learning models, performs better with an increasing number of training samples. In the experiment on increasing the neighborhood size, we observe that in the task of node classification,  $K = 3$  shows the best performance, but in link prediction  $K = 2$
has the best performance, with a significant drop in  $K = 3$ . Thus, for stability, we choose  $K = 2$  in our experiments. The trend in link prediction also shows that larger neighborhoods can lead to an increase in noise, which can negatively affect performance.
Figure 4: Performance of H-GRAM on challenging few-shot settings. The reported accuracies are multi-class classification accuracy averaged over five-fold runs of our model.
5.4 RQ4: Ablation Study
In this section, we aim to understand the contribution of the different components in our model. To this end, we compare variants of our model by (i) varying the base HNN model (HMLP, HAT, and HGCN), and (ii) deleting individual meta-learning components - H-ProtoNet implies H-GRAM without meta updates and H-MAML implies H-GRAM without prototypes. The model variants are compared on the real-world datasets and the results are presented in Table 4. The ablation study indicates that meta gradient updates and label prototypes contribute to  $\approx 16\%$  and  $\approx 6\%$  improvement in H-GRAM's performance, respectively. This clearly demonstrates the ability of label prototypes in encoding inductive biases and that of meta gradients in transferring the knowledge from metatraining to the meta-testing phase. Additionally, from our study on different HNN bases for H-GRAM, we note that the HGCN base outperforms the other bases of HMLP and HAT by  $\approx 19\%$  and  $\approx 2\%$ , respectively. Thus, we choose HGCN as the base in our final model.
Table 4: Ablation study results - H-ProtoNet and H-MAML can be considered H-GRAM's model variants without meta updates and label prototypes, respectively. H-GRAM(HMLP) and H-GRAM(HAT) represents the variant of H-GRAM with HMLP and HAT as base, respectively. Our final model, presented in the last row, uses HGCN as the base model. The columns report the average multi-class classification accuracy and  $95\%$  confidence interval over five-folds on different tasks.
Task	Node Classification	Link Prediction
Setup	<SG, DL>	<MG, SL>	<MG, DL>	<MG, SL>	<MG, SL>
Dataset	ogbn-arxiv	Tissue-PPI	Fold-PPI	FirstMM-DB	Tree-of-Life
H-ProtoNet	.389±.019	.559±.027	.398±.023	.799±.015	.716±.004
H-MAML	.407±.023	.762±.056	.502±.046	.777±.018	.739±.005
H-GRAM(HMLP)	.370±.036	.537±.044	.372±.036	.772±.028	.688±.019
H-GRAM(HAT)	.462±.032	.777±.028	.573±.048	.794±.023	.732±.021
H-GRAM(ours)	.472±.035	.786±.031	.584±.044	.804±.021	.742±.013
In this paper, we introduce H-GRAM, a scalable hyperbolic meta-learning model that is able to learn inductive biases from a support set and adapt to a query set with disjoint nodes, edges, and labels by transferring the knowledge. We have theoretically proven the effectiveness of node-centric subgraph information in HNN models, and used that to formulate a meta-learning model that can scale over large datasets. Our model is able to handle challenging few-shot learning scenarios and also outperform the previous Euclidean baselines in the area of meta-learning. Additionally, unlike previous HNN models, H-GRAM is also able to scale to large graph datasets.
[41] Fan Zhou, Chengtai Cao, Goce Trajcevski, Kunpeng Zhang, Ting Zhong, and Ji Geng. Fast network alignment via graph meta-learning. In IEEE INFOCOM 2020 - IEEE Conference on Computer Communications, pages 686-695, 2020. doi: 10.1109/INFOCOM41043.2020.9155456.
[42] Jie Zhou, Ganqu Cui, Shengding Hu, Zhengyan Zhang, Cheng Yang, Zhiyuan Liu, Lifeng Wang, Changcheng Li, and Maosong Sun. Graph neural networks: A review of methods and applications. AI Open, 1:57-81, 2020. ISSN 2666-6510. doi: https://doi.org/10.1016/j.aiopen.2021.01.001. URL https://www.sciencedirect.com/science/article/pii/S2666651021000012
[43] Marinka Zitnik and Jure Leskovec. Predicting multicellular function through multi-layer tissue networks. Bioinformatics, 33(14):i190-i198, 2017.
[44] Marinka Zitnik, Rok Sosic, Marcus W. Feldman, and Jure Leskovec. Evolution of resilience in protein interactomes across the tree of life. Proceedings of the National Academy of Sciences, 116(10):4426-4433, 2019. doi: 10.1073/pnas.1818013116. URL https://www.pnas.org/doi/abs/10.1073/pnas.1818013116