SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Nurendra Choudhary
Virginia Tech Arlington, VA, USA nurendra@vt.edu
Nikhil Rao  
Microsoft  
Sunnyvale, CA, USA  
nikhilrao@microsoft.com
Chandan K. Reddy  
Virginia Tech  
Arlington, VA, USA  
reddy@cs.vt.edu
The progress in hyperbolic neural networks (HNNs) research is hindered by their absence of inductive bias mechanisms, which are essential for generalizing to new tasks and facilitating scalable learning over large datasets. In this paper, we aim to alleviate these issues by learning generalizable inductive biases from the nodes' local subgraph and transfer them for faster learning over new subgraphs with a disjoint set of nodes, edges, and labels in a few-shot setting. We introduce a novel method, Hyperbolic GRAph Meta Learner (H-GRAM), that, for the tasks of node classification and link prediction, learns transferable information from a set of support local subgraphs in the form of hyperbolic meta gradients and label hyperbolic protonets to enable faster learning over a query set of new tasks dealing with disjoint subgraphs. Furthermore, we show that an extension of our meta-learning framework also mitigates the scalability challenges seen in HNNs faced by existing approaches. Our comparative analysis shows that H-GRAM effectively learns and transfers information in multiple challenging few-shot settings compared to other state-of-the-art baselines. Additionally, we demonstrate that, unlike standard HNNs, our approach is able to scale over large graph datasets and improve performance over its Euclidean counterparts.
Graphs are extensively used in various applications, including image processing, natural language processing, chemistry, and bioinformatics. With modern graph datasets ranging from hundreds of thousands to billions of nodes, research in the graph domain has shifted towards larger and more complex graphs, e.g., the nodes and edges in the traditional Cora dataset [29] are in the order of  $10^{3}$ , whereas, the more recent OGBN datasets [17] are in the order of  $10^{6}$ . However, despite their potential, Hyperbolic Neural Networks (HNNs), which capture the hierarchy of graphs in hyperbolic space, have struggled to scale up with the increasing size and complexity of modern datasets.
HNNs have outperformed their Euclidean counterparts by taking advantage of the inherent hierarchy present in many modern datasets [4, 10, 13]. Unlike a standard Graph Neural Network (GNN), a HNN learns node representations based on an "anchor" or a "root" node for the entire graph, and the operations needed to learn these embeddings are a function of this root node. Specifically, HNN formulations [13] depend on the global origin (root node) for several transformation operations (such as Möbius addition, Möbius multiplication, and others), and hence focusing on subgraph structures to learn representations becomes meaningless when their relationship to the root node is not considered. Thus, state-of-the-art HNNs such as HGCN [4], HAT [15], and HypE [8] require access to the entire dataset to learn representations, and hence only scale to experimental datasets (with  $\approx 10^3$  nodes). Despite this major drawback, HNNs have shown impressive performance on several research domains including recommendation systems [32], e-commerce [9], natural language processing [11], and knowledge graphs [5, 8]. It is thus imperative that one develops methods to scale HNNs to larger datasets, so as to realize their full potential. To this end, we introduce a novel method, Hyperbolic GRAph Meta Learner (H-GRAM), that utilizes meta-learning to learn infer
Figure 1: Meta-learning on hyperbolic neural networks. The procedure consists of two phases - (i) meta-training to update the parameters of the HNNs and learn inductive biases (meta gradients and label protonets), and (ii) meta-testing that initializes the HNNs with the inductive biases for faster learning over new graphs with a disjoint set of nodes, edges, or labels.
mation from local subgraphs for HNNs and transfer it for faster learning on a disjoint set of nodes, edges, and labels contained in the larger graph. As a consequence of meta-learning, H-GRAM also achieves several desirable benefits that extend HNNs' applicability including the ability to transfer information on new graphs (inductive learning), elimination of over-smoothing, and few-shot learning. We experimentally show that H-GRAM can scale to graphs of size  $\mathcal{O}(10^6)$ , which is  $\mathcal{O}(10^{3})$  times larger than previous state-of-the-art HNNs. To the best of our knowledge, there are no other methods that can scale HNNs to datasets of this size.
Recent research has shown that both node-level and edge-level tasks only depend on the local neighborhood for evidence of prediction [18, 19, 42]. Inspired by the insights from such research, our model handles large graph datasets using their node-centric subgraph partitions, where each subgraph consists of a root node and the  $k$ -hop neighborhood around it. In H-GRAM, the HNN formulations establish the root node as the local origin to encode the subgraph. We theoretically show that, due to the locality of tangent space transformations in HNNs (more details in Section 3), the evidence for a node's prediction can predominantly be found in the immediate neighborhood. Thus, the subgraph encodings do not lose a significant amount of feature information despite not having access to a "global" root node. However, due to the node-centric graph partitioning, the subgraphs are non-exhaustive, i.e., they do not contain all the nodes, edges, and labels, as previously assumed by HNNs. Thus, to overcome the issue of non-exhaustive subgraphs, we formulate four meta-learning problems (illustrated in Figure 1) that learn inductive biases on a support set and transfer it for faster learning on a query set with disjoint nodes, edges, or labels. Our model learns inductive biases in the meta-training phase, which contains two steps - HNN update and meta update. HNN updates are regular stochastic gradient descent steps based on the loss function for each support task. The updated HNN parameters are used to calculate the loss on query tasks and the gradients are accumulated into a meta-gradient for the meta update. In the meta-testing phase, the models are evaluated on query tasks with parameters post the meta updates, as this snapshot of the model is the most adaptable for faster learning in a few-shot setting. Our main contributions are as follows:
1. We theoretically prove that HNNs rely on the nodes' local neighborhood for evidence in prediction, as well as, formulate HNNs to encode node-centric local subgraphs with root nodes as the local origin using the locality of tangent space transformations.
2. We develop Hyperbolic GRAph Meta Learner (H-GRAM), a novel method that learns meta information (as meta gradients and label protonets) from local subgraphs and generalize it to new graphs with a disjoint set of nodes, edges, and labels. Our experiments show that H-GRAM can be used to generalize information from subgraph partitions, thus, enabling scalability.
3. Our analysis on a diverse set of datasets demonstrates that our meta-learning setup also solves several challenges in HNNs including inductive learning, elimination of over-smoothing, and few-shot learning in several challenging scenarios.
This section reviews the relevant work in the areas of hyperbolic neural networks and meta learning.
Hyperbolic Neural Networks: Due to their ability to efficiently encode tree-like structures, hyperbolic space has been a significant development in the modeling of hierarchical datasets [43, 44]. Among its different isometric interpretations, the Poincaré ball model is the most popular one and has been applied in several HNN formulations of Euclidean networks including the recurrent (HGRU [13]), convolution (HGCN [4]), and attention layers (HAT [15]). As a result of their performance gains on hierarchical graphs, the formulations have also been extended to applications in knowledge graphs for efficiently encoding the hierarchical relations in different tasks such as representation learning (MuRP [1], AttH [5]) and logical reasoning (HypE [8]). However, the above approaches have been designed for experimental datasets with a relatively small number of nodes (in the order of  $10^{3}$ ), and do not scale to real-world datasets. Hence, we have designed H-GRAM as a meta-learning algorithm to translate the performance gains of HNNs to large datasets in a scalable manner.
Graph Meta-learning: Few-shot meta-learning transfers knowledge from prior tasks for faster learning over new tasks with few labels. Due to their wide applicability, they have been adopted in several domains including computer vision [14, 34], natural language processing [22] and, more recently, graphs [18, 39]. One early approach in graph meta-learning is Gated Propagation Networks [23] which learns to propagate information between label prototypes to improve the information available while learning new related labels. Subsequent developments such as MetaR [7], Meta-NA [41] and G-Matching [39] relied on metric-based meta learning algorithms for relational graphs, network alignment and generic graphs, respectively. These approaches show impressive performance on few-shot learning, but are only defined for single graphs. G-Meta [18] extends the metric-based techniques to handle multiple graphs with disjoint labels. However, the method processes information from local subgraphs in a Euclidean GNN, and thus, is not as capable as hyperbolic networks in encoding tree-like structures. Thus, we model H-GRAM to encode hierarchical information from local subgraphs and transfer it to new subgraphs with disjoint nodes, edges, and labels.
3 The Proposed H-GRAM Model
In this section, we define the problem setup for different meta-learning scenarios and describe our proposed model, H-GRAM, illustrated in Figure ② For better clarity, we explain the problem setup for node classification and use HGCN as the exemplar HNN model. However, the provided setup can be extended to link prediction or to other HNN models, which we have evaluated in our experiments. The preliminaries of hyperbolic operations and meta-learning are provided in Appendix A.
Our problem consists of a group of tasks  $\mathcal{T}_i\in \mathcal{T}$  which are divided into a support set  $\mathcal{T}^s$  and query set  $\mathcal{T}^q$ , where  $\mathcal{T}^s\cap \mathcal{T}^q = \phi$ . Furthermore, each task  $\mathcal{T}_i$  is a batch of node-centric subgraphs  $S_{u}$  with a corresponding label  $Y_{u}$  (class of root node in node classification or root link in link prediction). The subgraphs  $S_{u}$  could be the partitions derived from a single graph or multiple graphs, both denoted by  $\mathcal{G}^{\cup} = \mathcal{G}^{s}\cup \mathcal{G}^{q}$ . We also define  $Y_{s} = \{Y_{u}\in \mathcal{T}^{s}\}$  and  $Y_{q} = \{Y_{u}\in \mathcal{T}^{q}\}$  as the set of labels in the support and query set respectively. The primary goal of meta-learning is to learn a predictor using the support set,  $P_{\theta_*(}Y_s|\mathcal{T}^s)$  such that the model can quickly learn a predictor  $P_{\theta}(Y_s|\mathcal{T}^s)$  on the query set. Following literature in this area [18], the problem categories are defined as follows:
Figure 2: An overview of the proposed H-GRAM meta-learning framework. Here, the input graphs  $\mathcal{G}^{\cup}$  are first partitioned into node-centric subgraph partitions. We theoretically show that encoding these subgraph neighborhoods is equivalent to encoding the entire graph in the context of node classification and link prediction tasks. H-GRAM then uses an HGCN encoder to produce subgraph encodings, which are further utilized to get label prototypes. Using the HGCN gradient updates and label prototypes, the HNN model's parameters  $P_{\theta^{*}}$  is updated through a series of weight updates and meta updates for  $\eta$  meta-training steps. The parameters are then transferred to the meta-testing stage  $P_{\theta^{*}\to \theta}$  and further trained on  $\mathcal{D}_{test}^{s}$  and evaluated on  $\mathcal{D}_{test}^{q}$ .
1. Single graph, shared labels  $\langle SG, SL \rangle$ : The objective is to learn the meta-learning model  $P_{\theta_* \to \theta}$ , where  $Y_s = Y_q$  and  $|\mathcal{G}^s| = |\mathcal{G}^q| = 1$ . It should be noted that the tasks are based on subgraphs, so  $|\mathcal{G}^s| = |\mathcal{G}^q| = 1 \Rightarrow |\mathcal{T}^s| = |T^q| = 1$ . Also, this problem setup is identical to the standard node classification task considering  $\mathcal{T}_j^q \in D_{test}$  to be the test set.
2. Single graph, disjoint labels  $\langle SG, DL \rangle$ : This problem operates on the same graph in the support and query set, however unlike the previous one, the label sets are disjoint. The goal is to learn the model  $P_{\theta_{s} \to \theta}$ , where  $|\mathcal{G}^{s}| = |\mathcal{G}^{q}| = 1$  and  $Y_{s} \cap Y_{q} = \phi$ .
3. Multiple graphs, shared labels  $\langle MG, SL \rangle$ : This problem setup varies in terms of the dataset it handles, i.e., the dataset can contain multiple graphs instead of a single one. However, our method focuses on tasks which contain node-centric subgraphs, and hence, the model's aim is the same as problem 1. The aim is to learn the predictor model  $P_{\theta_* \to \theta}$ , where  $Y_s = Y_q$  and  $|\mathcal{G}^s|, |\mathcal{G}^q| > 1$ .
4. Multiple graphs, disjoint labels  $\langle MG, DL \rangle$ : In this problem, the setup is similar to the previous one, but only with disjoint labels instead of shared ones, i.e., learn a predictor model  $P_{\theta_* \to \theta}$  where  $Y_s \cap Y_q = \phi$  and  $|\mathcal{G}^s|, |\mathcal{G}^q| > 1$ .
From the problem setups, we observe that, while they handle different dataset variants, the base HNN model operates on the  $(S_u,Y_u)$  pair. So, we utilize a hyperbolic subgraph encoder and prototypical labels to encode  $S_{u}$  and get a continuous version of  $Y_{u}$  for our meta-learning algorithm, respectively.
3.2 Hyperbolic Subgraph Encoder
In previous methods [16, 18, 38], authors have shown that nodes' local neighborhood provides some informative signals for the prediction task. While the theory is not trivially extensible, we use the local tangent space of Poincaré ball model to prove that the local neighborhood policy holds better for HNN models. The reason being that, while message propagation is linear in Euclidean GNNs [42], it is exponential in HNNs. Hence, a node's influence, as given by Definition [1] outside its neighborhood decreases exponentially.
Definition 1. The influence of a hyperbolic node vector  $x_{v}^{\mathbb{H}}$  on node  $x_{u}^{\mathbb{H}}$  is defined by the influence score  $\mathcal{I}_{uv} = \exp_0^c\left(\left\| \frac{\partial\log_0^x(x_u^{\mathbb{H}})}{\partial\log_0^x(x_v^{\mathbb{H}})}\right\|\right)$ .
Definition 2. The influence of a graph  $\mathcal{G}$  with set of vertices  $\mathcal{V}$  on a node  $u\in \mathcal{V}$  is defined as  $\mathcal{I}_{\mathcal{G}}(u) = \exp_0^c\left((\sum_{v\in \mathcal{V}}\log_0^c (\mathcal{I}_{uv})\right)$ .
Theorem 1. For a set of paths  $P_{uv}$  between nodes  $u$  and  $v$ , let us define  $D_{g\mu}^{p_i}$  as the geometric mean of nodes' degree in a path  $p_i \in P_{uv}$ ,  $p_{uv}$  as the shortest path, and  $\mathcal{I}_{uv}$  as the influence of node  $v$  on  $u$ . Also, let us say  $D_{g\mu}^{min} = \min \left\{D_{g\mu}^{p_i}\forall p_i \in P_{uv}\right\}$ , then the relation  $\mathcal{I}_{uv} \leq \exp_u^c\left(K / \left(D_{g\mu}^{min}\right)^{\| p_{uv}\|\}\right)$  (where  $K$  is a constant) holds for message propagation in HGCN models.
Theorem  $\boxed{1}$  shows that the influence of a node decreases exponent-of-exponentially  $(exp_u^c = \mathcal{O}(e^n))$  with increasing distance  $\| p_{uv}\|$ . Thus, we conclude that encoding the local neighborhood of a node is sufficient to encode its features for label prediction.
Definition 3. The information loss between encoding an entire graph  $\mathcal{G}$  and a subgraph  $S_{u}$  with root node  $u$  is defined as  $\delta_{\mathbb{H}}(\mathcal{G}, S_{u}) = \exp_{0}^{c} (\log_{0}^{c} (\mathcal{I}_{\mathcal{G}}(u)) - \log_{0}^{c} (\mathcal{I}_{S_{u}}(u))).$
Theorem 2. For a subgraph  $S_{u}$  of graph  $\mathcal{G}$  centered at node  $u$ , let us define a node  $v \in \mathcal{G}$  with maximum influence on  $u$ , i.e.,  $v = \arg \max_{t}(\{\mathcal{I}_{u}t, t \in \mathcal{N}(u) \setminus u\})$ . For a set of paths  $P_{uv}$  between nodes  $u$  and  $v$ , let us define  $D_{g\mu}^{p_i}$  as the geometric mean of degree of nodes in a path  $p_i \in P_{uv}$ ,  $\| p_{uv} \|$  is the shortest path length, and  $D_{g\mu}^{min} = \min \left\{D_{g\mu}^{p_i} \forall p_i \in P_{uv}\right\}$ . Then, the information loss is bounded by  $\delta_{\mathbb{H}}(\mathcal{G}, S_u) \leq \exp_u^c\left(K / \left(D_{g\mu}^{min}\right)^{\| p_{uv} \| + 1}\right)$  (where  $K$  is a constant).
Theorem 2 shows that encoding the local subgraph is a  $e^{\| p_{uv}\|}$  order approximation of encoding the entire graph, and thus, with high enough  $\| p_{uv}\|$ , the encodings are equivalent. Note that  $\| p_{uv}\|$  is equivalent to the subgraph's neighborhood size ( $k$  in  $k$ -hop). This shows that encoding a node's local neighborhood is sufficient to encode its features. Theorem proofs are provided in Appendix B. The above theorems provide us with the theoretical justification for encoding local subgraphs into our meta-learning framework. Hence, we partition the input  $\mathcal{G}^{\cup}$  into subgraphs  $S_u = V \times E$  centered at root node  $u$ , with  $|V|$  nodes,  $|E|$  edges, a neighborhood size  $k$ , and the corresponding label  $Y_u$ . The subgraphs are processed through an  $k$ -layer HGCN network and the Einstein midpoint [33] of all the node encodings are taken as the overall encoding of the subgraph  $S_u$ . For a given subgraph  $S_u = (A,X)$ , where  $A \in \mathbb{H}^{\| V\| \times \| V\|}$  is the local adjacency matrix and  $X \in \mathbb{H}^{\| V\| \times m}$  are the node feature vectors of  $m$  dimension, the encoding procedure given can be formalized as:
$$
h _ {u} = H G C N _ {\theta^ {*}} (A, X), \text {w h e r e} h _ {u} \in \mathbb {H} ^ {\| V \| \times d} \tag {1}
$$
$$
e _ {u} = \frac {\sum_ {i = 1} ^ {\| V \|} \gamma_ {i u} h _ {i u}}{\sum_ {i = 1} ^ {\| V \|} \gamma_ {i u}}, \text {w h e r e} \gamma_ {i u} = \frac {1}{\sqrt {1 - \| h _ {i u} \| ^ {2}}} \tag {2}
$$
where  $HGCN(A,X) \in \mathbb{H}^{\| V\| \times d}$  is the output of k-layer HGCN with  $d$  output units and  $e_{u} \in \mathbb{H}^{d}$  is the final subgraph encoding of  $S_{u}$ .  $\gamma_{iu}$  is the Lorentz factor of the hyperbolic vector  $h_{iu}$  that indicates its weightage towards the Einstein midpoint.
3.3 Label Prototypes
Label information is generally categorical in node classification tasks. However, this does not allow us to pass inductive biases from the support set to the query set. Hence, to circumvent this issue, we use prototypical networks [31] as our label encoding. Our approach constructs continuous label prototypes by using the mean of meta-training nodes' features that belong to the label. These prototypes are then employed to classify meta-testing samples based on their similarity to the corresponding meta-training label prototypes. This enables our model to handle new, non-exhaustive labels in an inductive manner, without the need for additional training data. The primary idea is to form continuous label prototypes using the mean of nodes that belong to the label. To this end, the continuous label prototype of a label  $y_{k}$  is defined as  $c_{k} = \frac{\sum_{Y_{u} = y_{k}}\gamma_{i}e_{i}}{\sum_{Y_{u} = y_{k}}\gamma_{i}}$ , where  $\gamma_{i} = \frac{1}{\sqrt{1 - \|e_{i}\|^{2}}}$ , and  $e_i\in \mathbb{H}^d$  is encoding of subgraphs  $S_{u}$  with labels  $Y_{u} = y_{k}$ . For each  $S_{u}$  with class  $y_{k}$ , we compute the class distribution vector as  $p_k = \frac{e^{(-d_c^c(e_u,c_k))}}{\sum_k e^{(-d_c^c(e_u,c_k))}}$ , where  $d_{\mathbb{H}}^c (e_u,c_k) = \frac{2}{\sqrt{c}}\tanh^{-1}(\sqrt{c}\| - e_u\oplus c_k\|)$  and the loss for HNN updates  $\mathcal{L}(p,y) = \sum_jy_i\log p_j$ , where  $y_{i}$  is the one-hot encoding of the ground truth. The class distribution vector  $p_k$  is a softmax over the hyperbolic distance of subgraph encoding to the label prototypes, which indicates the probability that the subgraph belongs to the class  $y_{k}$ . The loss function  $\mathcal{L}(p,y)$  is the cross-entropy loss between ground truth labels  $y$  and the class distribution vector  $p$ .
3.4 Meta-Learning
In the previous section, we learned a continuous label encoding that is able to capture inductive biases from the subgraph. In this section, we utilize the optimization-based MAML algorithm [12] to transfer the inductive biases from the support set to the query set. To this end, we sample a batch of tasks, where each task  $\mathcal{T}_i = \{S_i,Y_i\}_{i = 1}^{\|\mathcal{T}_i\|}$ . In the meta-training phase, we first optimize the HNN parameters using the Riemannian stochastic gradient descent (RSGD) [2] on support loss, i.e., for each  $\mathcal{T}_i^s\in \mathcal{T}_{train}^s:\theta_j^*\gets exp_{\theta_j^*}^c (-\alpha \nabla \mathcal{L}^s)$ , where  $\alpha$  is the learning rate of RSGD. Using the updated parameters  $\theta_j^*$ , we record the evaluation results on the query set, i.e., loss on task  $\mathcal{T}_i^q\in \mathcal{T}_{train}^q$  is  $\mathcal{L}_i^q$ . The above procedure is repeated  $\eta$  times post which  $\mathcal{L}_i^q$  over the batch of tasks  $\mathcal{T}_i^q\in \mathcal{T}_{train}^q$  is accumulated for the meta update  $\theta^{*}\gets exp_{\theta^{*}}^{c}(-\beta \nabla \sum_{i}\mathcal{L}_{i}^{q})$ . The above steps are repeated with the updated  $\theta^{*}$  and a new batch of tasks till convergence. The final updated parameter set  $\theta^{*}\rightarrow \theta$  is transferred to the meta-testing phase. In meta-testing, the tasks  $\mathcal{T}_i^s\in \mathcal{T}_{test}^s$  are used for RSGD parameter updates, i.e.,  $\mathcal{T}_i^s\in \mathcal{T}_{test}^s:\theta_j\gets exp_{\theta_j^j}^c (-\alpha \nabla \mathcal{L}^s)$  until convergence. The updated parameters  $\theta$  are used for the final evaluation on  $\mathcal{T}_i^q\in \mathcal{T}_{test}^q$ . Our meta-learning procedure is further detailed in Appendix C and the implementation code with our experiments is available at https://github.com/Akirato/HGRAM. Details on implementation and broader impacts are provided in Appendices 3.5 and E, respectively.
H-GRAM is primarily implemented in Pytorch [26], with geoopt [21] and GraphZoo [35] as support libraries for hyperbolic formulations. Our experiments are conducted on a Nvidia V100 GPU with 16 GB of VRAM. For gradient descent, we employ Riemannian Adam [28] with an initial learning rate of 0.01 and standard  $\beta$  values of 0.9 and 0.999. The other hyper-parameters were selected based on the best performance on the validation set ( $D_{val}$ ) under the given computational constraints. In our experiments, we empirically set  $k = 2$ ,  $d = 32$ ,  $h = 4$ , and  $\eta = 10$ . We explore the following search space and tune our hyper-parameters for best performance. The number of tasks in each batch are varied among 4, 8, 16, 32, and 64. The learning rate explored for both HNN updates and meta updates are  $10^{-2}$ ,  $5 \times 10^{-3}$ ,  $10^{-3}$  and  $5 \times 10^{-4}$ . The size of hidden dimensions are selected from among 64, 128, and 256. The final best-performing hyper-parameter setup for real-world datasets is presented in Table 5.
Our experiments aim to evaluate the performance of the proposed H-GRAM model and investigate the following research questions:
RQ1: Does our hyperbolic meta-learning algorithm outperform the Euclidean baselines on various meta-learning problems?
RQ2: How does our model perform and scale in comparison to other HNN formulations in standard graph problems?
RQ3: How does H-GRAM model's performance vary with different few-shot settings, i.e., different values of  $k$  and  $N$ ?
RQ4: What is the importance of different meta information components?