SlowGuess/NeurIPS_2023_OCR huggingface dataset:
[43] Ruiqing Xu, Chao Li, Junchi Yan, Cheng Deng, and Xianglong Liu. Graph convolutional network hashing for cross-modal retrieval. In Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, pages 982-988, 2019.
[44] Yi Xu, Lichen Wang, Yizhou Wang, and Yun Fu. Adaptive trajectory prediction via transferable gnn. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6520-6531, 2022.
[45] Liang Yang, Zesheng Kang, Xiaochun Cao, Di Jin, Bo Yang, and Yuanfang Guo. Topology optimization based graph convolutional network. In Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, pages 4054-4061, 2019.
[46] Donghan Yu, Ruohong Zhang, Zhengbao Jiang, Yuexin Wu, and Yiming Yang. Graph-revised convolutional network. In Machine Learning and Knowledge Discovery in Databases, pages 378-393, 2021.
[47] HaiYang Zhang, XiMing Xing, and Liang Liu. Dualgraph: A graph-based method for reasoning about label noise. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9654-9663, 2021.
A Proof of Theorem
Proof. To simplify the proof, let us assume that  $\mathbf{A} = \mathbb{1}_{\mathbb{R}^+}(\mathbf{A}) \in \{0, 1\}^{n \times n^6}$ . Consider node  $V_i$ . If  $\mathbf{A}_{ij} = 1$ , it implies that node  $V_j$  is one of the 1-hop neighbors of  $V_i$ . Now, if  $(\mathbf{A}^2)_{ij} = \mathbf{A}_i \cdot \mathbf{A}_{:j} = 1$ , it means that there exists an 1-hop neighbor of  $V_i$  that is also the 1-hop neighbor of  $V_j$ , indicating that node  $V_j$  is one of the 2-hop neighbors of  $V_i$ . Similarly, if  $(\mathbf{A}^3)_{ij} = (\mathbf{A})_i^2 \cdot \mathbf{A}_{:j} = 1$ , it signifies that there exists a 2-hop neighbor of  $V_i$  that is also the 1-hop neighbor of  $V_j$ , implying that node  $V_j$  is one of the 3-hop neighbors of  $V_i$ . By extension, if  $(\mathbf{A}^k)_{ij} = 1$ , it implies that node  $V_j$  is the one of  $k$ -hop neighbors of  $V_i$ . Based on this observation, for  $\forall j \in \{j \mid (\mathbf{A})_{ij} = 1 \cup (\mathbf{A}^2)_{ij} = 1 \cup \dots \cup (\mathbf{A}^k)_{ij} = 1\}$ , if node  $V_j$  is unlabeled, it indicates that all the  $\kappa$ -hop neighbors of  $V_i$  for  $\kappa \in \{1, \ldots, k\}$  are unlabeled nodes. Therefore, according to Definition 1, node  $V_i$  qualifies as a  $k$ -hop starved node.
Illustration of Theorem 1. To provide a clearer understanding, we present an example to illustrate the process of identifying  $k$ -hop starved nodes based on the given adjacency matrix. Fig. 3 depicts a graph consisting of 6 nodes, with 2 labeled and 4 unlabeled. The edges between the nodes are shown in the figure, with all edge weights set to 1 for simplicity. To identify the  $k$ -hop starved nodes, we need to determine the  $k$ -hop neighbors for each node. The steps below demonstrate the identification process of  $k$ -hop neighbors based on the given graph in Fig. 3, where the  $k$ -hop neighbors for each node are listed at the end of each row of the corresponding matrices:
Figure 3: A simple graph consisting of 6 nodes with 2 labeled and 4 unlabeled.
$$
\text {I d e n t i f y i n g 1 - h o p n e i g h b o r s b a s e d o n \mathbf {A} :} \quad \begin{array}{l l} & 1 \quad 2 \quad 3 \quad 4 \quad 5 \quad 6 \\ 1 & \left[ \begin{array}{c c c c c c} 1 & 0 & 1 & 0 & 1 & 0 \\ 0 & 1 & 1 & 1 & 0 & 0 \\ 1 & 1 & 1 & 0 & 0 & 1 \\ 0 & 1 & 0 & 1 & 0 & 0 \\ 1 & 0 & 0 & 0 & 1 & 0 \\ 0 & 0 & 1 & 0 & 0 & 1 \end{array} \right] \\ 2 \\ 3 \\ 4 \\ 5 \\ 6 \end{array} \quad \begin{array}{l} (V _ {3}, V _ {5}) \\ (V _ {3}, V _ {4}) \\ (V _ {1}, V _ {2}, V _ {6}) \\ (V _ {2}) \\ (V _ {1}) \\ (V _ {3}) \end{array}
$$
Since nodes  $V_{2}$  and  $V_{4}$  are labeled, we identify the 1-hop starved nodes as  $\{V_{1}, V_{5}, V_{6}\}$ .
$$
\text {I d e n t i f y i n g 2 - h o p n e i g h b o r s b a s e d o n} \mathbf {A} ^ {2}: \begin{array}{l l} & 1 \\ 1 & 2 \\ 2 & 3 \\ 4 & 5 \\ 6 & 6 \end{array} \left[ \begin{array}{c c c c c c} 1 & 2 & 3 & 4 & 5 & 6 \\ 3 & 1 & 2 & 0 & 2 & 1 \\ 1 & 3 & 2 & 2 & 0 & 1 \\ 2 & 2 & 4 & 1 & 1 & 2 \\ 0 & 2 & 1 & 2 & 0 & 0 \\ 2 & 0 & 1 & 0 & 2 & 0 \\ 1 & 1 & 2 & 0 & 0 & 2 \end{array} \right] \begin{array}{l} (V _ {2}, V _ {6}) \\ (V _ {1}, V _ {6}) \\ (V _ {4}, V _ {5}) \\ (V _ {3}) \\ (V _ {3}) \\ (V _ {1}, V _ {2}) \end{array}
$$
Now, we can identify 2-hop starved nodes from the set  $\{V_1, V_5, V_6\}$  as  $\{V_5\}$ .
$$
\text {I d e n t i f y i n g 3 - h o p n e i g h b o r s b a s e d o n} \mathbf {A} ^ {3}: \begin{array}{l l} & 1 \\ & 2 \\ & 3 \\ & 4 \\ & 5 \\ & 6 \end{array} \left[ \begin{array}{c c c c c c} 1 & 2 & 3 & 4 & 5 & 6 \\ 7 & 3 & 7 & 1 & 5 & 3 \\ 3 & 7 & 7 & 5 & 1 & 3 \\ 7 & 7 & 1 0 & 3 & 3 & 6 \\ 1 & 5 & 3 & 4 & 0 & 1 \\ 5 & 1 & 3 & 0 & 4 & 1 \\ 3 & 3 & 6 & 1 & 1 & 4 \end{array} \right] \begin{array}{l} (V _ {4}) \\ (V _ {5}) \\ (\varnothing) \\ (V _ {1}, V _ {6}) \\ (V _ {2}, V _ {6}) \\ (V _ {4}, V _ {5}) \end{array}
$$
Note that, applying the function  $\mathbb{1}_{\mathbb{R}^+}(\cdot)$  to  $\mathbf{A}$  does not affect the proof process.
We observe that there are no 3-hop starved nodes.
$$
\text {I d e n t i f y i n g 4 - h o p n e i g h b o r s b a s e d o n} \mathbf {A} ^ {4}: \begin{array}{l} 1 \\ 2 \\ 3 \\ 4 \\ 5 \\ 6 \end{array} \left[ \begin{array}{c c c c c c} 1 & 2 & 3 & 4 & 5 & 6 \\ 1 9 & 1 1 & 2 0 & 4 & 1 2 & 1 0 \\ 1 1 & 1 9 & 2 0 & 1 2 & 4 & 1 0 \\ 2 0 & 2 0 & 3 0 & 1 0 & 1 0 & 1 6 \\ 4 & 1 2 & 1 0 & 9 & 1 & 4 \\ 1 2 & 4 & 1 0 & 1 & 9 & 4 \\ 1 0 & 1 0 & 1 6 & 4 & 4 & 1 0 \end{array} \right] \begin{array}{l} (\emptyset) \\ (\emptyset) \\ (\emptyset) \\ (V _ {5}) \\ (V _ {4}) \\ (\emptyset) \end{array}
$$
We observe that there are no 4-hop starved nodes.
Note that, a node identified as  $k$ -hop starved node is also considered as a  $(k - 1)$ -hop starved node, as exemplified by node  $V_{5}$ . Consequently, if there are no  $k$ -hop starved nodes present, it follows that there are no  $(k + 1)$ -hop starved nodes.
Proof. Given that  $\mathbf{C} = \mathbf{A}[:,col\_mask] \in \mathbb{R}^{n \times c}$ ,  $\mathbf{C}$  models the affinities between all nodes and  $c$  labeled nodes. Consequently, if  $(row\_mask)_i = \mathbb{1}_{\mathbb{R}^-}(\mathbf{C}\mathbb{1}_c)_i = 1$ , indicating that  $(\mathbf{C}\mathbb{1}_c)_i = 0$ , we can deduce that there are no labeled 1-hop neighbors of node  $V_i$ . As a result, node  $V_i$  belongs to the set of 1-hop starved nodes, denoted as  $\mathsf{Set}_1(r) = \{V_i | i \in \mathsf{RM}_+\}$ . Additionally, it follows that  $\mathbf{U} = \mathbf{A}[row\_mask, col\_mask] = \mathbf{0} \in \mathbb{R}^{r \times c}$ . Considering  $\mathbf{R} = \mathbf{A}[row\_mask,:] \in \mathbb{R}^{r \times n}$ , where  $\mathbf{R}$  models the affinities between the  $r$  1-hop starved nodes and all nodes, we can examine each  $i \in \mathsf{RM}_+$  (where  $\mathsf{RM}_+$  denotes the indexes of all 1-hop starved nodes). If all 1-hop neighbors of  $V_i$  (i.e.,  $\forall j$  satisfying  $[\mathbb{1}_{\mathbb{R}^+}(\mathbf{R})]_{ij} = 1$ ) are 1-hop starved nodes (i.e.,  $j \in \mathsf{RM}_+$ ), then node  $V_i$  qualifies as a 2-hop starved node.
Illustration of Theorem 2. To facilitate understanding, we continue using Fig. 3 as an example for illustration. According to Theorem 2, the corresponding  $\mathbf{C}$ ,  $\mathbf{U}$ ,  $\mathbf{R}$  matrices are as follows:
$$
\begin{array}{c c c} & 1 & \\ \mathbf {C}: & 2 & \\ & 3 & \\ & 4 & \\ & 5 & \\ & 6 & \end{array} \quad \left[ \begin{array}{l l} 2 & 4 \\ 0 & 0 \\ 1 & 1 \\ 1 & 0 \\ 1 & 1 \\ 0 & 0 \\ 0 & 0 \end{array} \right] \quad ; \quad \mathbf {R}: \quad \begin{array}{c c c} & 1 & \\ 1 & 2 & 3 & 4 & 5 & 6 \\ 5 & \left[ \begin{array}{l l l l l l} 1 & 0 & 1 & 0 & 1 & 0 \\ 1 & 0 & 0 & 0 & 1 & 0 \\ 0 & 0 & 1 & 0 & 0 & 1 \end{array} \right] & \\ 6 & \end{array} \quad ; \quad \mathbf {U}: \quad \begin{array}{c c c} & 1 & \\ 5 & \\ & 6 & \end{array} \quad \left[ \begin{array}{l l} 2 & 4 \\ 0 & 0 \\ 0 & 0 \end{array} \right]
$$
Based on the  $\mathbf{C},\mathbf{U},\mathbf{R}$  matrices, we can determine that row_mask  $= [1,0,0,0,1,1]^{\top}$ ,  $\mathsf{RM}_{+} = \{1,5,6\}$ , the 1-hop starved nodes are  $V_{1}, V_{5}, V_{6}$ , and the 2-hop starved node is  $V_{5}$ .
Proof. Recall that  $\widehat{\rho}_{+} = \frac{\sum_{i\in\mathbb{RM}_{+}}\alpha\widehat{\mathbf{U}}_{ij}}{\sum_{i}\widetilde{\mathbf{A}}_{ij}}$  and  $\rho_{+} = \frac{\sum_{i\in\mathbb{RM}_{+}}\alpha\widetilde{\mathbf{U}}_{ij}}{\sum_{i}\widetilde{\mathbf{A}}_{ij}}$ . For  $\forall j\in \mathbb{CM}_{+}$ , if we randomly select  $\tau$  out of  $c$  labeled nodes as the supplementary adjacent points for each 1-hop starved node  $V_{i}$  then the probability of  $\widehat{\mathbf{U}}_{ij} > 0$  is  $\frac{\tau}{c}$ . Considering the  $j$ -th column  $(j\in \mathbb{CM}_{+})$ , if  $\sum_{i\in \mathbb{RM}_{+}}\alpha \widehat{\mathbf{U}}_{ij} = \sum_{i\in \mathbb{RM}_{+}}\alpha \widetilde{\mathbf{U}}_{ij}$  (i.e.,  $\widehat{\rho}_{+} = \rho_{+}$ ), it means that all the 1-hop starved nodes  $V_{i}$  (for  $i\in \mathbb{RM}_{+}$ ) select  $V_{j}$  as the supplementary adjacent point. Since there are  $r$  1-hop starved nodes, the probability of  $\widehat{\rho}_{+} = \rho_{+}$  is  $\left(\frac{\tau}{c}\right)^r$ . Moreover, since  $\tau < c$ , we have  $\widehat{\rho}_{+} = \frac{\sum_{i\in\mathbb{RM}_{+}}\alpha\widehat{\mathbf{U}}_{ij}}{\sum_{i}\widetilde{\mathbf{A}}_{ij}}\leq \rho_{+}$ , where  $\widehat{\rho}_{+} = \rho_{+}$  with only a probability of  $\left(\frac{\tau}{c}\right)^r$ .
Proof. Recall that  $\widehat{\rho}_{-} = \frac{\sum_{i \notin \mathbb{R}_{\mathbb{M}_{+}}} \widetilde{\mathbf{A}}_{ij} + \alpha \sum_{i \notin \mathbb{R}_{\mathbb{M}_{+}}} \mathbf{Q}_{ij}}{\sum_{i} \widetilde{\mathbf{A}}_{ij} + \alpha \sum_{i \notin \mathbb{R}_{\mathbb{M}_{+}}} \mathbf{Q}_{ij}}$  and  $\rho_{-} = \frac{\sum_{i \notin \mathbb{R}_{\mathbb{M}_{+}}} \widetilde{\mathbf{A}}_{ij}}{\sum_{i} \widetilde{\mathbf{A}}_{ij}}$ . For  $\forall j \in \mathbb{C}\mathbb{M}_{+}$ , if we randomly select  $\tau$  out of  $c$  labeled nodes as the supplementary adjacent points for each non-starved
Table 6: Detailed description of the datasets used in experiments.
Dataset / Statistic	Nodes	Edges	Features	Classes	Labeling Rate
ogbn-arxiv	169,343	1,166,243	128	40	53.70%
Cora140	2,708	5,429	1,433	7	5.17%
Cora390	2,708	5,429	1,433	7	14.40%
Citeseer120	3,327	4,732	3,703	6	3.61%
Citeseer370	3,327	4,732	3,703	6	11.12%
Pubmed	19,717	44,338	500	3	0.30%
node, then the probability of  $\mathbf{Q}_{ij} = 0$  is  $\left(1 - \frac{\tau}{c}\right)$ . Considering the  $j$ -th column  $(j \in \mathbb{CM}_+)$ , if  $\alpha \sum_{i \notin \mathbb{RM}_+} \mathbf{Q}_{ij} = 0$  (i.e.,  $\widehat{\rho}_- = \rho_-$ ), it means that all the non-starved nodes  $V_i$  (for  $i \notin \mathbb{RM}_+$ ) do not select  $V_j$  as the supplementary adjacent point. Since there are  $r$  1-hop starved nodes, the probability of  $\widehat{\rho}_- = \rho_-$  is  $\left(1 - \frac{\tau}{c}\right)^{n-r}$ . Moreover, since  $\sum_{i} \widetilde{\mathbf{A}}_{ij} > \sum_{i \notin \mathbb{RM}_+} \widetilde{\mathbf{A}}_{ij}(|\mathbb{RM}_| = r)$ , we have  $\widehat{\rho}_- \geq \rho_-$  where  $\widehat{\rho}_- = \rho_-$  with only a probability of  $\left(1 - \frac{\tau}{c}\right)^{n-r}$ .
Table 6 provides a comprehensive overview of the datasets used in our experiments, including various statistical characteristics. Please refer to the table for detailed information regarding the datasets. Note that, the original edge features are not used in the experiments as our goal is to infer a latent graph from the node features of the datasets.
B.2Baselines
In our experiments, we utilize publicly available code repositories provided by the respective authors and follow the hyperparameter settings outlined in their papers [4, 10, 13, 19, 46]. We rerun the codes for all the methods and record the best testing accuracy. Below, we summarize the implementation details and provide the code links for each of the used methods:
GCN+KNN [19]: https://github.com/tkipf/pygcn.
GCN&KNN [19]: https://github.com/tkipf/pygcn.
For GCN&KNN, we implement the codes by ourselves. For simplicity, we adopt the following Dirichlet energy [4] as the graph regularization loss:
$$
\mathcal {L} _ {\mathrm {r e g}} = \frac {1}{2 n ^ {2}} \sum_ {i = 1} ^ {n} \sum_ {j = 1} ^ {n} \widehat {\mathbf {A}} _ {i j} | | \mathbf {X} _ {i:} - \mathbf {X} _ {j:} | | ^ {2}.
$$
IDGL [4]: https://github.com/hugochan/IDGL.
GRCN [46]: https://github.com/PlusRoss/GRCN.
SLAPS [10]: https://github.com/BorealisAI/SLAPS-GNN.
Note that, SLAPS is a multi-task-based approach that involves training an additional self-supervision task for latent graph inference. To ensure computational efficiency, we set the maximum number of epochs to 1000 for ogbn-arxiv dataset. We adopt the following denoising autoencoder loss as the graph regularization loss [10]:
$$
\mathcal {L} _ {\mathrm {r e g}} = F \left(\mathbf {X} _ {i d x}, \mathrm {G N N} _ {\mathrm {D A E}} \left(\widetilde {\mathbf {X}}, A; \theta_ {\mathrm {G N N} _ {\mathrm {D A E}}}\right) _ {i d x}\right).
$$
where  $idx$  are the selected indices, and  $F$  is the binary cross-entropy loss or the mean-squared error loss [10].
LCGS [13]: https://github.com/hu-my/LCGS.
Table 7: Accuracy, parameters and FLOPs of SLAPS when using a different number of GNN layers.
Layers	Accuracy	Parameters	FLOPs
2	74.86 ± 0.79	645.76K	12.70G
4	70.24 ± 1.58	680.90K	13.39G
8	41.18 ± 0.88	751.17K	14.76G
C Further Discussions
C.1 Deeper GNNs
Why using deeper GNNs is not an optimal solution? This is because using deeper will bring up new issues as illustrated below. First, although increasing GNN to 4 layers reduces the number of starved nodes from near 3,000 to near 500 for the Citeseer120 dataset, the number of starved nodes still accounts for nearly  $15\%$ . In fact, the number of starved nodes depends on several factors, including the labeling rate of nodes, the graph sparsification process, and so on. How to select the number of layers of GNNs to reduce the starved nodes for different datasets and different LGI methods is not easy. Second, deeper GNNs typically provide inferior performance. To allow a  $k$ -hop starved node to receive information from labeled nodes, the GNNs need to have at least  $k + 1$  layers. However, as the number of layers  $k$  increases, the GNNs may still fail to propagate supervision information from labeled nodes to  $k$ -hop starved nodes due to the over-squashing issue [1]. Moreover, deeper GNNs will make the oversmoothing problem [10, 21] be even more severe. Empirically, we test the SLAPS method on the Pubmed dataset using a different number of layers, and the corresponding accuracies can be seen in Table 7. As we can see, the accuracy of SLAPS with a 8-layer GNN is extremely low, demonstrating that the generalization of GNNs cannot be guaranteed by simply increasing the number of layers. Moreover, deeper GNNs make the model more complexity. Table 7 lists the number of parameters and the float point operations (FLOPs) of SLAPS method with 2, 4, and 8-layer GNNs. We observe that as the number of layers increases, both the number of parameters and the FLOPs increase. In fact, the majority of existing LGI methods typically only adopt 2-layer GNNs for effectiveness and efficiency.
C.2 Degree Distribution
Why introducing connections between labeled and unlabeled nodes helps the models? We provide a basic insight to answer this question. As we know, an unlabeled node must belong to a specific class, and in the node classification task, a specific class must have some labeled nodes. As a result, we can derive the prior assumption that a good classification model will make an unlabeled node closer to some labeled nodes of the same class. That is why we select the closest labeled nodes as the supplementary adjacent points for the starved nodes. If we compulsorily add such connections between the unlabeled node and its closest labeled nodes, the prior information will be exploited during training, helping the models work better. It is worth noting that introducing connections between labeled and unlabeled nodes can lead to a substantial rise in the degrees of labeled nodes. A potential question is that whether altering the distribution of node degrees will have an adverse effect on the model. To be honest, we have not yet observed any adverse effects from the current experimental results. Nevertheless, it is still worth exploring whether the divergence in degree distributions will potentially hinder the model's performance. What the exact relationship is between the degree distributions and the model's performance in latent graph inference task is a difficult and open question that deserves in-depth exploration.
C.3 Definition of Regularization Graph
Why selecting  $k = 1$  to define the regularization graph between  $k$ -hop starved nodes and labeled ones? In fact, we can set any value for  $k$  to define the regularization graph. However, if we set  $k > 1$ , there will be a potential issue. For illustration, let us set  $k = 3$  and eliminate the 3-hop starved nodes by adding the regularization graph. In this case, 1-hop and 2-hop starved nodes will still exist since only 3-hop starved nodes are removed. To address this, a simple solution is to directly set  $k = 1$  to define the regularization graph. Why only setting  $k = 1$  works? According to the definition of starved nodes, we know that a  $k$ -hop starved node also qualifies as a  $(k - 1)$ -hop starved node. Therefore, if
Table 8: Test accuracy (%) of different models corresponding to the best validation performance.
GCN+KNN	67.28 ± 0.38	GRCN	68.62 ± 0.19	SLAPS	74.50 ± 0.33
GCN+KNN_U	72.76 ± 0.29	GRCN_U	72.30 ± 0.95	SLAPS_U	76.00 ± 0.41
GCN+KNN_R	72.98 ± 0.31	GRCN_R	72.14 ± 0.79	SLAPS_R	76.86 ± 0.83
we eliminate 1-hop starved nodes then all  $m$ -hop starved nodes for  $m > 1$  will be eliminated. Due to the effectiveness and simplicity, we therefore select  $k = 1$ .
C.4 Generalization Performance
To show the results of the models corresponding to the best validation performance, we conduct more experiments on the Pubmed dataset, and the experimental results are listed in Table 8. As we can see, our methods still improve the baselines, which further indicates their effectiveness.
D Limitation
According to Theorem 1, we can identify  $k$ -hop starved nodes for  $k \geq 1$ . Additionally, based on Theorem 2 and CUR matrix decomposition, we demonstrate the identification of  $k$ -hop starved nodes for  $k \in \{1, 2\}$ . How to identify  $k$ -hop starved nodes for  $k > 2$  based on CUR decomposition remains an interesting direction for further investigation. Nevertheless, it is worth noting that by eliminating all  $\kappa$ -hop starved nodes, there will be no  $k$ -hop starved nodes for  $\forall k > \kappa$ . Consequently, we think that only identifying  $k$ -hop starved nodes for  $k \in \{1, 2\}$  is sufficient and efficient to address the problem of supervision starvation.