SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Jianglin Lu<sup>1*</sup> Yi Xu<sup>1</sup> Huan Wang<sup>1</sup> Yue Bai<sup>1</sup> Yun Fu<sup>1,2</sup>
$^{1}$ Department of Electrical and Computer Engineering, Northeastern University
$^{2}$ Khoury College of Computer Science, Northeastern University
Project Page: https://jianglin954.github.io/LGI-LS/
Latent graph inference (LGI) aims to jointly learn the underlying graph structure and node representations from data features. However, existing LGI methods commonly suffer from the issue of supervision starvation, where massive edge weights are learned without semantic supervision and do not contribute to the training loss. Consequently, these supervision-starved weights, which may determine the predictions of testing samples, cannot be semantically optimal, resulting in poor generalization. In this paper, we observe that this issue is actually caused by the graph sparsification operation, which severely destroys the important connections established between pivotal nodes and labeled ones. To address this, we propose to restore the corrupted affinities and replenish the missed supervision for better LGI. The key challenge then lies in identifying the critical nodes and recovering the corrupted affinities. We begin by defining the pivotal nodes as  $k$ -hop starved nodes, which can be identified based on a given adjacency matrix. Considering the high computational burden, we further present a more efficient alternative inspired by CUR matrix decomposition. Subsequently, we eliminate the starved nodes by reconstructing the destroyed connections. Extensive experiments on representative benchmarks demonstrate that reducing the starved nodes consistently improves the performance of state-of-the-art LGI methods, especially under extremely limited supervision (6.12% improvement on Pubmed with a labeling rate of only 0.3%).
Graph neural networks (GNNs) [8, 12, 23, 33] have recently received considerable attention due to their strong ability to handle complex graph-structured data. GNNs consider each data sample as a node and model the affinities between nodes as the weights of edges. The edges as a whole constitute the graph structure or topology of the data. By integrating the graph topology into the training process of representation learning, GNNs have achieved remarkable performance across a wide range of tasks, such as classification [19, 37], clustering [31, 40], retrieval [5, 43], and recognition [35, 44].
Although effective, existing GNNs typically require a prior graph to learn node representations, which poses a major challenge when encountering incomplete or even missing graphs. This limitation has spurred the development of latent graph inference (LGI) [7, 10, 17, 22, 32], also known as graph structure learning [9, 24, 39, 42]. LGI aims to jointly learn the underlying graph and discriminative node representations solely from the features of nodes in an end-to-end fashion. By adaptively learning the graph topology, LGI models are empowered with great ability to remove noise and capture more complex structure of the data [13, 26, 27, 47]. Consequently, LGI emerges as a promising research topic with a broad range of applications, such as point cloud segmentation [41], disease prediction [6], multi-view clustering [31], and brain connectome representation [18].
*Corresponding author: JianglinLu@outlook.com.
However, many LGI methods suffer from a so-called supervision starvation (SS) problem [10], where a number of edge weights are learned without any semantic supervision during the graph inference stage. Specifically, given a  $k$ -layer GNN, if a node and all of its predefined neighbors (from 1- to  $k$ -hop) are unlabeled, the edge weights between this node and its neighbors will not contribute to the training loss and cannot be semantically optimal after training. This will lead to poor generalization performance since these under-trained weights are inevitably used to make predictions for testing samples. In this paper, we discover that the SS problem is actually caused by the graph sparsification operation, which severely destroys the important connections established between pivotal nodes and labeled ones. Based on this observation, we propose to restore the destroyed connections and replenish the missed supervision for better LGI.
Specifically, we first define the pivotal nodes as  $k$ -hop starved nodes. Then, the SS problem can be transformed into a more manageable task of eliminating starved nodes. We propose to identify the  $k$ -hop starved nodes based on the  $k$ -th power of a given adjacency matrix. After identification, we can diminish the starved nodes by incorporating a regularization adjacency matrix into the initial one. However, the above identification approach suffers from high computational complexity due to matrix multiplication. For example, given a 2-layer GNN, identifying 2-hop starved nodes requires at least  $\mathcal{O}(n^3)$ , where  $n$  denotes the number of nodes. To address this, we present a more efficient alternative solution inspired by CUR matrix decomposition [2, 3]. We find that with appropriate column and row selection, the  $U$  matrix obtained through CUR decomposition of the initial adjacency matrix is actually a zero matrix. Thus, we can eliminate the starved nodes by reconstructing the  $U$  matrix.
Although recovering the  $U$  matrix encourages more supervision, such strategy may cause a potential issue we call weight contribution rate decay. In other words, the weight contribution rate (WCR) of non-starved nodes decays as the number of starved nodes increases, resulting in a final latent graph that heavily relies on the regularization one. To tackle this issue, we propose two simple strategies, i.e., decreasing the WCR of starved nodes and increasing the WCR of non-starved nodes (see Sec. 3.3 for details). The main contributions of this paper can be summarized as follows:
- We identify that graph sparsification is the main cause of the supervision starvation (SS) problem in latent graph inference (LGI). Therefore, we propose to restore the affinities corrupted by the sparsification operation to replenish the missed supervision for better LGI.
- By defining  $k$ -hop starved nodes, we transform the SS issue into a more manageable task of removing starved nodes. Inspired by CUR decomposition, we propose a simple yet effective solution to determine the starved nodes and diminish them using a regularization graph.
- The proposed approach is model-agnostic and can be seamlessly integrated into various LGI models. Extensive experiments demonstrate that eliminating the starved nodes consistently enhances the state-of-the-art LGI methods, particularly under extremely limited supervision.
Let  $\mathbf{A}_{i:},\mathbf{A}_{:j},\mathbf{A}_{ij}$ , and  $\mathbf{A}^k$  denote the  $i$ -th row, the  $j$ -th column, the element at the  $i$ -th row and  $j$ -th column, and the  $k$ -th power of the matrix  $\mathbf{A}$ , respectively. Let  $\mathbf{1}_n$  represent an  $n$ -dimensional column vector with all elements being 1. Let  $\mathbb{1}_{\mathbb{R}^+}(\mathbf{A})$  be an element-wise indicator function that sets  $\mathbf{A}_{ij}$  to 1 if it is positive, and 0 otherwise. Conversely,  $\mathbb{1}_{\mathbb{R}^-}(\mathbf{A})$  sets  $\mathbf{A}_{ij}$  to 0 if it is positive, and 1 otherwise.
2.2 Latent Graph Inference
Definition 1 (Latent Graph Inference). Given a graph  $\mathcal{G}(\mathcal{V},\mathbf{X})$  containing  $n$  nodes  $\mathcal{V} = \{V_1,\ldots ,V_n\}$  and a feature matrix  $\mathbf{X}\in \mathbb{R}^{n\times d}$  with each row  $\mathbf{X}_i\colon \in \mathbb{R}^d$  representing the  $d$ -dimensional attributes of node  $V_{i}$ , latent graph inference (LGI) aims to simultaneously learn the underlying graph topology encoded by an adjacency matrix  $\mathbf{A}\in \mathbb{R}^{n\times n}$  and the discriminative  $d^{\prime}$ -dimensional node representations  $\mathbf{Z}\in \mathbb{R}^{n\times d^{\prime}}$  based on  $\mathbf{X}$ , where the learned  $\mathbf{A}$  and  $\mathbf{Z}$  are jointly optimal for certain downstream tasks  $\mathcal{T}$  given a specific loss function  $\mathcal{L}$ .
In general, an LGI model mainly consists of a latent graph generator  $\mathcal{P}_{\Phi}(\mathbf{X})$ :  $\mathbb{R}^{n\times d}\to \mathbb{R}^{n\times n}$  that generates an adjacency matrix  $\mathbf{A}$  based on the node features  $\mathbf{X}$ , and a node encoder  $\mathcal{F}_{\Theta}(\mathbf{X},\mathbf{A})$ :
$\mathbb{R}^{n\times d}\to \mathbb{R}^{n\times d^{\prime}}$  that learns discriminative node representations  $\mathbf{Z}$  based on  $\mathbf{X}$  and the learned A. In practice,  $\mathcal{F}_{\Theta}$  is typically implemented using a  $k$  -layer GNN, while  $\mathcal{P}_{\Phi}$  can be realized through different strategies such as full parameterization [17, 36], MLP [10, 25], or attentive network [4, 15]. In this paper, we adopt the most common settings from existing LGI literature [4, 10, 13, 25, 46], considering  $\mathcal{T}$  as the semi-supervised node classification task and  $\mathcal{L}$  as the cross-entropy loss.
2.3 Supervision Starvation
To illustrate the supervision starvation problem [10], we consider a general LGI model  $\mathcal{M}$  consisting of a latent graph generator  $\mathcal{P}_{\Phi}$  and a node encoder  $\mathcal{F}_{\Theta}$ . For simplicity, we ignore the activation function and assume that  $\mathcal{F}_{\Theta}$  is implemented using a 1-layer GNN, i.e.,  $\mathcal{F}_{\Theta} = \mathrm{GNN}_1(\mathbf{X},\mathbf{A};\Theta)$ , where  $\mathbf{A} = \mathcal{P}_{\Phi}(\mathbf{X})$ . For each node  $\mathbf{X}_{i:}$ , the corresponding node representation  $\mathbf{Z}_{i:}$  learned by the model  $\mathcal{M}$  can be expressed as:
$$
\mathbf {Z} _ {i:} = \mathbf {A} _ {i:} \mathbf {X} \boldsymbol {\Theta} = \left(\sum_ {j \in \Omega} \mathbf {A} _ {i j} \mathbf {X} _ {j:}\right) \boldsymbol {\Theta}, \tag {1}
$$
where  $\Omega = \{j\mid \mathbb{1}_{\mathbb{R}^{+}}(\mathbf{A})_{ij} = 1\}$  and  $\mathbf{A}_{ij} = \mathcal{P}_{\Phi}(\mathbf{X}_{i:},\mathbf{X}_{j:})$  . Consider the node classification loss:
$$
\min  _ {\mathbf {A}, \boldsymbol {\Theta}} \mathcal {L} = \sum_ {i \in \mathcal {Y} _ {L}} \sum_ {j = 1} ^ {| \mathcal {C} |} \mathbf {Y} _ {i j} \ln \mathbf {Z} _ {i j} = \sum_ {i \in \mathcal {Y} _ {L}} \mathbf {Y} _ {i:} \ln \mathbf {Z} _ {i:} ^ {\top} = \sum_ {i \in \mathcal {Y} _ {L}} \mathbf {Y} _ {i:} \ln \left(\left(\sum_ {j \in \Omega} \mathbf {A} _ {i j} \mathbf {X} _ {j:}\right) \boldsymbol {\Theta}\right) ^ {\top}, \tag {2}
$$
where  $\mathcal{V}_L$  represents the set of indexes of labeled nodes and  $|\mathcal{C}|$  denotes the size of label set. For  $\forall i\in \mathcal{V}_L$ ,  $j\in \Omega$ ,  $\mathbf{A}_{ij}$  is optimized via backpropagation under the supervision of label  $\mathbf{Y}_{i:}$ . For  $\forall i\notin \mathcal{V}_L$ , however, if  $j\notin \mathcal{V}_L$  for  $\forall j\in \Omega$ ,  $\mathbf{A}_{ij}$  will receive no supervision from any label and, as a result, cannot be semantically optimal after training. Consequently, the learning models exhibit poor generalization as the predictions of testing nodes inevitably rely on these supervision-starved weights. This phenomenon is referred to as supervision starvation (SS), where many edge weights are learned without any label supervision. It is easy to infer that this issue also persists in a  $k$ -layer GNN.
We may ask why this problem arises? In fact, the SS problem is caused by a common and necessary post-processing operation known as graph sparsification, which is employed in the majority of LGI methods [10, 25, 38, 46] to generate a sparse latent graph. To be more specific, graph sparsification adjusts the initial dense graph to a sparse one through the following procedure:
$$
\mathbf {A} _ {i j} = \left\{ \begin{array}{l l} \mathbf {A} _ {i j}, & \text {i f} \mathbf {A} _ {i j} \in \operatorname {t o p -} \kappa (\mathbf {A} _ {i:}) \\ 0, & \text {o t h e r w i s e ,} \end{array} \right. \tag {3}
$$
where  $\mathrm{top} - \kappa (\mathbf{A}_{i:})$  denotes the set of the top  $\kappa$  values in  $\mathbf{A}_{i}$ . After this sparsification operation, a significant number of edge weights are directly erased, including the crucial connections established between pivotal nodes and labeled nodes. Another question that may arise is: how many important nodes or connections suffer from this problem? We delve into this question in the next section.
3.1 Identification & Elimination of Starved Nodes
We first introduce the definitions of  $k$ -hop starved node and the corresponding  $k$ -hop starved weight.
Definition 2 ( $k$ -hop Starved Node). Given a graph  $\mathcal{G}(\mathcal{V}, \mathbf{X})$  consisting of  $n$  nodes  $\mathcal{V} = \{V_1, \ldots, V_n\}$  and the corresponding node features  $\mathbf{X}$ , for a  $k$ -layer graph neural network  $\mathrm{GNN}_k(\mathbf{X}; \Theta)$  with network parameters  $\Theta$ , the unlabeled node  $V_i$  is a  $k$ -hop starved node if, for  $\forall \kappa \in \{1, \ldots, k\}$ ,  $\forall V_j \in \mathcal{N}_{\kappa}(i)$ , where  $\mathcal{N}_{\kappa}(i)$  is the set of  $\kappa$ -hop neighbors of  $V_i$ ,  $V_j$  is unlabeled. Specifically, 0-hop starved nodes are defined as the unlabeled nodes<sup>2</sup>.
Based on Definition 2, we can define the  $k$ -hop starved weight as follows.
Here, we want to clarify that self-connections are not considered when defining  $k$ -hop neighbors.
Definition 3 ( $k$ -hop Starved Weight). If an edge exists between nodes  $V_{i}$  and  $V_{j}$ , the associated edge weight  $\mathbf{A}_{ij}$  is a  $k$ -hop starved weight if both of the  $V_{i}$  and  $V_{j}$  qualifies as  $(k - 1)$ -hop starved nodes<sup>3</sup>.
According to the definition presented, it is evident that  $k$ -hop starved weights are precisely the ones that receive no semantic supervision from labels. As a result, to address the SS problem, our focus should be on reducing the presence of  $k$ -hop starved nodes. The following theorem illustrates how we can identify such nodes based on a given initial adjacency matrix.
Theorem 1. Given a sparse adjacency matrix  $\mathbf{A} \in \mathbb{R}^{n \times n}$  with self-connections generated on graph  $\mathcal{G}(\mathcal{V},\mathbf{X})$  by a latent graph inference model with a  $k$ -layer graph neural network  $\mathrm{GNN}_k(\mathbf{X};\Theta)$ , the node  $V_i$  is a  $k$ -hop starved node, if  $\exists j \in \{1,\dots,n\}$ , such that  $[\mathbb{1}_{\mathbb{R}^+}(\mathbf{A})]_{ij}^k = 1$ , and for  $\forall j \in \{j \mid [\mathbb{1}_{\mathbb{R}^+}(\mathbf{A})]_{ij} = 1 \cup [\mathbb{1}_{\mathbb{R}^+}(\mathbf{A})]_{ij}^2 = 1 \cup \dots \cup [\mathbb{1}_{\mathbb{R}^+}(\mathbf{A})]_{ij}^k = 1\}$ ,  $V_j$  is unlabeled.
Proof. Please refer to the supplementary material for details.
To provide an intuitive perspective, we use two real-world graph datasets, namely Cora (2708 nodes) and Citeseer (3327 nodes), as examples. We calculate the number of  $k$ -hop starved nodes for  $k = 1, 2, 3, 4$ , based on their original graph topology. Fig. 1 shows the statistical results for the Cora140, Cora390, Citeseer120, and Citeseer370 datasets, where the suffix number represents the number of labeled nodes. From Fig. 1, we observe that as the value of  $k$  increases, the number of starved nodes decreases. This can be explained by the fact that as  $k$  increases, the nodes have more neighbors (from 1- to  $k$ -hop), and the possibility of having at least one labeled neighbor increases. Adopting a deeper GNN (larger  $k$ ) can thus mitigate the SS problem. However, it is important to consider that deeper GNNs result in higher computational consumption and may lead to poorer general
ization performance [1, 21, 30]. Furthermore, as shown in Fig. 1, even with a 4-layer GNN, there are still hundreds of 4-hop starved nodes in the Citesser120. Therefore, we believe that employing a deeper GNN is not the optimal solution to resolve the SS problem<sup>4</sup>.
In fact, Theorem 1 implicitly indicates how to alleviate the SS problem. Intuitively, a straightforward solution is to ameliorate the given adjacency matrix  $\mathbf{A}$  in order to reduce the number of starved nodes. This can be accomplished simply by reconstructing the connections between starved nodes and the labeled ones. Technically, we can achieve this by adding a regularization adjacency matrix  $\mathbf{B}$  to  $\mathbf{A}$ :
$$
\tilde {\mathbf {A}} = \mathbf {A} + \alpha \mathbf {B}, \tag {4}
$$
where  $\widetilde{\mathbf{A}}$  is the refined adjacency matrix,  $\mathbf{B}$  models the recovered affinities between staved nodes and labeled ones, and  $\alpha$  is a balanced parameter that controls the contribution of  $\mathbf{A}$  and  $\mathbf{B}$ . According to Theorem 1, we can identify the starved nodes based on  $\mathbf{A}$  and replenish the missed supervision through  $\mathbf{B}$ , thereby preventing  $\widetilde{\mathbf{A}}$  from being starved. Specifically, for each starved node  $V_{i}$ , we can search for at least one closest labeled node  $V_{l}$ , and restore at least one connection between  $V_{l}$  and  $V_{j}$  for  $j\in \{\mathcal{N}_{\kappa}(i)\cup i\}$  such that  $[\mathbb{1}_{\mathbb{R}^{+}}(\mathbf{B})]_{jl}^{\kappa} = 1$ , where  $\kappa$  can be arbitrarily chosen from the set of  $\{1,2,\ldots ,k\}$ . Although this strategy is effective, it may be computationally complex. Even with a small value of  $k$ , the computational cost of identifying  $k$ -hop starved nodes based on Theorem 1 is prohibitively expensive. For example, when identifying 2-hop starved nodes, the time complexity of computing  $\mathbf{A}^2$  alone reaches  $\mathcal{O}(n^3)$ . In the next section, we will propose a more efficient solution.
Figure 1: Illustration of  $k$ -hop starved nodes on different datasets. Obviously, the number of  $k$ -hop starved nodes decreases as the value of  $k$  increases.
is shown in Fig. 1, even with a 4-layer GNN, there is Citesser120. Therefore, we believe that employing a layer the SS problem  ${}^{4}$  .
$$
\mathbf {A} + \alpha \mathbf {B}, \tag {4}
$$
Is the recovered affinities between staved nodes and  $k$ -top controls the contribution of  $\mathbf{A}$  and  $\mathbf{B}$ . According to B, based on  $\mathbf{A}$  and replenish the missed supervision is served. Specifically, for each starved node  $V_{i}$ , we can find and restore at least one connection between  $V_{l}$  and  $V_{k-1}$ , where  $\kappa$  can be arbitrarily chosen from the set of all  $k$ , it may be computationally complex. Even with a non-valid  $k$ -hop starved nodes based on Theorem 1 is non-valid 2-hop starved nodes, the time complexity of this section, we will propose a more efficient solution.
Note that, for a  $k$ -layer GNN, a  $k$ -hop starved node also qualifies as a  $(k - 1)$ -hop starved node.
<sup>4</sup>We discuss this further in the supplementary material.
3.2 CUR Decomposition Makes Better Solution
Inspired by CUR matrix decomposition [2, 3], we propose an efficient alternative approach to identify the starved nodes. We first present the definition of CUR matrix decomposition.
Definition 4 (CUR Decomposition [2]). Given  $\mathbf{Q} \in \mathbb{R}^{n \times m}$  of rank  $\rho = \operatorname{rank}(\mathbf{Q})$ , rank parameter  $k < \rho$ , and accuracy parameter  $0 < \varepsilon < 1$ , construct column matrix  $\mathbf{C} \in \mathbb{R}^{n \times c}$  with  $c$  columns from  $\mathbf{Q}$ , row matrix  $\mathbf{R} \in \mathbb{R}^{r \times m}$  with  $r$  rows from  $\mathbf{Q}$ , and intersection matrix  $\mathbf{U} \in \mathbb{R}^{c \times r}$  with  $c, r$ , and  $\operatorname{rank}(\mathbf{U})$  being as small as possible, in order to reconstruct  $\mathbf{Q}$  within relative-error:
$$
\left\| \mathbf {Q} - \mathbf {C U R} \right\| _ {F} ^ {2} \leq (1 + \varepsilon) \left\| \mathbf {Q} - \mathbf {Q} _ {k} \right\| _ {F} ^ {2}. \tag {5}
$$
Here,  $\mathbf{Q}_k = \mathbf{U}_k\pmb{\Sigma}_k\mathbf{V}_k^T \in \mathbb{R}^{n\times m}$  is the best rank  $k$  matrix obtained via the singular value decomposition (SVD) of  $\mathbf{Q}$ .
With the definition of CUR decomposition, we can find a more efficient solution to identify the starved nodes. The following theorem demonstrates how we can accomplish this goal.
Theorem 2. Given a sparse adjacency matrix  $\mathbf{A} \in \mathbb{R}^{n \times n}$  with self-connections generated on graph  $\mathcal{G}(\mathcal{V},\mathbf{X})$ , construct  $\mathbf{C} = \mathbf{A}[:,\mathrm{col\_mask}] \in \mathbb{R}^{n \times c}$ , where  $\mathrm{col\_mask} \in \{0,1\}^n$  contains only  $c$  positive values corresponding to  $c$  labeled nodes, and  $\mathbf{R} = \mathbf{A}[row\_mask,:] \in \mathbb{R}^{r \times n}$  with row\_mask =  $\mathbb{1}_{\mathbb{R}^{-}}(\mathbf{C}\mathbb{1}_c) \in \{0,1\}^n$ . Then, (a)  $\mathbf{U} = \mathbf{A}[row\_mask,col\_mask] = \mathbf{0} \in \mathbb{R}^{r \times c}$ , where  $\mathbf{0}$  is a zero matrix, (b) the set of 1-hop starved nodes Set1(r) =  $\{V_i | i \in RM_+\}$ , where  $RM_+ \in \mathbb{N}^r$  indicates the set of indexes of positive elements from row\_mask, and (c) for each  $i \in RM_+$ ,  $V_i$  is a 2-hop starved node if, for  $\forall j$  satisfying  $[\mathbb{1}_{\mathbb{R}^+}(\mathbf{R})]_{ij} = 1$ ,  $j \in RM_+$ .
Theorem 2 provides a more efficient alternative for identifying  $k$ -hop starved nodes for  $k \in \{1, 2\}$ . In fact, the column matrix  $\mathbf{C}$  models the relationships between all nodes and  $c$  labeled nodes, the row matrix  $\mathbf{R}$  models the affinities between  $r$  1-hop starved nodes and the whole nodes, and the intersection matrix  $\mathbf{U}$  models the strength of connections between  $r$  1-hop starved nodes and  $c$  labeled nodes. Theorem 2 states that  $\mathbf{U} = \mathbf{0}$ , indicating that there are no connections between the starved nodes and the labeled ones. Based on this observation, we propose a simpler approach to reduce the number of starved nodes. Specifically, we rebuild the intersection matrix  $\mathbf{U}$  to ensure that the reconstructed  $\widetilde{\mathbf{U}} \neq \mathbf{0}^5$ . Consequently, Eq. (4) can be rewritten as:
$$
\widetilde {\mathbf {A}} = \mathbf {A} + \alpha \mathbf {B} = \mathbf {A} + \alpha \Gamma (\widetilde {\mathbf {U}}, n), \tag {6}
$$
where function  $\Gamma (\widetilde{\mathbf{U}},n)$  extends the matrix  $\widetilde{\mathbf{U}}\in \mathbb{R}^{r\times c}$  to an  $n\times n$  matrix by padding  $n - r$  rows of zeros and  $n - c$  columns of zeros in the corresponding positions.
The question now turns to how to reconstruct the intersection matrix  $\widetilde{\mathbf{U}}$ . For simplicity, we directly adopt the same strategy used in constructing  $\mathbf{A}$ . Specifically, for each row  $i$  of  $\widetilde{\mathbf{U}}$ , we establish a connection between  $V_{i}$  and  $V_{j}$  for  $\forall j \in \mathbb{C}\mathsf{M}_{+}$ , where  $\mathbb{C}\mathsf{M}_{+} \in \mathbb{N}^{c}$  represents the set of indexes of positive elements from col_mask. We then assign weights to these connections based on their distance. Note that, if we ensure each row of  $\widetilde{\mathbf{U}}$  has at least one weight greater than 0, there will be no  $\kappa$ -hop starved nodes for  $\forall \kappa > 1$ . This means that we do not need to feed the  $k$ -hop starved nodes satiated, simply feeding  $\kappa$ -hop ones for  $\forall \kappa < k$  makes  $k$ -hop starved nodes cease to exist. In addition, compared with the time complexity of  $\mathcal{O}(dn^{2})$  to construct  $\mathbf{A}$ , the time complexity of reconstructing  $\widetilde{\mathbf{U}}$  is  $\mathcal{O}(drc)$ . The additional computational burden is relatively negligible since  $c \ll n$ .
3.3 Weight Contribution Rate Decay
Directly replenishing the additional supervision encoded in  $\widetilde{\mathbf{U}}$  by Eq. (6) may cause a potential issue we refer to as weight contribution rate decay. To understand this issue, we consider the  $j$ -th column