Title: Latent Graph Inference with Limited Supervision
Venue: NeurIPS 2023
Accepted: yes
Keywords: Latent Graph Inference

CUR Matrix Decomposition

Graph Neural Networks
Abstract: Latent graph inference (LGI) aims to jointly learn the underlying graph structure and node representations from data features. However, existing LGI methods commonly suffer from the issue of supervision starvation, where massive edge weights are learned without semantic supervision and do not contribute to the training loss. Consequently, these supervision-starved weights, which determine the predictions of testing samples, cannot be semantically optimal, resulting in poor generalization. In this paper, we observe that this issue is actually caused by the graph sparsification operation, which severely destroys the important connections established between pivotal nodes and labeled ones. To address this, we propose to restore the corrupted affinities and replenish the missed supervision for better LGI. The key challenge then lies in identifying the critical nodes and recovering the corrupted affinities. We begin by defining the pivotal nodes as k-hop starved nodes, which can be identified based on a given adjacency matrix. Considering the high computational burden, we further present a more efficient alternative inspired by CUR matrix decomposition. Subsequently, we eliminate the starved nodes by reconstructing the destroyed connections. Extensive experiments on representative benchmarks demonstrate that reducing the starved nodes consistently improves the performance of state-of-the-art LGI methods, especially under extremely limited supervision (6.12% improvement on Pubmed with a labeling rate of only 0.3%).

Latent Graph Inference with Limited Supervision

Abstract

Latent graph inference (LGI) aims to jointly learn the underlying graph structure and node representations from data features. However, existing LGI methods commonly suffer from the issue of supervision starvation , where massive edge weights are learned without semantic supervision and do not contribute to the training loss. Consequently, these supervision-starved weights, which may determine the predictions of testing samples, cannot be semantically optimal, resulting in poor generalization. In this paper, we observe that this issue is actually caused by the graph sparsification operation, which severely destroys the important connections established between pivotal nodes and labeled ones. To address this, we propose to restore the corrupted affinities and replenish the missed supervision for better LGI . The key challenge then lies in identifying the critical nodes and recovering the corrupted affinities. We begin by defining the pivotal nodes as k -hop starved nodes , which can be identified based on a given adjacency matrix. Considering the high computational burden, we further present a more efficient alternative inspired by CUR matrix decomposition . Subsequently, we eliminate the starved nodes by reconstructing the destroyed connections. Extensive experiments on representative benchmarks demonstrate that reducing the starved nodes consistently improves the performance of state-of-the-art LGI methods, especially under extremely limited supervision (6.12% improvement on Pubmed with a labeling rate of only 0.3%).

1 Introduction

Graph neural networks (GNNs) [8,12,23,33] have recently received considerable attention due to their strong ability to handle complex graph-structured data. GNNs consider each data sample as a node and model the affinities between nodes as the weights of edges. The edges as a whole constitute the graph structure or topology of the data. By integrating the graph topology into the training process of representation learning, GNNs have achieved remarkable performance across a wide range of tasks, such as classification [19, 37], clustering [31, 40], retrieval [5, 43], and recognition [35, 44].

Although effective, existing GNNs typically require a prior graph to learn node representations, which poses a major challenge when encountering incomplete or even missing graphs. This limitation has spurred the development of latent graph inference (LGI) [7, 10, 17, 22, 32], also known as graph structure learning [9, 24, 39, 42]. LGI aims to jointly learn the underlying graph and discriminative node representations solely from the features of nodes in an end-to-end fashion. By adaptively learning the graph topology, LGI models are empowered with great ability to remove noise and capture more complex structure of the data [13,26,27,47]. Consequently, LGI emerges as a promising research topic with a broad range of applications, such as point cloud segmentation [41], disease prediction [6], multi-view clustering [31], and brain connectome representation [18].

However, many LGI methods suffer from a so-called supervision starvation (SS) problem [10], where a number of edge weights are learned without any semantic supervision during the graph inference stage. Specifically, given a k -layer GNN, if a node and all of its predefined neighbors (from 1 - to k -hop) are unlabeled, the edge weights between this node and its neighbors will not contribute to the training loss and cannot be semantically optimal after training. This will lead to poor generalization performance since these under-trained weights are inevitably used to make predictions for testing samples. In this paper, we discover that the SS problem is actually caused by the graph sparsification operation, which severely destroys the important connections established between pivotal nodes and labeled ones. Based on this observation, we propose to restore the destroyed connections and replenish the missed supervision for better LGI .

Specifically, we first define the pivotal nodes as k -hop starved nodes . Then, the SS problem can be transformed into a more manageable task of eliminating starved nodes. We propose to identify the k -hop starved nodes based on the k -th power of a given adjacency matrix. After identification, we can diminish the starved nodes by incorporating a regularization adjacency matrix into the initial one. However, the above identification approach suffers from high computational complexity due to matrix multiplication. For example, given a 2 -layer GNN, identifying 2 -hop starved nodes requires at least O ( n 3 ) , where n denotes the number of nodes. To address this, we present a more efficient alternative solution inspired by CUR matrix decomposition [2, 3]. We find that with appropriate column and row selection, the U matrix obtained through CUR decomposition of the initial adjacency matrix is actually a zero matrix. Thus, we can eliminate the starved nodes by reconstructing the U matrix.

Although recovering the U matrix encourages more supervision, such strategy may cause a potential issue we call weight contribution rate decay . In other words, the weight contribution rate (WCR) of non-starved nodes decays as the number of starved nodes increases, resulting in a final latent graph that heavily relies on the regularization one. To tackle this issue, we propose two simple strategies, i.e. , decreasing the WCR of starved nodes and increasing the WCR of non-starved nodes (see Sec. 3.3 for details). The main contributions of this paper can be summarized as follows:

We identify that graph sparsification is the main cause of the supervision starvation (SS) problem in latent graph inference (LGI). Therefore, we propose to restore the affinities corrupted by the sparsification operation to replenish the missed supervision for better LGI.

By defining k -hop starved nodes, we transform the SS issue into a more manageable task of removing starved nodes. Inspired by CUR decomposition, we propose a simple yet effective solution to determine the starved nodes and diminish them using a regularization graph.

The proposed approach is model-agnostic and can be seamlessly integrated into various LGI models. Extensive experiments demonstrate that eliminating the starved nodes consistently enhances the state-of-the-art LGI methods, particularly under extremely limited supervision.

2 Preliminaries

2.1 Notations

Let A i : , A : j , A ij , and A k denote the i -th row, the j -th column, the element at the i -th row and j -th column, and the k -th power of the matrix A , respectively. Let 1 n represent an n -dimensional column vector with all elements being 1 . Let ✶ R + ( A ) be an element-wise indicator function that sets A ij to 1 if it is positive, and 0 otherwise. Conversely, ✶ R -( A ) sets A ij to 0 if it is positive, and 1 otherwise. 2.2 Latent Graph Inference

Definition 1 (Latent Graph Inference) . Given a graph G ( V , X ) containing n nodes V = { V 1 , . . . , V n } and a feature matrix X ∈ R n × d with each row X i : ∈ R d representing the d -dimensional attributes of node V i , latent graph inference (LGI) aims to simultaneously learn the underlying graph topology encoded by an adjacency matrix A ∈ R n × n and the discriminative d ′ -dimensional node representations Z ∈ R n × d ′ based on X , where the learned A and Z are jointly optimal for certain downstream tasks T given a specific loss function L .

In general, an LGI model mainly consists of a latent graph generator P Φ ( X ) : R n × d → R n × n that generates an adjacency matrix A based on the node features X , and a node encoder F Θ ( X , A ) :

R n × d → R n × d ′ that learns discriminative node representations Z based on X and the learned A . In practice, F Θ is typically implemented using a k -layer GNN, while P Φ can be realized through different strategies such as full parameterization [17, 36], MLP [10,25], or attentive network [4, 15]. In this paper, we adopt the most common settings from existing LGI literature [4, 10, 13, 25, 46], considering T as the semi-supervised node classification task and L as the cross-entropy loss.

2.3 Supervision Starvation

To illustrate the supervision starvation problem [10], we consider a general LGI model M consisting of a latent graph generator P Φ and a node encoder F Θ . For simplicity, we ignore the activation function and assume that F Θ is implemented using a 1 -layer GNN, i.e. , F Θ = GNN 1 ( X , A ; Θ ) , where A = P Φ ( X ) . For each node X i : , the corresponding node representation Z i : learned by the model M can be expressed as:

where Y L represents the set of indexes of labeled nodes and |C| denotes the size of label set. For ∀ i ∈ Y L , j ∈ Ω , A ij is optimized via backpropagation under the supervision of label Y i : . For ∀ i / ∈ Y L , however, if j / ∈ Y L for ∀ j ∈ Ω , A ij will receive no supervision from any label and, as a result, cannot be semantically optimal after training. Consequently, the learning models exhibit poor generalization as the predictions of testing nodes inevitably rely on these supervision-starved weights. This phenomenon is referred to as supervision starvation (SS), where many edge weights are learned without any label supervision. It is easy to infer that this issue also persists in a k -layer GNN.

We may ask why this problem arises? In fact, the SS problem is caused by a common and necessary post-processing operation known as graph sparsification, which is employed in the majority of LGI methods [10,25,38,46] to generate a sparse latent graph. To be more specific, graph sparsification adjusts the initial dense graph to a sparse one through the following procedure:

where topκ ( A i : ) denotes the set of the top κ values in A i : . After this sparsification operation, a significant number of edge weights are directly erased, including the crucial connections established between pivotal nodes and labeled nodes. Another question that may arise is: how many important nodes or connections suffer from this problem? We delve into this question in the next section.

3 Methodology

3.1 Identification & Elimination of Starved Nodes

We first introduce the definitions of k -hop starved node and the corresponding k -hop starved weight.

Definition 2 ( k -hop Starved Node) . Given a graph G ( V , X ) consisting of n nodes V = { V 1 , . . . , V n } and the corresponding node features X , for a k -layer graph neural network GNN k ( X ; Θ ) with network parameters Θ , the unlabeled node V i is a k -hop starved node if, for ∀ κ ∈ { 1 , . . . , k } , ∀ V j ∈ N κ ( i ) , where N κ ( i ) is the set of κ -hop neighbors of V i , V j is unlabeled. Specifically, 0 -hop starved nodes are defined as the unlabeled nodes 2 .

Based on Definition 2, we can define the k -hop starved weight as follows.

Definition 3 ( k -hop Starved Weight) . If an edge exists between nodes V i and V j , the associated edge weight A ij is a k -hop starved weight if both of the V i and V j qualifies as ( k -1) -hop starved nodes 3 .

According to the definition presented, it is evident that k -hop starved weights are precisely the ones that receive no semantic supervision from labels. As a result, to address the SS problem, our focus should be on reducing the presence of k -hop starved nodes. The following theorem illustrates how we can identify such nodes based on a given initial adjacency matrix.

Theorem 1. Given a sparse adjacency matrix A ∈ R n × n with self-connections generated on graph G ( V , X ) by a latent graph inference model with a k -layer graph neural network GNN k ( X ; Θ ) , the node V i is a k -hop starved node, if ∃ j ∈ { 1 , . . . , n } , such that [ ✶ R + ( A )] k ij = 1 , and for ∀ j ∈ { j | [ ✶ R + ( A )] ij = 1 ∪ [ ✶ R + ( A )] 2 ij = 1 ∪ . . . ∪ [ ✶ R + ( A )] k ij = 1 } , V j is unlabeled.

Proof. Please refer to the supplementary material for details.

To provide an intuitive perspective, we use two real-world graph datasets, namely Cora ( 2708 nodes) and Citeseer ( 3327 nodes), as examples. We calculate the number of k -hop starved nodes for k = 1 , 2 , 3 , 4 , based on their original graph topology. Fig. 1 shows the statistical results for the Cora140, Cora390, Citesser120, and Citeseer370 datasets, where the suffix number represents the number of labeled nodes. From Fig. 1, we observe that as the value of k increases, the number of starved nodes decreases. This can be explained by the fact that as k increases, the nodes have more neighbors (from 1 - to k -hop), and the possibility of having at least one labeled neighbor increases. Adopting a deeper GNN (larger k ) can thus mitigate the SS problem. However, it is important to consider that deeper GNNs result in higher computational consumption and may lead to poorer general-

1-hop

2-hop

3-hop

4-hop

K-hop

0

500

1000

1500

2000

2500

3000

Number of Starved Nodes

Cora140

Cora390

Citeseer120

Citeseer370

ization performance [1, 21, 30]. Furthermore, as shown in Fig. 1, even with a 4 -layer GNN, there are still hundreds of 4 -hop starved nodes in the Citesser120. Therefore, we believe that employing a deeper GNN is not the optimal solution to resolve the SS problem 4 .

In fact, Theorem 1 implicitly indicates how to alleviate the SS problem. Intuitively, a straightforward solution is to ameliorate the given adjacency matrix A in order to reduce the number of starved nodes. This can be accomplished simply by reconstructing the connections between starved nodes and the labeled ones. Technically, we can achieve this by adding a regularization adjacency matrix B to A :

where ˜ A is the refined adjacency matrix, B models the recovered affinities between staved nodes and labeled ones, and α is a balanced parameter that controls the contribution of A and B . According to Theorem 1, we can identify the starved nodes based on A and replenish the missed supervision through B , thereby preventing ˜ A from being starved. Specifically, for each starved node V i , we can search for at least one closest labeled node V l , and restore at least one connection between V l and V j for j ∈ {N κ ( i ) ∪ i } such that [ ✶ R + ( B )] κ jl = 1 , where κ can be arbitrarily chosen from the set of { 1 , 2 , . . . , k } . Although this strategy is effective, it may be computationally complex. Even with a small value of k , the computational cost of identifying k -hop starved nodes based on Theorem 1 is prohibitively expensive. For example, when identifying 2 -hop starved nodes, the time complexity of computing A 2 alone reaches O ( n 3 ) . In the next section, we will propose a more efficient solution.

3.2 CUR Decomposition Makes Better Solution

Inspired by CUR matrix decomposition [2,3], we propose an efficient alternative approach to identify the starved nodes. We first present the definition of CUR matrix decomposition.

Definition 4 (CUR Decomposition [2]) . Given Q ∈ R n × m of rank ρ = rank ( Q ) , rank parameter k < ρ , and accuracy parameter 0 < ε < 1 , construct column matrix C ∈ R n × c with c columns from Q , row matrix R ∈ R r × m with r rows from Q , and intersection matrix U ∈ R c × r with c , r , and rank ( U ) being as small as possible, in oder to reconstruct Q within relative-error:

Here, Q k = U k Σ k V T k ∈ R n × m is the best rank k matrix obtained via the singular value decomposition (SVD) of Q .

With the definition of CUR decomposition, we can find a more efficient solution to identify the starved nodes. The following theorem demonstrates how we can accomplish this goal.

Theorem 2. Given a sparse adjacency matrix A ∈ R n × n with self-connections generated on graph G ( V , X ) , construct C = A [: , col _ mask ] ∈ R n × c , where col _ mask ∈ { 0 , 1 } n contains only c positive values corresponding to c labeled nodes, and R = A [ row _ mask, :] ∈ R r × n with row _ mask = ✶ R -( C ✶ c ) ∈ { 0 , 1 } n . Then, (a) U = A [ row _ mask,col _ mask ] = 0 ∈ R r × c , where 0 is a zero matrix, (b) the set of 1 -hop starved nodes Set 1 ( r ) = { V i | i ∈ RM + } , where RM + ∈ N r indicates the set of indexes of positive elements from row _ mask , and (c) for each i ∈ RM + , V i is a 2 -hop starved node if, for ∀ j satisfying [ R + ( R )] ij = 1 , j ∈ RM + .

✶ Proof. Please refer to the supplementary material for details.

̸

Theorem 2 provides a more efficient alternative for identifying k -hop starved nodes for k ∈ { 1 , 2 } . In fact, the column matrix C models the relationships between all nodes and c labeled nodes, the row matrix R models the affinities between r 1 -hop starved nodes and the whole nodes, and the intersection matrix U models the strength of connections between r 1 -hop starved nodes and c labeled nodes. Theorem 2 states that U = 0 , indicting that there are no connections between the starved nodes and the labeled ones. Based on this observation, we propose a simpler approach to reduce the number of starved nodes. Specifically, we rebuild the intersection matrix U to ensure that the reconstructed ˜ U = 0 5 . Consequently, Eq. (4) can be rewritten as:

where function Γ( ˜ U , n ) extends the matrix ˜ U ∈ R r × c to an n × n matrix by padding n -r rows of zeros and n -c columns of zeros in the corresponding positions.

The question now turns to how to reconstruct the intersection matrix ˜ U . For simplicity, we directly adopt the same strategy used in constructing A . Specifically, for each row i of ˜ U , we establish a connection between V i and V j for ∀ j ∈ CM + , where CM + ∈ N c represents the set of indexes of positive elements from col _ mask . We then assign weights to these connections based on their distance. Note that, if we ensure each row of ˜ U has at least one weight greater than 0, there will be no κ -hop starved nodes for ∀ κ > 1 . This means that we do not need to feed the k -hop starved nodes satiated, simply feeding κ -hop ones for ∀ κ < k makes k -hop starved nodes cease to exist. In addition, compared with the time complexity of O ( dn 2 ) to construct A , the time complexity of reconstructing ˜ U is O ( drc ) . The additional computational burden is relatively negligible since c ≪ n .

3.3 Weight Contribution Rate Decay

Directly replenishing the additional supervision encoded in ˜ U by Eq. (6) may cause a potential issue we refer to as weight contribution rate decay . To understand this issue, we consider the j -th column

vector ˜ A : j of ˜ A for j ∈ CM + . We define the weight contribution rates (WCR) of non-starved nodes ρ -and the WCR of starved nodes ρ + as:

From Eq. (7), we observe that the WCR of non-starved nodes ρ -decays as the number of starved nodes increases ( i.e. , ρ + increases). This means that ρ -becomes negligible if there are numerous starved nodes. As a result, the j -th column vector ˜ A : j of ˜ A for j ∈ CM + heavily relies on the j -th column vector ˜ U : j of the reconstructed intersection matrix ˜ U . This outcome is not desirable since the regularization imposed should not dominate a significant portion of the final adjacency matrix. Recalling our initial objective of properly refining the original latent graph to replenish the missed supervision, we will design two simple strategies to relieve this issue.

Decrease ρ + . On the one hand, we can decrease the WCR of starved nodes ρ + by selecting only τ ( τ < c ) labeled nodes as the supplementary adjacent points for each 1 -hop starved node. This results in a sparse intersection matrix ̂ U . For this strategy, we present the following proposition:

Proposition 1. Suppose that we randomly select τ out of c labeled nodes as the supplementary adjacent points for each 1 -hop starved node. If there are r 1 -hop starved nodes, then for ∀ j ∈

CM + , .

Proof. Please refer to the supplementary material for details.

Increase ρ -. On the other hand, we can improve the WCR of non-starved nodes ρ -by magnifying the weights of non-starved nodes. Specifically, we construct an additional regularization matrix Q ∈ R n × c , where its c columns correspond to the c labeled nodes and Q ij = 0 for ∀ i ∈ RM + . For ∀ i / ∈ RM + , we establish connections between the node V i and τ ( τ < c ) labeled nodes V j , and assign the corresponding weights Q ij using a similar strategy as in constructing ̂ U . By adding the additional regularization matrix Q , the refined adjacency matrix ̂ A can be expressed as:

Similarly, for this strategy, we have the following proposition:

Proposition 2. Suppose that we randomly select τ out of c labeled nodes as the supplementary adjacent points for each non-starved node. If there are r 1 -hop starved nodes, then for ∀ j ∈ CM + , we

Proof. Please refer to the supplementary material for details.

3.4 End-to-End Training

Note that the proposed approach is model-agnostic and can be seamlessly integrated into existing LGI models. In Sec. 4, we will apply our design to state-of-the-art LGI methods [4, 10, 13, 46] and compare its performance with the original approach. Therefore, we follow these methods and implement the node encoder F Θ using a 2 -layer GNN. Then, we calculate the cross-entropy loss L ce between the true labels Y and the predictions Z , as well as a graph regularization loss L reg on ̂ A :

where γ is a balanced parameter and Z = F Θ ( X , ̂ A ) = softmax ( ̂ A σ ( ̂ AXΘ 0 ) Θ 1 ) . According to different LGI methods, the graph regularization loss L reg can be different, such as Dirichlet energy [4] and self-supervision [10] (see supplementary material for more details). For fairness, in the experiments, we will adopt the same graph regularization loss as the comparison methods.

4 Experiments

4.1 Experimental Settings

Baselines. As mentioned earlier, the proposed regularization module can be easily integrated into most existing LGI methods. To evaluate its effectiveness, we select representative LGI methods as baselines, including IDGL [4], GRCN [46], SLAPS [10], and LCGS [13]. We also consider two additional baselines marked as GCN+KNN [10,19] and GCN&KNN [4,19]. GCN+KNN is a two-step method that first constructs a KNN graph based on feature similarities and then feeds the pre-constructed graph to a GCN for training. GCN&KNN is an end-to-end method that learns the latent graph and network parameters simultaneously. For methods that require a prior graph, we use the same KNN graph as in GCN+KNN. For GCN&KNN, we simply adopt the Dirichlet energy [4] as the graph regularization loss.

Datasets. Following the common settings of existing LGI methods [4, 10, 13, 46], we conduct experiments on four well-known benchmarks: Cora, Citeseer, Pubmed [19,34], and ogbn-arxiv [14]. For detailed dataset statistics, please refer to the supplementary material. For all datasets, we only provide the original node features for training. To test the performance under different labeling rates, for the Cora and Citeseer datasets, we add half of the validation samples to the training sets, resulting in Cora390 and Citeseer370, where the suffix number represents the total number of labeled nodes.

Implementation. We compare the above baselines with their corresponding CUR extension versions. Specifically, we consider two CUR extensions termed M_U and M_R (M refers to the baseline), where the former adopts the sparse intersection matrix ̂ U as the regularization, and the latter combines ̂ U and Q together (see Sec. 3.3 for details). In experiments, we practically select the τ closest labeled nodes as supplementary adjacent points for each row of ̂ U and Q . For postprocessing operations on the graph, such as symmetrization and normalization [4,10], we follow the baselines and adopt the same operations for fairness. We select the values of τ and α from the sets { 10 , 15 , 20 , 25 , 30 , 50 } and { 0 . 01 , 0 . 1 , 1 . 0 , 10 , 50 , 100 } , respectively. For other hyperparameters such as learning rate and weight decay, we follow the baselines and use the same settings. For each method, we record the best testing performance and report the average accuracy of five independent experiments, along with the corresponding standard deviation.

4.2 Comparison Results

Table 1 presents the comparison results on all used datasets. It is evident that our proposed CUR extensions consistently outperform the corresponding baselines, demonstrating the effectiveness of eliminating starved nodes. When considering the labeling rates of different datasets listed in the third row of Table 1, we observe that the lower the labeling rate, the greater the performance

0

250

500

750

1000

1250

1500

1750

2000

Epoch

0.0

0.2

0.4

0.6

0.8

1.0

Loss Value

0.40

0.45

0.50

0.55

0.60

0.65

0.70

0.75

Train Loss: GCN+KNN

Train Loss: GCN+KNN_U (ours)

Train Loss: GCN+KNN_R (ours)

Test Acc: GCN+KNN

Test Acc: GCN+KNN_U (ours)

Test Acc: GCN+KNN_R (ours)

(a) GCN+KNN

0

50

100

150

200

250

300

Epoch

0.2

0.4

0.6

0.8

1.0

1.2

Loss Value

0.2

0.3

0.4

0.5

0.6

0.7

Train Loss: GRCN

Train Loss: GRCN_U (ours)

Train Loss: GRCN_R (ours)

Test Acc: GRCN

Test Acc: GRCN_U (ours)

Test Acc: GRCN_R (ours)

(b) GRCN

200

300

400

500

600

700

800

900

1000

Epoch

0.4

0.6

0.8

1.0

1.2

1.4

1.6

Loss Value

0.40

0.45

0.50

0.55

0.60

0.65

0.70

0.75

Train Loss: SLAPS

Train Loss: SLAPS_U (ours)

Train Loss: SLAPS_R (ours)

Test Acc: SLAPS

Test Acc: SLAPS_U (ours)

Test Acc: SLAPS_R (ours)

(c) SLAPS

improvement achieved by our methods. Notably, on the Pubmed dataset with an extremely low labeling rate of 0 . 30% , the accuracy of our proposed methods (M_R) increase by 6 . 12% , 3 . 58% , and 2 . 26% compared to the basic models (M) of GCN+KNN, GRCN, and SLAPS, respectively. This is because the lower the labeling rate, the more starved nodes exist in the dataset (see Fig. 1 for an example). Our proposed methods aim to restore the destroyed affinities between starved nodes and labeled nodes, enabling us to leverage the semantic supervision missed by baselines, thereby achieving superior performance with extremely limited supervision.

Fig. 2 displays the training loss and testing accuracy curves of GCN+KNN, GRCN, SLAPS, and their CUR extensions on the Pubmed dataset. We omit the curves for the first 200 epochs of SLAPS because, during this stage, SLAPS focuses solely on learning the latent graph through an additional self-supervision task without involving the classification task. From Fig. 2, we observe that our proposed CUR extensions achieve higher testing accuracy compared to their corresponding baselines. In comparison with the CUR extensions of GCN+KNN, the CUR extensions of GRCN and SLAPS exhibit relatively stable accuracy results as the training epoch increases. This stability can be attributed to GRCN and SLAPS jointly learning the latent graph and network parameters in an end-to-end manner, whereas GCN+KNN pre-constructs a KNN graph and maintains a fixed graph during training without optimization. For GRCN, our proposed CUR extensions demonstrate lower training loss and higher testing accuracy, further indicating their improved generalization by removing starved nodes. Additionally, we observe relatively unstable training loss for the CUR extensions of SLAPS. The potential reason is that SLAPS introduces an additional graph regularization loss through a self-supervision task, while its CUR extensions aim to reconstruct the destroyed connections in the latent graph. As a result, the training process for the self-supervision task can exhibit some instability.

4.3 Discussion

We would like to explore the question that why our proposed methods yield a slight improvement on the Cora and Citeseer datasets, while achieving a substantial improvement on the Pubmed dataset . As shown in Table 2, when τ = 10 , there are 100 and 299 2 -hop starved nodes on the Cora140 and Citeseer120 datasets, respectively. However, when τ = 20 , there are no 2 -hop starved nodes on the Cora140 and Citeseer120 datasets, and the number of 1 -hop starved nodes also sharply decreases. In this scenario, since the comparison baselines all utilize a 2 -layer GNN, they are unaffected by 2 -hop starved nodes and only minimally affected by 1 -hop starved nodes. Consequently, our methods result in only slight improvements over the baselines on the Cora140 and Citeseer120 datasets. On the other hand, when τ = 20 , there are still 12 , 843 2 -hop starved nodes present on the Pubmed dataset. Since we effectively eliminate these starved nodes without requiring additional graph convolutional layers, our methods can provide notable benefits on this dataset with an extremely low labeling rate.

4.4 Ablation Study

In this subsection, we aim to explore and answer the following questions.

How many starved nodes should be eliminated? The results shown in Table 1 indicate that eliminating all starved nodes contributes to the performance improvement of the baselines. It is important to understand the relationship between the number of starved nodes removed and the corresponding performance improvement of the baselines. To investigate this, we randomly remain 10% , 20% , 40% , 60% , 80% starved nodes and evaluate the performance of GCN+CNN, GRCN, and SLAPS accordingly. The results on the Pubmed dataset are summarized in Table 3. We find that, for GCN+KNN, a smaller number of starved nodes leads to higher testing accuracy. For GRCN and SLAPS, however, we need to check the number of starved nodes to obtain optimal performance.

How τ affects the performance. Table 4 shows that the selection of τ slightly differentiates the performance of our proposed methods. In general, a larger value of τ leads to a higher improvement in performance. However, when τ is set too large, such as 50 , the performance starts to degrade. This degradation occurs due to the introduction of incorrect labels when τ exceeds a certain threshold.

How α affects the performance. Table 5 presents the sensitivity of parameter α on the Pubmed dataset. It is observed that a relatively larger value of α , such as 10 or 50, leads to a significant improvement in performance. This finding further emphasizes the effectiveness of our proposed regularization methods in enhancing the performance of existing LGI models.

5 Related Work

Latent Graph Inference. Given only the node features of data, latent graph inference (LGI) aims to simultaneously learn the underlying graph structure and discriminative node representations from the features of data [10, 24, 39]. For example, Jiang et al. [15] propose to infer the graph structure by combining graph learning and graph convolution in a unified framework. Yang et al. [45] model the topology refinement as a label propagation process. Jin et al. [16] explore some intrinsic properties of the latent graph and propose a robust LGI framework to defend adversarial attacks on graphs. Though effective, these methods require a prior graph to guide the graph inference process. Controversially,

some methods have been proposed to directly infer an optimal graph from the data. For example, Franceschi et al. [11] regard the LGI problem as a bilevel program task and learn a discrete probability distribution on the edges of the latent graph. Norcliffe-Brown et al. [29] focus on visual question answering task and propose to learn an adjacency matrix from image objects so that each edge is conditioned on the questions. Fatemi et al. [10] propose a self-supervision guided LGI method, called SLAPS, which yields supplementary supervision from node features through an additional self-supervision task. Our method is totally different from SLAPS as we provide supplementary supervision directly from the true labels. More importantly, our method is model-agnostic and can be easily integrated into most existing LGI methods.

CUR Matrix Decomposition. The CUR decomposition [2, 3] of a matrix Q ∈ R n × m aims to find a column matrix C ∈ R n × c with a subset of c < m columns of Q , and a row matrix R ∈ R r × m with a subset of r < n rows of Q , as well as an intersection matrix U ∈ R c × r such that the matrix multiplication of CUR approximates Q . Unlike the SVD decomposition of Q , the CUR decomposition obtains actual columns and rows of Q , which makes it useful in various applications [20, 28]. In our method, instead of seeking an optimal matrix approximation, we employ CUR decomposition to identify and eliminate the starved nodes. The extracted intersection matrix U is then reconstructed and served as a regularization term to provide supplementary supervision for better latent graph inference. To the best of our knowledge, we are the first to introduce CUR matrix decomposition into the field of graph neural networks.

6 Conclusion

In this paper, we analyze the common problem of supervision starvation (SS) in existing latent graph inference (LGI) methods. Our analysis reveals that this problem arises due to the graph sparsification operation, which destroys numerous important connections between pivotal nodes and labeled ones. Building upon this observation, we propose to recover the corrupted connections and replenish the missed supervision for improved graph inference. To this end, we begin by defining k -hop starved nodes and transform the SS problem into a more manageable task of reducing starved nodes. Then, we present two simple yet effective solutions to identify the starved nodes, including a more efficient method inspired by CUR matrix decomposition. Subsequently, we eliminate the starved nodes by constructing and incorporating a regularization graph. In addition, we propose two straightforward strategies to tackle the potential issue known as weight contribution rate decay. Extensive experiments conducted on representative benchmarks demonstrate that our proposed methods consistently enhance the performance of state-of-the-art LGI models, particularly under extremely limited supervision.

7 Acknowledgments and Disclosure of Funding

We are very grateful to Bahare Fatemi for her valuable discussion of our work. We thank the anonymous NeurIPS reviewers for providing us with constructive suggestions to improve our paper. This material is based upon work supported by the Air Force Office of Scientific Research under award number FA9550-23-1-0290.

References

Uri Alon and Eran Yahav. On the bottleneck of graph neural networks and its practical implications. In 9th International Conference on Learning Representations , 2021.

Christos Boutsidis and David P. Woodruff. Optimal cur matrix decompositions. In Proceedings of the Forty-Sixth Annual ACM Symposium on Theory of Computing , page 353-362, 2014.

HanQin Cai, Keaton Hamm, Longxiu Huang, and Deanna Needell. Robust cur decomposition: Theory and imaging applications. SIAM Journal on Imaging Sciences , 14(4):1472-1503, 2021.

Yu Chen, Lingfei Wu, and Mohammed Zaki. Iterative deep graph learning for graph neural networks: Better and robust node embeddings. In Advances in Neural Information Processing Systems , pages 19314-19326, 2020.

Yudong Chen, Sen Wang, Jianglin Lu, Zhi Chen, Zheng Zhang, and Zi Huang. Local graph convolutional networks for cross-modal hashing. In Proceedings of the 29th ACM International Conference on Multimedia , pages 1921-1928, 2021.

Luca Cosmo, Anees Kazi, Seyed-Ahmad Ahmadi, Nassir Navab, and Michael Bronstein. Latentgraph learning for disease prediction. In Medical Image Computing and Computer Assisted Intervention , pages 643-653, 2020.

Haitz Sáez de Ocáriz Borde, Anees Kazi, Federico Barbero, and Pietro Lio. Latent graph inference using product manifolds. In The Eleventh International Conference on Learning Representations , 2023.

Federico Errica, Marco Podda, Davide Bacciu, and Alessio Micheli. A fair comparison of graph neural networks for graph classification. In International Conference on Learning Representations , 2020.

Bahare Fatemi, Sami Abu-El-Haija, Anton Tsitsulin, Mehran Kazemi, Dustin Zelle, Neslihan Bulut, Jonathan Halcrow, and Bryan Perozzi. Ugsl: A unified framework for benchmarking graph structure learning. In the 2nd Annual Workshop on Topology, Algebra, and Geometry in Machine Learning (TAG-ML) at the 40 th International Conference on Machine Learning , 2023.

Bahare Fatemi, Layla El Asri, and Seyed Mehran Kazemi. Slaps: Self-supervision improves structure learning for graph neural networks. In Advances in Neural Information Processing Systems , pages 22667-22681, 2021.

Luca Franceschi, Mathias Niepert, Massimiliano Pontil, and Xiao He. Learning discrete structures for graph neural networks. In Proceedings of the 36th International Conference on Machine Learning , pages 1972-1982, 2019.

Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. In Advances in Neural Information Processing Systems , 2017.

Minyang Hu, Hong Chang, Bingpeng Ma, and Shiguang Shan. Learning continuous graph structure with bilevel programming for graph neural networks. In Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence , pages 3057-3063, 2022.

Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: Datasets for machine learning on graphs. In Advances in Neural Information Processing Systems , pages 22118-22133, 2020.

Bo Jiang, Ziyan Zhang, Doudou Lin, Jin Tang, and Bin Luo. Semi-supervised learning with graph learning-convolutional networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2019.

Wei Jin, Yao Ma, Xiaorui Liu, Xianfeng Tang, Suhang Wang, and Jiliang Tang. Graph structure learning for robust graph neural networks. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , page 66-74, 2020.

Anees Kazi, Luca Cosmo, Seyed-Ahmad Ahmadi, Nassir Navab, and Michael M. Bronstein. Differentiable graph module (dgm) for graph convolutional networks. IEEE Transactions on Pattern Analysis and Machine Intelligence , 45(2):1606-1617, 2023.

Byung-Hoon Kim, Jong Chul Ye, and Jae-Jin Kim. Learning dynamic graph representation of brain connectome with spatio-temporal attention. In Advances in Neural Information Processing Systems , pages 4314-4327, 2021.

Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In International Conference on Learning Representations , 2017.

Changsheng Li, Xiangfeng Wang, Weishan Dong, Junchi Yan, Qingshan Liu, and Hongyuan Zha. Joint active learning with feature selection via cur matrix decomposition. IEEE Transactions on Pattern Analysis and Machine Intelligence , 41(6):1382-1396, 2019.

Qimai Li, Zhichao Han, and Xiao-ming Wu. Deeper insights into graph convolutional networks for semi-supervised learning. In Proceedings of the AAAI Conference on Artificial Intelligence , 2018.

Ruoyu Li, Sheng Wang, Feiyun Zhu, and Junzhou Huang. Adaptive graph convolutional neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence , 2018.

Xiaorui Liu, Jiayuan Ding, Wei Jin, Han Xu, Yao Ma, Zitao Liu, and Jiliang Tang. Graph neural networks with adaptive residual. In Advances in Neural Information Processing Systems , pages 9720-9733, 2021.

Xin Liu, Jiayang Cheng, Yangqiu Song, and Xin Jiang. Boosting graph structure learning with dummy nodes. In Proceedings of the 39th International Conference on Machine Learning , pages 13704-13716, 2022.

Yixin Liu, Yu Zheng, Daokun Zhang, Hongxu Chen, Hao Peng, and Shirui Pan. Towards unsupervised deep graph structure learning. In Proceedings of the ACM Web Conference 2022 , page 1392-1403, 2022.

Jianglin Lu, Jingxu Lin, Zhihui Lai, Hailing Wang, and Jie Zhou. Target redirected regression with dynamic neighborhood structure. Information Sciences , 544:564-584, 2021.

Jianglin Lu, Hailing Wang, Jie Zhou, Yudong Chen, Zhihui Lai, and Qinghua Hu. Low-rank adaptive graph embedding for unsupervised feature extraction. Pattern Recognition , 113:107758, 2021.

Michael W. Mahoney and Petros Drineas. Cur matrix decompositions for improved data analysis. Proceedings of the National Academy of Sciences , 106(3):697-702, 2009.

Will Norcliffe-Brown, Stathis Vafeias, and Sarah Parisot. Learning conditioned graph structures for interpretable visual question answering. In Advances in Neural Information Processing Systems , 2018.

Kenta Oono and Taiji Suzuki. Graph neural networks exponentially lose expressive power for node classification. In 8th International Conference on Learning Representations , 2020.

ErLin Pan and Zhao Kang. Multi-view contrastive graph clustering. In Advances in Neural Information Processing Systems , pages 2148-2159, 2021.

Xingyue Pu, Tianyue Cao, Xiaoyun Zhang, Xiaowen Dong, and Siheng Chen. Learning to learn graph topologies. In Advances in Neural Information Processing Systems , pages 4249-4262, 2021.

Víctor Garcia Satorras, Emiel Hoogeboom, and Max Welling. E(n) equivariant graph neural networks. In Proceedings of the 38th International Conference on Machine Learning , pages 9323-9332, 2021.

Prithviraj Sen, Galileo Namata, Mustafa Bilgic, Lise Getoor, Brian Galligher, and Tina EliassiRad. Collective classification in network data. AI Magazine , 29(3):93, 2008.

Lei Shi, Yifan Zhang, Jian Cheng, and Hanqing Lu. Skeleton-based action recognition with directed graph neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2019.

Qingyun Sun, Jianxin Li, Hao Peng, Jia Wu, Xingcheng Fu, Cheng Ji, and Philip S Yu. Graph structure learning with variational information bottleneck. In Proceedings of the AAAI Conference on Artificial Intelligence , pages 4165-4174, 2022.

Petar Velickovic, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, Yoshua Bengio, et al. Graph attention networks. stat , 1050(20):10-48550, 2017.

Ruijia Wang, Shuai Mou, Xiao Wang, Wanpeng Xiao, Qi Ju, Chuan Shi, and Xing Xie. Graph structure estimation neural networks. In Proceedings of the Web Conference 2021 , page 342-353, 2021.

Yaohua Wang, Fangyi Zhang, Ming Lin, Senzhang Wang, Xiuyu Sun, and Rong Jin. Robust graph structure learning over images via multiple statistical tests. In Advances in Neural Information Processing Systems , 2022.

Yaohua Wang, Yaobin Zhang, Fangyi Zhang, Senzhang Wang, Ming Lin, YuQi Zhang, and Xiuyu Sun. Ada-NETS: Face clustering via adaptive neighbour discovery in the structure space. In International Conference on Learning Representations , 2022.

Yue Wang, Yongbin Sun, Ziwei Liu, Sanjay E Sarma, Michael M Bronstein, and Justin M Solomon. Dynamic graph cnn for learning on point clouds. Acm Transactions On Graphics (tog) , 38(5):1-12, 2019.

Qitian Wu, Wentao Zhao, Zenan Li, David Wipf, and Junchi Yan. Nodeformer: A scalable graph structure learning transformer for node classification. In Advances in Neural Information Processing Systems , 2022.

Ruiqing Xu, Chao Li, Junchi Yan, Cheng Deng, and Xianglong Liu. Graph convolutional network hashing for cross-modal retrieval. In Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence , pages 982-988, 2019.

Yi Xu, Lichen Wang, Yizhou Wang, and Yun Fu. Adaptive trajectory prediction via transferable gnn. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 6520-6531, 2022.

Liang Yang, Zesheng Kang, Xiaochun Cao, Di Jin, Bo Yang, and Yuanfang Guo. Topology optimization based graph convolutional network. In Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence , pages 4054-4061, 2019.

Donghan Yu, Ruohong Zhang, Zhengbao Jiang, Yuexin Wu, and Yiming Yang. Graph-revised convolutional network. In Machine Learning and Knowledge Discovery in Databases , pages 378-393, 2021.

HaiYang Zhang, XiMing Xing, and Liang Liu. Dualgraph: A graph-based method for reasoning about label noise. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 9654-9663, 2021.

Appendix

A Proof of Theorem

A.1 Proof of Theorem 1

Proof. To simplify the proof, let us assume that A = ✶ R + ( A ) ∈ { 0 , 1 } n × n 6 . Consider node V i . If A ij = 1 , it implies that node V j is one of the 1 -hop neighbors of V i . Now, if ( A 2 ) ij = A i : · A : j = 1 , it means that there exists an 1 -hop neighbor of V i that is also the 1 -hop neighbor of V j , indicating that node V j is one of the 2 -hop neighbors of V i . Similarly, if ( A 3 ) ij = ( A ) 2 i : · A : j = 1 , it signifies that there exists a 2 -hop neighbor of V i that is also the 1 -hop neighbor of V j , implying that node V j is one of the 3 -hop neighbors of V i . By extension, if ( A k ) ij = 1 , it implies that node V j is the one of k -hop neighbors of V i . Based on this observation, for ∀ j ∈ { j | ( A ) ij = 1 ∪ ( A 2 ) ij = 1 ∪ . . . ∪ ( A k ) ij = 1 } , if node V j is unlabeled, it indicates that all the κ -hop neighbors of V i for κ ∈ { 1 , . . . , k } are unlabeled nodes. Therefore, according to Definition 1, node V i qualifies as a k -hop starved node.

Illustration of Theorem 1. To provide a clearer understanding, we present an example to illustrate the process of identifying k -hop starved nodes based on the given adjacency matrix. Fig. 3 depicts a graph consisting of 6 nodes, with 2 labeled and 4 unlabeled. The edges between the nodes are shown in the figure, with all edge weights set to 1 for simplicity. To identify the k -hop starved nodes, we need to determine the k -hop neighbors for each node. The steps below demonstrate the identification process of k -hop neighbors based on the given graph in Fig. 3, where the k -hop neighbors for each node are listed at the end of each row of the corresponding matrices:

Identifying 1 -hop neighbors based on A :

1

2

3

5

6

4

Labeled

Unlabeled

Since nodes V 2 and V 4 are labeled, we identify the 1 -hop starved nodes as { V 1 , V 5 , V 6 } .

Identifying 2 -hop neighbors based on A 2 :

Now, we can identify 2 -hop starved nodes from the set { V 1 , V 5 , V 6 } as { V 5 } .

Identifying 3 -hop neighbors based on A 3 :

We observe that there are no 3 -hop starved nodes.

Identifying 4 -hop neighbors based on A 4 :

We observe that there are no 4 -hop starved nodes.

Note that, a node identified as k -hop starved node is also considered as a ( k -1) -hop starved node, as exemplified by node V 5 . Consequently, if there are no k -hop starved nodes present, it follows that there are no ( k +1) -hop starved nodes.

A.2 Proof of Theorem 2

Proof. Given that C = A [: , col _ mask ] ∈ R n × c , C models the affinities between all nodes and c labeled nodes. Consequently, if ( row _ mask ) i = ✶ R -( C ✶ c ) i = 1 , indicating that ( C ✶ c ) i = 0 , we can deduce that there are no labeled 1 -hop neighbors of node V i . As a result, node V i belongs to the set of 1 -hop starved nodes, denoted as Set 1 ( r ) = { V i | i ∈ RM + } . Additionally, it follows that U = A [ row _ mask,col _ mask ] = 0 ∈ R r × c . Considering R = A [ row _ mask, :] ∈ R r × n , where R models the affinities between the r 1 -hop starved nodes and all nodes, we can examine each i ∈ RM + (where RM + denotes the indexes of all 1 -hop starved nodes). If all 1 -hop neighbors of V i ( i.e., ∀ j satisfying [ ✶ R + ( R )] ij = 1 ) are 1 -hop starved nodes ( i.e., j ∈ RM + ), then node V i qualifies as a 2 -hop starved node.

Illustration of Theorem 2. To facilitate understanding, we continue using Fig. 3 as an example for illustration. According to Theorem 2, the corresponding C , U , R matrices are as follows:

Based on the C , U , R matrices, we can determine that row _ mask = [1 , 0 , 0 , 0 , 1 , 1] ⊤ , RM + = { 1 , 5 , 6 } , the 1 -hop starved nodes are V 1 , V 5 , V 6 , and the 2 -hop starved node is V 5 .

A.3 Proof of Proposition 1

Proof. Recall that ̂ ρ + = ∑ i ∈ RM + α ̂ U ij ∑ i ˜ A ij and ρ + = ∑ i ∈ RM + α ˜ U ij ∑ i ˜ A ij . For ∀ j ∈ CM + , if we randomly select τ out of c labeled nodes as the supplementary adjacent points for each 1 -hop starved node V i , then the probability of ̂ U ij > 0 is τ c . Considering the j -th column ( j ∈ CM + ), if ∑ i ∈ RM + α ̂ U ij = ∑ i ∈ RM + α ˜ U ij ( i.e., ̂ ρ + = ρ + ), it means that all the 1 -hop starved nodes V i (for i ∈ RM + ) select V j as the supplementary adjacent point. Since there are r 1 -hop starved nodes, the probability of ̂ ρ + = ρ + is ( τ c ) r . Moreover, since τ < c , we have ̂ ρ + = ∑ i ∈ RM + α ̂ U ij ∑ i ˜ A ij ≤ ρ + , where ̂ ρ + = ρ + with only a probability of ( τ c ) r .

A.4 Proof of Proposition 2

Proof. Recall that ̂ ρ -= ∑ i/ ∈ RM + ˜ A ij + α ∑ i/ ∈ RM + Q ij ∑ i ˜ A ij + α ∑ i/ ∈ RM + Q ij and ρ -= ∑ i/ ∈ RM + ˜ A ij ∑ i ˜ A ij . For ∀ j ∈ CM + , if we randomly select τ out of c labeled nodes as the supplementary adjacent points for each non-starved

node, then the probability of Q ij = 0 is ( 1 -τ c ) . Considering the j -th column ( j ∈ CM + ), if α ∑ i/ ∈ RM + Q ij = 0 ( i.e., ̂ ρ -= ρ -), it means that all the non-starved nodes V i (for i / ∈ RM + ) do not select V j as the supplementary adjacent point. Since there are r 1 -hop starved nodes, the probability of ̂ ρ -= ρ -is ( 1 -τ c ) n -r . Moreover, since ∑ i ˜ A ij > ∑ i/ ∈ RM + ˜ A ij ( | RM + | = r ), we have ̂ ρ -≥ ρ -, where ̂ ρ -= ρ -with only a probability of ( 1 -τ c ) n -r .

B Experimental Settings

B.1 Dataset Description

Table 6 provides a comprehensive overview of the datasets used in our experiments, including various statistical characteristics. Please refer to the table for detailed information regarding the datasets. Note that, the original edge features are not used in the experiments as our goal is to infer a latent graph from the node features of the datasets.

B.2 Baselines

In our experiments, we utilize publicly available code repositories provided by the respective authors and follow the hyperparameter settings outlined in their papers [4,10,13,19,46]. We rerun the codes for all the methods and record the best testing accuracy. Below, we summarize the implementation details and provide the code links for each of the used methods:

GCN+KNN [19]: https://github.com/tkipf/pygcn.

GCN&KNN [19]: https://github.com/tkipf/pygcn.

For GCN&KNN, we implement the codes by ourselves. For simplicity, we adopt the following Dirichlet energy [4] as the graph regularization loss:

IDGL [4]: https://github.com/hugochan/IDGL.

GRCN [46]: https://github.com/PlusRoss/GRCN.

SLAPS [10]: https://github.com/BorealisAI/SLAPS-GNN.

Note that, SLAPS is a multi-task-based approach that involves training an additional self-supervision task for latent graph inference. To ensure computational efficiency, we set the maximum number of epochs to 1000 for ogbn-arxiv dataset. We adopt the following denoising autoencoder loss as the graph regularization loss [10]:

where idx are the selected indices, and F is the binary cross-entropy loss or the mean-squared error loss [10].

LCGS [13]: https://github.com/hu-my/LCGS.

C Further Discussions

C.1 Deeper GNNs

Why using deeper GNNs is not an optimal solution? This is because using deeper will bring up new issues as illustrated below. First , although increasing GNN to 4 layers reduces the number of starved nodes from near 3 , 000 to near 500 for the Citeseer120 dataset, the number of starved nodes still accounts for nearly 15% . In fact, the number of starved nodes depends on several factors, including the labeling rate of nodes, the graph sparsification process, and so on. How to select the number of layers of GNNs to reduce the starved nodes for different datasets and different LGI methods is not easy . Second , deeper GNNs typically provide inferior performance. To allow a k -hop starved node to receive information from labeled nodes, the GNNs need to have at least k +1 layers. However, as the number of layers k increases, the GNNs may still fail to propagate supervision information from labeled nodes to k -hop starved nodes due to the over-squashing issue [1]. Moreover, deeper GNNs will make the oversmoothing problem [10,21] be even more severe. Empirically, we test the SLAPS method on the Pubmed dataset using a different number of layers, and the corresponding accuracies can be seen in Table 7. As we can see, the accuracy of SLAPS with a 8 -layer GNN is extremely low, demonstrating that the generalization of GNNs cannot be guaranteed by simply increasing the number of layers . Moreover , deeper GNNs make the model more complexity. Table 7 lists the number of parameters and the float point operations (FLOPs) of SLAPS method with 2 , 4 , and 8 -layer GNNs. We observe that as the number of layers increases, both the number of parameters and the FLOPs increase . In fact, the majority of existing LGI methods typically only adopt 2 -layer GNNs for effectiveness and efficiency.

C.2 Degree Distribution

Why introducing connections between labeled and unlabeled nodes helps the models? We provide a basic insight to answer this question. As we know, an unlabeled node must belong to a specific class, and in the node classification task, a specific class must have some labeled nodes. As a result, we can derive the prior assumption that a good classification model will make an unlabeled node closer to some labeled nodes of the same class. That is why we select the closest labeled nodes as the supplementary adjacent points for the starved nodes. If we compulsorily add such connections between the unlabeled node and its closest labeled nodes, the prior information will be exploited during training, helping the models work better. It is worth noting that introducing connections between labeled and unlabeled nodes can lead to a substantial rise in the degrees of labeled nodes. A potential question is that whether altering the distribution of node degrees will have an adverse effect on the model. To be honest, we have not yet observed any adverse effects from the current experimental results. Nevertheless, it is still worth exploring whether the divergence in degree distributions will potentially hinder the model's performance. What the exact relationship is between the degree distributions and the model's performance in latent graph inference task is a difficult and open question that deserves in-depth exploration.

C.3 Definition of Regularization Graph

Why selecting k = 1 to define the regularization graph between k -hop starved nodes and labeled ones? In fact, we can set any value for k to define the regularization graph. However, if we set k > 1 , there will be a potential issue. For illustration, let us set k = 3 and eliminate the 3 -hop starved nodes by adding the regularization graph. In this case, 1 -hop and 2 -hop starved nodes will still exist since only 3 -hop starved nodes are removed. To address this, a simple solution is to directly set k = 1 to define the regularization graph. Why only setting k = 1 works? According to the definition of starved nodes, we know that a k -hop starved node also qualifies as a ( k -1) -hop starved node. Therefore, if

we eliminate 1 -hop starved nodes then all m -hop starved nodes for m> 1 will be eliminated. Due to the effectiveness and simplicity, we therefore select k = 1 .

C.4 Generalization Performance

To show the results of the models corresponding to the best validation performance, we conduct more experiments on the Pubmed dataset, and the experimental results are listed in Table 8. As we can see, our methods still improve the baselines, which further indicates their effectiveness.

D Limitation

According to Theorem 1, we can identify k -hop starved nodes for k ≥ 1 . Additionally, based on Theorem 2 and CUR matrix decomposition, we demonstrate the identification of k -hop starved nodes for k ∈ { 1 , 2 } . How to identify k -hop starved nodes for k > 2 based on CUR decomposition remains an interesting direction for further investigation. Nevertheless, it is worth noting that by eliminating all κ -hop starved nodes, there will be no k -hop starved nodes for ∀ k > κ . Consequently, we think that only identifying k -hop starved nodes for k ∈ { 1 , 2 } is sufficient and efficient to address the problem of supervision starvation.

Peer reviews:
comments: SUMMARY:
The authors figure out that the graph sparsification operation results in the supervision starvation problem in latent graph inference (LGI). They propose to identify k-hop starved nodes and diminish the starved nodes by incorporating a regularization adjacency matrix into the initial one. They further reduce the computational cost by using CUR matrix decomposition and tackle the weight contribution rate decay problem via some simple strategies. The effectiveness of the proposed method is validated on well-known benchmarks.

Update after rebuttal:
The authors have addressed my concern through the rebuttal. The score remains unchanged.

STRENGTHS:
1. This paper is well-written and easy to follow.
2. This paper identifies the supervision starvation problem for LGI.
3. The proposed method is well-motivated and theoretically justified.
4. The proposed method is model-agnostic and can be seamlessly integrated into various LGI models.
5. The empirical results are significant on benchmarks.

WEAKNESSES:
From the analysis in Section 4.2, the proposed method may be sensitive to the hyper-paramters $\tau$ and $\alpha$.

QUESTIONS:
In Eq. (4), the refined matrix $\widetilde{\mathbf{A}}$ becomes a weighted adjacency matrix when the positive parameter $\alpha \neq 1$. Will tuning $\alpha$ help with methods developed for unweighted graphs?

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-06-23

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper points common LGI methods suffer from the issue of `supervision starvation`.\nIt also observes this issue is actually caused by the graph sparsification operation.\nTo address this problem, the paper proposes to restore the corrupted affinities and replenish the missed supervision.\nIt presents the `CUR matrix decomposition` to reduce the computational burden and eliminates the starved nodes by reconstructing the destroyed connections\nThe method is model-agnostic and can be seamlessly integrated into existing LGI methods.\nExtensive experiments show promising results.\n\nSTRENGTHS:\n1) Originality.\nThe identifies graph sparsification is the main cause of the supervision starvation (SS) in LGI.\nThe paper is solid in analyzing problems with insights.\n\n2) Quality.\nThis method is supported by both theoretical (Theorem 1, 2) and experimental aspects, having strong persuasiveness.\n\n3) Clarity.\nThe paper is well written and organized.\n\n4) Significance.\nLGI is a common task graph learning. This method can be potentially applied in many situations.\n\nWEAKNESSES:\n1) The paper should clearly explain why using `CUR Decomposition` is more efficient.\nThe experimental results only show a higher accuracy, but does not reflect the efficiency.\nSo this advantage is not well supported.\n\n2) The section3.2 and 3.3 are not easy to follow. \nThe reviewers and potential readers may prefer a popular version.\n\nQUESTIONS:\n1) The $L_{reg}$ should have a clear mathematical form in paper in Equal 9.\n\n2) The authors are encouraged to discuss more about SoTA GNNs [15,23,37,40] since the proposed approach is model-agnostic.\nThe reviewer is curious why these methods were mentioned in Section1 but not compared in experiments.\n\nLIMITATIONS:\nThe LGI is a common task in graph learning and all experiments are conducted on public and popular datasets.\nThe reviewer see no negative societal impact and limitations of this work.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a method for latent graph inference (aka graph structure learning) based on the idea of mitigating the supervision starvation problem present when jointly learning the underlying graph structure and node representations. The paper claims that the supervision starvation problem is caused by the sparsifier operation present in many existing models in the literature. It further proposes to recover (some) starved edges to resolve the issue and shows that reducing the number of starved edges consistently improves the performance on multiple benchmarks.\n\nSTRENGTHS:\nThe problem of latent graph inference and the supervision starvation problem present in many existing approaches are important directions that are recently gaining more and more attention. The paper proposes an approach to identify the starved edges and add them back to the graph structure that is backed up by CUR decomposition and theoretical results. The final analysis shows the proposed strategy improves the performance of state-of-the-art LGI methods. The method is shown to be more effective when supervision is less.\n\nWEAKNESSES:\nW1: One main weakness of the model is that the results are obtained using the average of the top five best testing performances. I understand that this setup was fixed across multiple baselines and datasets but to measure the generalization of the model, I would expect the results to be an average over testing performance of the models corresponding to the best validation performance. This is especially important as the charts in Figure 2 show a lot of instabilities in the test loss of multiple models.\n\nW2: the proposed model adds some edges to the graph structure and compares itself to the base model without those edges. I suggest adding a baseline as the base model + random edges to make sure the performance obtained is not just because there are more edges present.\n\nQUESTIONS:\nQ1: Can you elaborate more on what models M_U and M_R are referring to? My understanding so far is that M_U is referring to Equation 6 and M_R is referring to Equation 8.\n\nQ2: Can you explain what edge weights have been used on both existing and starved edges? Is this the real-valued edge from the output of the graph generator for both type of edges? \n\nQ3: What type of graph generators has been used to obtain the results in Table 1?\n\nLIMITATIONS:\nOne limitation of the model is that Theorem 2 only works for k of 1 and 2. For many academic datasets, using one or two layers of GNNs is enough. However, in many real-world applications, deeper GNNs are needed to send the long-range dependencies. The paper does not provide any insights on how the idea can be extended for higher values of ks.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

comments: SUMMARY:
This paper proposes a new method for the latent graph inference problem. The motivation of the new method is the existence of supervision starvation nodes caused by graph sparsification operation. To reduce the number of supervision starvation nodes, the authors propose a CUR matrix decomposition based method to add an additional adjacency matrix to the original sparse adjacency matrix with some constraints. The extensive experimental results show that the proposed idea is effective on various datasets and different base models.

STRENGTHS:
1.	The paper is well written. It is easy to follow the idea and understand the motivation and methodology design.
2.	Although the method is simple, it is effective and with sufficient analysis. 
3.	The performance looks promising.

WEAKNESSES:
The reason why some nodes may receive no supervision is also because of the only one-layer GNN. If there are multiple layers of GNNs, the supervision information will propagate in the graph to achieve semi-supervised training, which is the core of most of GNNs. If there are multiple layers of GNNs, is that still reasonable to define ‘supervision starvation’? In Line 155, why is deeper GNN not an optimal solution to reduce the starving nodes? From the Figure 1, we can see that increasing GNN to 4 layers could reduce the starving nodes from near 3,000 to only near 500. 

Since matrix A is dynamically generated from a latent graph generator, will the U or Q dynamically change as well? Are they dependent on generated A? If so, is that differentiable?

QUESTIONS:
see above

LIMITATIONS:
Yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a model-agnostic enhancement for current LGI (latent graph inference) methods, aiming to address the assumed issue of unlearned features in a substantial number of nodes and edges, which are believed to negatively impact LGL's generalization performance. It proposes to learn a weighted residual graph, focusing on unlabeled nodes without labeled neighbors and intending to link them with some labeled nodes. Additionally, the proposed method applies a sparsity constraint to the residual graph, preventing significant alterations to the underlying graph topology learned from the base LGI model. Finally, the paper evaluates the effectiveness of this approach by integrating it with several prior LGI models, and finds modest improvements in standard graph learning tasks.\n\nSTRENGTHS:\nThe paper's strength lies in its investigation of the important problem of optimizing the structure for neighborhood aggregation in graph neural networks. It introduces a method that addresses the issue of \"supervision starvation\" by adding labeled neighbors to initially supervision-starved nodes, which is demonstrated to bring benefit to existing LGI methods, especially on larger graphs.\n\nWEAKNESSES:\n1. The paper's proposed method shows moderate improvements, with limited statistical significance on small graphs (Cora and Citeseer). Larger graphs display more significant enhancements. The relationship between improvement and graph size should be explored for better insights. Further improvements for small graph scenarios are needed.\n\n2. The paper lacks clarity in explaining the generation of the residual adjacency matrix. The emphasis on the relationship with CUR decomposition should be weaken since the learning criteria for weights in matrix $\\tilde{U}$ are unrelated to reconstructing the target matrix $Q$. Furthermore, the method of selecting \"supplementary adjacent points\" from the labeled nodes for each 1-hop starved node remains unclear. The authors should be aware that \"randomly selecting a subset of $\\tau$ nodes from $c$ labeled nodes\" is a different statistical event compared to \"picking each labeled node at a probability of $\\frac{\\tau}{c}$\". Improved explanations of these aspects are necessary to avoid confusion and strengthen the paper's presentation.\n\n3. The paper lacks significant theoretical and technical novelty. Certain modeling choices are presented without adequate theoretical justification to demonstrate their importance or sensitivity analysis to show their insignificance. Specifically, the content in the paper should be enriched or reorganized to address the following research questions: (1) the reasons and mechanisms behind the significance of k-hop starved nodes in training a GNN through semi-supervised learning (i.e., why they are pivotal nodes), and (2) a comprehensive evaluation of the advantages and disadvantages of selecting k=1 to define the residual bipartite graph between k-hop starved nodes and labeled nodes, emphasizing how the benefits outweigh any drawbacks. Addressing these questions would enhance the paper's contributions and strengthen its overall impact in the field.\n\nGiven the aforementioned issues, a major revision of the current version of the paper is recommended before acceptance.\n\nQUESTIONS:\n1. Could the authors provide further elaboration on why the k-hop starved edges are defined as \"having at least one endpoint being a $k$-hop starved node\"? If the main objective is to identify edges whose weights are not trained in a $k$-layer GNN, wouldn't a more appropriate definition be \"both endpoints are $(k-1)$-hop starved nodes\"? (0-hop starved = unlabeled)\n\n2. Could the authors provide an analysis of why adding connections between 1-hop starved nodes and labeled nodes improves the model's generalization? It appears that while introducing supervision signals to some unlabeled nodes, these additional connections to the labeled nodes may make them less \"similar\" to the unlabeled nodes. For example, the degree distribution of labeled and unlabeled nodes might be similar in the original dataset, but after the structural modifications proposed in the paper, the labeled nodes may exhibit significantly higher degrees than the unlabeled nodes. Wouldn't such divergence in the distribution of labeled and unlabeled nodes negatively impact the model's generalization? An explanation of these potential effects would help clarify the benefits and implications of the proposed structural modifications.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-31"}