Stabilized Self-training with Negative Sampling on Few-labeled Graph Data
Authors: Ziang Zhou
Jieming Shi
Shengzhong Zhang
Zengfeng Huang
Qing Li
Venue: ICLR 2022 Submitted (2022)

Abstract: Graph neural networks (GNNs) are designed for semi-supervised node classification on graphs where only a small subset of nodes have class labels. However, under extreme cases when very few labels are available (e.g., 1 labeled node per class), GNNs suffer from severe result quality degradation. 
Specifically, we observe that existing GNNs suffer from unstable training process on few-labeled graph data, resulting to inferior performance on node classification. Therefore, we propose an effective framework, Stabilized self-training with Negative sampling (SN), which is applicable to existing GNNs to stabilize the training process and enhance the training data, and consequently, boost classification accuracy on graphs with few labeled data. In experiments, we apply our SN framework to two existing GNN base models (GCN and DAGNN) to get SNGCN and SNDAGNN, and evaluate the two methods against 13 existing solutions over 4 benchmarking datasets. Extensive experiments show that the proposed SN framework is highly effective compared with existing solutions, especially under settings with very few labeled data. In particular, on a benchmark dataset Cora with only 1 labeled node per class, while GCN only has 44.6% accuracy, SNGCN achieves 62.5% accuracy, improving GCN by 17.9%; SNDAGNN has accuracy 66.4%, improving that of the base model DAGNN (59.8%) by 6.6%.

Under review as a conference paper at ICLR 2022
STABILIZED SELF-TRAINING WITH NEGATIVE SAM- PLING ON FEW-LABELED GRAPH DATA
Anonymous authors Paper under double-blind review
ABSTRACT
Graph neural networks (GNNs) are designed for semi-supervised node classifica- tion on graphs where only small subset of nodes have class labels. However, under extreme cases when very few labels are available (e.g., labeled node per class), GNNs suffer from severe result quality degradation.
Specifically, we observe that existing GNNs suffer from unstable training process few-labeled graph data, resulting to inferior performance on node classifica- tion. Therefore. we propose an effective framework, Stabilized self-training with Negative sampling (SN), which is applicable to existing GNNs to stabilize the training process and enhance the training data, and consequently, boost classifica- tion accuracy on graphs with few labeled data. In experiments, we apply framework to two existing GNN base models (GCN and DAGNN) to get SNGCN and SNDAGNN, and evaluate the two methods against 13 existing solutions over 4 benchmarking datasets. Extensive experiments show that the proposed SN frame- work is highly effective compared with existing solutions, especially under set- tings with very few labeled data. In particular, on benchmark dataset Cora with only 1 labeled node per class, while GCN only has 44.6% accuracy. SNGCN achieves 62.5% accuracy, improving GCN by 17.9%; SNDAGNN has accuracy 66.4%, improving that of the base model DAGNN (59.8%) by 6.6%.
SN
INTRODUCTION
Graph is an expressive data model, representing objects and the relationships between objects as nodes and edges respectively. Graph data are ubiquitous with a wide range of real-world appli- cations, e.g., social network analysis (Qiu et al., 2018 Li & Goldwasser] 2019). traffic network prediction (Guo et al. 2019 Li ct al. 2019). protein interface prediction (Fout et al. 2017). recom- mendation systems (Fn et al. 2019 Yang et al. 2020a). Among these applications,
important
task to classify the nodes in a graph into various classes. However, one tough situation commonly existing the lack of sufficient labeled data, which are also expensive to collect.
To ease the situation, semi supervspe node classification on graphs has attracted much attention from both industry (Qiu et al. 2018 Li & Goldwasser 2019) and academia (Defferrard al. 2016
Hamilton et al. [ et al. [2188] et al. |2000] Li et al.. Klicpera et af.) 2019).
It aims to leverage small amount of labeled nodes and additionally large amount of unlabeled nodes in a graph to train an accurate classifier. There exists a collection of graph neural networks for semi-supervised node classification (Kipf & Welling 2017 Velickovic et al. 2018 Montt al.
2017 Hamilton et al. 2017: Klicpera et al. 2019: et al.) 2020). For instance, Graph convolution networks (GCNs) rely on message passing scheme called graph convolution that aggregates the neighborhood information of node, including node features and graph topology, to learn node representations, which can then be used in downstream classification tasks (Kipf & Welling 2017).
Despite the great success of GCNs, under the extreme cases when very few labels are given (e.g.. only one labeled node per class), the shallow GCN architecture, typically with two layers (Kipl & Welling 2017). cannot effectively propagate the training labels over the input graph, leading to inferior performance. In particular, as shown in our experiments, on a benchmark dataset Cora with 1 labeled node per class (Cora-1), 1), GCN is even less accurate than some unsupervised methods, such as DGI (Velickovic et al. 2019) and G2G (Bojchevski & Günnemann 2018). Recently, several latest studies try improve classification accuracy by designing deeper GNN architectures, egg., DAGNN

---

Under review as a conference paper at ICLR 2022
40
40
30
30
Percentago
Percentage
20
%
100
200
300
%
100
200
300
Epoch
Epoch
(a) GCN
(b) DAGNN
30
Percentago(%)
30
Parcnntage
20
20
10
%
200
%
100
200
300
Epoch
Epoch
SNGCN
(d) SNDAGNN
Figure 1: The distribution of predicted labels in different classes in Cora-l.
(Liu et al. 2020). which also address the over over-smoothing issue identified in (Xu et al. 2018 Li ct al. 2018 Chen et al. 2020a). However, these deep GNNs are still not directly designed tackle the scarcity of labeled data, especially when only very few labels are available.
After conducting in-depth study, we have important finding that existing GNNs suffer from unstable training process, when labeled nodes are few. In particular, on Cora dataset with 7 classes, for each run, randomly select 1 labeled node per class the training data for both GCN and DAGNN, and repeat 100 runs with 300 epochs per run, to get the average number of predicted labels percentage per class each epoch and also the standard deviation. The statistical results of GCN and DAGNN are shown in Figures (a) and ([b) respectively. z-axis is the epoch from to 300, and y-axis is the percentage of class the predicted node labels. There are 7 colored lines representing the average percentage of the predicted labels of the respective classes, as the epoch increases. The dashed lines are the ground-truth percentage of each class in the Cora dataset. The shaded areas in colors represent the standard deviation. Observe that in Figure [a). GCN has high variance at different runs when predicting node labels, and the variance keeps large late epochs, egg. 300, which indicates that GCN is quite unstable at different runs with 1 training label per class sampled randomly, leading to inferior classification accuracy as illustrated in our experiments. Moreover, shown in Figure [b). DAGNN also suffers from unstable training process. The variance of DAGNN relatively smaller than that of GCN, which provides a hint about why DAGNN performs better than GCN. Nevertheless, both GCN and DAGNN yield unstable training process with large variance. Since there is only 1 labeled node per class in Cora-1, at different runs, the randomly sampled training nodes can heavily influence the message passing process in GCN and DAGNN, depending on the connectivity of the training nodes to their neighborhoods over the graph topology, which result the unstable training process observed above.
To address the unstable training process of existing GNNs when only very few labeled data are available, we propose a framework, Stabilized self-training with Negative sampling (SN), which readily applicable to existing GNNs to improve classification accuracy via stabilized training pro- cess. In the proposed SN framework, at each epoch, we select a set of nodes with predicted labels pseudo labels and add such pseudo labels into training data to enhance the
of high confidence training of next epoch. To tackle the unstable issue of existing GNNs, we develop a stabilizing technique in self-training to balance the training. We then design a negative sampling regularization technique over pseudo labels to further improve node classification accuracy. In experiments, we apply our SN framework to GCN and DAGNN, denoted as SNGCN and SNDAGNN respectively.
2

---

Under review as a conference paper at ICLR 2022
Figures [c) and d report the average percentage and standard variance of the predicted labels per class per epoch of SNGCN and SNDAGNN on Cora-1 respectively. With our stabilized self- training technique, obviously, the variance of SNGCN in Figure decreases quickly and becomes stable as epoch increases, compared with Figure aa) of GCN. SNDAGNN is also more stable than DAGNN as shown in Figures (d] [Td respectively. As reported later in experiments, with the proposed SN framework, SNGCN achieves 62.5% node classification accuracy on Cora-I, signifi- cantly improving GCN 6%) by 17.9%, and SNDAGNN obtains 66.4% accuracy on Cora-I and outperforms DAGNN (59.8%) by substantial margin of 6.6%. We conduct extensive experiments on benchmarking datasets, and compare with 13 existing solutions, to evaluate the performance of the proposed SN framework. Experimental results demonstrate that our SN framework is able to significantly improve classification accuracy of existing GNNs when only few labels are available, and is also effective when training labels are sufficient.
2 RELATED WORK
In literature, there are two directions to address the scarcity of labeled data for semi-supervised
node classification: (i) explore multi-hop graph topological features to propagate the labels in L over the input graph, e.g., GCN (Kipf & Welling 2017) and DAGNN (Liu et al. 2020); (ii) enhance the training data by pseudo labels (self-training) (Li et al. 2018) or augmenting graph data by new
edges and features (Kong et al. 2020; Zhao et al. 202 Note that these two directions are not mutually exclusive, but can work together on lew-labeled graph data. Here we review the existing studies that are most relevant to this paper.
GNNs. There exist a large collection of GNNs, such as GCN, DAGNN, GAT, MoNet, and APPNP
(Kipf & Welling, 2017: Liu et al.] 2220;; Bruna al.] 2014 Henaff et al. [015 Defferrard et al.
2016 Velickovic et al. 2018 Monti et al. 201 7| Chen et al. 2020b Kliepera et al. 2019). We
introduce the details of GCN (Kipf & Welling 2017) and DAGNN (Liu et al. 2020 here. GCN
learns the representation of each node by iteratively aggregating the representations of its neighbors.
Specifically, GCN consists of k: > 0 layers, each with the same propagation rule defined as follows.
At the l-th layer, the representations H of previous layer are aggregated to get ((9)
H ((-) ),E é 1.2,..,k.
(I)
Á = D- is the graph laplacian, where Ā = A + I is the adjacency matrix of G after adding self-loops (I is the identity matrix) and D is diagonal matrix with D. -, A.j. W(O is trainable weight matrix of the ε-th layer, and is a nonlinear activation function. Initially, H = X. Note that GCN usually achieves superior performance with 1 1-layer or 2-layer models Kipp & Welling 2017). When applying multiple layers to leverage large receptive fields, the per- formance degrades severely, due to the over-smoothing issue identified in (Xu et al. 2018 Li et al.) 2018 Chen et al. 2020a). A recent deep GNN architecture, DAGNN, tackles the over-smoothing issue and achieves state-of-the- ther-rtt results by decoupling representation transformation and propaga- tion in GNNs (Liu et al., (020). Then it utilizes an adaptive adjustment mechanism to balance the information from local and global neighborhoods of each node. Specifically, the mathematical ex- pression of DAGNN is as follows. DAGNN uses a learnable parameters E Rex1 to adjust the weight of embeddings at different propagation level (from 1 to k). It processes data in the following way. Z = MLP E Raxe, He A Z E Ruxe { = 1,2, ..., k. S; = He E R ,, 1, 2,2, k, S. = [Sr, St... S.] E gaxe 1,2, k Xout = softmax( Ei=1 H Sz), where A is the L-th power of matrix A, @ is the Hadamard product, is dot product, MLP is the Multilayer Perceptron and softmax operation is on the second dimension.
Data Augmentation. Another way to address the situation of limited labeled data is to add pseudo labels to training dataset by self-training (Li et al., 2018), or enhance the graph data by adding new edges and features (Zhao et al. [021] Kong et al. (2220). Self-training itself is a general methodology [Scudder 1965) and is used in various domains in addition. It is used in word-sense disambiguation Yarowsky 1955: Hearst 1991), bootstrap for information extraction and learning subjective nouns Riloff & Jones 1999). and text classification (Nigam et al. 2000). In (Zhou et al. 2012). it suggests that selecting informative unlabeled data using guided search algorithm can significantly improve performance over standard self- self-training framework. Buchnik & Cohen (018) mainly consider self- training for diffusion-basee techniques. Recently, self-training has been adopted for semi-supervised
3

---

Under review as a conference paper at ICLR 2022
tasks graphs. For instance, Li et al. (2018) propose self-training and co co-training techniques for GCN. This self-training work selects the top-k confident predicted labels as pseudo labels. The co-training technique co-trains GCN with random walk model to handle few-labeled data. Com- pared with existing self-training work, our framework are different as shown later. In particular, our framework has different strategy to select pseudo labels and also has stabilizer address the deficiencies of existing GNNs; moreover, we propose a negative sampling regularization technique further boost accuracy. Besides, in existing work, if node is selected as a pseudo label, will never be moved out even if the pseudo label becomes obviously wrong in later epochs. On the other hand, in our framework, we update pseudo labels each epoch avoid such an issue. There also exist studies to augment the original graph data, which is different from self-training. For instance, Zhao et al. (202)) utilize link prediction to promote intra-class edges and demote inter-class edges in given graph. Kong et al. (2020) iteratively augment node features with gradient-based adversarial perturbations to enhance the performance of GNNs.
THE FRAMEWORK
3.1 PROBLEM FORMULATION
Let 9 EV) be graph consisting of node set V with cardinality n, a set of edges & of size m, each connecting two nodes in V. feature matrix X E laxdd, where d is the number of features in G. For every node U; V, it has a feature vector X; Rd where X; is the i-th row of X. Let e be the number of classes in G. We use L to denote the set of labeled nodes, and obviously V. v. Let L/ be the set of unlabeled nodes and U VV L. Each labeled node v; L has a one-hot vector Y, E 10, indicating the class label of v, Under the few-labeled setting. |C| < 12). A high-level definition of the semi-supervised node classification problem as follows.
Definition Given a graph G EV,E,X) set of labeled nodes L C and a groundiruth class label Y, E 13 per node v, E C, assuming that each node belongs to exactly one class, Semi-Supervised Node Classification predicts the labels of the unlabeled nodes.
per
In particular, the aim is to leverage the graph G with the labeled nodes in L. and to train forward predicting classification model/function f(G 0) that takes as input the graph G and a set of trainable parameters 0. The output of ff is matrix F E Raxe, with each i-th row F, [0, 1] representing the output probability vector of node v; E V (the 1-norm of F; is normalized to 1).
We adopt the widely used cross-entropy loss. For node us its loss of F; with respect to its true class label Y,, L(Y,, F,), is defined as follows.
L(Y,, F.)
Y. (n(s.))
1=1
where Yig is the j-th value in Y, and F,,, j is the j-th value in F,.
3.2 STABILIZED SELF-IRAINING
Recall that existing GNNs suffer from unstable training process as shown in Figure In this section, present the stabilized self-training technique that not only augments training data with pseudo labels but also stabilizes the training process. We first explain how to choose pseudo labels and then introduce loss function of stabilized self-training.
At certain epoch, given the matrix F E RR=x,, with each i-th row F, E [o, 1] representing the output probability vector of node v; E V. Let Yi, satisfying the following Eq. (2) be the predicted label of node v;. We say that, with confidence Fi,j. node is has class label Cy; i.e., the largest element F.,, in vector F; is called the confidence of node U4.
if = arg max, Fiy", Y1s Ru==t {
otherwise,
Then for every unlabeled node v; E U4 we can get N₁, the number nodes with the same predicted label as uy in Eq. (3). Recall that in Figure [ in existing GCN and DAGNN, the distribution of
4

---

Under review as a conference paper at ICLR 2022
the predieted labels is unstable with large variance during the training process; we also observe that most predicted labels are in the same class, especially at the early epochs. To reduce such unstable situation of existing GNNs, we use N, as a stabilizer in our framework to be explained shortly.
N₂ = Ko E u u =Y,}
For every unlabeled nodes in 24 it can have a predicted label. However, the confidence arg maxy, F.,, might be low. We do not want add such low-confidence labels into the training of the next epoch. Therefore, we only choose those unlabeled nodes with high-confidence predicted labels as pseudo labels to be augmented into the training data of next epoch. In particular, an unla- beled node vi is selected to be node with pseudo label in next epoch, if its confidence satisfies threshold B, as shown below. We use Ui to denote all unlabeled nodes selected with pseudo labels.
u' { v E U max F., 3}
where B E [0, 1] is threshold controlling the extent of cautious selection for self-training. A bigger threshold means stricter selection of the pseudo labels.
After explaining how to choose pseudo labels above, present the loss of our stabilized self- training technique in Eq. (5) In particular, we design N.=T as the stabilizer of the training process, to overcome the deficiencres of existing GNNs illustrated in Figure | The intuition is that, if an unlabeled node 2; is in pseudo label class with many nodes, its importance in the loss function is reduced. In other words, our stabilized self-training loss reduces the impact of classes with many pseudo labeled nodes, which is especially useful rectify the training process when the predictions in the early epochs are incorrect or less confident, compared with ground truth.
Ligst Σ YoLEU'
N +1 L(Y,, F,)
(5)
Compared with existing self-training techniques (L et al. 2018), our stabilized self- self-training tech- nique has major differences. First, we develop the stabilizer to re-weight the importance of pseudo labels in the loss function, so to address the unstable issue of existing GNNs. Second, we select only those nodes with high-confidence pseudo labels satisfying 3 threshold, and adaptively update the pscudo labels per epoch, meaning that pseudo label in previous epoch will be removed in the next epoch if its confidence becomes low, On the other hand, existing methods keep a pseudo label once it is selected and never remove it in later epochs (Li et al., (018)), which may harm the training quality if the pseudo label is wrong compared with ground truth.
3.3 NEGATIVE SAMPLING REGULARIZATION
Under extreme cases with very few labeled nodes (e.g. labeled node per class), we further design a negative sampling regularization technique for better performance. In existing studies, negative sampling is used as an unsupervised technique over node embeddings in network embedding meth- ods (Yang et al., 2020b; Velickovic et al. 2019, Yang et al.) 2020b). Here we customize it to the semi-supervised node classification task, and apply negative sampling over labels instead of embed- dings.
Intuitively, the label of node v should be distant to the label of another node u if these two nodes are faraway on the input graph G. Specifically, positive sample is node in L or Ut' We sample a set of positive samples from L U 2/' uniformly at random. The negative samples of a positive sample v are the nodes that are not directly connected to e, in graph G. For each positive sample Us in I, we sample a fixed-siza size set J, of negative samples uniformly at random.
For a nositive-negative pair (U,, Uz), compared with the Y, of vs E LUU', the intention is to let the output vector F, of by to be as different as possible. Here we use the symbol Y, to represent the pseudo label Y; for node is in Ut' ground-truth label Y; of node in L to avoid ambiguity. Denote as the all-one vector in Rs Then we have the following loss of all positive-negative pairs.
Tnng
Σ VI,EIN
Σ
I(Y,, 1. 1 F,)
6
I IJ.
5

---

Under review as a conference paper at ICLR 2022
Algorithm 1: SN Framework Over GNNS
Input: Graph 9 = (V.E, X) with labeled node set L and unlabeled node set 2
Output: the learned classifier [(; 0),
Generate initial parameter for model f(
for each epoch 0, 1,,2, ..., do
Use GNN to compute prediction F + f-G
Get high confidence set lt' and its stabilizing factor N,+I per node (S 3.2) 1
Get positive samples and corresponding negative samples using & U U2' and g (Section 3.3)
Get Lsotal of current epoch by Eq. (7) (Section
Update model parameters by 0 Adam Optimizer((, gragiiint Voliatal). if Convergence then
10
Break
12
end
13 end
3.4
OBJECTIVE FUNCTION AND ALGORITHM
Our final loss function is as follows, and it combines the stabilized self-training loss and negative sampling loss in Eq. 5 and Eq. (6) respectively.
Liotol
Σ Vo;EC
L(Y,, +1 Ao + AgLnnegg
C
where A and X2 are factors controlling the impact of these two losses.
Algorithm T shows the pseudo-code of our SN framework over GNNs, and it takes as input graph with labeled nodes L and unlabeled nodes U. Note that SN can be instantiated over either shallow deep GNN, e.g., GCN and DAGNN introduced in Section 2 The output of Algorithm [ the learned classification model f with trainable parameters 0. Line 3, SN initializes the trainable parameters A by Xavier (Glorot & Bengio 2010). Then from Lines 4 to 12, SN trains the classification model per epoch iteratively. until convergence or the max number of iterations is reached. Specifically, at Line SN first a GNN obtain the forward prediction output F. Then at Line 6, SN detects the pseudo-labeled set UI and obtain the stabilizer N:+I of each node oz in u', after which, at Line 7 we perform negative sampling to obtain positive samples I and negative samples J,. At Line 8, SN computes loss Ltotal of current epoch according to Eq. (7). And Line 9, SN updates model parameters 0 for next epoch by Adam optimizer (Kingma & Ba 2015).
EXPERIMENTS
We evaluate SN against 13 competitors for semi-supervised node classification on 4 benchmark graph datasets. All experiments conducted machine powered by Intel(R) Xeon(R) E5- 2603 v4 @ 1.70GHz CPU, 131GB RAM, 16.04. 1Ubuntu, and 4 Nvidia Geforce 1080ti Cards with Cuda version 10.2. Source codes of all competitors are obtained from the respective authors. Our SN framework is implemented in Python, using libraries including PyTorch (Paszke et al. 2019)) and PyTorch Geometric (Fey & Lenssen 2019). An anonymous link of our source code is provided.
4.1 DATASETS AND COMPETITORS
Datasets. Table I shows the statistics of the 4 real-world graphs used in our experiments. We list the number of nodes, edges, features and classes in each graph dataset respectively. Specifically, the 4 datasets are Cora (Sen et al. 2008), Citeseer (Sen et al.] 2008), Pubmed (Sen et al.) 2008). and widely used for benchmarking node
Core-full (Bojchevski & Günneeannn 2018), all of which
classification performance in existing studies (Sun et al.) 2020 Li et al. 2018 Liu et al. 2020
Notice that every node these graphs has ground-truth class label.
https::/anccyynous. http:/es...cceeeeeecceeeeeeeeeee 1

---

Under review as a conference paper at ICLR 2022
Table 1: Datasets
Cora
Citeseer
Pubmed
Cora-full
of Nodes
2708
3327
19717
19793
# of Edges
5429
4732
44338
65311
# of Features
1433
3703
500
8710
# of Classes
7
6
3
67
Competitors. We compare with 13 existing solutions, including LP (Label Propagation) (Wu et al,
2012), DeepWalk (Perozzi al. 2014), LINE (Tang al. 2013). G2G (Bojchevski & Günnemann
2018 DGI (Velickovic et al.] 2019), GCN (Kipf' & Welling 2017). GAT (Velickovic et al. 2018).
al. 2019). DAGNN (Liu al.) 2020), STs (Li et al. 2018), LCGCN and LCGAT in (Xu et al. 2020). In particular, GCN, GAT, MoNet, APPNP, and DAGNN are GNNs. DeepWalk, DGI, LINE, and G2G are unsupervised network embedding methods. STs represents the four variants in (Li et al. 2018), including Training,, Co- Co-Training, Union, and Intersection; we summarize the best results among them as the results of STs.
MoNet (Monti et al. 2017). APPNP (Klicpera
4.2 EXPERIMENTAL SETTINGS
We evaluate our framework and the competitors on semi-supervised node classification tasks with various settings. In particular, for each graph dataset, we repeat experiments on 100 random data splits as suggested in (Liu et al. 2020 Li et al., 2018) and report the average performance. For each graph dataset, we vary the number of labeled nodes per class in {1, 5, 10, 20), where 3, represent the very few-labeled settings. Following convention in existing work (Liu et al.] 2020), explain what a random data split is, as follows. For example, when the number of labeled nodes per class on Cora is (denoted as Cora-3), since Cora has 7 classes, we randomly pick 3 nodes per class, combining together as a training set of size 21 (i.e., the labeled node set L). and then, among the remaining nodes, we randomly select 500 nodes as a validation set, and 1000 nodes as a test set. Each data split consists of training set, a validation set, and test set as mentioned above. We use the classification accuracy on test set as evaluation metric. Specifically, accuracy is defined as the fraction of the testing nodes whose class labels are correctly predicted by the learned classifier.
4.3 IMPLEMENTATION DETAILS
We instantiate SN framework over the classic GCN model with layers and recent deep GNN architecture DAGNN to demonstrate the effectiveness and applicability of SN. The instantiation of SN over GCN and DAGNN are dubbed as SNGCN and SNDAGNN respectively. SNGCN and SNDAGNN has parameters (i) inherited from GCN and DAGNN and (ii) developed in SN. Hence, we first tune the best parameters of the base models under each classification task setting on each dataset and report this result for them for fair comparison.
Base models (GCN and DAGNN), In base models, we tune the parameters: a L2 regularization rate with search space in (1e-2, 5e-3, 1e-3, 5e-4, 1e-4, 5e-5, 0), dropout rate in (0.5, 0.8). For DAGNN, the level & of propagation after MLP is searched in {10, 15, 20}. We have the follow- ing parameters for base models in experiments: the number of hidden units of GCN and MLP (in DAGNN) is 64 units without bias; the number of layers of GCN and MLP (in DAGNN) is 2 layers; the learning rate of Adam Optimizer is 0.01; the activation function is RELU; U: the maximum number of training epochs is 1000. Moreover, early stopping is triggered when the validation loss is smaller than the average validation loss of previous 100 epochs, and the current epoch is beyond 500 epochs.
SN over GCN and DAGNN (SNGCN and SNDAGNN). After finding the best hyper parameters of the base models, we then tune the parameters in SN. X₁ searched in 10. 1} and A searched in {0, 0.1, 1}. Stabilizing enabler searches in True, False]. The number of positive and negative samples ([ (TT) is searched in (1, 10), (2, 5). (5, 2). (10, })}. For instance, (2, 5) means that we sample 2 positive nodes and then for each positive node, sample 5 negative nodes.
Competitors. We use the parameters suggested in the original papers of the competitors to tune their models, and report the best results of the competitors. Notice that for unsupervised network embedding methods, including WeepWalk, DGI, LINE, and G2G, after obtaining the embeddings, use logistic regression train node classifier over the embedding (Velickovic et al.! 2019).

---

Under review as a conference paper at ICLR 2022
Table 2: Accuracy results (in percentage) on Cora and CiteSeer respectively, averaged over 100 random data splits. (The best accuracy is in bold.)
# of Labels
Cora
CiteSeer
per class
3
5
10
20
1
5
10
20
GCN
44.6
63.8
71.3
77.2
81.4
40.4
53.5
61.0
65.8
69.5
SNGCN (ours)
62.5
72.8
75.8
80.7
82.5
56.2
66.4
68.0
70.2
72.1
DAGNN
59.8 72.4
76.7
80.8
83.7
46.5
58.8
63.6
67.9
71.2
SNDAGNN (ours)
66.4
77.6
79.8
82.2
84.1
65.9 48.5
67.9
69.8
72.1
LP
51.5 60.5 62.5
64.2
67.3
30.1
37.0
39.3
41.9
44.8
WeepWalk
40.4 53.8 59.4 65.4
69.9
28.3
34.7
38.1
42.0
45.6
LINE
49.4 62.6 63.4
71.1
74.0
28.0
34.7
38.0
43.1
48.5
G2G
54.5 68.1 70.9
73.8
75.8
45.1
56.4
60.3
63.1
65.7
DGT
55.3
70.9
72.6
76.4
77.9
46.1
59.2
64.1
67.6
68.7
STs
53.1
67.3
72.5
76.2
79.8
37.2
51.8
60.7
67.4 70.2
GAT
41.8
61.7
71.1
76.0
79.6
32.8
48.6
54.9
60.8 68.2
MoNet
43.4
61.2
70.9
76.1
79.3
38.8
52.9 59.7
64.6 66.9
APPNP
44.7
66.3
74.1
79.0
81.9
34.6
52.2 59.4
66.0 71.8
LCGCN
63.6
74.4
77.5
80.4
82.4
55.3
59.0
68.4
70.3 72.1
LCGAT
58.7
74.5
77.5
79.7
82.6
50.9
66.3
68.5
70.9
71.5
4.4 OVERALL RESULTS
Table 2 reports the classification accuracy (in percentage) of all methods on Cora and CiteSeer, when varying the number of labeled nodes per class in (1.3.5.10.20). The first and second rows report the performance of GCN and SNGCN. The third and fourth rows report the performance of DAGNN and SNDAGNN. Observe that SNGCN (resp. SNDAGNN) enhanced by SN frame- work significantly outperforms GCN (resp. DAGNN) under all task settings, and the performance gain of SNGCN is especially significant when the labels per class are few. For instance, on Cora-1, 1, SNGCN has accuracy 62.5%. while the accuracy of GCN is 44.6%, indicating that the proposed framework improves GCN by 17.9%. This demonstrates the power of the proposed SN framework to boost classification performance. Compared with other competitors, observe that SNDAGNN has the best performance under all settings of Cora (in bold), and SNGCN has the best performance on CiteSeer-1 and 3 and 20, while achieving similar performance compared with LCGCN and LCGAT on CiteSeer-5 and 10. Apart from the superior performance our SN framework over GCN and DAGNN, in Table 2 we can observe two interesting findings. First, under
settings (e.g.. Cora- I), unsupervised methods G2G (54.5%) and DGI (55.3%) achieve better perfor- mance than GCN (44.6%). One reason is that the unsupervised methods good at cases when no labeled data are available, while GCNs still require sufficient amount of labeled data. This finding demonstrates the intuition of the negative sampling regularization in Section 3.3] and also sheds light on possible future research to use unsupervised techniques to further enhance the performance of semi-supervised learning. Second, the performance gap between our methods and competitors enlarges as the number of labels per class decreases, which further illustrates the effectiveness of the proposed SN framework under extreme settings on graphs with very few-labeled nodes per class.
Table presents the classification accuracy of all methods under all settings in (1,3,5,10,20) on PubMed and Cora-full datasets. We exclude from this table, the inaccurate competitors (e.g., Deep- Walk and LINE) that are obviously outperformed by other competitors. Observe that on PubMed and Cora-full, SNDAGNN and SNGCN achieve the higher accuracy consistently than their respective base models GCN and DAGNN. For instance, on Cora-full-1, SNGCN achieves 30.8% accuracy, 6.3% better than GCN. Moreover, observe that on PubMed, SNDAGNN consistently outperforms all other competitors, e.g., LCGCN; Cora-full, SNGCN outperforms all competitors on and settings, and SNDAGNN has the best accuracy on 5, 10, and 20 settings.
In summary, the experimental results presented in Tables 2 and 3 validate the effectiveness of the proposed SN framework over GCN and DAGNN to boost classification performance, especially when only very few labeled nodes are available.
8

---

Under review as a conference paper at ICLR 2022
Table Accuracy results (in percentage) on PubMed and Cora-full respectively, averaged over 100 random data splits. ( The best accuracy is in bold.)
of Labels
PubMed
Cora-full
class
3
5
10
20
1
5
10
20
GCN
55.5 66.0
70.4
74.6
78.7
24.5
41.4
48.1
55.8
60.2
SNGCN (ours)
60.8
67.8
71.6
76.1
79.4
30.8
44.9
49.4
56.6
60.9
DAGNN
59.4 69.5
72.0
76.8
80.1
27.3
43.2
49.8
55.8
60.4
SNDAGNN (ours)
61.0 72.1
74.9
78.2
80.6
27.6
44.4
51.1
56.8
61.2
LP
55.7 61.9 63.5
65.2
66.4
26.3
32.4
35.1
38.0
41.0
G2G
55.2 64.5 67.4
72.0
74.3
25.8
36.4
43.3
49.3
54.3
DGI
55.1 63.4
71.8
73.9
26.2
46.5
55.3
STs
55.1 65.4 69.7
74.0
78.5
29.2
43.6
48.9
53.4
60.8
APPNP
54.8 66.9 70.8
76.0
79.4
24.3
41.5
48.5
55.3
60.1
GAT
52.7 64.4 69.4
73.7
73.5
24.8
41.0
47.5
54.7
59.9
LCGCN
56.6 69.2
72.6
74.6
80.0
26.7
43.9
49.2
55.9
60.5
LCGAT
49.5
59.2
62.3
70.2
65.3
27.4
43.2
48.4
55.0
60.1
bascline
baseline+S
baseline+N Accuracy %6
baseline+SN
Accuracy( %6
80
60
70
50
60
3
5
10
Number of labels per class
Number of labels per class
ablation GCN
(b) ablation on DAGNN
Figure 2: Ablation study of SN Cora.
4.5
ABLATION STUDY
We conduct ablation study evaluate the contributions of the techniques of SN presented in Sec- tion 3] Denote baseline+SN as the method with the whole SN framework enabled, baseline+S as the method with only stabilized self-training loss Eq. (5) enabled, and baseline+N as the method with only negative sampling loss in Eq. (6) enabled. Figures 2aaees_2aaa [aggure22aaad222222 and
the ablation results on baselines GCN and DAGNN respectively, on Cora when varying the number of labels per class in (1,3,5, 5, 10, 20}. Observe that the accuracies of baseline+S and baseline+N are always better than the baseline model, i.e., GCN and DAGNN. Also the accuracy of baseline+SN almost always the highest under all settings. The ablation study demonstrates the power of our proposed techniques to improve classification accuracy.
CONCLUSION
This paper presents Stabilized self- self-training with Negative sampling (SN), an effective framework for cemi-supervised node classification on few-labeled graph data. SN achieves superior performance on graphs with extremely few labeled nodes, through two main designs: stabilized self-training technique that adaptively selects and re-weights high-confidence pseudo labels based on the confi- dence distribution of current epoch to enhance the training process, and negative sampling regu- larization that further fully utilizes unlabeled data to train a high-quality classifier. The effectiveness of SN is extensively evaluated on 4 real graphs, compared against 13 existing solutions. Regard- ing future work, we plan to enhance SN by investigating other unsupervised techniques, and also implement SN top of more GNN architectures to further demonstrate its applicability.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Aleksandar Bojchevski and Stephan Günnemann. Deep gaussian embedding of graphs: Unsuper- vised inductive learning via ranking. In ICLR, 2018.
Joan Bruna, Wojciech Zaremba, Arthur Szlam, and Yann LeCun. Spectral networks and locally connected networks on graphs. In ICLR, 2014.
Eliav Buchnik and Edith Cohen. Bootstrapped graph diffusions: Exposing the power of nonlinear- ity. In Abstracts of the 2018 ACM International Conference Measurement and Modeling of Computer Systems, pp. 8-10, 2018.
Deli Chen, Yankai Lin, Wei Li, Peng Li, Jie Zhou, and Xu Sun. Measuring and relieving the over- smoothing problem for graph neural networks from the topological view. In AAAI, pp. 3438-3445, 2020a.
Ming Chen, Zhewei Wei, Zengfeng Huang, Bolin Ding, and Yaliang Li. Simple and deep graph convolutional networks. In ICML, pp. 1725-1735, 2020b.
Michaël Defferrard, Xavier Bresson, and Pierre Vandergheynst. Convolutional neural networks on graphs with fast localized spectral filtering. In NeurlPS, pp. 3837-3845, 2016.
Wenqi Fan, Yao Ma, Qing Li, Yuan He, Yihong Eric Zhao, Jiliang Tang, and Dawei Yin. Graph neural networks for social recommendation. In wWW. pp. 417-426, 2019.
Matthias Fey and Jan Eric Lenssen. Fast graph representation learning with pytorch geometric. arXiv preprint 1xXx::9933002288 2019.
Alex Fout, Jonathon Byrd, Basir Shariat, and Asa Ben-Hur. Protein interface prediction using graph convolutional networks. In NewrIPS, pp. 6530-6539, 2017.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In AISTATS, pp. 249-256, 2010.
Shengnan Guo, Youfang Lin, Ning Feng, Chao Song, and Huaiyu Wan. Attention based spatial- temporal graph convolutional networks for traffic flow forecasting. In AAAI, pp. 922-929, 2019.
William L. Hamilton, Zhitao Ying. and Jure Leskovec. Inductive representation learning large graphs. In NewrlPS, pp. 1024-1034, 2017.
Marti Hearst. Noun homograph disambiguation using local context in large text corpora. Using Corpora, pp. 185-188, 1991.
Mikael Henaff, Joan Bruna, and Yann LeCun. Deep convolutional networks graph-structured data. arXiv preprint arXiv:1506.05163 2015.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.
Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional net-
works. In ICLR, 2017.
Johannes Klicpera, Aleksandar Bojehevski, and Stephan Günnemann. Predict then propagate: Graph neural networks meet personalized pagerank. In ICLR, 2019.
Kezhi Kong, Guohao Li, Mucong Ding, Zuxuan Wu, Chen Zhu, Bernard Ghanem, Gavin Taylor, and Tom Goldstein. FLAG: adversarial data augmentation for graph neural networks. CORR, 2020.
Chang Li and Dan Goldwasser. Encoding social information with graph convolutional networks forpolitical perspective detection in news media. In ACL, pp. 2594-2604, 2019.
Jia Li, Zhichao Han, Hong Cheng, Jiao Su, Pengyun Wang, Jianfeng Zhang, and Lujia Pan. Predict- ing path failure in time-evolving graphs. In SIGKDD, pp. 1279-1289, 2019.
Qimai Li, Zhichao Han, and Xiao-Ming Wu. Deeper insights into graph convolutional networks for semi-supervvsed learning. In AAAI, pp. 3538-3388330 3538-3545, 2018.
10

---

Under review as a conference paper at ICLR 2022
Meng Liu, Hongyang Gao, and Shuiwang Ji. Towards deeper graph neural networks. In SIGKDD, pp. 338-348, 2020.
Federico Monti, Davide Boscaini, Jonathan Masci, Emanuele Rodolà, Jan Svoboda, and Michael M. Bronstein. Geometric deep learning on graphs and manifolds using mixture model cnns. In CVPR, pp. 5425-5425555 2017.
Kamal Nigam, Andrew McCallum, Sebastian Thrun, and Tom M. Mitchell. Text classification from labeled and unlabeled documents using EM. Mach. Learn., pp- 103-134, 2000.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zachary Vetito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang. Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. NeurlPS, pp. 8024-8035, 2019.
Bryan Perozzi, Rami AI-Rfou, and Steven Skiena. Deepwalk: online learning of social representa- tions. In SIGKDD, pp. 701-710, 2014.
Jiezhong Qiu, Jian Tang, Hao Ma, Yuxiao Dong, Kuansan Wang, and Jie Tang. Deepinf: Social influence prediction with deep learning. In SIGKDD, pp. 21 2110-2119, 2018.
Ellen Riloff and Rosie Jones. Learning dictionaries for information extraction by multi-level boot- strapping. In AAAI, pp. 474-479, 1999.
H. J. Scudder. Probability of error of some adaptive pattern-recognition machines. IEEE Trans. Inf. Theory, pp. 363-371, 1965.
Prithviraj Sen, Galileo Namata, Mustafa Bilgic, Lise Getoor, Brian Gallagher, and Tina Eliassi-Rad. Collective classification in network data. AI Mag., pp. 93-106, 2008.
Ke Sun, Zhouchen Lin, and Zhanxing Zhu. Multi-stage self-supervised learning for graph convolu- tional networks on graphs with few labeled nodes. In AAAI, pp. 5892-5899, 2020.
Jian Tang, Meng Qu, Mingzhe Wang, Ming Zhang, Jun Yan, and Qiaozhu Mei. LINE: large-scale information network embedding. In WWW. pp. 1067-1077, 2015.
Petar Veliekovic, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. Graph attention networks. In ICIR, 2018.
Petar Velickovic, William Fedus, William L. Hamilton, Pietro Liò, Yoshua Bengio, and R. Devon Hjelm. Deep graph infomax. In ICLR, 2019.
Xiao-Ming Wu, Zhenguo Li, Anthony Man-Cho So, John Wright, and Shih-Fu Chang. Learning with partially absorbing random walks. In NeurlPS, pp. 3086-3094, 2012.
Bingbing Xu, Junjie Huang, Liang Hou, Huawei Shen, Jinhua Gao, and Xueqi Cheng. Label- consistency based graph neural networks for semi-supervised node classification. In SIGIR, 2020.
Keyulu Xu, Chengtao Li, Yonglong Tian, Tomohiro Sonobe, Ken-ichi Kawarabayashi, and Stefanie Jegelka. Representation learning on graphs with jumping knowledge networks. In ICML, pp- 5449-5458, 2018
Gang Yang, Xiaofeng Zhang, and Yueping Li. Session-based recommendation with graph neural networks for repeat consumption. In ICCPR, pp. 519-524, 2020a.
Zhen Yang, Ming Ding, Chang Zhou, Hongxia Yang, Jingren Zhou, and Jie Tang. Understanding negative sampling in graph representation learning. In SIGKDD, pp. 1666-1676. 2020b.
David Yarowsky. Unsupervised word sense disambiguation rivaling supervised methods. In ACL, pp. 189-196, 1995.
Tong Zhao, Yozen Liu, Leonardo Neves, Oliver J. Woodford, Meng Jiang, and Neil Shah. Data augmentation for graph neural networks. In AAAI, 2021.
Yan Zhou, Murat Kantarcioglu, and Bhavani M. Thuraisingham. Self-training with selection-by- rejection. In ICDM, pp. 795-803, 2012.
11


Reviews:
[{"review_id": "u2YHUL6K_4o", "reviewer": "Reviewer_wfd3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1. This paper proposes a stabilized self-training technique which the pseudo labels are update adaptively per epoch and the weights of different pseudo labels in the loss function are proposed to address the unstable issue of existing GNNs.\n2. The author chooses negative samples from the nodes that are not directly connected and designs a new loss function to make positive and negative samples as different as possible.\n3. The classification accuracy are reported when varying the number of labeled nodes per class in {1,3,5,10,20}, the results show the good performance compared with the baselines.\n\nWeaknesses: \n1. The proposed algorithm has some new ideas, such as negative samples, but the model of GCN is not changed.\n2. The authors said that the unstable issue of existing GNNs can be addressed by the weights of different pseudo labels in the loss function, which named stabilizer in the paper. However, it has not been clearly proved by theoretical analysis or experiment. \n3. The choices of negative samples and positive samples have some uncertainties. The authors should show the effect of the uncertainties. \n4. Although the authors compared the proposed method with some unsupervised methods and supervised methods, some self-training or pseudo labeling methods for GCN or semi-supervised learning should be considered.\n5. The performance of the method is not fully verified by experiment, for example, (1) the importance of stabilizer and the number of positive and negative samples are not reported. (2) the performance comparison between the proposed pseudo labels and existing pseudo labels.\n\nMinor problem:\n1. The figures can not match with the words in the body of article. For example, “The dashed lines are the ground-truth percentage of each class in the Cora dataset” or “the average number of predicted labels in percentage per class at each epoch”\n2. There are many problems in the article, some of the description details remain in doubt, some of the statements are not exact. For example, (1) In the second paragraph of 3.2, “Fi representing the output probability vector of node vi ∈ V”, however, V is the node set. (2) In the last paragraph of 3.2, the author proposed “a pseudo label in previous epoch will be removed in the next epoch if its confidence becomes low.”, however, it hasn’t been depicted in the paper. \n", "summary_of_the_paper": "This paper presents a self-training GCN framework. It refines the predicted results as pseudo labels and pseudo negative labels to train GCN, which is applicable to existing GCNs to stabilize the training process and enhance the training data.", "summary_of_the_review": "The present paper presents an effective framework for semi-supervised node classification on few-labeled graph data. The paper is well organized and the motivation of this paper is clear. Although this paper has some innovation, but the model of GCN is not changed. Besides, some comparison experiments with pseudo labeling methods are lack. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "t0WpbOOGMEY", "reviewer": "Reviewer_DW1F", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1. This paper addresses an important problem of semi-supervised node classification with very few training labels. \n2. The few label problem is well motivated with a clear problem formulation. \n\nWeakness:\n1. The technical novelty of this paper is very limited. The proposed method is simply adapted from existing self-training and negative sampling approaches with incremental changes. \n2. The reweighing scheme for pseudo labels is somewhat ad-hoc. The authors need to provide more convincing justifications why the reweighing approach can help mitigate the training unstability.\n3. The proposed negative sampling method operates on labels rather than embeddings. However, When you choose negative samples that are not directly connected to a positive sample vi, there is no guarantee that the positive and negative sample would have different labels. How would you address this issue?\n4. The proposed method is not compared with more recent baselines on self-training methods, for example,\nZiang Zhou, Shenzhong Zhang, and Zengfeng Huang. 2019.   Dynamic Self-training  Framework  for  Graph  Convolutional Networks.arXiv preprintarXiv:1910.02684(2019)\nKe Sun, Zhanxing Zhu, and Zhouchen Lin. 2020.  Multi-Stage Self-Supervised Learning for Graph Convolutional Networks. InAAAI.\n5. The writing and clarity of the paper need be improved for better readability.", "summary_of_the_paper": "This paper proposes a stabilized self-training with negative sampling method to improve GNN performance on the node classification task when only few labels are provided for training. The main idea of this paper is to use the reweighted self-training method and the negative sampling method to stabilize GNN. ", "summary_of_the_review": "The technical novelty and originality of the paper are limited. The proposed method is an adaptation of the existing self-training and negative sampling strategies. This paper lacks comparisons with more recent self-training baselines to prove the effectiveness of the proposed method.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "slwVVHD-ocl", "reviewer": "Reviewer_s9Vo", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Both the problem setting and technical contribution of this paper is problematic. First of all, the proposed method is basically a bootstrapping-style self-training method, it needs to compare with other self-training methods instead claiming it can improve the performance. The comparison in the experiment is unfair, as proposed method utilizes way more training seeds. Second, the **unstable** phenomenon is not defined in a mathematical way. It's widely known that few training seeds leads to larger variance. The terminology here looks like proposing a new measurement without any mathematical support.\n\n1. **Lack of proper baselines**\nAs a self-training method, author should compare more baselines that considers additional training inputs. Also, experiments on larger benchmarks are necessary. It's a common sense that an algorithm can work well on Cora/Citeseer/PubMed because the network is simple and small.\n\n2. **Limited novelty**\nThe threshold-based detector has been widely used in out-of-distribution detection [1]. The proposed method is neither novel nor insightful.\n\n\n\n[1] Ren, Jie, et al. \"Likelihood Ratios for Out-of-Distribution Detection.\" Advances in Neural Information Processing Systems 32 (2019): 14707-14718.", "summary_of_the_paper": "This paper discusses *unstable* training procedure of graph neural network when training data is extremely limited. The author proposes use self-training and negative sampling to mitigate this issue. Specifically, it uses high-confidence prediction of the model as training seeds and control the importance by population. The result shows improvement on several node classification dataset.", "summary_of_the_review": "Overall, I recommend a rejection to this paper. The problem it studied is a common challenge (amount of training data vs. variance) in semi-supervised learning, while experiments are not conducted properly to support the effectiveness of it against other similar approaches. Moreover, the technical writing in the paper contains a lot of hand-waving example instead of sound mathematical justifications.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "GRST13devsi", "reviewer": "Reviewer_S1e4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths：\n1. The proposed problem is interesting in the graph learning community. The problem is similar to the topic of few-shot learning on graphs, which usually assumes that only a few labeled nodes or graphs are available. Compared with the existing few-shot learning methods, the paper assumes all the unlabelled nodes are available at the training stage and addresses it with popular training technics such as self-training and negative sampling.\n2. The paper is well organized with a clear problem definition and technical solution.\n3. The proposed two strategies are easy to be reproduced.\n\n\nWeaknesses:\n1. Although the paper tries to address the challenging and exciting problem, i.e., semi-supervised node classification on graphs with an extreme case when only a small subset of nodes have class labels, my major concern is that the self-training and negative sampling are standard methods and have been discussed in many graph learning papers, as referenced by this paper. Therefore, without significant technical improvement, I think the technical contribution is limited. \n2. I suggest the authors to make the Figure 1 more clear. It is not apparent to see the unstable training of GCN and DAGNN compared with the improved ones, due to the considerable overlap between 7 classes. For example, the variance of Class 3 in (a) seems to be smaller than the improved version shown in (c). \n3. For the stabilized self-training, the ablation experiments need to be performed to show the differences between the proposed stabilized self-training and existing self-training. \n\n4. Moreover, in implementing the stabilizer, a re-weighted pseudo-label loss (see Eq 5) is proposed, and the importance of each node is dependent on the number of their belonging classes. How to deal with the situation when a class has no node whose confidence is beyond the threshold? How to deal with the situation when a class dominates the most nodes on a graph? I suggest incorporating the neighbor information to design a more adaptive stabilizer.\n\n5. The negative sampling method, “The negative samples of a positive sample vi are the nodes that are not directly connected to vi in the graph G”, may bring false negative samples because even two nodes are not directly connected, they still have the same class. How to deal with this problem? \n6. I think the work can be better if the author considers problem 4 and 5. \n", "summary_of_the_paper": "This paper considers the semi-supervised node classification on graph data with only a few node labels available. Under this extreme situation, the paper demonstrates the unstable performances of existing GNNs, such as GCN and DAGNN. To address the unstable problem, the paper proposes two strategies consisting of pseudo-labelling based self-training and negative-sampling based regularization. The experiments show that the two strategies work well on graph data with only a few node labels.", "summary_of_the_review": "I think the main contribution is: the proposed problem is challenging and interesting, this paper considers the semi-supervised node classification on graph data with only a few node labels available. The main weakness is: the technical contribution is limited. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "8T3-_eeLVGp", "reviewer": "Reviewer_XgQF", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength \n\n1 - Propose a new method for node classification with few-shot labels.\n\n2 - Presentation is clear for me. \n\nWeakness\n\n1 - The novelty of this work is limited. \n\n2 - Miss important related works. \n\n3 - Experiments should be improved.   \n\nDetailed Review\n\nThe novelty of proposed method is limited for me. The major contribution: data augmentation strategy (pseudo label) and negative sampling regularization, is simple and intuitively, depending on the threshold value. There are many works proposing better data augmentation strategies or pseudo label generation, such as:\n\nGraph contrastive learning with augmentations, NeurIPS 2020 \n\nBig Self-Supervised Models are Strong Semi-Supervised Learners, NeurIPS 2020\n\nMore importantly, this paper lacks discussion and comparison of important related works. The authors seem to be not aware of graph few-shot learning works. There are many studies working on graph learning with few-shot labels, such as followings:\n\nGraph meta learning via local subgraphs, NeurIPS 2020\n\nGraph few-shot learning via knowledge transfer, AAAI 2020\n\nMeta-gnn: On few-shot node classification in graph meta-learning, CIKM 2019\n\nNode classification on graphs with few-shot novel labels via meta transformed network embedding, NeurIPS 2020", "summary_of_the_paper": "This paper proposes a self-training method with negative sampling for node classification on few-labeled graph data. The proposed method applies data augmentation (i.e., pseudo label) and negative sampling regularization to augment node classification model. Experiments are conducted to show that the proposed method outperforms some baseline methods. ", "summary_of_the_review": "The novelty of this work is limited. It lacks important related work (graph few-shot learning) discussion and comparison. In my opinion, this work is not suitable for ICLR. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper studies stability issues of GNN training when data are limited. The key contribution of this work is to use reweighted self-training and negative sampling to stabilize GNN. Multiple reviewers raised major concerns on the technical novelty, experimental setup, comparison, and results. No response was provided during discussion. I recommend this submission be rejected.
Decision: Reject