Stabilized Self-training with Negative Sampling on Few-labeled Graph Data

Specifically , we observe that existing GNNs suffer from unstable training process on few-labeled graph data , resulting to inferior performance on node classification . Therefore , we propose an effective framework , Stabilized self-training with Negative sampling ( SN ) , which is applicable to existing GNNs to stabilize the training process and enhance the training data , and consequently , boost classification accuracy on graphs with few labeled data . In experiments , we apply our SN framework to two existing GNN base models ( GCN and DAGNN ) to get SNGCN and SNDAGNN , and evaluate the two methods against 13 existing solutions over 4 benchmarking datasets . Extensive experiments show that the proposed SN framework is highly effective compared with existing solutions , especially under settings with very few labeled data . In particular , on a benchmark dataset Cora with only 1 labeled node per class , while GCN only has 44.6 % accuracy , SNGCN achieves 62.5 % accuracy , improving GCN by 17.9 % ; SNDAGNN has accuracy 66.4 % , improving that of the base model DAGNN ( 59.8 % ) by 6.6 % . 1 INTRODUCTION . Graph is an expressive data model , representing objects and the relationships between objects as nodes and edges respectively . Graph data are ubiquitous with a wide range of real-world applications , e.g. , social network analysis ( Qiu et al. , 2018 ; Li & Goldwasser , 2019 ) , traffic network prediction ( Guo et al. , 2019 ; Li et al. , 2019 ) , protein interface prediction ( Fout et al. , 2017 ) , recommendation systems ( Fan et al. , 2019 ; Yang et al. , 2020a ) . Among these applications , an important task is to classify the nodes in a graph into various classes . However , one tough situation commonly existing is the lack of sufficient labeled data , which are also expensive to collect . To ease the situation , semi-supervised node classification on graphs has attracted much attention from both industry ( Qiu et al. , 2018 ; Li & Goldwasser , 2019 ) and academia ( Defferrard et al. , 2016 ; Hamilton et al. , 2017 ; Velickovic et al. , 2018 ; Liu et al. , 2020 ; Li et al. , 2018 ; Klicpera et al. , 2019 ) . It aims to leverage a small amount of labeled nodes and additionally a large amount of unlabeled nodes in a graph to train an accurate classifier . There exists a collection of graph neural networks for semi-supervised node classification ( Kipf & Welling , 2017 ; Velickovic et al. , 2018 ; Monti et al. , 2017 ; Hamilton et al. , 2017 ; Klicpera et al. , 2019 ; Liu et al. , 2020 ) . For instance , Graph convolution networks ( GCNs ) rely on a message passing scheme called graph convolution that aggregates the neighborhood information of a node , including node features and graph topology , to learn node representations , which can then be used in downstream classification tasks ( Kipf & Welling , 2017 ) . Despite the great success of GCNs , under the extreme cases when very few labels are given ( e.g. , only one labeled node per class ) , the shallow GCN architecture , typically with two layers ( Kipf & Welling , 2017 ) , can not effectively propagate the training labels over the input graph , leading to inferior performance . In particular , as shown in our experiments , on a benchmark dataset Cora with 1 labeled node per class ( Cora-1 ) , GCN is even less accurate than some unsupervised methods , such as DGI ( Velickovic et al. , 2019 ) and G2G ( Bojchevski & Günnemann , 2018 ) . Recently , several latest studies try to improve classification accuracy by designing deeper GNN architectures , e.g. , DAGNN ( Liu et al. , 2020 ) , which also address the over-smoothing issue identified in ( Xu et al. , 2018 ; Li et al. , 2018 ; Chen et al. , 2020a ) . However , these deep GNNs are still not directly designed to tackle the scarcity of labeled data , especially when only very few labels are available . After conducting an in-depth study , we have an important finding that existing GNNs suffer from unstable training process , when labeled nodes are few . In particular , on Cora dataset with 7 classes , for each run , we randomly select 1 labeled node per class as the training data for both GCN and DAGNN , and repeat 100 runs with 300 epochs per run , to get the average number of predicted labels in percentage per class at each epoch and also the standard deviation . The statistical results of GCN and DAGNN are shown in Figures 1 ( a ) and 1 ( b ) respectively . x-axis is the epoch from 0 to 300 , and y-axis is the percentage of a class in the predicted node labels . There are 7 colored lines representing the average percentage of the predicted labels of the respective classes , as the epoch increases . The dashed lines are the ground-truth percentage of each class in the Cora dataset . The shaded areas in colors represent the standard deviation . Observe that in Figure 1 ( a ) , GCN has high variance at different runs when predicting node labels , and the variance keeps large at late epochs , e.g. , 300 , which indicates that GCN is quite unstable at different runs with 1 training label per class sampled randomly , leading to inferior classification accuracy as illustrated in our experiments . Moreover , as shown in Figure 1 ( b ) , DAGNN also suffers from unstable training process . The variance of DAGNN is relatively smaller than that of GCN , which provides a hint about why DAGNN performs better than GCN . Nevertheless , both GCN and DAGNN yield unstable training process with large variance . Since there is only 1 labeled node per class in Cora-1 , at different runs , the randomly sampled training nodes can heavily influence the message passing process in GCN and DAGNN , depending on the connectivity of the training nodes to their neighborhoods over the graph topology , which result to the unstable training process observed above . To address the unstable training process of existing GNNs when only very few labeled data are available , we propose a framework , Stabilized self-training with Negative sampling ( SN ) , which is readily applicable to existing GNNs to improve classification accuracy via stabilized training process . In the proposed SN framework , at each epoch , we select a set of nodes with predicted labels of high confidence as pseudo labels and add such pseudo labels into training data to enhance the training of next epoch . To tackle the unstable issue of existing GNNs , we develop a stabilizing technique in self-training to balance the training . We then design a negative sampling regularization technique over pseudo labels to further improve node classification accuracy . In experiments , we apply our SN framework to GCN and DAGNN , denoted as SNGCN and SNDAGNN respectively . Figures 1 ( c ) and 1 ( d ) report the average percentage and standard variance of the predicted labels per class per epoch of SNGCN and SNDAGNN on Cora-1 respectively . With our stabilized selftraining technique , obviously , the variance of SNGCN in Figure 1 ( c ) decreases quickly and becomes stable as epoch increases , compared with Figure 1 ( a ) of GCN . SNDAGNN is also more stable than DAGNN as shown in Figures 1 ( d ) and 1 ( b ) respectively . As reported later in experiments , with the proposed SN framework , SNGCN achieves 62.5 % node classification accuracy on Cora-1 , significantly improving GCN ( 44.6 % ) by 17.9 % , and SNDAGNN obtains 66.4 % accuracy on Cora-1 and outperforms DAGNN ( 59.8 % ) by a substantial margin of 6.6 % . We conduct extensive experiments on 4 benchmarking datasets , and compare with 13 existing solutions , to evaluate the performance of the proposed SN framework . Experimental results demonstrate that our SN framework is able to significantly improve classification accuracy of existing GNNs when only few labels are available , and is also effective when training labels are sufficient . 2 RELATED WORK . In literature , there are two directions to address the scarcity of labeled data for semi-supervised node classification : ( i ) explore multi-hop graph topological features to propagate the labels in L over the input graph , e.g. , GCN ( Kipf & Welling , 2017 ) and DAGNN ( Liu et al. , 2020 ) ; ( ii ) enhance the training data by pseudo labels ( self-training ) ( Li et al. , 2018 ) or augmenting graph data by new edges and features ( Kong et al. , 2020 ; Zhao et al. , 2021 ) . Note that these two directions are not mutually exclusive , but can work together on few-labeled graph data . Here we review the existing studies that are most relevant to this paper . GNNs . There exist a large collection of GNNs , such as GCN , DAGNN , GAT , MoNet , and APPNP ( Kipf & Welling , 2017 ; Liu et al. , 2020 ; Bruna et al. , 2014 ; Henaff et al. , 2015 ; Defferrard et al. , 2016 ; Velickovic et al. , 2018 ; Monti et al. , 2017 ; Chen et al. , 2020b ; Klicpera et al. , 2019 ) . We introduce the details of GCN ( Kipf & Welling , 2017 ) and DAGNN ( Liu et al. , 2020 ) here . GCN learns the representation of each node by iteratively aggregating the representations of its neighbors . Specifically , GCN consists of k > 0 layers , each with the same propagation rule defined as follows . At the ℓ-th layer , the representations H ( ℓ−1 ) of previous layer are aggregated to get H ( ℓ ) . H ( ℓ ) = σ ( ÂH ( ℓ−1 ) W ( ℓ ) ) , ℓ = 1 , 2 , ... , k. ( 1 ) Â = D̃− 1 2 ÃD̃ − 1 2 is the graph laplacian , where Ã = A + I is the adjacency matrix of G after adding self-loops ( I is the identity matrix ) and D̃ is a diagonal matrix with D̃ii = ∑ j Ãij . W ( ℓ ) is a trainable weight matrix of the ℓ-th layer , and σ is a nonlinear activation function . Initially , H ( 0 ) = X . Note that GCN usually achieves superior performance with 1-layer or 2-layer models ( Kipf & Welling , 2017 ) . When applying multiple layers to leverage large receptive fields , the performance degrades severely , due to the over-smoothing issue identified in ( Xu et al. , 2018 ; Li et al. , 2018 ; Chen et al. , 2020a ) . A recent deep GNN architecture , DAGNN , tackles the over-smoothing issue and achieves state-of-the-art results by decoupling representation transformation and propagation in GNNs ( Liu et al. , 2020 ) . Then it utilizes an adaptive adjustment mechanism to balance the information from local and global neighborhoods of each node . Specifically , the mathematical expression of DAGNN is as follows . DAGNN uses a learnable parameter s ∈ Rc×1 to adjust the weight of embeddings at different propagation level ( from 1 to k ) . It processes data in the following way . Z = MLP ( X ) ∈ Rn×c , Hℓ = Â ℓ · Z ∈ Rn×c , ℓ = 1 , 2 , ... , k , Sℓ = Hℓ · s ∈ R n×1 , ℓ = 1 , 2 , ... , k , Ŝℓ = [ Sℓ , Sℓ , ... , Sℓ ] ∈ R n×c , ℓ = 1 , 2 , ... , k , Xout = softmax ( ∑k ℓ=1 Hℓ⊙ Ŝℓ ) , where Â ℓ is the ℓ-th power of matrix Â , ⊙ is the Hadamard product , · is dot product , MLP is the Multilayer Perceptron and softmax operation is on the second dimension . Data Augmentation . Another way to address the situation of limited labeled data is to add pseudo labels to training dataset by self-training ( Li et al. , 2018 ) , or enhance the graph data by adding new edges and features ( Zhao et al. , 2021 ; Kong et al. , 2020 ) . Self-training itself is a general methodology ( Scudder , 1965 ) and is used in various domains in addition . It is used in word-sense disambiguation ( Yarowsky , 1995 ; Hearst , 1991 ) , bootstrap for information extraction and learning subjective nouns ( Riloff & Jones , 1999 ) , and text classification ( Nigam et al. , 2000 ) . In ( Zhou et al. , 2012 ) , it suggests that selecting informative unlabeled data using a guided search algorithm can significantly improve performance over standard self-training framework . Buchnik & Cohen ( 2018 ) mainly consider selftraining for diffusion-based techniques . Recently , self-training has been adopted for semi-supervised tasks on graphs . For instance , Li et al . ( 2018 ) propose self-training and co-training techniques for GCN . This self-training work selects the top-k confident predicted labels as pseudo labels . The co-training technique co-trains a GCN with a random walk model to handle few-labeled data . Compared with existing self-training work , our framework are different as shown later . In particular , our framework has a different strategy to select pseudo labels and also has a stabilizer to address the deficiencies of existing GNNs ; moreover , we propose a negative sampling regularization technique to further boost accuracy . Besides , in existing work , if a node is selected as a pseudo label , it will never be moved out even if the pseudo label becomes obviously wrong in later epochs . On the other hand , in our framework , we update pseudo labels in each epoch to avoid such an issue . There also exist studies to augment the original graph data , which is different from self-training . For instance , Zhao et al . ( 2021 ) utilize link prediction to promote intra-class edges and demote inter-class edges in a given graph . Kong et al . ( 2020 ) iteratively augment node features with gradient-based adversarial perturbations to enhance the performance of GNNs .

Summary: This paper discusses *unstable* training procedure of graph neural network when training data is extremely limited. The author proposes use self-training and negative sampling to mitigate this issue. Specifically, it uses high-confidence prediction of the model as training seeds and control the importance by population. The result shows improvement on several node classification dataset.