Stabilized Self-training with Negative Sampling on Few-labeled Graph Data

Specifically , we observe that existing GNNs suffer from unstable training process on few-labeled graph data , resulting to inferior performance on node classification . Therefore , we propose an effective framework , Stabilized self-training with Negative sampling ( SN ) , which is applicable to existing GNNs to stabilize the training process and enhance the training data , and consequently , boost classification accuracy on graphs with few labeled data . In experiments , we apply our SN framework to two existing GNN base models ( GCN and DAGNN ) to get SNGCN and SNDAGNN , and evaluate the two methods against 13 existing solutions over 4 benchmarking datasets . Extensive experiments show that the proposed SN framework is highly effective compared with existing solutions , especially under settings with very few labeled data . In particular , on a benchmark dataset Cora with only 1 labeled node per class , while GCN only has 44.6 % accuracy , SNGCN achieves 62.5 % accuracy , improving GCN by 17.9 % ; SNDAGNN has accuracy 66.4 % , improving that of the base model DAGNN ( 59.8 % ) by 6.6 % . 1 INTRODUCTION . Graph is an expressive data model , representing objects and the relationships between objects as nodes and edges respectively . Graph data are ubiquitous with a wide range of real-world applications , e.g. , social network analysis ( Qiu et al. , 2018 ; Li & Goldwasser , 2019 ) , traffic network prediction ( Guo et al. , 2019 ; Li et al. , 2019 ) , protein interface prediction ( Fout et al. , 2017 ) , recommendation systems ( Fan et al. , 2019 ; Yang et al. , 2020a ) . Among these applications , an important task is to classify the nodes in a graph into various classes . However , one tough situation commonly existing is the lack of sufficient labeled data , which are also expensive to collect . To ease the situation , semi-supervised node classification on graphs has attracted much attention from both industry ( Qiu et al. , 2018 ; Li & Goldwasser , 2019 ) and academia ( Defferrard et al. , 2016 ; Hamilton et al. , 2017 ; Velickovic et al. , 2018 ; Liu et al. , 2020 ; Li et al. , 2018 ; Klicpera et al. , 2019 ) . It aims to leverage a small amount of labeled nodes and additionally a large amount of unlabeled nodes in a graph to train an accurate classifier . There exists a collection of graph neural networks for semi-supervised node classification ( Kipf & Welling , 2017 ; Velickovic et al. , 2018 ; Monti et al. , 2017 ; Hamilton et al. , 2017 ; Klicpera et al. , 2019 ; Liu et al. , 2020 ) . For instance , Graph convolution networks ( GCNs ) rely on a message passing scheme called graph convolution that aggregates the neighborhood information of a node , including node features and graph topology , to learn node representations , which can then be used in downstream classification tasks ( Kipf & Welling , 2017 ) . Despite the great success of GCNs , under the extreme cases when very few labels are given ( e.g. , only one labeled node per class ) , the shallow GCN architecture , typically with two layers ( Kipf & Welling , 2017 ) , can not effectively propagate the training labels over the input graph , leading to inferior performance . In particular , as shown in our experiments , on a benchmark dataset Cora with 1 labeled node per class ( Cora-1 ) , GCN is even less accurate than some unsupervised methods , such as DGI ( Velickovic et al. , 2019 ) and G2G ( Bojchevski & Günnemann , 2018 ) . Recently , several latest studies try to improve classification accuracy by designing deeper GNN architectures , e.g. , DAGNN ( Liu et al. , 2020 ) , which also address the over-smoothing issue identified in ( Xu et al. , 2018 ; Li et al. , 2018 ; Chen et al. , 2020a ) . However , these deep GNNs are still not directly designed to tackle the scarcity of labeled data , especially when only very few labels are available . After conducting an in-depth study , we have an important finding that existing GNNs suffer from unstable training process , when labeled nodes are few . In particular , on Cora dataset with 7 classes , for each run , we randomly select 1 labeled node per class as the training data for both GCN and DAGNN , and repeat 100 runs with 300 epochs per run , to get the average number of predicted labels in percentage per class at each epoch and also the standard deviation . The statistical results of GCN and DAGNN are shown in Figures 1 ( a ) and 1 ( b ) respectively . x-axis is the epoch from 0 to 300 , and y-axis is the percentage of a class in the predicted node labels . There are 7 colored lines representing the average percentage of the predicted labels of the respective classes , as the epoch increases . The dashed lines are the ground-truth percentage of each class in the Cora dataset . The shaded areas in colors represent the standard deviation . Observe that in Figure 1 ( a ) , GCN has high variance at different runs when predicting node labels , and the variance keeps large at late epochs , e.g. , 300 , which indicates that GCN is quite unstable at different runs with 1 training label per class sampled randomly , leading to inferior classification accuracy as illustrated in our experiments . Moreover , as shown in Figure 1 ( b ) , DAGNN also suffers from unstable training process . The variance of DAGNN is relatively smaller than that of GCN , which provides a hint about why DAGNN performs better than GCN . Nevertheless , both GCN and DAGNN yield unstable training process with large variance . Since there is only 1 labeled node per class in Cora-1 , at different runs , the randomly sampled training nodes can heavily influence the message passing process in GCN and DAGNN , depending on the connectivity of the training nodes to their neighborhoods over the graph topology , which result to the unstable training process observed above . To address the unstable training process of existing GNNs when only very few labeled data are available , we propose a framework , Stabilized self-training with Negative sampling ( SN ) , which is readily applicable to existing GNNs to improve classification accuracy via stabilized training process . In the proposed SN framework , at each epoch , we select a set of nodes with predicted labels of high confidence as pseudo labels and add such pseudo labels into training data to enhance the training of next epoch . To tackle the unstable issue of existing GNNs , we develop a stabilizing technique in self-training to balance the training . We then design a negative sampling regularization technique over pseudo labels to further improve node classification accuracy . In experiments , we apply our SN framework to GCN and DAGNN , denoted as SNGCN and SNDAGNN respectively . Figures 1 ( c ) and 1 ( d ) report the average percentage and standard variance of the predicted labels per class per epoch of SNGCN and SNDAGNN on Cora-1 respectively . With our stabilized selftraining technique , obviously , the variance of SNGCN in Figure 1 ( c ) decreases quickly and becomes stable as epoch increases , compared with Figure 1 ( a ) of GCN . SNDAGNN is also more stable than DAGNN as shown in Figures 1 ( d ) and 1 ( b ) respectively . As reported later in experiments , with the proposed SN framework , SNGCN achieves 62.5 % node classification accuracy on Cora-1 , significantly improving GCN ( 44.6 % ) by 17.9 % , and SNDAGNN obtains 66.4 % accuracy on Cora-1 and outperforms DAGNN ( 59.8 % ) by a substantial margin of 6.6 % . We conduct extensive experiments on 4 benchmarking datasets , and compare with 13 existing solutions , to evaluate the performance of the proposed SN framework . Experimental results demonstrate that our SN framework is able to significantly improve classification accuracy of existing GNNs when only few labels are available , and is also effective when training labels are sufficient . 2 RELATED WORK . In literature , there are two directions to address the scarcity of labeled data for semi-supervised node classification : ( i ) explore multi-hop graph topological features to propagate the labels in L over the input graph , e.g. , GCN ( Kipf & Welling , 2017 ) and DAGNN ( Liu et al. , 2020 ) ; ( ii ) enhance the training data by pseudo labels ( self-training ) ( Li et al. , 2018 ) or augmenting graph data by new edges and features ( Kong et al. , 2020 ; Zhao et al. , 2021 ) . Note that these two directions are not mutually exclusive , but can work together on few-labeled graph data . Here we review the existing studies that are most relevant to this paper . GNNs . There exist a large collection of GNNs , such as GCN , DAGNN , GAT , MoNet , and APPNP ( Kipf & Welling , 2017 ; Liu et al. , 2020 ; Bruna et al. , 2014 ; Henaff et al. , 2015 ; Defferrard et al. , 2016 ; Velickovic et al. , 2018 ; Monti et al. , 2017 ; Chen et al. , 2020b ; Klicpera et al. , 2019 ) . We introduce the details of GCN ( Kipf & Welling , 2017 ) and DAGNN ( Liu et al. , 2020 ) here . GCN learns the representation of each node by iteratively aggregating the representations of its neighbors . Specifically , GCN consists of k > 0 layers , each with the same propagation rule defined as follows . At the ℓ-th layer , the representations H ( ℓ−1 ) of previous layer are aggregated to get H ( ℓ ) . H ( ℓ ) = σ ( ÂH ( ℓ−1 ) W ( ℓ ) ) , ℓ = 1 , 2 , ... , k. ( 1 ) Â = D̃− 1 2 ÃD̃ − 1 2 is the graph laplacian , where Ã = A + I is the adjacency matrix of G after adding self-loops ( I is the identity matrix ) and D̃ is a diagonal matrix with D̃ii = ∑ j Ãij . W ( ℓ ) is a trainable weight matrix of the ℓ-th layer , and σ is a nonlinear activation function . Initially , H ( 0 ) = X . Note that GCN usually achieves superior performance with 1-layer or 2-layer models ( Kipf & Welling , 2017 ) . When applying multiple layers to leverage large receptive fields , the performance degrades severely , due to the over-smoothing issue identified in ( Xu et al. , 2018 ; Li et al. , 2018 ; Chen et al. , 2020a ) . A recent deep GNN architecture , DAGNN , tackles the over-smoothing issue and achieves state-of-the-art results by decoupling representation transformation and propagation in GNNs ( Liu et al. , 2020 ) . Then it utilizes an adaptive adjustment mechanism to balance the information from local and global neighborhoods of each node . Specifically , the mathematical expression of DAGNN is as follows . DAGNN uses a learnable parameter s ∈ Rc×1 to adjust the weight of embeddings at different propagation level ( from 1 to k ) . It processes data in the following way . Z = MLP ( X ) ∈ Rn×c , Hℓ = Â ℓ · Z ∈ Rn×c , ℓ = 1 , 2 , ... , k , Sℓ = Hℓ · s ∈ R n×1 , ℓ = 1 , 2 , ... , k , Ŝℓ = [ Sℓ , Sℓ , ... , Sℓ ] ∈ R n×c , ℓ = 1 , 2 , ... , k , Xout = softmax ( ∑k ℓ=1 Hℓ⊙ Ŝℓ ) , where Â ℓ is the ℓ-th power of matrix Â , ⊙ is the Hadamard product , · is dot product , MLP is the Multilayer Perceptron and softmax operation is on the second dimension . Data Augmentation . Another way to address the situation of limited labeled data is to add pseudo labels to training dataset by self-training ( Li et al. , 2018 ) , or enhance the graph data by adding new edges and features ( Zhao et al. , 2021 ; Kong et al. , 2020 ) . Self-training itself is a general methodology ( Scudder , 1965 ) and is used in various domains in addition . It is used in word-sense disambiguation ( Yarowsky , 1995 ; Hearst , 1991 ) , bootstrap for information extraction and learning subjective nouns ( Riloff & Jones , 1999 ) , and text classification ( Nigam et al. , 2000 ) . In ( Zhou et al. , 2012 ) , it suggests that selecting informative unlabeled data using a guided search algorithm can significantly improve performance over standard self-training framework . Buchnik & Cohen ( 2018 ) mainly consider selftraining for diffusion-based techniques . Recently , self-training has been adopted for semi-supervised tasks on graphs . For instance , Li et al . ( 2018 ) propose self-training and co-training techniques for GCN . This self-training work selects the top-k confident predicted labels as pseudo labels . The co-training technique co-trains a GCN with a random walk model to handle few-labeled data . Compared with existing self-training work , our framework are different as shown later . In particular , our framework has a different strategy to select pseudo labels and also has a stabilizer to address the deficiencies of existing GNNs ; moreover , we propose a negative sampling regularization technique to further boost accuracy . Besides , in existing work , if a node is selected as a pseudo label , it will never be moved out even if the pseudo label becomes obviously wrong in later epochs . On the other hand , in our framework , we update pseudo labels in each epoch to avoid such an issue . There also exist studies to augment the original graph data , which is different from self-training . For instance , Zhao et al . ( 2021 ) utilize link prediction to promote intra-class edges and demote inter-class edges in a given graph . Kong et al . ( 2020 ) iteratively augment node features with gradient-based adversarial perturbations to enhance the performance of GNNs .

Summary: This paper proposes a self-training method with negative sampling for node classification on few-labeled graph data. The proposed method applies data augmentation (i.e., pseudo label) and negative sampling regularization to augment node classification model. Experiments are conducted to show that the proposed method outperforms some baseline methods.