FASG: Feature Aggregation Self-training GCN for Semi-supervised Node Classification

1 INTRODUCTION . Graph convolutional network ( GCN ) can be seen as the migration of convolutional neural network ( CNN ) on non-Euclidean structure data . Due to its its excellent ability in representation learning , GCN has achieved significant success in many graph-based learning tasks , including node clustering , graph classification and link prediction ( Dwivedi et al. , 2020 ) . Kipf & Welling ( 2016 ) proposed a GCN mode from the perspective of spectrogram theory and validated its effectiveness on semi-supervised node classification task . Subsequent models such as GraphSAGE ( Hamilton et al. , 2017 ) , GAT ( Veličković et al. , 2017 ) , SGCN ( Wu et al. , 2019 ) and APPNP ( Klicpera et al. , 2018 ) designed more sophisticated neighborhood aggregation functions from spatial or spectral views . These methods obtain much more effective results on semi-supervised node classification than traditional methods such as MLP , DeepWalk ( Perozzi et al. , 2014 ) , etc . However , the prediction accuracy of such GCN models depends largely on the quantity and quality of supervised information , and it will decrease significantly when the quantity of labeled nodes is quite small ( Li et al. , 2018 ) . The main reason lies that scarce supervised information is difficult to be spread far away in the graph so that unlabeled nodes are hardly to make full use of supervised information for prediction . Addressing the above issue , many studies have been devoted to improving the representation ability by designing multi-layer GCN model ( Li et al. , 2019 ) . However , the representation ability of GCN , as illustrated in Kipf & Welling ( 2016 ) , can hardly be improved by simply stacking layers just like MLP . Moreover , stacking too many layers tends to cause over-smoothing ( Xu et al. , 2018 ) that makes all node embeddings indistinguishable . Alternatively , Li et al . ( 2018 ) proposed to improve the reasoning ability of GCN models by applying self-training techniques on the training . Rather than trying to enhance the expressive ability of the model , the self-training strategy prefers to expand the supervised information by adding unlabeled nodes with high confidences to the training set at each round . Following this line , Sun et al . ( 2019 ) proposed a multi-stage self-training strategy ( M3S ) to enrich the training set , which uses deep cluster ( Caron et al. , 2018 ) and an aligning mechanism to generate pseudo-labels of nodes for updating of the training set . Later , Zhou et al . ( 2019 ) proposed a dynamic self-training framework to continuously refresh the training set by directly using the output of GCN without a checking part . In general these self-training algorithms generate pseudolabels using relatively simple checking mechanism , which may introduce false labels as supervision information and prevent the improvement of prediction accuracy . In this paper , we propose a novel feature aggregation self-training GCN ( FASG ) algorithm for semisupervised node classification . We firstly propose a lightweight classifier that applies linear SVM on aggregated node features , and validate that it achieves comparable performance to popular GCN approaches . Furthermore , this classifier is served as a checking part in the multi-round training process to generate pseudo-labels , which are used to filter unreliable nodes when expanding the supervised information . By fully considering the structural information of graph nodes , the newly developed checking part is able to improve the accuracy of the generated pseudo-labels and finally boost the node classification . Finally , we illustrate that the proposed self-training strategy can be integrated with various existing GCN models to improve the prediction performance . The proposed algorithms is validated in comparison with several state-of-the-art baseline algorithms in three public benchmarks , and the experimental results illustrate that the proposed algorithm outperforms all compared algorithms in general on all benchmarks . We will release the source code upon publication of this paper . 2 RELATED WORK . In the past decade CNN has achieved great success in many areas of machine learning ( Krizhevsky et al. , 2012 ; LeCun et al. , 1998 ; Sermanet et al. , 2012 ) , but its applications are mainly restricted in dealing with Euclidean structure data ( Bruna et al. , 2013 ) . Consequently , in recent years more and more studies are devoted to learning the representation on non-Euclidean structure data such as graph . Graph neural network ( GNN ) plays an important role in the field of graph representation learning , which can learn the representation of nodes or the whole graph . There are many famous GNN architectures including GCN ( Kipf & Welling , 2016 ) , graph recurrent neural network ( Hajiramezanali et al. , 2019 ) and graph autoencoder ( Pan et al. , 2018 ) . As one of the most important architecture of GNN , GCN can be roughly categorized into spectral and spatial approaches . The spectral approaches ( Bruna et al. , 2013 ) define convolution operation by Laplacian feature decomposition of the graph , thereby filtering the graph structure in the spectral domain . On the basis of the Chebyshev polynomial ( Defferrard et al. , 2016 ) of the graph Laplacian matrix , Kipf & Welling ( 2016 ) proposed a much simper GCN framework that limits the filter to the first-order neighbor around each node . On the other hand , spatial approaches implement convolution in spatial domain by defining aggregation functions and transform functions . Notable work includes GraphSAGE ( Hamilton et al. , 2017 ) that transformed representation learning into a formal pattern called aggregation and combination and proposed several effective aggregation strategies such as mean-aggregator and max-aggregator , and GAT ( Veličković et al. , 2017 ) that focuses on the diversity in connected nodes and leverages selfattention mechanism to learn the important information in neighborhoods . Although these models have achieved far better performance on node classification than traditional methods , they still suffer from scarce supervised information due to the limitation on GCN layers making it hard to transform the supervised information to the entire graph . Self-training is an ancient and classic topic in the NLP field before deep learning era ( Hearst , 1991 ; Riloff et al. , 1999 ; Rosenberg et al. , 2005 ; Van Asch & Daelemans , 2016 ) , and has recently been introduced into semi-supervised node classification . For making full use of supervised information to improve the prediction accuracy , Li et al . ( 2018 ) proposed to improve GCN model by self-training mechanism , which trains and applies a base model in rounds , and adds nodes with high confidences as supervision after each round . The newly added nodes are expect to be beneficial to predict rest nodes so as to enhance the final performance of the model . Following this line , the M3S training algorithm Sun et al . ( 2019 ) pretrains a model over the labeled data , and then assigns pseudo-labels to highly confident unlabeled samples that are considered as labeled data for the next round of the training . Later , Zhou et al . ( 2019 ) proposed a dynamic self-training GCN that generalizes and simplifies previous by directly using the output of GCN without a checking part to continuously refresh the training set . Similarly , Yang et al . ( 2020 ) proposed self-enhanced GNN ( SEG ) to improve the quality of the input data using the outputs of existing GNN models . These self-training methods expand the labeled node set with relatively simple checking mechanism or even directly using the output of GCN , as a result they may introduce noise as supervision and thus hurt the final prediction performance . 3 PRELIMINARIES . An attributed relational graph of n nodes can be represented by G = ( V , E , X ) , where V = { v1 , v2 , ... , vn } denotes the set of n nodes , and E = { eij } is the edge set . X = { x1 , x2 , ... xn } ∈ Rn×d is the set of attributes of all nodes , where xi is the d-dimensional attribute vector associated with node vi . Adjacency matrix A = { aij } ∈ Rn×n denotes the topological structure of graph G , where aij > 0 if there is an edge eij between node vi and vj and aij = 0 otherwise . For semi-supervised node classification , the node set V can be split into a labeled node set VL ∈ V with attributes XL ∈ X and an unlabeled one VU = V \VL with attributes XU = X\XL . We assume each node belongs to exactly one class , and denote YL = { yi } the ground-truth labels of node set VL where yi is the class label of node vi ∈ VL . The aim of semi-supervised node classification is to learn a classifier from the graph and known node labels YL , and use it to predict labels for unlabeled nodes VU . Define a classifier fθ : ( ỸL , ỸU ) ← fθ ( X , A , YL ) , where θ denotes the parameters of model , ỸL and ỸU are the predicted labels of nodes VL and VU respectively . In general , we want the predict labels ỸL is close to the ground-truth labels YL as possible in favor of θ∗ = argmin θ d ( ỸL , YL ) = argmin θ d ( fθ ( X , A , YL ) , YL ) , ( 1 ) where d ( · , · ) is a distance measure between two label sets . In recent years GCN has become a popular model for semi-supervised node classification , which aggregates a structural feature for each node and use the formed features , rather than the initial attributes X , for label prediction . 4 THE PROPOSED METHOD . In this section , we will elaborate our proposed framework , namely feature aggregation self-training GCN ( FASG ) , for semi-supervised node classification . Firstly , we do analysis of pseudo-labels to explain the importance of checking part in self-training framework . Secondly , we illustrate the design of checking part in our framework and show the superiority of our checking part in graph networks . Then we elaborate every part of the framework and display the FASG training algorithm . Finally we integrate our framework with various GNN models . 4.1 ANALYSIS OF PSEUDO-LABELS . It is common for existing self-training GCN models to assign pseudo-labels to highly confident nodes and expand them as supervised information . Therefore , the quality of the generated pseudolabels is crucial for node classification and the wrongly introduced supervision information may hurt the final prediction performance . Table 1 summarizes the prediction accuracy of the GCN model ( Kipf & Welling , 2016 ) on Cora when it is trained given different ratios of falsely labeled nodes . It is shown that the accuracy decreases significantly with the ratio of bad training nodes increasing . 4.2 CHECKING PART WITH FEATURE AGGREGATION . To guarantee the quality of the generated pseudo-labels , we develop a delicate checking part in the assistance of feature aggregation . The implementation of feature aggregation can be described as Xaggre = D̃−1ÃX , where D is digree matrix of the graph , D̃ = D + I , Ã = A+ I . We use deep graph library DGL ( Wang et al. , 2019 ) to implement feature aggregation . For illustration of the effectiveness of feature aggregation , we apply t-SNE ( Maaten & Hinton , 2008 ) to visualize the aggregated features of each node on the Cora dataset in Fig 1 , where feati denotes the features aggregated from the original features for i times . As shown in Fig . 1 ( a ) , the original node features are mixed together and are difficult to distinguish . As the fusion of node features going deeper from feat1 to feat4 , nodes with the same label tend to aggregate into clusters in 2-D space . However , the cluster boundaries become blur again after the aggregation goes up to a certain level , e.g .. feat15 and feat20 . Furthermore , we apply linear svm ( Cortes & Vapnik , 1995 ) on aggregated features feat5 to form a classifier , and report its performance in Table 2 in comparison with several GCN models on three citation networks , Cora , CiteSeer and PubMed . Clearly , this relatively simple classifier is able to achieve comparable performance with popular GCN models due to the representation ability of aggregated features . As for the self-training mechanism , we employ the above classifier that combines feature aggregation with line svm to serve as check part for generation of pseudo-labels of nodes . In Fig . 2 , we compare the quality of pseudo-labels generated by different checking mechanisms including plain self-training method ( Li et al. , 2018 ) , deep cluster Sun et al . ( 2019 ) and the proposed checking part with feature aggregation . It is shown that our method introduces less bad training nodes than the compared methods in different label rates on both Cora and CiteSeer , which accounts for the better performance on node classification shown in Sec . 5 .

Summary: This paper proposes a self-training based semi-supervised framework for node classification using Graph Neural Networks when the amount of labelled data is very limited. Self-training is performed by incorporating highly confident samples with their corresponding predicted class as the pseudo label. Authors show that incorporation of correct pseudo labels is a crucial step as the performance degrades rapidly with the incorporation of wrong labels. This work ensures high quality of pseudo labels by a "checking part" with feature aggregation. Aggregated features with linear SVM performs comparably with GNN methods.