FASG: Feature Aggregation Self-training GCN for Semi-supervised Node Classification
Authors: Gongpei Zhao
Tao Wang
Yidong Li
Yi Jin
Venue:  (2021)

Abstract: Recently, Graph Convolutioal Networks (GCNs) have achieved significant success in many graph-based learning tasks, especially for node classification, due to its excellent ability in representation learning. Nevertheless, it remains challenging for GCN models to obtain satisfying prediction on graphs where few nodes are with known labels. In this paper, we propose a novel self-training algorithm based on GCN to boost semi-supervised node classification on graphs with little supervised information. Inspired by self-supervision strategy, the proposed method introduces an ingenious checking part to add new nodes as supervision after each training epoch to enhance node prediction. In particular, the embedded checking part is designed based on aggregated features, which is more accurate than previous methods and boosts node classification significantly. The proposed algorithm is validated on three public benchmarks in comparison with several state-of-the-art baseline algorithms, and the results illustrate its excellent performance.

Under review as a conference paper at ICLR 2021
FASG: FEATURE AGGREGATION SELF-TRAINING GCN FOR SEMI-SUPERVISED NODE CLASSIFICATION
Anonymous authors
Paper under double-blind review
ABSTRACT
Recently, graph convolutioal networks (GCNs) have achieved significant success in many graph-based learning tasks, especially for node classification, due to its excellent ability in representation learning. Nevertheless, it remains challenging for GCN models to obtain satisfying predictions on graphs where few nodes are with known labels. this paper, we propose novel self-training algorithm based on GCN to boost semi-supervised node classification on graphs with little su- pervised information. Inspired by self-supervision strategy, the proposed method introduces an ingenious checking part to add new nodes supervision after each training epoch to enhance node prediction. In particular, the embedded checking part is designed based on aggregated features, which is more accurate than previ- ous methods and boosts node classification significantly. The proposed algorithm is validated on three public benchmarks in comparison with several state-of-the-art baseline algorithms, and the results illustrate its excellent performance.
INTRODUCTION
Graph convolutional network (GCN) can be seen the migration of convolutional neural network (CNN) on non- Eucliidenn structure data. Due to its its excellent ability in representation learning, GCN has achieved significant success in many graph-based learning tasks, including node cluster- ing, graph classification and link prediction (Dwivedi et al.) 2020). Kipf & Welling (2016) pro- posed GCN mode from the perspective of spectrogram theory and validated its effectiveness on semi-supervise node classification task. Subsequent models such as GraphSAGE (Hamilton et al.)
2017]. GAT (Veličković et al., (017). SGCN (Wu et al. 2019) and APPNP (Klicpera et al. 2018) de- signed more sophisticated neighborhood aggregation functions from spatial or spectral views. These methods obtain much more effective results on cemi-supervised node classification than traditional methods such MLP, DeepWalk (Perozzi et al. 2014), etc. However, the prediction accuracy of such GCN models depends largely on the quantity and quality of supervised information, and it will decrease significantly when the quantity of labeled nodes is quite small (Li et al. 2018). The main reason lies that scarce supervised information is difficult to be spread far away in the graph so that unlabeled nodes are hardly to make full use of supervised information for prediction.
Addressing the above issue, many studies have been devoted to improving the representation ability by designing multi-layer GCN model (Li al. 2019). However, the representation ability of GCN, as illustrated in Kipf & Welling (2016). can hardly be improved by simply stacking layers just like MLP. Moreover, stacking too many layers tends to cause over-smoothing (Xu al. 2018) that makes all node embeddings indistinguishable Alternatively, ei al. (2018) proposed to improve the reasoning ability of GCN models by applying self-training techniques on the training. Rather than trying to enhance the expressive ability of the model, the self self-training strategy prefers to expand the supervised information by adding unlabeled nodes with high confidences to the training set each round. Following this line, Sun et al. (2019) proposed a multi-stage self-training strategy (M3S) to enrich the training set, which uses deep cluster (Caron et al.) (018)) and an aligning mechanism to generate pseudo-labels of nodes for updating of the training set. Later, Zhou et al. (2019) proposed a dynamic self-training framework to continuously refresh the training set by directly using the output of GCN without checking part. In general these self- self-training algorithms generate pseudo- labels using relatively simple checking mechanism, which may introduce false labels as supervision information and prevent the improvement of prediction accuracy.

---

Under review as a conference paper at ICLR 2021
In this paper, we propose feature aggregation self-training GCN (FASG) algorithm for semi- supervised node classification. We firstly propose a lightweight classifier that applies linear SVM on aggregated node features, and validate that it achieves comparable performance to popular GCN approaches. Furthermore, this classifier is served as a checking part in the multi-round training process generate pseudo-labels, which are used to filter unreliable nodes when expanding the supervised information. By fully considering the structural information of graph nodes, the newly developed checking part is able improve the accuracy of the generated pseudo-labels and finally boost the node classification. Finally, we illustrate that the proposed self- training strategy can be integrated with various existing GCN models to improve the prediction performance.
paper,
The proposed algorithms is validated in comparison with several state-of-the-eae art baseline algorithms in three public benchmarks, and the experimental results illustrate that the proposed algorithm out- performs all compared algorithms in general on all benchmarks. We will release the source code upon publication of this paper.
RELATED WORK
In the past decade CNN has achieved great success in many areas of machine learning (Krizhevsky al.) 2012: LeCun et al., 1998 Sermanet al.] 2012), but its applications are mainly restricted in dealing with Euclidean structure data (Bruna et al. (203)). Consequently, in recent years more and more studies are devoted to learning the representation on non-Euclidean structure data such as graph.
Graph neural network (GNN) plays an important role the field of graph representation learning, which can learn the representation of nodes or the whole graph. There are many famous GNN ar- chitectures including GCN (Kipf & Welling) 2016), graph recurrent neural network (Hajiramezanali eet al. 2019) and graph autoencoder (Pan et al. 2018). As one of the most important architecture of GNN, GCN can be roughly categorized into spectral and spatial approaches. The spectral ap- proaches (Bruna et al. 2013) define convolution operation by Laplacian feature decomposition of the graph, thereby filtering the graph structure in the spectral domain. On the basis of the Chebyshev polynomial (Defferrard al., 2016) of the graph Laplacian matrix, Mairix & Welling (2016) proposed a much simper GCN framework that limits the filter to the first-order neighbor around each node. On the other hand, spatial approaches implement convolution in spatial domain by defining aggregation functions and transform functions. Notable work includes GraphSAGE (Hamilton et al. (017)) that transformed representation learning into a formal pattern called aggregation and combination and proposed several effective aggregation strategies such as mean-age aagn-agggegator and max-aggregator, and GAT (Veličković et al., 2017) that focuses on the diversity in connected nodes and leverages self- attention mechanism to learn the important information in neighborhoods. Although these models have achieved far better performance on node classification than traditional methods, they still suffer from scarce supervised information due to the limitation on GCN layers making hard to transform the supervised information to the entire graph.
Self-training is an ancient and classic topic in the NLP field before deep learning era (Hearst 1991 Riloft et al., 1999; Rosenberg et al. 2005 Van Asch & Daelemans 2016), and has recently been introduced into semi-supervised node classification. For making full use of supervised information to improve the prediction accuracy, Li et al. (2018) proposed to improve GCN model by self-training mechanism, which trains and applies base model in rounds, and adds nodes with high confidences as supervision after each round. The newly added nodes are expect to be beneficial to predict rest nodes so as to enhance the final performance of the model. Following this line, the M3S training algorithm Sun et al. (2019) pretrains a model over the labeled data, and then assigns pseudo-labels to highly confident unlabeled samples that are considered as labeled data for the next round of the training. Later, Zhou et al. (2019) proposed a dynamic self-training GCN that generalizes and simplifies previous by directly using the output of GCN without a checking part to continuously refresh the training set. Similarly, (Yang et al. (2020) proposed self-enhunced GNN (SEG) to improve the quality of the input data using the outputs of existing GNN models. These self seff-training methods expand the labeled node set with relatively simple checking mechanism even directly using the output of GCN, as a result they may introduce noise as supervision and thus hurt the final prediction performance.
2

---

Under review as a conference paper at ICLR 2021
3 PRELIMINARIES
An attributed relational graph of n nodes can be represented by G = E, where V {vi, "., Un
denotes the Ruad is the set of attributes of all nodes, where xt; the d-dimensional attribute vector associated with node Vg- Adjacency matrix A = {ay}] E Rm denotes the topological structure of graph G, where azj > if there is an edge Eij between node vs and V/ and ag otherwise.
nodes, and E' {egg is the edge set. X (2,, .n.
For semi-supervised node classification, the node set V can be split into a labeled node set Vz E V with attributes XL assume each node belongs to exactly one class, and denote Yz {vi} the ground-truth labels of node set Vz where yE the class label of node 22 VZ.
and an unlabeled one Vu VVV with attributes Xu X We
The aim of supervise node classification is to learn classifier from the graph and known node labels Yz, and use it to predict labels for unlabeled nodes Ve. Define classifier fo (Yz, Yu) t- fo( A, Yr.,, where 0 denotes the parameters of model, Y, and Yv are the predicted labels of nodes Vz and Vv respectively. In general, we want the predict labels Yz is close to the ground-truth labels Yz as possible in favor of
0 arg min di (Yz, YL) = arg min A,L)),
(1)
where -)- is a distance measure between two label sets.
In recent years GCN has become a popular model for semi-supervised node classification, which aggregates a structural feature for each node and use the formed features, rather than the initial attributes X, for label prediction.
THE PROPOSED METHOD
In this section, we will elaborate our proposed framework, namely feature aggregation self-training GCN (FASG), for semi-supervised node classification. Firstly, we do analysis of pseudo-labels to explain the importance
checking part self-training framework. Secondly, we illustrate design of checking part in our framework and show the superiority of our checking part in graph networks. Then we elaborate every part the framework and display the FASG training algorithm. Finally integrate our framework with various GNN models.
4.1 ANALYSIS OF PSEUDO-LABELS
It is common for existing self-training GCN models to assign pseudo-labels to highly confident nodes and expand them as supervised information. Therefore, the quality of the generated pseudo- labels is crucial for node classification and the wrongly introduced supervision information may hurt the final prediction performance. Table [ summarizes the prediction accuracy of the GCN model (Kipf & Welling 2016) Cora when it is trained given different ratios of falsely labeled nodes. It is shown that the accuracy decreases significantly with the ratio bad training nodes increasing.
Table I: Performance of GCN model trained with different ratio bad training nodes.
Ratio
0%
10%
20%
30%
40%
50% %
60 %
Ace 81.9%
77.9%
71.5%
69.1%
65.4%
57.9%
56.3%
4.2 CHECKING PART WITH FEATURE AGGREGATION
To guarantee the quality of the generated pseudo-labels, we develop delicate checking part in the assistance of feature aggregation. The implementation of feature aggregation can be described as
Xaggre = Đ ÁX, where D is digree matrix of the graph, D = D + I, Á = A I. We use deep graph library DGL. (Wang et al. 2019) to implement feature aggregation.
3

---

Under review as a conference paper at ICLR 2021
For illustration of the effectiveness of feature aggregation, apply t-SNE (Maaten & Hinton 2008 to visualize the aggregated features of each node on the Cora dataset in Fig | where feat, denotes the features aggregated from the original features for times. As shown in Fig. aa), the original node features are mixed together and are difficult to distinguish. As the fusion of node features going deeper from feat to feats, nodes with the same label tend to aggregate into clusters in 2-1) space. However, the cluster boundaries become blur again after the aggregation goes up to a certain level, e.g.. feat and feat2o.
Furthermore, we apply linear svm (Cortes & Vapnik (995) on aggregated features feals to form classifier, and report its performance Table comparison with several GCN models on three citation networks, Cora, CiteSeer and PubMed. Clearly, this relatively simple classifier is able to achieve comparable performance with popular GCN models due to the representation ability of aggregated features.
As for the self-training mechanism, we employ the above classifier that combines feature aggrega- tion with line sym to serve check part for generation of pseudo-labels of nodes. In Fig. ? we compare quality of pseudo-label, generated by different checking mechanisms including plain self-training method (Li et al.) 2018), deep cluster Sun et al. (2019) and the proposed checking part with feature aggregation. It is shown that our method introduces less bad training nodes than the compared methods in different label rates both Cora and CiteSeer, which accounts for the better performance on node classification shown Sec.
(a) feata
(b) feal
(c) fealz
(d) feats
(e) feals
(l) featis
(g) feal20
(h) fealio0
Figure 1: T-SNE visualization of aggregated features in the Cora dataset.
Table 2: Performance of aggregation and classic methods
Dataset
MLP
ChebyNet
GCN
GAT
feat5+svm
Cora 55.1%
81.2%
81.5%
83.0%
83.2%
CiteSeer
46.5%
69.8%
70.3%
72.5%
72.3%
PubMed 71.4%
74.4%
79.0%
79.0%
78.8%
4.3 MULTI-STAGE SELF-TRAINING FRAMEWORK
The overall framework of the proposed feature aggregation self-training GCN (FASG) algorithm is illustrated in Fig. 3 Instead of using deep cluster and aligning mechanism, we firstly apply feature aggregation and linear SVM classifier build checking part. After each training round
4

---

Under review as a conference paper at ICLR 2021
Sellirannngg Feature
16
Deepolamer
18
Feature
16-
16-
bord fwiinin indes
10-
10
come-0.5%%
curoe113
Lilenenrr 40 5%
chlowwer %
Iabel
(a) falsely labeled nodes on Cora
(b) falsely labeled nodes on CiteSeer
Figure Comparison of the number of falsely labeled nodes introduced by different checking mechanism.
both the output GCN confidence and the checking part to choose reliable nodes as supervised ones the next round. The training iterates K rounds and then output the final predictions of unlabeled nodes.
Enlarge training set from unlabeled nodes
Aggregation--Linarr SVM
Preudo-labels
GCN-Input layer
GCN Nak-OOutpttllyae
N
Prediction
round==K?
X
Jayers
X
X
X
Figure 3: The overall FASG framework for semi-supervised node classification.
The proposed FASG algorithm described in details in Algo. At the beginning, we concatenate features from feato to feat1o and them into linear SVM to build the checking part. At each round if the output of a node predicted by the GCN model consistent with its pseudo-label generated by the checking part, then we tend expand this node with high certainty to the supervised set. To avoid expanding too much nodes at one time, only t nodes with top confidence are checked at each round. Note that the base GCN model in Algo. is not specified, i.c. the proposed FASG algorithms can be integrated with various GCN models to boost node classification, of which the effectiveness is validated in Table. 6

---

Under review as a conference paper at ICLR 2021
Algorithm The FASG Algorithm
Input:
G EV )): the input graph.
A: the adjacent matrix of graph G.
L, U: the labeled and unlabeled node set of respectively.
GCNcoonu()): the base GCN model.
M-M()): the linear SVM classifier.
K: the number of self-training rounds.
Output:
predictions of all the unlabeled nodes Yo:
1: Conduct feature aggregation to generate feat feats.
2: Form concatenation feat +
3: Generate pseudo-labels Y = SV (feat,,, L, U).
4: Let L' L, U x= U;
5: for k = to K do
6: Train GCN model and get predictions and confidence matix:
Yu, M GCNconv
L', U*).
7:
for each class j do Select nodes (Uj1...., Ujr} } in U" with top confidences.
8:
9:
for i = 1 to t do
if yソı equals 8/y then
+ I' (o U U (ugr)
12: end if
13: end for
14: end for
end for
16:
Compute the final predictions Yu CetNNonv(A,,,, L',U')
17:
return Yu
5 EXPERIMENTS
5.1 EXPERIMENTAL SETTINGS
We conduct experiments three open graph datasets derived from citation networks (including Cora, CiteSeer, PubMed) (Sen et al. 2008 for the semi-supervised node classification task. In these citation networks nodes denote documents whose features formed by bag-of- of-words repre- sentations, and edges denote their relationships with labels indicating what field the corresponding documents belong
Though our framework can be integrated with various GNN models, we choose plain GCN Kiipf & Welling 2016) the base model in Algo. 1 in this experiment. Specifically, set the number of GCN layers n 22 learning rate 1r=le-2, training epochs=600, weight Sec for the 10 in Algo. T Similar to the M3S algorithm (Sun et al. 2019),
GCN model, and fix regard the option of rounds K as hyper-parameter and assign the most suitable K for each testing of different label rate. We choose as 40.5,,4, for Cora dataset, 30,25,15,10.1 10.10 for CiteSeer and 5,4,3 for PubMed. The label rate indicates the amount of labeled nodes, which are randomly chosen from the whole node set under an extra measures that is to guarantee the balance between different classes. For each trial repeat the testing 10 times and report the mean accuracy.
also
5.2 COMPARISON WITH BASELINE ALGORITHMS
The compared baseline algorithms in this experiment include traditional learning method such as Vece2Vec (Grover & Leskovec 2016) .PP (w et al., 2012) and classic GNN approach such as GCN (Kipf & Welling 2016). GAT Veličković et al., (017) and ChebNet (Defferrard et al.) (016). We also include Co-training and Self-training proposed by Li et al. (2018) and other self-learning based approaches MultiStage (Sun et al., 2019). MBS (Sun et al. 2019 as baseline. The relevant experimental settings and results are all taken from original papers.

---

Under review as a conference paper at ICLR 2021
Table 3: Comparison of prediction accuracy Cora dataset
Label Rate
0.5%
1%
2%
3%
4%
Nede2Vec
32.4%
44.4%
50.2%
54.5%
57.4%
LP
57.6%
61.0%
63.5%
64.3%
65.7%
Cheby
38.0%
52.0%
62.4%
70.8%
74.1%
GCN
50.6%
58.4%
70.0%
75.7%
76.5%
GAT
49.0%
60.7%
72.8%
77.7%
79.7%
Co-training
53.9%
57.0%
69.7%
74.8%
75.6%
Self-training
56.8%
60.4%
71.7%
76.8%
77.7%
MultiStage
61.1%
63.7%
74.4%
76.1%
77.2%
M3S
61.5%
67.2%
75.6%
77.8%
78.0%
FASG
62.8%
68.5%
76.1%
78.0%
80.3%
Table Comparison of prediction accuracy on CiteSeer dataset
Label Rate
0.5%
%
2%
3%
4 4%
Node2Vec
24.9%
29.1%
34.4%
35.7%
38.8%
LP
37.7%
41.6%
41.9%
44.4%
44.8%
Cheby
31.7%
42.8%
59.9%
66.2%
68.3%
GCN
44.8%
54.7%
61.2%
67.0%
69.0%
GAT
44.6%
53.7%
64.6%
66.4%
69.3%
Co-training
42.0%
50.0%
58.3%
64.7%
65.3%
Self-training
51.4%
57.1%
64.1%
67.8%
68.8%
MultiStage
53.0%
57.8%
63.8%
68.0%
69.0%
M3S
56.1%
62.1%
66.4%
70.3%
70.5%
FASG
58.3%
66.4%
70.2%
70.3%
70.8%
The comparison of these algorithms on the three benchmarks is summarized in Tables and respectively. It is observed that GNN-based approaches surpass traditional learning approaches in general on all three datasets. By adopting multi-rounds training strategy and expanding the super- vised information iteratively, the algorithms based on self-training mechanism achieve remarkable improvement in prediction accuracy, especially when the label rate is quite small. Furthermore, the proposed FASG algorithm outperforms all baseline algorithms in tested scenarios. The superior- ity of our method derives from the delicate checking part based feature aggregation, which is able to guarantee the high quality expanded supervised information as illustrated Fig. 2
5.3
ABLATION STUDIES
5.3.
THE NUMBER OF TRAINING ROUNDS
Ascenat grandds
rodes grond
Arount
700
AG5
200
C200
1.0
Grounci
Gruncak
(a) Accuracy
(b) New added nodes
(c) Bad training nodes
Figure Accuracy, new added nodes and bad training nodes each round

---

Under review as a conference paper at ICLR 2021
Table 5: Comparison of prediction accuracy
PubMed dataset
Label Rate
0.03%
0.05%
0.1%
Node2Vec
37.2%
38.2%
42.9%
LP
58.3%
61.3%
63.8%
Cheby
40.4%
47.3%
51.2%
GCN
51.1%
58.0%
67.5%
GAT
50.6%
59.1%
65.0%
Co-training
55.5%
61.6%
67.8%
Self-training
56.3%
63.6%
70.0%
MultiStage
57.4%
64.3%
70.2%
M3S
59.2%
64.4%
70.6%
FASC
60.1%
65.2%
70.7%
In order to reveal how our algorithm is affected by the number of training rounds K, we report the numbers of newly added nodes, bad training nodes and the prediction accuracy on the CiteSeer dataset for different label rates with increasing K from 0 to 50. Note that when K is 0, the framework degrades to the plain GCN model. As shown in Fig. Ma), accuracies grow rapidly during the first few rounds for all label rates. For a small label rate (e.g. 0.005), the accuracy tends grow continuously with K increasing. On the contrary, for a relatively large label rate (e.g. 0.04) the accuracy will reach the peak rapidly with small K and saturate afterward.
Fig. bb) shows the number of newly added nodes after each training round, which is consistent with the change of the accuracy in Fig. a). There numbers of newly nodes that expanded pervision information at each round for a small label rate, so the accuracy is improved continuously.
While, for relatively large label rate, the number of newly added nodes drops markedly after few training rounds, which results in the saturation of the accuracy.
Table 6: Performance of our framework integrated with different base GNN models
GNN Model
GCN
GAT
APPNP
GS-M
GS-P
Acc of normal GNN
44.8%
46.8%
45.7%
40.9%
34.9%
Acc in our framework
58.3%
59.8%
57.6%
59.1%
51.0%
5.3.2 INTEGRATION WITH DIFFERENT BASE GCN MODELS
As described in Sec. the proposed FASG algorithm can be integrated with various base GCN models to improve their prediction performances. For validation, we combine FASG with several popular GCN models, and report their prediction accuracy on the CiteSeer dataset with label rate 0.5% in Table where GS-M and GS-P represent GraphSage with mean and maxpool aggregator respectively. It is shown that all tested base GCN models achieve similar performances, and they are all benefitted significantly in prediction accuracy by applying our FASG to expand supervised information iteratively.
CONCLUSION
In this paper, firstly analyzed the limitations of plain GCN models in dealing with semi- supervised node classification tasks, and subsequently proposed a feature aggregation self-training GCN algorithm aiming to improve the prediction accuracy. Our algorithm iteratively expand reliable nodes into the supervised set by checking both the GCN outputs and the pseudo-labels of nodes that are generated through applying linear SVM classifier on the aggregated features. This checking mechanism is able to provide supervised information of better quality than previous methods and boosts the final node classification significantly. In experiments, the proposed algorithm outper- forms state-of-the-are baseline algorithms in general on all tested benchmarks, especially when the ratio labeled nodes is quite small.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Joan Bruna, Wojciech Zaremba, Arthur Szlam, and Yann LeCun. Spectral networks and locally connected networks on graphs. arXiv preprint arXiv:1312.6200 2013.
Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustering for unsu- pervised learning of visual features. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 132-149, 2018.
Corinna Cortes and Vladimir Vapnik. Support- networks. Machine learning, 20(3):273-297, 1995.
Michaël Defferrard, Xavier Bresson, and Pierre Vandergheynst. Convolutional neural networks on graphs with fast localized spectral filtering. Advances in neural information processing systems, pp. 3844-3852, 2016.
Vijay Prakash Dwivedi, Chaitanya K. Joshi, Thomas Laurent, Yoshua Bengio, and Xavier Bresson. Benchmarking graph neural networks. arXiv preprint arXiv:2003.00982 2020.
Aditya Grover and Jure Leskovec. node2vec: Scalable feature learning for networks. In Proceedings of the 22nd ACM SIGKDD international conference Knowledge discovery and data mining, pp. 855-864, 2016.
Ehsan Hajiramezanali, Arman Hasanzadeh, Krishna Narayanan, Nick Duffield, Mingyuan Zhou, and Xiaoning Qian. Variational graph recurrent neural networks. In Advances in neural informa- tion processing systems, pp. 10701-10711, 2019.
Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs.
In Advances neural information processing systems, pp. 1024-1034, 2017.
Marti Hearst. Noun homograph disambiguation using local context in large text corpora. Using Corpora, pp. 185-188, 1991.
Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional net-
works. arXiv preprint
2016.
Johannes Klicpera, Aleksandar Bojchevski. and Stephan Günnemann. Predict then propagate: Graph neural networks meet personalized pagerank. arXiv preprint arXiv:1810.05999,, 2018.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convo- lutional neural networks. In Advances in neural information processing systems, pp. 1097-1105, 2012.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86( (621))2277-2324, 1998.
Guohao Li, Mathias Muller, Ali Thabet, and Bernard Ghanem. Deepgens: Can gens go as deep as cnns? In Proceedings of the IEEE International Conference on Computer Vision, pp. 9267-9276, 2019.
Qimai Li, Zhichao Han, and Xiao-Ming Wu. Deeper insights into graph convolutional networks for semi-supervised learning. arXiv preprint urXiv:180I.076i0 2018.
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, (Nov):2579-2605, 2008
Shirui Pan, Ruiqi Hu, Guodong Long, Jing Jiang, Lina Yao, and Chengqi Zhang. Adversarially regularized graph autoencoder for graph embedding. arXiv preprint 1rXiv:1802.044000 2018.
Bryan Perozzi, Rami Al-Rfou, and Steven Skiena. Deepwalk: Online learning of social repre-
sentations. In Proceedings of the 20th ACM SIGKDD international conference Knowledge discovery and data mining, pp. 701-710, 2014.
Ellen Riloff, Rosie Jones, et al. Learning dictionaries for information extraction by multi-level bootstrapping. In AAAI/IAAI, pp. 474-479, 1999.

---

Under review as a conference paper at ICLR 2021
Chuck Rosenberg, Martial Hebert, and Henry Schneiderman. Semi-supervised self-training of object detection models. 2005.
Prithviraj Sen, Galileo Namata, Mustafa Bilgic, Lise Getoor, Brian Galligher, and Tina Eliassi-Rad. Collective classification in network data. Al magazine, 29(3):93-93, 2008
Pierre Sermanet, Soumith Chintala, and Yann LeCun. Convolutional neural networks applied to house numbers digit classification. In Proceedings of the 2t International Conference on Pattern Recognition (ICPR2012), pp- 3288-3291. IEEE, 2012.
Ke Sun, Zhanxing Zhu, and Zhouchen Lin. Mtlti-stage self-supervised learning for graph convolu- tional networks. arXiv preprint arXiv: /902./ 2019.
Vincent Van Asch and Walter Daelemans. Predicting the effectiveness of self-training: Application to sentiment classification. arXiv preprint arXiv:1601.0 101.0288 2016.
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. arXiv preprint arXiv: 1710. 10903, 2017.
Minjie Wang, Lingfan Yu, Da Zheng, Quan Gan, Yu Gai, Zihao Ye, Mufei Li, Jinjing Zhou, Qi Huang, Chao Ma, et al. Deep graph library: Towards efficient and scalable deep learning on graphs. arXiv preprint arXiv:1909.01313. 2019.
Felix Wu, Tianyi Zhang, Amauri Holanda de Souza Jr, Christopher Fifty, Tao Yu, and Kilian Q Weinberger. Simplifying graph convolutional networks. arXiv preprint
2019.
Xiao-Ming Wu, Zhenguo Li, Anthony M So, John Wright, and Shih-Fu Chang. Learning with partially absorbing random walks. In Advances in neural information processing systems, pp- 3077-3085, 2012.
Keyulu Xu, Weihua Hu, Jure Leskovee, and Stefanic Jegelka. How powerful are graph neural networks? arXiv preprint arXiv:1810.008226 2018.
Han Yang, Xiao Yan, Xinyan Dai, and James Cheng- Self-enhanced gnn: Improving graph neural networks using model outputs. arXiv preprint arXiv:2002.07518, 2020.
Ziang Zhou, Shenzhong Zhang, and Zengfeng Huang. Dynamic self-training framework for graph convolutional networks. arXiv preprint arXiv:1910.02684 2019.
10


Reviews:
[{"review_id": "WSxIgbI7FH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This manuscript proposes FASG, a self-training model with GCN to improve node classification in graph. FASG introduces a checking part to add new nodes as supervision to enhance classification model. Experiments on several datasets show that FASG is better than some baseline methods. \n\nPros\n1. The problem is important. \n2. The presentation is good. \n3. Introduce a new method to generate nodes with pseudo-labels. \n\nCons\n1. The novelty of this work limited. According to my understanding, the contribution lies in the checking part and the checking part just uses GCN and SVM for generating pseudo-labeled nodes. In my opinion, it is a general choice and the novelty is limited. \n\n2. The improvement over baseline methods is not significant. Most improvement percentages are smaller than 1% especially in Cora and PubMed datasets. \n\n3. Experiments should be improved. Baseline methods are relatively weak, there are many work [1,2,3] for graph pre-training or self-training that could be compared and discussed.  \n\n[1] Strategies for Pre-training Graph Neural Networks, ICLR 2020\n\n[2] GCC: Graph Contrastive Coding for Graph Neural Network Pre-Training, KDD 2020\n\n[3] Gpt-gnn: Generative pre-training of graph neural networks, KDD 2020\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "D6U1I1zHCr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a self-training algorithm based on GCN to improve the semi-supervised node classification on graphs. The key idea is to add new nodes with high confidence as supervision to enlarge the labeled nodes. Although the experimental results show the proposed method outperforms or performs similarly to baseline methods, the paper has several weaknesses. First the presented approach is not clearly introduced, with inconsistent statements on building the checking part, and lack of details on how to calculate the confidence to add the new nodes. Second, the novelty of the presented approach is limited, as adding unlabeled samples with high confidence is not a novel idea. Third, the paper writing should be improved, as there are errors.  \n\nIn Table 2, GAT has better or similar performance comparing to feat5+SVM. Why not using GAT feature aggregation for building the checking part? And, in page 5, it says “At the beginning, we concatenate features from feat0 to feat10 and put them into a linear SVM to build the checking part. ”.  So, it is confusing how the checking part is built. In Algorithm 1,  feat1 to feat9 were concatenated in line 2. The statements are not consistent in the whole paper.\n\nIn the line 6 of Algorithm 1, Train GCN model and get predictions and confidence matix:   How the confidence is calculated? \n\nThere are writing errors to correct, such as   Due to its its excellent,   confidence matix: \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "Au0mWE9chC7", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### Summary\nThis paper proposes a self-training based semi-supervised framework for node classification using Graph Neural Networks when the amount of labelled data is very limited. Self-training is performed by incorporating highly confident samples with their corresponding predicted class as the pseudo label. Authors show that incorporation of correct pseudo labels is a crucial step as the performance degrades rapidly with the incorporation of wrong labels. This work ensures high quality of pseudo labels by a \"checking part\" with feature aggregation. Aggregated features with linear SVM performs comparably with GNN methods.\n\n### Strong Points\n 1. Well motivated paper with good performance\n 2. Proposed approach uses self-training based approach with linear SVM which performs well when the amount of labelled data is scarce.\n 3. The framework can be easily incorporated with any GNN based approaches.\n\n### Weak Points\n 1. I am little surprised that the feature aggregation performs comparably to GCN considering the fact that it is essentially the forward pass of GCN. The improvement could be influenced more because of incorporating the extra examples using self-training than feature aggregation.\n 2. Generally easy (very similar to training distribution) examples end up with high confidence scores. Incorporating such nodes might make hard nodes (closer to decision boundary) harder to classify, thoughts on this point is required.\n 3. Choice of \"t\" seemed adhoc, incorporation of how \"t\" impacted the performance of the model would be interesting to see.\n \n### Other comments\n * Section 4.1: It was not clear to me how exactly the \"bad nodes\" were introduced to the model. A little more details would be helpful for readers.", "extra_scores": {}, "extra_text": {}}, {"review_id": "A7zZTNVsMP0", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes an algorithm combining SVM and GCN to solve the node classification problem in label-less scenarios. The proposed model uses a self-training mechanism to generate labels and features and integrates SVM to improve the confidence level of the labels.\n\nStrong points:\n-It first uses relationship information among nodes for training an SVM. The algorithm based on the SVM-checking mechanism achieves remarkable improvement in prediction accuracy, especially when the label rate is quite small.\n\nWeak points: \n- The description in the SVM classifier experiment section is not clear and detailed enough. How does it deal with different multi-class classification problems?   Perhaps, it should verify whether updating the SVM with pseudo-labels can improve the performance of the model in the case of low labeling rates.\n\n- According to the analysis in the paper, GAT and SVM have similar performance. Why did the authors choose to use SVM as the checking part rather than GAT? Some experiments may be required to show the advantages of using SVM checking.\n\n- The proposed method should be compared with some previous SOTA methods (e.g., Union and Intersection method proposed in \"Deeper Insights into Graph Convolutional Networks for Semi-Supervised Learning.\" (2018).)\n\n- The proposed method is not innovative enough, because it just employs a different check algorithm.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a self-training idea for GCN models to help improve the node classification. The reviewers agreed that the technical contribution of the proposed approach is limited and the performance improvement seems marginal. 
Decision: Reject