DYNAMIC SELF-TRAINING FRAMEWORK  FOR GRAPH CONVOLUTIONAL NETWORKS
Authors: Ziang Zhou
Shenzhong Zhang
Zengfeng Huang
Venue:  (2020)
Keywords: self-training
semi-supervised learning
graph convolutional networks

Abstract: Graph neural networks (GNN) such as GCN, GAT, MoNet have achieved state-of-the-art results on semi-supervised learning on graphs. However, when the number of labeled nodes is very small, the performances of GNNs downgrade dramatically. Self-training has proved to be effective for resolving this issue, however, the performance of self-trained GCN is still inferior to that of G2G and DGI for many settings. Moreover, additional model complexity make it more difficult to tune the hyper-parameters and do model selection. We argue that the power of self-training is still not fully explored for the node classification task. In this paper, we propose a unified end-to-end self-training framework called \emph{Dynamic Self-traning}, which generalizes and simplifies prior work. A simple instantiation of the framework based on GCN is provided and empirical results show that our framework outperforms all previous methods including GNNs, embedding based method and  self-trained GCNs by a noticeable margin. Moreover, compared with standard self-training, hyper-parameter tuning for our framework is easier.

Under review as a conference paper at ICLR 2020
DYNAMIC SELF-TRAINING FRAMEWORK FOR GRAPH CONVOLUTIONAL NETWORKS
Anonymous authors
Paper under double-bling review
ABSTRACT
Graph neural networks (GNN) such as GCN, GAT, MoNet have achieved state- of-the-art results on semi-supervised learning on graphs. However, when the number of labeled nodes is very small, the performances of GNNs downgrade dramatically. Self-training has proved to be effective for resolving this issue, however, the performance of self-trained GCN is still inferior to that of G2G and DGI for many settings. Moreover, additional model complexity make it more difficult to tune the hyper-parameters and do model selection. We argue that the power of self-training is still not fully explored for the node classification task. In this paper, propose unified end-to-ent welf-training framework called Dynamic Self-traming, which generalizes and simplifies prior work. A simple instantiation of the framework based GCN is provided and empirical results show that our framework outperforms all previous methods including GNNs, embedding based method and standard self- self-training,
GCNs by a noticeable margin. Moreover, compared with
tuning for our framework is easier.
INTRODUCTION
Graphs or networks can be used to model any interactions between entities such as social interactions (Facebook, Twitter), biological networks pprotein-protein interaction), and citation networks. There has been an increasing research interest in deep learning on graph structured data, e.g.. (Bruna et al.
2014: Defferrard et al., 2016; Monti et al. [017 Kipf & Welling 2017] Hamilton et al. 2017
Velickovic et al. 2018 8 Tang et al.) 2015 Perozzi et al. 2014).
Semi-supervised node classification on graphs is fundamental learning task with many applications. Classic methods rely on some underly diffusion process to propagate label information. Recently, network embedding approaches have demonstrate outstanding performance on node classification (Tang et al.) 2015 Grover & Leskovee 2016 Bojchevski & Günnemann
2018). This approach first learns a liwer-dimensional embedding for each node in an unsupervised manner, and then the embeddings are used to train supervised classifier for node classification, e.g., logistic regression or multi-layer perceptron (MLP). Graph neural networks (GNN) are semi-supervised models and have achieved state-of-the-art performance on many benchmark data sets (Monti et al. 2017 Kipf & Welling, 2017] Velickovic et al.) 2018). GNNs generalize convolution to graph structured data and typically have clear advantage when the number of training examples is reasonably large. However, when there are very few labeled nodes, GNNs is outperformed by embedding based method (as shown by our experimental results), e.g., G2G from (Bojchevski & Günnemann, 2018 and DGI from (Veličković et al. 2019).
To overcome this limitation of GCNs (Kipf & Welling 2017). Li et al. (Li et al., 2018) propose to apply self-training and co-training techniques (Scudder) 1965). The idea of these techniques to augment the original training set by adding in some unlabeled examples together with their label predictions. Such "pseudo-label" information is either from the base model trained on the original training set (self-training) or another learning algorithm (co-training). The results from (Li et al. 2018) demonstrate the effectiveness of co-training and self-training. However, among the four variants implemented in (Li et al. 2018), there is not a single one that achieves the best performance across different settings; and from experiments, G2G and DGI outperforms all the four variants when the number of labels from each class is less than 10. There are clear restrictions in prior self- self-training approaches. First, the pseudo-label set is incremental only, i.e., after unlabeled

---

Under review as a conference paper at ICLR 2020
example is added to the training set, it will never be deleted and its pseudo-label will never change even its prediction and/or the corresponding margin has changed drastically. Secondly, all the pscudo-labels are considered equal, although they may have very different classification margins. Furthermore, it introduces extra hyper-parameters such as the number of unlabeled nodes to be added into the training set and the total number of self-training iterations. The performance gain is sensitive to such parameters and their optimal values may differ for different data sets and label rates (Buchnik) Cohen 2018).
To fully understand and explore the power of self-training on the node classification task, we propose novel self self-train framework, named Dynamic Self- traininn which is general, flexible, and easy to use. We provide simple instantiation of the framework based on GCN (Kipi & Welling 2017) and empirically show that outperforms state-of-art methods including GNNs, self-trained GCN (Li et al., (0188), and embedding based methods. Our framework has the following distinguishing features compared with (Li et al., 2018 Buchnik & Cohen 2018).
1. We augment the training set and recalculate the pscudo-labels after each epoch. So the number self-training iterations is the same as the number of epochs and the pseudo-label assigned to an unlabeled example may change during the training process.
2. In stead of inserting a fixed number of new pscudo-labels with highest margin in each iteration, we use a threshold-basee rule, i.e., insert unlabeled node if and only if its classification margin above the threshold.
3. The pscudo-label set is dynamic. When the margin of an unlabeled node is above the threshold, we activate it by adding it to the loss function, but if the margin of this node becomes lower than the threshold in a later epoch, we will deactivate it.
4. We assign (dynamic) personalized weight each active pseudo-label proportional to its current classification margin. The total pseudo-label loss is thus the weighted sum of losses corresponds all pseudo-labels.
2
PRELIMINARIES
2.1 GRAPH NOTATION AND PROBLEM DEFINITION
In the problem, we are given an undirected graph with node attributes G (V, F, X), where V is the vertex set, E is the edge set. Here, X is the feature matrix, the i-th row of which, denoted as J, is the feature vector of node i. We assume each node belongs to exactly one class and use yi to denote the class label of the i-th node. The aim is to design learning algorithms to predict the labels of all nodes based on the labels of small set of training nodes provided in the beginning. We use (t) to denote the set of nodes whose distance to node i is most k. L V is the set of labeled nodes and 4 = V \ L is the set of unlabeled nodes.
2.2 GRAPH CONVOLUTIONAL NETWORKS
GCN introduced in (Kipf & Welling 2017) is graph neural network model for semi-supervised classification. GCN learns the representations of each node by iteratively aggregating the embeddings of its neighbors. Specifically, GCN consists of L > layers each with the same propagation rule defined as follows. In the 1-th layer, the hidden representations (I-1) are averaged among one-hop neighbors as:
H = a A ((-))))))))
(1)
Here, Ā= A + In is the adjacency matrix of G after adding self-loops (Jn, is the identity matrix), ò is a diagonal matrix with D., = Ej Auj. w(1)) is trainable weight matrix of the I-th layer, and ar is nonlinear activation function; H gaxd denotes hidden feature matrix of the 1-th layer and (o) = X and fi = H(()) represents the output of i-th node.
We use 1(y;, J&) to denote the classification loss of node i, which is typically the cross entropy function. Thus, loss function used by GCN is of the form:
11ii. f.)
(2)
NEC
2

---

Under review as a conference paper at ICLR 2020
For k-layer GCN, the receptive field of each training example is its order-k neighborhood. When there are only few training samples, we need to increase the number of layers in order to cover most of the unlabeled nodes. However, deeper GCN will cause the problem of over-smoothing, i.c., critical features of the vertices may be smoothed through the iterative averaging process, which makes nodes from different class indistinguishable (Xu et al. 2018 Li et al. 2018).
2.3 SELF TRAINING
Recently (Li ct al, 2018) apply self-training to overcome these limitations of GCNs. Self-training a natural and general approach to semi-supervised learning, which is particularly well- well-motivated in the context of node classification (Buchnik & Cohen 2018 Li et al. 2018). Assume we have base model/algorithm for the learning problem, which takes as input a set of labeled examples and makes predictions for other examples. Typically, for each unlabeled node, the base algorithm will also return an associated margin or confidence score. The self-training framework trains and applies the base model in rounds, where the end of each round, the highest-confidence predictions are converted to become new labeled examples in the next round of training and prediction. Thus, the receptive fields of all the labeled nodes increases and will eventually cover the entire graph, which resolve the issue of GCNs without adding more layers.
OUR METHOD
3.1
GENERALIZED TRAF-NNG FRAMEWORK
Algorithm 1: Dynamic Self-training Framework
Generate initial parameter 6 for model f(;;') :), and the initial confidence score vector Sy for each epoch & = 1, 2, do Compute prediction fv + f-G (1-) 3
Update confidence score Sy + UCU()v
Update model parameter by confidence score. 0c +- (P(( if stopping criteria is met then
Break
end end
Sun et al. (Sun et al.] 2019) proposed Multi-stage Training Framework as generalization for self- training method in (Li et al. 2018). Inspired by this, propose more generalized end-to-end self-training framework named Dynamic Self-training Framework shown in algorithm []] Instead of operating data split, we maintain confidence score in each iteration. There is no specified training stages here, but we update the confidence value for each unlabeled node after every epoch.
forward predicting function with backward trainable parame- ters. The graph data G and the trainable parameters 8 the input of this function, and the output of this model is collected into fv E R where fo denotes the output vector (before assigned with label) of node E V. and C = dr. is the number of classes. Then we construct the confidence score vector Sy E R from the model output fo using a function UC, which can be instantiated in many forms. For example, Algorithm 2 iillstrates how standard multi-stage self-training GCN implement this part. Finally we update the model parameters using a specified algorithm such as gradient descent, where the confidence score vector plays role. The confidence score participates the parameter updating process in an end-to-end manner. An example of this part can be seen in section 3.3
Consider the original model f(;:))
3.2 PSEUDO LABEL METHOD
Define the pscudo label J. E Rdd of i-th node which satisfies:
if j arg maxy Siz
Vij 0 otherwise
(3)
3

---

Under review as a conference paper at ICLR 2020
Algorithm 2: Update confidence score for Multi-stage Self-training GCN the stage is currently switched then for each class k do
3
Find the top m vertices v in fv and v E4 Change the value of in Sy end
return Sy
end
Leee 2013)
introduced pseudo label version of semi-supervised losses:
L. Lvn fi) A Σ Tiū ft),
SEC
VEU
where À 7 [C]. n'
[41, E a slyperpaaamenn and the additive term Eieu (ū: ft) the pseudo label loss. Here, À measures how much the pseudo label term influence the training process. This is equivalent to Entropy Regularization for classification problems (Lee 2013).
3.3 SOFT LABEL CONFIDENCE
In standard multi-stage self-training methods, a node just has two states: in the training set or not, which corresponds binary-valued confidences (0, 1]; and in most cases, if node is added in training set, it will be kept there. This simple setting hinders learning in some cases. For instance, the classifier puts wrongly labeled node into the training set, which is of high possibility in preliminary training epochs, it will persistently learn wrong knowledge from this node. Worse still, another wrongly adding more possible. This negative feedback loop may contribute poor classifier. Moreover, original labeled nodes and added nodes in the training are treated equally, which too restricted and may harm the learning; explicitly distinguishing them in the training process could be beneficial. To resolve these problems, we introduce mechanism named Seft Label Confidence as the confidence updating component in algorithm|]] [ which computes a personalized confidence value for each node, and the training set is dynamically changing except the ground truth labels. Based on the pseudo label loss we propose the loss wrapped by soft label confidence:
extremely
L = Σ (Ve f:) + > o(f:))(Ue.
SEC
BEL
Here is a function mapping from Rd to R. defined confidence function. While there are other possible choices for a, in our method we adopt threshold based function:
a(fi) n
max(ReLLU(f 1)),
(6)
Here B E (0, 1) is a hyper-parameter as threshold, n', denotes the number of nodes whose pseudo label belongs to class c', c is the class which i-th node's pseudo label belongs to, and is the all 1 vector. We introduce n',, here to balance the categories of pseudo labels, because pseudo labels could be initially extremely unbalanced and lead to a poor classifier in practice.
Although (fi)) depends St. and thus function of network's weights, we will block the flow of gradient through a (ff.) for the following reasons: Firstly, confidence function is pon-differentiaable in most cases. Secondly, we allow the gradient flow through a the optimizer may tend find a solution that satisfies max(fi) < 8, Vi E V, since for such solution, o(fa) 0 for all and the pseudo label loss zero, which does no good to self self-supervised learning. So we use the following way to compute the gradient:
at ow!
Aly,, S1) aWL
a. f4) +x SEU
E
+
(ff)
OW! 8.1
8,t
4 RELATED WORK
Graph Convolutional Network The work of GNNs seeks generalizations of the convolution
operator to graph structured data. One way do this is apply convolution in the spectral domain,
4

---

Under review as a conference paper at ICLR 2020
where the eigenvectors of the graph Laplacian considered as the Fourier basis (Bruna al. 2014; Henaff et al. 2015; Defferrard et al. 2016 Kipf & Welling (017). Such spectral methods learns hidden layer representations that encode both graph structure and node features simultancously, Kipf and Welling (Kipt & Welling 2017) simplify previous spectral techniques by restricting the propagation to a 1-hop neighborhood in each layer. (Chen al. 2018) propose fast GCNs, which improves the training speed of the original GCN. GAT of (Velickovic et al. 2018) allows for assigning different importances nodes of the same neighborhood via attention mechanisms. (Xu al. 2018)
introduce JK networks, which adjust the influence radii of each node adaptively. Another direction that generalizes convolutions to graph structured data, namely non-spectral approaches, define
convolutions directly in the spatial domain (Duvenaud et al.] 2015 Atwood & Towsley| [2016] Monti
al. 2017). Such methods are easier to be adapted to do inductive learning (Hamilton et al. [2017]
Velickovic et al., 2018 (Bojchevski & Günneeann 2018). However, few-shot learning remains
challenge for this class of methods.
Label Propagation Unlike GNNs, which propagate node representations, the classic Label Propa- gation (LP) method (Zhu et al., (003) iteratively propagates (soft) labels. More specifically, in each iteration, each unlabeled node obtains a new soft label that is the aggregation of the soft labels from the previous iteration of its neighbors. The key LP is design an effective propagation rule; for some propagation rules, the algorithm may not converge and/or the accuracy may not improve over iterations. Thus, one often needs to specify a stopping criteria and a validation set for model selection. LP can also be used as the base algorithm in the self self-training framework.
Self-training Self-training is natural and general approach to semi cemi-supervised learning (Scudder 1965) and has been widely used in the NLP literature. Self-training is used by (Yarowsky 1995 Hearst 1991) for word sense disambiguation. (Riloff et al. (999) used self-training in the form of
bootstrapping for information extraction and later for learning subjective nouns. (Riloff et al. 2003
with (Nigam et al.) 2000) using EM for text classification. Self-training has been used for object
H Huang & Harper 2009 [Sagae] 2010) shows how effective can self-training be in parsing. (Wang al.) [007]] Huang et al. 2009 Qi et al. 2009) introduce self-training techniques to part of speech tagging, and
recognition (Rosenberg ct al. 2005; Zhou et al. 2012). (McClosky et al. 2006 2008
(Kozareva et al. 2005 Liu et 201 3a) adopt self-training in named entity recognition. (Van Asch &
Daelemans] 2016 Drury et al. 2011 Liu et al.] (013b) used self-training in sentiment classification.
Recently, self self-training has also been successfully applied on node classification. Li et al. (Li et al.) 2018) study self- aaff-rraning GCNs; Buchnik and Cohen (Buchnik & Cohen 2018) mainly consider the effect self-training for liffusion-based techniques. In pseudo-label method of (Lee) 2013), for unlabeled data, their leeudollabels are recalculated every weights update. However, they don't ton'tenn assign weight to each unlabeled data.
As for the self-training algorithm itself, (Chen et al., (011) shows that selecting highly confident instances with pre-defined threshold may not perform well. (McClosky et al. 2006) produce ranked list of n-best predicted parses and selected the best one. (Rosenberg et al. 2005) shows that a training data selection metric that is defined independently of the detector greatly outperforms a selection metric based on the detection confidence generated by the detector. (Zhou et al., 2012)) suggests that selecting more informative unlabelled data using guided search algorithm can significantly improve performance over standard self- framework. Most recently, (Levatié et al. 2017) proposed proposed an algorithm to automatically select appropriate threshold.
Network Embedding Node classification is also one of the main applications of network embed- ding methods, which learns dimeen-immniinaal representation for each node in an unsupervised manner, followed by a supervised classifier layer for node classification (Perozzi et al., 2014, Tang et al. 2015 Grover & Leskovee 2016)] Wang et al. 2016 2 (Bojchevski & Günnemann] 2018). A recent work of (Bojchevski & Günnemann (018)) proposes Graph2Gaass. This method embeds each node as a Gaussian distribution according to a novel ranking similarity based on the shortest path distances between nodes, A distribution embedding naturally captures the uncertainty about the representation. DGI (Veličković et al.] 2019) is an embedding method based on GCNs, the unsupervised objective of which is to maximize mutual information. The work of Embedding approaches achieve competitive performance in node classification tasks, while the learned representations also prove to be extremely useful for other downstream applications.
5

---

Under review as a conference paper at ICLR 2020
5 EVALUATION
5.1 DATASET
We conduct the evaluation on four benchmark citation datasets: Cora, Citeseer, Pubmed (Sen et 2008), and Core-full (Bojchevski & Günnemann, 2018). Each of these four datasets is undirected graph with node feature. Each node document and the edges denote the citation relationship: the feature of node is the bag-of-words representation of the document. The number of layers in GCN is two by default, and thus the receptive field of each labeled node its order-2 neighborhood. We measure the fraction of nodes which is covered by the 2-hop neighbors of all labeled nodes. i.e., ( where is the set of labeled nodes randomly sampled from V. Here we report the 2-hop coverage ratio on the four datasets when the label rates 1 %% and 0.5% respectively. We summarize the information datasets in Table
Table I: Summary of datasets
Cora
Citeseer
Pubmed
Cora-full
# of Nodes
2708
3327
19717
18703
# of Edges
5429
4732
44338
81 124
of Features
1433
3703
500
8710
of Classes Coverage(O.5
7
14.78%
6.64%
21.58%
27.19%
Coveragel I %)
24.78%
12.14%
34.60%
47.42%
5.2 EXPERIMENT SETTINGS
We evaluate models on semi- semi-supervised node classification tasks with varying label rates. Instead of evaluating on a fixed data split as (Kipf Welling 2017 Velickovic et al. 2018), we mainly consider random splits as (Li et al. 2018) does. In detail, for a given label rate, we randomly generate 100 different splits on each dataset. In each split, there a labeled set with prespecified size for training, and in this set each class contains the same number of labeled nodes. As in (Li et al. 20188, we don't t use a validation set, and all the remaining nodes will be used for testing. For simplicity, we will refer to task in the form of dataset-l, where is the number of labeled nodes per class. For example, Cora-1 denotes the classification task on dataset Cora with one seed per class.
5.3 IMPLEMENTATION DETAILS
For all the models al. 2014 Tang al. 2015 Grover Leskovec 2016 Wang et al.
2016 BBojchevsii & Günnemann 2018 Velickovic et al.) 201 Monti et al.) 2017) except for GCN based methods, settings of
are the same as suggested original papers. All GCN based methods including GCN, Self-training GCN, Co-training GCN, Intersection GCN, Union GCN, and DSGCN share the same setting of
following (Shchur et al.. 20188: one hidden
layer with 64 units, dropout rate 0.8, Adam optimizer (Kingma & Ba 2015) with learning rate 10 a L2 regularization with weight 10 10-3, We train other GCN based methods for fixed epochs of 200, while DSGCN is trained for 600 epochs in few-label tasks such 1, 5, 10 tasks. Because 20 50 labels per class implies ample supervised information, we train DSGCN for 200 epochs these tasks. The four variants of (Li et al. 2018): Self-training GCN, Co-training GCN. Intersection GCN and Union GCN follow original self-training settings in (Li et al. 2018). For DSGCN, we use a threshold of 0.6 when the number labels per class is below 3, and set the threshold 0.75 for label rate above 3 but below 10. Otherwise, the threshold is 0.9 by default.
5.4 RESULT ANALYSIS
The numerical results are summarized Table and Table The highest accuracy in each column highlighted in bold and the top are underlined. We group all models into three categories: GNN variants GCN, GCNN, GAT, MoNet), unsupervised embedding methods (DeepWalk, DGL, LINE, G2G) and GCN with self- self-training (Co-training, Self-training, Union and Intersection, DSGCN).

---

Under review as a conference paper at ICLR 2020
Table 2: Summary of results in terms of mean classification accuracy (in percent) 100 random splits in different tasks. Unsupervised approaches first learn lower-dimensions embedding for each node in an unsupervised manner, and then the embeddings are used train a supervised classifier for node classification. Here we use logistic regression the classifier for unsupervised embeddings.
Citeseer
Cora
# of Labels
3
5
10
20
50
1
3
5
10
20
50
LP
30.1
37.0
39.3
41.9
44.8
49.5
51.5
60.5
62.5
64.2
67.3
71.7
DeepWalk
28.3
34.7
38.1
42.0
45.6
50.7
40.4
53.8
59.4
65.4
69.9
74.2
LINE
28.0
34.7
38.0
43.1
48.5
54.6
49.4
62.6
63.4
71.1
74.0
76.5
G2G
45.1
56.4
60.3
63.1
65.7
68.2
54.5
68.1
70.9
73.8
75.8
77.0
DGI
46.1
59.2
64.1
67.6
68.7
72.3
55.3
70.9
72.6
76.4
77.9
78.7
GCN
36,4
50.3
57.5
63.2
68.8
72.2
42.4
61.6
68.4
75.1
80.2
83.5
GAT
32.8
48.6
54.9
60.8
68.2
71.5
41.8
61.7
71.1
76.0
79.6
83.4
MoNet
38.8
52.9
59.7
64.6
66.9
69.9
43.4
61.2
70.9
79.3
83.9
Co-training
36.7
49.0
55.0
60.7
65.9
70.0
53.1
65.7
70.2
73.8
78.7
82.5
Self-training
34.6
50.0
58.7
67.4
69.1
71.3
40.6
63.9
71.1
75.5
79.1
81.6
Union
37.2
50.8
55.9
64.4
67.5
70.6
50.1
67.3
72.5
76.2
79.8
82.4
Intersection
35.3
51.8
60.7
67.1
70.2
72.2
43.1
64.4
69.5
73.
78.4
82.0
DSGCN
53.2
63.9
65.8
67.6
70.5
72.4
62.5
72.3
75.5
77.7
80.8
83.8
Table Summary of results in terms of mean classification accuracy(in percent) over 100 random splits in different tasks. GNN variants are excluded due to limited computation resources. Pubmed
Cora-full
# of Labels
3
5
10
20
50
1
3
5
10
20
50
LP GCN
55.7
61.9
63.5
65.2
66.4
67.5
26.3
32.4
35.1
38.0
41.0
46.0
41.3
54.9
63.6
71.2
77.8
81.0
26.4
42.8
49.3
54.4
61.2
65.4
Co-training Self-training
55.1
64.7
69.0
73.5
77.9
80.5
28.3
38.1
42.8
48.5
53.8
62.2
49.7
62.7
67.2
70.6
76.5
79.3
28.7
43.6
48.9
53.4
60.8
64.4
Union
55.1
65.4
69.7
74.0
78.5
80.9
29.2
43.3
48.4
52.9
59.2
62.2
Intersection
52.7
63.4
67.8
70.6
75.9
79.0
26.8
37.7
44.4
51.5
58.4
62.1
DSGCN
55.8
67.1
70.2
74.7
77.8
81.0
30.9
45.6
51.3
57.5
61.4
64.8
Comparison Between GNN Variants and Embedding Methods As unsupervised methods, G2G and DGI outperform all GNN variants in very few labels cases, c.g., 1 and per class on both Cora and Citeseer. Observing that LP performs well in Cora-l while other feature propagation methods not, we can naturally conclude that in dataset with graph structure, concentrating more on the unsupervised information (both strong manifold structure(Li et al., 2018) and feature patterns) model compared to just utilizing supervised information, in the case
will improve remi-supervise of low label rate. When label rate goes higher, all GNN variants enjoy better accuracies compared to unsupervised models. Hence empirically verify the strong generalization ability of GNNs when the supervised information is sufficient. Sun et al. (Sun et al., 2019) has demonstrated the limitation of GCN in few labels case, and here we find that these convolution based methods suffer from inefficient propagation of label information as well, which can seen as the intrinsic drawbacks of semi-supervised graph convolution based methods.
Comparison Between Self-training GCNs and All Other Models In all few-label tasks, self- training strategies improve over GCN by a remarkable margin. Except for tasks with 50 labels per class, the best accuracy is always obtained by self- self-training GCN. Even in extreme one-label case, where unsupervised information is more vital, DSGCN outperforms G2G by a margin of 6.2% in Cora and 9.2% in Citeseer. We conclude that self-training strategy is capable of utilizing unsupervised information more effectively. Thus it significantly helps classification. Additionally, four naive self- training GCNs implemented in (Li et al. 2018) are worse than GCN when label rate goes higher, e.g., Cora-50 and Cora-full-5, which manifests that inappropriate self-traine training strategies will sometimes degrade the performance of the base model. Hence there is trade-off: capturing unsupervised signals, or learning supervised information well. However, DSGCN holds a good balance here. doesn't 't show much decrease compared GCN even in the worst task, Cora-full-She where the

---

Under review as a conference paper at ICLR 2020
Cora-
C
Lora-20
Comr- 50
Gitecn-30
Choaee-0
Shayeno Cho00
M
Test accuracies in training process. Models with different threshold are denoted with different colors, which can be distinguished in legend. Specifically, threshold I represents that the model is equal original GCN.
Figure 1:
accuracy only decreases by 0.6%; in all other cases it is always better than GCN. This demonstrates that the dynamic self-training framework not only helps the original model to capture unsupervised information, but also retains the learning ability when there are enough labels.
Comparison of Self-training GCNs By applying a simpler and more general self-training strategy, DSGCN outperforms other self-training based GCNs with considerable margins in most cases. In Citesecr-L, the margin even reaches 14. 1% compared with the best strategy among Co-training, Self- training, Union and Intersection. This empirically supports the advantage of DSGCN for tackling a wide range of classification tasks over conventional seff-training methods.
Effeet of Threshold Here we discuss how the important parram-rrretter 3 influence the perfor- mance of DSGCN. We train DSGCN with different threshold: 0.45, 0.6, 0.75, 0.9, 1.0 for 1000 epochs dataset Cora and Citeseer for the same split with the same initialized weights. We conduct these experiments on tasks with different seed numbers, the results are presented in figure [ As shown in figure [ when labels are very few, DSGCN with relatively lower threshold / demonstrate a clear improvement in accuracy over the original GCN. Besides, GCN's accuracy curve erratically fluctuates while the curve of DSGCN with a low threshold does not. Thus, observe that the stability of the base model is also improved by wrapping it into the dynamic self- eeff-training framework. When more labels are provided, all models tend be stable and a low threshold could harm the training process.
6 CONCLUSION
In this paper. we firstly introduce a novel self-training framework. This framework generalizes and simplifies prior work, providing customizable modules as extension for multi-stage self-training. Then we instantiate this framework based on GCN and empirically compare this model with number of methods on different dataset splits. Result of experiments suggests that when labels are few, the proposed DSGCN not only outperform all previous models with noticeable margins in accuracy but also enjoy better stability in the training process. Overall, the Dynamic Self-training Framework is powerful for few-label tasks graph data, and provides a novel perspective self- self-training techniques.
REFERENCES
James Atwood and Don Towsley. Diffusion--convllttion neural networks. In Advances in Neural Information Processing Systemx, pp. 1993-2001, 2016.
8

---

Under review as a conference paper at ICLR 2020
Aleksandar Bojchevski and Stephan Günnemann. Deep gaussian embedding of graphs: Unsupervised inductive learning via ranking. International Conference on Learning Representations, 2018.
Joan Bruna, Wojciech Zaremba, Arthur Szlam, and Yann LeCun. Spectral networks and locally connected networks on graphs. International Conference Learning Representations, 2014.
Eliav Buchnik and Edith Cohen. Bootstrapped graph diffusions: Exposing the power nonlinear- ity. In Abstracts of the 2018 ACM International Conference on Measurement and Modeling of Computer Systems, pp. 8-10. ACM, 2018.
Jie Chen, Tengfei Ma, and Cao Xiao. Fastgen: fast learning with graph convolutional networks via importance sampling. International Conference Learning Representations, 2018.
Minmin Chen, Kilian Q Weinberger, and John Blitzer. Co-training for domain adaptation. In
Advances in neural information processing systems, pp. 2456-2464, 2011.
Michaël Defferrard, Xavier Bresson, and Pierre Vandergheynst. Convolutional neural networks on graphs with fast localized spectral filtering. In Advances in Neural Information Processing Systems, pp. 3844-3854 2016.
Brett Drury, Luis Torgo, and Jose Joao Almeida. Guided self training for sentiment classification. In Proceedings of Workshop on Robust Unsupervised and Semisupervised Methods in Natural Language Processing, pp. 9-16, 2011.
David K Duvenaud, Dougal Maclaurin, Jorge Iparraguirre, Rafael Bombarell, Timothy Hirzel, Alán Aspuru- and Ryan P Adams. Convolutional networks on graphs for learning molecular fingerprints. In Advances in neural information processing systems, pp. 2224 22322, 2015.
Aditya Grover and Jure Leskovec. node2vec: Scalable feature learning for networks. In Proceedings of the 22nd ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 855-864. ACM, 2016.
Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. In Advances in Neural Information Processing Systems, pp. 1024-1034, 2017.
Marti Hearst. Noun homograph disambiguation using local context in large text corpora. Using Corpora, pp. 185-188, 1991.
Mikael Henaff, Joan Bruna, and Yann LeCun. Deep convolutional networks on graph-structured data. arXiv preprini arXiv: 1506:050660555 2015.
Zhonggiang Huang and Mary Harper. Self-training pcfg grammars with latent annotations across languages. In Proceedings of the 2009 conference on empirical methods in natural language processing: Volume 2- Volume 2, pp. 832-841. Association for Computational Linguistics, 2009.
Zhonggiang Huang, Vladimir Eidelman, and Mary Harper. Improving a simple bigram hmm part- of-speech tagger by latent annotation and self-training. In Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics, Companion Volume: Short Papers. pp. 213-216. Association for Computational Linguistics, 2009.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. International Conference on Learning Representations, 2015.
Thomas Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. International Conference on Learning Representations, 2017.
Zoritssa Kozareva, Boyan Bonev, and Andres Montoyo. Self-training and co-training applied to spanish named entity recognition. In Mexican International conference on Artificial Intelligence, pp. 770-779. Springer, 2005.
Dong-Hyun Hyun Lee. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In Workshop on Challenges in Representation Learning, ICML, volume 3, pp. 2013.

---

Under review as a conference paper at ICLR 2020
Jurica Levatić, Michelangelo Ceci, Dragi Kocev, and Sašo Džeroski. Self training for multi-targer regression with tree ensembles. Knowledge-Baad Based Systems, 123:41-60, 60, 2017.
Qimai Li, Zhichao Han, and Xiao-Ming Wu. Deeper insights into graph convolutional networks for semi-supervised learning. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018.
Qian Liu, Bingyang Liu, Dayong Wu, Yue Liu, and Xueqi Cheng. A self-learning template approach for recognizing named entities from web text. In Proceedings of the Sixth International Joint Conference on Natural Language Processing, pp. 1143, 2013a.
Zhiguang Liu, Xishuang Dong, Yi Guan, and Jinfeng Yang. Reserved self toff-traaning: A semi-supervised sentiment classification method for chinese microblogs. In Proceedings of the Sixth International Joint Conference on Natural Language Processing, pp. 455 2013b.
David McClosky, Fugene Charniak, and Mark Johnson. Effective self-training for parsing. In Proceedings of the main conference on human language technology conference of the North American Chapter of the Association of Computational Linguistics, pp. 152-159. Association for Computational Linguistics, 2006.
David McClosky, Eugene Charniak, and Mark Johnson. When is self- self-training effective for parsing? In Proceedings of the 22nd International Conference on Computational Linguistics- Volume 1, pp. 561-568 Association for Computational Linguistics, 2008
Federico Monti, Davide Boscaini, Jonathan Masci, Emanuele Rodola, Jan Svoboda, and Michael M Bronstein. Geometric deep learning graphs and manifolds using mixture model cnns. In Proc. CVPR, volume I. pp. 2017.
Kamal Nigam, Andrew Kachites McCallum, Sebastian Thrun, and Tom Mitchell. Text classification from labeled and unlabeled documents using em. Machine learning, 3902-3):033-13,, 2000.
Bryan Perozzi, Rami Al-Rfou, and Steven Skiena. Deepwalk: Online learning of social representa- tions. In Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 701-710. ACM, 2014.
Yanjun Qi, Pavel Kuksa, Ronan Collobert, Kunihiko Sadamasa, Koray Kavukcuoglu, and Jason Weston. Semi-supervised sequence labeling with self-learned features. In 2009 Ninth IEEE International Conference on Data Mining, pp. 428-437, IEEE, 2009.
Ellen Riloff, Rosie Jones, et al. Learning dictionaries for information extraction by multi-level bootstrapping. In AAAI/IAII, pp. 474 479, 1999.
Ellen Riloff, Janyce Wiebe, and Theresa Wilson. Learning subjective nouns using extraction pattern bootstrapping. In Proceedings of the seventh conference on Natural language learning at HLT. NAACL 2003- 4, pp. 25-32. Association for Computational Linguisties, 2003.
Chuck Rosenberg, Martial Hebert, and Henry Schneiderman. Semi-supervised self-training of object detection models. WACV/OTTION, 2, 2005.
Kenji Sagae. Self-training without reranking for parser domain adaptation and its impact semantic role labeling. In Proceedings of the 2010 Workshop on Domain Adaptation for Natural Language Processing, pp. 37-44. Association for Computational Linguistics, 2010.
H Scudder. Probability of error of some adaptive pattern-recognition machines. IEEE Transactions on Information Theory, 11(3):363-371, 1965.
Prithviraj Sen, Galileo Namata, Mustafa Bilgic, Lise Getoor, Brian Galligher, and Tina Cliassi-Rad Collective classification in network data. Al magazine, 29(3):93 93, 2008
Oleksandr Shchur, Maximilian Mumme, Aleksandar Bojchevski, and Stephan Günnemann. Pitfalls of graph neural network evaluation. CoRR, abs/181 1.5668. 2018. URL http:://rxxvv.ozg
abs 1811. 05868
Ke Sun, Zhanxing Zhu, and Zhouchen Lin. Multi Multi-stage self superviee learning for graph convolu- tional networks. arXiv preprint arXiv: 1902. 1038, 2019.
10

---

Under review as a conference paper at ICLR 2020
Jian Tang, Meng Qu, Mingzhe Wang, Ming Zhang, Jun Yan, and Qiaozhu Mei. Line: Large-scale information network embedding. In Proceedings of the 24th International Conference on World Wide Web, pp. 1067-1077, 2015.
Vincent Van Asch and Walter Daelemans. Predicting the effectiveness of self-training: Application to sentiment classification. arXiv preprint urXiv::6011 1601.03288, 2016.
Petar Velickovic, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. International Conference on Learning Representations, 2018.
Petar Veličković, William Fedus, William L Hamilton, Pietro Liò, Yoshua Bengio, and R Devon Hjelm. Deep graph infomax. International Conference on Learning Representations, 2019.
Daixin Wang, Peng Cui, and Wenwu Zhu. Structural deep network embedding. In Proceedings of the 22nd ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 1234. ACM, 2016.
Wen Wang, Zhongqiang Huang, and Mary Harper. Semi-supervised learning for part-of-speech tagging of mandarin transcribed speech. In 2007 IEEE International Conference on Acoustics, volume 4, pp. IV-137. IEEE, 2007.
Speech and Signal Processing-IAASSP
Felix Wu, Amauri Souza, Tianyi Zhang, Christopher Fifty, Tao Yu, and Kilian Weinberger. Sim- plifying graph convolutional networks. In International Conference on Machine Learning, pp. 6861-6871, 2019.
Keyulu Xu, Chengtao Li, Yonglong Tian, Tomohiro Sonobe, Ken-ichi Kawarabayashi, and Stefanie Jegelka. Representation learning on graphs with jumping knowledge networks. International Conference on Machine Learning, 2018.
David Yarowsky. Unsupervised word sense disambiguation rivaling supervised methods. In 33rd annual meeting of the association for computational linguistics, 1995.
Yan Zhou, Murat Kantarcioglu, and Bhavani Thuraisingham. Self-training with selection -eyeeiion----reectioonn In 2012 IEEE 12th international conference data mining, pp. 795-803. IEEE, 2012.
Xiaojin Zhu, Zoubin Ghahramani, and John D Lafferty. Semi-supervised learning using gaussian fields and harmonic functions. In Proceedings of the 20th International conference on Machine learning, pp. 1199 2003.

---

Under review as a conference paper at ICLR 2020
A APPENDIX: ADDITIONAL EXPERIMENTS
We also test our self-training methods on other GNNs as well, e.g., SGC(Wu et al. 2019). GAT (Velickovic et al. 2018), and GraphSage (Hamilton al. 2017). For the three GNN models, settings of hyper-paramettrs are the same as suggested in original papers. And our dynamic self-training framework share the same setting of
one hidden layer with 32 units, dropout rate 0.7, Adam optimizer (Kingma& & Ba 2015). a Lg regularization with weight 5 and set the threshold to 0.9. Clearly, our dynamic self-training framework achieves similar improvements on all the three base models. The numerical results are summarized in Table 4 framework, these models enjoys noticeable increase performance.
We can see equipped with our DS
Table Summary of results terms of mean classification accuracy (in percent) over 50 random splits in different tasks(the results of GAT experiments are from Table
Citeseer
Cora
# of Labels
5
10
20
50
5
10
20
50
SGC
55.5
63.7
69.0
72.6
63.5
72.5
75.9
78.9
DS-SGC
59.6
65.0
69.7
73.4
65.0
73.4
76.2
78.9
GAT
54.9
60.8
68.2
71.5
71.1
76.0
79.6
83.4
DS-GAT
58.3
67.0
70.8
73.4
71.9
77.1
81.0
83.6
GraphSAGE
59.7
65.4
68.8
72.1
69.3
75.3
79.2
82.5
DS-GraphSAGE
60.6
66.3
69.5
72.6
72.5
78.4
81.0
84.0
To evaluate the computation overhead introduced by dynamic self-training framework, test the total training time for various models. Intuitively the computational cost will only slightly increase. The reason is that the computational cost of the original GCN model is dominated by previous layers, where the entire graph is included. So even if all nodes become pseudo labels, the size of the entire network is increased by at most factor of 2, and the number of parameters remains the same. Therefore, the computational costs will increase by at most small constant in theory. We have also verified this empirically. We record the training time of base models before and after applying our framework. In the experiments, the training size is 20 per class, the number of epoch is 200, and the time is the average time (in seconds) of runs. The numerical results can be seen Tabel
Table 5: Total training time for various models in seconds(s), implemented on PyG.
Citeseer
Cora
GCN DSGCN
3.0
2.6
SGC
1.4
1.3
DS-SGC
7.4
GAT
DS-GAT
11.9
8.7
GraphSAGE DS-GraphSAGE
8.7
8.5
12


Reviews:
[{"review_id": "r1lIXkR6YH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes an approach for learning graph convolutional networks for inferring labels on the nodes of a partially labeled graph  when only limited amount of labeled nodes are available.\n\nThe proposal is inspired from Graph convolution Networks with the idea of overcoming the major drawback of these models that lies of their behavior in case of limited coverage of the labeled nodes, which implies using deeper versions of the model leading at the price of what the authors call the over-smoothing problem.  \n\nThe main idea here consists in relying on self training to get a better coverage of labeled nodes enabling learning with less deep models, this translates to a simple and intuitive algorithm. Using self training is not new in GCN but the way it is used here, computing adaptively a threshold for incorporating pseudo labels and using weights according to the confidence off predictions is new.\n\nExperimental results are reported on citation datasets and compared with many baselines show similar results as baselines when the coverage increases up to 50 labeled nodes /class, but the method brings significant improvements when the coverage is low (e.g. only few, <20, labels /class). \n\nAlthough the difference with previous approaches do not look like a huge step, the method seems to be quite justified empirically and achieve real good results wrt state of the art.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "H1eZGztV9B", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "#Summary\n\nThis paper proposes a generalised self-training framework to build a Graph Neural Network to label graphs.  Of importance is the dynamic nature of the self-training. The authors do not change the GCN but extend the self-training portion as per the prior GCN paper by introducing Dynamic Self-Training that keeps a confidence score of labels predicted for unlabelled nodes.\n\n# Comments\n\nThis is a very interesting paper in terms of looking at the effects of changing the self-training framework to better utilise the underlying structure. As such we can exploit information from other nodes that are yet to be labelled.\n\n1. As the self-training is going on, are there different computational costs or are they about the same?\n2. For CiteSeer 20 and 50, why does \\beta = 0.45 switch from the other experiments?\n3. Will such self-training be useful for general NN self-training procedures\n4. If we had soft-labelling or uncertainty on which label each node has, how would the dynamic self-training be changed?\n\n#Other notes\nPlease remove the \n\nAn appendix\nYou may include other additional sections here", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "ByxRt-tpFr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper propose to modify the existing work [1] of self-training framework for graph convolutional networks. It tracks three limitations of [1] and propose three  use a threshold-based rule to insert new pseudo-labels and dynamic change the pseudo-label set. Moreover personalized weight are assigned to each activate pseudo-label proportional to its current classification margin. Evaluation of the proposed framework is performed on four networks for semi-supervised node classification task with varying label rates.\nPros:\n1. This work tracks and addresses the limitations of existing work.\n2. Authors conduct experiments on multiple dataset with varying 2-hop coverage ratio. \n3. The overall paper is well written, except some typos, e.g. in page 6, section 5.1 \"Each of three dataset is ......\". Should \"three\" be \"four\".  \nCons:\n1. The proposed framework makes modification on the existing work, which is a good extension but the novelty is limited.\n2. The gap of the experiment results between the proposed method and the baseline methods are quite small.\n3. Only GCN instantiation are provided, it is suggested to evaluate the effectiveness on the other GNN variants, such as GraphSage, GAT and MoNet.\n[1] Li et al. Deeper insights into graph convolutional networks for semi-supervised learning.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper is develops a self-training framework for graph convolutional networks where we have partially labeled graphs with a limited amount of labeled nodes. The reviewers found the paper interesting. One reviewer notes the ability to better exploit available information and raised questions of computational costs. Another reviewer felt the difference from previous work was limited, but that the good results speak for themselves. The final reviewer raised concerns on novelty and limited improvement in results. The authors provided detailed responses to these queries, providing additional results.

The paper has improved over the course of the review, but due to a large number of stronger papers, was not accepted at this time.
Decision: Reject