Auxiliary learning induced graph convolutional networks
Authors: Gengchen Duan
Taisong Jin
Rongrong Ji
Ling Shao
Baochang Zhang
Feiyue Huang
YONGJIAN WU
Venue: NeurIPS 2021 Submitted (2021)
Keywords: graph neural networks
meta learning
auxiliary learning

Abstract: In this article, we propose a novel auxiliary learning induced graph convolutional network in a multi-task fashion. Specifically, both the link prediction and pseudo label generation are used as two auxiliary tasks to complement the primary task of node classification. Those two auxiliary tasks are jointly trained with the primary node classification task via the graph meta-learning strategy. The experimental results demonstrate that the proposed method consistently and significantly outperforms the existing methods and achieves state-of-the-art node classification results on the benchmark citation network datasets. 

Auxiliary Learning Induced Graph Convolutional Networks
Anonymous Author(s)
Affiliation Address
email
Abstract
Graph convolutional networks (GCNs) have recently achieved great success in many applications. However they suffer from an incomplete amnotation problem for complex graph-structured data. In this paper, we introduce novel auxiliary learning method for GCNs in a multi-task fashion, which can efficiently enrich the data annotations. Specifically, both link prediction and label generation are used as two auxiliary tasks to complement the primary task of node classification. These two auxiliary tasks are jointly trained with the primary node classification task via graph meta-learning strategy. The experimental results demonstrate that the proposed method consistently and significantly outperforms existing methods and achieves state-of-te offte-of-thearr results on several benchmark citation network datasets.
6
10
Introduction
"
Graph-structured data is ubiquitous in real-world applications. However, general deep learning 13 methods, such as convolutional neural networks (CNNs), cannot adapt to graph-structured data directly, because the nodes in graph have different numbers of neighbors, which often lose the ranking information. To handle the graph data effectively, graph convolutional networks (GCNs) have recently been proposed and used in many applications such biomolecular prediction (1 and recommendation systems 22]
15
17
Previous methods focus designing models that can extract information from both the graph 19 topology and node features. Specifically, existing GCN methods typically design different propagation strategies
B 4 (5) for each network layer and stack more network layers [6 7 g to derive larger receptive fields. However, the neighborhood aggregation is essentially type of Laplacian smoothing and stacking too many layers may result in over-smoothing [9]. These drawbacks of existing methods
21
23 limit further performance enhancement.
24 In this paper, we try to explore the bottleneck of node classification from another point of view, i.e.,
25 from the training data itself. As shown in Fig. [ graph-structured data has different properties from grid-like data such an image. The most obvious difference is that the nodes in graph are connected 27 by edges, This causes two main issues whien comes to annotating graph-structured data, resulting in that existing methods cannot fully leverage the graph-structured information. First, the edges in most
29 graph data for semi- supervised node classification are unweighted. This arbitrary edge indication setting cannot effectively reflect the detailed graph structures. Besides, graph-strure structrrd data may be contaminated with noisy edges. These noisy edges cannot represent the true pairwise relationships between nodes. Second, using one-hot labels to train graph- model is inappropriate. One-hot
31
39 labels are widely used various machine learning tasks, assigning training sample to a single class. However, nodes in graph are connected;
nodes with different classes may have relations. In 35 this scenario, it is more suitable to use soft labels to assign a node to multiple classes, with different probabilities indicating which class the node possibly belongs
38
Submitted to 35th Conference on Neural Information Processing Systems (NeurIPS 2021). Do not distribute.

---

43 we add two auxiliary tasks to enrich To fully leverage the graph-structured information for enhancing the node classification performance of GCNs, 40 we introduce an auxiliary learning 44 the topology information of a graph
37
Image
Graph
38
/
/
39
2
2
2
2
41 scheme to the GCN framework in by softening the node labels and re- weighting the edges. Experimental results show that model achieves state-of-the-arr node classification per- formance several benchmark cita- 50
42 a multi-task fashion. To this end,
3
3
3
3
2
4
4
4
4
3
46
Figure The difference between image data and graph- structured data. The nodes in graph-structured data connected. Further, in most graph data for the node classifi- cation task, edges are unweighted and noisy.
48
tion network datasets. Our contribu- tions follows:
51
52 (I) We propose two auxiliary tasks to capture more accurate graph information and enhance the model performance. The auxiliary link prediction task ensures that the model captures more graph topology information and generates probabilistic edges. The auxiliary label generation task softens the one-hot labels and generates aseudo-labels for unlabeled nodes.
54
56 (2) The reconstructed edges and pseudo sudo labels derived via the two auxiliary tasks are iteratively
57
updated with a node classifier of the primary task based on a meta auxiliary learning strategy, resulting state-of-the-ar node classification performance.
58
2
Related Work
Over the past few years, GCNs have achieved significant breakthroughs in graph data representation. Generally, existing GCNs can be divided into spectral-based methods and apatial-based methods.
61
62
The spectral- based methods use graph spectral theory to define the graph convolutional operation 63 in graph Fourier domain. Spectral CNN [0]] follows these mathematical foundations, assuming that a convolutional filter is a set of learnable parameters. To reduce computational complexity.
64
ChebNet [ approximates a graph convolutional filter as Chebyshev polynomials of the eigenvalues. GCN [12 introduces a first-order approximation of ChebNet and proposes a renormalization trick to alleviate numerical instabilities and exploding/vanishing gradients. DualGCN [13] introduces dual
65
66
67
68
GCN architecture with two graph convolutional layers in parallel to encode both local and global structural information.
69
70
The spatial-based methods define feature aggregation in the spatial domain directly, which is more efficient, general, and flexible [14. The key challenge for these spatial- based methods is to apply the convolution operation for different-sizeed neighborhoods. while at the same time maintaining the weight sharing property. Neural network for graphs (NN4G) [15] the first spatial- method,
71
72
73
74 applying the graph convolutional operation in the spatial space. Diffusion convolutional neural networks (DCNN) [6 consider graph convolutions as diffusion processes to efficiently learn features that are invariant under isomorphism. Message passing neural networks (MPNN) [77 model graph convolution as message passing process among the nodes. The graph attention network (GAT) introduce masked self-attentional layers to assign different weights to adjacent nodes, leading to learnable filter weights. The mixture model network (MoNet) [18] introduces pseudo-coordinatt to assign different weights to the neighbors of each node. To achieve weight sharing across different
75
76
77
3
79
80
81 nodes, some spatial-based models attempt to rank a node's neighbors via certain criteria or metries, which transforms the graph-structured data into grid data for further processing. The large-scale
83 graph convolutional network (LGCN) [9] ranks a node's neighbors via the node feature values. Then, multiple ID convolutional layers are stacked for feature aggregation. Approximate personalized propagation of neural predictions (APPNP) [4] takes the personalized PageRank algorithm as the model propagation method avoid over-smoothing when stacking more layers or increasing the size of the neighborhood.
85
87
Multi-task learning is designed to simultaneously learn set of related but different tasks for ensuring 89 that a learning model can derive the best performance across tasks. Different from multi-task learning, auxiliary learning is only concerned with model performance on the primary task. For

---

(9:4)
Classilication result
GON Layer
GON Layer
GCN Layer
Debail of the GCN layer
Input graph
Compute message
Aggregete and Update
Figure 2: Network architecture of the vanilla GCN model. The vanilla GCN contains multiple GCN layers. Each layer captures the graph structure generate the hidden embeddings from the previous layer (For the first layer, it is the original feature of the node) input, and obtains the output through the message calculation, aggregation and update step. The last layer uses a softmax function to generate classification probabilities for each node.
91 instance, Deepstereo 20 leverages auxiliary learning to predict the relative poses of multiple cameras for unsupervised monocular depth estimation. To improve the performance of conversational speech recognition, auxiliary learning 21 is applied to low-level representations. Compared to the common learning scheme, adopts meta-learning to automatically generate the auxiliary task labels. Pseudo Label 23 is sepe-ssee learning method, where deep neural network is trained using both labeled and unlabeled data. For unlabeled data, the model picks up the class that has the maximum predicted probability as the true label train itself. MPL [24] extends the Pseudo Label [23] via meta-learning strategy, where the pseudo-labels are not generated by itself, but by teacher network.
93
94
auxiliary learning can enhance learning performance. For instance, MAXL
85
22
87
98
99
The existing GCNs are designed for single task, where the properties of graph- structurd data are not fully explored. We introduce the auxiliary learning scheme to leverage more detailed graph topology information for enhancing node classification performance.
101
3
Method
104
3.1
Preliminaries
105
Given graph G = (V,E, V is set of nodes and E is set of the edges connecting the related nodes. X E R xd represents the features matrix of the nodes, where d is the dimension of the node features and N | is the number of nodes.
106
107
108
The proposed method adopts the vanilla GCN 122 as the backbone network, taking the graph adjacency matrix A, labeled training set ttrain and original features X of the nodes inputs to perform the semi-supervised node classification task. Based on the MPNN framework [17], each layer of the vanilla GCN is defined in three parts:
110
112
(1) Message computation: The message of node v; and its neighbor node vy is calculated, where j (i), as:
(k)
h (w), jE (i) UN(.
/ deg(()deg(ū)
114 Here, deg(i) is the degree of node and ((k) Rx- are the learnable parameters of the 115 1th layer, where es is the size of the hidden embedding.
(2) Aggregation: The messages of node and its neighbors are aggregated by summing them up:
m (R) = Σ m (k)
(2)
W())
jE(1)UN()
3

---

Link predictor
Label ganerator
Feature Extrector
Decoder
Labol Generator
ABAN
0000
2000
0000
0000
0010
Gax
Arcem
hay
Feature Extrastor
Liccoder
Feature Extrastor
Clacaller
dwan
0000 0800
88BE
0000
Backbone model
The overall network architecture. Our method consists of three networks: backbone network for the primary task and two auxiliary task networks. (1) The backbone network is a vanilla GCN, which predicts the classification results of each node. (2) The first auxiliary task network is link predictor, which focuses on the link prediction task and generates a probabilistic edge structure the input of the backbone network. (3) The second auxiliary task network is label generator, which employs the label generation task to generate the pseudo soft labels for supervising the node classifier. The parts connected by the dashed arrow share the same parameters, and the red slash the arrow indicates a stop-gradient (detach) operation.
Figure 3:
(k) where ()() denotes the aggregated message.
(3) Feature updating: Finally, the hidden representation is updated with the aggregated message. For the vanilla GCN, the update function can be considered as applying a non-linear operation to the aggregated message:
119
120
softmax((m) m ((a) h
k
(3)
[eLUU(m (
otherwise.
121 Here, is the number of model layers. The last layer of the vanilla GCN should output the classification probability via softmax function. Otherwise, ReLU operation is used.
123
3.2
Multi-Task Network Architecture
In this paper, propose auxiliary learning induced GCN for semi-supervised node classification (see Fig. 3). To enhance the node classification performance the backbone vanilla GCN, we design two auxihary tasks: link prediction and pseudo-label generation.
125
127 The first K layers of K- K-layer vanilla GCN model can be considered as a feature extractor, while the last layer can be considered as classifier. Denoting the feature extractor and node classifier of the backbone model as he, and fer, respectively, where and 02 are the learnable parameters, the proposed two auxiliary task networks are defined as follows.
129
3.2.1
Link Predictor
To enrich the edge information, we design auxiliary link prediction task to infer the missing edges and present the probability of edge existence. The link predictor we propose contains decoder R(-) and the feature extractor hag of the backbone model.
133
134
135 The feature extractor hog (+) takes reduced adjacency matrix A, which corresponds to the sampled edge set E, C E, and node features X as inputs to generate the hidden embedding:
H hag (X, Ag).
(4)
137
Then, the decoder computes the similarity between each node based on H, predict the edge existence probabilities. We simply use an inner-prous mnner-product calculator with a sigmoid function as the implementation of decoder R(-). The similarity of hidden embeddings can be computed
138
139
Tú R(h, h,) a (h,h²))
(5)
where h, and hy denote the hidden embeddings of node and node by in H., respectively.

---

141
3.2.2
Label Generator
142
Although using the one-hot label is inappropriate, it is difficult to obtain soft labels with manual annotations for real graph-structured data. To tackle this issue, we introduce an auxiliary label generation task to generate soft labels that reflect the tendency of different classes each node belongs to. Label generator I6 [X, A) is a vanilla GCN, where yp are the learnable parameters and A the adjacency matrix. The label generation network predicts the label distribution of each node based on the graph structure A and the raw features X of the nodes, as:
143
145
147
Y = 9p (X, A),
(6)
148
where Y are the predicted pseudo labels for guiding the training of the backhone network and the label generator.
149
150
3.2.3
Node classifier
151
The node classifier carries out the primary semi-supervised node classification task in our model. Compared the vanilla GCN model, add a graph reconstruction step the beginning. The graph adjacency matrix reconstructed with the hidden embeddings contains richer edge information since the proposed auxiliary link prediction task can enhance the graph topology capturing ability of the feature extractor hap (-). However, the hidden embeddings derived by the feature extractor change rapidly in the first few iterations, resulting
152
153
154
155
156
changing reconstructed adjacency matrix. Directly applying the reconstructed adjacency matrix to the entire backbone network increases the training instability. Thus, we only apply it as the input of the feature extractor heg (+), while the classifier fo (-) still adopts the original adjacency matrix as input.
157
158
159
160
In each training iteration, the feature extractor ha first generates the hidden embeddings of the nodes H = hag (X, A). The decoder uses H to reconstruct the adjacency matrix Arecon = R(H, H). Then, the feature extractor takes the reconstructed adjacency matrix Areco compute the hidden embeddings Hjecon = he, (X, Arecou).
161
162
163
Finally, the computed hidden embeddings Hrecon and the original graph adjacency matrix A are fed to the classifier to obtain the final classification results:
185
Y
fo, (Hrecon: A).
186
3.3 Auxiliary Training Phases
187
In addition to the backbone network, the proposed method contains two auxiliary task networks. In the following, we will introduce the objectives for the node classifier fo, link predictor heg, and label generator 9Gp in order, and then leverage a meta auxiliary learning scheme to train the proposed multi-task network.
188
169
170
3.3.1 Training the Node Classifier fo,
The purpose of the node classifier fe, (-) is to carry out the graph-based semi-supervised node classification task, which yields the final prediction results. Naturally, classification loss between the predicted result and the real categories of nodes to supervise the training process. At the same time, to reflect the tendency of the class a node belongs to, the training should make the prediction labels of the classifier be close to the pscudo soft labels generated by label generator (.).
175
necessary to use the
175
177
In the th iteration, denoting the pseudo soft labels as Y = (. (X, A), the real labels used in training as Y train, and the prediction results of the node classifier of the backbone network So, as
178
179
((1) = ffgo (hogo) (X, ((doo)) ), A), the objective for the node classifier fo, (-) is defined as
180
2/0 = LCE f(t)) train? Ytrain) + LMSE ((0) Yro(t)).
(8)
181
The objective contains two parts: the loss on the real training labels, and the loss on the generated pseudo soft labels. LCE denotes the cross-entropy loss and LMSE denotes the mean squared loss. Although this objective can be used to update the learnable parameters 02 of the feature extractor he (.), it generates the hidden embeddings used for node classification, only employ it to
183
5

---

185
supervise the learning of the node parameters 81 of the classifier fe, (-). To avoid adding unnecessary supervision for generating the reconstructed graph adjacency matrix Arezons we add a goopggrad operation (detach) which is formulated as follows:
186
187
(1) recon
detach (R(H, H))-
188
3.3.2
Auxiliary Training for the Feature Extractor hog
189
The feature extractor haz is shared by the link prediction module and the backbone network. In each iteration, we first randomly sample certain percentage of edges from the real edge set & to form sampled edge set & C E. Denoting the reduced graph adjacency matrix as As. which corresponds to the sampled edge set Eg, and applying a message passing operation with Ag
190
191
192
H
Pogo A
(10)
193
then the objective for the feature extractor is defined as
(0) c
CCE(R R ( H.)) A).
(11)
Here R(H.)) H represents the correlation between the hidden embeddings of each pair of nodes.
194
195
3.3.3
Auxiliary Training for the Label Generator Sip
196
The label generator de is a vanilla GCN model used to predict the label distribution for each node, which naturally needs to be supervised by a classification loss. In the 7th iteration, a label generator first generates the prediction results using the original graph adjacency matrix A and the node features
197
198
X, which is formulated as Y 9. (X, A). Denoting the training labels as Y Ytrain the objective
199
200
of the label generator IGp can be formulated as
LCF (o((t) train Ytrain).
(12)
201
3.3.4
Meta-Learning Based Training Strategy
202
The final node classification results only depend on the primary task, while the performance of the other modules, including the label generator and link predictor are not our ultimate concern. Simply using the classification loss to train the label generator Ie or using the link prediction loss to train the feature extractor log cannot provide the effects of auxiliary learning. Thus, it crucial to make the node classifier fe, perform better after it is trained with the pseudo soft labels while taking the hidden embeddings derived by the feature extractor he, as inputs. To this end, we use meta auxiliary learning to update the model parameters.
203
204
205
206
207
209
For the auxiliary label generation task, we consider not only the classification performance of the label generator 9., but also the auxiliary effect on the node classifier. We assume that the classifier parameters 02 are updated with gradient descent based on the pseudo soft labels in the 7th iteration,
210
211
81 0 v/ 8, MMSE (SMSEEYY() ((t), yr((t))
(13)
212
A direct way to evaluate the auxiliary effect of the pseudo soft labels is to compute the classification loss of the prediction given by the node classifier using the updated parameters 8/, as
213
c = Lce(fe; (fe; (hogo) (X, Arecon), A) Arrsiin Ytrain).
(14)
This loss quantify how much performance improvement the classifier gains from the two auxiliary tasks. Because 0' is updated with the pscudo soft labels generated by (-), the loss Lt is also a function of yp. This means that the objective could be used to supervise the learning of yp. Note
215
that C requires the gradient of the gradient to be computed (25), which can be considered as a meta-learning strategy. The final objective of the label generator 9. with meta-leaming is formulated
217
219 as
( - C + CL'
(15)
Similar to the label generator 9 (-), the link predictor h2, should derive the effective node embed- dings to enhance the node classification performance. Since the hidden embeddings used for node
220

---

Algorithm AL-GCN
Input: Graph adjacency matrix A, the node features X, the data labels Y train of a training set. Output: A feature extractor h6g. node classifier fo.
1: Initialize learnable parameters 0,, 02,
2: while not converged do node classifier training phase
3:
4:
Y + 9p (X, A) hog A)
5:
6:
Areon = detach (R(ac R(H,
7:
Hrezon f h4g X,, Arcen)
8:
Y t fo. (Hrocons A)
Co, + LCE( train' traii) + LMSE Y Update: 01 Adam (L
9:
11:
# meta-learning preparation
12:
Compute: 8f +- 81 Vo, LMSE Y Y L t LCE(SO) (oo, (he, (X, Arcoo), A Irains Y (usin)
13:
14:
# label generator training phase Y + 9x(X, A)
15:
C.p f CY train: Y teaam) ) C51
17:
Update: ip t A s)
feature extractor training phase
19:
A, + RandomSample(A
20:
H. e (X,,(, A.)
Cø + LCE (R(H,, H,), A) C5'
22:
Update: 02 + Adam(Le,
end while
222
classification are derived from the feature extractor hag the objective defined in Eq. 14 can also be considered function of Og. Thus, the objective of the feature extractor ha, with meta-learning defined as
223
2/3 C
(16)
meta
225
This objective means that, except for the link prediction task, the feature extractor he, should ensure that the generated hidden embedding enhance the classification accuracy of the node classifier fa. updated with the pseudo soft labels. The overall training process shown Alg.
228
227
Experiments
228
229
4.1
Experimental Settings and Compared Methods
We demonstrate the classification performance of our method via semi-supervised document classifi- cation on three citation network datasets, including Cora, Citeseer, and Pubmed [26), where nodes represent the documents and edges are citation links. Dataset statistics summarized in Table [
231
233
The proposed method a novel GCN, which is leveraged to carry out a node classification task. We compare our method with several popular graph-based node classification methods including GCN [2], GAT 31, DualGCN [13), SGC (27). and APPNP [4). As we use a three-layer GCN as
235
the backbone in proposed model, compare both two-layer and three-layer GCNs, denoted as GCN2 and GCN3, respectively.
237
For the semi- cemi-supervised node classification, we use all node features but only 20 labels per class for training and 500 nodes the validation set. We train the proposed method for maximum of 400 epochs using Adam [28] with learning rate of 0.01. We use the well-trained parameters, which achieve the best performance on validation set during accuracy on test set of 1,000 labeled examples. We run each method 100 times and compute average classification accuracy on single NVIDIA GTX 1080Ti GPU. To implement all the
239
241
training phases, to evaluate classification
243
compared methods more conveniently, use PyG 29 the graph-based learning framework.

---

Table I: Dataset statistics.
Dataset
Nodes
Edges
Features
Classes
Cora
2,708
5,278
1,433
7
CiteSeer
3,327
4,552
3,703
6
Cora
19,717
44,324
500
3
Table 2: Classification results on the datasets (bold: best, underline: runner-up).
Method
Cora
Citeseer
Pubmed
GAT 33
82.5 + 0.8%
71.4 0.7%
78.4 + 0.4%
DualGCN 13
83.4 + 0.5%
72.6 + 0.6%
79.9 + 0.3%
SGC 27
=
78.2 = 0.5%
APPNP 4
0.4%
71.6 0.5%
79.8 = 0.3%
GCN2 12
81.5
71.5 0.5%
79.2 = 0.4%
AL-GCN2 (ours)
82.3
72.6 13.6::00 0.5%
79.6 = 0.5%
0.5%
GCN3 12
80.7 50.7=1.2%
68.0 = 1.4%
77.7 = 0.5%
AL-GCN3 (ours)
84.7 0.4%
72.3 0.5%
81.4 0.6%
245
4.2
Experimental Results
246
In this section, we provide the experimental results of the node classification, an ablation study, and the visualization of hidden embeddings. More experimental results, such as parameter and model robustness studies can be found in the supplementary materials.
247
We conduct the node classification task three citation network datasets. As shown in Table 2 our method consistently and significantly enhances the learning performance compared to the other methods. In particular, for the Cora dataset, the proposed method is superior GCN by 4.9%.
250
251
252
Compared to the other methods, our model considers more graph-structured information via auxiliary learning. Thus, it is consistently and significantly superior the compared methods, achieving
253
254
ofte-of-the-ar results.
255
4.3
Ablation Studies
256
4.3.1
On the Auxiliary Learning Modules
257 To determine how the link predictor (P) and the pseudo label generator (G) affect the node classi- fication performance, apply the following two ablation models: (1) Vanilla GCN with the link predictor, termed GCN+P. (2) Vanilla GCN with the lable generator, termed GCN+G. We compare GCN+G and GCN+P with the proposed method, AL-GGN (GCN+P+G), and the original GCN. Table 3 lists the results. As can be seen, the proposed method consistently outperforms GCN+P and GCN+G. Specifically, compared the link predictor, the label generator has more effect on the Citeseer dataset. However, for the Pubmed dataset, the link predictor has much more effect on the
259
261
262
263
264
learning performance.
285
4.3.2
On the Reconstructed Graph Adjacency Matrix
286
For the backbone network, a reconstructed graph adjacency matrix via link prediction task used input. To determine how the reconstructed graph adjacency matrix affects the node classification performance, we compare the proposed model with the following two models: (1) A model that
266
takes the original graph adjacency matrix without the reconstructed one the input of the backbone network, termed w/o-recG; (2) A model that takes only the reconstructed graph adjacency matrix as the input of the backbone network, termed w/o-oriG.
270
272
As shown Table 4 our method consistently outperforms w/o-recG and w/o-oriG. This can be attributed to the fact that the reconstructed graph adjacency matrix via the link predictor can capture
274
detailed topology information graph and the fixed original graph adjacency matrix increases
the training stability.

---

Table 3: The ablation experiment results.
Method
Cora
Citeseer
Pubmed
GCN
80.7 + 1.2%
68.0 = 1.4%
77.7 0.5%
GCN+P
83.0 + 0.7%
70.6 + 0.8%
81.3 = 0.6%
GCN+G
83.0 + 0.6%
71.7 + 0.8%
78.8 0.6%
GCN+P+G (AL-GCN)
84.7 + 0.4%
72.3 + 0.5%
81.4 + 0.6%
Table 4: The ablation experiment results in terms of classification accuracy (in percent).
Method
Cora
Citeseer
Pubmed
GCN
80.7 + 1.2%
68.0 + 1.4%
77.7 0.5%
w/o-reeG
84.5 + 0.5%
71.9 + 0.6%
80.1 + 0.5%
w/o-oriG
84.1 + 0.6%
71,4
80.6 + 1.5%
AL-GCN
84.7 + 0.4%
7 0.5%
81.4 + 0.6%
(a) GCN
(b) GAT
(c) AI-GCN
Figure 4: Visualization of the hidden embedding obtained by different methods via I-SNE algorithm.
276
4.4
Visualization of Hidden Embeddings
To determine how the hidden embeddings affect the learning performance, use the visualization tool I-SNE 30 to observe their distribution. As shown in Fig. 4 the embedding results of GCN and GAT are denser, and the separation of different clusters not obvious. contrast, the node distributions learned by our proposed method are more separate, with most of the nodes from the same classes being close each other, resulting obvious cluster structures. These experimental results demonstrate that the proposed method can capture more detailed structure information of graph, including the nodes and edges, resulting in more effective hidden embeddings.
278
279
281
283
Conclusion
284
We have proposed novel graph convolutional network for semi-supervised node classification. Different from existing methods, the proposed model focuses on enriching the graph data and adopts meta auxiliary learning to enhance the representations of nodes and edges in graph. To enrich node label information, an auxiliary label generator is used to generate pseudo probabilistic labels. Meanwhile, auxiliary link predictor is used generate probabilistic edges enrich the graph structure information. The enriched node and edge information can iteratively enhance the performance of the node classification task. Experimental results on several benchmark citation datasets show that the proposed model is superior to the existing methods. For future work, we note that real-world data is usually contaminated by noise, which results a robustness problem for graph learning methods. We plan to extend our model to handle noisy data by designing a more robust
286
288
280
282
293
295
learning method for graph-structured data.

---

References
296
297
[1] David Duvenaud, Dougal Maclaurin, Jorge Aguilera-Iparrauuin Iparraguirre, Rafael
298
Timothy Hirzel, Alán Aspuru-Guzik, and Ryan P. Adams. Convolutional networks on graphs for learning molecular fingerprints. In Advances in Neural Information Processing Systems, pages 2224-2232, 2015.
299
300
301
[2] Rex Ying, Ruining He. Kaifeng Chen, Pong Eksombatchai, William L. Hamilton, and Jure Leskovec. Graph convolutional neural networks for web-scale recommender systems. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 974-983, 2018.
302
303
304
305
[3] Petar Velickovic, Guillem Cucurull, Aranixa Casanova, Adriana Romero, Pictro Liò, and Yoshua Bengio. Graph attention networks. In 6th International Conference on Learning Representations, 2018.
308
307
308
[4] Johannes Klicpera, Aleksandar Bojchevski, and Stephan Günnemann. Predict then propagate: Graph neural networks meet personalized pagerank. In 7th International Conference on Learning Representations, 2019.
309
310
[5] Xiao Wang, Meiqi Zhu, Deyu Bo, Peng Cui, Chuan Shi, and Jian Pei. AM-GCN: adaptive multi- channel graph convolutional networks. In Proceedings of the 26t h ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 1243-1253, 2020.
312
313
[6] Keyulu Xu, Chengtao Li, Yonglong Tian, Tomohiro Sonobe, Ken-ichi Kawarabayashi, and Stefanie Jegelka. Representation learning on graphs with jumping knowledge networks. In Proceedings of the 35th International Conference on Machine Learning, volume 80, pages 5449-5458, 2018.
315
316
317
318
[7] Lingxiao Zhao and Leman Akoglu. Pairnorm: Tackling oversmoothing in gnns. In 8th International Conference on Learning Representations, 2020.
88] Ming Chen, Zhewei Wei. Zengfeng Huang, Bolin Ding, and Yaliang Li. Simple and deep graph convolutional networks. In Proceedings of the 37th International Conference on Machine Learning, volume 1 119, pages 1725-1735, 2020.
321
322
323
99] Qimai Li, Zhichao Han, and Xiao-Ming Wu. Deeper insights into graph convolutional networks for semi-supervised learning. In Proceedings of the Thirty- AAAI Conference on Artificial Intelligence, pages 3538-3545 2018
324
326
[10] Joan Bruna, Wojeiech Zaremba, Arthur Szlam, and Yann LeCun. Spectral networks and locally connected networks on graphs. In 2nd International Conference on Learning Representations, 2014.
327
329
[11] Michaël Defferrard, Xavier Bresson, and Pierre Vandergheynst. Convolutional neural networks on graphs with fast localized spectral filtering. In Advances in Neural Information Processing Systems, pages 3837-3845, 2016.
330
332
[12] Thomas N. Kipf and Max Welling, Semi-supervised classification with graph convolutional networks. In Sth International Conference on Learning Representations, 2017.
333
334
[13] Chenyi Zhuang and Qiang Ma. Dual graph convolutional networks for graph-based semi- supervised classification. In Proceedings of the 2018 World Wide Web Conference on World Wide Web, pages 499-508, 2018
335
336
337
[14] Zonghan Wu, Shirui Pan, Fengwen Chen, Guodong Long, Chengqi Zhang, and Philip S. Yu. A comprehensive survey on graph neural networks. CoRR,
338
2019.
339
[15]
Alessio Micheli. Neural network for graphs: A contextual constructive approach. JEEE Trans. Neural Networks, 20(3):498-51,, 2009.
340
[16]
James Atwood and Don Towsley. Diffusion--comvoutionn neural networks. In Advances in Neural Information Processing Systems, pages 1993-2001,, 2016.
342
343
Justin Gilmer, Samuel S. Schoenholz, Patrick Riley, Oriol Vinyals, and George E. Dahl. Neural message passing for quantum chemistry. In Proceedings of the 34th International Conference on Machine Learning, volume 70, pages 1263-1272, 2017.
345
346
[18] Federico Monti, Davide Boscaini, Jonathan Masci, Emanuele Rodolà, Jan Svoboda, and Michael M. Bronstein. Geometric deep learning graphs and manifolds using mixture
10

---

348
model cnns. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, pages 5425-5434, 2017.
349
350
[19]
Hongyang Gao, Zhengyang Wang, and Shuiwang Ji. Large-scale learnable graph convolutional networks. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 1416-1424, 2018.
351
352
[00]
John Flynn, Ivan Neulander, James Philbin, and Noah Snavely. Deepstereo: Learning to predict new views from the world's imagery. CoRR, abs/1506.06825, 2015.
354
355
[21]
Shubham Toshniwal, Hao Tang, Liang Lu, and Karen Livescu. Multitask learning with low-level auxiliary tasks for encoder-decoder based speech recognition. In 18th Annual Conference of the International Speech Communication Association, pages 3532-3536, 2017.
358
357
[22]
Timothy M. Hospedales, Antreas Antoniou, Paul Micaelli, and Amos J. Storkey. Meta-learning in neural networks: A survey. CoRR, abs/2004.05439, 2020.
359
360
[3
Dong-Hyun Lee. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In Workshop on Challenges in Representation Learning, volume 3, 2013.
362
[24]
Hieu Pham, Qizhe Xie, Zihang Dai, and Quoc V. Le. Meta pseudo labels. CoRR, abs/2003. 10580, 2020.
383
[25]
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agmostic meta-learning for fast adap- tation of deep networks. In Proceedings of the 34th International Conference on Machine Learning, volume 70, pages 1126-1135, 2017.
385
387
[66] Prithviraj Sen, Galileo Namata, Mustafa Bilgic, Lise Getoor, Brian Galligher, and Tina Eliassi- Rad. Collective classification in network data. Magazine, 29(3):93, September 2008
368
369
[77]
Felix Wu, Amauri H. Souza Jr., Tianyi Zhang, Christopher Fifty, Tao Yu, and Kilian Q. Wein- berger. Simplifying graph convolutional networks. In Proceedings of the 36th International Conference Machine Learning, volume 97, pages 6861-6871, 2019.
370
371
[28] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In 3rd
373
International Conference on Learning Representations, 2015.
374
[29]
Matthias Fey and Jan Eric Lenssen. Fast graph representation learning with pytorch geometric. CoRR, abs//903,00 002428, 2019.
375
376
[30] Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, (Nov):2579-266) 2008
377
Checklist
378
379
1. For all authors...
380
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
383
(c) Did you discuss any potential negative societal impacts of your work? [No]
304
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
385
386
2. If you are including theoretical results...
387
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
388
(b) Did you include complete proofs of all theoretical results? [N/A]
389
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main exper- imental results (either in the supplemental material or as a URL)? [Yes] The codes are included in the supplemental material and the datasets can be downlowded by the codes automaticly.
391
393
384
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] They are included in the supplemental material.
395
396
(c) Did you report error bars (e-g., with respect to the random seed after running experi- ments multiple times)? [Yes] See Section 4.1
11

---

(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Section 4.1
399
4. If you using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] See Section 4.1
(b) Did you mention the license of the assets? [No]
403
(c) Did you include any new assets either in the supplemental material or as URL? [No]
404
(d) Did you discuss whether and how consent was obtained from people whose data you'te 'r using/curating? [No]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [No]
407
408
5. If you used crowdsourcing or conducted research with human subjects...
409
(a) Did you include the full text of instructions given participants and screenshots, applicable? [N/A]
410
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
412
413
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
414
12


Reviews:
[{"review_id": "gJ75LuYIpzH", "reviewer": "Reviewer_t7yf", "summary": "The paper proposes a new auxiliary learning method to improve the existing GCNs. Considering that the edges in many graphs are unweighted and nodes labels are often one-hot, the authors added two auxiliary tasks to enrich the topology information in the graph node classification task. One is link prediction which generates probabilistic edges; and the other is to generate soft labels. The probabilistic edges and soft labels are iteratively updated by a meta auxiliary learning strategy in a smart way, and the results on three standard benchmark datasets outperform some baselines.\n", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The motivation of the paper is reasonable but maybe not so significant. It is limited to the node classification task on graphs with binary edges. In practice many graphs have weights or attributes on edges, and the application of most GNNs are also not limited to the node classification task. \n\nThe paper combines the ideas of auxiliary learning and meta pseudo labels to enrich the topology information. It is novel in the context of graph learning, but some concepts or descriptions are confusing. For example, I am not sure whether the label generation can be regarded as an auxiliary task, since g_\\phi is a totally different model which does not share any parameters with the target model in the main task.\n\nThe experiments are weak. As the area of deep graph learning is developing so fast, just using the three small graph datasets with standard settings are not enough nowadays. It cannot give us enough insights why and how the proposed method works. For example, 1. the advantage of using probability edges instead of original ones may be making the model not only more expressive but also robust. Can you make some perturbations on the edges and see how the performance changes? 2. The utilization of teacher model and pseudo labels maybe make the unlabeled data more efficiently used, so can you change the proportion of labeled/unlabeld data  to show the power of your model? Or can you experiment on larger datasets such as OGB? \nThis kind of experiments will make the paper more solid. Otherwise, it is not convincing. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "\nI do not see any negative social impact.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "flhX81onnv8", "reviewer": "Reviewer_rGes", "summary": "This paper proposes a novel meta-learning framework that utilizes auxiliary learning tasks of link predictions and label generations to improve the performance of GCNs on node classifications tasks. The meta-learning algorithm adopts the idea from [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://arxiv.org/abs/1703.03400) so that the models for auxiliary tasks are updated in a way that has higher impacts on the primary task.  The authors use two self-supervised learning tasks, edge predictions and label generation, as the auxiliary tasks.  The method performs favorably in the three standard public graph data sets with higher prediction accuracy.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper is generally well written and well structured. I appreciate the authors' efforts to keep the discussions organized and easy to understand. The experiment's setting is reasonable and the results of the experiment suggest that the proposed method is able to achieve higher prediction accuracy. And the design choices are properly examined in the ablation study.\n\nUsing self-supervised tasks as auxiliary learning tasks in GNNs has been discussed in a few pieces of literature recently, such as [1][2]. Different from prior arts, this paper presents a novel meta-learning algorithm to the GNNs that using the impacts on the primary tasks to direct the update of the auxiliary tasks. Thus the primary tasks might suffer less from degeneration, which is common for multi-task learning.  However, I am curious about the difference between this method and the one in [3], which is general auxiliary learning using self-supervised tasks and a similar meta-learning. In general, I find the contribution of this paper quite incremental.\n\n\n[1] Multi-Stage Self-Supervised Learning for Graph Convolutional Networks on Graphs with Few Labeled Nodes\n\n[2] When Does Self-Supervision Help Graph Convolutional Networks? \n\n[3] Self-Supervised Generalisation with Meta Auxiliary Learning, NeurIPS 2019.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The limitations and potential negative sociteal impacts are not addressed. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "XfeRLxYTwIf", "reviewer": "Reviewer_KGBu", "summary": "In this paper, they introduced a method for node classification in graphs using Graph Convolutional Networks (GCNs) in multi-task fashion to address incomplete annotation problem. Two auxiliary tasks, link prediction and label generation, are trained with the target task (node classification) simultaneously using a graph meta-learning approach. \n", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\n\n\nStrengths:\n\n- In general, paper is well-written and well-organized; e.g.,  Algorithm 1 is self-descriptive. There are some small confusion for me that I mention later.\n\n-  I like the idea of using two related auxiliary tasks to improve node classification problem; specifically, they used two link prediction and label generation tasks, in which they leveraged graph-structured information to improve the performance of target task (node classification).\n\n- The experiments are thorough, they cover both qualitative and qualitative analysis, the ablation studies were interesting\n\nNotes:\n\n- Figure 1 is a little bit confusing for me. I would suggest using the citation network example instead of image example in the figure 1 to be consistent with the experiment data.\n\n- For figure 2 and 3, I would recommend adding explanation related to color-coded features and nodes.for example, what green and blue colored nodes mean? it will be also better if they could add some explanation in the captions of tables.\n\n- In the link prediction task, if I understand it correctly, they used the similarity of every two node to compute if edge is between them? What is the threshold to infer there is an edge between two nodes?\n\n- In the link prediction task, it would be useful to mention the intuition why they use a sample of edges for each iteration? Is it related to time or complexity issues?\n\n- In equation 8, for calculating the loss, I think for the psudo labels, you could use the cross entropy loss instead of MSE. What is the intuition of using MSE?\n\n- In section 4.3.2, I do not understand the difference between \"w/o-oriG\" and \"AL-GCN\". My assumption is that both of them use the reconstructed adjacency matrix for backbone model. What is the difference between these two?  \n\n- In comparison with the other node classification methods, I would recommend adding a new paraphrase explaining why we should use \"AL-GCN\" instead of other methods; basically what is the winner characteristics of this method comparing others. Does this have better time complexity? Or does it perform better in semi-supervised setting? I understand it outperforms previous ones with respect to accuracy, but we could argue other benefits of using this approach\n\n- I think this framework could be model-agnostic. I means we will be able to use GATs instead of of GCN? This will be nice future work.\n\n- In line 56, one extra \"sudo\"?\n\n**update: After receiving authors' response, I would like to appreciate their time for addressing my concerns. My evaluation on the paper is reflected after reading their response.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "As far as I understand, for the limitations, they mentioned that they plan to address the noise in the data by designing more robust method.\n\nThey did not mentioned any potential negative societal impact in the work; and I do not see there could be any negative societal impact in this work as well.", "review_text": "", "extra_scores": {"ethical_concerns": "I have not observed any ethical issues in this work.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper has merit and received mixed reviews, but overall it is too incremental. I think there are several avenues along which the paper can be strengthened, some of which are mentioned in the discussion. Pursuing those, it seems likely to get to a paper with critical mass for a top tier ML conference. The submitted paper is not quite there yet.
Decision: Reject