Auxiliary Learning Induced Graph Convolutional Networks
Year: 2021

Abstract:
Graph convolutional networks (GCNs) have recently achieved great success in 1 many applications. However they suffer from an incomplete annotation problem 2 for complex graph-structured data. In this paper, we introduce a novel auxiliary 3 learning method for GCNs in a multi-task fashion, which can efficiently enrich the 4 data annotations. Specifically, both link prediction and label generation are used as 5 two auxiliary tasks to complement the primary task of node classification. These 6 two auxiliary tasks are jointly trained with the primary node classification task 7 via a graph meta-learning strategy. The experimental results demonstrate that the 8 proposed method consistently and significantly outperforms existing methods and 9 achieves state-of-the-art results on several benchmark citation network datasets. 10

text: Graph convolutional networks (GCNs) have recently achieved great success in1 many applications. However they suffer from an incomplete annotation problem2 for complex graph-structured data. In this paper, we introduce a novel auxiliary3 learning method for GCNs in a multi-task fashion, which can efficiently enrich the4 data annotations. Specifically, both link prediction and label generation are used as5 two auxiliary tasks to complement the primary task of node classification. These6 two auxiliary tasks are jointly trained with the primary node classification task7 via a graph meta-learning strategy. The experimental results demonstrate that the8 proposed method consistently and significantly outperforms existing methods and9 achieves state-of-the-art results on several benchmark citation network datasets.10
1 Introduction11
Graph-structured data is ubiquitous in real-world applications. However, general deep learning12 methods, such as convolutional neural networks (CNNs), cannot adapt to graph-structured data13 directly, because the nodes in a graph have different numbers of neighbors, which often lose the14 ranking information. To handle the graph data effectively, graph convolutional networks (GCNs)15 have recently been proposed and used in many applications such as biomolecular prediction [1] and16 recommendation systems [2].17
Previous methods focus on designing models that can extract information from both the graph18 topology and node features. Specifically, existing GCN methods typically design different propagation19 strategies [3, 4, 5] for each network layer and stack more network layers [6, 7, 8] to derive larger20 receptive fields. However, the neighborhood aggregation is essentially a type of Laplacian smoothing21 and stacking too many layers may result in over-smoothing [9]. These drawbacks of existing methods22 limit further performance enhancement.23
In this paper, we try to explore the bottleneck of node classification from another point of view, i.e.,24 from the training data itself. As shown in Fig. 1, graph-structured data has different properties from25 grid-like data such as an image. The most obvious difference is that the nodes in a graph are connected26 by edges. This causes two main issues whien it comes to annotating graph-structured data, resulting in27 that existing methods cannot fully leverage the graph-structured information. First, the edges in most28 graph data for semi-supervised node classification are unweighted. This arbitrary edge indication29 setting cannot effectively reflect the detailed graph structures. Besides, graph-structured data may be30 contaminated with noisy edges. These noisy edges cannot represent the true pairwise relationships31 between nodes. Second, using one-hot labels to train a graph-based model is inappropriate. One-hot32 labels are widely used in various machine learning tasks, assigning a training sample to a single class.33 However, nodes in a graph are connected; even nodes with different classes may have relations. In34 this scenario, it is more suitable to use soft labels to assign a node to multiple classes, with different35 probabilities indicating which class the node possibly belongs to.36
Submitted to 35th Conference on Neural Information Processing Systems (NeurIPS 2021). Do not distribute.
Image Graph
2
2
2
1
1
3 3
1 1
1 1
1 1
1
1
1
1
(1) We propose two auxiliary tasks to capture more accurate graph information and enhance the52 model performance. The auxiliary link prediction task ensures that the model captures more graph53 topology information and generates probabilistic edges. The auxiliary label generation task softens54 the one-hot labels and generates pseudo-labels for unlabeled nodes.55
(2) The reconstructed edges and pseudo sudo labels derived via the two auxiliary tasks are iteratively56 updated with a node classifier of the primary task based on a meta auxiliary learning strategy, resulting57 in state-of-the-art node classification performance.58
2 Related Work59
Over the past few years, GCNs have achieved significant breakthroughs in graph data representation.60 Generally, existing GCNs can be divided into spectral-based methods and spatial-based methods.61
The spectral-based methods use graph spectral theory to define the graph convolutional operation62 in a graph Fourier domain. Spectral CNN [10] follows these mathematical foundations, assuming63 that a convolutional filter is a set of learnable parameters. To reduce computational complexity,64 ChebNet [11] approximates a graph convolutional filter as Chebyshev polynomials of the eigenvalues.65 GCN [12] introduces a first-order approximation of ChebNet and proposes a renormalization trick to66 alleviate numerical instabilities and exploding/vanishing gradients. DualGCN [13] introduces a dual67 GCN architecture with two graph convolutional layers in parallel to encode both local and global68 structural information.69
The spatial-based methods define feature aggregation in the spatial domain directly, which is more70 efficient, general, and flexible [14]. The key challenge for these spatial-based methods is to apply71 the convolution operation for different-sized neighborhoods, while at the same time maintaining the72 weight sharing property. Neural network for graphs (NN4G) [15] is the first spatial-based method,73 applying the graph convolutional operation in the spatial space. Diffusion convolutional neural74 networks (DCNN) [16] consider graph convolutions as diffusion processes to efficiently learn features75 that are invariant under isomorphism. Message passing neural networks (MPNN) [17] model graph76 convolution as a message passing process among the nodes. The graph attention network (GAT)77 [3] introduce masked self-attentional layers to assign different weights to adjacent nodes, leading to78 learnable filter weights. The mixture model network (MoNet) [18] introduces pseudo-coordinates to79 assign different weights to the neighbors of each node. To achieve weight sharing across different80 nodes, some spatial-based models attempt to rank a node’s neighbors via certain criteria or metrics,81 which transforms the graph-structured data into grid data for further processing. The large-scale82 graph convolutional network (LGCN) [19] ranks a node’s neighbors via the node feature values. Then,83 multiple 1D convolutional layers are stacked for feature aggregation. Approximate personalized84 propagation of neural predictions (APPNP) [4] takes the personalized PageRank algorithm as the85 model propagation method to avoid over-smoothing when stacking more layers or increasing the size86 of the neighborhood.87
Multi-task learning is designed to simultaneously learn a set of related but different tasks for ensuring88 that a learning model can derive the best performance across all tasks. Different from multi-task89 learning, auxiliary learning is only concerned with model performance on the primary task. For90
<latexit sha1_base64="xEfXpeR/Fa79GZHZXEkfLplChxE=">AAACV3icbVFNSwMxEM2uX7V+VT16CRahPVh2paIXoeDFYwVrhW4t2TTbxibZNckKJeRPipf+FS+a1lX8Ggi8eTPzMnmJM0aVDoKZ5y8tr6yuldbLG5tb2zuV3b1bleYSkw5OWSrvYqQIo4J0NNWM3GWSIB4z0o0nl/N694lIRVNxo6cZ6XM0EjShGGlHDSoi4kiP48RwOzAP9t7UJnULL2CUSIRNaE2kHqU2ZtEmuRmSka3R+o/0oW6t/dQZf+kch/UvtlsoDyrVoBEsAv4FYQGqoIj2oPIcDVOccyI0ZkipXhhkum+Q1BQzYstRrkiG8ASNSM9BgThRfbPwxcIjxwxhkkp3hIYL9vuEQVypKY9d53xP9bs2J/+r9XKdnPcNFVmuicAfFyU5gzqFc5PhkEqCNZs6gLCkbleIx8g5qt1XlJ0J4e8n/wW3J42w2Ti9blZbrcKOEjgAh6AGQnAGWuAKtEEHYPACXr0lb9mbeW/+ql/6aPW9YmYf/Ah/9x0TebbR</latexit>
<latexit sha1_base64="+Tu+mvVFlWMCdmfnb9sBdyu+r8M=">AAACRnicbVBNaxsxEJ11vp0vtz3mImIC9sXshoT2UjD00lNIIE4C3s2ilbW2bEm7SNqAEfvreum5t/yEXHpoCb1WtjeQ2BkQPN6beTN6Sc6ZNr7/6NXW1jc2t7Z36rt7+weHjQ8fb3RWKEJ7JOOZukuwppxJ2jPMcHqXK4pFwultMvk2028fqNIsk9dmmtNI4KFkKSPYOCpuRKHAZpSkdlTGlpX3tjVpl+grmtNKWExM2Qp1IWI7Dpl0LSgkRb7QCeb2omyxNirRi49wPuPKpx03mn7HnxdaBUEFmlDVZdz4FQ4yUggqDeFY637g5yayWBlGOC3rYaFpjskED2nfQYkF1ZGdx1CiE8cMUJop96RBc/b1hMVC66lIXOfsWL2szcj3tH5h0i+RZTIvDJVksSgtODIZmmWKBkxRYvjUAUwUc7ciMsLKZeeSr7sQguUvr4Kb005w1jm/Omt2u1Uc23AEx9CCAD5DF77DJfSAwA94gj/w1/vp/faevX+L1ppXzXyCN1WD/wA8sp0=</latexit>
<latexit sha1_base64="Cru5BVsOASECHoYReBIwFRt/mog=">AAACA3icbVDLSsNAFL2pr1pfUXe6GSxCBSmJVHThoiCCywr2AW0ok+mkHTp5MDMRSgi48VfcuFDErT/hzr9x0gbR6oGBM+fcy733uBFnUlnWp1FYWFxaXimultbWNza3zO2dlgxjQWiThDwUHRdLyllAm4opTjuRoNh3OW2748vMb99RIVkY3KpJRB0fDwPmMYKVlvrmXqXnYzUimCet9Bh9f67So75ZtqrWFOgvsXNShhyNvvnRG4Qk9mmgCMdSdm0rUk6ChWKE07TUiyWNMBnjIe1qGmCfSieZ3pCiQ60MkBcK/QKFpurPjgT7Uk58V1dmO8p5LxP/87qx8s6dhAVRrGhAZoO8mCMVoiwQNGCCEsUnmmAimN4VkREWmCgdW0mHYM+f/Je0Tqp2rXp6UyvXL/I4irAPB1ABG86gDtfQgCYQuIdHeIYX48F4Ml6Nt1lpwch7duEXjPcvQ5yXRQ==</latexit>
<latexit sha1_base64="pY/ClCGqBGnEDcJm6l7QNvlJoPM=">AAAB8XicbVC7SgNBFL0bXzG+opY2g0GwCrsS0cIiYGMZwTwwWcLsZDYZMju7zNwVwpK/sLFQxNa/sfNvnCRbaOKBgcM59zLnniCRwqDrfjuFtfWNza3idmlnd2//oHx41DJxqhlvsljGuhNQw6VQvIkCJe8kmtMokLwdjG9nfvuJayNi9YCThPsRHSoRCkbRSo+9iOIoCLPOtF+uuFV3DrJKvJxUIEejX/7qDWKWRlwhk9SYrucm6GdUo2CST0u91PCEsjEd8q6likbc+Nk88ZScWWVAwljbp5DM1d8bGY2MmUSBnZwlNMveTPzP66YYXvuZUEmKXLHFR2EqCcZkdj4ZCM0ZyokllGlhsxI2opoytCWVbAne8smrpHVR9WrVy/tapX6T11GEEziFc/DgCupwBw1oAgMFz/AKb45xXpx352MxWnDynWP4A+fzB81jkP4=</latexit>
<latexit sha1_base64="Cru5BVsOASECHoYReBIwFRt/mog=">AAACA3icbVDLSsNAFL2pr1pfUXe6GSxCBSmJVHThoiCCywr2AW0ok+mkHTp5MDMRSgi48VfcuFDErT/hzr9x0gbR6oGBM+fcy733uBFnUlnWp1FYWFxaXimultbWNza3zO2dlgxjQWiThDwUHRdLyllAm4opTjuRoNh3OW2748vMb99RIVkY3KpJRB0fDwPmMYKVlvrmXqXnYzUimCet9Bh9f67So75ZtqrWFOgvsXNShhyNvvnRG4Qk9mmgCMdSdm0rUk6ChWKE07TUiyWNMBnjIe1qGmCfSieZ3pCiQ60MkBcK/QKFpurPjgT7Uk58V1dmO8p5LxP/87qx8s6dhAVRrGhAZoO8mCMVoiwQNGCCEsUnmmAimN4VkREWmCgdW0mHYM+f/Je0Tqp2rXp6UyvXL/I4irAPB1ABG86gDtfQgCYQuIdHeIYX48F4Ml6Nt1lpwch7duEXjPcvQ5yXRQ==</latexit>
<latexit sha1_base64="jpDvRQYjmmhsYjhkn3eweWYZXkE=">AAAB+XicbVDLSsNAFL2pr1pfUZduBotQNyWRii5cFNx0WcE+oI1lMp20QyeTMDMplNA/ceNCEbf+iTv/xkmbhbYeGDiccy/3zPFjzpR2nG+rsLG5tb1T3C3t7R8cHtnHJ20VJZLQFol4JLs+VpQzQVuaaU67saQ49Dnt+JP7zO9MqVQsEo96FlMvxCPBAkawNtLAtvsh1mM/SBvzp7TiXs4HdtmpOgugdeLmpAw5mgP7qz+MSBJSoQnHSvVcJ9ZeiqVmhNN5qZ8oGmMywSPaM1TgkCovXSSfowujDFEQSfOERgv190aKQ6VmoW8ms5xq1cvE/7xeooNbL2UiTjQVZHkoSDjSEcpqQEMmKdF8ZggmkpmsiIyxxESbskqmBHf1y+ukfVV1a9Xrh1q5fpfXUYQzOIcKuHADdWhAE1pAYArP8ApvVmq9WO/Wx3K0YOU7p/AH1ucP836TMw==</latexit>
instance, Deepstereo [20] leverages auxiliary learning to predict the relative poses of multiple cameras91 for unsupervised monocular depth estimation. To improve the performance of conversational speech92 recognition, auxiliary learning [21] is applied to low-level representations. Compared to the common93 learning scheme, meta auxiliary learning can enhance learning performance. For instance, MAXL94 [22] adopts meta-learning to automatically generate the auxiliary task labels. Pseudo Label [23] is95 a semi-supervised learning method, where a deep neural network is trained using both labeled and96 unlabeled data. For unlabeled data, the model picks up the class that has the maximum predicted97 probability as the true label to train itself. MPL [24] extends the Pseudo Label [23] via a meta-learning98 strategy, where the pseudo-labels are not generated by itself, but by a teacher network.99
The existing GCNs are designed for a single task, where the properties of graph-structured data100 are not fully explored. We introduce the auxiliary learning scheme to leverage more detailed graph101 topology information for enhancing node classification performance.102
3 Method103
3.1 Preliminaries104
Given a graph G = {V, E ,X}, V is a set of nodes and E is a set of the edges connecting the related105 nodes. X ∈ RN×d represents the features matrix of the nodes, where d is the dimension of the node106 features and N = |V| is the number of nodes.107 The proposed method adopts the vanilla GCN [12] as the backbone network, taking the graph108 adjacency matrix A, labeled training set Ytrain, and original features X of the nodes as inputs to109 perform the semi-supervised node classification task. Based on the MPNN framework [17], each110 layer of the vanilla GCN is defined in three parts:111
(1) Message computation: The message of node vi and its neighbor node vj is calculated, where112 j ∈ N (i), as:113
m (k) j = 1√ deg(i)deg(j) h (k−1) j W
(k), j ∈ {i} ⋃ N (i). (1)
Here, deg(i) is the degree of node vi and W(k) ∈ Rck−1×ck are the learnable parameters of the114 kth-layer, where ck is the size of the hidden embedding.115
(2) Aggregation: The messages of node vi and its neighbors are aggregated by summing them up:116
m (k) N (i) =
∑
j∈{i}∪N (i) m
(k) j , (2)
where m(k)N (i) denotes the aggregated message.117
(3) Feature updating: Finally, the hidden representation is updated with the aggregated message. For118 the vanilla GCN, the update function can be considered as applying a non-linear operation to the119 aggregated message:120
hki =
{ softmax(m
(k) N (i)), if k = K
ReLU(m (k) N (i)), otherwise.
(3)
Here, K is the number of model layers. The last layer of the vanilla GCN should output the121 classification probability via a softmax function. Otherwise, a ReLU operation is used.122
3.2 Multi-Task Network Architecture123
In this paper, we propose an auxiliary learning induced GCN for semi-supervised node classification124 (see Fig. 3). To enhance the node classification performance of the backbone vanilla GCN, we design125 two auxiliary tasks: link prediction and pseudo-label generation.126
The first K−1 layers of a K-layer vanilla GCN model can be considered as a feature extractor, while127 the last layer can be considered as a classifier. Denoting the feature extractor and node classifier of128 the backbone model as hθ2 and fθ1 , respectively, where θ1 and θ2 are the learnable parameters, the129 proposed two auxiliary task networks are defined as follows.130
3.2.1 Link Predictor131
To enrich the edge information, we design an auxiliary link prediction task to infer the missing edges132 and present the probability of edge existence. The link predictor we propose contains a decoder R(·)133 and the feature extractor hθ2 of the backbone model.134
The feature extractor hθ2(·) takes reduced adjacency matrix As which corresponds to the sampled135 edge set Es ⊂ E , and node features X as inputs to generate the hidden embedding:136
Hs = hθ2(X,As). (4) Then, the decoder computes the similarity between each node based on Hs to predict the edge137 existence probabilities. We simply use an inner-product calculator with a sigmoid function as the138 implementation of decoder R(·). The similarity of two hidden embeddings can be computed as139
rij = R(hi,hj) = σ(hih T j ), (5)
where hi and hj denote the hidden embeddings of node vi and node vj in Hs, respectively.140
3.2.2 Label Generator141
Although using the one-hot label is inappropriate, it is difficult to obtain soft labels with manual142 annotations for real graph-structured data. To tackle this issue, we introduce an auxiliary label143 generation task to generate soft labels that reflect the tendency of different classes each node belongs144 to. Label generator gϕ(X,A) is a vanilla GCN, where ϕ are the learnable parameters and A is the145 adjacency matrix. The label generation network predicts the label distribution of each node based on146 the graph structure A and the raw features X of the nodes, as:147
Ŷg = gϕ(X,A), (6)
where Yg are the predicted pseudo labels for guiding the training of the backbone network and the148 label generator.149
3.2.3 Node classifier150
The node classifier carries out the primary semi-supervised node classification task in our model.151 Compared to the vanilla GCN model, we add a graph reconstruction step at the beginning. The graph152 adjacency matrix reconstructed with the hidden embeddings contains richer edge information since153 the proposed auxiliary link prediction task can enhance the graph topology capturing ability of the154 feature extractor hθ2(·). However, the hidden embeddings derived by the feature extractor change155 rapidly in the first few iterations, resulting in a changing reconstructed adjacency matrix. Directly156 applying the reconstructed adjacency matrix to the entire backbone network increases the training157 instability. Thus, we only apply it as the input of the feature extractor hθ2(·), while the classifier158 fθ1(·) still adopts the original adjacency matrix as input.159 In each training iteration, the feature extractor hθ2(·) first generates the hidden embeddings of the160 nodes H = hθ2(X,A). The decoder uses H to reconstruct the adjacency matrix Arecon = R(H,H).161 Then, the feature extractor takes the reconstructed adjacency matrix Arecon to compute the hidden162 embeddings Hrecon = hθ2(X,Arecon).163
Finally, the computed hidden embeddings Hrecon and the original graph adjacency matrix A are fed164 to the classifier to obtain the final classification results:165
Ŷf = fθ1(Hrecon,A). (7)
3.3 Auxiliary Training Phases166
In addition to the backbone network, the proposed method contains two auxiliary task networks. In167 the following, we will introduce the objectives for the node classifier fθ1 , link predictor hθ2 , and168 label generator gϕ in order, and then leverage a meta auxiliary learning scheme to train the proposed169 multi-task network.170

heading: 3.3.1 Training the Node Classifier fθ1171
text: The purpose of the node classifier fθ1(·) is to carry out the graph-based semi-supervised node172 classification task, which yields the final prediction results. Naturally, it is necessary to use the173 classification loss between the predicted result and the real categories of nodes to supervise the174 training process. At the same time, to reflect the tendency of the class a node belongs to, the training175 should make the prediction labels of the classifier be close to the pseudo soft labels generated by176 label generator gϕ(·).177 In the tth iteration, denoting the pseudo soft labels as Ŷg(t) = gϕ(t)(X,A), the real labels used in178 training as Ytrain, and the prediction results of the node classifier of the backbone network fθ1 as179 Ŷf(t) = f
θ (t) 1 (h θ (t) 2 (X,A
(t) recon),A), the objective for the node classifier fθ1(·) is defined as180
L(t)θ1 = LCE(Ŷ f(t) train,Ytrain) + LMSE(Ŷf(t), Ŷg(t)). (8)
The objective contains two parts: the loss on the real training labels, and the loss on the generated181 pseudo soft labels. LCE denotes the cross-entropy loss and LMSE denotes the mean squared loss.182 Although this objective can be used to update the learnable parameters θ2 of the feature extractor183 hθ2(·), as it generates the hidden embeddings used for node classification, we only employ it to184
supervise the learning of the node parameters θ1 of the classifier fθ1(·). To avoid adding unnecessary185 supervision for generating the reconstructed graph adjacency matrix Arecon, we add a stop-gradient186 operation (detach) which is formulated as follows:187
A(t)recon = detach(R(H,H)). (9)

heading: 3.3.2 Auxiliary Training for the Feature Extractor hθ2188
text: The feature extractor hθ2 is shared by the link prediction module and the backbone network. In each189 iteration, we first randomly sample a certain percentage of edges from the real edge set E to form a190 sampled edge set Es ⊂ E . Denoting the reduced graph adjacency matrix as As, which corresponds to191 the sampled edge set Es, and applying a message passing operation with As as192
H(t)s = hθ(t)2 (X,A(t)s ), (10)
then the objective for the feature extractor is defined as193
L(t)θ2 = LCE(R(H (t) s ,H (t) s ),A). (11)
Here R(H(t)s ,H (t) s ) represents the correlation between the hidden embeddings of each pair of nodes.194

heading: 3.3.3 Auxiliary Training for the Label Generator gϕ195
text: The label generator gϕ is a vanilla GCN model used to predict the label distribution for each node,196 which naturally needs to be supervised by a classification loss. In the tth iteration, a label generator197 first generates the prediction results using the original graph adjacency matrix A and the node features198 X, which is formulated as Ŷg(t) = gϕ(t)(X,A). Denoting the training labels as Ytrain, the objective199 of the label generator gϕ can be formulated as200
L(t)ϕ = LCE(Ŷg(t)train,Ytrain). (12)
3.3.4 Meta-Learning Based Training Strategy201
The final node classification results only depend on the primary task, while the performance of the202 other modules, including the label generator and link predictor are not our ultimate concern. Simply203 using the classification loss to train the label generator gϕ or using the link prediction loss to train204 the feature extractor hθ2 cannot provide the effects of auxiliary learning. Thus, it is crucial to make205 the node classifier fθ1 perform better after it is trained with the pseudo soft labels while taking the206 hidden embeddings derived by the feature extractor hθ2 as inputs. To this end, we use meta auxiliary207 learning to update the model parameters.208
For the auxiliary label generation task, we consider not only the classification performance of the209 label generator gϕ, but also the auxiliary effect on the node classifier. We assume that the classifier210 parameters θ1 are updated with gradient descent based on the pseudo soft labels in the tth iteration,211
θ′1 = θ (t) 1 − η∇θ1LMSE(Ŷf(t), Ŷg(t)). (13)
A direct way to evaluate the auxiliary effect of the pseudo soft labels is to compute the classification212 loss of the prediction given by the node classifier using the updated parameters θ′1, as213
Lf ′ = LCE(fθ′1(hθ(t)2 (X,Arecon),A)train,Ytrain). (14)
This loss can quantify how much performance improvement the classifier gains from the two auxiliary214 tasks. Because θ′1 is updated with the pseudo soft labels generated by gϕ(·), the loss Lf ′ is also a215 function of ϕ. This means that the objective could be used to supervise the learning of ϕ. Note216 that ∇ϕLf ′ requires the gradient of the gradient to be computed [25], which can be considered as a217 meta-learning strategy. The final objective of the label generator gϕ with meta-learning is formulated218 as219
L(t)ϕ−meta = L(t)ϕ + Lf ′. (15)
Similar to the label generator gϕ(·), the link predictor hbθ2 should derive the effective node embed-220 dings to enhance the node classification performance. Since the hidden embeddings used for node221
Algorithm 1 AL-GCN Input: Graph adjacency matrix A, the node features X, the data labels Ytrain of a training set. Output: A feature extractor hθ2 , a node classifier fθ1
1: Initialize learnable parameters θ1, θ2, ϕ 2: while not converged do 3: # node classifier training phase 4: Ŷg ← gϕ(X,A) 5: H← hθ2(X,A) 6: Arecon = detach(R(H,H)) 7: Hrecon ← hθ2(X,Arecon) 8: Ŷf ← fθ1(Hrecon,A) 9: Lθ1 ← LCE(Ŷftrain,Ytrain) + LMSE(Ŷf , Ŷg)
10: Update: θ1 ← Adam(Lθ1 , θ1) 11: # meta-learning preparation 12: Compute: θ′1 ← θ1 − η∇θ1LMSE(Ŷf , Ŷg) 13: Lf ′ ← LCE(f bθ′1(h b θ2 (X,Arecon),A)train,Ytrain) 14: # label generator training phase 15: Ŷg ← gϕ(X,A) 16: Lϕ ← LCE(Ŷgtrain,Ytrain) + Lf ′ 17: Update: ϕ← Adam(Lϕ, ϕ) 18: # feature extractor training phase 19: As ← RandomSample(A) 20: Hs ← hθ2(X,As) 21: Lθ2 ← LCE(R(Hs,Hs),A) + Lf ′ 22: Update: θ2 ← Adam(Lθ2 , θ2) 23: end while
classification are derived from the feature extractor hθ2 , the objective defined in Eq. 14 can also be222 considered as a function of θ2. Thus, the objective of the feature extractor hθ2 with meta-learning is223 defined as224
L(t)θ2−meta = L (t) θ2 + Lf ′. (16) This objective means that, except for the link prediction task, the feature extractor hθ2 should ensure225 that the generated hidden embedding enhance the classification accuracy of the node classifier fθ1226 updated with the pseudo soft labels. The overall training process is shown in Alg. 1.227
4 Experiments228
4.1 Experimental Settings and Compared Methods229
We demonstrate the classification performance of our method via semi-supervised document classifi-230 cation on three citation network datasets, including Cora, Citeseer, and Pubmed [26], where nodes231 represent the documents and edges are citation links. Dataset statistics are summarized in Table 1.232
The proposed method is a novel GCN, which is leveraged to carry out a node classification task.233 We compare our method with several popular graph-based node classification methods including234 GCN [12], GAT [3], DualGCN [13], SGC [27], and APPNP [4]. As we use a three-layer GCN as235 the backbone in our proposed model, we compare both two-layer and three-layer GCNs, denoted as236 GCN2 and GCN3, respectively.237
For the semi-supervised node classification, we use all node features but only 20 labels per class for238 training and 500 nodes as the validation set. We train the proposed method for a maximum of 400239 epochs using Adam [28] with a learning rate of 0.01. We use the well-trained parameters, which240 achieve the best performance on the validation set during the training phases, to evaluate classification241 accuracy on a test set of 1,000 labeled examples. We run each method 100 times and compute242 the average classification accuracy on a single NVIDIA GTX 1080Ti GPU. To implement all the243 compared methods more conveniently, we use PyG [29] as the graph-based learning framework.244
In this section, we provide the experimental results of the node classification, an ablation study, and246 the visualization of hidden embeddings. More experimental results, such as parameter and model247 robustness studies can be found in the supplementary materials.248
We conduct the node classification task on three citation network datasets. As shown in Table 2,249 our method consistently and significantly enhances the learning performance compared to the other250 methods. In particular, for the Cora dataset, the proposed method is superior to GCN by 4.9%.251 Compared to the other methods, our model considers more graph-structured information via auxiliary252 learning. Thus, it is consistently and significantly superior to the compared methods, achieving253 state-of-the-art results.254
4.3 Ablation Studies255
4.3.1 On the Auxiliary Learning Modules256
To determine how the link predictor (P) and the pseudo label generator (G) affect the node classi-257 fication performance, we apply the following two ablation models: (1) Vanilla GCN with the link258 predictor, termed GCN+P. (2) Vanilla GCN with the lable generator, termed GCN+G. We compare259 GCN+G and GCN+P with the proposed method, AL-GCN (GCN+P+G), and the original GCN.260 Table 3 lists the results. As can be seen, the proposed method consistently outperforms GCN+P261 and GCN+G. Specifically, compared to the link predictor, the label generator has more effect on the262 Citeseer dataset. However, for the Pubmed dataset, the link predictor has much more effect on the263 learning performance.264
4.3.2 On the Reconstructed Graph Adjacency Matrix265
For the backbone network, a reconstructed graph adjacency matrix via a link prediction task is used266 as input. To determine how the reconstructed graph adjacency matrix affects the node classification267 performance, we compare the proposed model with the following two models: (1) A model that268 takes the original graph adjacency matrix without the reconstructed one as the input of the backbone269 network, termed w/o-recG; (2) A model that takes only the reconstructed graph adjacency matrix as270 the input of the backbone network, termed w/o-oriG.271
As shown in Table 4, our method consistently outperforms w/o-recG and w/o-oriG. This can be272 attributed to the fact that the reconstructed graph adjacency matrix via the link predictor can capture273 the detailed topology information of a graph and the fixed original graph adjacency matrix increases274 the training stability.275
4.4 Visualization of Hidden Embeddings276
To determine how the hidden embeddings affect the learning performance, we use the visualization277 tool t-SNE [30] to observe their distribution. As shown in Fig. 4, the embedding results of GCN278 and GAT are denser, and the separation of different clusters is not obvious. In contrast, the node279 distributions learned by our proposed method are more separate, with most of the nodes from the280 same classes being close to each other, resulting in obvious cluster structures. These experimental281 results demonstrate that the proposed method can capture more detailed structure information of a282 graph, including the nodes and edges, resulting in more effective hidden embeddings.283
5 Conclusion284
We have proposed a novel graph convolutional network for semi-supervised node classification.285 Different from existing methods, the proposed model focuses on enriching the graph data and286 adopts meta auxiliary learning to enhance the representations of nodes and edges in a graph. To287 enrich node label information, an auxiliary label generator is used to generate pseudo probabilistic288 labels. Meanwhile, an auxiliary link predictor is used to generate probabilistic edges to enrich the289 graph structure information. The enriched node and edge information can iteratively enhance the290 performance of the node classification task. Experimental results on several benchmark citation291 datasets show that the proposed model is superior to the existing methods. For future work, we note292 that real-world data is usually contaminated by noise, which results in a robustness problem for graph293 learning methods. We plan to extend our model to handle noisy data by designing a more robust294 learning method for graph-structured data.295
References296
[1] David Duvenaud, Dougal Maclaurin, Jorge Aguilera-Iparraguirre, Rafael Gómez-Bombarelli,297 Timothy Hirzel, Alán Aspuru-Guzik, and Ryan P. Adams. Convolutional networks on graphs298 for learning molecular fingerprints. In Advances in Neural Information Processing Systems,299 pages 2224–2232, 2015.300
[2] Rex Ying, Ruining He, Kaifeng Chen, Pong Eksombatchai, William L. Hamilton, and Jure301 Leskovec. Graph convolutional neural networks for web-scale recommender systems. In302 Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery and303 Data Mining, pages 974–983, 2018.304
[3] Petar Velickovic, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and305 Yoshua Bengio. Graph attention networks. In 6th International Conference on Learning306 Representations, 2018.307
[4] Johannes Klicpera, Aleksandar Bojchevski, and Stephan Günnemann. Predict then propagate:308 Graph neural networks meet personalized pagerank. In 7th International Conference on309 Learning Representations, 2019.310
[5] Xiao Wang, Meiqi Zhu, Deyu Bo, Peng Cui, Chuan Shi, and Jian Pei. AM-GCN: adaptive multi-311 channel graph convolutional networks. In Proceedings of the 26t h ACM SIGKDD International312 Conference on Knowledge Discovery and Data Mining, pages 1243–1253, 2020.313
[6] Keyulu Xu, Chengtao Li, Yonglong Tian, Tomohiro Sonobe, Ken-ichi Kawarabayashi, and314 Stefanie Jegelka. Representation learning on graphs with jumping knowledge networks. In315 Proceedings of the 35th International Conference on Machine Learning, volume 80, pages316 5449–5458, 2018.317
[7] Lingxiao Zhao and Leman Akoglu. Pairnorm: Tackling oversmoothing in gnns. In 8th318 International Conference on Learning Representations, 2020.319
[8] Ming Chen, Zhewei Wei, Zengfeng Huang, Bolin Ding, and Yaliang Li. Simple and deep320 graph convolutional networks. In Proceedings of the 37th International Conference on Machine321 Learning, volume 119, pages 1725–1735, 2020.322
[9] Qimai Li, Zhichao Han, and Xiao-Ming Wu. Deeper insights into graph convolutional networks323 for semi-supervised learning. In Proceedings of the Thirty-Second AAAI Conference on Artificial324 Intelligence, pages 3538–3545, 2018.325
[10] Joan Bruna, Wojciech Zaremba, Arthur Szlam, and Yann LeCun. Spectral networks and locally326 connected networks on graphs. In 2nd International Conference on Learning Representations,327 2014.328
[11] Michaël Defferrard, Xavier Bresson, and Pierre Vandergheynst. Convolutional neural networks329 on graphs with fast localized spectral filtering. In Advances in Neural Information Processing330 Systems, pages 3837–3845, 2016.331
[12] Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional332 networks. In 5th International Conference on Learning Representations, 2017.333
[13] Chenyi Zhuang and Qiang Ma. Dual graph convolutional networks for graph-based semi-334 supervised classification. In Proceedings of the 2018 World Wide Web Conference on World335 Wide Web, pages 499–508, 2018.336
[14] Zonghan Wu, Shirui Pan, Fengwen Chen, Guodong Long, Chengqi Zhang, and Philip S. Yu. A337 comprehensive survey on graph neural networks. CoRR, abs/1901.00596, 2019.338
[15] Alessio Micheli. Neural network for graphs: A contextual constructive approach. IEEE Trans.339 Neural Networks, 20(3):498–511, 2009.340
[16] James Atwood and Don Towsley. Diffusion-convolutional neural networks. In Advances in341 Neural Information Processing Systems, pages 1993–2001, 2016.342
[17] Justin Gilmer, Samuel S. Schoenholz, Patrick F. Riley, Oriol Vinyals, and George E. Dahl.343 Neural message passing for quantum chemistry. In Proceedings of the 34th International344 Conference on Machine Learning, volume 70, pages 1263–1272, 2017.345
[18] Federico Monti, Davide Boscaini, Jonathan Masci, Emanuele Rodolà, Jan Svoboda, and346 Michael M. Bronstein. Geometric deep learning on graphs and manifolds using mixture347
model cnns. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, pages348 5425–5434, 2017.349
[19] Hongyang Gao, Zhengyang Wang, and Shuiwang Ji. Large-scale learnable graph convolutional350 networks. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge351 Discovery and Data Mining, pages 1416–1424, 2018.352
[20] John Flynn, Ivan Neulander, James Philbin, and Noah Snavely. Deepstereo: Learning to predict353 new views from the world’s imagery. CoRR, abs/1506.06825, 2015.354
[21] Shubham Toshniwal, Hao Tang, Liang Lu, and Karen Livescu. Multitask learning with low-level355 auxiliary tasks for encoder-decoder based speech recognition. In 18th Annual Conference of the356 International Speech Communication Association, pages 3532–3536, 2017.357
[22] Timothy M. Hospedales, Antreas Antoniou, Paul Micaelli, and Amos J. Storkey. Meta-learning358 in neural networks: A survey. CoRR, abs/2004.05439, 2020.359
[23] Dong-Hyun Lee. Pseudo-label: The simple and efficient semi-supervised learning method for360 deep neural networks. In Workshop on Challenges in Representation Learning, volume 3, 2013.361
[24] Hieu Pham, Qizhe Xie, Zihang Dai, and Quoc V. Le. Meta pseudo labels. CoRR, abs/2003.10580,362 2020.363
[25] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adap-364 tation of deep networks. In Proceedings of the 34th International Conference on Machine365 Learning, volume 70, pages 1126–1135, 2017.366
[26] Prithviraj Sen, Galileo Namata, Mustafa Bilgic, Lise Getoor, Brian Galligher, and Tina Eliassi-367 Rad. Collective classification in network data. AI Magazine, 29(3):93, September 2008.368
[27] Felix Wu, Amauri H. Souza Jr., Tianyi Zhang, Christopher Fifty, Tao Yu, and Kilian Q. Wein-369 berger. Simplifying graph convolutional networks. In Proceedings of the 36th International370 Conference on Machine Learning, volume 97, pages 6861–6871, 2019.371
[28] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In 3rd372 International Conference on Learning Representations, 2015.373
[29] Matthias Fey and Jan Eric Lenssen. Fast graph representation learning with pytorch geometric.374 CoRR, abs/1903.02428, 2019.375
[30] Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine376 learning research, 9(Nov):2579–2605, 2008.377
Checklist378
1. For all authors...379 (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s380 contributions and scope? [Yes]381 (b) Did you describe the limitations of your work? [Yes]382 (c) Did you discuss any potential negative societal impacts of your work? [No]383 (d) Have you read the ethics review guidelines and ensured that your paper conforms to384 them? [Yes]385 2. If you are including theoretical results...386
(a) Did you state the full set of assumptions of all theoretical results? [N/A]387 (b) Did you include complete proofs of all theoretical results? [N/A]388
3. If you ran experiments...389 (a) Did you include the code, data, and instructions needed to reproduce the main exper-390 imental results (either in the supplemental material or as a URL)? [Yes] The codes391 are included in the supplemental material and the datasets can be downlowded by the392 codes automaticly.393
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they394 were chosen)? [Yes] They are included in the supplemental material.395
(c) Did you report error bars (e.g., with respect to the random seed after running experi-396 ments multiple times)? [Yes] See Section 4.1.397
(d) Did you include the total amount of compute and the type of resources used (e.g., type398 of GPUs, internal cluster, or cloud provider)? [Yes] See Section 4.1.399
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...400 (a) If your work uses existing assets, did you cite the creators? [Yes] See Section 4.1.401 (b) Did you mention the license of the assets? [No]402 (c) Did you include any new assets either in the supplemental material or as a URL? [No]403 (d) Did you discuss whether and how consent was obtained from people whose data you’re404 using/curating? [No]405 (e) Did you discuss whether the data you are using/curating contains personally identifiable406 information or offensive content? [No]407 5. If you used crowdsourcing or conducted research with human subjects...408
(a) Did you include the full text of instructions given to participants and screenshots, if409 applicable? [N/A]410
(b) Did you describe any potential participant risks, with links to Institutional Review411 Board (IRB) approvals, if applicable? [N/A]412
(c) Did you include the estimated hourly wage paid to participants and the total amount413 spent on participant compensation? [N/A]414

Review summaries:
The paper proposes a new auxiliary learning method to improve the existing GCNs. Considering that the edges in many graphs are unweighted and nodes labels are often one-hot, the authors added two auxiliary tasks to enrich the topology information in the graph node classification task. One is link prediction which generates probabilistic edges; and the other is to generate soft labels. The probabilistic edges and soft labels are iteratively updated by a meta auxiliary learning strategy in a smart way, and the results on three standard benchmark datasets outperform some baselines. 

This paper proposes a novel meta-learning framework that utilizes auxiliary learning tasks of link predictions and label generations to improve the performance of GCNs on node classifications tasks. The meta-learning algorithm adopts the idea from [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://arxiv.org/abs/1703.03400) so that the models for auxiliary tasks are updated in a way that has higher impacts on the primary task.  The authors use two self-supervised learning tasks, edge predictions and label generation, as the auxiliary tasks.  The method performs favorably in the three standard public graph data sets with higher prediction accuracy.