Auxiliary learning induced graph convolutional networks

Graph convolutional networks ( GCNs ) have recently achieved great success in1 many applications . However they suffer from an incomplete annotation problem2 for complex graph-structured data . In this paper , we introduce a novel auxiliary3 learning method for GCNs in a multi-task fashion , which can efficiently enrich the4 data annotations . Specifically , both link prediction and label generation are used as5 two auxiliary tasks to complement the primary task of node classification . These6 two auxiliary tasks are jointly trained with the primary node classification task7 via a graph meta-learning strategy . The experimental results demonstrate that the8 proposed method consistently and significantly outperforms existing methods and9 achieves state-of-the-art results on several benchmark citation network datasets.10 1 Introduction11 Graph-structured data is ubiquitous in real-world applications . However , general deep learning12 methods , such as convolutional neural networks ( CNNs ) , can not adapt to graph-structured data13 directly , because the nodes in a graph have different numbers of neighbors , which often lose the14 ranking information . To handle the graph data effectively , graph convolutional networks ( GCNs ) 15 have recently been proposed and used in many applications such as biomolecular prediction [ 1 ] and16 recommendation systems [ 2 ] .17 Previous methods focus on designing models that can extract information from both the graph18 topology and node features . Specifically , existing GCN methods typically design different propagation19 strategies [ 3 , 4 , 5 ] for each network layer and stack more network layers [ 6 , 7 , 8 ] to derive larger20 receptive fields . However , the neighborhood aggregation is essentially a type of Laplacian smoothing21 and stacking too many layers may result in over-smoothing [ 9 ] . These drawbacks of existing methods22 limit further performance enhancement.23 In this paper , we try to explore the bottleneck of node classification from another point of view , i.e.,24 from the training data itself . As shown in Fig . 1 , graph-structured data has different properties from25 grid-like data such as an image . The most obvious difference is that the nodes in a graph are connected26 by edges . This causes two main issues whien it comes to annotating graph-structured data , resulting in27 that existing methods can not fully leverage the graph-structured information . First , the edges in most28 graph data for semi-supervised node classification are unweighted . This arbitrary edge indication29 setting can not effectively reflect the detailed graph structures . Besides , graph-structured data may be30 contaminated with noisy edges . These noisy edges can not represent the true pairwise relationships31 between nodes . Second , using one-hot labels to train a graph-based model is inappropriate . One-hot32 labels are widely used in various machine learning tasks , assigning a training sample to a single class.33 However , nodes in a graph are connected ; even nodes with different classes may have relations . In34 this scenario , it is more suitable to use soft labels to assign a node to multiple classes , with different35 probabilities indicating which class the node possibly belongs to.36 Submitted to 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Do not distribute . Image Graph 2 2 2 1 1 3 3 1 1 1 1 1 1 1 1 1 1 ( 1 ) We propose two auxiliary tasks to capture more accurate graph information and enhance the52 model performance . The auxiliary link prediction task ensures that the model captures more graph53 topology information and generates probabilistic edges . The auxiliary label generation task softens54 the one-hot labels and generates pseudo-labels for unlabeled nodes.55 ( 2 ) The reconstructed edges and pseudo sudo labels derived via the two auxiliary tasks are iteratively56 updated with a node classifier of the primary task based on a meta auxiliary learning strategy , resulting57 in state-of-the-art node classification performance.58 2 Related Work59 Over the past few years , GCNs have achieved significant breakthroughs in graph data representation.60 Generally , existing GCNs can be divided into spectral-based methods and spatial-based methods.61 The spectral-based methods use graph spectral theory to define the graph convolutional operation62 in a graph Fourier domain . Spectral CNN [ 10 ] follows these mathematical foundations , assuming63 that a convolutional filter is a set of learnable parameters . To reduce computational complexity,64 ChebNet [ 11 ] approximates a graph convolutional filter as Chebyshev polynomials of the eigenvalues.65 GCN [ 12 ] introduces a first-order approximation of ChebNet and proposes a renormalization trick to66 alleviate numerical instabilities and exploding/vanishing gradients . DualGCN [ 13 ] introduces a dual67 GCN architecture with two graph convolutional layers in parallel to encode both local and global68 structural information.69 The spatial-based methods define feature aggregation in the spatial domain directly , which is more70 efficient , general , and flexible [ 14 ] . The key challenge for these spatial-based methods is to apply71 the convolution operation for different-sized neighborhoods , while at the same time maintaining the72 weight sharing property . Neural network for graphs ( NN4G ) [ 15 ] is the first spatial-based method,73 applying the graph convolutional operation in the spatial space . Diffusion convolutional neural74 networks ( DCNN ) [ 16 ] consider graph convolutions as diffusion processes to efficiently learn features75 that are invariant under isomorphism . Message passing neural networks ( MPNN ) [ 17 ] model graph76 convolution as a message passing process among the nodes . The graph attention network ( GAT ) 77 [ 3 ] introduce masked self-attentional layers to assign different weights to adjacent nodes , leading to78 learnable filter weights . The mixture model network ( MoNet ) [ 18 ] introduces pseudo-coordinates to79 assign different weights to the neighbors of each node . To achieve weight sharing across different80 nodes , some spatial-based models attempt to rank a node ’ s neighbors via certain criteria or metrics,81 which transforms the graph-structured data into grid data for further processing . The large-scale82 graph convolutional network ( LGCN ) [ 19 ] ranks a node ’ s neighbors via the node feature values . Then,83 multiple 1D convolutional layers are stacked for feature aggregation . Approximate personalized84 propagation of neural predictions ( APPNP ) [ 4 ] takes the personalized PageRank algorithm as the85 model propagation method to avoid over-smoothing when stacking more layers or increasing the size86 of the neighborhood.87 Multi-task learning is designed to simultaneously learn a set of related but different tasks for ensuring88 that a learning model can derive the best performance across all tasks . Different from multi-task89 learning , auxiliary learning is only concerned with model performance on the primary task . For90 < latexit sha1_base64= '' xEfXpeR/Fa79GZHZXEkfLplChxE= '' > AAACV3icbVFNSwMxEM2uX7V+VT16CRahPVh2paIXoeDFYwVrhW4t2TTbxibZNckKJeRPipf+FS+a1lX8Ggi8eTPzMnmJM0aVDoKZ5y8tr6yuldbLG5tb2zuV3b1bleYSkw5OWSrvYqQIo4J0NNWM3GWSIB4z0o0nl/N694lIRVNxo6cZ6XM0EjShGGlHDSoi4kiP48RwOzAP9t7UJnULL2CUSIRNaE2kHqU2ZtEmuRmSka3R+o/0oW6t/dQZf+kch/UvtlsoDyrVoBEsAv4FYQGqoIj2oPIcDVOccyI0ZkipXhhkum+Q1BQzYstRrkiG8ASNSM9BgThRfbPwxcIjxwxhkkp3hIYL9vuEQVypKY9d53xP9bs2J/+r9XKdnPcNFVmuicAfFyU5gzqFc5PhkEqCNZs6gLCkbleIx8g5qt1XlJ0J4e8n/wW3J42w2Ti9blZbrcKOEjgAh6AGQnAGWuAKtEEHYPACXr0lb9mbeW/+ql/6aPW9YmYf/Ah/9x0TebbR < /latexit > < latexit sha1_base64= '' +Tu+mvVFlWMCdmfnb9sBdyu+r8M= '' > AAACRnicbVBNaxsxEJ11vp0vtz3mImIC9sXshoT2UjD00lNIIE4C3s2ilbW2bEm7SNqAEfvreum5t/yEXHpoCb1WtjeQ2BkQPN6beTN6Sc6ZNr7/6NXW1jc2t7Z36rt7+weHjQ8fb3RWKEJ7JOOZukuwppxJ2jPMcHqXK4pFwultMvk2028fqNIsk9dmmtNI4KFkKSPYOCpuRKHAZpSkdlTGlpX3tjVpl+grmtNKWExM2Qp1IWI7Dpl0LSgkRb7QCeb2omyxNirRi49wPuPKpx03mn7HnxdaBUEFmlDVZdz4FQ4yUggqDeFY637g5yayWBlGOC3rYaFpjskED2nfQYkF1ZGdx1CiE8cMUJop96RBc/b1hMVC66lIXOfsWL2szcj3tH5h0i+RZTIvDJVksSgtODIZmmWKBkxRYvjUAUwUc7ciMsLKZeeSr7sQguUvr4Kb005w1jm/Omt2u1Uc23AEx9CCAD5DF77DJfSAwA94gj/w1/vp/faevX+L1ppXzXyCN1WD/wA8sp0= < /latexit > < latexit sha1_base64= '' Cru5BVsOASECHoYReBIwFRt/mog= '' > AAACA3icbVDLSsNAFL2pr1pfUXe6GSxCBSmJVHThoiCCywr2AW0ok+mkHTp5MDMRSgi48VfcuFDErT/hzr9x0gbR6oGBM+fcy733uBFnUlnWp1FYWFxaXimultbWNza3zO2dlgxjQWiThDwUHRdLyllAm4opTjuRoNh3OW2748vMb99RIVkY3KpJRB0fDwPmMYKVlvrmXqXnYzUimCet9Bh9f67So75ZtqrWFOgvsXNShhyNvvnRG4Qk9mmgCMdSdm0rUk6ChWKE07TUiyWNMBnjIe1qGmCfSieZ3pCiQ60MkBcK/QKFpurPjgT7Uk58V1dmO8p5LxP/87qx8s6dhAVRrGhAZoO8mCMVoiwQNGCCEsUnmmAimN4VkREWmCgdW0mHYM+f/Je0Tqp2rXp6UyvXL/I4irAPB1ABG86gDtfQgCYQuIdHeIYX48F4Ml6Nt1lpwch7duEXjPcvQ5yXRQ== < /latexit > < latexit sha1_base64= '' pY/ClCGqBGnEDcJm6l7QNvlJoPM= '' > AAAB8XicbVC7SgNBFL0bXzG+opY2g0GwCrsS0cIiYGMZwTwwWcLsZDYZMju7zNwVwpK/sLFQxNa/sfNvnCRbaOKBgcM59zLnniCRwqDrfjuFtfWNza3idmlnd2//oHx41DJxqhlvsljGuhNQw6VQvIkCJe8kmtMokLwdjG9nfvuJayNi9YCThPsRHSoRCkbRSo+9iOIoCLPOtF+uuFV3DrJKvJxUIEejX/7qDWKWRlwhk9SYrucm6GdUo2CST0u91PCEsjEd8q6likbc+Nk88ZScWWVAwljbp5DM1d8bGY2MmUSBnZwlNMveTPzP66YYXvuZUEmKXLHFR2EqCcZkdj4ZCM0ZyokllGlhsxI2opoytCWVbAne8smrpHVR9WrVy/tapX6T11GEEziFc/DgCupwBw1oAgMFz/AKb45xXpx352MxWnDynWP4A+fzB81jkP4= < /latexit > < latexit sha1_base64= '' Cru5BVsOASECHoYReBIwFRt/mog= '' > AAACA3icbVDLSsNAFL2pr1pfUXe6GSxCBSmJVHThoiCCywr2AW0ok+mkHTp5MDMRSgi48VfcuFDErT/hzr9x0gbR6oGBM+fcy733uBFnUlnWp1FYWFxaXimultbWNza3zO2dlgxjQWiThDwUHRdLyllAm4opTjuRoNh3OW2748vMb99RIVkY3KpJRB0fDwPmMYKVlvrmXqXnYzUimCet9Bh9f67So75ZtqrWFOgvsXNShhyNvvnRG4Qk9mmgCMdSdm0rUk6ChWKE07TUiyWNMBnjIe1qGmCfSieZ3pCiQ60MkBcK/QKFpurPjgT7Uk58V1dmO8p5LxP/87qx8s6dhAVRrGhAZoO8mCMVoiwQNGCCEsUnmmAimN4VkREWmCgdW0mHYM+f/Je0Tqp2rXp6UyvXL/I4irAPB1ABG86gDtfQgCYQuIdHeIYX48F4Ml6Nt1lpwch7duEXjPcvQ5yXRQ== < /latexit > < latexit sha1_base64= '' jpDvRQYjmmhsYjhkn3eweWYZXkE= '' > AAAB+XicbVDLSsNAFL2pr1pfUZduBotQNyWRii5cFNx0WcE+oI1lMp20QyeTMDMplNA/ceNCEbf+iTv/xkmbhbYeGDiccy/3zPFjzpR2nG+rsLG5tb1T3C3t7R8cHtnHJ20VJZLQFol4JLs+VpQzQVuaaU67saQ49Dnt+JP7zO9MqVQsEo96FlMvxCPBAkawNtLAtvsh1mM/SBvzp7TiXs4HdtmpOgugdeLmpAw5mgP7qz+MSBJSoQnHSvVcJ9ZeiqVmhNN5qZ8oGmMywSPaM1TgkCovXSSfowujDFEQSfOERgv190aKQ6VmoW8ms5xq1cvE/7xeooNbL2UiTjQVZHkoSDjSEcpqQEMmKdF8ZggmkpmsiIyxxESbskqmBHf1y+ukfVV1a9Xrh1q5fpfXUYQzOIcKuHADdWhAE1pAYArP8ApvVmq9WO/Wx3K0YOU7p/AH1ucP836TMw== < /latexit > instance , Deepstereo [ 20 ] leverages auxiliary learning to predict the relative poses of multiple cameras91 for unsupervised monocular depth estimation . To improve the performance of conversational speech92 recognition , auxiliary learning [ 21 ] is applied to low-level representations . Compared to the common93 learning scheme , meta auxiliary learning can enhance learning performance . For instance , MAXL94 [ 22 ] adopts meta-learning to automatically generate the auxiliary task labels . Pseudo Label [ 23 ] is95 a semi-supervised learning method , where a deep neural network is trained using both labeled and96 unlabeled data . For unlabeled data , the model picks up the class that has the maximum predicted97 probability as the true label to train itself . MPL [ 24 ] extends the Pseudo Label [ 23 ] via a meta-learning98 strategy , where the pseudo-labels are not generated by itself , but by a teacher network.99 The existing GCNs are designed for a single task , where the properties of graph-structured data100 are not fully explored . We introduce the auxiliary learning scheme to leverage more detailed graph101 topology information for enhancing node classification performance.102 3 Method103 3.1 Preliminaries104 Given a graph G = { V , E , X } , V is a set of nodes and E is a set of the edges connecting the related105 nodes . X ∈ RN×d represents the features matrix of the nodes , where d is the dimension of the node106 features and N = |V| is the number of nodes.107 The proposed method adopts the vanilla GCN [ 12 ] as the backbone network , taking the graph108 adjacency matrix A , labeled training set Ytrain , and original features X of the nodes as inputs to109 perform the semi-supervised node classification task . Based on the MPNN framework [ 17 ] , each110 layer of the vanilla GCN is defined in three parts:111 ( 1 ) Message computation : The message of node vi and its neighbor node vj is calculated , where112 j ∈ N ( i ) , as:113 m ( k ) j = 1√ deg ( i ) deg ( j ) h ( k−1 ) j W ( k ) , j ∈ { i } ⋃ N ( i ) . ( 1 ) Here , deg ( i ) is the degree of node vi and W ( k ) ∈ Rck−1×ck are the learnable parameters of the114 kth-layer , where ck is the size of the hidden embedding.115 ( 2 ) Aggregation : The messages of node vi and its neighbors are aggregated by summing them up:116 m ( k ) N ( i ) = ∑ j∈ { i } ∪N ( i ) m ( k ) j , ( 2 ) where m ( k ) N ( i ) denotes the aggregated message.117 ( 3 ) Feature updating : Finally , the hidden representation is updated with the aggregated message . For118 the vanilla GCN , the update function can be considered as applying a non-linear operation to the119 aggregated message:120 hki = { softmax ( m ( k ) N ( i ) ) , if k = K ReLU ( m ( k ) N ( i ) ) , otherwise . ( 3 ) Here , K is the number of model layers . The last layer of the vanilla GCN should output the121 classification probability via a softmax function . Otherwise , a ReLU operation is used.122 3.2 Multi-Task Network Architecture123 In this paper , we propose an auxiliary learning induced GCN for semi-supervised node classification124 ( see Fig . 3 ) . To enhance the node classification performance of the backbone vanilla GCN , we design125 two auxiliary tasks : link prediction and pseudo-label generation.126 The first K−1 layers of a K-layer vanilla GCN model can be considered as a feature extractor , while127 the last layer can be considered as a classifier . Denoting the feature extractor and node classifier of128 the backbone model as hθ2 and fθ1 , respectively , where θ1 and θ2 are the learnable parameters , the129 proposed two auxiliary task networks are defined as follows.130 3.2.1 Link Predictor131 To enrich the edge information , we design an auxiliary link prediction task to infer the missing edges132 and present the probability of edge existence . The link predictor we propose contains a decoder R ( · ) 133 and the feature extractor hθ2 of the backbone model.134 The feature extractor hθ2 ( · ) takes reduced adjacency matrix As which corresponds to the sampled135 edge set Es ⊂ E , and node features X as inputs to generate the hidden embedding:136 Hs = hθ2 ( X , As ) . ( 4 ) Then , the decoder computes the similarity between each node based on Hs to predict the edge137 existence probabilities . We simply use an inner-product calculator with a sigmoid function as the138 implementation of decoder R ( · ) . The similarity of two hidden embeddings can be computed as139 rij = R ( hi , hj ) = σ ( hih T j ) , ( 5 ) where hi and hj denote the hidden embeddings of node vi and node vj in Hs , respectively.140 3.2.2 Label Generator141 Although using the one-hot label is inappropriate , it is difficult to obtain soft labels with manual142 annotations for real graph-structured data . To tackle this issue , we introduce an auxiliary label143 generation task to generate soft labels that reflect the tendency of different classes each node belongs144 to . Label generator gϕ ( X , A ) is a vanilla GCN , where ϕ are the learnable parameters and A is the145 adjacency matrix . The label generation network predicts the label distribution of each node based on146 the graph structure A and the raw features X of the nodes , as:147 Ŷg = gϕ ( X , A ) , ( 6 ) where Yg are the predicted pseudo labels for guiding the training of the backbone network and the148 label generator.149 3.2.3 Node classifier150 The node classifier carries out the primary semi-supervised node classification task in our model.151 Compared to the vanilla GCN model , we add a graph reconstruction step at the beginning . The graph152 adjacency matrix reconstructed with the hidden embeddings contains richer edge information since153 the proposed auxiliary link prediction task can enhance the graph topology capturing ability of the154 feature extractor hθ2 ( · ) . However , the hidden embeddings derived by the feature extractor change155 rapidly in the first few iterations , resulting in a changing reconstructed adjacency matrix . Directly156 applying the reconstructed adjacency matrix to the entire backbone network increases the training157 instability . Thus , we only apply it as the input of the feature extractor hθ2 ( · ) , while the classifier158 fθ1 ( · ) still adopts the original adjacency matrix as input.159 In each training iteration , the feature extractor hθ2 ( · ) first generates the hidden embeddings of the160 nodes H = hθ2 ( X , A ) . The decoder uses H to reconstruct the adjacency matrix Arecon = R ( H , H ) .161 Then , the feature extractor takes the reconstructed adjacency matrix Arecon to compute the hidden162 embeddings Hrecon = hθ2 ( X , Arecon ) .163 Finally , the computed hidden embeddings Hrecon and the original graph adjacency matrix A are fed164 to the classifier to obtain the final classification results:165 Ŷf = fθ1 ( Hrecon , A ) . ( 7 ) 3.3 Auxiliary Training Phases166 In addition to the backbone network , the proposed method contains two auxiliary task networks . In167 the following , we will introduce the objectives for the node classifier fθ1 , link predictor hθ2 , and168 label generator gϕ in order , and then leverage a meta auxiliary learning scheme to train the proposed169 multi-task network.170

Summary: This paper proposes a novel meta-learning framework that utilizes auxiliary learning tasks of link predictions and label generations to improve the performance of GCNs on node classifications tasks. The meta-learning algorithm adopts the idea from [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://arxiv.org/abs/1703.03400) so that the models for auxiliary tasks are updated in a way that has higher impacts on the primary task.  The authors use two self-supervised learning tasks, edge predictions and label generation, as the auxiliary tasks.  The method performs favorably in the three standard public graph data sets with higher prediction accuracy.