Graph inference learning for semi-supervised classification

1 INTRODUCTION . Graph , which comprises a set of vertices/nodes together with connected edges , is a formal structural representation of non-regular data . Due to the strong representation ability , it accommodates many potential applications , e.g. , social network ( Orsini et al. , 2017 ) , world wide data ( Page et al. , 1999 ) , knowledge graph ( Xu et al. , 2017 ) , and protein-interaction network ( Borgwardt et al. , 2007 ) . Among these , semi-supervised node classification on graphs is one of the most interesting also popular topics . Given a graph in which some nodes are labeled , the aim of semi-supervised classification is to infer the categories of those remaining unlabeled nodes by using various priors of the graph . While there have been numerous previous works ( Brandes et al. , 2008 ; Zhou et al. , 2004 ; Zhu et al. , 2003 ; Yang et al. , 2016 ; Zhao et al. , 2019 ) devoted to semi-supervised node classification based on explicit graph Laplacian regularizations , it is hard to efficiently boost the performance of label prediction due to the strict assumption that connected nodes are likely to share the same label information . With the progress of deep learning on grid-shaped images/videos ( He et al. , 2016 ) , a few of graph convolutional neural networks ( CNN ) based methods , including spectral ( Kipf & Welling , 2017 ) and spatial methods ( Niepert et al. , 2016 ; Pan et al. , 2018 ; Yu et al. , 2018 ) , have been proposed to learn local convolution filters on graphs in order to extract more discriminative node representations . Although graph CNN based methods have achieved considerable capabilities of graph embedding by optimizing filters , they are limited into a conventionally semi-supervised framework and lack of an efficient inference mechanism on graphs . Especially , in the case of few-shot learning , where a small number of training nodes are labeled , this kind of methods would drastically compromise the performance . For example , the Pubmed graph dataset ( Sen et al. , 2008 ) consists ∗Corresponding author : Zhen Cui . ( b ) The process of Graph inference learning . We extract the local representation from the local subgraph ( the circle with dashed line The red wave line denote the node reachability from to . d t th h bilit f d t th d of 19,717 nodes and 44,338 edges , but only 0.3 % nodes are labeled for the semi-supervised node classification task . These aforementioned works usually boil down to a general classification task , where the model is learnt on a training set and selected by checking a validation set . However , they do not put great efforts on how to learn to infer from one node to another node on a topological graph , especially in the few-shot regime . In this paper , we propose a graph inference learning ( GIL ) framework to teach the model itself to adaptively infer from reference labeled nodes to those query unlabeled nodes , and finally boost the performance of semi-supervised node classification in the case of a few number of labeled samples . Given an input graph , GIL attempts to infer the unlabeled nodes from those observed nodes by building between-node relations . The between-node relations are structured as the integration of node attributes , connection paths , and graph topological structures . It means that the similarity between two nodes is decided from three aspects : the consistency of node attributes , the consistency of local topological structures , and the between-node path reachability , as shown in Fig . 1 . The local structures anchored around each node as well as the attributes of nodes therein are jointly encoded with graph convolution ( Defferrard et al. , 2016 ) for the sake of high-level feature extraction . For the between-node path reachability , we adopt the random walk algorithm to obtain the characteristics from a labeled reference node vi to a query unlabeled node vj in a given graph . Based on the computed node representation and between-node reachability , the structure relations can be obtained by computing the similar scores/relationships from reference nodes to unlabeled nodes in a graph . Inspired by the recent meta-learning strategy ( Finn et al. , 2017 ) , we learn to infer the structure relations from a training set to a validation set , which can benefit the generalization capability of the learned model . In other words , our proposed GIL attempts to learn some transferable knowledge underlying in the structure relations from training samples to validation samples , such that the learned structure relations can be better self-adapted to the new testing stage . We summarize the main contributions of this work as three folds : • We propose a novel graph inference learning framework by building structure relations to infer unknown node labels from those labeled nodes in an end-to-end way . The structure relations are well defined by jointly considering node attributes , between-node paths , and graph topological structures . • To make the inference model better generalize to test nodes , we introduce a meta-learning procedure to optimize structure relations , which could be the first time for graph node classification to the best of our knowledge . • Comprehensive evaluations on three citation network datasets ( including Cora , Citeseer , and Pubmed ) and one knowledge graph data ( i.e. , NELL ) demonstrate the superiority of our proposed GIL in contrast with other state-of-the-art methods on the semi-supervised classification task . 2 RELATED WORK . Graph CNNs : With the rapid development of deep learning methods , various graph convolution neural networks ( Kashima et al. , 2003 ; Morris et al. , 2017 ; Shervashidze et al. , 2009 ; Yanardag & Vishwanathan , 2015 ; Jiang et al. , 2019 ; Zhang et al. , 2020 ) have been exploited to analyze the irregular graph-structured data . For better extending general convolutional neural networks to graph domains , two broad strategies have been proposed , including spectral and spatial convolution methods . Specifically , spectral filtering methods ( Henaff et al. , 2015 ; Kipf & Welling , 2017 ) develop convolution-like operators in the spectral domain , and then perform a series of spectral filters by decomposing the graph Laplacian . Unfortunately , the spectral-based approaches often lead to a high computational complex due to the operation of eigenvalue decomposition , especially for a large number of graph nodes . To alleviate this computation burden , local spectral filtering methods ( Defferrard et al. , 2016 ) are then proposed by parameterizing the frequency responses as a Chebyshev polynomial approximation . Another type of graph CNNs , namely spatial methods ( Li et al. , 2016 ; Niepert et al. , 2016 ) , can perform the filtering operation by defining the spatial structures of adjacent vertices . Various approaches can be employed to aggregate or sort neighboring vertices , such as diffusion CNNs ( Atwood & Towsley , 2016 ) , GraphSAGE ( Hamilton et al. , 2017 ) , PSCN ( Niepert et al. , 2016 ) , and NgramCNN ( Luo et al. , 2017 ) . From the perspective of data distribution , recently , the Gaussian induced convolution model ( Jiang et al. , 2019 ) is proposed to disentangle the aggregation process through encoding adjacent regions with Gaussian mixture model . Semi-supervised node classification : Among various graph-related applications , semi-supervised node classification has gained increasing attention recently , and various approaches have been proposed to deal with this problem , including explicit graph Laplacian regularization and graphembedding approaches . Several classic algorithms with graph Laplacian regularization contain the label propagation method using Gaussian random fields ( Zhu et al. , 2003 ) , the regularization framework by relying on the local/global consistency ( Zhou et al. , 2004 ) , and the random walkbased sampling algorithm for acquiring the context information ( Yang et al. , 2016 ) . To further address scalable semi-supervised learning issues ( Liu et al. , 2012 ) , the Anchor Graph regularization approach ( Liu et al. , 2010 ) is proposed to scale linearly with the number of graph nodes and then applied to massive-scale graph datasets . Several graph convolution network methods ( Abu-El-Haija et al. , 2018 ; Du et al. , 2017 ; Thekumparampil et al. , 2018 ; Velickovic et al. , 2018 ; Zhuang & Ma , 2018 ) are then developed to obtain discriminative representations of input graphs . For example , Kipf et al . ( Kipf & Welling , 2017 ) proposed a scalable graph CNN model , which can scale linearly in the number of graph edges and learn graph representations by encoding both local graph structures and node attributes . Graph attention networks ( GAT ) ( Velickovic et al. , 2018 ) are proposed to compute hidden representations of each node for attending to its neighbors with a self-attention strategy . By jointly considering the local- and global-consistency information , dual graph convolutional networks ( Zhuang & Ma , 2018 ) are presented to deal with semi-supervised node classification . The critical difference between our proposed GIL and those previous semi-supervised node classification methods is to adopt a graph inference strategy by defining structure relations on graphs and then leverage a meta optimization mechanism to learn an inference model , which could be the first time to the best of our knowledge , while the existing graph CNNs take semi-supervised node classification as a general classification task . 3 THE PROPOSED MODEL . 3.1 PROBLEM DEFINITION . Formally , we denote an undirected/directed graph as G = { V , E , X , Y } , where V = { vi } ni=1 is the finite set of n ( or |V| ) vertices , E ∈ Rn×n defines the adjacency relationships ( i.e. , edges ) between vertices representing the topology of G , X ∈ Rn×d records the explicit/implicit attributes/signals of vertices , and Y ∈ Rn is the vertex labels of C classes . The edge Eij = E ( vi , vj ) = 0 if and only if vertices vi , vj are not connected , otherwise Eij 6= 0 . The attribute matrix X is attached to the vertex set V , whose i-th row Xvi ( or Xi· ) represents the attribute of the i-th vertex vi . It means that vi ∈ V carries a vector of d-dimensional signals . Associated with each node vi ∈ V , there is a discrete label yi ∈ { 1 , 2 , · · · , C } . We consider the task of semi-supervised node classification over graph data , where only a small number of vertices are labeled for the model learning , i.e. , |VLabel| |V| . Generally , we have three node sets : a training set Vtr , a validation set Vval , and a testing set Vte . In the standard protocol of prior literatures ( Yang et al. , 2016 ) , the three node sets share the same label space . We follow but do not restrict this protocol for our proposed method . Given the training and validation node sets , the aim is to predict the node labels of testing nodes by using node attributes as well as edge connections . A sophisticated machine learning technique used in most existing methods ( Kipf & Welling , 2017 ; Zhou et al. , 2004 ) is to choose the optimal classifier ( trained on a training set ) after checking the performance on the validation set . However , these methods essentially ignore how to extract transferable knowledge from these known labeled nodes to unlabeled nodes , as the graph structure itself implies node connectivity/reachability . Moreover , due to the scarcity of labeled samples , the performance of such a classifier is usually not satisfying . To address these issues , we introduce a meta-learning mechanism ( Finn et al. , 2017 ; Ravi & Larochelle , 2017 ; Sung et al. , 2017 ) to learn to infer node labels on graphs . Specifically , the graph structure , between-node path reachability , and node attributes are jointly modeled into the learning process . Our aim is to learn to infer from labeled nodes to unlabeled nodes , so that the learner can perform better on a validation set and thus classify a testing set more accurately .

Summary: This paper proposes to leverage the between-node-path information into the inference of conventional graph neural network methods. Specifically, the proposed method treats the nodes in training set as a reference corpus and, when infering the label of a specific node, makes this node "attend" to the reference corpus, where the "attention" weights are calculated based on the node representations and the between-node paths. (The paper used different terms about the "attention".)