Iterative Deep Graph Learning for Graph Neural Networks

1 INTRODUCTION . Recent years have seen a significantly growing amount of interest in graph neural networks ( GNNs ) , especially on efforts devoted to developing more effective GNNs for node classification ( Li et al. , 2016 ; Kipf & Welling , 2016 ; Hamilton et al. , 2017a ) , graph classification ( Ying et al. , 2018b ; Ma et al. , 2019 ) , and graph generation ( Samanta et al. , 2018 ; Li et al. , 2018b ; You et al. , 2018 ) . Encouraged by their huge success , GNNs have widely been used in a variety of domain specific applications such as machine reading comprehension ( Chen et al. , 2019a ) , semantic parsing ( Xu et al. , 2018b ) , natural language generation ( Chen et al. , 2019b ) , and healthcare informatics ( Gao et al. , 2019 ) . Despite GNNs ’ powerful ability of learning expressive node embeddings , unfortunately , GNNs can only be used when graph-structured data is available . Many real-world applications naturally admit network-structured data like social networks or graph-structured data like chemical compounds . However , it is questionable if these intrinsic graph-structures are optimal for the supervised downstream tasks . This is partially because the raw graphs were constructed from the original feature space , which may not reflect the “ true ” graph topology after feature extraction and transformation . Another potential reason is that real-world graphs are often noisy due to the inevitably error-prone data measurement or collection . More importantly , many applications such as those in natural language processing may only have non-graph structured data or even just the original feature matrix , requiring additional graph construction from the original data matrix to formulate graph data . Independently , there has been an increasing amount of work studying the dynamic model of interacting systems utilizing implicit interaction models ( Sukhbaatar et al. , 2016 ; Hoshen , 2017 ; Van Steenkiste et al. , 2018 ) . There models can be viewed as message passage based graph nerual networks ( Gilmer et al. , 2017 ) that pass messages over the fully connected graph through the message passing function ( Sukhbaatar et al. , 2016 ) or the use of an attention mechanism ( Hoshen , 2017 ; Van Steenkiste et al. , 2018 ) . This has been further extended by Kipf et al . ( 2018 ) , where they addressed the problem by inferring an explicit interaction structure using a variational graph autoencoder . However , these methods can not be directly applicable to joint learning the graph structure and graph representations when the graph is noisy or even not available . More recently , Franceschi et al . ( 2019 ) presented a new approach for jointly learning the graph and the parameters of GNNs , where they learnt a discrete probability distribution on the edges of the graph by approximately solving a bilevel program . Their experimental results have shown promising performance in both cases where the input graph is either corrupted or not available . However , this approach has severe scalability issue since it needs to learn N2 number of ( Bernoulli ) random variables to model joint probability distribution on the edges of the graph consisting of N number of vertices . More importantly , it can only be used for transductive setting , which means this method can not consider new nodes during the testing . To address these limitations , in this paper , we propose an Iterative Deep Graph Learning ( IDGL ) framework for jointly learning the graph structure and the GNN parameters that are optimized towards the prediction task at hand . Our IDGL framework consisting of five parts : 1 ) a graph learning neural network to generate a graph topology ; 2 ) a graph regularization neural network for controlling the smoothness , connectivity and sparsity of the generated graph ; 3 ) a graph embedding neural network for generating node embeddings ; 4 ) an iterative method to dynamically stop learning when the optimal graph is found ; and 5 ) a prediction neural network for performing a downstream prediction task . In particular , we present a graph learning neural network that casts a graph learning problem as a data-driven similarity metric learning task for constructing a graph . We then adapt techniques for learning graphs from smooth signals ( Kalofolias , 2016 ) to serve as graph regularization . More importantly , we propose a novel iterative method to search for hidden graph structure that augments the initial graph structure toward an optimal graph for the supervised prediction tasks . We highlight three contributions of our approach as follows : • We propose an end-to-end graph learning framework for jointly learning graph structure and graph embedding simultaneously . The proposed approach can cope with both transductive training and inductive training . • We propose to iteratively learn a better graph structure with updated node embeddings , and in the meanwhile , learn better node embeddings with the updated graph structure . In addition , the iterative method dynamically stops when the learned graph structure approaches close enough to the optimal graph based on our proposed stopping criterion . • We cast graph structure learning problem as similarity metric learning problem and leverage an adapted graph regularization for controlling smoothness , connectivity and sparsity of the generated graph . • Our extensive experiments demonstrate that our model can consistently outperform or match state-of-the-art baselines . 2 AN ITERATIVE DEEP GRAPH LEARNING FRAMEWORK . With this paper we address the challenging problem of automatic graph structure learning for GNNs . We are given a set of n objects V associated with a feature matrix X P Rdˆn encoding the feature descriptions of the objects . The goal is to automatically learn the graph structure G , typically in the form of an adjacency matrix A P Rnˆn , underlying the set of objects , which will be consumed by a GNN-based model for a downstream prediction task . Unlike most existing methods that construct graphs based on hand-crafted rules or features as a preprocessing step , our proposed Iterative Deep Graph Learning ( IDGL ) framework formulates the problem as an iterative learning problem that jointly learns the graph structure and the GNN parameters iteratively in an end-to-end manner . The overall model architecture is shown in Fig . 1 . 2.1 GRAPH LEARNING AS SIMILARITY METRIC LEARNING . In traditional graph theory , various methods have been explored to construct a graph from data points . These methods usually apply a metric function to compute the similarity between pairs of nodes during preprocessing , and then consume the constructed graph in a downstream task . Unlike these methods , in this work , we design a learnable metric function for graph structure learning , which will be jointly trained with the prediction model dedicated to a downstream task . 2.1.1 SIMILARITY METRIC LEARNING . Common options for metrics include cosine similarity , radial basis function ( RBF ) kernel and attention mechanisms . A good metric function is supposed to be learnable and expressively powerful . We design a weighted cosine similarity as our metric function , defined as , sij “ cospw d vi , w d vjq ( 1 ) where d denotes the Hadamard product , and w is a learnable weight vector which has the same dimension as the input vectors vi and vj , and learns to highlight different dimensions of the vectors . After preliminary experiments , we have found extending Eq . ( 1 ) to a multi-head version to be beneficial , similar to Vaswani et al . ( 2017 ) ; Veličković et al . ( 2017 ) . Specifically , we use m weight vectors ( each vector representing one perspective ) to compute m independent similarity matrices using Eq . ( 1 ) , and take their average as the final similarity S : skij “ cospwk d vi , wk d vjq , sij “ 1 m m ÿ k “ 1 skij ( 2 ) Intuitively , skij computes the cosine similarity between the two input vectors vi and vj , for the k-th perspective where each perspective considers one part of the semantics captured in the vectors . 2.1.2 GRAPH SPARSIFICATION VIA ε-NEIGHBORHOOD An adjacency matrix ( same for a metric ) is supposed to be non-negative while sij ranges between r´1 , 1s . In addition , many underlying graph structures are much more sparse than a fully connected graph , which is not only computationally expensive but also makes little sense for most applications . We hence proceed to extract a symmetric sparse adjacency matrix A from S by considering only the ε-neighborhood for each node . Specifically , we mask off those elements in S which are smaller than certain non-negative threshold ε. Aij “ `` sij sij ą ε 0 otherwise ( 3 ) 2.2 GRAPH REGULARIZATION . In graph signal processing ( Shuman et al. , 2013 ) , each column of the feature matrix X can be considered as a graph signal . A widely adopted assumption for graph signals is that values change smoothly across adjacent nodes . Given an undirected graph with symmetric weighted adjacency matrix A , the smoothness of a set of graph signals x1 , . . . , xn P Rd is usually measured by the Dirichlet energy ( Belkin & Niyogi , 2002 ) , ΩpA , Xq “ 1 2 ÿ i , j Aij ||xi ´ xj ||2 “ trpXTLXq ( 4 ) where trp¨q denotes the trace of a matrix , L “ D´A is the graph Laplacian , and D “ ř j Aij is the degree matrix . As can be seen , minimizing ΩpA , Xq forces adjacent nodes to have similar features , thus enforces smoothness of the graph signals on the graph associated to A . However , solely minimizing the above smoothness loss will result in the trivial solution A “ 0 . Also , it is desirable to have control of how sparse the resulting graph is . Following Kalofolias ( 2016 ) , we impose additional constraints to the learned graph , fpAq “ ´β1T logpA1q ` γ||A||2F ( 5 ) where || ¨ ||F denotes the Frobenius norm of a matrix . As we can see , the first term penalizes the formation of disconnected graphs via the logarithmic barrier , and the second term controls sparsity by penalizing large degrees due to the first term . In this work , we borrow the above techniques , and apply them as regularization terms to the graph learned by Eqs . ( 2 ) and ( 3 ) . As shown in Eq . ( 6 ) , the overall graph regularization loss is defined as the sum of the above losses , which is able to control the smoothness , connectivity and sparsity of the resulting graph where α , β and γ are all non-negative hyperparameters . LG “ αΩpA , Xq ` fpAq ( 6 ) 2.3 ITERATIVE METHOD FOR JOINT GRAPH STRUCTURE AND REPRESENTATION LEARNING . 2.3.1 JOINT GRAPH STRUCTURE AND REPRESENTATION LEARNING . We expect the graph structure underlying a set of objects to serve two purposes : on the one hand , it should respect the semantic relations among the objects , which is enforced by the metric function ( Eq . ( 2 ) ) and the smoothness loss ( Eq . ( 4 ) ) ; on the other hand , it should suit the needs of the downstream prediction task . Compared to previous works which directly optimize the adjacency matrix based on either some graph regularization loss ( Kalofolias & Perraudin , 2017 ) , or some task-dependent prediction loss ( Franceschi et al. , 2019 ) , we propose to learn an optimal similarity metric function as well as the GNN parameters by minimizing a joint loss function combining both the prediction loss defined on the downstream task and the graph regularization loss , namely , L “ Lpred ` LG . Note that our graph learning framework is agnostic to various GNNs and prediction tasks . In this paper , we adopt a two-layered GCN ( Kipf & Welling , 2016 ) where the first layer maps the node features to the intermediate embedding space ( Eq . ( 7 ) ) , and the second layer further maps the intermediate node embeddings to the output space ( Eq . ( 8 ) ) . Z “ ReLUprAXW1q ( 7 ) py “ σprAZW2q ( 8 ) Lpred “ ` ppy , yq ( 9 ) where rA is the normalized adjacency matrix , σp¨q is a task-dependent output function , and ` p¨q is a task-dependent loss function . For instance , for node classification problem , σp¨q is a softmax function for predicting a probability distribution over a set of classes , and ` p¨q is a cross-entropy function for computing the prediction loss . We now discuss how to obtain the normalized adjacency matrix rA . For some problems when an initial graph is available , our preliminary experiments showed that it is harmful to totally discard the initial graph structure . Previous works ( Veličković et al. , 2017 ; Jiang et al. , 2019 ) inject the initial graph structure into the graph learning mechanism by performing masked attention , which might limits its graph learning ability . This is because there is no way for their methods to learn weights for those edges that do not exist in the initial graph , but carry useful topological information . With the assumption that the optimal graph structure is potentially a small shift from the initial graph structure , we combine the learned graph structure with the initial graph structure as follows , rA “ λL0 ` p1´ λq Aij ř j Aij ( 10 ) where L0 is the normalized adjacency matrix of the initial graph , defined as , L0 “ D´1 { 20 A0D ´1 { 2 0 , and D0 is its degree matrix . The adjacency matrix learned by Eqs . ( 2 ) and ( 3 ) is row normalized such that each row sums to 1 . A hyperparameter λ is used to balance the trade-off between the learned graph structure and the initial graph structure . If such an initial graph structure is not available , we instead use a kNN graph constructed based on cosine similarity . Algorithm 1 : IDGL : Iterative Deep Graph Learning Framework Input : X , yr , A0s Parameters : m , ε , α , β , γ , λ , δ , T , ηr , ks Output : Θ , rAptq , py 1 rA0 Ð kNNpX , kqs // Init . A0 to kNN graph if A0 is unavailable 2 Ap0q , rAp0q Ð tX , A0u using Eqs . ( 2 ) , ( 3 ) and ( 10 ) // Learn the adj . matrix 3 Zp0q Ð t rAp0q , Xu using Eq . ( 7 ) // Compute node embeddings 4 Lp0qpred Ð t rA p0q , Zp0q , yu using Eqs . ( 8 ) and ( 9 ) // Compute prediction loss 5 Lp0qG Ð tA p0q , Xu using Eqs . ( 4 ) – ( 6 ) // Compute graph regularization loss 6 Lp0q Ð Lp0qpred ` L p0q G // Compute joint loss 7 tÐ 0 8 while pt “ “ 0 or ||Aptq ´Apt´1q||2F ą δ||Ap0q||2F q and t ă T do 9 tÐ t ` 1 10 Aptq , rAptq Ð tZpt´1q , A0u using Eqs . ( 2 ) , ( 3 ) and ( 10 ) // Refine the adj . matrix 11 sAptq Ð t rAptq , rAp0qu using Eq . ( 11 ) 12 Zptq Ð t sAptq , Xu using Eq . ( 7 ) // Refine node embeddings 13 pyÐ t sAptq , Zptqu using Eq . ( 8 ) // Compute task output 14 Lptqpred Ð tpy , yu using Eq . ( 9 ) 15 LptqG Ð tA ptq , Xu using Eqs . ( 4 ) – ( 6 ) 16 Lptq Ð Lptqpred ` L ptq G 17 end 18 LÐ Lp0q ` řt i “ 1 L piq { t 19 if Training then 20 Back-propagate L to update model weights Θ 21 end

Summary: This paper leverages metric learning to learn graph structure jointly with the learning of graph embedding. Firstly, it defines the similarity between any pair of nodes as the cosine similarity of nodal representations learned from attributes from nodes. Some tricks such as multi-head and sparsification are applied to learned cosine similarity to enhance the performance. Secondly, the authors introduce several graph regularizations to make the learned graph smooth, connected, sparse and non-trivial. Finally, the learned graph is linearly combined with the existing graph, using a hyperparameter \lambda. The convergence and time complexity are analyzed. The authors design experiments on five datasets. The paper also contains some issues: