Graph Learning Network: A Structure Learning Algorithm

1 INTRODUCTION . Data is organically structured ( and can be represented as a graph ) as relations exist between its elements ( nodes on such graph ) , e.g. , networks , images , proteins , etc . Hence , learning to predict this structure from its components plays an important role in understanding the data and the process that generated it . We consider the problem of predicting the structure of a given set of points ( which we assume are the nodes of a graph ) and an initial structure ( connections of the points ) . Simultaneously , we aim to learn to predict these structures according to some prior information . Predicting the structure and nodes ’ information of a graph is not a new task . Existing approaches tend to focus on one of these tasks , and ignore the properties of the other in their solutions . For instance , generative graph models ( Grover et al. , 2018 ; Li et al. , 2018 ; Simonovsky & Komodakis , 2018 ; You et al. , 2018 ) create graphs ( mainly the structure ) and ignore the features involved on this process . Node classification methods ( Defferrard et al. , 2016 ; Kipf & Welling , 2017 ; Lee et al. , 2018 ; Zhang et al. , 2018 ) work closely with the features of the nodes , but assume that the graph structure is fixed and given , both of which restrict the problems that can be solved . And link prediction methods ( Grover & Leskovec , 2016 ; Kipf & Welling , 2016 ; Perozzi et al. , 2014 ) are a compromise in between . However , they work looking at pairs of nodes at a time and , commonly , ignore the whole structure of the graph to make its predictions . On the contrary , in this paper , we present a simple yet effective method to predict the structure of a given set of points , that we assume have an underlying graph structure , and create node embeddings of their original features that are more robust for further processing ( e.g. , classification ) . Our proposal comprises a two step repetitive process that obtains expected node embeddings , and then use them to obtain the best prediction of the structure given the information at that step . These steps are repeated on a refinement process , and encoded as layers in a neural network . Our contribution is the definition of two prediction functions ( for nodes ’ features and adjacency ) , that let us extract the most probable structure given a set of points and their feature embeddings , respectively . We also present a layer-wise architecture that define our iterative process and our prediction functions , and a learning framework that let us learn , on and end-to-end fashion , how to predict the structure given a family of graphs . We demonstrate that our proposed method can effectively extract relevant features and generate graph representations on diverse settings . Additionally , we introduce a synthetic dataset that contains patterns that can be controlled and mapped into graphs to evaluate the robustness of existing methods . We present more details regarding the dataset on Appendix A . 2 GRAPH LEARNING NETWORK . Given a set of vertices V = { vi } , such that every element vi is a feature vector , we intend to predict its structure as a set of edges between the vertices , E = { ( vi , vj ) : vi , vj ∈ V } . In other words , we want to learn the edges of the graph G = ( V , E ) that maximize the relations between the vertices given some prior patterns , i.e. , a family of graphs . To achieve this , we perform two alternating tasks for a given number of times ( this defines our architecture ) , akin to an expectation-maximization process . At each step , we transform the nodes ’ features through convolutions on the graph ( Kipf & Welling , 2017 ) to learn better representations to predict their structure . Then , we use these transformed features to predict the next structure , which is represented through an adjacency matrix . The learned convolutions on the graph represent a set of responses on the nodes that will reveal their relations . These responses are combined to create or delete connections between the nodes , and encoded into the adjacency matrix . The sequential application of these steps recover effective relations on nodes , even when trained on families of the graphs . We represent this process , which is illustrated in Fig . 1 , through a neural network that is trained in an end-to-end fashion to learn , both , the convolution kernels and the structure-predicting functions . 2.1 NODE EMBEDDINGS . At a given step , l , on the alternating process , we have the dl hidden features , H ( l ) ∈ Rn×dl , of the n nodes , and the set of edges ( structure ) encoded into an adjacency matrix A ( l ) ∈ [ 0 , 1 ] n×n that represent our graph . As introduced , our first step is to produce the features of the next step , H ( l+1 ) , through the embedding function , hl , H ( l+1 ) = hl ( H ( l ) , A ( l ) ) . ( 1 ) In our proposal , we intend to transform each node ’ s features locally by using the information of its neighborhood on the graph . Hence , we use the convolutional graph operation proposed by Kipf & Welling ( 2017 ) hl ( H ( l ) , A ( l ) ) = σl ( τ ( A ( l ) ) H ( l ) W ( l ) ) , ( 2 ) where W ( l ) ∈ Rdl×dl+1 is the learnable weights of the convolution kernel for the lth step , σl is an activation function , and τ ( · ) is a symmetric normalization transformation of the adjacency matrix , defined by τ ( A ( l ) ) = ( D̂ ( l ) ) − 12 ( A ( l ) + In ) ( D̂ ( l ) ) − 12 , ( 3 ) where D̂ ( l ) is the degree matrix of the graph plus the identity , that is , D̂ ( l ) = D ( l ) + In , ( 4 ) where D ( l ) is the degree matrix of A ( l ) , and In is the identity matrix of size n×n . Unlike previous work ( Kipf & Welling , 2017 ) , we are computing convolutions that will have different neighborhoods at each step defined by the changing A ( l ) . In summary , this step allow us to learn a response function , defined by the weights W ( l ) of the kernel , that embed the node ’ s features into a suitable form to predict the structure of the graph . 2.2 ADJACENCY MATRIX PREDICTION . After obtaining the nodes embedding , H ( l ) , we predict the adjacency matrix , A ( l ) , given these embedding values . In general , that step is defined as A ( l+1 ) = gl ( H ( l ) , A ( l ) ) . ( 5 ) We explore two definitions for gl : a general function that depends on all the nodes of the graph , fl , and a convolution-based operation , cl , that depends on the local connections of the graph . The former , is defined as gl ( H ( l ) , A ( l ) ) = fl ( H ( l ) ) = σl ( f̃l ( H ( l ) ) ) , ( 6 ) where f̃l is approximated with a set of fully connected layers , and a non-linear function σl . In our experiments , we settled for two consecutive layers that reduce the encoding space before predicting the adjacency matrix for the next step , i.e. , with a transformation from Rn×dl → R1×1024 → Rn×n . ( Note that this definition does not depend on the previous predicted structure , A ( l−1 ) , as it relies only on the encoded information from the features of the nodes , H ( l ) . Nevertheless , for compatibility , we use the same signature for both forms of the gl function . ) The second form , cl , is based on a convolution operation that processes the information of the graph locally and transforms it into a predicted adjacency . The transformation function first computes a scored adjacency , which may be interpreted as the probability of linking the nodes , defined by αl ( H ( l ) , A ( l ) ) = σl ( τ ( A ( l ) ) H ( l ) U ( l ) ) , ( 7 ) where σl is a non-linear function , τ ( · ) is the symmetric normalization transformation ( 3 ) , and U ( l ) ∈ Rdl×n is the learnable weight matrix for the linear combinations of the nodes ’ features H ( l ) . In other words , the αl function broadcasts the information of the nodes ’ neighborhoods ( as determined by the adjacency on the previous step , A ( l ) ) , and , at each edge , creates a score of the possible adjacency as a linear combination of the nodes ’ features restricted to the existing structure . Then , once again , a linear combination of the combined neighborhood ’ s information ( 7 ) is created by gl ( H ( l ) , A ( l ) ) = cl ( H ( l ) , A ( l ) ) = σl [ τ ( A ( l ) ) V ( l ) αl ( H ( l ) , A ( l ) ) ] , ( 8 ) where σl is a non-linear function , αl ( · , · ) is the approximated adjacency representation ( 7 ) , and the V ( l ) ∈ Rn×n matrix is the learnable weights that create the prediction of each edge by combining the previous scores . This operation restricts the combination of features to the local structure through the multiplication of the normalized adjacency , τ ( A ( l ) ) . For the last output matrix , A ( L ) , the output of the cL function passes through a fully connected layer of size n× n , and then through a sigmoid function . After extracting the prediction of the last layer , A ( L ) , to convert the predicted value into an edge , we use a simple threshold operation Aoi = { 1 if A ( L ) i ≥ , 0 otherwise , ( 9 ) where A ( L ) i is the ith edge value of the final prediction , is a threshold that defines what will be considered edge , and i is every index of the edges in the adjacency matrix . 2.3 LEARNING FRAMEWORK . We are assuming that we have a family of undirected graphs , G = { Gi } , that have a particular structure pattern that we are interested in . We will use each of the graphs , Gi = ( Vi , Ai ) , to learn the parameters , Θ , of our model that minimize the loss function ( 16 ) on each of them . The structure of each graph is used as ground truth , A∗i = Ai . The graph is predicted by the set of node embedding , hl ( 1 ) , and the adjacency prediction , gl ( 5 ) , functions that depend on a set of parameters Θ , i.e. , our model is defined by GLN = { hl , gl ; Θ } Ll=0 . ( 10 ) Our input comprises the vertices , H ( 0 ) = Vi , and some structure for training . In our experiments , we used the identity , A ( 0 ) = I . However , other structures can be used as well . In the following , we describe our learning framework to obtain the parameters of the functions hl and gl , for every l. To simplify the notation we will omit the parameters on the losses and in their functions . Given the combinations of pairs of vertices on a graph , the total number of pairs with an edge ( positive class ) is , commonly , fewer than pairs without an edge ( negative class ) . In order to handle the imbalance between the two binary classes ( edge , no edge ) , we used the HED-loss function ( Xie & Tu , 2015 ) that is a class-balanced cross-entropy loss function . Then we consider the edge-class objective function as Lclass = −β ∑ i∈Y+ logP ( Aoi ) − ( 1− β ) ∑ j∈Y− logP ( Aoj ) , ( 11 ) where Aoi is the indexed predicted edge for the ith pair of vertices , where the index comes from an enumeration of the positive ( edge ) and negative ( no edge ) class of the pairs of vertices on the ground-truth graph denoted by Y+ and Y− , respectively ; β = |Y+|/|Y | and 1 − β = |Y−|/|Y | are the proportion of positive and negative pairs of vertices on the A∗ graph , and Y = Y+ ∪ Y− ; and P ( · ) is the probability of a pair of vertices to be of a given class given by the last layer , such as P ( Aoi ) = A ( L ) i . ( 12 ) Individually penalizing the ( class ) prediction of each edge is not enough to model the structure of the graph . Hence , we compare the whole structure of the predicted graph , Ao , with its ground truth , A∗ . By treating the edges on the adjacency matrices as regions on an image , we try to maximize the intersection over union ( Rahman & Wang , 2016 ) of the structural regions . Then we consider the objective function , Lstruct = 1− Ao ∩A∗ Ao ∪A∗ = 1− ∑ i , j A o i , jA ∗ i , j∑ i , j A o i , j +A ∗ i , j −Aoi , jA∗i , j . ( 13 ) On the other hand , the predictions in each layer A ( l ) , where l ∈ { 0 , . . . , L } , must be symmetric in an undirected graph . ( This restriction can be removed in case of working with directed graphs , without loss of generalization . ) To guarantee that , we penalize the symmetry of our predictions by a mean square loss function . Hence , we used the symmetry loss function Lsym = 1 L L∑ l=0 ∥∥∥A ( l ) −A ( l ) > ∥∥∥2 , ( 14 ) where · > is the transposition operator . We also regularize all the parameters Θ in the model by Lreg = ∑ θ∈Θ ‖θ‖2 . ( 15 ) Finally , we aim to minimize the total loss that is the sum of all of the previous ones , defined by L = λ1Lclass + λ2Lstruct + λ3Lsym + λ4Lreg , ( 16 ) where λ1 , λ2 , λ3 , and λ4 are hyper-parameters that define the contribution of each loss to the learning process .

Summary: Authors propose a supervised method for predicting adjacency matrix for a set of points. Loss function consists of 4 terms: intersection over union loss with respect to target adjacency, cross entropy loss, symmetry penalty and L2 regularization of parameters. Learning process consists of alternating node feature updates parametrized by GCN-like layers and updates of the adjacency matrix (different across layers).