CGNF: Conditional Graph Neural Fields

1 INTRODUCTION . Graph is an expressive representation for diverse datasets . Node classification on graphs has many real world applications . For example , to classify publications we build citation networks where publications are nodes and the research fields are node labels ( Hamilton et al. , 2017 ) . To study the protein-disease relations we could construct protein-protein interaction ( PPI ) graphs where each protein is a node and diseases are treated as node labels . Node classification on graphs is a general task that creates knowledge and has real social impact , e.g. , by exploring the similarities between proteins , we can learn novel protein target for treating diseases and use that knowledge to facilitate drug repurposing and safety monitoring ( Zitnik et al. , 2018 ; Ma et al. , 2018 ) . Many earlier works take connectivity-based approaches to classify nodes , which mainly operate on the graph structure alone . The underlying assumption is that nodes that are closely connected in the graph should have similar labels . Among the most successful connectivity-based methods are node embedding techniques including ( Perozzi et al. , 2014 ; Grover & Leskovec , 2016a ; Tang et al. , 2015a ; Hamilton et al. , 2017 ; Kipf & Welling , 2016 ) . More recently , features of neighboring nodes are included in learning node embedding . These feature-based approaches incorporate graph connectivities via regularization ( Zhu et al. , 2003 ; Belkin et al. , 2006 ) or aggregation over local neighborhoods ( Defferrard et al. , 2016 ; Kipf & Welling , 2017 ; Chen et al. , 2018a ) . Among them , the graph convolutional networks ( GCN ) become one of the most successful methods of node embedding for arbitrary graphs with node features ( Kipf & Welling , 2017 ; Defferrard et al. , 2016 ; Hamilton et al. , 2017 ; Chen et al. , 2018a ) . Although GCN gets good node embeddings and node classification performance via aggregating node features and graph connectivities , the correlation of node labels was not exploited . In practice , the labels of neighborhood nodes in a graph are often mutually impacted . For example , in citation networks the labels ( fields of research ) for neighborhood nodes ( publications ) are often consistent : a paper citing another tends to appear in similar or related fields to the cited one . Unfortunately , we are not able to simply treat labels as additional node features because label information are not available in test set . Instead , the key question is how to leverage label correlation in addition to node features in node classification . We notice that energy-based approaches such as the conditional random fields ( CRF ) ( Lafferty et al. , 2001 ) are desirable for capturing structured labels including correlation of node labels . However , they may involve non-trivial optimization to perform predictions . In this paper we propose a new deep architecture to extend the GCN by defining an energy function to capture the dependencies between labels for better node embedding and classification . In particular , we model the pairwise label relation with a pairwise energy function and propose a new model integrating graph node embedding and conditional random fields . The proposed model , coined conditional graph neural fields ( CGNF ) , has the following contributions : • CGNF is the first work that bridges GCN and CRF to directly capture label correlations for more accurate node classification in graphs . Experimental results on several graph datasets demonstrated the advantage of CGNF over several baselines including GCN . • We designed a specific graph-based energy function and proposed an efficient way for approximate training as well as two efficient inference schemes for predicting test labels . • Experimental results on several graph datasets demonstrate our advantage over other graph neural networks such as GCN . 2 BACKGROUND . Graph Convolutional Networks Over the past few years , several graph convolutional network models emerged to compute informative node feature representations for arbitrary graphs ( Kipf & Welling , 2017 ; Defferrard et al. , 2016 ; Hamilton et al. , 2017 ; Chen et al. , 2018a ) . GCN models learn node embeddings in the following manner : Given each graph node initially attached with a feature vector , the embedding vector of each node are the transformed weighted sum of the feature vectors of its neighbors . All nodes are simultaneously updated to perform a layer of forward propagation over neural networks . The deeper the network , the larger the local neighborhood . Thus global information is disseminated to each graph node for learning better node embeddings . Specifically , given an undirected graph with input features matrix of nodes X and adjacency matrix of the underlying graph A , a multi-layer neural network is constructed on the graph with the following layer-wise propagation rule : H ( l+1 ) = σ ( D̃− 1 2 ÃD̃− 1 2H ( l ) W ( l ) ) ( 1 ) where Ã = A + I is the adjacency matrix with self-connections . I is the identity matrix , D is a diagonal matrix such that D̃ii = ∑ j Ãij , W ( l ) is a layer-specific parameter matrix , H ( l ) is the node representation in the lth layer , H ( 0 ) = X , and σ is an activation function ( e.g . ReLU or sigmoid ) . However , labels and label correlation is not modeled in GCN . Conditional Random Fields The conditional random fields ( CRF ) is an undirected probabilistic graphical model commonly used for structured prediction tasks , e.g . named entity recognition and image segmentation . Formally , given input feature x ∈ Rd , CRF aims at finding the label set y that maximizes the conditional probability P ( y|x ) . In particular , CRF defines a joint probability over cliques { xc } in the graph . A clique is any fully connected subset of the the graph . CRF represents P ( y|x ) as a product of clique potential Φc ( xc , yc ) . Each potential function Φc depends only on part of the feature input x denoted as xc and a label subset ya . This factorization can allow us to represent P ( y|x ) much more efficiently ( Sutton et al. , 2012 ) . By making sure the factorization form of P ( y|x ) sum to 1 , partition function Z ( x ) is used for normalization use as follows : P ( y|x ) = 1 Z ( x ) ∏ c Φa ( xc , yc ) ( 2 ) where Z ( x ) = ∑ y′c ∏ c Φa ( xc , y ′ c ) is a normalizer required for a valid probability distribution . 3 CGNF METHOD 3.1 TASK FORMULATION . Consider the problem of multi-class node classification . The input graph G = { X , Y , A } includes node feature matrix X , adjacency matrix A , and the output node label matrix Y . More specifically , we have X = [ x1 , x2 , · · · , xN ] ᵀ ∈ RN×D and Y = [ y1 , y2 , · · · , yN ] ᵀ ∈ RN×M , where N is the number of nodes in the graph , D is the dimension of node features , and M is the number of distinct labels . The ith node has a feature vector xi , and a one-hot label vector yi . The graph connectivity is specified through the adjacency matrix A ∈ RN×N , where Ai , j ∈ A indicates the edge weight between node i and node j . We follow convention to denote Â = D̃− 1 2 ÃD̃− 1 2 as a normalized version of Ã , which will be used in our model later . We follow the traditional setting of GCN and formalize the learning in semi-supervised setting where we assume we observe the labels Ytr only for some of the nodes Xtr . Our goal is to predict the labels Ŷte for other unlabeled nodes Xte . To start with , we consider { Xtr , Ytr , Atr } as training set , and { Xte , Yte , Ate } as test set . 3.2 THE CGNF MODEL Given input graph G = { X , Y , A } as previous described , our goal is to predict node labels Y for graph G. We can learn node embeddings and use them to make such predictions . In this paper , we follow ( Kipf & Welling , 2017 ) and use a two-layer GCN model given by Eq . 3 . H = f ( X , A ) = Softmax ( ÂReLu ( ÂXW 0 ) W 1 ) ( 3 ) Here W 0 ∈ RD×S , W 1 ∈ RS×M are weight parameters of GCN . Softmax function is applied rowwise to make each row { hi } N1 of GCN prediction H ∈ RN×M . However , the training of GCN is conditionally independent for each node , and label correlation is ignored by this model . To fill this gap , we aim to integrate label correlations into the GCN model . Energy based models , such as CRF , are flexible ways to define the variable dependencies . So in a general graph , considering both the impact of node features and label dependency , we can define the energy function as belows . E ( Y , X , A ) = Ec ( Yc , Xc , A ) = ∑ i ψ ( yi , xi ) + γ ∑ ( i , j ) ∈E , i < j φ ( yi , yj , Ai , j ) ( 4 ) where c is a clique , E ∈ RM is the edge set , the unary potential ψ ( · ) is designed to measure the compatibility between observed node xi and its label yi ; and pairwise potential φ ( · ) is meant for capturing label correlation ; γ is a mixture weight . With this energy function , we can derive the Gibbs distribution , i.e . the conditional probability P ( Y |X , A ) as given by Eq . 5 . P ( Y |X , A ) = exp ( −E ( Y , X , A ) ) ∑ Y ′∈Y exp ( −E ( Y ′ , X , A ) ) = exp ( −E ( Y , X , A ) ) Z ( X , A ) ( 5 ) where Z = ∑ Y ′∈Y exp ( −E ( Y ′ , X , A ) ) is the normalizer ( or called partition function ) , E is the energy function which is an indicator of the likelihood of the corresponding relationships within the clique , with a higher energy configuration having lower probability ; Y is the label space consisting of unique label assignments Y ′ . Then our objective is to maximize this conditional probability . To enhance Eq . 4 by the neural network representations of GCN , the energy function can be reformulated as : E ( Y , X , A ) = ∑ i ψ ( yi , hi ) + γ ∑ ( i , j ) ∈E , i < j φ ( yi , yj , Âi , j ) ( 6 ) where hi is the previously mentioned GCN prediction H on node i and Âi , j is an element in the normalized adjacency matrix Â . Separating the energies for each node , we can further express the energy function as Eq . 7 . E ( Y , X , A ) = ∑ i ψ ( yi , hi ) + γ 2 ∑ j∈N ( i ) φ ( yi , yj , Âi , j ) ( 7 ) where N ( i ) is the neighborhood of node i . Specifically , in a graph we define the unary potential for node i as the prediction loss of GCN , denoted as ψ ( yi , hi ) . We also define the pairwise potential between node i and node j as φ ( yi , yj , Âi , j ) , which is impacted by two factors : the normalized edge weights Âi , j and label correlation weights Uyi , yj . The two potential functions are computed as Eq . 8. ψ ( yi , hi ) = − log p ( yi|hi ) = − ∑ k yi , k log hi , k ( 8 ) φ ( yi , yj , Âi , j ) = −2Âi , jUyi , yj where yi and yj are the corresponding label index of label vector yi and yj ; and Uyi , yj ∈ U is the learnable correlation/transition weight between label yi and yj . Like in the traditional CRF , when all components for conditional probability in Eq . 5 are introduced throughly above , a common way is to use negative log likelihood as training objective function in Eq . 9 to find optimal parameters like W 0 , W 1 , U : − logP ( Y |X , A ) = E ( Y , X , A ) + logZ ( X , A ) ( 9 ) = E ( Y , X , A ) + log ∑ Y ′ exp ( −E ( Y ′ , X , A ) ) Using the energy based conditional probability , the inference of new labels is comparably easy in this model . Notice that in Eq . 9 Since P ( Y |X , A ) ∝ exp ( −E ( Y , X , A ) ) , after getting the parameters in the training phase , the inference of the test labels will simply be minY E ( Y , X , A ) . However , how to optimize the parameters by minimizing the negative log likelihood is a great challenge . The computation of the normalizer Z ( X , A ) is often intractable , which causes a big problem in training . In the next section , we will introduce two ways to solve the problem .

Summary: This paper proposed a combination of graph neural networks and conditional random field to model the correlation between node labels in the output.  In typical graph neural nets the predictions for nodes are conditionally independent given the node representations.  This paper proposes to use a CRF to compensate for that.  In terms of the approach, the authors used GCNs to produce unary potentials for the CRF, and have the pairwise potentials on each edge to model the correlation of labels of neighboring nodes.  Learning is done by optimizing pseudo-likelihood and the energy loss, while inference is performed through a couple heuristic processes.