Neural Structured Prediction for Inductive Node Classification

This paper studies node classification in the inductive setting , i.e. , aiming to learn a model on labeled training graphs and generalize it to infer node labels on unlabeled test graphs . This problem has been extensively studied with graph neural networks ( GNNs ) by learning effective node representations , as well as traditional structured prediction methods for modeling the structured output of node labels , e.g. , conditional random fields ( CRFs ) . In this paper , we present a new approach called the Structured Markov Network ( SMN ) , which combines the advantages of both worlds . SMN defines flexible potential functions of CRFs with GNNs . However , learning such a model is nontrivial as it involves optimizing a maximin game with high-cost inference . Inspired by the underlying connection between the joint and marginal distributions defined by Markov networks , we propose to solve an approximate version of the optimization problem as a proxy , which yields a near-optimal solution , making learning more efficient . Extensive experiments on two settings show our approach outperforms many competitive baselines . 1 INTRODUCTION . Graph-structured data ( e.g. , World Wide Web ) are ubiquitous in the real world , covering a variety of applications . This paper studies node classification , a fundamental problem in graph machine learning . Most existing efforts of node classification focus on the transductive setting ( Kipf & Welling , 2017 ; Veličković et al. , 2018 ) , i.e. , using a small set of labeled nodes in a graph to classify the rest of nodes . In this paper , we study node classification in the inductive setting ( Hamilton et al. , 2017 ) , which is receiving growing interest in the graph machine learning community . Given a set of training graphs with all the nodes labeled , we aim to learn a classifier to infer the nodes in unlabeled test graphs . This problem has been recently studied with graph neural networks ( GNNs ) ( Kipf & Welling , 2017 ; Hamilton et al. , 2017 ; Gilmer et al. , 2017 ; Veličković et al. , 2018 ) . GNNs infer the marginal label distribution of each node by learning useful node representations based on node features and edges . Specifically , the representation of each node is iteratively updated by the representations of its neighbors . Once a GNN is learned on training graphs , it can be further applied to test graphs to infer node labels . Owing to the high capacity of nonlinear neural architectures , GNNs achieve impressive results on many datasets . However , one limitation of GNNs is that they ignore the joint dependency of node labels , and thus node labels are predicted separately without modeling structured output . Indeed , modeling structured output has been widely explored by the literature of structured prediction ( BakIr et al. , 2007 ) . Structured prediction methods predict node labels collectively , so the label prediction of each node can be improved according to the predicted labels of neighboring nodes . One representative approach is the conditional random field ( CRF ) ( Lafferty et al. , 2001 ) . A CRF models the joint distribution of node labels with Markov networks , and thus training CRFs becomes a learning task in graphical models , while predicting node labels corresponds to an inference task . Typically , the potential functions in CRFs are parameterized as log-linear functions , which suffer from low model capacities . One remedy for this is to define potential functions with GNNs ( Ma et al. , 2018 ; Qu et al. , 2019 ) . However , most of the effective methods for learning CRFs involve a maximin game ( Wainwright & Jordan , 2008 ; Sutton & McCallum , 2012 ) , making learning often hard to converge , especially when GNNs are used to parameterize potential functions . Besides , as learning CRFs requires doing inference on the graphical models , the combined model requires a long run time . In this paper , we address these challenges by proposing SMN ( Structured Markov Network ) , which is high in capacity , efficient in learning , and able to model the joint dependency of node labels . SMN is inspired by theoretical works in graphical models ( Wainwright & Jordan , 2008 ) , which reveal close connections between the joint label distribution and the node/edge marginal label distribution in a Markov network . Based on that , we approximate the original optimization problem with a proxy problem , where the potential functions in CRFs are defined by combining a collection of node/edge pseudomarginal distributions , which are parameterized by GNNs that satisfy a few simple constraints . This proxy problem can be easily solved by maximizing the data likelihood on each node and edge , which yields a near-optimal joint label distribution on training graphs . Once the model is learned , we apply it to test graphs and run loopy belief propagation ( Murphy et al. , 1999 ) to infer node labels . Experiments on two settings against both GNNs and CRFs prove the effectiveness of our approach . 2 RELATED WORK . Graph neural networks ( GNNs ) perform node classification by learning useful node representations . The idea is to iteratively update the representation of each node according to its own features as well as its neighboring nodes . Well-known models include graph convolutional networks ( Kipf & Welling , 2017 ) , graph attention networks ( Veličković et al. , 2018 ) , and message passing neural networks ( Gilmer et al. , 2017 ) . Most earlier efforts focus on designing GNNs for node classification in the transductive setting ( Yang et al. , 2016 ; Kipf & Welling , 2017 ; Gao & Ji , 2019 ; Xhonneux et al. , 2020 ) , and many recent works move to study the inductive setting ( Hamilton et al. , 2017 ; Gao et al. , 2018 ; Chiang et al. , 2019 ; Li et al. , 2019 ; Chen et al. , 2020a ; Zeng et al. , 2020 ) . Because of high model capacity and efficient end-to-end training , GNNs achieve impressive results on inductive node classification . Despite the great success , GNNs only try to model the marginal distribution of each node label and predict node labels separately without considering their joint dependency . In contrast , SMN models joint distributions of node labels by combining GNNs with CRFs , which can thus predicts node labels collectively to improve the result . See Sec . 4.3 for the detailed discussion . Another type of approach for inductive node classification is structured prediction , which focuses on modeling the dependency of node labels , so that the predicted node labels are more consistent . One representative approach is structured SVM ( Tsochantaridis et al. , 2005 ; Finley & Joachims , 2008 ; Sarawagi & Gupta , 2008 ) , but it lacks a probabilistic interpretation to handle the uncertainty of the prediction . Another representative probabilistic approach is conditional random field ( Lafferty et al. , 2001 ; Sutton & McCallum , 2006 ) , which models the distribution of output spaces by using a Markov network . CRFs have been proven effective in many applications , such as POS tagging ( Lafferty et al. , 2001 ) , shallow parsing ( Sha & Pereira , 2003 ) , image labeling ( He et al. , 2004 ) , and sequence labeling ( Lample et al. , 2016 ; Ma & Hovy , 2016 ; Liu et al. , 2018 ) . Nevertheless , the potential functions in CRFs are typically defined as log-linear functions , suffering from low model capacity . There are also some recent works trying to combine GNNs and CRFs . Some works use GNNs to solve inference problems in graphical models ( Dai et al. , 2016 ; Satorras et al. , 2019 ; Zhang et al. , 2020 ; Chen et al. , 2020b ; Satorras & Welling , 2020 ) . In contrast , our approach uses GNNs to parameterize the potential functions in CRFs , which is in a similar vein to Ma et al . ( 2018 ) ; Qu et al . ( 2019 ) ; Ma et al . ( 2019 ; 2021 ) ; Wang et al . ( 2021 ) . Among them , Ma et al . ( 2018 ) and Qu et al . ( 2019 ) optimize the pseudolikelihood ( Besag , 1975 ) for model learning , and Wang et al . ( 2021 ) optimizes a cross-entropy loss on each single node , which can yield poor approximation of the true joint likelihood ( Koller & Friedman , 2009 ; Sutton & McCallum , 2012 ) . Our approach instead solves a proxy problem , which yields a near-optimal solution to the original problem of maximizing likelihood , and thus achieves superior results . For Ma et al . ( 2019 ) , they model joint dependency of node labels by designing a generative model , which is evaluated in the transductive setting . Different from them , our approach designs a discriminative model based on CRF , which has weaker assumptions on the underlying generative process , and we focus on the inductive setting . For Ma et al . ( 2021 ) , they study continuous node labels , which is different from our setting of node classification with discrete labels . Lastly , learning CRFs has been widely explored in graphical models . Many methods perform learning by solving a maximin game as a surrogate for data likelihood maximization ( Sutton & McCallum , 2009 ; 2012 ; Wiseman & Kim , 2019 ) . Nevertheless , these maximin games are often hard to optimize . To avoid solving the maximin game , we follow Wainwright et al . ( 2003 ) and build an approximate optimization problem as a proxy , which is much easier to solve and yields a near-optimal solution . 3 PRELIMINARY . 3.1 PROBLEM DEFINITION . This paper focuses on inductive node classification ( Hamilton et al. , 2017 ) , where we are given some labeled training graphs and unlabeled test graphs . Each training graph is represented as ( y∗V , xV , E ) , where xV and y∗V are features and labels of a set of nodes V , and E is a set of edges . For each test graph ( xṼ , Ẽ ) , only node features xṼ and edges Ẽ are observed . Based on that , we aim to solve : • Learning . On training graphs , learning a probabilistic model to approximate p ( yV |xV , E ) . • Inference . For each test graph , inferring node labels y∗ Ṽ according to the distribution p ( yṼ |xṼ , Ẽ ) . The problem can be solved by GNNs and CRFs . Next , we introduce the details of these methods . 3.2 GRAPH NEURAL NETWORKS . For inductive node classification , graph neural networks ( GNNs ) learn node representations to predict marginal label distributions of nodes . GNNs assume all node labels are independent conditioned on node features and edges , so the joint label distribution is factorized into a set of marginals as below : pθ ( yV |xV , E ) = ∏ s∈V pθ ( ys|xV , E ) . ( 1 ) Each marginal distribution pθ ( ys|xV , E ) is modeled as a categorical distribution over label candidates , and the label probabilities are computed by applying a linear softmax classifier to the representation of node s. In general , node representations are learned via the message passing mechanism ( Gilmer et al. , 2017 ) , which brings high capacity to GNNs . Also , owing to the factorization in Eq . ( 1 ) , learning and inference can be easily solved in GNNs , where we simply need to compute loss and make prediction on each node separately . However , GNNs approximate only the marginal label distributions of nodes on training graphs , which may generalize badly and result in poor approximation of node marginal label distributions on test graphs . Also , the labels of different nodes are separately predicted according to their own marginal label distributions , yet the joint dependency of node labels is ignored .

Summary: This paper proposes a CRF for classifying nodes in graphs where the CRF has a potential for each node and edge in the graph. GNNs are used for computing the potentials, one GNN for node potentials and one GNN for edge potentials. For general graphs, computing the partition function is intractable, so approximations are used during both learning and inference.  Learning draws from prior work and combines learning pseudomarginals of nodes and edges with GNNs and optionally some steps of "refinement" by optimizing a maximin game equivalent to likelihood maximization. Inference uses sum-product or max-product loopy BP (they perform similarly, though max-product is slightly better). The procedure is called a "Structured Markov Network" (SMN). Experimental results on several graph node classification tasks show that SMNs outperform several baselines, including both CRFs with standard training (using approximate inference during training) and CRFs trained with pseudolikelihood.