Neural Structured Prediction for Inductive Node Classification

This paper studies node classification in the inductive setting , i.e. , aiming to learn a model on labeled training graphs and generalize it to infer node labels on unlabeled test graphs . This problem has been extensively studied with graph neural networks ( GNNs ) by learning effective node representations , as well as traditional structured prediction methods for modeling the structured output of node labels , e.g. , conditional random fields ( CRFs ) . In this paper , we present a new approach called the Structured Markov Network ( SMN ) , which combines the advantages of both worlds . SMN defines flexible potential functions of CRFs with GNNs . However , learning such a model is nontrivial as it involves optimizing a maximin game with high-cost inference . Inspired by the underlying connection between the joint and marginal distributions defined by Markov networks , we propose to solve an approximate version of the optimization problem as a proxy , which yields a near-optimal solution , making learning more efficient . Extensive experiments on two settings show our approach outperforms many competitive baselines . 1 INTRODUCTION . Graph-structured data ( e.g. , World Wide Web ) are ubiquitous in the real world , covering a variety of applications . This paper studies node classification , a fundamental problem in graph machine learning . Most existing efforts of node classification focus on the transductive setting ( Kipf & Welling , 2017 ; Veličković et al. , 2018 ) , i.e. , using a small set of labeled nodes in a graph to classify the rest of nodes . In this paper , we study node classification in the inductive setting ( Hamilton et al. , 2017 ) , which is receiving growing interest in the graph machine learning community . Given a set of training graphs with all the nodes labeled , we aim to learn a classifier to infer the nodes in unlabeled test graphs . This problem has been recently studied with graph neural networks ( GNNs ) ( Kipf & Welling , 2017 ; Hamilton et al. , 2017 ; Gilmer et al. , 2017 ; Veličković et al. , 2018 ) . GNNs infer the marginal label distribution of each node by learning useful node representations based on node features and edges . Specifically , the representation of each node is iteratively updated by the representations of its neighbors . Once a GNN is learned on training graphs , it can be further applied to test graphs to infer node labels . Owing to the high capacity of nonlinear neural architectures , GNNs achieve impressive results on many datasets . However , one limitation of GNNs is that they ignore the joint dependency of node labels , and thus node labels are predicted separately without modeling structured output . Indeed , modeling structured output has been widely explored by the literature of structured prediction ( BakIr et al. , 2007 ) . Structured prediction methods predict node labels collectively , so the label prediction of each node can be improved according to the predicted labels of neighboring nodes . One representative approach is the conditional random field ( CRF ) ( Lafferty et al. , 2001 ) . A CRF models the joint distribution of node labels with Markov networks , and thus training CRFs becomes a learning task in graphical models , while predicting node labels corresponds to an inference task . Typically , the potential functions in CRFs are parameterized as log-linear functions , which suffer from low model capacities . One remedy for this is to define potential functions with GNNs ( Ma et al. , 2018 ; Qu et al. , 2019 ) . However , most of the effective methods for learning CRFs involve a maximin game ( Wainwright & Jordan , 2008 ; Sutton & McCallum , 2012 ) , making learning often hard to converge , especially when GNNs are used to parameterize potential functions . Besides , as learning CRFs requires doing inference on the graphical models , the combined model requires a long run time . In this paper , we address these challenges by proposing SMN ( Structured Markov Network ) , which is high in capacity , efficient in learning , and able to model the joint dependency of node labels . SMN is inspired by theoretical works in graphical models ( Wainwright & Jordan , 2008 ) , which reveal close connections between the joint label distribution and the node/edge marginal label distribution in a Markov network . Based on that , we approximate the original optimization problem with a proxy problem , where the potential functions in CRFs are defined by combining a collection of node/edge pseudomarginal distributions , which are parameterized by GNNs that satisfy a few simple constraints . This proxy problem can be easily solved by maximizing the data likelihood on each node and edge , which yields a near-optimal joint label distribution on training graphs . Once the model is learned , we apply it to test graphs and run loopy belief propagation ( Murphy et al. , 1999 ) to infer node labels . Experiments on two settings against both GNNs and CRFs prove the effectiveness of our approach . 2 RELATED WORK . Graph neural networks ( GNNs ) perform node classification by learning useful node representations . The idea is to iteratively update the representation of each node according to its own features as well as its neighboring nodes . Well-known models include graph convolutional networks ( Kipf & Welling , 2017 ) , graph attention networks ( Veličković et al. , 2018 ) , and message passing neural networks ( Gilmer et al. , 2017 ) . Most earlier efforts focus on designing GNNs for node classification in the transductive setting ( Yang et al. , 2016 ; Kipf & Welling , 2017 ; Gao & Ji , 2019 ; Xhonneux et al. , 2020 ) , and many recent works move to study the inductive setting ( Hamilton et al. , 2017 ; Gao et al. , 2018 ; Chiang et al. , 2019 ; Li et al. , 2019 ; Chen et al. , 2020a ; Zeng et al. , 2020 ) . Because of high model capacity and efficient end-to-end training , GNNs achieve impressive results on inductive node classification . Despite the great success , GNNs only try to model the marginal distribution of each node label and predict node labels separately without considering their joint dependency . In contrast , SMN models joint distributions of node labels by combining GNNs with CRFs , which can thus predicts node labels collectively to improve the result . See Sec . 4.3 for the detailed discussion . Another type of approach for inductive node classification is structured prediction , which focuses on modeling the dependency of node labels , so that the predicted node labels are more consistent . One representative approach is structured SVM ( Tsochantaridis et al. , 2005 ; Finley & Joachims , 2008 ; Sarawagi & Gupta , 2008 ) , but it lacks a probabilistic interpretation to handle the uncertainty of the prediction . Another representative probabilistic approach is conditional random field ( Lafferty et al. , 2001 ; Sutton & McCallum , 2006 ) , which models the distribution of output spaces by using a Markov network . CRFs have been proven effective in many applications , such as POS tagging ( Lafferty et al. , 2001 ) , shallow parsing ( Sha & Pereira , 2003 ) , image labeling ( He et al. , 2004 ) , and sequence labeling ( Lample et al. , 2016 ; Ma & Hovy , 2016 ; Liu et al. , 2018 ) . Nevertheless , the potential functions in CRFs are typically defined as log-linear functions , suffering from low model capacity . There are also some recent works trying to combine GNNs and CRFs . Some works use GNNs to solve inference problems in graphical models ( Dai et al. , 2016 ; Satorras et al. , 2019 ; Zhang et al. , 2020 ; Chen et al. , 2020b ; Satorras & Welling , 2020 ) . In contrast , our approach uses GNNs to parameterize the potential functions in CRFs , which is in a similar vein to Ma et al . ( 2018 ) ; Qu et al . ( 2019 ) ; Ma et al . ( 2019 ; 2021 ) ; Wang et al . ( 2021 ) . Among them , Ma et al . ( 2018 ) and Qu et al . ( 2019 ) optimize the pseudolikelihood ( Besag , 1975 ) for model learning , and Wang et al . ( 2021 ) optimizes a cross-entropy loss on each single node , which can yield poor approximation of the true joint likelihood ( Koller & Friedman , 2009 ; Sutton & McCallum , 2012 ) . Our approach instead solves a proxy problem , which yields a near-optimal solution to the original problem of maximizing likelihood , and thus achieves superior results . For Ma et al . ( 2019 ) , they model joint dependency of node labels by designing a generative model , which is evaluated in the transductive setting . Different from them , our approach designs a discriminative model based on CRF , which has weaker assumptions on the underlying generative process , and we focus on the inductive setting . For Ma et al . ( 2021 ) , they study continuous node labels , which is different from our setting of node classification with discrete labels . Lastly , learning CRFs has been widely explored in graphical models . Many methods perform learning by solving a maximin game as a surrogate for data likelihood maximization ( Sutton & McCallum , 2009 ; 2012 ; Wiseman & Kim , 2019 ) . Nevertheless , these maximin games are often hard to optimize . To avoid solving the maximin game , we follow Wainwright et al . ( 2003 ) and build an approximate optimization problem as a proxy , which is much easier to solve and yields a near-optimal solution . 3 PRELIMINARY . 3.1 PROBLEM DEFINITION . This paper focuses on inductive node classification ( Hamilton et al. , 2017 ) , where we are given some labeled training graphs and unlabeled test graphs . Each training graph is represented as ( y∗V , xV , E ) , where xV and y∗V are features and labels of a set of nodes V , and E is a set of edges . For each test graph ( xṼ , Ẽ ) , only node features xṼ and edges Ẽ are observed . Based on that , we aim to solve : • Learning . On training graphs , learning a probabilistic model to approximate p ( yV |xV , E ) . • Inference . For each test graph , inferring node labels y∗ Ṽ according to the distribution p ( yṼ |xṼ , Ẽ ) . The problem can be solved by GNNs and CRFs . Next , we introduce the details of these methods . 3.2 GRAPH NEURAL NETWORKS . For inductive node classification , graph neural networks ( GNNs ) learn node representations to predict marginal label distributions of nodes . GNNs assume all node labels are independent conditioned on node features and edges , so the joint label distribution is factorized into a set of marginals as below : pθ ( yV |xV , E ) = ∏ s∈V pθ ( ys|xV , E ) . ( 1 ) Each marginal distribution pθ ( ys|xV , E ) is modeled as a categorical distribution over label candidates , and the label probabilities are computed by applying a linear softmax classifier to the representation of node s. In general , node representations are learned via the message passing mechanism ( Gilmer et al. , 2017 ) , which brings high capacity to GNNs . Also , owing to the factorization in Eq . ( 1 ) , learning and inference can be easily solved in GNNs , where we simply need to compute loss and make prediction on each node separately . However , GNNs approximate only the marginal label distributions of nodes on training graphs , which may generalize badly and result in poor approximation of node marginal label distributions on test graphs . Also , the labels of different nodes are separately predicted according to their own marginal label distributions , yet the joint dependency of node labels is ignored .

Summary: The paper aims to break the label independence assumption in existing GNNs. To this end, a combination of GNNs and CRF is proposed, named SMN. SMN improves GNNs by modeling the joint distribution among discrete node labels; SMN is a promising alternative to CRF as it provides efficient learning and inference procedures. Empirically, SMN achieves better node-level and graph-level prediction accuracies than several existing models, with minor additional runtime overhead.