CopulaGNN: Towards Integrating Representational and Correlational Roles of Graphs in Graph Neural Networks

1 INTRODUCTION . Graphs , as flexible data representations that store rich relational information , have been commonly used in data science tasks . Machine learning methods on graphs ( Chami et al. , 2020 ) , especially Graph Neural Networks ( GNNs ) , have attracted increasing interest in the research community . They are widely applied to real-world problems such as recommender systems ( Ying et al. , 2018 ) , social network analysis ( Li et al. , 2017 ) , and transportation forecasting ( Yu et al. , 2017 ) . Among the heterogeneous types of graph-structured data , it is worth noting that graphs usually play diverse roles in different contexts , different datasets , and different tasks . Some of the roles are relational , as a graph may indicate certain statistical relationships of connected observations ; some are representational , as the topological structure of a graph may encode important features/patterns of the data ; some are even causal , as a graph may reflect causal relationships specified by domain experts . It is crucial to recognize the distinct roles of a graph in order to correctly utilize the signals in the graph-structured data . In this paper , we distinguish the representational role and the correlational role of graphs in the context of node-level ( semi- ) supervised learning , and we investigate how to design better GNNs that take advantage of both roles . In a node-level prediction task , the observed graph in the data may relate to the outcomes of interest ( e.g. , node labels ) in multiple ways . Conceptually , we call that the graph plays a representational 1The code is available at https : //github.com/jiaqima/CopulaGNN . role if one can leverage it to construct better feature representations . For example , in social network analysis , aggregating user features from one ’ s friends is usually helpful ( thanks to the well-known homophily phenomenon ( McPherson et al. , 2001 ) ) . In addition , the structural properties of a user ’ s local network , e.g. , structural diversity ( Ugander et al. , 2012 ) and structural holes ( Burt , 2009 ; Lou & Tang , 2013 ) , often provide useful information for making predictions about certain outcomes of that user . On the other hand , sometimes a graph directly encodes correlations between the outcomes of connected nodes , and we call it playing a correlational role . For example , hyper-linked Webpages are likely to be visited together even if they have dissimilar content . In spatiotemporal predictions , the outcome of nearby locations , conditional on all the features , may still be correlated . We note that the graph structure may provide useful predictive information through both roles but in distinct ways . While both the representational and the correlational roles are common in graph-structured data , we find that , through a simulation study , many existing GNN models are incapable of utilizing the correlational information encoded in a graph . Specifically , we design a synthetic dataset for the node-level regression . The node-level outcomes are drawn from a multivariate normal distribution , with the mean and the covariance as functions of the graph to reflect the representational and correlation roles respectively . We find that when the graph only provides correlational information of the node outcomes , many popular GNN models underperform a multi-layer perceptron which does not consider the graph at all . To mitigate this deficiency of GNNs , we propose a principled solution , the Copula Graph Neural Network ( CopulaGNN ) , which can take a wide range of GNNs as the base model and improve their capabilities of modeling the correlational graph information . The key insight of the proposed method is that , by decomposing the joint distribution of node outcomes into the product of marginal densities and a copula density , the representational information and correlational information can be separately modeled . The former is modeled by the marginal densities through a base GNN while the latter is modeled by a Gaussian copula . The proposed method also enjoys the benefit of easy extension to various types of node outcome variables including continuous variables , discrete count variables , or even mixed-type variables . We instantiate CopulaGNN with normal and Poisson marginal distributions for continuous and count regression tasks respectively . We also implement two types of copula parameterizations combined with two types of base GNNs . We evaluate the proposed method on both synthetic and real-world data with both continuous and count regression tasks . The experimental results show that CopulaGNNs significantly outperform their base GNN counterparts when the graph in the data exhibits both correlational and representational roles . We summarize our main contributions as follows : 1 . We raise the question of distinguishing the two roles played by the graph and demonstrate that many existing GNNs are incapable of utilizing the graph information when it plays a pure correlational role . 2 . We propose a principled solution , the CopulaGNN , to integrate the representational and correlational roles of the graph . 3 . We empirically demonstrate the effectiveness of CopulaGNN compared to base GNNs on semi-supervised regression tasks . 2 RELATED WORK . There have been extensive existing works that model either the representational role or the correlational role of the graph in node-level ( semi- ) supervised learning tasks . However , there are fewer methods that try to model both sides simultaneously , especially with a GNN . Methods focusing on the representational role . As we mentioned in Section 1 , the graph can help construct better node feature representations by both providing extra topological information and guiding node feature aggregation . There have been vast existing studies on both directions , and among them we can only list a couple of examples . Various methods have been proposed to leverage the topological information of graph-structured data in machine learning tasks , such as graph kernels ( Vishwanathan et al. , 2010 ) , node embeddings ( Perozzi et al. , 2014 ; Tang et al. , 2015 ; Grover & Leskovec , 2016 ) , and GNNs ( Xu et al. , 2018 ) . Aggregating node features on an attributed graph has also been widely studied , e.g. , through feature smoothing ( Mei et al. , 2008 ) or GNNs ( Kipf & Welling , 2016 ; Hamilton et al. , 2017 ) . In this work , we restrict our focus on the GNN models , which have been the state-of-the-art graph representation learning method on various tasks . Methods focusing on the correlational role . On the other hand , there has also been extensive literature on modeling the dependence of variables on connected nodes in a graph . One group of methods is called the graph-based regularization ( Zhu et al. , 2003 ; Li et al. , 2019 ) , where it is assumed that the variables associated with linked objects change smoothly and pose an explicit similarity regularization among them . The correlational role of the graph is also closely related to undirected graphical models ( Lauritzen , 1996 ; Jordan et al. , 2004 ; Wainwright & Jordan , 2008 ) . In graphical models , the edges in a graph provide a representation of the conditional ( in ) dependence structure among a set of random variables , which are represented by the node set of the graph . Finally , there has been a line of research that combines graphical models with copulas and leads to more flexible model families ( Elidan , 2010 ; Dobra et al. , 2011 ; Liu et al. , 2012 ; Bauer et al. , 2012 ) . Our proposed method integrates the benefits of copulas and GNNs to capture both the representational and correlational roles . Methods improving GNNs by leveraging the correlational graph information . A few methods explicitly leverage the correlational graph information to improve the GNN training , but most of them focus on a classification setting ( Qu et al. , 2019 ; Ma et al. , 2019 ) . A recent study ( Jia & Benson , 2020 ) that we have been aware of only lately shares a similar motivation to ours , yet our methodology differs significantly . In particular , Jia & Benson ( 2020 ) apply a multivariate normal distribution to model the correlation of node outcomes , which can be viewed as a special case of our proposed CopulaGNN when a Gaussian copula with normal marginals is used . Our method not only generalizes to other marginals ( we show the effectiveness of some of them ) , but also has a more flexible parameterization on the correlation matrix of the copula distribution . In addition , we differ with these previous works by explicitly distinguishing the two roles of the graph in the data . 3 SIMULATING THE TWO ROLES OF THE GRAPH . In this section , we investigate , through a simulation study , the representational and correlational roles of the graph in the context of node-level semi-supervised learning . 3.1 NODE-LEVEL SEMI-SUPERVISED LEARNING . We start by formally introducing the problem of node-level semi-supervised learning . A graph is a tuple : G = ( V , E ) , where V = { 1 , 2 , . . . , n } is the set of n nodes ; E ∈ V × V is the set of edges and let s = |E| be the number of edges . The graph is also associated with X ∈ Rn×d and y ∈ Rn , which are the node features and outcome labels . In the semi-supervised learning setting , we only observe the labels of 0 < m < n nodes . Without loss of generality , we assume the labels of nodes { 1 , 2 , . . . , m } are observed and those of { m+ 1 , . . . , n } are missing . Therefore , the label vector y can be partitioned as y = ( yTobs , y T miss ) T . The goal of a node-level semi-supervised learning task is to infer ymiss based on ( yobs , X , G ) . 3.2 SYNTHETIC DATA . To simulate the representational and correlational roles of the graph , we first design a synthetic dataset by specifying the joint distribution of y conditional on X and G. In particular , we let the joint distribution of the node outcomes take the form of y|X , G ∼ N ( µ ( X , G ) , Σ ( G ) ) , for some µ , Σ to be specified . In this way , the graph G plays a representational role through µ ( X , G ) and a correlational role through Σ ( G ) . Specifically , we generate synthetic node-level regression data on a graph with n nodes and s edges ( see Appendix A.1 for the whole procedure ) . We first randomly generate a feature matrix X ∈ Rn×d0 . AssumeA is the adjacency matrix of the graph , D is the degree matrix , and L = D−A is the graph Laplacian . Let Ã = A + I and D̃ = D + I . Given parameters wy ∈ Rd0 , we generate the node label vector y ∼ N ( µ , Σ ) , where , for some γ > 0 , τ > 0 , and σ2 > 0 , ( a ) µ = D̃−1ÃXwy , Σ = σ2I ; ( b ) µ = Xwy , Σ = τ ( L+ γI ) −1 ; ( c ) µ = D̃−1ÃXwy , Σ = τ ( L+ γI ) −1 . Depending on how ( µ , Σ ) are configured , we get three types of synthetic data settings : ( a ) , ( b ) , and ( c ) . Intuitively , the graph plays a pure representational role in setting ( a ) since the label of a node depends on the aggregated features of its local neighborhood and the node labels are independent conditional on the node features . In setting ( b ) , the graph plays a pure correlational role ; while the means of node labels only depend on their own node features , the node labels are still correlated conditional on the features , and the correlation is determined by the graph structure . Finally , setting ( c ) is a combination of ( a ) and ( b ) where the graph plays both representational and correlational roles . In the rest of this section , we test the performance of a few widely used GNNs under setting ( a ) and ( b ) to examine their capabilities of utilizing the representational and correlational information . We defer the experimental results under setting ( c ) to Section 5.2 for ease of reading .

Summary: The paper presents a new model based on the Graphical Neural Network (GNN). The proposed model adopts probability distributions called copulas and is called the Copula Graphical Neural Network (CopulaGNN). Two parametrizations of the CopulaGNN are given, and the learning of the proposed model is discussed. Experiments suggest that the CopulaGNN outperforms existing GNNs and MLP in almost all setups.