CopulaGNN: Towards Integrating Representational and Correlational Roles of Graphs in Graph Neural Networks
Authors: Jiaqi Ma
Bo Chang
Xuefei Zhang
Qiaozhu Mei
Venue: ICLR 2021 Poster (2021)
Keywords: Graph Neural Network
Gaussian Copula
Gaussian Graphical Model

Abstract: Graph-structured data are ubiquitous. However, graphs encode diverse types of information and thus play different roles in data representation. In this paper, we distinguish the \textit{representational} and the \textit{correlational} roles played by the graphs in node-level prediction tasks, and we investigate how Graph Neural Network (GNN) models can effectively leverage both types of information. Conceptually, the representational information provides guidance for the model to construct better node features; while the correlational information indicates the correlation between node outcomes conditional on node features. Through a simulation study, we find that many popular GNN models are incapable of effectively utilizing the correlational information. By leveraging the idea of the copula, a principled way to describe the dependence among multivariate random variables, we offer a general solution. The proposed Copula Graph Neural Network (CopulaGNN) can take a wide range of GNN models as base models and utilize both representational and correlational information stored in the graphs. Experimental results on two types of regression tasks verify the effectiveness of the proposed method.

Published as a conference paper at ICLR 2021
COPULAGNN: TOWARDS INTEGRATING REPRESEN- TATIONAL AND CORRELATIONAL ROLES OF GRAPHS IN GRAPH NEURAL NETWORKS
Jiaqi Ma
Bo Chang
Xuefei Zhang
School of Information University of Michigan jf lagima ma @mii ch edu
Department of Statisties University of British Columbia
Department of Statistics
University of Michigan
bchang wwt.sttttttt......eeoeeoeeeeeeet
xfzhange uumi ch edu
Qiaozhu Mei
School of Information and Department of EECS University of Michigan qme i @ tumiie ch. edu
ABSTRACT
Graph-structured data are ubiquitous. However, graphs encode diverse types of information and thus play different roles data representation. this paper, we distinguish the representational and the correlational roles played by the graphs in node-level prediction tasks, and we investigate how Graph Neural Network (GNN) models effectively leverage both types of information. Conceptually, the representational information provides guidance for the model to construct bet- ter node features; while the correlational information indicates the correlation be- tween node outcomes conditional on node features. Through a simulation study, we find that many popular GNN models are incapable of effectively utilizing the correlational information. By leveraging the idea of the copula, a principled way describe the dependence among multivariate random variables, we offer gen- eral solution. The proposed Copula Graph Neural Network (CopulaGNN) can take a wide range of GNN models as base models and utilize both representa- tional and correlational information stored in the graphs. Experimental results on two types regression tasks verify the effectiveness of the proposed method"
INTRODUCTION
Graphs, flexible data representations that store rich relational information, have been commonly used in data science tasks. Machine learning methods on graphs (Chami et al. 2020), especially Graph Neural Networks (GNNs), have attracted increasing interest in the research community. They are widely applied to real-world problems such as recommender systems (Ying et al. 2018). social network analysis (Li et al.) 2017), and transportation forecasting (Yu al. (017). Among the het- erogeneous types of graph-structune different contexts, different datasets, and different tasks. Some of the roles are relational, as graph may indicate certain statistical relationships of connected observations: some are representational, as the topological structure graph may encode important features/patterns of the data; some are even causal, as a graph may reflect causal relationships specified by domain experts.
data, it is worth noting that graphs usually play diverse roles in
crucial recognize the distinct roles of a graph in order correctly utilize the signals in the graph-structured data. In this paper, we distinguish the representational role and the correlational role of graphs in the context node-level )superiiuee learning, and we investigate how to design better GNNs that take advantage of both roles.
In node-level prediction task, the observed graph in the data may relate to the outcomes of interest (e.g., node labels) in multiple ways. Conceptually, we call that the graph plays representational
'The code is available at wttptttp///// httppa:/gittub.... com/ agi ma mammaaa//mmaaooooe aGNN

---

Published as a conference paper at ICLR 2021
role if one can leverage it to construct better feature representations, For example, in social network analysis, aggregating user features from one's ' friends is usually helpful (thanks to the well- homophily phenomenon (McPherson al., (200T). In addition, the structural properties of user's local network, e.g., structural diversity (Ugander et al., 2012) and structural holes (Burt 2009 Lou & TTagg] 2013), often provide useful information for making predictions about certain outcomes of that user. On the other hand, sometimes a graph directly encodes correlations between the outcomes of connected nodes, and we call playing a correlational role. For example, hyper-linked Webpages are likely to be visited together even if they have dissimilar content. In spatiotemporal predictions, the outcome of nearby locations, conditional on all the features, may still be correlated. We note that the graph structure may provide useful predictive information through both roles but in distinct ways.
While both the representational and the correlational roles are common in graph-structured data, we find that, through a simulation study, many existing GNN models are incapable of utilizing the correlational information encoded in graph. Specifically, we design a synthetic dataset for the node-level regression. The node-level outcomes are drawn from a multivariate normal distribution, with the mean and the covariance functions of the graph to reflect the representational and corre- lation roles respectively. We find that when the graph only provides correlational information of the node outcomes, many popular GNN models underperform multi-layer perceptron which does not consider the graph at all.
To mitigate this deficiency of GNNs, we propose a principled solution, the Copula Graph Neural Network (CopulaGNN), which can take wide range of GNNs as the base model and improve their capabilities of modeling the correlational graph information. The key insight of the proposed method is that, by decomposing the joint distribution of node outcomes into the product of marginal densities and copula density, the representational information and correlational information can be separately modeled. The former is modeled by the marginal densities through base GNN while the latter is modeled by a Gaussian copula. The proposed method also enjoys the benefit of easy extension to various types of node outcome variables including continuous variables, discrete count variables, or even mixed-type variables. We instantiate CopulaGNN with normal and Poisson marginal distributions for continuous and count regression tasks respectively. We also implement two types of copula parameterizations combined with two types of base GNNs.
We evaluate the proposed method on both synthetic and real-world data with both continuous and count regression tasks. The experimental results show that CopulaGNNs significantly outperform their base GNN counterparts when the graph in the data exhibits both correlational and representa- tional roles. We summarize our main contributions as follows:
1. We raise the question of distinguishing the roles played by the graph and demonstrate that many existing GNNs are incapable of utilizing the graph information when it plays pure correlational role.
2. We propose principled solution, the CopulaGNN, to integrate the representational and correlational roles of the graph.
3. We empirically demonstrate the effectiveness of CopulaGNNN compared to base GNNs on semi-supervised regression tasks.
2 RELATED WORK
There have been extensive existing works that model either the representational role or the corre- lational role of the graph in node-level (remi-)supervised learning tasks. However, there are fewer methods that try to model both sides simultaneously, especially with a GNN.
Methods focusing on the representational role. As we mentioned in Section T. the graph can help construct better node feature representations by both providing extra topological information and guiding node feature aggregation. There have been vast existing studies on both directions, and among them we can only list couple of examples. Various methods have been proposed to leverage the topological information of graph-sttucturee data in machine learning tasks, such as graph kernels (Vishwanathan et al. 2010), node embeddings (Perozzi al.) 2014 Tang al. 2015 Grover & Leskovee (2016), and GN (Xu et al. 2018). Aggregating node features on an attributed graph has also been widely studied, e.g., through feature smoothing (Mei et al. 2008) GNNs (Kipf
2

---

Published as a conference paper at ICLR 2021
& Welling 2016;
Hamilton al. 2017). In this work, we restrict our focus the GNN models, which have been the state-of-the-an art graph representation learning method on various tasks.
Methods focusing on the correlational role. On the other hand, there has also been extensive liter- ature on modeling the dependence of variables on connected nodes in graph. One group of methods called the graph-baser based regularization (Zhu et al. 2003 Li et al. 2019), where it assumed that the variables associated with linked objects change smoothly and pose an explicit similarity regulariza- tion among them. The correlational role of the graph is also closely related to undirected graphical models (Lauritzen 1996 Jordan et al. 2004 Wainwright & Jordan 2008). In graphical models, the edges in a graph provide a representation of the conditional (in) (in)dependence structure among a set of random variables, which are represented by the node set of the graph. Finally, there has been line of research that combines graphical models with copulas and leads to more flexible model fam- ilies [Elidan] 2010; Dobra al., [201]] [Lu et al. 2012 Bauer al., 2012). Our proposed method integrates the benefits of copulas and GNNs to capture both the representational and correlational roles.
Methods improving GNNs by leveraging the correlational graph information. A few methods explicitly leverage the correlational graph information to improve the GNN training, but most of them focus on classification setting (Qu al., 2019; Ma et al. 2019). A recent study (Jia & Benson 2020) that we have been aware of only lately shares a similar motivation to ours, yet our methodology differs significantly. In particular, Jia & Benson (2020) apply multivariate normal distribution to model the correlation of node outcomes, which can be viewed as a special case of our proposed CopulaGNN when Gaussian copula with normal marginals is used. Our method not only generalizes to other marginals (we show the effectiveness of some of them), but also has a more flexible parameterization on the correlation matrix of the copula distribution. In addition, differ with these previous works by explicitly distinguishing the two roles of the graph in the data.
3 SIMULATING THE TWO ROLES OF THE GRAPH
In this section, we investigate, through a simulation study, the representational and correlational roles of the graph in the context of node-level semi-supervised learning.
3.1 NODE-LEVEL SEMI-SUPERVISED LEARNING
We start by formally introducing the problem of node-level semi-supervised learning. A graph is tuple: g (V, E), where V (1....... n) is the set of n nodes; & E V is the set of edges and let 8 = |&| be the number of edges. The graph is also associated with X E RRad and E RR. which are the node features and outcome labels.
the semi-supervised learning setting, we only nodes. Without loss of generality, we assume the labels of nodes
observe the labels of ( < are observed and those of {m + I,....,1, n are missing. Therefore, the label vector y
{1, 2.... m) can be partitioned as y yy Yniss) YT. The goal of a node-level semi-supervised learning task is
to infer Ymiss based on (Yobs, X, K.
3.2 SYNTHETIC DATA
To simulate the representational and correlational roles of the graph, we first design a synthetic dataset by specifying the joint distribution of y conditional on X and G. In particular, we let the joint distribution of the node outcomes take the form of y g ~ ( 9), E(G),, for some μ, to be specified. In this way, the graph 9 plays representational role through pXX, g) and correlational role through E(g).
Specifically, we generate synthetic node-level regression data on graph with n nodes and edges (see Appendix A. for the whole procedure). We first randomly generate feature matrix X E d. Assume A the adjacency matrix of the graph, D is the degree matrix, and D A is A + and D = D + I. Given parameters wy E R.do we generate the node label vector y N NM,, E), where, for some y 0, T 0, and o² >
the graph Laplacian. Let
(a) # D- AXw,, w.. Σ I²I: (b) f = Xury E = 7 (L+
3

---

Published as a conference paper at ICLR 2021
MLP
GCN
SAGE
GAT
APPNP
MLP
GCN
SAGE
GAT
APPNP
0.2
2
0.1
0.2
d² 2.5
6¹ 6--5
6'-10
0'-20
T=0.5
T= Configuration
5.5.0
Configuration
(a) μl = D A E
1²I.
(b) F = Xw,y E = r(L + +
Figure The coefficient of determination R (the higher the better) of GNNs and MIP when the graph plays (a) the representational role or (b) the correlational role. For each configuration, the results are aggregated from 100 trials. In (a), all GNNs outperform MLP; (b), all GNNs underperform MLP.
(0) μ P= D IA E == (L +
Depending on how (H, Σ) are configured, we get three types of synthetic data settings: (a), (b), and (c). Intuitively, the graph plays a pure representational role in setting (a) since the label of node depends on the aggregated features of its local neighborhood and the node labels are independent conditional on the node features.
setting (b), the graph plays a pure correlational role; while the means of node labels only depend on their own node features, the node labels are still correlated conditional on the features, and the correlation is determined by the graph structure. Finally, setting (c) is a combination of (a) and (b) where the graph plays both representational and correlational roles.
In the rest of this section, test the performance of few widely used GNNs under setting (a) and (b) to examine their capabilities of utilizing the representational and correlational information. We defer the experimental results under setting (c) Section 5.2 for ease of reading.
3.3 SIMULATION STUDY
Simulation Setup. We set the number of nodes n = 300, the number of edges s = 5000, and the feature dimension do 10. Elements of both W, and wy are generated from i.i.d. standard normal distribution. For setting (a), we vary o² E (2.5, 5, 10, 20). For settings (b) and (c), we set y = 0.1 and vary r E (0.5, 1, 2, 5}. We test common GNN models, GCN (Kipf & Welling 2016), GraphSAGE (Hamilton et al. 2017) (denoted as SAGE), GAT (Veličković et al. 2018). and
APPNP (Klicpera et al. 2018)) as well as the multi-layer perceptron (MLP).
Simulation Results. First, we observe that all types of GNNs outperform MLP under setting (a) (Figure Tap. where the graph plays a pure representational role. This is not surprising as the architectures of the GNNs encode similar feature aggregation structure as the data. However, under setting (b) (Figure 1b) where the graph plays pure correlational role, all 4 types of GNNs underperform MLP. This suggests that a majority of popular GNN models might be incapable of fully utilizing the correlational graph information.
Motivated by our findings in the simulation study, in the following section, we seek for methods that augment existing GNN models in order to better utilize both representational and correlational information in the graph.
4 COPULA GRAPH NEURAL NETWORK
In this section, we propose principled solution called the Copula Graph Neural Network (Copu- laGNN). At the core of our method is the application of copulas, which are widely used for modeling multivariate dependence. In the rest of this section, first provide a brief introduction to copulas (more detailed expositions can be found in the monographs by Joe (2014) and Czado (2019)), then present the proposed CopulaGNN and its parameterization, learning, and inference.
4

---

Published as a conference paper at ICLR 2021
4.1 INTRODUCTION TO COPULAS
In order to decompose the joint distribution of the labels y into representational and correlational components, make of copulas, which widely used in multivariate statistics model the joint distribution of a random vector.
General formulation. Sklar's theorem Sklar 1959) states that any multivariate joint distribution F of random vector Y (Yi,..., Yx) can be written in terms of one-dimensional marginal distributions (v) = P(Y; S y) and a copula [0, 1 + [0, 1] that describes the dependence structures among variables:
F(y.,..... Vy.) = C(F (y))))
In other words, one can decompose joint distribution into two components: the marginals and the copula. Such decomposition allows two-step approach to modeling joint distribution: (1) learning the marginals F;; (2) learning the copula C, where various parametric copula families are available. Furthermore, copula C can also be regarded as the Cumulative Distribution Func- tion (CDF) of corresponding distribution on the unit hypercube [0, 1]". Its copula density de- noted by c(ui...., un)) == 8" C((,,, u2..... Uh M,a) //8] ... Sun. The Probability Density Function (PDF) of a random vector can be represented by its corresponding copula density. the random
vector Y is continuous, its PDF can be written
f(y) c(u
Un
where is the PDF of Y; u4 (i;))) and c is the copula density. For discrete random vectors, the form of the Probability Mass Function (PMF) is more complex. See Appendix B.2 for details.
Gaussian copula. One of the most popular copula family is the Gaussian copula. When the joint distribution F is multivariate normal with mean of O and a covariance matrix of Σ, the corresponding copula is the Gaussian copula:
C'u,, U2, Uni 2) 2r (2-4-)(()) (ui), & (u,); R),
where (-; 0, R) is the multivariate normal CDF, is the correlation matrix of E, and + (-)(() is the quantile function of the univariate standard normal distribution. Its copula density is
1 (u)" I.) (.)))-(()))
c(u,, U2 ; Σ) (det R)- exp
where I is the identity matrix of size n and = (u1, u2...., Un)
4.2 THE PROPOSED MODEL
Recall that our goal is model both representational and correlational graph information in the conditional joint distribution of the node outcomes,
fly: X. X. = c(u1, Un X. 9) II fol: 9,
which can be decomposed into the copula density and marginal densities. In this formulation, the representational information and correlational information are naturally separated into the marginal densities fi, for densities and the copula density are conditional on the node features and the graph G. Next, need to choose proper distribution family for each of these densities and parameterize the distribution parameters as functions of X and G.
..... n, and the copula density c, respectively. Note that both the marginal
Choice of distribution family and parameterization for the copula density. For the distribution family, we choose the Gaussian copula as the copula family, c(u),... Un Σ(X, G; 0)), where the form of {.: and the learnable parameters 0 remain to be specified. To leverage the correlational graph information, Σ the Gaussian copula density. Let K = Σ be the precision matrix; if two nodes and not linked in the graph, constrain the corresponding (i. (ih entry in K be O. In other
draw connection between the graph structure & and the covariance matrix
5

---

Published as a conference paper at ICLR 2021
words, the absence of an edge between nodes i and j leads to their outcome variables y; and y being conditionally independent given all other variables. The motivation of parameterizing the precision matrix K instead of the covariance matrix Σ is closely related to undirected graphical models (Lauritzen] 1996] Jordan et al. 2004 Wainwright & Jordan, 2008), where the conditional dependence structure among set of variables is fully represented by edges in underlying graph. In our use case, we could view our assumption on K as graphical model among random variables (y1, (y1....: Vn), where the underlying graph structure known.
The conditional independence assumption has significantly reduced the number of non-zero entries in K be estimated. However, without any further constraints, there are still |8| free parameters growing with the graph size, which can hardly be estimated accurately given only one observation of (yi..... ). In practice, we consider two ways of parametrizing K with fewer parameters.
Two-parameter parametrization. A rather strong but simple constraint is to assume the non-zero off- off-diagonal entries of K have the same value or they are proportional to the corresponding entries in the normalized adjacency matrix, and introduce two global parameters controlling the overall strength of correlation. For example, we could have K = T -L + 7 as what we did in the simulation study in Section 3 or K = ( D- ADD -1/2) as used in Jia & Benson 2020).
where (T, 7) or (a, 3) are learnable parameters.
Regression-based parametrization. We further propose a more flexible parameterization that allows the non-zero entries in K to be estimated by a regressor taking node features inputs. In pariticular, for any (i,j)-pair corresponding to non-zero entry of K, we A., softplus(h (h(r, ;;;; 0)), where h is a two-layer MI.P that takes the concatenation of æ; and æ, as input and outputs a scalar. Let D be the degree matrix if we treat Á as a weighted adjacency matrix, and set the precision matrix K In D A. This parameterization improves the flexibility on estimating K while keeping the number of learnable parameters 0 independent of the graph size n. It also ensures that K is positive-detee definite and thus invertible.
Choice of distribution families and parameterization for the marginal densities. One benefit of the copula framework the flcxibility on the choice of distribution families for the marginal densities. In this work, choose the marginal densities to be normal distributions if the labels are continuous variables, and we choose them to be Poisson distributions the are discrete. We denote the i-th marginal density function by (w n: (X, 9; 0)) and the corresponding CDF by F. (y (X, G; 0)), where (= O) denotes the distribution parameters to be specified.
If the i-th marginal distribution takes the form of normal distribution N(a;, of), then (X, G; 0) = (X, G; of (X, G; 0)). We define μ: (X, G: 0) as the output of base GNN model for node i, and of (X, G; 0) the i-th diagnoal element of the covariance matrix E(X, G: 0) as we specified in the Gaussian copula. If the i-th marginal distribution takes the form of a Poisson distribution Pois( then n. (X, 9: 0) (X, G; 0), and we define (X, 9; 0) as the output of base GNN model for node i. Either way, the representational role of the graph is reflected in the location parameters (1A; or X₂) computed by a base GNN model. In practice, we can also choose other distribution families such as the log-normal or negative binomial, depending on our belief on the true distributions of the node outcomes. One can even choose different distribution families for different nodes simultaneously if necessary.
4.3 MODEL LEARNING AND INFERENCE
For simplicity of notation, we write n) (X, 9: and E(X, 9) as ny: and Σ throughout this section.
Model learning. The model parameters & are learned by maximizing the likeliodd on the observed node labels. Given the partition of y, we can further partition the covariance matrix Σ accordingly:
y
Yobs Yuiss
and Σ
Eno
E
Eo1 Σ11
where yobs = (y...... ym) and ymiss = (ym+1..... Vn). In other words, Eno and En are the covariance matrices of the observed and missing nodes. We further denote u, Filys 77) for i 1,.... . n, Upbs = (uj...., umm), and Umiss = (Um+I...., u,); that is, u; is the probability integral transform of the i-th label ys. According to Equation (T. the joint density can be written

---

Published as a conference paper at ICLR 2021
the product of the copula density and the marginal densities. Therefore the loss function, i... the negative log- nog-liieihhood is
m
(()) log fyyari g) -log cf Uubsi Eoo)
Σ
log f( m:).
(3)
The parameters are learned end-to-end using standard optimization algorithms such as Adam (Kingma & Ba 2015).
Model inference.
At inference time, we are interested in the conditional distribution (ymmss Mobs; X, g). The inference of the conditional distribution can be done via sampling. Since fly: X, G) is modeled by the Gaussian copula, we have
Φp (1hubs) ④ <0-1((unn (Umiss)
.V (
(un
Roo R10
Ro1 R11
where R is the correlation matrix corresponding to the covariance matrix E. By the property of the multivatiate normal distribution, the conditional distribution of & (umiss) (uoba) is also multivatiate normal:
(umiss) & (uobs) ~ M (Rio-R - (Uobb)) Ru1 R10 Rot)
(4)
This provides way draw samples from f yloss: X, 9), which we describe in Algorithm |
Algorithm 1: Model inference by sampling.
Input:
The node features X, the graph G, the observed node labels yl..... Um. the learned parameters O, the marginal CDF functions Fo(:: ; ns (X, 9; 0)) their inverse F (1: (X, O)) for i = 1,..., n, and the number of samples L. Output: Predicted missing node labels ù., i = + I.
for i m + 1, m. + ...., do L 0;
for i 1,2..... m do
u: t F.(y:: (X, 9: 0)); =i (u,):
Zots < [Z1,..., 2m
R < the correlation matrix corresponding to Σ(X, 9)0);
Mound R1n R Zobs:
Dxond -- R11 R1o Roj Roi:
10
for L = 1,2,... do
11
lm 1>m+t)..., =-1²" ~ (( Econd); for i = m + 1,m 2,...... do
12
yf < F.- ( n (:
13
& - 8. + /
14
15
return D., i = m + +l..... I.....R; R
Scalability. Finally, make brief remark the scalability of the proposed method. During model learning, the calculation of log c( Eoo) in Eq. (3) involves the log-determinant of the precision matrix K and its submatrix. During model inference, both the conditional mean and variance involve the evaluation of R but they can be transformed (via Schur complement) in form that only requires solving linear system with a submatrix of K as the coefficients. Thanks to the sparse structure of K, both (one-step) learning and inference can be made efficient with computation cost linear the graph size if K well-conditioned. Jii & Benson (2020) provide an introduction of numerical techniques that can efficiently calculate the log-determinant (and its gradients) of a sparse matrix, as well as the solution of a linear system with sparse coefficients, which can be directly applied to accelerate our proposed method. In addition, real-world graphs often present community structures, we can further improve the scalability of the proposed method by enforcing block structure for the precision matrix.

---

Published as a conference paper at ICLR 2021
Table 1: Experiment results on the synthetic data under setting (c) as described in Sections 3.2 and
The average test R² from 100 trials is reported (the larger the better). The asterisk markers, " # and ***, indicate the difference between variant of CopulaGNN and its GNN base model is statistically significant by a pairwise (-test at significance levels of 0.1, 0.05, and 0.01, respectively. The (+) error bar denotes the standard error of the mean.
3.3
7 0.5
1.0
== 2.0
5.0
MLP
0.624 0.011
0,549 + 0.014
0,437 + 0.018
0.193 + 0.020
GCN
0.673 0.022
0.563 + 0.034
0.384 0.055
0.174 1 0.032
a--C-GCN
0.669 0.024
0.568 + 0.033
0,408 0.050
0.200 + 0.035
R-C-GEN
0.706 + 0,0174**
0.617 + 0.023***
0,489 + 0.034
0,217 + 0.029
SAGE
0.733 0.013
0.644 + 0.020
0.507 0.030
0.262 + 0.025
a--C-SAGE
0.741 0.013**
0.650 + 0.019
0,518 0.029
0.281 + 0.024
R-C-SAGE
0.754 0.010
0.665 + 0.017
0.540 + 0.024
0.290 + 0.022
5
EXPERIMENTS
5.1 GENERAL SETUP
We instantiate CopulaGNN with either GCN or GraphSAGE as the base GNN models, and imple- ment both the two-parameter parameterization (the (a, B) parameterization, denoted by "a:3-C-", where "C" stands for copula) and the regression-based parameterization (denoted by "R-C-"). In combination, we have four variants of CopulaGNN: a:3-C-GCN, R-C-GCN, aB-C-SAGE, and R- C-SAGE. When the outcome outcome is a count variable, the Poisson margin is used. In particular, in the former case, the o:3-C- GNN degenerates to the Correlation GNN proposed by Jia & Benson (2020). We compare different variants of CopulaGNN with their base GNN counterparts, as well an MLP model, two types of regression tasks: continuous outcome variables and count outcome variables. More experiment details can be found in Appendix A.3
continuous variable, the normal margin is used; and when the
5.2 REGRESSION WITH CONTINUOUS OUTCOME VARIABLES
We use two groups of datasets with continuous outcome variables. The first group is the synthetic data of setting (c) as described in Sections 3.2 and 3.3 where a graph provides both representational and correlational information. The second group includes four regression tasks constructed from the U.S. Election data (Jia & Benson 2020). We use the coefficient of determination R² to measure the model performance.
Results. For the synthetic datasets (Table Th. we vary the value of т, which controls the overall increases, the labels become noisier and
magnitude of the label covariance. Unsurprisingly, the test R² of all models decreases. In all configurations, R-C-GCN and R-C-SAGE respectively outperform their base model counterparts, GCN and SAGE, by significant margins. This verifies the proposed method when the graph provides both representational and cor-
the effectiveness relational information. Another interesting observation is that GCN outperforms MLP when 5 is small (0.5 and 1.0), but underperforms MLP when 7 becomes large (2.0 and 5.0), whereas R-C- GCN consistently outperforms MLP. Note that 7 can also be viewed as the tradeoff between the representational role and the correlational role served by the graph. The correlational role of the graph will have more influence on the outcome variables when 7 becomes larger. This explains the intriguing observation: GCN fails to utilize the correlational information and its advantages on the representational information diminish as T increases.
For the U.S. Election dataset (Table 2 we observe that all variants of CopulaGNN significantly outperform their base GNN counterparts. is interesting that the simpler two-parameter param- eterization outperforms the regression-based parameterization in most setups this dataset. One possible explanation is that the outcome variables that are connected in the graph tend to have strong correlations, since adjacent counties usually have similar statistics. This is indeed suggested by Jia & Benson (2020). The Unemployment task in particular, where the two woo-parameter parameterization appears have the largest advantage, is shown to have the strongest correlation.
8

---

Published as a conference paper at ICLR 2021
Table 2: Experiment results on regression tasks of the U.S. Election dataset (with continuous out- come variables). The average test R from 10 trials is reported (the larger the better). The asterisk markers and the (+) error bar indicate the same meaning in Table
Education
Election
Income
Unemployment
MLP
0.660 + 0.004
0,400 + 0.003
0.597 + 0.006
0,400 + 0.004
GCN
0.418 0.004
0.472
0.002
0.607
0.003
0.572 0.010
a,9-C-GCN
0.452 + 0.001
0.558
0.002***
0.635
0.001
0.750 0.004*..
R-C-GCN
0.454 + 0.002*.*
0.578 + 0.005
0.661 + 0.001
0.654 + 0.009
SAGE
0.677 + 0.004
0,565
0.005
0.714
0.628 0.005
aB-C-SAGE
0.709 + 0.003
0.700
0.003
0.775
0.003***
0.813 0.004***
R-C-SAGE
0.707 + 0.003*.*
0.692 + 0.005
0.763 + 0.003***
0.695 + 0.003***
Table 3: Experiment results on regression tasks with count outcome variables. The average test R² (deviance) from 50 trials reported (the larger the better). The asterisk markers and the (+t) error bar indicate the same meaning as in Table [
EMNLP
Wiki-Chameleon
Wiki-Squirrel
MLP
0.125 0.006
0.347 + 0.008
0.439 0.004
GCN
0.609 + 0.002
0.408 = 0.008
0.470 + 0.006
aB3-C-GNN
0.630 + 0.001
0.405 + 0.007
0.476 + 0.006***
R-C-GCN
0.657 0.001 *.*
0.424 + 0.007*
0.490
0.006***
SAGE
0.711 + 0.002
0.343 = 0.009
0.539 + 0.004
a-9-CSAGE
0.721 + 0.002
0.352 + 0.009**
0.548 0.004***
R-C-SAGE
0.734 0.002
0.360 + 0.008
0.551 0.004***
5.3 REGRESSION WITH COUNT OUTCOME VARIABLES
We use two groups of datasets with count outcome variables. The first group consists of two Wikipedia datasets: Wiki-Chameleon and Wiki-Squirrel (Rozemberczki al. 2019); both are page- page networks of Wikipedia pages with the visiting traffic as node labels. The second group is a co-citation network of papers at the EMNLP conferences. The goal is to predict the overall number of citations of each paper (including citations from outside EMNLP). We use the R² deviinnee an Ra measure for count data (Cameron & Windmeijer 1996). measure the model performance.
Results. The results of the count regression tasks are shown in Table 3 Intuitively, hyper-linked web pages or co-cited papers are more likely to be visited or cited together, therefore leading to correlated outcome variables captured by the graph. Indeed, we observe that the different variants of CopulaGNN outperform their base model counterparts in almost all setups. However, as the correlation may not be as strong as in the U.S. Election dataset, we observe that the regression- based parameterization (R-C-GCN and R-C-SAGE) has a greater advantage.
CONCLUSION
In this work, we explicitly distinguish the representational and correlational roles of the graph rep- resentation of data. We demonstrate through simulation study that many popular GNN models are incapable of fully utilizing the correlational graph information. Furthermore, we propose Copu- laGNN, a prineipled method that improves upon wide range of GNNs achieve better prediction performance when the graph plays both representational and correlational roles. Compared with the corresponding base GNN models, multiple variants of CopulaGNN yield consistently superior results on both synthetic and real- real-world datasets for continuous and count regression tasks.
ACKNOWLEDGEMENT
Jiaqi Ma and Qiaozhu Mei were in part supported by the National Science Foundation under grant numbers 1633370 and 1620319.

---

Published as a conference paper at ICLR 2021
REFERENCES
Alexander Bauer, Claudia Czado, and Thomas Klein. Pair-copula constructions for non-gaussian dag models. Canadian Journal of Statistics, 40(1):86-109, 2012.
Ronald S Burt. Structural holes: The social structure of competition. Harvard university press, 2009.
A Colin Cameron and Frank AG Windmeijer. R-squared measures for count data regression models with applications health-care utilization. Journal of Business Economic Statistics, 14(2): 209-220, 1996.
Ines Chami, Sami Abu-El-Haija, Bryan Perozzi, Christopher Ré, and Kevin Murphy. Machine learning on graphs: A model and comprehensive taxonomy. arXiv preprint arXiv:2005.0367. 2020.
Claudia Czado. Analyzing dependent data with vine copulas. Lecture Notes in Statistics, Springer, 2019.
Adrian Dobra, Alex Lenkoski, et al. Copula Gaussian graphical models and their application to modeling functional disability data. The Annals of Applied Statistics, 5(2A))969-993 2011.
Gal Elidan. Copula bayesian networks. In Advances in neural information processing systems, pp- 559-567, 2010.
Aditya Grover and Jure Leskovee. node2vec: Scalable feature learning for networks. Proceedings of the 22nd ACM SIGKDD international conference Knowledge discovery and data mining, pp. 855-864,70 2016.
Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. In Advances in Neural Information Processing Systems, pp. 1024-1034, 2017.
Peter D Hoff, Adrian E Raftery, and Mark S Handcock. Latent space approaches to social network analysis. Journal of the american Statistical association, 97(460):1000-1098, 2002.
Junteng Jia and Austion R Benson. Residual correlation in graph neural network regression. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 588-598, 2020.
Harry Joe. Dependence Modeling with Copulas. Chapman & Hall CRC Press, Boca Raton, FL., 2014.
Michael Jordan et Graphical models. Statistical science, 19(1):140-153: 140-155, 2004.
Hannes Kazianka and Jürgen Pilz. Copula-based geostatistical modeling of continuous and discrete data including covariates. Stochastic environmental research and risk assessment, 24(5):661-61) 2010.
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. In International Conference on Learning Representations, 2015.
Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional net- works. arXiv preprint arXiv:1609.02000 2016.
Johannes Klicpera, Aleksandar Bojchevski, and Stephan Günnemann. Predict then propagate: Graph neural networks meet personalized pagerank. In International Conference on Learning Representations, 2018.
Steffen Lauritzen. Graphical models, volume 17. Clarendon Press, 1996.
Cheng Li, Jiaqi Ma, Xiaoxiao Guo, and Qiaozhu Mei. Deepcas: An end-to-end predictor of infor- mation cascades. In Proceedings of the 26th international conference on World Wide Web. pp- 577-586, 2017.
10

---

Published as a conference paper at ICLR 2021
Tianxi Li, Elizaveta Levina, Ji Zhu, et al. Prediction models for network-linked data. The Annals of Applied Statistics, 13(1):132-164, 2019.
Han Liu, Fang Han, Ming Yuan, John Lafferty, Larry Wasserman, al. High-dimensional semi-
parametric Gaussian copula graphical models. The Annals of Statistics, 40(4):2293-2326, 2012.
Tiancheng Lou and Jie Tang. Mining structural hole spanners through information diffusion in social networks. In Proceedings of the 22nd international conference World Wide Web, pp. 825-836, 2013.
Jiaqi Ma, Weijing Tang. Ji Zhu, and Qiaozhu Mei. A flexible generative framework for graph-based semi-supervised learning. arviv preprint arXiv: 10769, 2019.
Miller McPherson, Lynn Smith-Lovin, and James M Cook. Birds of feather: Homophily in social networks. Annual review of sociology, 27(11:415 2001.
Qiaozhu Mei, Duo Zhang, and ChengXiang Zhai. general optimization framework for smoothing language models graph structures. In Proceedings of the 31 annual international ACM SIGIR conference on Research and development in information retrieval, pp. 611-618, ACM, 2008.
Bryan Perozzi, Rami AI-Rfou, and Steven Skiena. Deepwalk: Online learning of social repre- sentations. In Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 701-710, 2014.
Meng Qu, Yoshua Bengio, and Jian Tang. Gmnn: Graph markov neural networks. arXiv preprint arXiv:1905.06214 2019.
Benedck Rozemberezki, Carl Allen, and Rik Sarkar. Multi-scale attributed node embedding. arXiv preprint UrXiv:1909.13022 2019.
A. Sklar. Fonetions de répartition dimensions leurs marges. Publications de I'Institut de Statistique de l'Université de Paris, 8:229-231, 1959.
Jian Tang, Meng Qu, Mingzhe Wang, Ming Zhang. Jun Yan, and Qiaozhu Mei. Line: Large-scale information network embedding. In Proceedings of the 24th international conference on world wide web, pp. 1067-1077. International World Wide Web Conferences Steering Committee, 2015.
Jie Tang, Jing Zhang, Limin Yao, Juanzi Li, Li Zhang, and Zhong Su. Ametminer: Extraction and mining of academic social networks. In KDD '08, pp. 990-998, 2008
Johan Ugander, Lars Backstrom, Cameron Marlow, and Jon Kleinberg. Structural diversity in social contagion. Proceedings of the National Academy of Sciences, 109(16):5922-5966, 2012.
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. Graph attention networks. In International Conference Learning Representations, 2018.
S Vichy N Vishwanathan, Nicol N Schraudolph, Risi Kondor, and Karsten M Borgwardt. Graph kernels. The Journal of Machine Learning Research, 11:1201-1242, 2010.
Martin Wainwright and Michael Jordan. Graphical models, exponential families, and variational inference. Foundations and Trends in Machine Learning. 1(1-2):1-305, 2008.
Keyulu Xu, Weihua Hu, Jure Leskovee, and Stefanic Jegelka. How powerful are graph neural
networks? arXiv preprint arXiv:1810.00820 2018.
Rex Ying, Ruining He, Kaifeng Chen, Pong Eksombatchai, William L Hamilton, and Jure Leskovec.
Graph convolutional neural networks for web-scale recommender systems. In Proceedings the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 974 983. 2018.
Bing Yu, Haoteng Yin, and Zhanxing Zhu. Spatio-temporal graph convolutional networks: A deep learning framework for traffic forecasting. arXiv preprint arXiv:1709.0000 04875, 2017.
Xiaojin Zhu, Zoubin Ghahramani, and John D Lafferty. Semi-supervised learning using gaussian fields and harmonic functions. In Proceedings of the 20th International conference on Machine learning (ICMI-03), pp. 912-919, 2003.
11

---

Published as a conference paper at ICLR 2021
A EXPERIMENT DETAILS
A.1 DETAILS OF SYNTHETIC DATA GENERATION
We generate the synthetic data with the following procedure.
1. Sample node feature matrix X ~ (O, Id,), where X E Raxdo and do is the feature dimension.
2. Generate the graph in way similarly to latent space model (Hoff et al. 2002). First compute the latent variables Z = XW where Wg E Rdoxd is a given weight matrix. Then calculate the latent distance l|z; 2; ll for each node pair (i.j), 1 S
I n. Finally assign edges between the m pairs nodes with the shortest latent distances to form graph with n nodes and edges.
3. Assume A is the adjacency matrix of the graph, D is the degree matrix, and L = D A
is the graph Laplacian. Let Á = A + I and D = D + I. Given parameters w, E IRd, generate the node label vector y ~ N (H, Σ), where, for some 7 > 0, T > 0, and o² > 0,
(a) = D-1 wy == o²
(b) = Xwy, E = T -
(c) = D , E = T -(L+71)-
A.2 SIMULATION DETAILS FOR SECTION
For each configuration we randomly generate 100 datasets with different seeds. For each dataset, we randomly split the nodes into training, validation, and test sets equally to form a semi-supervised learning task.
We set the number of layers as 2 and the total hidden units as 16 for all models. We use the Adam optimizer (Kingma & Bal 2015) with initial learning rate 0.01 train all models by minimizing the MSE loss, with early stopping on the validation set. Finally, we report the 122 score on the test set.
A.3 EXPERIMENT DETAILS FOR SECTION
Training details. For all neural networks involved in the experiments, we set the number of layers as 2 and the number of hidden units 16. We use the Adam optimizer train all the models and apply early stopping on a validation set. For the real-world datasets, the initial learning rate is chosen from (0.01, 0.001} on the validation set.
The U.S. Election dataset. The nodes in the election data are U.S. counties and edges connect adjacent counties on the map. Each county is associated with demographic and election statistics. In each of the regression tasks, one statistic is selected as the node outcome and the remaining statistics are used as the node features. The four regression tasks are named by the outcome statistics: Education. Election, Income, Unemployment. We randomly split the data into training, validation, and test sets with ratio 6:2:2 following Jia & Benson 22000), and we refer to their work for more details of the datasets.
The Wikipedia datasets. Each of the two Wikipedia datasets consists of graph Wikipedia, where each node is a Wikipedia page related to the animal of the page title and edges reflect mutual hyper-links between the pages. The node features are principal components of binary indicators for the presence of certain nouns. The count outcome variable of each node label is the monthly traffic in the unit of thousands.
The EMNLP dataset. This dataset is constructed from the DBLP citation data provided by AMiner (Tang et all, 20088.. We first extract a set of papers published on the EMNLP conference and treat each paper as a node. Then we construct graph where two papers have edge if they are cited simultaneously by at least two EMNLP papers. The node features are principal components of the bag-of- oo-wddd of paper titles and abstracts well as the year of publication. The node label is the number of citations of each paper from outside EMNLP. For both types of datasets, we randomly split the data into training, validation, and test sets with ratio 1:1:1.
12

---

Published as a conference paper at ICLR 2021
B MORE DETAILS ABOUT COPULAS
B.1 TWO-DIMENSIONAL EXAMPLES
Figure shows PDFs of two-dimensional distributions constructed using different parametric copu- las; the marginal distributions are all standard normal.
Gaussian copula
t-copula
Frank copula
-1-
2
-2
MTCJ copula
Joe copula
Gumbel copula
0-
-1
2
Figure 2: Density functions of two-dimensional distributions constructed using copulas. The marginal distributions are all standard normal. See Joe (2014) for the definitions of these parametric copulas.
B.2 APPROXIMATING THE COPULAS FOR DISCRETE RANDOM VARIABLES
If the random vector Y is discrete, the copula representation of its PMF is more complex. Take n = as an example, the PMF of Y (Y1, Y2) is
((y) = P(Yi = ,/, Y2 = 1/2
P(Y S y1 S 122 IP(| < y1, Y2 S v2) P(Y S 01, Y2 < 92) P(Y, < yı, Y2 < v2) C(u12 U22) - C(u11, u22) - C( u21) + C'un,, u21).
where
lim (xx (1,))) and u:2 F((ss). In general, the PMF has the following
form:
2
f(y) Σ
(-1))++ +5n CCn Ch Unja): )
(5)
=1=1
J,=1
which is computationally intractable because there are 2n summands. Kazianka & Pilz (2010) pro- pose an approximation of the above PMF based the generalized quantile transform. smooths the CDF of ordinal discrete variables from step function to piece-wise linear one:
n
f(y) = c(ui, (n II f(()).
(6)
where v; (un + /2 has been shown that the approximation works well as long as the marginal variance is not too small. We apply this method to approximate PMF to handle discrete random variables.
13


Reviews:
[{"review_id": "sHfHVbbomWr", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes to use Copula function to model the correlational role in the node-level prediction tasks. From my own perspective, I quite enjoy this work. In particular, it uses a simple simulation example to uncover the importance of correlational role, i.e., most of the GNNs approaches might perform worse than simple MLPs as they are incapable of using the correlation graph information. This observation is simple, however, would provide deep insights into the current GNNs literature. \n\n[Introduction & Related Work]\nThe authors clearly state their motivation and precisely introduce the mentioned works. Readers would get a clear understanding on the background information. \n\n[Simulation the two roles]\nThe authors use simple simulation examples to show the importance of correlation graph information. \n\n[Copula Graph Neural Networks]\nThe authors introduce the detail architecture design for the model. Although the design looks reasonable, I have several questions as follows:\n\n1, the authors mainly used the Gaussian Copula as the correlation modeling choice and incorporate the graph structure in the precision matrix. I can totally understand that this setting can reduce the number of parameters. What if the precision matrix is without the graph structure and is simply placed with sparse priors?  \n\n2, following the above question, it looks like Gaussian Copula is the only suitable choice for Copula functions. How can one use other copula functions, as we cannot incorporate the graph structure and the other copula functions needs complex structures (e.g. C-vines or D-vines) to model the high dimensional data?\n\n3, the two-parameter and regression parameterisation seems quite complicate in fitting the precision matrix. Thus, can these parameterisation method also be used for other copula functions?\n\n[Experiments]\nThe experimental part can be sufficient to verify the effectiveness of the authors' proposed model. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "pqvM9HyaAx", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThis paper proposes a novel framework for the use of graph neural networks for node-level prediction when these predictions have correlations across nodes that cannot be entirely explained by the node features. In a nutshell, the idea is to propose a probabilistic model for the joint node-level output whose mean can be parametrized as a GNN and whose correlation is determined by a precision matrix with the same support as the graph at hand.\n\nOverall:\n\nMy evaluation is positive. Although the presentation could be streamlined and made clearer (see comments below), the explicit recognition for the need of a learnable module for the correlation that is separated from the GNN seems to be a fundamental insight that can shed light on multiple existing methods.\n\nPros:\n\n1 - The synthetic example in Section 3.2 nicely illustrates the shortcoming of some existing methods.\n\n2 - The generic framework is applicable on top of existing GNNs architectures. As such, it can be applicable to a wide range of settings\n\nCons:\n\n1 - The presentation is hard to follow at some points. For example, the “Model Inference” paragraph is key for the implementation of the proposed method. However, it is presented in only a handful of lines without any discussion on the different steps. Maybe even an Algorithm environment would be preferred for this.\n\n2 - It is unclear to me the true value in doing the presentation around copulas. Section 4.1 is too short in the sense that it would be difficult for someone that does not know what a copula is beforehand to actually pick up the necessary notions from there. The key idea, formally stated in Section 4.2, can be explained without resorting to copulas, simply in terms of marginal distributions and covariances. What is exactly the value of introducing the copula formalism? Please make more emphasis on this.\n\n++++++++++++++++++++++++++++++++++++++++++++\n\nEdit after author response: We thank the authors for their response to my concerns and those of other reviewers.", "extra_scores": {}, "extra_text": {}}, {"review_id": "hDy820MO6fD", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes to distinguish the representational and correlational roles of the graph representation of data. The authors proposes CopulaGNN that decomposes the joint distribution of node labels into the product of marginal densities and a copula density, which models the representational information and correlational information separately. The experimental results demonstrate the effectiveness of the proposed method. The idea of introducing copula into GNN is novel and interesting, and this paper is basically well written and well organized. However, I have the following concerns:\n\n1. Characterizing representational role and correlational role is the key motivation of the proposed method, but I'm not quite clear about their definitions. As far as I understand, the representational role means graph homophily and the correlational role means graph topology. But homophily and topology are not two disjoint concepts since homophily says that two connected nodes are likely to have the same feature, which is exactly based on (local) topology. I think a better way to decompose the model is to say that, the label of a node depends on not only its feature but also the graph topology.\n\n2. The authors decompose the joint distribution of node labels into marginal distributions of node labels as well as a copula density. The marginal distributions characterize the representational information (homophily) and the copula density characterizes the correlational information (topology). Here the marginal distribution f_i(y_i; X, G) is not only dependent on its own feature but only the feature of its neighbors, which is not reasonable since the marginal distribution should only characterize the node itself. This is also connected to the above argument that the dependence of node label can be decomposed into its feature and graph topology. Therefore, it seems more reasonable to write eq. 2 as f(y; X, G) = c(u_1, ..., u_n; G) \\prod f_i (y_i; X). Only in this way we can say that we \"decompose\" the joint distribution of node labels. The original eq.2 does not decompose anything since X and G appear in both copula density and marginal distributions. I'm not saying that the current eq.2 do not work, but it \"seems\" more elegant to do as what I said above. Otherwise the original eq.2 is just like fitting the joint distribution with two complete modules then simply combining them together.\n\n3. Another issue that confuses me is that, the studied problem is formulated as learning joint distribution of node labels, but we actually only have one observed sample, i.e., the whole graph. Although the authors have tried their best to reduce the parameters of the model, it is still prone to overfitting if you maximize the likelihood only on one sample.\n\n4. Modeling the problem as learning joint distributions will cause another issue that doing inference is too time confusing, since we need to infer labels of all nodes as the same time even if we only need the label of one node. In addition, as the authors said, multiple sampling rounds may be required to calculate the average.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "EhnSqG24jgo", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "##########################################################################\n\nSummary:\n\nThe paper presents a new model based on the Graphical Neural Network (GNN). The proposed model adopts probability distributions called copulas and is called the Copula Graphical Neural Network (CopulaGNN). Two parametrizations of the CopulaGNN are given, and the learning of the proposed model is discussed. Experiments suggest that the CopulaGNN outperforms existing GNNs and MLP in almost all setups.\n\n \n##########################################################################\n\nReasons for score: \n\nUnlike many existing GNNs which focus only on the representational role, the proposed CopulaGNN can be used for the modelling of not only representational information but also correlational information. My concern is that, although various families of copulas exist, it is not clear why the authors consider only the Gaussian copula. This is not consistent considering the fact that multiple types of (marginal) outcome variables are discussed in the paper.\n\n##########################################################################\n\nPros: \n\n(1) Many existing GNNs focus only on the representational role. Some methods are available for modelling the dependence of variables (or correlational information). The proposed method is nice because, unlike most existing methods, the CopulaGNN enables us to model both the representational and correlational information. This is due to the advantage of copula modelling in which the joint distribution of node outcomes can be decomposed into the product of marginal densities and a copula density.\n\n(2) Copula is a probability distribution which can be used for flexible modelling. Because of this benefit, the proposed CopulaGNN can also be used as a flexible model.\n\n(3) The experiments consider both discrete and continuous outcome variables both of which potentially appear in practice.\n \n##########################################################################\n\nCons: \n\n(1) Many families of copulas have been proposed in the literature. However the paper considers only the Gaussian copula. This is not a consistent approach because the authors consider multiple types of (marginal) outcome variables in the paper.\n\n(2) Considering that the GNN has a graph structure, one possible choice is to use the vine copula instead of the Gaussian copula as a copula for the GNN. Since the vine copula is a multivariate copula with graphical structure, this copula might match well with the GNN in terms of theoretical and interpretational aspects.\n\n(3) It is known that the learning of the copula is not unique if outcome variables are discrete. Does this fact cause any problem in the experiments in Section 5.3?\n\n##########################################################################\n\nQuestions during rebuttal period: \n\nPlease address and clarify the cons above.\n\n#########################################################################\n\nTypos: \n\n(1) p.4, Section 3.3, l.1: Since $n$ is assumed to be greater than $m$ in Section 3.1, it does make sense to assume $n=300$ and $m=5000$ here.\n\n(2) p.6, l.13 up: Equation 1 -> Equation (1)\n\n\n---\n\n### Updates:\n\nThanks for the authors' response. My major concerns were related to the comments (1) and (2) in Cons, but it is now clear why the authors consider only the Gaussian copula in the paper. My other concerns are addressed, too. I upgraded my rating after reading the authors' response.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Three referees support accept and one indicates reject. The issues pointed out by the reviewer who proposed rejection should be properly reflected in the final version.

First, regarding the synthetic experiment that illustrates the shortcomings of the existing GNN models, three reviewers, including myself, judged quite interesting. However, note the opinion of one reviewer that it is more appropriate to separate the influence of feature x and graph structure in the label generation method and each independently contribute to label generation. This part should be more justified in the final version.

In addition, it was pointed out that the expressive power of the model may be limited according to the parameterization type of the precision matrix, and there is a limitation that there may be a disadvantage in inference because it is a copula-based probabilistic model. I think this characteristic is actually a fundamental limitation of the proposed method. However, three reviewers, including myself, thought that it was an interesting framework as a role that can complement the message passing architecture, and decided that the possibility of the proposed method was worth publishing. However, in order to reinforce this argument a little more, it would be better if the final version verifies it with more diverse GNN architectures and datasets.
Decision: Accept (Poster)