Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: ##########################################################################

Summary:

The paper presents a new model based on the Graphical Neural Network (GNN). The proposed model adopts probability distributions called copulas and is called the Copula Graphical Neural Network (CopulaGNN). Two parametrizations of the CopulaGNN are given, and the learning of the proposed model is discussed. Experiments suggest that the CopulaGNN outperforms existing GNNs and MLP in almost all setups.

 
##########################################################################

Reasons for score: 

Unlike many existing GNNs which focus only on the representational role, the proposed CopulaGNN can be used for the modelling of not only representational information but also correlational information. My concern is that, although various families of copulas exist, it is not clear why the authors consider only the Gaussian copula. This is not consistent considering the fact that multiple types of (marginal) outcome variables are discussed in the paper.

##########################################################################

Pros: 

(1) Many existing GNNs focus only on the representational role. Some methods are available for modelling the dependence of variables (or correlational information). The proposed method is nice because, unlike most existing methods, the CopulaGNN enables us to model both the representational and correlational information. This is due to the advantage of copula modelling in which the joint distribution of node outcomes can be decomposed into the product of marginal densities and a copula density.

(2) Copula is a probability distribution which can be used for flexible modelling. Because of this benefit, the proposed CopulaGNN can also be used as a flexible model.

(3) The experiments consider both discrete and continuous outcome variables both of which potentially appear in practice.
 
##########################################################################

Cons: 

(1) Many families of copulas have been proposed in the literature. However the paper considers only the Gaussian copula. This is not a consistent approach because the authors consider multiple types of (marginal) outcome variables in the paper.

(2) Considering that the GNN has a graph structure, one possible choice is to use the vine copula instead of the Gaussian copula as a copula for the GNN. Since the vine copula is a multivariate copula with graphical structure, this copula might match well with the GNN in terms of theoretical and interpretational aspects.

(3) It is known that the learning of the copula is not unique if outcome variables are discrete. Does this fact cause any problem in the experiments in Section 5.3?

##########################################################################

Questions during rebuttal period: 

Please address and clarify the cons above.

#########################################################################

Typos: 

(1) p.4, Section 3.3, l.1: Since $n$ is assumed to be greater than $m$ in Section 3.1, it does make sense to assume $n=300$ and $m=5000$ here.

(2) p.6, l.13 up: Equation 1 -> Equation (1)


---

### Updates:

Thanks for the authors' response. My major concerns were related to the comments (1) and (2) in Cons, but it is now clear why the authors consider only the Gaussian copula in the paper. My other concerns are addressed, too. I upgraded my rating after reading the authors' response.
review_title: Graphical Neural Network using copulas

final_score: rating: 7: Good paper, accept
review_content: This paper proposes to use Copula function to model the correlational role in the node-level prediction tasks. From my own perspective, I quite enjoy this work. In particular, it uses a simple simulation example to uncover the importance of correlational role, i.e., most of the GNNs approaches might perform worse than simple MLPs as they are incapable of using the correlation graph information. This observation is simple, however, would provide deep insights into the current GNNs literature. 

[Introduction & Related Work]
The authors clearly state their motivation and precisely introduce the mentioned works. Readers would get a clear understanding on the background information. 

[Simulation the two roles]
The authors use simple simulation examples to show the importance of correlation graph information. 

[Copula Graph Neural Networks]
The authors introduce the detail architecture design for the model. Although the design looks reasonable, I have several questions as follows:

1, the authors mainly used the Gaussian Copula as the correlation modeling choice and incorporate the graph structure in the precision matrix. I can totally understand that this setting can reduce the number of parameters. What if the precision matrix is without the graph structure and is simply placed with sparse priors?  

2, following the above question, it looks like Gaussian Copula is the only suitable choice for Copula functions. How can one use other copula functions, as we cannot incorporate the graph structure and the other copula functions needs complex structures (e.g. C-vines or D-vines) to model the high dimensional data?

3, the two-parameter and regression parameterisation seems quite complicate in fitting the precision matrix. Thus, can these parameterisation method also be used for other copula functions?

[Experiments]
The experimental part can be sufficient to verify the effectiveness of the authors' proposed model. 
review_title: Deep insights into the Graph Neural Networks

final_score: rating: 7: Good paper, accept
review_content: Summary:

This paper proposes a novel framework for the use of graph neural networks for node-level prediction when these predictions have correlations across nodes that cannot be entirely explained by the node features. In a nutshell, the idea is to propose a probabilistic model for the joint node-level output whose mean can be parametrized as a GNN and whose correlation is determined by a precision matrix with the same support as the graph at hand.

Overall:

My evaluation is positive. Although the presentation could be streamlined and made clearer (see comments below), the explicit recognition for the need of a learnable module for the correlation that is separated from the GNN seems to be a fundamental insight that can shed light on multiple existing methods.

Pros:

1 - The synthetic example in Section 3.2 nicely illustrates the shortcoming of some existing methods.

2 - The generic framework is applicable on top of existing GNNs architectures. As such, it can be applicable to a wide range of settings

Cons:

1 - The presentation is hard to follow at some points. For example, the “Model Inference” paragraph is key for the implementation of the proposed method. However, it is presented in only a handful of lines without any discussion on the different steps. Maybe even an Algorithm environment would be preferred for this.

2 - It is unclear to me the true value in doing the presentation around copulas. Section 4.1 is too short in the sense that it would be difficult for someone that does not know what a copula is beforehand to actually pick up the necessary notions from there. The key idea, formally stated in Section 4.2, can be explained without resorting to copulas, simply in terms of marginal distributions and covariances. What is exactly the value of introducing the copula formalism? Please make more emphasis on this.

++++++++++++++++++++++++++++++++++++++++++++

Edit after author response: We thank the authors for their response to my concerns and those of other reviewers.
review_title: A useful perspective on correlation on GNNs

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper proposes to distinguish the representational and correlational roles of the graph representation of data. The authors proposes CopulaGNN that decomposes the joint distribution of node labels into the product of marginal densities and a copula density, which models the representational information and correlational information separately. The experimental results demonstrate the effectiveness of the proposed method. The idea of introducing copula into GNN is novel and interesting, and this paper is basically well written and well organized. However, I have the following concerns:

1. Characterizing representational role and correlational role is the key motivation of the proposed method, but I'm not quite clear about their definitions. As far as I understand, the representational role means graph homophily and the correlational role means graph topology. But homophily and topology are not two disjoint concepts since homophily says that two connected nodes are likely to have the same feature, which is exactly based on (local) topology. I think a better way to decompose the model is to say that, the label of a node depends on not only its feature but also the graph topology.

2. The authors decompose the joint distribution of node labels into marginal distributions of node labels as well as a copula density. The marginal distributions characterize the representational information (homophily) and the copula density characterizes the correlational information (topology). Here the marginal distribution f_i(y_i; X, G) is not only dependent on its own feature but only the feature of its neighbors, which is not reasonable since the marginal distribution should only characterize the node itself. This is also connected to the above argument that the dependence of node label can be decomposed into its feature and graph topology. Therefore, it seems more reasonable to write eq. 2 as f(y; X, G) = c(u_1, ..., u_n; G) \prod f_i (y_i; X). Only in this way we can say that we "decompose" the joint distribution of node labels. The original eq.2 does not decompose anything since X and G appear in both copula density and marginal distributions. I'm not saying that the current eq.2 do not work, but it "seems" more elegant to do as what I said above. Otherwise the original eq.2 is just like fitting the joint distribution with two complete modules then simply combining them together.

3. Another issue that confuses me is that, the studied problem is formulated as learning joint distribution of node labels, but we actually only have one observed sample, i.e., the whole graph. Although the authors have tried their best to reduce the parameters of the model, it is still prone to overfitting if you maximize the likelihood only on one sample.

4. Modeling the problem as learning joint distributions will cause another issue that doing inference is too time confusing, since we need to infer labels of all nodes as the same time even if we only need the label of one node. In addition, as the authors said, multiple sampling rounds may be required to calculate the average.

review_title: An interesting work, but the way of decomposing joint distribution may not be elegant

Meta-review: Three referees support accept and one indicates reject. The issues pointed out by the reviewer who proposed rejection should be properly reflected in the final version.

First, regarding the synthetic experiment that illustrates the shortcomings of the existing GNN models, three reviewers, including myself, judged quite interesting. However, note the opinion of one reviewer that it is more appropriate to separate the influence of feature x and graph structure in the label generation method and each independently contribute to label generation. This part should be more justified in the final version.

In addition, it was pointed out that the expressive power of the model may be limited according to the parameterization type of the precision matrix, and there is a limitation that there may be a disadvantage in inference because it is a copula-based probabilistic model. I think this characteristic is actually a fundamental limitation of the proposed method. However, three reviewers, including myself, thought that it was an interesting framework as a role that can complement the message passing architecture, and decided that the possibility of the proposed method was worth publishing. However, in order to reinforce this argument a little more, it would be better if the final version verifies it with more diverse GNN architectures and datasets.
Decision: Accept (Poster)