Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The authors propose to extend Graph Nets (GNs) by allowing edge-bound uncertainty and edge-level conditioning using a Variational Bayes formulation.   
main_review: The paper is clear although (necessarily) not self-contained and it therefore becomes at times hard to follow for someone who is not deep into the subject matter.  The authors should perhaps try to better explain the flexibility and the generalisation capacity of the model proposed: would the model break or suffer catastrophic failures if the nodes are not placed in a quasi regular lattice? The reader would assume that as the turbines placement becomes more irregular, it becomes more difficult to leverage and transfer information from various parts of the network, but does the edge modelling give more robustness in these cases? Quantifying these notions via empirical experimentation would  improve the paper impact.  
Some empirical comparison with Gaussian Processes for the turbine application case is missing.  
Further improvements to the paper could include a more detailed discussion on the implications and limitations of factorizing the graph priors by the edges, nodes and global variables separately.  
Finally, perhaps a more representative artificial case could be devised to showcase the improved capacity to model relational information since in the proposed example the edges simply encode the distance between nodes and this is a type of information that can be readily derived given that the node attributes include the node coordinates, i.e. it is not something that is accessible only through modelling an relational process that is independent from the node process.  

###After authors' reply
Many thanks to the authors for the effort in replying all our concerns. After reviewing the comments raised by the other reviewers I find myself sharing several of their concerns, especially the need of a much improved clarity in explaining the relationship of their proposal to Neural Processes. I maintain that for non-expert readers it would be beneficial to be presented with intuitive artificial cases where the relational nature of the objective is clear and unquestionable.   

limitations_and_societal_impact: The authors have very briefly addressed the limitations and potential negative societal impact of their work.
review_title: null

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: This paper introduces Relational VAE, which is an extension of NPs for graph-structured data. However, the selling point of the paper is not that much clear.  
main_review: The paper is hard to follow, motivation is not very clear, and seems to have some notational mistakes. I had to read it multiple times and still do not know what exactly the authors plan to do with their model. 

I also believe the authors change some ML notations which are not necessary and sometimes are incorrect, starting from the title. For example, they mentioned their model as Relation VAE, however, they really proposed a new version of NP for graph-structured input data. Even from the lens of VAE, the relation in this model is the input graphs. As stated in lines 117-120, the authors stated " Inferring graph connectivity or generating graphs, however, falls out of the scope of this work. In RVAE the focus is generative modeling of graph structured data with an apriori known connectivity, with [...]." Or they clearly could use index $c$ instead of $h$ to refer to the context/reference points.  The article has the unfortunate tendency to contains a few unproven (or wrong) claims. For instance, "In this work, the above mentioned approaches, are generalized and unified in the proposed Relational Variational Autoencoder (RVAE) model", which would deserve some empirical/theoretical evidence. For example, the goal of NRI, is inferring the relation between objects in dynamic systems, where the input might not be structured. I cannot see how the proposed model is extending this. 

My main concern with this paper is novelty: the method seems to be a straightforward application of NP/CVAE to a different data domain on which variational auto-encoders are being used. Comparing to NP, this paper only considered the observed samples are graphs. So, they only need to change the architectures to GNNs. For example, Equation (4) is similar to NP equation in [ Garnelo et al, NeurIPS 2018].  

Concerning the experiments, I am not an expert in wind farm monitoring data, but I believe that as an ML conference, the authors do not clearly define and motivate the problem for a reader. Besides that, they could compare with some other existing baselines or include some sort of ablation study. 

Overall, I think that this paper would need to be extremely solid on the experimental side with potentially further experiments, e.g. on node classification and more detailed analysis with respect to label rates of nodes, robustness to missing edges etc, in order to make up for the somewhat limited contributions in terms of novelty. Apart from that,  plotting $G_z$, showing the benefit of uncertainty modeling in the proposed method using pavpu or etc. would be helpful. For now, I cannot recommend this paper for acceptance at NeurIPS.


There are multiple notational mistakes. 

1) = in equation 10 is missed in the second line
2) I think the ; in $p_\theta(G_z; G_h^{(i)})$ in equation 4 is not correct
3) I believe an expectation is missed in Eq 4 
4) writing a loss for each sample in Eq 4 is not meaningful, while $G_z$ does not have any index. 

If the model is Bayesian Graph modeling, the authors need to discuss and compare it with other Bayesian Graph modelings as well. 
[1] Bayesian Graph Neural Networks with Adaptive Connection Sampling, ICML 2020
[2] Bayesian Graph Convolutional Neural Networks for Semi-Supervised Classification, AAAI 2019
[...] There are multiple other related papers

Equation 6 assumes the normal distribution for the relations. This might lead to a very dense graph. 

I cannot follow lines 91 - 95 in that place. You might want to put them in the discussion or related works. 

Validation could be used for early stopping. It is not clear to me how you used test data for this.

It is not clear how the authors change the name of the model from RVAE to CRVAE and what are the differences.


limitations_and_societal_impact: The complexity of the model and dealing with large graphs can also be considered as the limitation of the model.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper proposes a generalized formulation of a probabilistic model for graph-structured data that incorporates graph networks (GN) (or rather, provides a probabilistic extension thereof). The authors show that the proposed framework is general enough to subsume existing approaches such as Neural Relational Inference (NRI) and Neural Processes (NP). The method is empirically validated on simulated and real-world data, with emphasis placed on the modelling of wind farm data.
main_review: This paper is generally well-written and reasonably easy to follow. There are a few grammatical mistakes that commonly occur throughout the paper; a non-exhaustive list is provided in the Misc Issues section of this review. I checked the details of the mathematical equations and was convinced that the proposed framework is technically sound.  

However, it was difficult to assess the significance and originality of this paper's main contribution, and I was not totally convinced. The proposed framework is indeed quite general and shown to include existing methods such Neural Relational Inference (NRI) and Neural Processes (NP) as special cases. On one hand, generalizations such as this are often important contributions since by providing a framework under which to unify existing approaches, they can help provide a better understanding of the existing approaches, in particular how and why they work, and clarify the conditions and assumptions under which they are closely related. On the other hand, there is a fine line between generality and vagueness, the latter case being that something is so general it ceases to be particularly insightful or interesting.

In Section 2 "Methods", there is nothing particularly novel and original. The framework formalizes the various components of a graph -- the nodes, edges, and global attributes, and considers their various taxonomies as deterministic or stochastic, observed or hidden, dynamic or static. A subset of the observed components makes up the conditioning or context variables and another makes up the state variables. A graph-structured latent variable consisting of vertices, edges, and global variables is assumed. A joint distribution over these variables is then specified, which factorizes into distributions over the vertices, edges, and global variables, respectively. All of this is straightforward and fundamental to probabilistic modeling.

> Sidebar question: Eq (3) what specifically are these prior distributions? I couldn't find these specified anywhere. 

Next, the procedure for approximate inference over the graph-structured latent variable is described. Again, this is just a description of the standard application of the principles of variational inference using amortized inference with a Graph Network, with the addition of scaling the KL divergence terms using a $\beta$ hyperparameter as in $\beta$-VAEs. 

> Sidebar comment: line 76-77 "A GN is proposed for inferring the parameters." - in more precise terms, you're *amortizing* the inference on these parameters using GNs. Consider perhaps invoking this term to make this more explicit.

Regarding the experimental results, there is a concern that the method might be overfitted to wind farm data, and might fail to generalize well beyond this context. On the one hand, I was pleased to see the method applied to a real-world problem, particularly one that has a high potential for positive societal impact (contributing to better clean energy solutions). On the other hand, as a methodological contribution, I would have liked to see results on a more diverse range of problems, both real and synthetic. The other issue is the lack of comparisons against baselines. On the real wind farm dataset, the CVAE is the only baseline considered. On the 1D regression dataset, NP is the only baseline considered. 

In summary, I find the contribution to be fairly significant but somewhat incremental. In particular, the originality and novelty are somewhat limited. The paper is communicated clearly and of reasonably high quality. The experiments are mostly well-executed but may be over-specialized to the domain of wind farm modeling.

### Misc Issues

- line 212: "The CRVAE models are compared to a two-layer MLP-based CVAE trained with the arbitrary conditioning objective [30]" - this is the first occurrence of the acronym "CVAE". From the context and subsequent citation, it's possible to guess that it refers to the conditional variational autoencoder, but the reader shouldn't need to guess.
- Please consider rasterizing the figures on Page 8 (the culprit is most likely Figure 3) which causes my PDF viewer to lag on a reasonably powerful computer.
- line 309, reference: please sort your references alphabetically by the first author's last name
- line 328, citation [7]: "bariational bayes"

The following sentences all have extraneous commas (probably non-exhaustive):

- line 60: "The full graph state, is denoted by"
- line 114: "in this work, the above mentioned approaches, are generalized and unified"
- line 128: "the input-output observation cases D, are split as"
- line 146: "The aforementioned models, feature a global latent variable"
- line 209: "for using composite aggregators, is also due to the physics of the problem" 


limitations_and_societal_impact: The authors recognize the potential negative societal impacts of their work, in particular, its potential malicious uses in de-anonymization of social networks and exploiting vulnerabilities in transportation networks.
review_title: null

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: This paper proposes a framework for a general probabilistic generative model for attributed graphs.
The proposed model assumes that the nodes, edges, or global attributes of a graph have many different properties, which can be deterministic or stochastic, and observable or unobservable, and so on.
The proposed model formulates a stochastic dependency between the conditioning graph variable G_h and the state graph variable G_s similar to beta-VAE, assuming the existence of several latent variables in addition. 
The paper shows that the proposed model encompasses many existing Bayesian graph models and evaluate its performance using wind farm dataset and artificial simulation dataset with ELBO or test data log-likelihood.
main_review: ## Originality: 
In Sec. 2, the proposed model is explained with many equations. As far as I understood, the variables of the graph data are separated into conditioning and state, and the probabilistic dependency between these two groups of variables and latent variables is assumed to have a dependency structure similar to that of beta-VAE. 
As far as I know, no VAE-like graph model has ever been proposed to model the process of graph data generation in such a general way, so there is a  concrete novelty at this point.

However,  it is unclear that what kind of benefits we can expect from the proposed model. This is because there are no observations or assumptions about the data to be modeled. In turn, there are no new ideas or inspirations to explain the characteristics of the data efficiently in the graphical models.
I felt that there is no strong answer given to the question, "We have created a new model, but is it beneficial?" throughout the paper. 

## Quality

I think there is a room for improving this manuscript as I write in the following  "clarity" and "significance" issues. 

 ## Clarity 

In the introduction, the motivation for this research is not explained intensively. Therefore, it is difficult to answer "Why is this research necessary? What is the value of this research?'' 
Therefore, readers may find it difficult to understand these importantquestions. This problem is not resolved for me even after reading the paper to the end.

In L17-18, it is written ''the ubiquity of noisy structured data and systems with stochastic or partially observable
 interactions of industrial relevance (e.g. wind farms and urban transportation networks)''. 
For most readers, this sentence alone does not explain the motivation of the study at all. This is because the readers are unfamiliar with these examples and have no knowledge of what they want to do with these data. 

(A) what kind of uncertantiies exist in these data
(B) what kind of problems caused by the uncertainties
and (C) why existing GNNs and graph generation models are insufficient to solve them
Given concrete explanations of all A, B, and C, then, I think we can finally reach the understanding that a general form of a probabilistic graph generation model is necessary.

The same difficulty in understanding due to lack of explanation affects each technical discussion in Sec 2.
- L37-65 Why did you make the settings for each attributes so general (stochastic/deterministic, observable/non-observable, static/dynamic)? 
In many DNN/GNN researches, it is of interest to choose appropriate inductive biases into the model to perform good at chosen tasks or goals. The formulation of Sec. 2 seems to go in the opposite direction, and I don't understand the intention or the motivations for a general formulation like this. 

In Sec. 4, the examination of specific data begins.  Unfortunately, the explanation is not concise and easy to understand. 
Since the paper introduces many symbols and conditional probability distributions in Sec. 2, I would like to describe the data with these symbols or equations:  for example, which symbol corresponds to an element of wind fard data, and which term of the conditional probability distribution will be important. 
Clearer description of the data wouldl help to convey the appropriateness of choosing counterparts (CVAE and NP) for experiments. In the current manuscript, it is very difficult to understand the appropriateness of the choices. 

## Significance:
The lack of explanation of motivation described in the Quality section makes it difficult for me to evaluate the significance of this paper. If we do not fully understand the reason ''Why is this study necessary?'', it is difficult to evaluate the significance of the content of the paper. 

I think that the current experiment is insufficient to experimentally demonstrate the usefulness of the proposed RVAE. 
(I) The proposed RVAE model is quite broad and general in its formulation.
(ii) The effective latent variables and conditioning variables are expected to be different from dataset to dataset.
Given (i) and (ii), it is necessary to conduct a thorough ablation study on many different datasets to evaluate the proposed method accurately.
By testing such experiments on several real-world datasets with different characteristics, we can see that RVAE is highly effective on any dataset, thanks to the general formulation.

Thus, in my opinion, the paper needs more experiments in various datasets. Ideally, the counterpart models should variy across dataset to dataset reflecting the different nature of datasets and tasks. 

Figs. 3, 4 I think that it is insufficient to convince the readers of the usefulness of the proposed method even qualitatively unless the visualization of the estimation results by other methods is included in addition to the estimation results by RVAE.

(+) a broad and general formulation for VAE-like graph generation models
(--) motivations are not explained well
(-) description of datasets can be improved clearer and more informative way
(--) experimental validations are insufficient to show the superiority of a proposed general framework

## After author feedback

Thank you authors for the detailed clarifications. 
Your feedbacks resolves several of my misunderstandings. Thus I raise my scores +1. 

At the same time, the feedbacks reveal the manuscript lacks these important explanations. 
Also I still believe the experimental validations are not sufficient for convincing non-expert readers like me. 
Therefore I cannot vote for acceptance before confirming the revised manuscript fulfills these requirements. 


limitations_and_societal_impact: No specific comments
review_title: null

Meta-review: This submission considers a probabilistic model for graph-structured data that incorporates graph networks. It also attempts to connect/extend CVAEs, NPs for graph data. However, the reviewers remained unconvinced that the generalisation provides theoretical insights or practical benefits. In addition, all reviewers agreed the exposition of the work is hard to follow and the experiment section requires more work. Thus, the paper in its current form is not ready to be accepted.
Decision: Reject