Relational VAE: A Continuous Latent Variable Model for Graph Structured Data
Authors: Charilaos Mylonas
Imad Abdallah
Eleni Chatzi
Venue: NeurIPS 2021 Submitted (2021)
Keywords: Graph Neural Networks
Variational Inference
Stochastic Gradient Variational Bayes
Wind Farms
Neural Processes

Abstract: Graph Networks (GNs) enable the fusion of prior knowledge and relational reasoning with flexible function approximations. In this work, a general GN-based model is proposed which takes full advantage of the relational modeling capabilities of GNs and extends these to probabilistic modeling with Variational Bayes (VB). To that end, we combine complementary pre-existing approaches on VB for graph data and propose an approach that relies on graph-structured latent and conditioning variables. It is demonstrated that Neural Processes can also be viewed through the lens of the proposed model. We show applications on the problem of structured probability density modeling for simulated and real wind farm monitoring data, as well as on the meta-learning of simulated Gaussian Process data. We release the source code, along with the simulated datasets.

In the context considered herein, modeled entities (nodes) and modeled relations (edges) may feature 20 a state, which may not be fully observed and/or stochastic. The same may also holds for global Relational V: for Graph Structured Data
A Continuous Latent Variable Model
Anonymous Author(s)
Affiliation Address email
Abstract
Graph Networks (GNs) enable the fusion of prior knowledge and relational reason- ing with flexible function approximations. is proposed which takes full advantage of the relational modeling capabilities of GNs and extends these to probabilistic modeling with Variational Bayes (VB). To that end, combine complementary pre-existing approaches VB for graph data and propose an approach that relies on graph-structured latent and conditioning variables. It is demonstrated that Neural Processes also be viewed through the lens of the proposed model. We show applications on the problem of structured probability density modeling for simulated and real wind farm monitoring data, well as on the meta-learning of simulated Gaussian Process data. We release the source code, along with the simulated datasets.
2
this work, a general GN-based model
4
10
11
Introduction
12
Graph Neural Networks (GNNs) have been established effective tool for representation learning on graph structured data. Graph structured data are routinely employed to represent entities and relations among them. The present work focuses in representation uncertainty and generative
14
15
16 modeling for attributed directed graph data with continuous attributes. The initiating motivation for
17 this work is the ubiquity of noisy structured data and systems with stochastic or partially observable interactions of industrial relevance (e.g. wind farms and urban transportation networks).
18
21 (graph) attributes. At the same time, nodes and relations may possess dynamic partially observed
22 state, which we may infer directly from data. Both the node states and edge states are not fully
23 observed and non-deterministic, which amply motivates probabilistic extensions of graph networks.
24 In essence, this work proposes a method that 1) exploits the relational structure of data and 2) allows
25 for learning flexible distributions over entity and relation attributes. Several partially overlapping approaches for this problem exist. A short review of such prior approaches is offered in section 3 allow for stronger generalization in novel settings. The main contribution of this work is to propose approach to transfer the potent combinatorial generalization and modeling capabilities of GNNs the problem of modeling conditional distributions of structured data.
26
27
Modeling entities and relations has been shown empirically
28
3
29
2
Methods
31
Attributed graphs Following (2). global attribute augmented graphs are denoted by G (V,E, where V (vi)==::- with V; E Rd denoting the nodes (vertices) of the graph,
38
Submitted to 35th Conference on Neural Information Processing Systems (NeurIPS 2021). Do not distribute.

---

34 8: : {(ek: Sk, Sk, k=) designating the set of edges, with edge attributes E R sk, E N'
35 denote the head (sender) and tail (receiver) nodes of the modeled relation, while u E Rd is the global attribute.
36
37
Graph Networks (GN) (or GraphNets) are composite functions that receive and return attributed graphs. The full GN block consists of edge update, a node update and a global update block. Each block contains a corresponding function 6º, Φ" Φ". The edge update function uses edge, node and global data. The edge block is followed by aggregation step p pf where edge messages accumulated according to permutation invariant function, e.g. a mean function. The node update uses (optionally) the global state, the aggregated edge state and the current node state. Finally, a global block aggregates with permutation invariant functions the edge and node properties (p--0, p²-"), and optionally uses the global state for updating the global variable state. Different parts of the full GN computation may be omitted. Several Graph Neural Network architectures can be cast as special
39
40
41
42
43
44
45
cases of GNs by omitting certain features or by special choices of the different functions involved 21. In what follows, when referring to GNNs, the most general and expressive GN layer is implied 47
46
48
except otherwise specified.
49
In the proposed model, entities (nodes), relations (edges) and global attributes contain both determin- istic and stochastic variables. These variables in turn, may be observable not directly observable. Both observable and unobservable attributes may be deterministic or stochastic (static or evolving). In what follows, part of the observable quantities is referred to as conditioning or confext. The node, edge and global observable quantities are denoted as v², e u² where h signifies that a variable corresponds conditioning. Conditioning variables may either correspond to conditioning with known dynamic quantities or static quantities. Common instantiations of such conditioning are positional encoding for vertices, relative position for edges between vertices and time of day as a global variable. The node, edge and global variables that correspond to the rest of the states
51
52
53
54
55
56
57
58 (stochastic, evolving, unobserved) are denoted by v" ed, u In essence, the conditioning attributes can be used to create conditioning graph variable GA, = (Vn. En, uy) and state graph variable
59
60 G2 = (Vz, Ez, uz). The full graph state, is denoted by Ga (V₂ Vn: &g U Er, u, u,) where U denotes set union. Since part of the node, edge and global attributes may be stochastic, graph structured latent variable G= = (Vz, Ez. uz) is assumed. The graph structure may also be determined
61
62
through the edge variables as in (6) but we restrict our model to pre-determined graph structure
63
64 in this work. The following model proposed for the joint distribution of the graph structured observations
P(Gai Gn) / p(G G:)((G:;;)) Gn)dG2
(1)
66 where p(G = Vr (EE; E Ex)p(u,; is the distribution of the latent variables given Gh- A prior distribution conditioned on G2, is assumed for the latent variable, which is further factorized
68 along each edge and node latent separately, i.e.,
p(G:; Gr) = pp000 (V.i
(V=: (E); &n (t)
(uns uz)
N
N
II p(vi:v!)) II p(e e u(u";u²).
(3)
69 An approximate posterior (i.e., recognition model) is assumed for G2 as 4o
G1) together
70 with a generative model for G.. po Gz] Gn). In correspondence with the Variational Autoen-
71 coder (VAE) 77). we seek to learn the generative model parameters O and inference model param-
72 eters & simultaneously. Assuming independent identically distributed (i.i.d.) graph observations
73 (G()) ....G()). the Evidence Lower Bound (ELBO) for the marginal log- lig-liielihood reads
C(0, Φi G.), G(a)) =E (GG cc( [log po po G Ge: (00
D dl pe (Ge; ((0))
(4)
74. We seek to perform fast and scalable approximate inference over the G2 graph variable and at the
75 same time take advantage of the relational structure in the data. A particularly convenient choice for parametrizing G is to assume parametric distribution over edges, nodes and globals. A GN is

---

77
proposed for inferring the parameters. For a graph structured observation observation Gz, we write
(4) (G.IG: Nifo (Ga; - (G2; (G,:
V" 2
(5)
E 0 (8) (G -N) (Gz; Gh), S4 (Gz:
(6)
(1)) (G. -N (Gzi G Jux GG))
Ug ds
The functions f and f. implemented by GN allow for taking into account in general 79 manner relational information while inferring over V2, Eg and ug. In practice a shared, single GN,
80 Las (-) is used. The parametrization for vertices, edges and global variables are the corresponding
81 states of the GN at the final message passing step. In a similar manner, a GN generator network,
82 Ire (-), is used for pa. Since the prior and posterior are factorized over nodes, edges and the global
83 variable of each graph datapoint, the ELBO is split accordingly as
[(O, Φ: G ) =E vo [lo po GG
ByDxL (
(G
(
33 (G.
(G.: G ((
-B. (G2
Ilre (Gz; (G G(
(8)
where Sy. .gg Bu can be used for controlling disentanglement as in VAE or the rute-distottion 84
85
characteristics of the model [9] or for preventing posterior collapse and aiding training through Kl--annelling [10 [T] In a similar manner to VAEs, the approach to representing distributions over graph data with a distribution that factorizes over V. & and u allows for defining alternative evidence lower bounds for variational Bayes. Note that the distribution does not need to be factorized along the elements of the latent vector. This allows stright-forwant extensions using more flexible distributions
86
87
86
90
[2]. A generative model based on normalizing flows that uses shift-scale transformations [13] has
91
already been proposed in [4] for graph generation. The Relational VAE (RVAE) model proposed can be extended as a hierarchical VAE [15] yielding a model akin to Doubly Stochastic Variational Neural Process (DSNPV) [16), which uses global and node variables. Finally, Neural Processes
92
94
[8] (NP) and other graph encoder-decoded models [6 19 21 22 are closely related to the
proposed model.
3 Related work
96
37
GNN Encoder-decoder models In Neural Relational Inference (NRI) G discrete edge latent vari- ables are inferred from node representations and a re-parametrized Gumbel Softmar distribution is used[23] 224. A coarse representation of the computational graphs of NRI, NPs and the RVAE shown in Figure I In [19] graphs are modeled from global continuous latent variables, which are subsequently used for graph generation through an adjacency matrix. In Graph VAE [20) the global variable together with graph-structure
98
99
100
101
102
conditioning variable is used for generation. In Graphite [21] a latent variable for each node is inferred from the encoder, while the edge variables (i.e.. symmetric adjacency matrix) is inferred through efficient iterated message passing. Similarly, uses separate latent variable for every node and graph convolutional encoder. Several of the aforementioned works take advantage of recent advances in low-variance gradient estimates for distributions over latent variables, as in Variational Autoencoders (VAFs) via the reparametrization trick [7]. The overlapping traits of the aforementioned are the treatment of edge, node and global variables. In Table I l summary of the relational modeling capabilities of various graph encoder-decoder models is offered. Note that the table highlights only the relevant parts to this work together with several important and influential design choices for graph representa-
103
104
105
the VariationalGAEg2
106
107
108
110
112 tion learning were not touched upon. For instance, the graph convolutional models of some of the aforementioned works offer the important advantage of scalability and small computation cost.
114 In this work, the above mentioned approaches, are generalized and unified in the proposed Relational
115 Variational Autoencoder (RVAE) model. Note that it is not difficult to yield explicit graph connectivity
116 in RVAE as in NRI 6 since the type and existence of connection can be seen as a categorical
variable. See also Figure I (b), where a sketch of NRI is offered. Inferring graph connectivity or
3

---

118 generating graphs, however, falls out of the scope of this work. In RVAE the focus is generative 119 modeling of graph structured data with an apriori known connectivity, with attributed nodes and edges, which optionally may include glohal attribute that influences both entities and relations.
Table 1: Features of different related Bayesian graph network encoder-decoder models (see also
Figure Tp. For the NP models that contain latent variable, it is straightforward to combine deterministic global encoder for the context inputs at test time [26]. The attributes with subscript denote that the model performs optimization using an ELBO objective. The attributes with superscript h denote whether the models may facilitate deterministic conditioning for the corresponding graph attribute at test time.
Latent
Conditioning
Name
V.
E
11
Vh.
Eh
11
Architecture notes
CNP
17
DeepSet encoder, GN node block Attention encoder/decoder
AuCNP
27
/
Decoder edge cond. through cross-attention
ConvCNP
28
V
SetConv encoder
NP
18
DeepSet encoder
GAE
20
SS V
Graph conv. encoder
GariationalGAE
22
Graph convolutions
Graphite
21
/
Iterative decoder
NRI
66
MP encoder/decoder
MPNP
29
MP encoder/decoder
DSVNP
16
V p(V*ju", V", ydh)) S
RVAE (this work)
MP encoder/decoder
121 Neural processes In Neural Processes [T, we consider set of mappings F: + y where X' {x,}, F₁ RN= .y: {u, 34 E RN,, A particular draw of function f ~ F, is modeled as f (xx)) = go (04, 2) where ~ p(2) is a high dimensional random vector (c.g. a standard normal) and ge neural network and 0 denotes the parameters of g. Given set of nm input-output observations D {(11:.m )... from m different realizations of f (potentially different in number), we want to learn distribution over p(2 (D). Under the NP approximation, assuming observation noise ys ~ (gg(( 2), o²), the distribution of y is defined as
123
125
127
T
Vin.iln = p(2) II N (),
128 In practice, the input-output observation cases D, are split as DOUT DC UD" where C denotes a set of points with observations in X' and y and T denotes a set of points where we only observe
129
130
X' (i.e., the inputs). This can be cast as a conditional generative model for p(yr| (2r, DC, yc)
pe pyrlaa,, 2) 2)p(()/cc (0////..)))) where the conditioning is the fully observed context pairs. The ELBO used for optimization
132
g(2
log logrlur (ellrlrr,.co Uc) 2 Bestsppuury
log po + log q(2 DOUT)
EE
E
log po(un12, x.)] - DKz((9(21DDU) (2
(10)
133
Note that the above variational objective has an intuitive interpretation, as a reconstruction loss (first part) and Kullback-Leibler divergence between the approximate posterior distributions predicted when using both C' U T and when using only C' (the context set). In (30 a similar loss function was proposed with the motivation of training VAËs that can be used with arbitrary conditioning masks. By considering the set of observations as nodes in disconnected graph, (i.e., V: (vil(za, b
134
135
136
137
138
and training while masking the context output nodes yic, the same objective is retrieved. Therefore, following the nomenclature of (2), we can instantiate a NP from the proposed model, by using arbitrary conditioning as described in [30] a DeepSet
140
as an encoder and only a node-block as
decoder as shown in Figure

---

Conditioning
Graph
Gnuph Indap.
Graph Indep.
Graph Indep.
Indep.
Full GN
Fall GN
Conditioning
V
V2 17
(b)
(c)
Figure 1: (a) Proposed architecture with single message passing step the encoder and decoder (b) the Neural Relational Inference model of [6] (c) The Neural Process model [1]. For direct correspondence between the present work and (6) and [T8] the notations of the other works are included in the figure (e.g. p²-u in the Neural Process model).
142
The NP framework has been extended to take advantage of special inductive biases, such as the relation of observation and target nodes in Attentive Conditional Neural Processes (AttNP) 27 or the translation equivariance in Convolutional Conditional Neural Processes (ConvCNP) (28]. More recently, relational inductive biases employed in Message Passing Neural Processes (MPNP) The aforementioned models, feature a global latent variable u, which inferred from the context points and parametrizes the distribution aforementioned works target non-relational data. Nevertheless, MPMPs does not directly implement boun-bound uncertainty edge-level conditioning, which is the most pronounced difference RVAE. Similar to this work, in DSNPV [16] NP that allows for both node and global latent variables was proposed, which in addition, employs hierarchical VAE 15). The motivation of DSNPV is to include node-context information, which in the conditional RVAE is also supported by design through Vy. RVAE attempts to merge the complementary strengths of the aforementioned models in representation of uncertainty, with a focus towards modeling graph structured continuous data, Finally, contrast Functional Neural Processes [32) we do not deal with inferring graph of dependencies among latent variables, yet hierarchical RVAE adaptations may also manage such tasks.
144
146
29
functions. With the exception of MPMP, the
148
150
151
152
153
154
155
156
157
Graph Gaussian processes Sharing the motivation this work, i.e., taking advantage relational information and learning joint distributions of graph structured data, in [33 GPs were defined over graphs with undirected binary (positive or negative) edges and applied to semi-supervised learning problems. In (44 the authors applied GPs trained with variational approximations for semi-supervised
159
160
161
learning on graphs that contain non-attributed edges. In (35) GP-based approaches are fused with deep learning for learning graph (e.g. network) structured signals.
162
Results
163
164
4.1
Wind farm operational data
185
A real-worid industrial application, where relational structure is inherent in the observed data, is found in modeling of operational data of wind turbines positioned in a farm. The wind turbines (nodes) feature static variables, such as their power production characteristics and their position, as well as dynamic variables such as their current operational state. The actual operational state of a turbine is only known up to a certain precision from historical data, (i.e., Supervisory Control and Data Acquisition (SCADA) data), which usually limited to 10 minute statistics. Due to the stochasticity of the wind excitation, compounded by incomplete information due to coarse measurements, there uncertainty associated with the actual operational state of a wind turbine.
188
167
168
169
170
172
Wind turbines arranged in wind farm interact through the so-called wakes, which are travelling
5

---

174
vortices that affect the power production and vibrations of downstream turbines. The magnitude of wake effects is related to large scale turbulence (which is a global dynamic variable), wind orientation (which is a global dynamic variable), to upwind turbine nacelle orientation (which is node dynamic variable), the relative position between the two turbines (an edge static variable), the rotor diameter and the distance between the two turbines. The interaction is one-way directional but can change directionality depending on the wind orientation. The effect of wakes is stochastic due to turbulence. For robust wind power prediction, monitoring, control, and maintenance planning, we want to infer the distribution of operational characteristics of a wind farm conditioned on turbine characteristics and farm layout. Of crucial importance is the inclusion of stochastic variables in the interactions (i.e., edges) of the considered graph. Static graph edges, used as part of the graph conditioning, are constructed by considering the spatial proximity and relative position of pairs of turbines. The goal is to generalize directly to unseen farm configurations while learning directly on real condition monitoring data (zero-shot generalization) but at the same time to yield uncertainty estimates.
175
176
177
178
179
181
182
183
185
186
187
188
Graph machine learning in wind farm modeling In 36 a GNN was trained on simulated data for wind power prediction. Recently, in (37 GNNs were applied as a surrogate model to more accurate fluid dynamic simulations. With the architectural advancements proposed in this work, we extend the wind farm relational modeling literature by providing a solution for representation of uncertainty in wind turbine interactions. Morcover, we empirically show in real wind farm data that significant accuracy improvements are possible through the incorporation of the proposed relational modeling and variational Bayes approach.
189
191
192
193
RVAE deta (10 % missing!
RVAE data 120 missing)
RVAE data 150 missing)
data
1000000
00
-18
Figure 2: Imputation qualitative results for wind speed. The imputed points are marked with dark
circle on the background. The mean absolute percentage error is reported, which is computed as N (Iv?' v; D//D/ where vf is the actual value of node i, NT the number of target turbines and v; is the CRVAE prediction.
1/N" Σ Lim1(e
195
4.2
Real wind farm SCADA dataset
196
Conditional RVAE models (CRVAE) were trained with with a 80/20 train/test split on a dataset that includes 6 months of 10- average SCADA data readings. Since the goal is to compare the fitting capability of the models and not model selection, no validation set was employed. Early stopping with patience of 2500 steps was used (test set evaluation every 500 steps). The larger RVAE models that also yield the best performance had not converged at the 10th epoch. The 20% of turbine data are randomly masked during training. A batch size of 16 was used for all models. In order to make a fair comparison no regularization or KL-annealing was used. A small learning rate of
198
200
202
203
5 10-5 and the Adam optimizer (38) with default parameters was used for all the runs. The final
204
ELBOs for all models are shown in Table 2 A mean aggregation function and composite aggregation
205
function consisting of concatenation of mean,mor and min aggregators were used, Due to the
206
concatenation operation, the composite aggregators result in slightly larger networks. Aligned with recent results on GN performance when using composite aggregation functions [39 we find that networks with the mean - mar - min aggregator indeed yield better performance. The motivation, however, for using composite aggregators, also due to the physies of the problem. By using
207
208
209
210
such aggregators it is easier to discriminate the un-waked part of the farm and the waked turbines.
211
More concretely, turbines at the upstream boundary of the farm have larger power production and this directional effect can easily be masked using the mean aggregation. The CRVAE models are compared to a two-layer MLP-based CVAE trained with the arbitrary conditioning objective (30) of
212
213
varying sizes, with the largest CVAE model number of parameters corresponding to the number of
215
parameters of the best performing RCVAE. The largest CVAE model was the porst-perf of
the evaluated CVAE models.

---

217
The CVAE model with the smallest size has slightly better performance compared to the RVAE model that performs no message passing on the encoder part, and therefore ignores relational inductive biases when inferring Gz. All but one of the CRVAE models strongly outperform the CVAE models 220 by a large margin which is attributed to the effective use of relational inductive biases. To further support this claim, in the supplemental material (section A.J) gradient sensitivities are plotted and it observed that the imputation results for masked turbines depend on upstream turbines. Qualitative imputation results are shown in Figure 2
218
222
Table 2: Test set ELBO on Anholt SCADA dataset after 10 epochs. Numbers in parentheses are the standard deviations of the ELBO estimates in the test set (higher is better). The same node, edge and global latent sizes were used (N (NC=). "(comp.)" stands for the composite mean-max-min aggregator. All MLPs are 3 layer ReLU MLPs. The superscript denotes results that were not derived from early stopping.
Steps
Model
mlp units
N size
cnc.
dec.
agg.
params
ELBO
CRVAE
64
32
mean
184,717
1.96, (0.30)
64
32
mean
341,517
6.99(0.29)
64
32
2
2
mean
498,317
7.48(0.61)*
64
32
2
(comp.)
522,893
8.11(0.48)
64
32
(comp.)
679,693
(.70().53)'
CVAE
128
64
77,194
2.12(0.10)
256
64
252,554
1.17(0.16)
384
96
563,146
1.23(0.09)
224
Effect of inferring edge latents &g The introduction of continuous edge-related latent variables is overlooked in a
225
226
large part of the literature. Wake effect modeling is an appli- cation that may benefit from edge latent variables. We test
Table 3: Effect of edge latent variables.
Results based on 3 runs for each case.
227
228
the effect of edge latent variables by setting 8gg = 0 while
Case
log p [..; G.).
Range
229
still using Gn. The results of this experiment are shown
80 = 1. Be 0.
4.16
+0.43
Table 3] The inclusion of the KI. term with respect to edge latent variables seems to improve the reconstruction error achieved by the model.
1.80
+1.21
231
233
4.3
Wind farm sinulation dataset
234
The steady-state wind farm wake simulator FLORIS 44 was used. A dataset of wake effect simulations and preprocessing tools for demonstrating the wind farm modeling approach adopted
235
236
herein released part of this work. In what follows we test the generalization capabilities of trained RVAE to novel geometric configurations. A single farm configuration is used for training and another is used for testing. Both farms are simulated with random wind characteristics such 238
237
239
as direction and average wind speed. An example output from the simulation can be found in the supplemental material. The train and test farm configurations can also be found in the supplemental material.
240
241
Qualitative results The RVAE model is able capture the orientation-dependdee wake deficit for each turbine separately on the test wind farm as shown in Figure 3] Furthermore, we use a single turbine as probe and position it on regular grid while keeping turbine on a fixed position (0,0). By inspecting the wind speed predicted at the probe turbine. we can map the wake deficit in 2D behind the source turbine. This is shown Figure 4 The spatial dependence of the wake deficit
243
244
245
246
247 is also shown computed from FLORIS and the error in RVAE estimation. For distances larger than 200m the wind deficit accurately predicted. Note that this result is from a model trained on operational data from single simulated windfarm. When the turbines are very close (< 200m) the
248
249
250
wakes are not predicted correctly, but this an expected effect since the RVAE never encounters turbines at these distances. Wake effects estimated with the RVAE are slightly lower than those derived from the simulation as shown in Figure However, the RVAE seems to capture the intricate wind crientation-dependee effects which depend the farm layout.
252
7

---

maxiwd
1340
2340
2000
1500
1000
500
2000 1500
1000 1300 2000
Figure 3: Wind deficits on the simulated test farm and estimates from the trained RVAE. Each point associated with a turbine is plotted in a 2D polar coordinate system centered on the turbine. Each point is plotted towards the orientation of the incoming wind. The distance from the origin is proportional to the wake deficit, estimated as max(v) v where is mean power and mean wind.
RUAC aind make delicits Upaind spred:: mys
PLCRIS wind speed delicits
Reletive wind defett errer 2-..-
Up ndt sprecd m//s
750
250
-75
-250
200
460
650
66c
1000
1000
1400
200
40
de
1000
1200
1400
200
800
900
800
3006
1988
1403
wind speed make deleiis
FLORIS wind spced make defiit
wind
202
-500
3400
Figure 4: Learned spatial distribution of wake related wind speed deficit, evaluated as W(0,0) ((ryy) where ((0,0) is the the wind speed at the up-wind turbine and (æ. y) denotes the wind speed for probe turbine positioned at ((aa))
4.4
1D regression
255
In order to further demonstrate the versatility of the RVAE in modeling structured data, and in order make the connection to NPs (18) clearer, in what follows RVAE adapted for node data imputation is presented [30]. The dataset consists of sets of points sampled from a zero-mean 1D)
257
Gaussian process with squared exponential kernel. The pairs of input points {(I1mm.. : V1:... m
259
are used as node features to construct set of context and target graphs, where m corresponds to different GP realizations. Each graph contains set of edges e; which encode the relative position of the observation points. The edge features between observations at points 24, Ii are defined as [ .ry) é -c-l, where is function of the cutoff distance for edge creation. Note that the construction of such edge features endows the model with translation equivariance, In contrast MPNPs, [29]. the edges are the same the context and target graphs. The loss function used is the same as in Equation 1O, Both ps and qo are implemented as GNs. The outputs of the GNs parameterize Gaussian, I.C.,
280
281
283
264
285
286
(=(=) (()
Po(y)D) (Nn((D)),
(11)
The values of " are replaced with 0 when fed through the encoder and an additional binary feature b for the node, which denotes masking, appended to the node tuple. The b feature zero for the
287
268
unmasked nodes and 1 for the masked nodes. The masked input denoted by D The union of
289
the masked target input with the context dataset is denoted by T Instead of using two different
functions for the prior of p(2 (TANCC in [30], and posterior network q(> (TU)) and in order to

---

272
keep the conditional RVAE model closer to the NP formulation, the approximate posterior (i.e., the encoder of the RVAE) is used also for the learned prior. The decoder pe receives as node conditioning (and optionally edge conditioning) the æT values and the global latent variable u², Fach realization of u corresponds to a different context set which in turn corresponds to different sampled GP.
273
274
275
276
More information about he training setup can be found in the appendix.
277
The NP is implemented by defining DeepSet encoder, a global latent variable of the same size as the NP MLP. The same latent variable size for nodes and edges was used for each experiment, which is the same as the core size. All aggregation functions are mean aggregations. Experiments
278
279
280
were performed with different number of message passing steps, and inclusion of either the relative observation position as an edge feature or the absolute node position z; for each feature. The models are tested in un-seen GP realizations and the negative log- log-likelihood of predictions are reported in
281
282
283
Table 4 The RVAE models compute edge, node and global variables. The test datasets contain points with x E [0, 1] and x E [1 ranges in order to test the generalization capability of the proposed model in translation. Since the adge-blocks only ever receive translation equivariant inputs from
284
285
the dataset, the RVAE models generalize well in the E [1, 2] range. This is presented only as an example of how special equivariant inductive biases may be implemented in RVAE. It is observed that the full RVAE model does not perform well when only the node features are available. As with NPs, it was empirically found that models yield better results with more training.
287
289
Table 4: Test set log likelihoods ID GP regression with Conditional RVAE. The results are based on a set of 5000 unseen GP samples, each with 50 context and 50 target points. The models were trained only on points with in the [0. 1] range. Values in parentheses are standard deviations of the mini-hatches. RVAF denotes a model where all latent variables are used (edge node and global).
Model
size (mlp/z/MP Steps)
Only cond. on nodes GA. = (Vhs..) E [1,2]
Cond. on edges and nodes
GR. = (Vn, Er. :)
E [0,
I E [0, 1]
E [1.
CRVAE
64/64/0
117)94(3.11)
-4..59(4.10)
0.33().04)
-0.21 (0.13)
64/64/1
-12.55(2.51)
-9.79(2.51)
0.36(0.07)
0.08(0.06)
64/64/2
0.98 NA
0.67(0.08) NA
NP
64/64/NA
-1.34(0.07)
-11.13(3.08)
128/128/NA
-1.08(0.11)
-31.74(14.08)
NA
NA
Conclusions and broader impact
290
This work introduces an attributed graph approach to the probabilistic modeling of relations within entities and their properties. The approach is verified and validated on wake effect simulations and actual data from wind turbines placed within a wind farm; characteristic example that may be modeled as a graph. We also find some connections to the NP literature which we demonstrate by adapting the proposed method perform typical NP benchmark which is 1D regression for GP data.
292
293
294
285
296
We introduce a method for data-driven wake effect modeling for wind farms that accounts for uncer- tainty. The proposed method fuses physical intuition, flexible function approximation through GNs, and variational Bayes through re-pan re-parametrized gradients. Better and more computationally efficient wake effect modeling can lead to improvements in terms of accuracy and computational efficiency in analysis for wind farm siting [] farm layout optimization [2). wind farm control optimization [g) and ultimately power production improvements, as well as more robust to uncertainties maintenance planning. Ultimately, the aforementioned lead wind energy being more attractive clean energy solution.
296
300
302
303
304
Graph data are naturally used to model social, transportation and communication networks. Possible negative implications of any graph ML work relate to possible malicious uses of analysis in such networks, such as de-amonymization in social networks [44, and vulnerability exploitation on transportation networks.
306
308

---

References
309
[1] Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals, and George E Dahl. Neural message passing for quantum chemistry. In International Conference on Machine Learning, pages 1272-1272. PMLR, 2017.
311
312
313
[] Peter W Battaglia, Jessica Hamrick, Victor Bapst, Alvaro Gonnallz, Vinicius Zambaldi, Mateusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, et al. Relational inductive biases, deep learning, and graph networks. arXiv preprint arXiv: 1806.0/261, 2018.
314
315
316 [3] Alvaro Gonbalzz, Jonathan Godwin, Tobias Pfaff, Rex Ying, Jure Leskovec, and Peter Battaglia. Learning to simulate complex physies with graph networks. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 8459-8468. PMLR, 13-18 Jul 2020.
317
318
319
[4] Tobias Pfaff, Meire Fortunato, Alvaro Sanchez-Gonzalez and Peter W Battaglia. Learning mesh-basca simulation with graph networks. arXiv preprint arXiv:2010.03409, 2020.
321
[5] Vinicius Zambaldi, David Raposo, Adam Santoro, Victor Bapst, Yujia Li, Igor Babuschkin, Karl Tuyls, David Reichent, Timothy Lillicrap, Edward Lockhart, et al. Relational deep reinforcement leamning arXiv preprint or xrx: 100.0/ 03000 2018.
323
324
325
[6] Thomas Kipf, Ethan Fetaya, Kuan-Chich Wang. Max Welling, and Richard Zemel. Neural relational inference for interacting systems. In International Conference on Machine Learning, pages 2688-2697. PMLR, 2018.
326
327
328
[7] Diederik Kingma and Max Welling, Auto-encoding bariational bayes. arXiv preprint 1xXvv13312.6114 2013.
329
330
[8] Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and A Alexander CeandeerLerchee Lerchner. beta-vae: Learning basic visual concepts with constrained variational framework. 2016.
331
332
333
19] Alexander Alemi, Ben Poole, Ian Fischer, Joshua Dillon, Rif A. Saurous, and Kevin Murphy. Fixing broken ELBO In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Mochine Learning Research, pages 159-168. PMLR, 10-15 Jul 2018.
334
336
337
[10] Samuel R Bowman, Luke Vilmis, Oriol Vinyals, Andrew M Dai, Rafal Jozefowicz, and Samy Bengio. Generating sentences from a continuous space. arXiv preprint urXiv:1511.06344 2015.
338
[11] Hao Fu, Chunyuan Li, Xiaodong Liu, Jianfeng Gao, Asli Celikyilmaz, and Lawrence Carin. Cyclical annealing schedule: A simple approach to mitigating kl vanishing. arXiv preprint arXiv: /903. 10145, 2019.
340
341
[12] Danilo Rezende and Shakir Mohamed. Variational inference with normalizing flows. Francis Bach and David Blei, editors, Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pages 1530-1538, Lille, France, 07-09 Jul 2015. PMLR.
342
343
344
[13] Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real nvp. arXiv preprint arXiv: 100,00800,,, 2016.
345
346
[14] Jenny Liu, Aviral Kumar, Jimmy Ba, Jamie Kiros, and Kevin Swersky. Graph normalizing flows arXiv preprint Xiv: 1905. 13177, 2019.
347
348
[15] Casper Kaae Sønderby, Tapani Raiko, Lars Maaløc, Søren Kaae Sønderby, and Ole Winther. Ladder variational autoencoders. arXiv preprint 1602.0:600..22200 2016.
349
350
[16] Qi Wang and Herke Van Hoof. Doubly stochastic variational inference for neural processes with hierarchical latent variables. In International Conference on Machine Learning, pages 10018-10028. PMLR, 2020.
351
352
[17] Marta Gamelo, Dan Rosenbaum, Christopher Maddison, Tiago Ramalho, David Saxton, Murray Shanahan, Yee Whye Tch, Danilo Rezende, and SM Ali Eslami. Conditional neural processes. In International Conference on Machine Learning, pages 1704-1713. PMLR, 2018
353
354
355
[18] Marta Gamelo, Jonathan Schwarz, Dan Rosenbaum, Fabio Viola, Danilo J Rezende, SM Eslami, and Yee whye Teh. Neural processes. arXiv preprint 107.0020.. 2018.
357
[19] Mohammadanin Tavakoli and Pierre Baldi. Continuous represenattonooooooooo of molecules using graph variational autoencoder. arXiv preprint 2XXv:2004.00112. 2020.
10

---

359
[20]
Martin Simonovsky and Nikos Komodakis. Graphvae: Towards generation of small graphs using variational autoencoders. In International Conference on Artificial Neural Networks, pages 412-422. Springer, 2018.
[21]
Aditya Grover, Aaron Zweig, and Stefano Ermon. Graphite: Iterative generative modeling of graphs. In International conference on machine learning, pages 2434-2444. PMLR, 2019.
382
383
[22]
Thomas N Kipf and Max Welling. Variational graph auto-cneoders. arXiv preprint arXiv: 1611v3661.007305 2016.
364
365
[23]
Eric Jang. Shixiang Gu, and Ben Poole. Categorical reparameterizzaioo with gumbel-softmax, arXiv preprint 1.01 1,01144, 2016.
366
367
[24] Chris J Maddison, Andriy Mnih, and Yee Whye Tch. The concrete distribution: A continuous relaxation of discrete random variables. arXiv preprint 16Iiv:6111.000 2016.
368
369
[25] Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wienstra. Stochastic backpropagation and ap- proximate inference in deep generative models. In International conference machine learning, pages 1278-1286. PMLR, 2014.
371
372
[26]
Yann Dubois, Jonathan Gordon, and Andrew YK Foong. Neural process family. http:://anndubs itchub.io/Neurall. Pwrc-errocssssssee
373
September 2020.
374
[27]
Hyunjik Kim, Andriy Mnih, Jonathan Schwarz, Marta Gamelo, Ali Eslami, Dan Rosenbaum, Oriol Vinyals, and Yee Whye Teh. Attentive neural processes. arXiv preprint arXiv:/901.0000 090661 2019.
375
376
[28] Jonathan Gordon, Wessel P Bruinsma, Andrew YK Foong, James Requeima, Yann Dubois, and Richard E Turner. Convolutional conditional neural processes. arXiv preprint arXiv:/910. 13556, 2019.
378
[29] Ben Day, Cätälina Cangea, Arian R Jamasb, and Pietro Liò. Message passing neural processes, arXiv preprint arXiv:2009. 13895, 2020.
379
[30] Oleg Ivanov, Michael Figurnov, and Dmitry Vetrov. Variational autoencoder with arbitrary conditioning. arXiv preprint arXiv: 0238000222222 2018
381
382
[31] Manzil Zaheer, Satwik Kottrr, Siamak Ravanbakhsh, Barnabas Poczos, Ruslan Salakhutdinov, and Alexan- der Smola. Deep sets. arXiv preprint 17Xiv:1703.06114 2017.
383
384
[32] Christos Louizos, Xiahan Shi, Klamer Schutte, and Max Welling. The functional neural process. arXiv preprint Xrxiv:99666 08324, 2019.
385
386
[33] Wei Chu Vikas Sindhwani, Zoubin Ghahramani, and S Sathiya Keerthi, Relational learning with gaussian processes. In Advances in Neural Information Processing Systems 19: Proceedings of the 2006 Conference, volume 19, page 289. MIT Press, 2007.
387
388
[34] Yin Cheng Ng. Nicolò Colombo, and Ricardo Silva. Bayesian semi-supervised learning with graph gaussian processes. arXiv preprint /00,::/9,,904379, 2018.
390
391
[35] Naiqi Li. Wenjie Li, Jifeng Sun, Yinghua Gao. Yong Jiang, and Shu-Tar Xia. Stochastic deep gaussian processes over graphs. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages
392
393
Curran Associates.
Inc., 2020.
[36] Junyoung Park and Jinkyoo Park. Physics-inducee graph neural network: An application to wind-farm power estimation. Energy, 187:11588: 2019.
396
[37] James Bleeg. A graph neural network surrogate model for the prediction of turbine interaction loss. In Journal of Physics: Conference Series, volume 1618, page 062054. IOP Puhlishing, 2020.
398
399
[38] Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint arXiv:1412.6988 2014.
400
401
[39] Gabriele Corso, Luca Cavalleri, Dominique Beaini, Pietro Lid, and Petar Veličković. Principal neighbour- hood aggregation for graph nets. arXiv preprint arXiv:2004.05718, 2020.
402
403
[40]
NREL. FLORIS. Version 2.2.0, 2020.
404
[41] JK Lundquist, KK Vivie. D Kaffine, and JM Tomaszewski. Costs and consequences of wind turbine wake effects arising from uncoordinated wind energy development. Nature Energy, 4(1):26 34, 2019.
11

---

408
[42] Nicolas Kirchner Bossi and Fernando Porté-Agel. Multi-objective wind farm layout optimization with unconstrained area shape. In Wind Energy Science Conference 2019 (WESC 2019), number POST_TALK, TALK,
408
2019.
[43] Michael F Howland. Sanjiva K Lele, and John O Dabiri. Wind farm power optimization through wake steering. Proceedings of the National Academy of Sciences, (66622)):4445-111 14500, 2019.
410
[44]
Shouling Ji, Prateek Mittal, and Raheem Beyah. Graph data anonymization, de-anonymization attacks, and de anonymizability quantification: A survey, IEEE Communications Surveys & Tutorials, 19(2):1305 1326.
412
413
2016.
Checklist
415
1. For all authors...
416
(a) Do main claims made the abstract and introduction accurately reflect the paper's contribu- tions and scope? [Yes]
417
418
(b) Did you describe the limitations of your work? [Yes] As noted in section 3 the main limitation
419
of RVAF is that it cannot be used directly as a graph generation model.
420
(C) Did you discuss any potential negative societal impacts of your work? [Yes]
421
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
422
2. If you are including theoretical results...
423
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
424
(b) Did you include complete proofs of all theoretical results? [N/A]
425
3. If you ran experiments..
426
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (cither in the supplemental material or as a URL)? [TODO/The code and dataset will be shared within the dcadline for providing supplemental materials.
427
428
429
(b) Did you specify all the training details (e.g.. data splits, hyperparameters, how they were chosen)?
430
[Yes] see seefsubsection 4.4 and subsection 4.2
431
(c) Did you report error hars (c.g., with respect to the random seed after running experiments multiple times)? [N/A] For experiments in Table [and Table 2 the standard deviation of the batch statistics for single
432
reported since the differences of the models were pronounced. For the results in Table 3 the statisties reported are based on 3 independent runs.
434
435
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] All reported experiments were run on a personal desktop running Ubuntu 20.04, with an n Vidia RTX2080 GPU and 16GB of RAM.
436
437
4. If you using existing assets (e.g.. code, data, models) curating/releasing new assets...
439
(a) If your work uses existing assets, did you cite the creators? [Yes] Orsted energy, formerly known as Dong energy provided us with a proprietary SCADA dataset for the Anholt farm.
440
441
(b) Did you mention the license of the assets? [Yes] The Anholt SCADA datasct was made available to us under an non-disclosure agreement. The simulated wind farm dataset will be released under
442
the CC-BY 4.0 license.
(c) Did you include any new assets either in the supplemental material or as URL? [Yes]
445
(d) Did you discuss whether and how consent was obtained from people whose data you're 're us- ing/eurating? [N/A]
447
(e) Did you discuss whether the data you are using/curating contains personally identifiable informa- tion or offensive content? [N/A]
449
5. If you used crowdsourcing or conducted research with human subjects...
450
(a) Did you include the full text of instructions given to participants and screenshots, if applicable?
451
[N/A]
452
(b) Did you descrihe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
453
454
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
455
12


Reviews:
[{"review_id": "yPZqDTBua1M", "reviewer": "Reviewer_UzkQ", "summary": "This paper proposes a framework for a general probabilistic generative model for attributed graphs.\nThe proposed model assumes that the nodes, edges, or global attributes of a graph have many different properties, which can be deterministic or stochastic, and observable or unobservable, and so on.\nThe proposed model formulates a stochastic dependency between the conditioning graph variable G_h and the state graph variable G_s similar to beta-VAE, assuming the existence of several latent variables in addition. \nThe paper shows that the proposed model encompasses many existing Bayesian graph models and evaluate its performance using wind farm dataset and artificial simulation dataset with ELBO or test data log-likelihood.", "questions": "", "limitations": "", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Originality: \nIn Sec. 2, the proposed model is explained with many equations. As far as I understood, the variables of the graph data are separated into conditioning and state, and the probabilistic dependency between these two groups of variables and latent variables is assumed to have a dependency structure similar to that of beta-VAE. \nAs far as I know, no VAE-like graph model has ever been proposed to model the process of graph data generation in such a general way, so there is a  concrete novelty at this point.\n\nHowever,  it is unclear that what kind of benefits we can expect from the proposed model. This is because there are no observations or assumptions about the data to be modeled. In turn, there are no new ideas or inspirations to explain the characteristics of the data efficiently in the graphical models.\nI felt that there is no strong answer given to the question, \"We have created a new model, but is it beneficial?\" throughout the paper. \n\n## Quality\n\nI think there is a room for improving this manuscript as I write in the following  \"clarity\" and \"significance\" issues. \n\n ## Clarity \n\nIn the introduction, the motivation for this research is not explained intensively. Therefore, it is difficult to answer \"Why is this research necessary? What is the value of this research?'' \nTherefore, readers may find it difficult to understand these importantquestions. This problem is not resolved for me even after reading the paper to the end.\n\nIn L17-18, it is written ''the ubiquity of noisy structured data and systems with stochastic or partially observable\n interactions of industrial relevance (e.g. wind farms and urban transportation networks)''. \nFor most readers, this sentence alone does not explain the motivation of the study at all. This is because the readers are unfamiliar with these examples and have no knowledge of what they want to do with these data. \n\n(A) what kind of uncertantiies exist in these data\n(B) what kind of problems caused by the uncertainties\nand (C) why existing GNNs and graph generation models are insufficient to solve them\nGiven concrete explanations of all A, B, and C, then, I think we can finally reach the understanding that a general form of a probabilistic graph generation model is necessary.\n\nThe same difficulty in understanding due to lack of explanation affects each technical discussion in Sec 2.\n- L37-65 Why did you make the settings for each attributes so general (stochastic/deterministic, observable/non-observable, static/dynamic)? \nIn many DNN/GNN researches, it is of interest to choose appropriate inductive biases into the model to perform good at chosen tasks or goals. The formulation of Sec. 2 seems to go in the opposite direction, and I don't understand the intention or the motivations for a general formulation like this. \n\nIn Sec. 4, the examination of specific data begins.  Unfortunately, the explanation is not concise and easy to understand. \nSince the paper introduces many symbols and conditional probability distributions in Sec. 2, I would like to describe the data with these symbols or equations:  for example, which symbol corresponds to an element of wind fard data, and which term of the conditional probability distribution will be important. \nClearer description of the data wouldl help to convey the appropriateness of choosing counterparts (CVAE and NP) for experiments. In the current manuscript, it is very difficult to understand the appropriateness of the choices. \n\n## Significance:\nThe lack of explanation of motivation described in the Quality section makes it difficult for me to evaluate the significance of this paper. If we do not fully understand the reason ''Why is this study necessary?'', it is difficult to evaluate the significance of the content of the paper. \n\nI think that the current experiment is insufficient to experimentally demonstrate the usefulness of the proposed RVAE. \n(I) The proposed RVAE model is quite broad and general in its formulation.\n(ii) The effective latent variables and conditioning variables are expected to be different from dataset to dataset.\nGiven (i) and (ii), it is necessary to conduct a thorough ablation study on many different datasets to evaluate the proposed method accurately.\nBy testing such experiments on several real-world datasets with different characteristics, we can see that RVAE is highly effective on any dataset, thanks to the general formulation.\n\nThus, in my opinion, the paper needs more experiments in various datasets. Ideally, the counterpart models should variy across dataset to dataset reflecting the different nature of datasets and tasks. \n\nFigs. 3, 4 I think that it is insufficient to convince the readers of the usefulness of the proposed method even qualitatively unless the visualization of the estimation results by other methods is included in addition to the estimation results by RVAE.\n\n(+) a broad and general formulation for VAE-like graph generation models\n(--) motivations are not explained well\n(-) description of datasets can be improved clearer and more informative way\n(--) experimental validations are insufficient to show the superiority of a proposed general framework\n\n## After author feedback\n\nThank you authors for the detailed clarifications. \nYour feedbacks resolves several of my misunderstandings. Thus I raise my scores +1. \n\nAt the same time, the feedbacks reveal the manuscript lacks these important explanations. \nAlso I still believe the experimental validations are not sufficient for convincing non-expert readers like me. \nTherefore I cannot vote for acceptance before confirming the revised manuscript fulfills these requirements. \n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No specific comments", "review_text": "", "extra_scores": {"ethical_concerns": "No specific concerns.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "FLEn5pxylEA", "reviewer": "Reviewer_zzT8", "summary": "This paper proposes a generalized formulation of a probabilistic model for graph-structured data that incorporates graph networks (GN) (or rather, provides a probabilistic extension thereof). The authors show that the proposed framework is general enough to subsume existing approaches such as Neural Relational Inference (NRI) and Neural Processes (NP). The method is empirically validated on simulated and real-world data, with emphasis placed on the modelling of wind farm data.", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper is generally well-written and reasonably easy to follow. There are a few grammatical mistakes that commonly occur throughout the paper; a non-exhaustive list is provided in the Misc Issues section of this review. I checked the details of the mathematical equations and was convinced that the proposed framework is technically sound.  \n\nHowever, it was difficult to assess the significance and originality of this paper's main contribution, and I was not totally convinced. The proposed framework is indeed quite general and shown to include existing methods such Neural Relational Inference (NRI) and Neural Processes (NP) as special cases. On one hand, generalizations such as this are often important contributions since by providing a framework under which to unify existing approaches, they can help provide a better understanding of the existing approaches, in particular how and why they work, and clarify the conditions and assumptions under which they are closely related. On the other hand, there is a fine line between generality and vagueness, the latter case being that something is so general it ceases to be particularly insightful or interesting.\n\nIn Section 2 \"Methods\", there is nothing particularly novel and original. The framework formalizes the various components of a graph -- the nodes, edges, and global attributes, and considers their various taxonomies as deterministic or stochastic, observed or hidden, dynamic or static. A subset of the observed components makes up the conditioning or context variables and another makes up the state variables. A graph-structured latent variable consisting of vertices, edges, and global variables is assumed. A joint distribution over these variables is then specified, which factorizes into distributions over the vertices, edges, and global variables, respectively. All of this is straightforward and fundamental to probabilistic modeling.\n\n> Sidebar question: Eq (3) what specifically are these prior distributions? I couldn't find these specified anywhere. \n\nNext, the procedure for approximate inference over the graph-structured latent variable is described. Again, this is just a description of the standard application of the principles of variational inference using amortized inference with a Graph Network, with the addition of scaling the KL divergence terms using a $\\beta$ hyperparameter as in $\\beta$-VAEs. \n\n> Sidebar comment: line 76-77 \"A GN is proposed for inferring the parameters.\" - in more precise terms, you're *amortizing* the inference on these parameters using GNs. Consider perhaps invoking this term to make this more explicit.\n\nRegarding the experimental results, there is a concern that the method might be overfitted to wind farm data, and might fail to generalize well beyond this context. On the one hand, I was pleased to see the method applied to a real-world problem, particularly one that has a high potential for positive societal impact (contributing to better clean energy solutions). On the other hand, as a methodological contribution, I would have liked to see results on a more diverse range of problems, both real and synthetic. The other issue is the lack of comparisons against baselines. On the real wind farm dataset, the CVAE is the only baseline considered. On the 1D regression dataset, NP is the only baseline considered. \n\nIn summary, I find the contribution to be fairly significant but somewhat incremental. In particular, the originality and novelty are somewhat limited. The paper is communicated clearly and of reasonably high quality. The experiments are mostly well-executed but may be over-specialized to the domain of wind farm modeling.\n\n### Misc Issues\n\n- line 212: \"The CRVAE models are compared to a two-layer MLP-based CVAE trained with the arbitrary conditioning objective [30]\" - this is the first occurrence of the acronym \"CVAE\". From the context and subsequent citation, it's possible to guess that it refers to the conditional variational autoencoder, but the reader shouldn't need to guess.\n- Please consider rasterizing the figures on Page 8 (the culprit is most likely Figure 3) which causes my PDF viewer to lag on a reasonably powerful computer.\n- line 309, reference: please sort your references alphabetically by the first author's last name\n- line 328, citation [7]: \"bariational bayes\"\n\nThe following sentences all have extraneous commas (probably non-exhaustive):\n\n- line 60: \"The full graph state, is denoted by\"\n- line 114: \"in this work, the above mentioned approaches, are generalized and unified\"\n- line 128: \"the input-output observation cases D, are split as\"\n- line 146: \"The aforementioned models, feature a global latent variable\"\n- line 209: \"for using composite aggregators, is also due to the physics of the problem\" \n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors recognize the potential negative societal impacts of their work, in particular, its potential malicious uses in de-anonymization of social networks and exploiting vulnerabilities in transportation networks.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6 hours"}, "extra_text": {}}, {"review_id": "BcnyYWhr9Fn", "reviewer": "Reviewer_2qz2", "summary": "This paper introduces Relational VAE, which is an extension of NPs for graph-structured data. However, the selling point of the paper is not that much clear.  ", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is hard to follow, motivation is not very clear, and seems to have some notational mistakes. I had to read it multiple times and still do not know what exactly the authors plan to do with their model. \n\nI also believe the authors change some ML notations which are not necessary and sometimes are incorrect, starting from the title. For example, they mentioned their model as Relation VAE, however, they really proposed a new version of NP for graph-structured input data. Even from the lens of VAE, the relation in this model is the input graphs. As stated in lines 117-120, the authors stated \" Inferring graph connectivity or generating graphs, however, falls out of the scope of this work. In RVAE the focus is generative modeling of graph structured data with an apriori known connectivity, with [...].\" Or they clearly could use index $c$ instead of $h$ to refer to the context/reference points.  The article has the unfortunate tendency to contains a few unproven (or wrong) claims. For instance, \"In this work, the above mentioned approaches, are generalized and unified in the proposed Relational Variational Autoencoder (RVAE) model\", which would deserve some empirical/theoretical evidence. For example, the goal of NRI, is inferring the relation between objects in dynamic systems, where the input might not be structured. I cannot see how the proposed model is extending this. \n\nMy main concern with this paper is novelty: the method seems to be a straightforward application of NP/CVAE to a different data domain on which variational auto-encoders are being used. Comparing to NP, this paper only considered the observed samples are graphs. So, they only need to change the architectures to GNNs. For example, Equation (4) is similar to NP equation in [ Garnelo et al, NeurIPS 2018].  \n\nConcerning the experiments, I am not an expert in wind farm monitoring data, but I believe that as an ML conference, the authors do not clearly define and motivate the problem for a reader. Besides that, they could compare with some other existing baselines or include some sort of ablation study. \n\nOverall, I think that this paper would need to be extremely solid on the experimental side with potentially further experiments, e.g. on node classification and more detailed analysis with respect to label rates of nodes, robustness to missing edges etc, in order to make up for the somewhat limited contributions in terms of novelty. Apart from that,  plotting $G_z$, showing the benefit of uncertainty modeling in the proposed method using pavpu or etc. would be helpful. For now, I cannot recommend this paper for acceptance at NeurIPS.\n\n\nThere are multiple notational mistakes. \n\n1) = in equation 10 is missed in the second line\n2) I think the ; in $p_\\theta(G_z; G_h^{(i)})$ in equation 4 is not correct\n3) I believe an expectation is missed in Eq 4 \n4) writing a loss for each sample in Eq 4 is not meaningful, while $G_z$ does not have any index. \n\nIf the model is Bayesian Graph modeling, the authors need to discuss and compare it with other Bayesian Graph modelings as well. \n[1] Bayesian Graph Neural Networks with Adaptive Connection Sampling, ICML 2020\n[2] Bayesian Graph Convolutional Neural Networks for Semi-Supervised Classification, AAAI 2019\n[...] There are multiple other related papers\n\nEquation 6 assumes the normal distribution for the relations. This might lead to a very dense graph. \n\nI cannot follow lines 91 - 95 in that place. You might want to put them in the discussion or related works. \n\nValidation could be used for early stopping. It is not clear to me how you used test data for this.\n\nIt is not clear how the authors change the name of the model from RVAE to CRVAE and what are the differences.\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The complexity of the model and dealing with large graphs can also be considered as the limitation of the model.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4 hours"}, "extra_text": {}}, {"review_id": "B1BTnRENdGg", "reviewer": "Reviewer_4JjX", "summary": "The authors propose to extend Graph Nets (GNs) by allowing edge-bound uncertainty and edge-level conditioning using a Variational Bayes formulation.   ", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is clear although (necessarily) not self-contained and it therefore becomes at times hard to follow for someone who is not deep into the subject matter.  The authors should perhaps try to better explain the flexibility and the generalisation capacity of the model proposed: would the model break or suffer catastrophic failures if the nodes are not placed in a quasi regular lattice? The reader would assume that as the turbines placement becomes more irregular, it becomes more difficult to leverage and transfer information from various parts of the network, but does the edge modelling give more robustness in these cases? Quantifying these notions via empirical experimentation would  improve the paper impact.  \nSome empirical comparison with Gaussian Processes for the turbine application case is missing.  \nFurther improvements to the paper could include a more detailed discussion on the implications and limitations of factorizing the graph priors by the edges, nodes and global variables separately.  \nFinally, perhaps a more representative artificial case could be devised to showcase the improved capacity to model relational information since in the proposed example the edges simply encode the distance between nodes and this is a type of information that can be readily derived given that the node attributes include the node coordinates, i.e. it is not something that is accessible only through modelling an relational process that is independent from the node process.  \n\n###After authors' reply\nMany thanks to the authors for the effort in replying all our concerns. After reviewing the comments raised by the other reviewers I find myself sharing several of their concerns, especially the need of a much improved clarity in explaining the relationship of their proposal to Neural Processes. I maintain that for non-expert readers it would be beneficial to be presented with intuitive artificial cases where the relational nature of the objective is clear and unquestionable.   \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have very briefly addressed the limitations and potential negative societal impact of their work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This submission considers a probabilistic model for graph-structured data that incorporates graph networks. It also attempts to connect/extend CVAEs, NPs for graph data. However, the reviewers remained unconvinced that the generalisation provides theoretical insights or practical benefits. In addition, all reviewers agreed the exposition of the work is hard to follow and the experiment section requires more work. Thus, the paper in its current form is not ready to be accepted.
Decision: Reject