Relational VAE: A Continuous Latent Variable Model for Graph Structured Data

Graph Networks ( GNs ) enable the fusion of prior knowledge and relational reason-1 ing with flexible function approximations . In this work , a general GN-based model2 is proposed which takes full advantage of the relational modeling capabilities of3 GNs and extends these to probabilistic modeling with Variational Bayes ( VB ) . To4 that end , we combine complementary pre-existing approaches on VB for graph data5 and propose an approach that relies on graph-structured latent and conditioning6 variables . It is demonstrated that Neural Processes can also be viewed through the7 lens of the proposed model . We show applications on the problem of structured8 probability density modeling for simulated and real wind farm monitoring data , as9 well as on the meta-learning of simulated Gaussian Process data . We release the10 source code , along with the simulated datasets.11 1 Introduction12 Graph Neural Networks ( GNNs ) [ 1 , 2 ] have been established as an effective tool for representation13 learning on graph structured data . Graph structured data are routinely employed to represent entities14 and relations among them . The present work focuses in representation of uncertainty and generative15 modeling for attributed directed graph data with continuous attributes . The initiating motivation for16 this work is the ubiquity of noisy structured data and systems with stochastic or partially observable17 interactions of industrial relevance ( e.g . wind farms and urban transportation networks ) .18 In the context considered herein , modeled entities ( nodes ) and modeled relations ( edges ) may feature19 a state , which may not be fully observed and/or stochastic . The same may also holds for global20 ( graph ) attributes . At the same time , nodes and relations may possess a dynamic partially observed21 state , which we may infer directly from data . Both the node states and edge states are not fully22 observed and non-deterministic , which amply motivates probabilistic extensions of graph networks.23 In essence , this work proposes a method that 1 ) exploits the relational structure of data and 2 ) allows24 for learning flexible distributions over entity and relation attributes . Several partially overlapping25 approaches for this problem exist . A short review of such prior approaches is offered in section 3.26 Modeling entities and relations has been shown empirically to allow for stronger generalization27 [ 3 , 4 , 5 ] in novel settings . The main contribution of this work is to propose an approach to transfer the28 potent combinatorial generalization and modeling capabilities of GNNs to the problem of modeling29 conditional distributions of structured data.30 2 Methods31 Attributed graphs Following [ 2 ] , global attribute augmented graphs are denoted by G =32 ( V , E , u ) where V : { vi } i=1 : Nv with vi ∈ Rd v denoting the nodes ( vertices ) of the graph,33 Submitted to 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Do not distribute . E : { ( ek , sk , rk ) } k=1 : Ne designating the set of edges , with edge attributes ek ∈ Rd e , sk , ri ∈ N134 denote the head ( sender ) and tail ( receiver ) nodes of the modeled relation , while u ∈ Rdu is the35 global attribute.36 Graph Networks ( GN ) ( or GraphNets ) are composite functions that receive and return attributed37 graphs . The full GN block consists of an edge update , a node update and a global update block . Each38 block contains a corresponding function φe , φv , φu . The edge update function uses edge , node and39 global data . The edge block is followed by an aggregation step ρe→n , where edge messages are40 accumulated according to a permutation invariant function , e.g . a mean function . The node update41 uses ( optionally ) the global state , the aggregated edge state and the current node state . Finally , a global42 block aggregates with permutation invariant functions the edge and node properties ( ρe→u , ρv→u ) ,43 and optionally uses the global state for updating the global variable state . Different parts of the full44 GN computation may be omitted . Several Graph Neural Network architectures can be cast as special45 cases of GNs by omitting certain features or by special choices of the different functions involved46 [ 2 ] . In what follows , when referring to GNNs , the most general and expressive GN layer is implied47 except otherwise specified.48 In the proposed model , entities ( nodes ) , relations ( edges ) and global attributes contain both determin-49 istic and stochastic variables . These variables in turn , may be observable or not directly observable.50 Both observable and unobservable attributes may be deterministic or stochastic ( static or evolving ) .51 In what follows , a part of the observable quantities is referred to as conditioning or context . The node,52 edge and global observable quantities are denoted as vh , eh , uh where h signifies that a variable53 corresponds to conditioning . Conditioning variables may either correspond to conditioning with54 known dynamic quantities or static quantities . Common instantiations of such conditioning are55 positional encoding for vertices , relative position for edges between vertices and time of day as56 a global variable . The node , edge and global variables that correspond to the rest of the states57 ( stochastic , evolving , unobserved ) are denoted by vd , ed , ud . In essence , the conditioning attributes58 can be used to create a conditioning graph variable Gh = ( Vh , Eh , uh ) and a state graph variable59 Gx = ( Vx , Ex , ux ) . The full graph state , is denoted by Gd = ( Vx ∪ Vh , Ex ∪ Eh , ux ∪ uh ) where60 ∪ denotes set union . Since part of the node , edge and global attributes may be stochastic , a graph61 structured latent variable Gz = ( Vz , Ez , uz ) is assumed . The graph structure may also be determined62 through the edge variables as in [ 6 ] , but we restrict our model to a pre-determined graph structure63 in this work . The following model is proposed for the joint distribution of the graph structured64 observations65 p ( Gx ; Gh ) = ∫ p ( Gx|Gz ; Gh ) p ( Gz ; Gh ) dGz ( 1 ) where p ( Gz ; Gh ) = p ( Vz ; Vh ) p ( Ez ; Eh ) p ( uz ; uh ) is the distribution of the latent variables given Gh.66 A prior distribution conditioned on Gh is assumed for the latent variable , which is further factorized67 along each edge and node latent separately , i.e.,68 p ( Gz ; Gh ) = p ( V ) ( Vz ; Vh ) p ( E ) ( Ez ; Eh ) p ( u ) ( uh ; uz ) ( 2 ) = Nv∏ i=1 p ( vzi ; v h i ) · Ne∏ k=1 p ( ezk ; e h k ) · p ( uz ; uh ) . ( 3 ) An approximate posterior ( i.e. , recognition model ) is assumed for Gz as qφ ( Gz|Gx ; Gh ) together69 with a generative model for Gx , pθ ( Gx|Gz ; Gh ) . In correspondence with the Variational Autoen-70 coder ( VAE ) [ 7 ] , we seek to learn the generative model parameters θ and inference model param-71 eters φ simultaneously . Assuming independent identically distributed ( i.i.d . ) graph observations72 { G ( 1 ) x , . . . G ( i ) x } , the Evidence Lower Bound ( ELBO ) for the marginal log-likelihood reads73 L ( θ , φ ; G ( i ) x , G ( i ) h ) =Eqθ ( Gz|G ( i ) x ; G ( i ) h ) [ log pθ ( G ( i ) x |Gz ; G ( i ) h ) ] −DKL ( qφ ( Gz|G ( i ) x ; G ( i ) h ) ||pθ ( Gz ; G ( i ) h ) ) ( 4 ) We seek to perform fast and scalable approximate inference over the Gz graph variable and at the74 same time take advantage of the relational structure in the data . A particularly convenient choice75 for parametrizing Gz is to assume a parametric distribution over edges , nodes and globals . A GN is76 proposed for inferring the parameters . For a graph structured observation observation Gx , we write77 Vz ∼ q ( V ) φ ( Gz|Gx ; Gh ) = N ( f µ ( V ) qφ ( Gx ; Gh ) , f σ2 ( V ) qφ ( Gx ; Gh ) ) ( 5 ) Ez ∼ q ( E ) φ ( Gz|Gx ; Gh ) = N ( f µ ( E ) qφ ( Gx ; Gh ) , f σ2 ( E ) qφ ( Gx ; Gh ) ) ( 6 ) uz ∼ q ( u ) φ ( Gz|Gx ; Gh ) = N ( f µ ( u ) qφ ( Gx ; Gh ) , f σ2 ( u ) qφ ( Gx ; Gh ) ) . ( 7 ) The functions f µ ( · ) · and f σ2 ( · ) · are implemented by a GN to allow for taking into account in a general78 manner relational information while inferring over Vz , Ez and uz . In practice a shared , single GN,79 fqφ ( · ) is used . The parametrization for vertices , edges and global variables are the corresponding80 states of the GN at the final message passing step . In a similar manner , a GN generator network,81 gpθ ( · ) , is used for pθ . Since the prior and posterior are factorized over nodes , edges and the global82 variable of each graph datapoint , the ELBO is split accordingly as83 L ( θ , φ ; G ( i ) x , G ( i ) h ) =Eqθ ( Gz|G ( i ) x ; G ( i ) h ) [ log pθ ( G ( i ) x |Gz ; G ( i ) h ) ] − βVDKL ( q ( V ) φ ( Gz|G ( i ) x ; G ( i ) h ) ||p ( V ) θ ( Gz ; G ( i ) h ) ) − βEDKL ( q ( E ) φ ( Gz|G ( i ) x ; G ( i ) h ) ||p ( E ) θ ( Gz ; G ( i ) h ) ) − βuDKL ( q ( u ) φ ( Gz|G ( i ) x ; G ( i ) h ) ||p ( u ) θ ( Gz ; G ( i ) h ) ) ( 8 ) where βV , βE , βu can be used for controlling disentanglement as in β–VAE [ 8 ] or the rate-distortion84 characteristics of the model [ 9 ] or for preventing posterior collapse and aiding training through85 KL-annealing [ 10 , 11 ] . In a similar manner to VAEs , the approach to representing distributions over86 graph data with a distribution that factorizes over V , E and u allows for defining alternative evidence87 lower bounds for variational Bayes . Note that the distribution does not need to be factorized along the88 elements of the latent vector . This allows straight-forward extensions using more flexible distributions89 [ 12 ] . A generative model based on normalizing flows that uses shift-scale transformations [ 13 ] has90 already been proposed in [ 14 ] for graph generation . The Relational VAE ( RVAE ) model proposed91 can be extended as a hierarchical VAE [ 15 ] yielding a model akin to Doubly Stochastic Variational92 Neural Process ( DSNPV ) [ 16 ] , which uses global and node variables . Finally , Neural Processes93 [ 17 , 18 ] ( NP ) and other graph encoder-decoder models [ 6 , 19 , 20 , 21 , 22 ] are closely related to the94 proposed model.95 3 Related work96 GNN Encoder-decoder models In Neural Relational Inference ( NRI ) [ 6 ] discrete edge latent vari-97 ables are inferred from node representations and a re-parametrized Gumbel− Softmax distribution98 is used [ 23 , 24 ] . A coarse representation of the computational graphs of NRI , NPs and the RVAE99 is shown in Figure 1 . In [ 19 ] graphs are modeled from global continuous latent variables , which100 are subsequently used for graph generation through an adjacency matrix . In GraphVAE [ 20 ] the101 global variable together with a graph-structured conditioning variable is used for generation . In102 Graphite [ 21 ] a latent variable for each node is inferred from the encoder , while the edge variables103 ( i.e. , symmetric adjacency matrix ) is inferred through efficient iterated message passing . Similarly,104 the VariationalGAE [ 22 ] uses a separate latent variable for every node and a graph convolutional105 encoder . Several of the aforementioned works take advantage of recent advances in low-variance106 gradient estimates for distributions over latent variables , as in Variational Autoencoders ( VAEs ) via107 the reparametrization trick [ 7 , 25 ] . The overlapping traits of the aforementioned are the treatment108 of edge , node and global variables . In Table 1 a summary of the relational modeling capabilities of109 various graph encoder-decoder models is offered . Note that the table highlights only the relevant110 parts to this work together with several important and influential design choices for graph representa-111 tion learning were not touched upon . For instance , the graph convolutional models of some of the112 aforementioned works offer the important advantage of scalability and small computation cost.113 In this work , the above mentioned approaches , are generalized and unified in the proposed Relational114 Variational Autoencoder ( RVAE ) model . Note that it is not difficult to yield explicit graph connectivity115 in RVAE as in NRI [ 6 ] since the type and existence of a connection can be seen as a categorical116 variable . See also Figure 1 ( b ) , where a sketch of NRI is offered . Inferring graph connectivity or117 generating graphs , however , falls out of the scope of this work . In RVAE the focus is generative118 modeling of graph structured data with an apriori known connectivity , with attributed nodes and119 edges , which optionally may include a global attribute that influences both entities and relations.120 Neural processes In Neural Processes ( NP ) [ 17 , 18 ] , we consider a set of mappings F : X → Y121 where X : { xi } , xi ∈ RNx , Y : { yi } , yi ∈ RNy . A particular draw of a function f ∼ F , is modeled122 as f ( xi ) = gθ ( xi , z ) where z ∼ p ( z ) is a high dimensional random vector ( e.g . a standard normal ) 123 and gθ is a neural network and θ denotes the parameters of g. Given a set of nm input-output124 observations D : { ( x1 : nm , y1 : nm ) fm } from m different realizations of f ( potentially different in125 number ) , we want to learn a distribution over z ∼ p ( z|D ) . Under the NP approximation , assuming126 observation noise yi ∼ N ( gθ ( xi , z ) , σ2 ) , the distribution of y is defined as127 p ( z , y1 : n|x1 : n ) = p ( z ) n∏ j=1 N ( yi|g ( xi , z ) , σ2 ) . ( 9 ) In practice , the input-output observation cases D , are split as DC∪T = DC ∪ DT , where C denotes128 a set of points with observations in X and Y and T denotes a set of points where we only observe129 X ( i.e. , the inputs ) . This can be cast as a conditional generative model for p ( yT |xT , xC , yC ) =130 pθ ( yT |xT , z ) p ( z|xC , yC ) , where the conditioning is the fully observed context pairs . The ELBO131 used for optimization is132 log p ( yT |xT , xC , yC ) ≥Eqφ ( z|DC∪T ) [ ∑ i∈T log pθ ( yi|z , xi ) + log q ( z|DC ) q ( z|DC∪T ) ] Eqφ ( z|DC∪T ) [ ∑ i∈T log pθ ( yi|z , xi ) ] −DKL ( qφ ( z|DC∪T ) ||qφ ( z|DC ) ) ( 10 ) Note that the above variational objective has an intuitive interpretation , as a reconstruction loss ( first133 part ) and a Kullback-Leibler divergence between the approximate posterior distributions predicted134 when using both C ∪ T and when using only C ( the context set ) . In [ 30 ] a similar loss function was135 proposed with the motivation of training VAEs that can be used with arbitrary conditioning masks . By136 considering the set of observations as nodes in a disconnected graph , ( i.e. , V : { vi| ( xi , yi ) } i=1 : Nv ) 137 and training while masking the context output nodes yC , the same objective is retrieved . Therefore,138 following the nomenclature of [ 2 ] , we can instantiate a NP from the proposed model , by using139 arbitrary conditioning as described in [ 30 ] , a DeepSet [ 31 ] as an encoder and only a node-block as a140 decoder as shown in Figure 1.141 The NP framework has been extended to take advantage of special inductive biases , such as the142 relation of observation and target nodes in Attentive Conditional Neural Processes ( AttNP ) [ 27 ] or143 the translation equivariance in Convolutional Conditional Neural Processes ( ConvCNP ) [ 28 ] . More144 recently , relational inductive biases were employed in Message Passing Neural Processes ( MPNP ) 145 [ 29 ] . The aforementioned models , feature a global latent variable uz which is inferred from the146 context points and parametrizes the distribution over functions . With the exception of MPMP , the147 aforementioned works target non-relational data . Nevertheless , MPMPs does not directly implement148 edge-bound uncertainty or edge-level conditioning , which is the most pronounced difference to RVAE.149 Similar to this work , in DSNPV [ 16 ] a NP that allows for both node and global latent variables150 was proposed , which in addition , employs a hierarchical VAE [ 15 ] . The motivation of DSNPV is151 to include node-context information , which in the conditional RVAE is also supported by design152 through Vh . RVAE attempts to merge the complementary strengths of the aforementioned models153 in representation of uncertainty , with a focus towards modeling graph structured continuous data.154 Finally , in contrast to Functional Neural Processes [ 32 ] we do not deal with inferring a graph of155 dependencies among latent variables , yet hierarchical RVAE adaptations may also manage such tasks.156 Graph Gaussian processes Sharing the motivation of this work , i.e. , taking advantage of relational157 information and learning joint distributions of graph structured data , in [ 33 ] GPs were defined over158 graphs with undirected binary ( positive or negative ) edges and applied to semi-supervised learning159 problems . In [ 34 ] the authors applied GPs trained with variational approximations for semi-supervised160 learning on graphs that contain non-attributed edges . In [ 35 ] GP-based approaches are fused with161 deep learning for learning graph ( e.g . network ) structured signals.162 4 Results163 4.1 Wind farm operational data164 A real-world industrial application , where relational structure is inherent in the observed data , is165 found in modeling of operational data of wind turbines positioned in a farm . The wind turbines166 ( nodes ) feature static variables , such as their power production characteristics and their position,167 as well as dynamic variables such as their current operational state . The actual operational state168 of a turbine is only known up to a certain precision from historical data , ( i.e. , Supervisory Control169 and Data Acquisition ( SCADA ) data ) , which is usually limited to 10 minute statistics . Due to170 the stochasticity of the wind excitation , compounded by incomplete information due to coarse171 measurements , there is uncertainty associated with the actual operational state of a wind turbine.172 Wind turbines arranged in a wind farm interact through the so-called wakes , which are travelling173 vortices that affect the power production and vibrations of downstream turbines . The magnitude174 of wake effects is related to large scale turbulence ( which is a global dynamic variable ) , to wind175 orientation ( which is a global dynamic variable ) , to upwind turbine nacelle orientation ( which is a176 node dynamic variable ) , the relative position between the two turbines ( an edge static variable ) , the177 rotor diameter and the distance between the two turbines . The interaction is one-way directional but178 can change directionality depending on the wind orientation . The effect of wakes is stochastic due to179 turbulence . For robust wind power prediction , monitoring , control , and maintenance planning , we180 want to infer the distribution of operational characteristics of a wind farm conditioned on turbine181 characteristics and farm layout . Of crucial importance is the inclusion of stochastic variables in182 the interactions ( i.e. , edges ) of the considered graph . Static graph edges , used as part of the graph183 conditioning , are constructed by considering the spatial proximity and relative position of pairs of184 turbines . The goal is to generalize directly to unseen farm configurations while learning directly on185 real condition monitoring data ( zero-shot generalization ) but at the same time to yield uncertainty186 estimates.187 Graph machine learning in wind farm modeling In [ 36 ] a GNN was trained on simulated data188 for wind power prediction . Recently , in [ 37 ] GNNs were applied as a surrogate model to more189 accurate fluid dynamic simulations . With the architectural advancements proposed in this work , we190 extend the wind farm relational modeling literature by providing a solution for representation of191 uncertainty in wind turbine interactions . Moreover , we empirically show in real wind farm data that192 significant accuracy improvements are possible through the incorporation of the proposed relational193 modeling and variational Bayes approach.194 1/NT ∑NT i=1 ( |vTi − v̂i T | ) /|vTi | where vTi is the actual value of node i , NT the number of target turbines and v̂i is the CRVAE prediction . 4.2 Real wind farm SCADA dataset195 Conditional RVAE models ( CRVAE ) were trained with with a 80/20 train/test split on a dataset that196 includes 6 months of 10-minute average SCADA data readings . Since the goal is to compare the197 fitting capability of the models and not model selection , no validation set was employed . Early198 stopping with patience of 2500 steps was used ( test set evaluation every 500 steps ) . The larger RVAE199 models that also yield the best performance had not converged at the 10th epoch . The 20 % of turbine200 data are randomly masked during training . A batch size of 16 was used for all models . In order201 to make a fair comparison no regularization or KL-annealing was used . A small learning rate of202 5 · 10−5 and the Adam optimizer [ 38 ] with default parameters was used for all the runs . The final203 ELBOs for all models are shown in Table 2 . Amean aggregation function and composite aggregation204 function consisting of a concatenation of mean , max and min aggregators were used . Due to the205 concatenation operation , the composite aggregators result in slightly larger networks . Aligned with206 recent results on GN performance when using composite aggregation functions [ 39 ] we find that207 networks with the mean−max−min aggregator indeed yield better performance . The motivation,208 however , for using composite aggregators , is also due to the physics of the problem . By using209 such aggregators it is easier to discriminate the un-waked part of the farm and the waked turbines.210 More concretely , turbines at the upstream boundary of the farm have larger power production and211 this directional effect can easily be masked using the mean aggregation . The CRVAE models are212 compared to a two-layer MLP-based CVAE trained with the arbitrary conditioning objective [ 30 ] of213 varying sizes , with the largest CVAE model number of parameters corresponding to the number of214 parameters of the best performing RCVAE . The largest CVAE model was the worst-performing of215 the evaluated CVAE models.216 The CVAE model with the smallest size has slightly better performance compared to the RVAE model217 that performs no message passing on the encoder part , and therefore ignores relational inductive218 biases when inferring Gz . All but one of the CRVAE models strongly outperform the CVAE models219 by a large margin which is attributed to the effective use of relational inductive biases . To further220 support this claim , in the supplemental material ( section A.1 ) gradient sensitivities are plotted and it221 is observed that the imputation results for masked turbines depend on upstream turbines . Qualitative222 imputation results are shown in Figure 2.223 Effect of inferring edge latents Ez The introduction of224 continuous edge-related latent variables is overlooked in a225 large part of the literature . Wake effect modeling is an appli-226 cation that may benefit from edge latent variables . We test227 the effect of edge latent variables by setting βE = 0 while228 still using Gh . The results of this experiment are shown in229 Table 3 . The inclusion of the KL term with respect to edge230 latent variables seems to improve the reconstruction error231 achieved by the model.232 4.3 Wind farm simulation dataset233 The steady-state wind farm wake simulator FLORIS [ 40 ] was used . A dataset of wake effect234 simulations and preprocessing tools for demonstrating the wind farm modeling approach adopted235 herein is released as part of this work . In what follows we test the generalization capabilities of a236 trained RVAE to novel geometric configurations . A single farm configuration is used for training237 and another one is used for testing . Both farms are simulated with random wind characteristics such238 as direction and average wind speed . An example output from the simulation can be found in the239 supplemental material . The train and test farm configurations can also be found in the supplemental240 material.241 Qualitative results The RVAE model is able to capture the orientation-dependent wake deficit for242 each turbine separately on the test wind farm as shown in Figure 3 . Furthermore , we use a single243 turbine as a probe and position it on a regular grid while keeping a turbine on a fixed position ( 0,0 ) .244 By inspecting the wind speed predicted at the probe turbine , we can map the wake deficit in 2D245 behind the source turbine . This is shown in Figure 4 . The spatial dependence of the wake deficit246 is also shown as computed from FLORIS and the error in RVAE estimation . For distances larger247 than 200m the wind deficit is accurately predicted . Note that this result is from a model trained on248 operational data from a single simulated windfarm . When the turbines are very close ( < 200m ) the249 wakes are not predicted correctly , but this is an expected effect since the RVAE never encounters250 turbines at these distances . Wake effects estimated with the RVAE are slightly lower than those251 derived from the simulation as shown in Figure 3 . However , the RVAE seems to capture the intricate252 wind orientation-dependent effects which depend on the farm layout.253 4.4 1D regression254 In order to further demonstrate the versatility of the RVAE in modeling structured data , and in255 order to make the connection to NPs [ 18 ] clearer , in what follows an RVAE adapted for node data256 imputation is presented [ 30 ] . The dataset consists of sets of points sampled from a zero-mean 1D257 Gaussian process with a squared exponential kernel . The pairs of input points { ( x1 : nm , y1 : nm ) m } 258 are used as node features to construct a set of context and target graphs , where m corresponds to259 different GP realizations . Each graph contains a set of edges ei which encode the relative position260 of the observation points . The edge features between observations at points xi , xj are defined as261 f ( xi , xj ) = e −c·|xi−xj |2 where c is a function of the cutoff distance for edge creation . Note that262 the construction of such edge features endows the model with translation equivariance . In contrast263 to MPNPs , [ 29 ] , the edges are the same in the context and target graphs . The loss function used264 is the same as in Equation 10 . Both pφ and qθ are implemented as GNs . The outputs of the GNs265 parameterize a Gaussian , i.e.,266 qφ ( z|D ) = N ( µφ ( D ) , σ2φ ( D ) ) , pθ ( y|D ) = N ( µθ ( D ) , σ2θ ( D ) ) . ( 11 ) The y values ofDT are replaced with 0 when fed through the encoder and an additional binary feature267 b for the node , which denotes masking , is appended to the node tuple . The b feature is zero for the268 unmasked nodes and 1 for the masked nodes . The masked input is denoted by DT\b . The union of269 the masked target input with the context dataset is denoted by DT\b∪C . Instead of using two different270 functions for the prior of p ( z|DT\b∪C ) as in [ 30 ] , and posterior network q ( z|DT∪C ) and in order to271 keep the conditional RVAE model closer to the NP formulation , the approximate posterior ( i.e. , the272 encoder of the RVAE ) is used also for the learned prior . The decoder pθ receives as node conditioning273 ( and optionally edge conditioning ) the xT values and the global latent variable uz . Each realization274 of uz corresponds to a different context set which in turn corresponds to a different sampled GP .275 More information about he training setup can be found in the appendix.276 The NP is implemented by defining a DeepSet encoder , a global latent variable of the same size277 as the NP MLP . The same latent variable size for nodes and edges was used for each experiment,278 which is the same as the core size . All aggregation functions are mean aggregations . Experiments279 were performed with different number of message passing steps , and inclusion of either the relative280 observation position as an edge feature or the absolute node position xi for each feature . The models281 are tested in un-seen GP realizations and the negative log-likelihood of predictions are reported in282 Table 4 . The RVAE models compute edge , node and global variables . The test datasets contain points283 with x ∈ [ 0 , 1 ] and x ∈ [ 1 , 2 ] ranges in order to test the generalization capability of the proposed284 model in translation . Since the edge-blocks only ever receive translation equivariant inputs from285 the dataset , the RVAE models generalize well in the x ∈ [ 1 , 2 ] range . This is presented only as an286 example of how special equivariant inductive biases may be implemented in RVAE . It is observed287 that the full RVAE model does not perform well when only the node features are available . As with288 NPs , it was empirically found that models yield better results with more training.289 Conclusions and broader impact290 This work introduces an attributed graph approach to the probabilistic modeling of relations within291 entities and their properties . The approach is verified and validated on wake effect simulations and292 actual data from wind turbines placed within a wind farm ; a characteristic example that may be293 modeled as a graph . We also find some connections to the NP literature which we demonstrate by294 adapting the proposed method to perform a typical NP benchmark which is 1D regression for GP295 data.296 We introduce a method for data-driven wake effect modeling for wind farms that accounts for uncer-297 tainty . The proposed method fuses physical intuition , flexible function approximation through GNs,298 and variational Bayes through re-parametrized gradients . Better and more computationally efficient299 wake effect modeling can lead to improvements in terms of accuracy and computational efficiency in300 analysis for wind farm siting [ 41 ] farm layout optimization [ 42 ] , wind farm control optimization [ 43 ] 301 and ultimately power production improvements , as well as more robust to uncertainties maintenance302 planning . Ultimately , the aforementioned lead to wind energy being a more attractive clean energy303 solution.304 Graph data are naturally used to model social , transportation and communication networks . Possible305 negative implications of any graph ML work relate to possible malicious uses of analysis in such306 networks , such as de-anonymization in social networks [ 44 ] , and vulnerability exploitation on307 transportation networks.308 References309 [ 1 ] Justin Gilmer , Samuel S Schoenholz , Patrick F Riley , Oriol Vinyals , and George E Dahl . Neural message310 passing for quantum chemistry . In International Conference on Machine Learning , pages 1263–1272.311 PMLR , 2017.312 [ 2 ] Peter W Battaglia , Jessica B Hamrick , Victor Bapst , Alvaro Sanchez-Gonzalez , Vinicius Zambaldi , Mateusz313 Malinowski , Andrea Tacchetti , David Raposo , Adam Santoro , Ryan Faulkner , et al . Relational inductive314 biases , deep learning , and graph networks . arXiv preprint arXiv:1806.01261 , 2018.315 [ 3 ] Alvaro Sanchez-Gonzalez , Jonathan Godwin , Tobias Pfaff , Rex Ying , Jure Leskovec , and Peter Battaglia.316 Learning to simulate complex physics with graph networks . In Hal Daumé III and Aarti Singh , editors,317 Proceedings of the 37th International Conference on Machine Learning , volume 119 of Proceedings of318 Machine Learning Research , pages 8459–8468 . PMLR , 13–18 Jul 2020.319 [ 4 ] Tobias Pfaff , Meire Fortunato , Alvaro Sanchez-Gonzalez , and Peter W Battaglia . Learning mesh-based320 simulation with graph networks . arXiv preprint arXiv:2010.03409 , 2020.321 [ 5 ] Vinicius Zambaldi , David Raposo , Adam Santoro , Victor Bapst , Yujia Li , Igor Babuschkin , Karl Tuyls,322 David Reichert , Timothy Lillicrap , Edward Lockhart , et al . Relational deep reinforcement learning . arXiv323 preprint arXiv:1806.01830 , 2018.324 [ 6 ] Thomas Kipf , Ethan Fetaya , Kuan-Chieh Wang , Max Welling , and Richard Zemel . Neural relational325 inference for interacting systems . In International Conference on Machine Learning , pages 2688–2697.326 PMLR , 2018.327 [ 7 ] Diederik P Kingma and Max Welling . Auto-encoding bariational bayes . arXiv preprint arXiv:1312.6114,328 2013.329 [ 8 ] Irina Higgins , Loic Matthey , Arka Pal , Christopher Burgess , Xavier Glorot , Matthew Botvinick , Shakir330 Mohamed , and Alexander Lerchner . beta-vae : Learning basic visual concepts with a constrained variational331 framework . 2016.332 [ 9 ] Alexander Alemi , Ben Poole , Ian Fischer , Joshua Dillon , Rif A. Saurous , and Kevin Murphy . Fixing333 a broken ELBO . In Jennifer Dy and Andreas Krause , editors , Proceedings of the 35th International334 Conference on Machine Learning , volume 80 of Proceedings of Machine Learning Research , pages335 159–168 . PMLR , 10–15 Jul 2018.336 [ 10 ] Samuel R Bowman , Luke Vilnis , Oriol Vinyals , Andrew M Dai , Rafal Jozefowicz , and Samy Bengio.337 Generating sentences from a continuous space . arXiv preprint arXiv:1511.06349 , 2015.338 [ 11 ] Hao Fu , Chunyuan Li , Xiaodong Liu , Jianfeng Gao , Asli Celikyilmaz , and Lawrence Carin . Cyclical339 annealing schedule : A simple approach to mitigating kl vanishing . arXiv preprint arXiv:1903.10145 , 2019.340 [ 12 ] Danilo Rezende and Shakir Mohamed . Variational inference with normalizing flows . In Francis Bach and341 David Blei , editors , Proceedings of the 32nd International Conference on Machine Learning , volume 37 of342 Proceedings of Machine Learning Research , pages 1530–1538 , Lille , France , 07–09 Jul 2015 . PMLR.343 [ 13 ] Laurent Dinh , Jascha Sohl-Dickstein , and Samy Bengio . Density estimation using real nvp . arXiv preprint344 arXiv:1605.08803 , 2016.345 [ 14 ] Jenny Liu , Aviral Kumar , Jimmy Ba , Jamie Kiros , and Kevin Swersky . Graph normalizing flows . arXiv346 preprint arXiv:1905.13177 , 2019.347 [ 15 ] Casper Kaae Sønderby , Tapani Raiko , Lars Maaløe , Søren Kaae Sønderby , and Ole Winther . Ladder348 variational autoencoders . arXiv preprint arXiv:1602.02282 , 2016.349 [ 16 ] Qi Wang and Herke Van Hoof . Doubly stochastic variational inference for neural processes with hierarchical350 latent variables . In International Conference on Machine Learning , pages 10018–10028 . PMLR , 2020.351 [ 17 ] Marta Garnelo , Dan Rosenbaum , Christopher Maddison , Tiago Ramalho , David Saxton , Murray Shanahan,352 Yee Whye Teh , Danilo Rezende , and SM Ali Eslami . Conditional neural processes . In International353 Conference on Machine Learning , pages 1704–1713 . PMLR , 2018.354 [ 18 ] Marta Garnelo , Jonathan Schwarz , Dan Rosenbaum , Fabio Viola , Danilo J Rezende , SM Eslami , and355 Yee Whye Teh . Neural processes . arXiv preprint arXiv:1807.01622 , 2018.356 [ 19 ] Mohammadamin Tavakoli and Pierre Baldi . Continuous representation of molecules using graph variational357 autoencoder . arXiv preprint arXiv:2004.08152 , 2020.358 [ 20 ] Martin Simonovsky and Nikos Komodakis . Graphvae : Towards generation of small graphs using variational359 autoencoders . In International Conference on Artificial Neural Networks , pages 412–422 . Springer , 2018.360 [ 21 ] Aditya Grover , Aaron Zweig , and Stefano Ermon . Graphite : Iterative generative modeling of graphs . In361 International conference on machine learning , pages 2434–2444 . PMLR , 2019.362 [ 22 ] Thomas N Kipf and Max Welling . Variational graph auto-encoders . arXiv preprint arXiv:1611.07308,363 2016.364 [ 23 ] Eric Jang , Shixiang Gu , and Ben Poole . Categorical reparameterization with gumbel-softmax . arXiv365 preprint arXiv:1611.01144 , 2016.366 [ 24 ] Chris J Maddison , Andriy Mnih , and Yee Whye Teh . The concrete distribution : A continuous relaxation of367 discrete random variables . arXiv preprint arXiv:1611.00712 , 2016.368 [ 25 ] Danilo Jimenez Rezende , Shakir Mohamed , and Daan Wierstra . Stochastic backpropagation and ap-369 proximate inference in deep generative models . In International conference on machine learning , pages370 1278–1286 . PMLR , 2014.371 [ 26 ] Yann Dubois , Jonathan Gordon , and Andrew YK Foong . Neural process family . http : //yanndubs.372 github.io/Neural-Process-Family/ , September 2020.373 [ 27 ] Hyunjik Kim , Andriy Mnih , Jonathan Schwarz , Marta Garnelo , Ali Eslami , Dan Rosenbaum , Oriol Vinyals,374 and Yee Whye Teh . Attentive neural processes . arXiv preprint arXiv:1901.05761 , 2019.375 [ 28 ] Jonathan Gordon , Wessel P Bruinsma , Andrew YK Foong , James Requeima , Yann Dubois , and Richard E376 Turner . Convolutional conditional neural processes . arXiv preprint arXiv:1910.13556 , 2019.377 [ 29 ] Ben Day , Cătălina Cangea , Arian R Jamasb , and Pietro Liò . Message passing neural processes . arXiv378 preprint arXiv:2009.13895 , 2020.379 [ 30 ] Oleg Ivanov , Michael Figurnov , and Dmitry Vetrov . Variational autoencoder with arbitrary conditioning.380 arXiv preprint arXiv:1806.02382 , 2018.381 [ 31 ] Manzil Zaheer , Satwik Kottur , Siamak Ravanbakhsh , Barnabas Poczos , Ruslan Salakhutdinov , and Alexan-382 der Smola . Deep sets . arXiv preprint arXiv:1703.06114 , 2017.383 [ 32 ] Christos Louizos , Xiahan Shi , Klamer Schutte , and Max Welling . The functional neural process . arXiv384 preprint arXiv:1906.08324 , 2019.385 [ 33 ] Wei Chu Vikas Sindhwani , Zoubin Ghahramani , and S Sathiya Keerthi . Relational learning with gaussian386 processes . In Advances in Neural Information Processing Systems 19 : Proceedings of the 2006 Conference,387 volume 19 , page 289 . MIT Press , 2007.388 [ 34 ] Yin Cheng Ng , Nicolò Colombo , and Ricardo Silva . Bayesian semi-supervised learning with graph389 gaussian processes . arXiv preprint arXiv:1809.04379 , 2018.390 [ 35 ] Naiqi Li , Wenjie Li , Jifeng Sun , Yinghua Gao , Yong Jiang , and Shu-Tao Xia . Stochastic deep gaussian391 processes over graphs . In H. Larochelle , M. Ranzato , R. Hadsell , M. F. Balcan , and H. Lin , editors,392 Advances in Neural Information Processing Systems , volume 33 , pages 5875–5886 . Curran Associates,393 Inc. , 2020.394 [ 36 ] Junyoung Park and Jinkyoo Park . Physics-induced graph neural network : An application to wind-farm395 power estimation . Energy , 187:115883 , 2019.396 [ 37 ] James Bleeg . A graph neural network surrogate model for the prediction of turbine interaction loss . In397 Journal of Physics : Conference Series , volume 1618 , page 062054 . IOP Publishing , 2020.398 [ 38 ] Diederik P Kingma and Jimmy Ba . Adam : A method for stochastic optimization . arXiv preprint399 arXiv:1412.6980 , 2014.400 [ 39 ] Gabriele Corso , Luca Cavalleri , Dominique Beaini , Pietro Liò , and Petar Veličković . Principal neighbour-401 hood aggregation for graph nets . arXiv preprint arXiv:2004.05718 , 2020.402 [ 40 ] NREL . FLORIS . Version 2.2.0 , 2020.403 [ 41 ] JK Lundquist , KK DuVivier , D Kaffine , and JM Tomaszewski . Costs and consequences of wind turbine404 wake effects arising from uncoordinated wind energy development . Nature Energy , 4 ( 1 ) :26–34 , 2019.405 [ 42 ] Nicolas Kirchner Bossi and Fernando Porté-Agel . Multi-objective wind farm layout optimization with406 unconstrained area shape . In Wind Energy Science Conference 2019 ( WESC 2019 ) , number POST_TALK,407 2019.408 [ 43 ] Michael F Howland , Sanjiva K Lele , and John O Dabiri . Wind farm power optimization through wake409 steering . Proceedings of the National Academy of Sciences , 116 ( 29 ) :14495–14500 , 2019.410 [ 44 ] Shouling Ji , Prateek Mittal , and Raheem Beyah . Graph data anonymization , de-anonymization attacks , and411 de-anonymizability quantification : A survey . IEEE Communications Surveys & Tutorials , 19 ( 2 ) :1305–1326,412 2016.413 Checklist414 1 . For all authors ... 415 ( a ) Do the main claims made in the abstract and introduction accurately reflect the paper ’ s contribu-416 tions and scope ? [ Yes ] 417 ( b ) Did you describe the limitations of your work ? [ Yes ] As noted in section 3 the main limitation418 of RVAE is that it can not be used directly as a graph generation model.419 ( c ) Did you discuss any potential negative societal impacts of your work ? [ Yes ] 420 ( d ) Have you read the ethics review guidelines and ensured that your paper conforms to them ? [ Yes ] 421 2 . If you are including theoretical results ... 422 ( a ) Did you state the full set of assumptions of all theoretical results ? [ N/A ] 423 ( b ) Did you include complete proofs of all theoretical results ? [ N/A ] 424 3 . If you ran experiments ... 425 ( a ) Did you include the code , data , and instructions needed to reproduce the main experimental426 results ( either in the supplemental material or as a URL ) ? [ TODO ] The code and dataset will be427 shared within the deadline for providing supplemental materials.428 ( b ) Did you specify all the training details ( e.g. , data splits , hyperparameters , how they were chosen ) ? 429 [ Yes ] see subsection 4.4 and subsection 4.2430 ( c ) Did you report error bars ( e.g. , with respect to the random seed after running experiments431 multiple times ) ? [ N/A ] For experiments in Table 1 and Table 2 the standard deviation of the432 batch statistics for a single run are reported since the differences of the models were pronounced.433 For the results in Table 3 the statistics reported are based on 3 independent runs.434 ( d ) Did you include the total amount of compute and the type of resources used ( e.g. , type of GPUs,435 internal cluster , or cloud provider ) ? [ Yes ] All reported experiments were run on a personal436 desktop running Ubuntu 20.04 , with an nVidia RTX2080 GPU and 16GB of RAM.437 4 . If you are using existing assets ( e.g. , code , data , models ) or curating/releasing new assets ... 438 ( a ) If your work uses existing assets , did you cite the creators ? [ Yes ] Orsted energy , formerly known439 as Dong energy provided us with a proprietary SCADA dataset for the Anholt farm.440 ( b ) Did you mention the license of the assets ? [ Yes ] The Anholt SCADA dataset was made available441 to us under an non-disclosure agreement . The simulated wind farm dataset will be released under442 the CC-BY 4.0 license.443 ( c ) Did you include any new assets either in the supplemental material or as a URL ? [ Yes ] 444 ( d ) Did you discuss whether and how consent was obtained from people whose data you ’ re us-445 ing/curating ? [ N/A ] 446 ( e ) Did you discuss whether the data you are using/curating contains personally identifiable informa-447 tion or offensive content ? [ N/A ] 448 5 . If you used crowdsourcing or conducted research with human subjects ... 449 ( a ) Did you include the full text of instructions given to participants and screenshots , if applicable ? 450 [ N/A ] 451 ( b ) Did you describe any potential participant risks , with links to Institutional Review Board ( IRB ) 452 approvals , if applicable ? [ N/A ] 453 ( c ) Did you include the estimated hourly wage paid to participants and the total amount spent on454 participant compensation ? [ N/A ] 455

Summary: This paper proposes a generalized formulation of a probabilistic model for graph-structured data that incorporates graph networks (GN) (or rather, provides a probabilistic extension thereof). The authors show that the proposed framework is general enough to subsume existing approaches such as Neural Relational Inference (NRI) and Neural Processes (NP). The method is empirically validated on simulated and real-world data, with emphasis placed on the modelling of wind farm data.