Integrating Bayesian Network Structure into Residual Flows and Variational Autoencoders
Authors: Jacobie Mouton
Rodney Stephen Kroon
Venue: Accepted by TMLR ()

Abstract: Deep generative models have become more popular in recent years due to their scalability and representation capacity. Unlike probabilistic graphical models, they typically do not incorporate specific domain knowledge. As such, this work explores incorporating arbitrary dependency structures, as specified by Bayesian networks, into variational autoencoders (VAEs). This is achieved by developing a new type of graphical normalizing flow, which extends residual flows by encoding conditional independence through masking of the flow’s residual block weight matrices, and using these to extend both the prior and inference network of the VAE. We show that the proposed graphical VAE provides a more interpretable model that generalizes better in data-sparse settings, when practitioners know or can hypothesize about certain latent factors in their domain. Furthermore, we show that graphical residual flows provide not only density estimation and inference performance competitive with existing graphical flows, but also more stable and accurate inversion in practice as a byproduct of the flow’s Lipschitz bounds.

and Published in Transactions on Machine Learning Research (04/2023)
Integrating Bayesian Network Structure into Residual Flows and Variational Autoencoders
Jacobie Mouton
1hout123@gmate com
Computer Science Division
Stellenbosch University
South Africa
Steve Kroon
kroon www..acccaaaa.
Computer Science Division Stellenbosch University
National Institute for Theoreticol and Compulational Sciences South Africa
Reviewed on OpenReview: https: // openreview. net/ permffrmm? id=0sKXlwant
Abstract
Deep generative models have become more popular in recent years due to their scalabil- ity and representation capacity. Unlike probabilistic graphical models, they typically do not incorporate specific domain knowledge. As such, this work explores incorporating ar- bilrary dependency structures, specified by Bayesian networks, into variational autoen- type of graphical normalizing flow, coders (VAEs). This is achieved by developing independence through masking of the which extends residual flows by encoding conditional flow's residual block weight matrices, and using these to extend both the prior and inference network of the VAF. We show that the proposed graphical VAE provides a more inter- pretable model that generalizes better in data-sparse settings, when practitioners know or can hypothesize about certain latent factors in their domain. Furthermore, we show that graphical residual flows provide not only density estimation and inference performance com- petitive with existing graphical flows, but also more stable and accurate inversion in practice as byproduct of the flow's Lipschitz bounds.
1 Introduction
Normalizing flows (NFs) RRzendee & Mohamed, 2015 Tabak & Turner 2013 have proven be useful tool in many machine learning problems. Typically parameterized by neural networks, NFss represent complex probability distributions as bijective transformations of simple base distribution, while tracking the change in density through the multivariate change-of-variables formula. Variational autoencoders (VABs) (Kingma & Welling 2014 Rezende et al. 2014) also provide a powerful framework for constructing deep latent variable models, and unlike bijective flows, allow practitioners to specify the dimensionality of the latent space. By positing and fitting generic model of the data-generating process, VAΣs allow one generate new samples and reason probabilistically about the data and its underlying representation. Despite the success of VAEs, they typically use overly simple latent distributions, e.g. fully-factorized Gaussian distributions for both the prior and variational posterior. Subsequent work has explored incorporating more complex latent variable distributions and have shown that this results in improved performance. For example, NTs can be included as part of the VAE's encoder network (Kingmaa et al., 2016), entangling the latent variables non- non-linearly to obtain a richer class of approximate posteriors. The prior distribution can also be made more complex, for example by stacking layers of latent variables create a hierarchical structure (Sonderby et al., 2016). This increases the flexibility of the true posterior, leading to improved empirical results (Kingma al. 2016

---

Published in Transactions on Machine Learning Research (04/2023)
In contrast to data-driven deep generative models such as NEs and VAEs, Bayesian networks (BNs) model distributions as structured product of conditional distributions, allowing practitioners to specify both ex- pert knowledge and quantitative information in simple and interpretable manner. The aim of this work is to combine the strengths of both: the simplicity and interpretability of BNs, and the scalability and representation capacity of deep generative models. Traditional VAEs, as well as those with flow-enriched inference networks and/or stacked layers of latent variables, do not allow one to directly control the depen- dence structure encoded by the model. We therefore propose an approach to incorporate rich conditional distributions for arbitrary dependency structures into VAEs. We do this by extending both the prior and inference network with graphical NFs (Wehenkel & Louppe 2021 Weilbach et al.) 2020), which allow one to encode an arbitrary BN structure into these distributions through the NF architecture.
The flow making up the prior of such a structured VAE will be used in both transformation directions in the normalizing direction for density estimation and the inverse direction for sample generation. However, existing graphical flows do not emphasize providing stable and efficient inversion. While NFs are theoretically invertible, stable inversion is not always guaranteed in practice (Behrmann al., 2021): if the Lipschitz constant of the inverse flow transformation is too large, numerical errors may be amplified. To address this, our work proposes a new graphical flow that encodes domain knowledge from a BN into a residual flow (Chen et al., 2019). Residual flows ensure theoretical invertibility by imposing a Lipschitz bound on the transformation. Graphical residual flows (GRFs) encode a predefined dependency structure by masking the residual blocks' weights, and obtain stable inversion in practice as a byproduct of the Lipschitz constraint.
The contribution of this work is therefore two-fold First, we propose graphical residual flows as graphical NF that can stably and efficiently be inverted in practice. We compare the GRF to existing approaches on both density estimation and inference tasks and confirm that this method yields competitive performance. Our model exhibits accurate inversion that is also more time-efficient than alternative graphical flows with similar task performance. GRFss are therefore an attractive alternative to existing approaches when flow is required to perform reliably in both directions. Second, we propose a structured VAE, termed the structured invertible residual network (SIReN) VAE, that employs GRTs to encode a predefined dependency structure over the latent and observed variables. We identify posterior collapse (Razavi et al.) 2019 where some latent dimensions become inactive and are effectively ignored by the model-as an issue with STREN-VAE, as this phenomenon is influenced by the encoded structure. We consider various existing techniques to alleviate this phenomenon, and show that they lead to improved performance. Finally, we empirically show this model's potential for better generalization in data-sparse settings, as well as its ability to provide more interpretable latent spaces when practitioners know or can hypothesize about latent factors in their domain.
2 Background & Related Work
2.1 Bayesian Networks
Let P be the joint distribution over variables X' = independent given Xx in P, if P(X,, X,,xx, = P(xx)x))((xxxxx)) Now, let g be a DAG with vertices corresponding to elements of X, and let Pa denote the parent vertices of X; in G. P factorizes according to 9 if P can be expressed as the following product of conditional distributions (Koller & Friedman 2009): P(x)= = Ix.ex )..)). In this setting, the BN graph G provides a compact encoding of various conditional indopendence assumptions about P(X), as indicated by the absence of dirccted edges between certain vertices. We aim to capture these conditional independencies the generative models we construct.
We say that X; and Xj are conditionally
2.2 Normalizing Flows
An NF (Rezende & Mohamed, 2015 Tabak & Turner 2013 consists of a simple base distribution and a bijective transformation that provides mapping between this base distribution and a more complex data distribution. The incurred change in density is tracked via the hhange-of-variable formula. For density
1 An implementation of the GRF and STReN-VARE as well as our experimental code, can be found at httpt::/tttllbb.coooo Bleased/grf-ane and-rr-----------veeee
2

---

Published in Transactions on Machine Learning Research (04/2023)
estimation, an NF Fx-1()), is used to transform a sample from the data distribution, p, to sample from the base distribution, Po (Papamakarios et al. 2017]. The sample's density is then given by
logp(x) = logpo(fx=>x(x)) log [det (J)x,.(())]
(1)
where JF(a) denotes the Jacobian of the transformation F at a. For amortized variational inference, where the aim is to infer the latent posterior z ~ q(-1x) for a given observation x, the flow Fe-z(-) instead models the generative direction of the mapping: it transforms a sample from the basc distribution to a sample from the latent posterior (Kingma el al. 2016). The density of the generated sample is given by
log (z)x) logPo(E) log |det (J
(2)
Flows are typically trained to implement a specific transformation direction, depending on the task. The name normalizing flow in reference to the procedure of letting a variable 'flow" through a series of transfor- mations that 'normalizes' a complex data distribution into a simpler known base distribution. We refer to a flow that instcad transforms (samples from) the base distribution to (samples from) the data distribution generalive flown The need for accurate and elficient inversion arises when using either Fx- to generate new X from E ~ Po: or when using Fe-x to compute the density for a z not generated by the flow. In principle, the trained flow can be inverted either analytically (if possible) or by using numerical methods. However, inversion may be slow or numerically unstable unless suitable modelling choices are made.
Recent approaches to constructing NFs can be divided into two main categories: finite and continuous (or infinitesimal) flows. Finite flows (Tabak & Turner 2013) create a complex bijective mapping by composing simpler transformations- which we refer to as flow steps wwhile continuous flows (Chen et al., 2018 define the flow transformation implicitly in terms of an ordinary differential equation (ODE). One type of finite flow cnsures tractable computation of the Jacobian determinant by enforcing an autoregressive dependency structure over the variables such that the Jacobian is triangular. Another type of finite flow, known as a residual flow, applies the update x(++1) = x(1) + (x())) at each flow step t and ensures invertibility and tractability by applying suitable restrictions to Jt- Since one type of residual flow, known as a contractive residual flow, forms the basis of our proposed graphical residual flow, we discuss it in more detail below.
2.2.1 Contractive Residual Flows
Behrmann et al. (2019) show how to construct a finite NF by changing the normalization scheme of a traditional residual network's weights. Consider a residual network (He al. 2016), F(x) = offo....ff composed of blocks x(++1) := (x(()) = x(9) + 9x(x()), with x()) x. 1' is invertible if all of its component transformations are invertible, which holds if all ge are contractive, i.e. Lip(g.) < 1, where Tip(-) denotes the Lipschitz constant for transformation. [ehrmann et al. (2019) ensure Lip(g)) < 1 by implementing g: as composition of activations h with Lip(h) < 1 (e.g. LipSwish (Chen et al., 2019 ) and affine layers with weight matrices W2 satisfying [W |ls < 1. Here, || II> is the spectral norm, which can be computed with a power iteration (Gouk et al.) 2021 The spectral norm of W; is constrained to [0, c] by normalizing W2 as
W
W = min(m, [[W]la))
(3)
||
A (non-trivial) upper bound the Lipschitz constant of a contractive residual block g. also implies Lipschitz bounds for fr and f- the Lipschitz constant of the forward mapping, Lip(ft), is upper bounded by 1 1 Lip(g)), while the Lipschitz constant of the inverse mapping, Lip(f="). ), is upper bounded by 1/(1 - Lip(g))). This bi-Iipschiyiivity is an attractive property for stable and efficient inversion (Behrmann et al. 2021) To allow scaling to highor dimensions, Chen et al. (2019) use a "Russian roulette" unbiased and tractable estimate for the Jacobian determinant. A drawback is that it has unpredictable time and memory usage.
2.2.2 Normalizing Flows with Graphical Structures
Incorporating the dependency structure of a BN into NFs has bogun to receive attention in the past two years. For finite flows, Wehenkel & Louppe (2021) consider only autoregressive flows where each step can be
2:Normalixing flow widely used for flows in both directions, but make this distinction for clarity.
3

---

Published in Transactions on Machine Learning Research (04/2023)
constructed from normalizer and conditioner function. A normalizer applies bijective transformation to its input and is partially parameterized by the output of the conditioner, which controls the dependen- cies between the variables such that the Jacobian of the flow step is triangular. They consider two specific normalizer functions. The first is the affine transformation: f(x = exp(s) O x m, where @ denotes element-wise multiplication. Here, (m,s) s) is the putput of the (arbitrarily complex) conditioner. To encode the dependency structure, the conditioner is constrained such that (m4, is only a function of z;'s parents in the corresponding BN, denoted by XPa(s): for i = I,.... D. Second they consider the monotonic normal- izer [Wehenkel & Louppe 2019): fi(]//c,)) = Jo² 9:(4, c.) di + (c:),, where c; is the conditioner output, and 9s and 3, are two neural networks with strictly positive and a real scalar output, respectively. Although one could implement. D separale conditioner functions that each take different subset of variables, XPa(t): as input, Wehenkel & Louppe (2021) choose to use only single neural network. To ensure that the correct independencies are still maintained, they perform D passes through this neural network, masking out those inputs during forward pass i that are not in XPa(t)-
Weilbach et al. (2020) propose incorporating a graphical structure into continuous NFs. They consider a neural ODE system (Chen et al. 2018), xxx = fxx, where the layers of the neural network f take the form h(x, 0) tanh
ih (U] bo n2((), where the bΙ are time-dependent linear gating functions. They incorporate graphical dependency structure by applying a binary adjacency matrix mask to the weight matrix W. This mask ensures that output i is dependent on only the inputs corresponding to æ; and its parents in the corresponding BN at each layer in the network. A drawback of this masking approach is that it restricts the width of each layer of the neural network to be the same as the dimension of x
2.3 Variational Autoencoders
Let po(x,2) z) be deep latent variable model over observed variables x and latent variables z, parameterized by neural networks with parameters 0. Optimizing 0 using maximum (marginal) likelihood estimation is intractable in this setting bocause one typically cannot easily compute the evidence, (x).. By employing amortized variational inference, one can however maximize the evidence lower bound (ELBO):
(x 2 CELBO (x) = Ezzag [log z) - log go
(4)
where the inference network with variational parameters & outputs an approximation go to the true posterior p(z)x). The resulting model, that simultancously optimizes 0 and ø, is known as a variational autoencoder (VAE). The vanilla. approach to constructing a VAE assumes that all the latent variables are independent in the prior and conditionally independent given x in the approximate posterior, and also uses simple distributions for the prior, likelihood and posterior:
p(z) _N(0,
μu, log σ = EncooderNeurrlNeet(( ()
μ, log DecoderNeuralNNet(2:.))
46 xz) = N(H, diag(o²))
(x(x NNu,, diag(o"))
where DecoderNeuralNett and Encaaer/eeuraleet are neural networks parameterized by and o, respectively.
3 Graphical Residual Flows
Assume a contractive residual flow F = fr f1, where each residual block It, 1. -= 1,..., fully-connected neural network with a single hidden layer o and activation function A(-), where Lip(h) < 1:
T, is
(((x(t=1)) = (t-1) We h(W1 (t-1)) + b1) + b2
such that Lip(g:) < 1. Similar to the work of Wehenkel & Louppe (2021) and Weilbach et al. (0200). we can encode the graphical structure of a BN, by ensuring that oulput i of each Jr is only a function of those inputs corresponding to x, and its
Here, W; indicates a normalized weight matrix as in Equation
Wellbach al. (020)) therefore suggest incorporating additional auxiliary variables. The rest. this discussion is easily extended to residual blocks with more hidden layers.
4

---

Published in Transactions on Machine Learning Research (04/2023)
parents in the BN graph. This can be achieved by suitably masking the weight matrices of each of the above residual blocks before applying spectral normalization. Given a BN graph, 9, over the components of x E R2, let W = W; C M,, i == 2, be the new masked weight matrices where the M, are binary masks ensuring that component j of the residual block's output is only a function of the inputs corresponding to {2z} U The update to ((*11) in block Se is then defined as follows:
x (0) = (t-1) + W3 h(W1 x(-1++))) + lb
(6)
The masks are constructed according to a new variant of MADE (Germain et al. 2015) which allows one to encode not only autoregressive, but arbitrary graphical structures. To construct these masks, we assign a specifie subset of variables to each unit in the neural network: input units are assigned their corresponding unit sets { and output units are as- signed the sets { U Pay.- Each hidden unit is randomly assigned one of the sets associated with the input and output units. To ensure each output unit has at least one valid path connecting it to the impul, we ensure that each of the sets {1;) least one unit in each hidden layer. See Figure [ for an illustrated example. A mask is then constructed by zeroing out weight between two units in suc- cessive layers if the set assigned to the unit in the later layer is not a superset of the set assigned to the unit in the earlier layer Our proposed masking scheme overcomes the shortcom- ings of those used by Wehenkel & Louppe (2021) and Weilbachi et al. (2020) in that requires only single pass through the network and allows arbitrary hidden layer widths. Since we are enforcing the BN's DAG structure between the flow's variables, autoregressive structure. If the variables were ordered according to their topological ordering in the BN, the update applied to each variable would only be conditioned on variables with a strictly lower index (though not necessarily on all variables with a lower index). Unlike standard residual flows, this construction results in a triangular Jacobian for which the determinant is easy to compute ezactly as the product of the matrix's diagonal terms.
x
(m1)
(21.22.00 22
[4)
W
{2) (2) 22
na)
(ay)
(3g
124)
i= L....., D, is assigned
W
{ml}
(22)
(ma)
{ma)
Figure 1: The update to x(t-1) at flow step of a GRF. Edges removed by the masks are not shown. The remaining edges encode the structure of the given BN.
are in effect encoding 'sparse'
Inversion
The inverse of this flow does not have an analytical solution (Behrmann et al. 2019 Instead,
each block can be inverted numerically using Newton-like fixed-point method (Song et al. 2019 To compule x = -y), the following update is applied until convergence:
( =x (x) - a (diag(JJ, (xiag((,(((()))) (x()))) [((x())))]
using the initialization (0) = y and letting 0 < a < 2 (which ensures local convergence (Sogg et al. 2019 We use this instead of the Banach fixed-point approach employed by Behrmann et al. (2019), since the convergence rate of the latter is dependent on the Lipschitz constant of the residual block and takes longer to converge for larger bounds. We empirically demonstrate that the convergence rate of (7) does not have this dependence (see Appendix B.3.I). It is thus preferable if we wish to use larger Lipschitz bounds for the residual blocks, which allows for more expressive flow steps. Although T only guarantees local and not global convergence like the Banach approach, we did not find this to be issue with the datasets used.
Induced Dependency Structure For a single-step normalizing GRF, the conditional independencies of the distribution learned by the flow will correspond exactly to the conditional independencies specified by the encoded BN (Wehenkel & Louppe 2021). When multiple flow steps are composed, additional dependencies are however induced-ing in multi-step GRF each variable will ultimately receive information from all of its ancestors in the BN via the intermediate representations of its parents previous steps, with the induced dependency structure corresponding to the BN's transitive closure. Because this 'information leakage' is
5

---

Published in Transactions on Machine Learning Research (04/2023)
only through intermediate representations of each variable's parents, nonetheless still expect GRFs to incorporate a strong enough inductive bias to encourage the variables to adhere to the desired dependency structure. See Appendix A.3 and B.3.3 for details and empirical results supporting this argument.
Variational Inference In the case where latent variables z are present, one typically only has access to the forward BN that models the generating process for an observation x. That is, the BN generally encodes the following generic factorization of the joint: p(x, 2) = -(x(z)p(Z)) To perform inference, we first invert the BN structure using the faithful inversion algorithm of Webb et al. (2018) to obtain BN structure encoding domain knowledge about the dependencies between the components of z once x is observed. This allows us to construct a GRF over the latent variables that is conditioned on the observation: each block is constructed as (0() := (( + W/2 hwW, (z(t)) @x) + b1) + b2, where @ denotes concatenation. Such a GRF is typically applied in the generative direction to infer the latent representation of a given observation.
LipMish Activation Function Since Equations 1 and ( contain the derivatives of the residual block activation functions through the Jacobian term, the gradients used for training will depend on their sec- ond derivatives. It is thus desirable to use smooth non-monotonic functions that adhere to the Lipschitz bounds (such as LipSwish |Chen et al. 2019)), because unlike common monotonic activations, these typically do not have a vanishing second derivative in the region where the first derivative approaches 1. In our model, we use an activation we call "LipMish": LipMish(x)) = (a/1.088) -tanh paothssolllussoofiiii plus(softplus(() -z)). which is a scaled version of the non-monotonic Mish function (Misra 2020), ensuring that Lip s 1 for all B, where B allows for different degrees of curvature and is passed through softplus(-) to ensure it is nonnegative.
4 VAEs with Structured Invertible Residual Networks
If we wish to construct VAE, and have prior knowledge about the data-generating process, then it seems beneficial to incorporate this knowledge in the VAE. In this work, we assume access to a BN specifying the dependency structure over D observed and K latent variables. Our goal is to suitably incorporate this information into the VAE's encoder and decoder networks. Using @ for the decoder network's 's parameters, this mcans that its likelihood po (x())) and prior (z) should idcally factorize according to the BN's conditional independenncies. However, Webb et al. (2018) also showed the value of encoding the generative model's true inverted dependency structure as far as possible in the VAE's inference network. That is, the structure of the variational posterior should respect knowledge about p(z(x) which can be deduced from the factorization of p(x, z). Approximating the posterior p(z[x) in such way requires inverting the BN (so that edges go from x to z) while taking into account the independencies encoded by the model. As discussed in the context of inference with flows, we use Webb et al. (2018)'s 's proposed algorithm for obtaining such a faithful inverse.
We use GRFs to incorporate these desired structures into the generative and inference network of a VAE, yielding the structured invertible residual network (STReN) VAF. For an observed sample x, the encoder network (below left), with parameters o, is defined as a GRF conditioned on x. The subscript g denotes that this is a generative flow, and for all our investigations, we set Po to (O,, For sample z from the encoder network, the decoder (below right) uses normalizing flow for the prior density, and fully-factore factored Gaussian likelihood, with parameters output by a network denoted by DecoderNN:
log (z) log Po (GRF, (2; 0)) + log (eet (2))
z = GRF g( (<; where c~ Po log (z|x) = log - log (ee( (ctt(JaRF,(E)
μ. log DecoderNN(z; 9)
p lz) (N diag((a))))))
The subscript n above denotes normalizing flow. To encode the conditional independencies between the latent and observed variables, DecoderNN is also masked according to the scheme discussed for GRFs. Note that GRF, must be inverted to generate samples from this VAE, making sample generation slower than with regular VAEs. A key benefit of GRFs over other graphical flows, however, is that they guarantee stable inversion. The inversion time per flow step of a GRF is also relatively low compared to other graphical flows with similar modelling capability. This motivates our use of GRFs instead of her existing graphical flows.
6

---

Published in Transactions on Machine Learning Research (04/2023)
g(z/x) po(e) det(JRRF, (E))
E
GRF
pla 2)
pole) [det(JeRF_
Z
GRE
GRE
Inv. B.v.BN
Z
BN
GRE
DecoderNN
Po
X
X
Figure SIReN- VAE encodes the BN's graphical structure into the decoder (right) via masking of the normalizing GRF (GRF,) and decoder neural network (DecoderNN) weights. The inference network (left) similarly encodes the inverted BN structure in its generating GRF (GRF,).
5 Experiments
We evaluate the proposed GRF and STReN-VAE models range of synthetic and real-world datasets that each have associated true hypothesized BN graph. The synthetic datasets are generated from fully specified BNs. All models were trained using the Adam optimizer with an initial lcarning rate of cither 0.01 or 0.001 and batch size of 100. The learning rate was decreased by a factor of 10 each time no improvement in the loss was observed for a set number of consecutive epochs, until a minimum learning rate of 10-6 was reached, at which point training was terminated. The initial learning rate and duration before learning rate reduction was chosen based on the lowest validation loss obtained over the grid {0.01, 0.001) {10, 20, 30).
5.1 Graphical Residual Flows
We first compare GRF to two existing approaches- the graphical NF (GNF) of Wehenkel & Louppe (022 for which we consider both affine and monotonic normalizers (denoted by GNF-A and GNF-M, respectively) and the structured conditional continuous NF (SCCNF) presented in Weilbach al. (2020). Our experiments use three synthetic datasets: the Arithmetic Circuit dataset (Weilbach et al. 2020 Wehenkel & Louppe, 2021), adaptation of the Tree dataset (Wehenkel & Louppe] 2021 well as linear Gaussian BN, EColi, adapted from the repository of (Scutari (2022). We also consider two real-word datasets, namely Protein (Sachs et al. 2005) and MEHRA (Vitolo et al. 2018). Further information is given in Appendix B.1 To provide more informative comparisons between the flows, we train two models per task for each approach. The first is a smaller model with a maximum capacity of 5000 trainable parameters, denoted by a subscript S, e.g., GRFs. We also train larger model with maximum capacity of 15000 paraneters, denoted by the subscript L. For further information on the flow architectures, see Appendix B.2 Our experiments consider the relative performance of the flows with respect to tasks in both the generative and normalizing directions, well as the efficiency and accuracy of flow inversion.
5.1.1 Density Estimation & Inference
Table ! provides the negative log- likelihoodd (NLL) and the negative ELBO achieved by each model on the test set of the various datasets for the density estimation and variational inference tasks, respectively. For density estimation we assumed all variables were obscrved. Inference is only performed for the synthetic datasets, since we require access to the true model p(x, 2) to compute the ELBO. We find that GRFs provide performance competitive with GNF-M and SCCNF, with the GRF models achieving the best NLL and ELBO on the majority of the datasets. GNF-A, with its reliance on simple affine transformations, is unable to match the performance of the other approaches for these primary modelling tasks.
7

---

Published in Transactions on Machine Learning Research (04/2023)
Table Density estimation and inference performance. NLL and -ELBO are the averages on the test set (lower is better) over five runs, with standard deviation given in the subscript. Bold indicates the best result in each group. The number of edges (E) and observed (D) and latent (K) variables in the datasets' associated BNs are also provided. Inference was not performed the real-world datasets (where K = 0).
Density Estimation (NLL.)
Variational Inference (- (-ELBO)
BN
D
K
E
Flow
Small Budget
Large Budget
Small Budget
Large Budget
Arithmetic Circuit
GNF-A
1.2620.02
10.18
1.90
1.59
GNF-M
1.1940.07
1.14
3.9610.19
3.92:0.08
2
6 8
SCONF 0.86=0.01
0.000
4.01
3.97
GRF 1.25
1.11
4.19
3.71
GNF-A 9.32+0.00
2.32
2.3620.04
2.3840.05
Tree
GNF-M
8.65
8.65
1.72:0.01
1.70
SCONF
8.59
8.000
1.7840.01
1.7640.01
GRF
8.64.0.01
8.64g0.00
1.7440.00
1.71
GNF-A
1.93
6.92
Protein
GNF-M
3.00
5.48
20
SCCNF
4.88
-5.60+0.06
GRF
5 5.2640.02
-6.11
GNF-A
40.11
40.11
34.98=u.00
34.98
EEColi
GNF-M
40.13+0.01
40.13
34.99
34.98
29
15
59
SCCNF
40.12
40.08
35.24
35.24
GRF 40.0620.00
40.06
34.96.0.00
34.96=0.00
MEHRA
GNF-A 12.90
12.93
GNF-M
11.7440.02
11.67:0.02
10
0
10
SCONF
11.80
11.76g0.02
GRF 11.66
11.61+0.08
We also compared the density estimation performance of GRF to the vanilla residual flow (RF) of Chen et al. (2019). The results are given in Table 2 The chosen RF architectures are similar to those of the GRFs they are being compared against: only the residual block hid- den layer widths were reduced to ensure that the models comply with the respective size budgets. GRF performs the best on the majority of the datasets, specifically the synthetic datasets where the true dependency structure is known, but also on the real-world Protein dataset in the large model setting. We also found that GRFs typi- cally required less time to perform density estimate cal- culations than REs, which allowed for faster training. This to be expected, given that the Jacobian deter- minant for the GRFss can be computed directly, whereas RFs must use the Russian roulette estimator. However, GRF is noticeably outperformed by RF the real-world MEHRA datasct. Note that GRF has the best perfor- mance of all the graphical flows on this dataset, suggesting that its poorer performance may be a result of the BN not fully capturing the dependencies in the data, rather than a shortcoming of GRF. The clear supe- riority of RF over the graphical flows may cause one question the veracity of the assumed BN structure. In such case, the difference in results between vanilla RFs and GRFs can serve a prompt further investigate and refine the BN structure.
Table 2: NLL achieved by GRF compared to the vanilla residual flow (RF) of
Chen al. 2019
BN
Flow
Small Budget
Large Budget
Arithmetic Circuit
GRF
10.5
10.1
RF
1.27
1.20
GRF
8.64
8.64
Tree
8.6640.02
8.62
GRF
5.2622.01
6.11
Protein
RF
5.7820.13
55.4
GRF
40.06-2.00
40.064t..0
EColi
RF
40.30
40.12.0.12
GRF
11.6644.oz
11.61
MEHRA
RF
8.68+0.06
8.67
8

---

Published in Transactions on Machine Learning Research (04/2023)
Table 3: Inversion performance on 100 test samples from the Tree dataset. Bold indicates the best results in each column. N and a are not applicable for SCCNF. Ranges indicate different optimal settings for N and a for different samples. Inversion time is measured for the smallest N - 50 that allowed the most samples in the batch to have reconstruction error of less than 10-4, and is the time taken to invert the entire batch.
Small Budget
Large Budget
Converged
Inversion time (ms)
Converged within 50 steps
Inversion time (ms)
Flow
NV
N
within 50 steps
GNF-M
100
1
0.9 1.0
53.67
98
4 -47
0.4 1.0
488.51
SCCNF
98
140.51
94
392.97
GRF
100
3-5
1.0
49.73
100
4-6
0.9-1.0
122.27
5.1.2 Inversion
The main reason for introducing GRFs is their potential to provide more accurate inversion. Having estab- lished that they provide competitive key task performance, we now invostigate the inversion accuracy and efficiency of GRFs compared to alternative graphical flows with similar task performance, namely GNF-M and SCCNF. GNF-M and GRF were inverted using the Newton-like inversion procedure in 7 while SCCNF was inverted by executing the integration in the opposite direction. For GNF-M and GRF, we considered values for the step-size a in the set {0.1 x t t = 1,....19) 19), and the inversion process was deemed to have converged when reconst rection error of less than 10 was achieved. To better illustrate potential inversion instability, we performed this inversion on a per-data-point basis for 100 test samples from the Tree datasot; the results for the other datasets are given in Appendix [B..1] For each data point, we note the that required the fewest iterations, N, for convergence. Table 3 summarizes these results, including the number of samples for which the desired reconstruction error was achieved for GNF-M and GRF with N s 50. We also include the inversion time for the entire batch, using the settings that allowed the most samples to achieve the desired reconstruction error.
One of the main paradigms for enforcing global inversion stability using Lipschits-coonstranned flow trans- formations (Behrmann al.) 2021). For GRFs, this stability is automatically achieved as a byproduct of the flow design, and we see that GRF shows excellent inversion accuracy. GNF-M, depending on the archi- tecture and learned weights, has either potentially very large Lipschitz bounds, no global bounds at all. This helps to explain its poorer inversion results. As illustrated in Figure 3 numerically inverting GNF-M can lead to very large reconstruction errors. While SCCNFs have global Lipschitz bounds, these are not controlled during training and numerical instability can occur. For further information see Appendix
1040
GNF-M;
1040
GRF,
Reconstucton error
107
107
0.7
10*
Reconstuction
10
1.0
1.2
10'
10
10-2
10-2
10a9
10a9
1
6
11
16
26
31
36
41
46
50
1
6
11
21
26
31
36
41
45
Number of iterations per flow step
Number of Iterations per flow step
Figure 3: Reconstruction error when inverting GNF-M and GRF on 100 test samples from the Tree dataset. Note the large reconstruction error of GNF-M (left) for different values of the step-size, a, compared to the fast convergence of GRF (right) to the correct inverse.

---

Published in Transactions on Machine Learning Research (04/2023)
5.2 SIReN-VAL
As with GRFs, we evaluate the SIReN-VAB model various synthetic and real-world datasets that each have an associated BN graph. For each BN, all leaf nodes are considered observed, and the rest are taken to be latent. Since VAËs are typically used to encode information into a lower-dimensional representation, we only consider the datasets for which there are fewer latent than observed variables, ie. EColi and MEHRA. Additionally, we use an altered version of the Arithmetic Circuit dataset, where the number of obscrved variables has been increased from 2 to 10, as well as a. larger lincar Gaussian BN, Arth (Scutari, 2022). To better compare the effect of the encoded structure, we train three SIReN-VAE models that each incorporate different structure. These are either fully-connected (SIReN-VAEFc) random (SIReN- VAERand), or adhere to the true dependencios (SIReN-VAErrue) For STReN-VAEpe, each observed variable is conditioned on all latent variables. SIReN-VAERand encodes a BN graph with the same number of edges as the graph encoded by STReN-VAFTToe but where these edges have been assigned to random pairs of vertices. All models used the same latent dimension as SUReN-VAETrur Details on the model architectures for each dataset is given in Appendix B.2 We also compare the results to those of vanilla VAE, and a model (VAE+IAF+ME posterior were implemented with masked autoregressive flow (MAF) (Papamakarios et al., 2017) and an inverse autoregressive flow (IAF) (Kingma et al. 2016), respectively. The decoder network of both these models, as well as the encoder network for the vanilla VAE, had similar architectures to the DecoderNN used in the STREN-VAF, models, i.e., the same activation functions, number of hidden layers and hidden layer widths were used. The MAF and IAF each consisted of three transformation steps, where the order of the input features was reversed in successive steps, as recommended by Kingma et al. (2016).
that is similar to STReN-VAFFC except that the latent prior and approximate
5.2.1 Performance in a Data-sparse Setting
We compared training the models on the full training sets of the respective datasets against using much smaller training sets consisting of only 2 x [g] instances, where [g] = D K. We noted each model's average negative log-evides log-evidence on the test set over five independent runs. The log-evidence per test point is estimated
using 50 importance-woighted samples as in Burda ct al. (0066). We also determined the average number of latent variables that collapsed during training. To measure whether a specific variable has collapsed, we use the statistic presented by [urda et al. (2016): A. Varx<<po (Ez=g(=(x)[22), This is based on the assumption that if latent dimension encodes uschul information about the data, then its posterior mcan would be expected to vary as the observations change. As in Burda et al.. (2016), is deemed inactive if A, S 0.01. These results are given in Table + In these initial experiments, STReN- is not able to match the performance of HIReN-VAEPe in terms of log-evidence on most of the datasets when enough training data is available. Furthermore, using GRFs to construct the latent space for fully-connected SIReN-VAES in this setting provided comparable performance to using affine autoregressive flows with feature shullligg bet ween flow steps. In all cases, STReN-VAFRand did significantly worse. This shows that the encoded structure does play significant role in modelling performance and that using the true (or hypothesised) BN structure aids in learning appropriate observational and latent distributions. When only limited training data is available, STReN-VAFTue clearly outperforms the other models, and achieves a higher log-evidence on most of the datasets. We speculate that the increased sparsity of the neural network weights, in line with the true BN independenciess poses an easier learning task and results in better generalization performance.
5.2.2 Addressing Posterior Collapse
As seen in Table 4 SIReN-VAETuue suffers from posterior collapse. We are especially motivated in this setting to avoid this in order to learn meaningful latent distributions in line with the provided BN structure. We first investigate whether the position of a latent variable within the BN graph affects the likelihood that it will collapse, which would suggest that the encoded structure further contributes to inactive latents apart from known causes. We visualize in Figure g the logarithm of the average value of A₂ for each latent variable over the five runs used to generate the results in Table 4 for the two datasets in which posterior collapse occurred for the full training set. If the encoded structure plays no role, we expect the indices of collapsod variables to be arbitrary, resulting in all latent variables having similar average activity score. This is what we would expect for the vanilla VAF, where all are independent and have the same prior distribution.
10

---

Published in Transactions on Machine Learning Research (04/2023)
achieved by each model when trained on different sized training sets, well as the number collapsed latent variables. Each entry corresponds to the average over five runs with standard deviation given in the subscript. Bold indicates the best result in each group.
Table 4: Negative log-evidence
2 x [9] training samples
All training samples
D
K
E
Model
log
# #Inactive Units
log
#Inactive Units
VAE
13.90
0.00.1..00
9.7940.01
2.00
Arithmetic Circuit 2
VAE+IAF
13.27
0.00
9.76=0.00
0.00=0.00
10
5
15
STReN-VARE Frc
13. 13.14
0.00
9.7611..2
0.00 10.00
SIReN-VAL VREn
14.86+1.39
2.00
11.09
1.20
SIReN-VAETrue
12.2940.84
2.60
10.03
2.00
VAE
40.91+1.07
0.00+0.00
35.02=0.02
4.33+1.25
TAE+IAF++M
39.57+1.33
0.00
35.04
0.00
BColi
29
15
59
SIReN-VARE VAEFC
38.84+0.22
0.00
35.02
0.00
STReN-VA MAReN-VATE Rand
43.69
2.00
41.63
1.80
DIReN-VAETrue
37.5440.33
4.20
34.99
0.00
VAE
64.2242.14
0.00+0.00
37.45
23.3346.34
42.79.0.25
2.40
37.5940.01
1.80
Arth
67
40
150
SIReN-VAERO
VAEFC
43.35.+0.00
2.40+1.36
37.5640.07
6.33+1.89
STReN-VAL VARN-VAERRA VAERad
52.74416.16
3.10
40.81
5.20
SIReN-VAETH
41.56440.58
19.40
37.7340.05
14.80+0.33
VAE
10.57
0.00
7.65.0.02
0.00 10.00
MEHRA
VAE+IAF+MN
10.42+0.39
0.00
7.61+0.02
0.00
7
3
10
SIReN-VARE VFC
10.5640.32
0.50
7.58
0.00
SIReN-VABH VAERand 10.65+0.01
1.80
8.92+0.23
0.00
STReN-VAFTrue
10.60
1.33
8.37
0.00
Considering Figure this is indeed the case as the average activity of the vanilla VAE's latent variables is far more uniform (compared to SIReN-VAEErue). Interestingly, for SIReN-VAEpe the higher the index of the latent, the lower its activity on average. For the fully-connected BNs, each variable is conditioned on all variables with a lower index. This suggests that the model may tend to de-emphasize latent variables that are conditioned on many others. Figure Souure]shows showw that in SIReNVAAETue, posterior collapse tends to happen for certain variables far more than for others. Further inspecting the positions of the collapsed & in the BNs shows that the corresponding vertices typically only share edges with other latent variables, and not with any observed variables. This aligns with the findings of Burda et al. (2016), who observed that the latents in higher layers of hierarchical VAE (which do not directly influence any x) more prone to collapse.
Since we would like the learned latent distribution of STReN-VAERtH VAETru to respect the provided BN structure and utilize all the latent dimensions, the fact that aspects of the encoded structure make posterior collapse more likely, makes doubly important for us to try to address in some way. We therefore invest igate the efficacy warm-up (WU) (Bowman et al. 2016) and aaportancc-weighhee objectives in combatting posterior collapse, with the results given in Table 5 Since we have found that a certain subset. of latent. variables is more prone collapse than others, compare two different approaches to warm-up. The standard, more naïve, approach applies the warm-up factor to all latent variables, and is denoted by WUan- We also apply the warm-up term to only those latent variables that typically collapsed, as identified in Figure 4 We denote this 'retrained' approach by WUgelect- We do not apply warm-up if no posterior collapse occurred on the full training set as indicated in Table 4 For further information, see Appendix B.2 For comparison, we also apply these techniques to SIReN-VALE VAL
(IW) (Burda et al. 2016)
combat posterior callapse. Although this approach generally provided slight performance gains, comes significant time memory see Appendix B.3.4
We also considered using lower-variance gradient estimates Tucker al. 2018
11

---

Published in Transactions on Machine Learning Research (04/2023)
Arithmetic Circuit
Vanille
FC
0.75
0.54
0.72
-0.46
-2.25
True
-6.07
0.82
2.07
-4.69
1.86
Arth
11.83
1.98
-2.63
-1.45
-2.19
-2.50
-1.65
-2.15
-2.56
-2.12
-1.87
-2.51
-1.76
11.38
2.80
-2.00
-2.27
-2.42
2.84
-1.47
4.15
22.94
-2.58
-1.1
-1.81
-2.09
-3.00
-2.43
11.86
-1.88
3.08
1.80
4 4.79
0.66
0.53
10
-0.02 0.02
-0.20
0.59
-1.11
.0.73
1 3.30
-111
1100
1.40
1.75
-168
1.63
2.02
-1.93
2.56
2.36
3.07
3.12
-3 3.06
3.15
3.70
3.70
62
-3
-
4.16
4.42
-.99
4.79
5.48
5.46
3.62
-6 0..0
2 2.44
4.00
.6.10
3,99
..13
5.58
0.24
3.63
5.61
5.00
3.16
5.19
1.008
3.65
2.13
5.93
1.91
3.46
5.60
4.01
4.49
$.39
2.29
5.92
-2 2.05
5.04
0.06
-6.04
2..3
2.37
5.87
3.
5
8
244
277
Figure 4: The (logarithm of the) average of the posterior collapse statistic A,,, for each of the latent variables
of the Arithmetic Circuit and Arth datasets. The top, middle and bottom bar for each dataset corresponds the vanilla. VAF, STReN-VAFre Frc and STReN-VAFTHus respectively. The lower this value, the darker the corresponding block. Note that log(0.01) -4.6 is the point which latent variable is deemed collapsed.
Table 5: Performance of SIReN-VAE when applying posterior collapse mitigation techniques, namely: warm- up (WU) and importance weighted objective (TWAF). Warm-up is applied to either all latents (WUan), or to selected subset that is prone to collapse (WUselect)
2 x | training samples
All training samples
Model
log p(x)
#Tnactive Units
logp(x)
# #Inactive Units
STReN-IWAEFe
12.7810.43
0.00=0,00
9.73
0.00
Arithmetic
Circuit
SIReN-IWe SHIN--WAETroee WUsll
11.8810.27
0.00=0.00
9.8610.08
1.00
11.96
0.00=00.00
9.80
0.00
SIReN-IWAEre
38.8610.28
0.00=0000
35.0440.01
0.00
EColi
DIReN-IWAETo
38.42
1.60
34.98
0.00
S
43.0240.80
7.40
37.3210.15
1.00
Arth
STReN-IWAE Exroo WUal
41.09
15.00
37.48 40.47
10.20
SUReN-IWAETTo.. W Unntect
41.3110.43
17. 17.60
37.42
7.00
TTReN-IWAFpc
10.47
0.33
7.58
0.00
MEHRA
SIReN-IWAETh..
10.36+0.27
0.50
8.19
0.00
Table [5] shows that these techniques help to improve the performance of SIReN-VAETr. and reduce the number of inactive latent variables. However, when enough data is available, SIReN-VAEThe still does not quite match the performance of SIReN-VAErC, especially on the real-world MEHRA datasct. This could be because BNs constructed for real-world domains tend to focus more on the key influences between variables and neglect smaller interactions, which could limit VAEET 's ability to achieve comparable log-ovidence scores. Promisingly, STReN-VAET, performed the best on MEHRA the data-sparso setting when employing an IW objective. is perhaps precisely the coarser- grained -oarser-rrained structure encoded by SIReN. VA VAFruue that prevents it from overfitting and allows better generalization performance.
12

---

Published in Transactions on Machine Learning Research (04/2023)
5.2.3 Interpretability of the Latent Space
One of the anticipated benefits of incorporating hypothesized BN structures into a VAE is more in- terpretable latent space. Since we have access to the true latent variables of the datasets investigated whether this is being realized by estimating the mutual information (Mf) between each component of the true latent, x", and each component of the latent inferred by the model, 2, denoted by ,ZZ,,z) for i,j I,... K, using the MT neural estimator (MINE) of Belghazi et al. (2018). The samples used to estimale MI were obtained by sampling a single point Zn from (x)))x)) for each pair (xn.z,) the training set. Figure gives these results. We used the
models that achieved the best log-cvidence as given in Table J
above. In Figure B we also plot the MI belween pairs of true latent and observed variables, and pairs of
latent and observed variables sampled from SIReN- LRRe-VAETTue for the EColi dataset.
Arilhmelic Circuit 2
EColl
Arth
MEHRA
5-
True ppts)
20
Trun
35
10
10 15
25 30 Inferred qa42(s)
Inferred 2 oylajat
Inferred 2 9gl2iaa
Inferred aalzjk
Figure 5: Visualization of the MI between latent variables. Each entry corresponds to MI between the true and inferred latent variables, I(27, 2j) for i,j 1,.... . K, computed using the MI neural estimator (MINE) of Belghazi al. (2018). The higher the MI, the darker the associated entry.
If STReN- LoteN-VAErroe learns meaningful latent representa- tions, with each z; affecting its neighbourhood in way similar to the true hidden variable, would expect the MI
Thue -h piz
between z and to be high and for (z));z)) s I(zj. z)).
Visually, this means that we expect the matrix plots of
Figure 5 to be approximately symmetric and to have a
noticeable line on the diagonal. This holds the most no- ticeably for the FColi dataset. Although arguably not as distinct, similar trends evident for the other datasets well, e.g. for the real-world MEHRA dataset, two out of the three latent variables of the model have very high MI with their true counterparts. The two plots in Figure 0 are also nearly identical, meaning that SIReN-VE VRRNN-VAETuue closely mimics the dependencies of the true model in terms the MI between variables. The results support our hypothesis that SIReN-VAEntu should provide better interpretability in that gains further insight into which latent. variables directly influence each other. This could aid in more trolled conditional sample generation, and could allow one to associate specific roles with each latent factor of SIReN-VAE corresponding to their meaning in the original BN, as for example in the MEHRA BN where the latent variables represent atmospheric conditions.
p(z
Truo x ppx(()
Model > paiz)
10
15
20
Madel Pulzt
Modal Palx(p)
Figure 6: MI between latent and observed vari- ables for the EColi dataset.
6oor the synthetic datasets, the latent variables were sampled from the known models. For the real-world MEHRA dataset, assumed certain dimensions latent our purposes, and such have
llttent. data.
13

---

Published in Transactions on Machine Learning Research (04/2023)
6 Discussion
Limitations Future Work BNs are commonly defined over sets of both continuous and discrete variables. Extending GREs and STReN-VAEs to support discrete variables would thus greatly broaden their applicability. One desirable direction for future work is therefore to investigate integrating gradient
estimation approaches for discrete variables (e.g. Bengio et al. (0133) ; Tran et al. (2019) into these models.
In this work, we assumed access to an appropriate BN describing the dependency structure between the variables of interest. However, in most real-world settings one does not typically have access to such a structure, or one only has partial knowledge about the dependency graph. Therefore it is of interest to learn or refine the dependency structure from data. This idea has been explored in the context of normalizing flows and VAEs (e.g. (Wehenkel & Louppe 2021: He et al. 2019 and is an interesting future line of work to incorporate into the GRF and SIReN-VAL VAE models.
VAEs are typically applied to high-dimenssonal data like images and audio. As mentioned above, suitable BN structures are usually not readily available in these settings. In certain contexts, one might posit an appropriate latent structure, but it is not clear how to associate specific portions of the observation with specific latents a more fine-grained way than using full connectivity between certain latents and all input features. Initial experiments using SIReN-VAB (with a convolutional decoder) on syuthetically generated images using known sparse latent dependency structure and full connectivity between latents and observed features did not provide interpretable latent variables. Further exploration is therefore required to determine how best to apply STReN-VAF VAE directly to such high-dimensional data, and what sensible way might be to identify meaningful finer dependency structures between latent variables and observations.
Conclusion We proposed the GRF as an alternative to existing graphical NFs such as GNF-M (Wehenkel & Louppe] 2021) and SCCNF (Weilbach et al.) 2020). While these flows provide very good modelling capability, they do not ensure stable inversion. GRFs the other hand, exhibited comparable modelling performance, while being designed for reliable and more time-efficienl inversion. The GRF is thus suitable option to use when some assumed dependency structure is available and where the flow may be required to perform reliably in both directions. This is for example needed in the prior of our proposed STReN-VAR model, which is an approach for incorporating prior information from an arbitrary BN graph into a VAE. Encoding this structure does not lead to significantly better generalization performance than simply using a fully-connected structure when enough data is available. The key benefits of SIReN- VAE are its ability to provide more muanced interpretabilitty and to allow stable training with good generalization training sets. Practitioners who elicit BNs typically omit unknown factors, because they cannot in general use them in current modelling frameworks. Based on our results, we suggest that SIReN-VAETTu might be of use when practitioners know or can hypothesize about the dependency structure of certain latent factors in their domain, especially they wish to interpret the model according to the chosen dependency structure. STReN-VAFTue could also facilitate higher quality data augmentation from limited real-world data.
small
7
Acknowledgements
We would like to thank DeepMind for financially supporting J. Mouton through a scholarship, as well as the reviewers for helping fine-tune the ideas in this paper.
References
Jens Behrmann, Will Grathwohl, Ricky T. Q. Chen, David Duvenaud, and Jörn-Henrik Jacobsen. Invertible residual networks. In Proceedings of the 36th International Conference on Machine Learning, pp. 573 582. PMLR, 2019.
Jens Behrmann, Paul Vicol, Kuan-Chich Wang, Roger Grosse, and Jörn-Henrik Jacobsen. Understanding and mitigating exploding inverses in invertible neural networks. In Proceedings of The 24th International Conference on Artificial Intelligence and Statistics, pp. 1792-1800 PMLR, 2021.
14

---

Published in Transactions on Machine Learning Research (04/2023)
Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeshwar, Sherjil Ozair, Yoshua Bengio, Aaron Courville, and Devon Hjelm. Mutual information neural estimation. In Jennifer Dy and Andreas Krause (eds.), Pro- ceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 531-540. PMLR, 2018.
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXio:1308.3432, 2013.
Samuel R. Bowman, Luke Vilnis, Oriol Vinyals, Andrew Dai, Rafal Jozelowicz, and Samy Bengio. Generating sentences from a continuous space. In Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning. Association for Computational Linguisties, 2016. doi: 10.18653/vv /015/////1166--/// 1002.
Yuri Burda, Roger Grosse, and Ruslan Salakhutdinov. Importance weighted autoencoders. In 4th Interna- tional Conference on Learning Representations, 2016.
Ricky T. Q. Chen, Yulia Rubanova, Jesse Bettencourt, and David Duvenaud. Neural ordinary differential equations. In Advances in Neural Information Processing Systems, pp. 1572-6583, Red Hook, NY, USA, 2018. Curran Associates, Inc.
Ricky T. Q. Chen, Jens Behrmann, David Duvenaud, and Jörn-Henrik Jacobsen. Residual flows for invertible generative modeling. In Advances in Neural Information Processing Systems, pp. 9916 9926, Red Hook, NY, USA, 2019. Curran Associates, Inc.
Herbert Federer. Geometric Measure Theory. Springer, 1996. ISBN 978-3-540-60656-7
Mathieu Germain, Karol Gregor, lain Murray, and Hugo Larochelle. MADE: Masked autoencoder for distribution estimation. In Proceedings of the 32nd International Conference on Machine Learning, pp- 881 889. PMLR, 2015.
Henry Gouk, Eibe Frank, Bernhard Pfahringer, and Michael J. Cree. Regularisation of neural networks by enforcing Lipschitz continuily. Machine Learning, 110(2):393 416, 2021.
Will Grathwohl, Ricky T. Q. Chen, Jesse Bettencourt, Πlya Sutskever, and David Duvenaud. FFJORD: free-form continuous dynamics for scalable reversible generative models. In 7th International Conference on Learning Representations, 2019.
Jiawei He, Yu Gong, Joseph Marino, Greg Mori, and Andreas Lehrmann. Variational autoencoders with jointly optimized latent dependency structure. In International Conference on Learning Representations, 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In
IEEE Conference Computer Vision and Pattern Recognation (CVPR), pp. 770-778. IEEE, 2016.
Diederik P. Kingma and Max Welling. Auto-encoding variational Bayes. Tn 2nd International Conference on Learning Representations, pp. 14 16, 2014.
Dicderik P. Kingma, Tim Salimans, Rafal Jozefowicz, Xi Chen, Πya Sutskever, and Max Welling. Improving variational inference with inverse autoregressive flow. In Advances in Neural Information Processing Systems, pp.
Red Hook, NY, USA, 2016. Curran Associates, Inc.
Daphne Koller and Nir Friedman. Probabilistic Graphical Models: Principles and Techsriques Adaptive Computation and Machine Learning. The MIT Press, 2009. ISBN 0262013193.
Gábor Melis, András György, and Phil Blunsom. Mutual information constraints for Monte Carlo objectives to prevent posterior collapse especially in language modelling. Journal of Machine Learning Research, 23 (75):1 36, 2022.
Diganta Misra. Mish: A self regularized non-monotonic activation function. In 31st British Machine Vision Conference, 2020.
15

---

Published in Transactions on Machine Learning Research (04/2023)
George Papamakarios, Theo Pavlakou, and lain Murray, Masked autoregressive flow for density estimation. In Advances in Neural Information. Processing Systems, pp. Red Hook, NY, USA, 2017. Curran Associates, Inc.
Ali Razavi, Aäron van den Oord, Ben Poole, and Oriol Vinyals. Preventing posterior collapse with delta- VAEs. In 7th International Conference on Learning Representations. Open .ee,, 2019.
Danilo Rezende and Shakir Mohamed. Variational inference with normalizing flows. In Proceedings of the 32nd International Conference on Machine Learning, pp. 1530 1538. PMLR, 2015.
Danilo Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximato infer- ence in deep generative models. In Proceedings of the 31st International Conference on Machine Learning, volume 32, pp. 1278 1286, 2014.
Karen Sachs, Omar Perez, Dana Pe'er, Douglas A. Lauffenbuger,, and Garry P. Nolan. Causal protein- signaling networks derived from multiparameter single-cell data. Science, 308(5721):523 529, 2005.
Marco Scutari. Bayesian network repository, 2022. URL http////t///////oooooororrerortr
Casper Kaae Sonderby, Tapani Raiko, Lars Maalee, Soren Kaae Sønderby, and Ole Winther, Ladder varia- tional autoencoders. Tn D. Lee, M. Sugiyama, U. Luxburg, T. Guyon, and R. Garnett (eds.), Advances in Newral Information Processing Systems, volume 29. Curran Associates, Inc., 2016.
Yang Song, Chenlin Meng, and Stefano Ermon. MintNet: Building invertible neural networks with masked convolutions. In Advances in Neurol Information Processing Systems, pp. 11004 11014. Curran Associates, Inc., 2019.
E. G. Tabak and Cristina V. Turner. A family of nonparametric density estimation algorithms. Communi- cations on Pure and Applied Mathematics, 66(2):145 164, 2013.
Dustin Tran, Keyon Vala, Kumar Agrawal, Laurent Dinh, and Ben Poole. Discrete flows: Invertible gener- ative models of discrete data. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché Buc, E. Fox, and R. Garnett (eds.), Advances Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
George Tucker, Dictcrich Lawson, Shixiang Gu, and Chris J. Maddison. Doubly reparamctorized gradicnt estimators for Monte Carlo objectives. In 1st Symposium on Advances in Ayprozimate Bayesian Inference, pp. 1-14, 2018
Lorenz Vaitl, Kim A Nicoli, Shinichi Nakajima, and Pan Kessel. Gradients should stay on path: Better estimators of the reverse- and forward KL-divergence for normalizing flows. Machine Learning: Science and Technology, 3(4), 2022. doi: 10.1088/2633 215632225533392555
Claudia Vitolo, Marco Scutari, Mohamed Ghalaieny, Allan Tucker, and Andrew Russell. Modeling air pollution, climate, and health data using Bayesian networks: A case study of the English regions. Earth and Space Science, 5(4):76-85 88, 2018. doi: https://dbo.orr/11.0000/// /ttp:////////boo0////00000/00700 2012000702001..00002
Stefan Webb, Adam Golinski, Robert Zinkov, N. Siddharth, Tom Rainforth, Yee Whye Tch, and Frank Wood. Faithful inversion of generative models for effective amortized inference. In Advances in Neural Information Processing Systems, pp. 3070-3080, Red Hook, NY, USA, 2018. Curran Associates, Inc.
Antoine Wehenkel and Gilles Louppe. Unconstrained monotonic neural networks. In Advances in Neural Information Processing Systems, Red Hook, NY, USA, 2019. Curran Associates, Inc.
Antoine Wehenkel and Gilles Louppe. Graphical normalizing flows. In Proceedings of The 24th International Conference Artificial Intelligence and Statistics, pp. 37-45. PMLR, 2021.
Christian Weilbach, Boyan Beronov, Frank Wood, and William Harvey. Structured conditional continu- ous normalizing flows for efficient amortized inference in graphical models. In Proceedings of the 23rd International Conference on Artificial Intelligence and Stalistics, pp. 4441-4451. PMLR, 2020.
16

---

Published in Transactions on Machine Learning Research (04/2023)
A Appendix: Additional Theoretical Background
A.1 Extending MADE for Arbitrary Graphical Structures
Suppose a joint distribution factorizes as
p(x) = II Pa
(8)
for DAG G. Pag, denotes the parents of x; in G. For a given neural network that takes x as input, the goal is to have the output units associated with z; be computed from only those input units associated with æ; and its parents. This means that there should be no computational paths between an input and an output unit if there is no direct dependency between the associated variables in g. This can be achieved by applying masking matrix to the weights of each neural network layer (which can be of arbitrary width) such that at least one weight on any such computational path is set to zero.
We follow a similar approach to MADE (Germain et al. 2015). which constructs masks for an implicit fully-connected BN. We begin by assigning a specific subset of variables to each unit in the neural network. Specifically, each input unit is assigned unit set containing its corresponding imput variable: {æz]. Each output unit is assigned {a,} U Pag. Lastly, each hidden unit is randomly assigned one of the following sets: {z,} or {ax) U Pag. where i can be any of 1..... M
consisting of its associated variable and that variable's parents in the BN:
A correct mask can then be constructed by ensuring that it zeroes out any weight between two neural network units if the set assigned to the unit in the next layer is not a superset of the set assigned to the unit in the previous layer. This has the implication that any path from input to output for any variable has a single associated set switch from {a.} to {x,} UP?..
A.2 Invertibility of Graphical Normalizing Flows Practice
The Lipschitz constants of the forward and inverse transformation of an invertible neural network quantifies its worst-case stability. Bounds on these values play an important role in understanding and mitigating possible exploding inverses.
No global bounds can be placed on the Lipschitz constant of this type of flow, which complicates the task of ensuring stable inversion in all scenarios. [Behrmann et al. (2021) provide the following simple illustration of why GNF-A only has local Lipschitz bounds. Assume x consists of two variables, zo and ø1, where In is dependent on Ty in the corresponding BN. Let [F(x)] = æ1 exp(s(20)) be the transformation applied to 21 by a single-step GNF-A, where a(rr)) = [s(x)] is the second output dimension of the condit ioner function s() and the dependence on only xo has been made explicit. The output of the conditioner function [m(x)]1 is taken to be 0 for simplicity. Then
GNF-A
O[F(x)) dexp(s(<o)) = x1 cxp(s(zo))s((au))
= 21
(9)
Ozo
Oro
Thus, if 21 may grow arbitrarily large, this derivative will be unbounded, which could allow the Jacobian, Jp(x), to have an unbounded Frobenius norm. Due to the equivalence of norms in finite dimensions, this in turn can induce an unbounded spectral norm of the Jacobian. Lastly, we consider the following theorem from Federer (1996): if F RRD + D is a Lipschitz continuous and differentiable function under the Euclidean norm, then
Lip(F) = sup || XERO
(10)
where 11- denotes the spectral norm. Based on Equation 10 we conclude that if the spectral norm of the Jacobian is unbounded, then no global Lipschitz bound can be obtained.
Too prevent situations where there valid paths from an input the corresponding output, we require least. one unit in each hidden layer associated with (x,).
17

---

Published in Transactions on Machine Learning Research (04/2023)
GNF-M We can employ similar illustration to investigate the Lipschitz bounds of a GNF with monotonic transformations. Again, assume consists of two variables, 4o and 21, where depends corresponding BN. The transformation applied to by a single-step GNF-M is then given by [F(x)] = Jo" htt, ei (zo)) dt++ ((1 (mo)). We take the partial derivative of the above transformation and apply Leibniz's integral rule and the chain rule:
2o in the
I ((tt) dt (3(c)(2a) axo a c1 de, (2o) dt +
Ø
Ozo
axo 03(c)(z0))
(11)
(c1() dxg
Oxo
The integrand above the product of the derivatives of two neural networks with respect to their inputs. For general networks, this integrand's shape will depend not only on the chosen activation functions, but also the weights obtained during training. If x may grow arbitrarily large, and if the area under the curve given by the integrand is not bounded by some maximun value as I1 increases, then can apply similar reasoning as above to show that this flow has no global Lipschitz bounds. Thus, either the architecture of the flow must be adapted to ensure that this integral remains bounded as function of 21 > 0, or other techniques must be used to improve local stability, as discussed in Behrmann et al. (2021]
SCCNF Given that the flow is defined by a neural ODE, de have that the Lipschitz constant for both the forward and inverse transformation are upper bounded by + Behrmann et al. 2021).
dx(() (x((), f), where E [0, 1], we
A.3 Dependence Structure Induced by GRFs
We investigate whether the distribution represented by a given GRF does indeed encode all the conditional independencies specified by the provided BN. We first consider a normalizing GRF with a single flow step, F(x)= =6, 6, which encodes the following BN chain structure: applied to each of the dimensions be given by Fo(ro) Ep, (z graphical illustration of this flow is given in Figure 7a Note that these individual bijective transformations are implemented using single residual block neural network. We treat them separately here to simplify the discussion. We are interested in whether the distribution represented by the flow, p(x), respects the conditional independence assumptions specified by this BN.
20 + 21 + x2. Let the bijective transformations
6 and (2 = 62. A
After a single transformation step, the distribution of 22 will only depend on ø1 as desired, since it can computed as: 1og2(22 (1)) = log po (F2(1z; + log (Jr, (#2; x1))]- This illustrates how the distri- bution represented by normalizing GRF with a single transformation step will adhere to the conditional independencies specified by the BN, which is in line with the argument presented by Wehenkel & Louppe h
20
22
Z
222
Fo
F1
F
Fo
F
F
60
(a) Normalizing GRF
(b) Generative GRF
Figure 7: A graphical illustration of the transformation applied by (a) one-slep normalizing GRF and (b) a one-step generative GRF when encoding the chain dependency structures: Zo - 21 + 22 and x + 20 + 31 22, respectively. The variables of the base distribution are represented by co. C1 and c2. Undirectod odges represent bijective transformation (Fu, F1 or F2) between the associated variables with the small indicating the direction of the forward mapping of the flow. Directed edges indicate the additional variables these bijective transformations conditioned as enforced by the presented masking scheme.
18

---

Published in Transactions on Machine Learning Research (04/2023)
Additional dependencies are however introduced when the number of flow steps is increased. Considering Figure 8 which depicts 2-step normalizing GRF, one
20
Z1
32
can note that when computing the latent representation of T2, information will 'loak' from To via the intermediate transformations of the observed variables. If enough translormation steps are applied, the distribution of any observed variable will ultimately depend on all its ancestors in the BN graph. As result, the encoded structure may end up corresponding to the transitive closure of the original BN structure.
60.2
$1.
c1,1
C1.
Next, consider generative GRF with a single flow step, 2 = F(c,a:) for c~ po, which encodes the following BN chain structure: I - - 21 + 22. Again, let the bijective transformations applied to each of the dimensions be given by 20 = Folso: x), 21 = (o) and 22 = (1( where each transformation is conditioned on a subset of e, and only Fo conditioned on the observation, v. A graphical illustration of this flow is given in Figure 7b We are inter- ested in whether the distribution of the generated samples, (((2x), respects the conditional independence assumptions specified by this BN, such as that is conditionally independent of 2n and given 21. That is, (22//20, 21. x)- = (22/221) should hold in the distribution represented by the flow. We can compute:
Figure 8: A normalizing GRF with two transforma- tion steps.
log g( Ro. x)) = log (F2 (22;61) (1)) + log dee (2236)))
log (F2 (z2 FT (z13e0))) + log (del( r (z2; FT (2136))))
= log (z2; FT (=1: (=o:))))) + log det([ ( PT (21;Fo (=o:=))))
By expanding the expression in this way, we make clear the direct dependence of 22 on and x knowing only 21 is not sufficient to specify ( 20, x). This dependence arises from the fact that the bijective transformation between 22 and 62 is only specified once Ej is known, and 6 is a function of both ž and zo. Thus, ((2g) z0. 2) & q(22|21). In this way, each variable could be dependent on all its ancestors in the BN, and the dependency structure induced by the flow may again ultimately correspond to the transitive closure of the encoded BN.
We therefore have that for both normalizing and generative GRE, the dependency structure induced by the flow could correspond to the encoded BN's transitive closure. In the worst-case scenario, this transitive closure corresponds to a fully-connected graph, in which case one would secmingly not have gained any benefit from encoding the given structure this way. The dependencies induced by these graphical flows are arguably more subtle, however. For normalizing GRFs, each variable only receives information from its ancestors via intermediate bijective transformations of its parents. Even though there some 'information leakage', it would not be unrcasonable to expect that the distribution of a given variable will be more strongly influenced by its parents, rather than by the polentially 'diluted' information the variable receives about the rest of its ancestors. The degree of information leakage for generative GRFs is even less clear. This is primarily because each variable is in fact never a direct function of its parents, but rather depends on bijective transformations of these parents where each bijection itself conditioned on those variables' parents. For example, we showed that knowing only the parent of 22 is not sufficient to calculate its density (if' is unknown), and one additionally needs knowledge of its ancestors, including the observation x. When performing density estimation with generative flow, this dependence is thus already introduced after only one flow step. When generating new samples, one however still needs three flow transformations for a generated sample of to contain any information about the observed state (similar to how 61,2 only receives information about xp after two steps in the normalizing GRF depicted in Figure D]. Taken together, even though GRFss cannot in general guarantee that the distribution represented by the flow respects the independence statements specified by the encoded BN, we still expect these flows to incorporate a strong inductive bias that encourages the variables to adhere to the desired dependency structure (see A Appendix B.3.3).
19

---

Published in Transactions on Machine Learning Research (04/2023)
B Appendix: Datasets & Experiments
B.1 Bayesian Network Datasets
The synthotic arithmotic circuit BN the same as used that by Weilbach et al. (2020) and Wehenkel Louppe (2021). For density estimation tasks, all variables are observed. For amorlized inference tasks, variables 20 to 25 are latent, while z0 and 21 are observed. This distribution consists of heavy-tailed densities which linked through non-lincar dependencics.
Arithmetic Circuit
20 Laplace(5, 5, 1)
Au
F
21 Laplace( 22
22 ((tanh(2) 20 + - 2.8). 0.1)
23 ~ N(20 x 0.1)
24 ~ (7,
FD
25 (taanh(z2 + 24), 0.1)
30 V213,01)
x1
(22,, 0.1).
Figure 9: Arithmetic Circuit BN.
Tree This another synthetic dataset. It is adapted from the model given in Wehenkel & Louppe (2021)
obtain a fully-specified model for which the joint density can be computed closed form (as needed for
the inference tasks) see Figure 10 Instead of using the circles 2D dataset from Grathwohl et al. (2019)
in Wehenkel & Louppe (2021) p, the first two variables
sampled from a 2D Gaussian mixture model.
GMM2, which consists of two equally weighted components with means at (1, and (-1,-1) and shared covariance matrix 0.2 x Iz. As in Wehenkel & Louppe (2021), the second pair of variables is sampled from a
GMM with 8 equally weighted components with means at (0, 1.5), (1, 1), (1.5, 0), (1, 11), (0, -1.5), (-1, 11), (-1.5,0) 0) and (-1,1) 1) and shared covariance matrix 0.1 x Ig.
20. GMM2
22
23
2/ GMMs
24 ~ (z), 1)
ZA
25
25 N(min(22, 23),
2g N (5 (in((2 + 25) + cos(24 + 2s))).
FO
Figure 10: Tree BN.
EColi The FCCli dataset was generated using a fully-specified BN adapted from the BN repository of Scu- tari 2022).. All vertices are (conditionally) Gaussian, with means given by a linear combination of the parents. Cortain leaf vertices were removed from the original BN (nmpc and ftsJ) in order to obtain a BN with fewer latent. than observed variables. making thr BN more applicable for the STReN-VAE setting as well. The following vertices were considered to be latent: {asnA, atpD, b1191, cspA, cspG, dnaK, eutG, fixC, icdA, lacA, lacY, sucA, yedE, ygcE, yheI), while the rest were considered observed. See Figure Tla for diagram of the BN.
This real-world dataset consists of 11 observed variables containing information about multiple phosphorylated human proteins (Sachs al.) 2005 see Figure 11b for a diagram of the BN structure. The BN structure encodes the cellular signalling network then believed to exist between these proteins.
Protein
20

---

Published in Transactions on Machine Learning Research (04/2023)
MEHRA The Multi-dmmensional Environment-Healer Risk Analysis dataset (MEHRA) was assembled by Vitolo et al. (2018) to help model air pollution, climate and health in English regions using BN. We only consider the subgraph of the BN obtained during the study, corresponding to the continuous variables, see Figure 11c As such, only a subset of the original dataset corresponding to a fixed set discrete variables used. This reduced datasel consisted of all observations of the continuous variables for the following sctting the observed variables: (Region=Greater London Authority; Zone=Greater London Urban Area; Type=Traffic Urban; Year=2014; Season=Winter).
11
11
27
28
21
19
13
24
18
8
3
12
15
10
25
16
17
13
14
20
22
14
FColi
Plas
PKC
wd
pm10
t2m
P38
SST
Akt
bTh
WS
no2
8n2
pm25
(b) Protein
(c) MEHRA
Figure 11: BN graphs associated with the EColi, Protein and MEHRA datasets.
Since VAEs are typically used lo encode information into lower-dinensionn representation, we only consider the datasets from the GRF investigation for which there are fewer latent than observed variables. Thus, in addition to EColi and MEHRA presented above we also use the following datasets:
Arth The Arth dataset was generated using a fully-specified BN from the BN repository Scutari (0022 All vertices are (conditionally) Gaussian, with means given by linear combination of the parents. All leaf vertices were considered be observed, and the rest are latent. See Figure 12 for the corresponding BN graph.
21

---

Published in Transactions on Machine Learning Research (04/2023)
56
58
36
66
33
30
39
38
13
26
Figure 12: Arth BN graph.
Arithmetic Circuit This Arithmetic Circuit BN is an adaptation the one described above Figure g where we have added more obscrved variables. Variables 2n to 24 are latent, while ap to 29 are observed. See Figure 13 This distribution consists of heavy-tailed densities which linked through non-linear dependencies.
20 (tann((zo + - 2.8).
21
20
21 (tann(22),1.1)
22 ((tanh(2) + 23). 0.1)
20 ~ Laplace(5, 1)
23 N/22 8,0.1)
Z3
21 Laplace(-2 22
24 Ni(o(zs 7), 1.1)
22 M(120 21)/7.9 - - 0.1)
I
23 NN((2 x 2.)/6.1,0.1)
23 (7,
Z
F
26 ((2,,
21 (tanh(22 + 22),
27 ~ N(2,,
F
Zg
28 (tannh2z))2211 2.11
F6
Fs
3g Nssin(z2), 1.1).
Figure 13: Arithmetic Circuit BN.
22

---

Published in Transactions on Machine Learning Research (04/2023)
B.2 Model Architectures & Experiment Setup
GRF We train two models per task for each of the approaches. The first a smaller model with a maximum capacity of 5000 trainable paramcters, denoted by subscript The second, larger model has a maximum capacity of 15000 parameters, denoted by the subscript L. Table [ details the flow architectures. Since proposed masking scheme be used drop-in replacement, we compare the different graphical models using this scheme to encode domain knowledge in GRF, GNF and SCCNF.
Table 6: Flow architectures. The hidden layer width and choice of activation function are the settings used for the conditioner neural network for GNF-A and GNF-M, the residual block for GRF, and the main flow transformation neural network for SCCNF, respectively.
Number of parameters
Number of flow steps*
Hidden layer width
Activation
BN
Flow
function
GNF-Ag
4416
200
ReLU
GNF-Mg
4552
2
50
ReLU
Arithmetic circuit.
SCCNF's
4621
140
Tanh
GRFs
4296
8
125
LipMish
GNF-A,
14796
9
300
ReLU
GNR-M
14508
4
125
ReLU
SCCNF,
13312
150
Tanh
GRFZ
14518
17
200
LipMish
GNF-Ag
4832
4
250
ReLU
GNF-Ms
4934
75
ReLU
SCONEs
4415
125
Tanh
Tree
GRFs
4576
8
125
LipMish
GNE-A 14490
10
300
ReLU
GNF-Mr 14208
150
ReLU
SCCNF,
13828
140
Tanh
GRFZ
14625
15
215
LipMish
GNF-Ag
4812
4
175
ReLU
GNF-Mg
4897
150
ReLU
SCCNEs
4788
3
150
Tanh
Protein
GRFs
4788
9
100
LipMish
GNE-A
13788
9
225
ReLU
GNF-Mr
14691
3
150
ReLU
SCCNFL
14765
170
Tanh
GRF,
1896
28
100
LipMish
GNF-As
4852
4
140
ReLU
GNF-Ms
4664
100
ReLU
SCCNF$ 8
4699
3
250
Tanh
EColi
GRFs
4347
9
100
TipMish
GNF-Ar
14472
9
190
ReLU
GNF-M
13992
100
ReLU
SCCNF,
13165
300
Tanh
GRFz
14944
16
200
LipMish
GNF-As
4760
4
150
ReLU
GNF-Ms
4976
125
ReLU
MEHRA
SCCNF$
4355
3
150
Tanh
GRFs
4797
9
125
LipMish
GN
14220
200
ReLU
GNE-M
14928
125
ReLU
SCCNFI
14214
175
Tanh
GRFZ 14382
17
200
LipMish
*or SCCNE, this instead refers the number of layers the neural network.
23

---

Published in Transactions on Machine Learning Research (04/2023)
SIReN-VAE Table provids the network architectures used in the SIReN-VAE experiments. The models had the same network architectures for all of the datasets, except Arth. For Arth, the hidden layers of the neural networks had a width of 200 units, not 100. The weights of the decoder neural network and all residual blocks were masked according to the given BN structure.
When applying warm-up, a default warm-up period of 100 epochs was used in all cases. We used aeportance-weighter samples for the Arithmetic Circuit and MEHRA datasets when employing the importance- objective, and only 8 for the Gaussian BNs, EColi and Arth, since further increas- ing the number of samples did not lead to notably better results.
Table 7: Model architectures used for the graphical datasets. All GRFs have 5 transformation steps, with each residual block having the same architecture given below. D indicates the number of observed variables, and K the number of latent variables associated with each dataset's s BN. Although not indicated here, the weight matrices of the linear layers in the decoder neural network and residual blocks of SIReN-VAB are masked according to the encoded BN structure. For GRF,, g: the input dimension of the first lincar layer of each residual block larger to accommodate conditioning on the observation.
Model
Architecture
Encoder
Linear(D,100) + ReLU + Jinear(100,K3
VAE
Decoder
Lincar(K,100) ReLU - Linear (100,Dx2)
Encoder
GRF,
SIReN-VAE
GRF, Linear(K,100) ReLU + Linear(100,Dx2)
Decoder
Residual Block
(1n) + LipMish + Linear Linear(100,K)
B.3
Additional Results
B.3.1 Flow Inversion
gives the inversion performance of the Banach and Newton-like (Song et al. 2019 fixed-point approaches on the Arithmetic Circuit dataset. Table summarizes the inversion performance of the different flow models on the rest of the datasets, with Figures 15 to 18 plotting the reconstruction error of GNF-M and GRF for different values of in when varying the number of iterations used at each step while inverting these flows.
Figure 14
10a
Newton (c-0.99)
10a
(c=0.99)
Reconstruction error
Newton (c-0.85)
Banach (c=0.85)
10-1
Newton (c=0.7)
Banach (c=0.7)
10-2
10-3
10-1
60
120
150
180
Number of Iterations performed per block
Figure 14: Using the Newton-like inversion procedure of Equation requires far fewer ilerations per block accurately invert a GRF than using the Banach fixed-point approach. The plot shows the average recon- struction error (log-scale) for 100 samples from the Arithmetic Circuit test Note that all the plots for the Newton-like inversion procedure, corresponding to different values of the Lipschitz-bount bound hyperparameter c, overlap. Similar results were observed for the other datasets as well.
24

---

Published in Transactions on Machine Learning Research (04/2023)
Table 8: Comparison of the inversion performance for the different flow models on 100 test data points from the various datasets. Bold indicates the best results in each column. N and a are not applicable for SCCNF. Ranges indicate different optimal settings for N and for different data points. Inversion time measured for the smallest N 50 that allowed the most data points in the batch have a reconstruction error of less than 10- and is the time taken to invert the entire batch.
Small Budget
Large Budget
Converged within
Converged within
Inversion time (ms)
Inversion time (ms)
BN
Flow
N
N
50 steps
50 steps
GNF-M
99
1-42
0.3 1.1
226. 17
100
1.0
141.83
Arithmetic Circuit
SCCNF
82
294.63
97
540.38
GRF
100
4-5
1.0
50.15
100
3-4
1.0
91.88
GNF-M
97
9 50
0.5 1.4
145.88
100
32
0.8 1.2
268.28
Protein
SCCNF
93
186.08
81
890.38
GRF
100
5-7
0.9-1.0
71.90
100
4-8
0.9-1.0
265. 265.23
GNF-M
100
7-45
0.4
182.40
100
6 8
1.0
98.90
EColi
SCCNF
23
121.66
516.53
GRF
100
5
1.0
57.91
100
4-5
1.0
101.90
GNF-M
100
3-11
0.9-1.0
32.91
100
3-7
0.8-1.1
62.81
MEHRA
SCCNF
100
94.27
99
190.92
GRF
100
1.0
48.15
100
1.0
89.69
10a
10
0.2
Reconstruction error
10a
0.6
Reconstrution
0.7
10a
10
0.9
-
1
0.7
0.9
10
1013
1.0
1.2
10-4
10-4
1
6
11
16
21
26
31
36
41
46
50
6
11
16
21
26
31
36
41
46
50
Number of iterations flow
Numberr of iterations per flow step
(a) GNF-Ms
(b) GNF-M
10a
10F
10a
10a
Reconstruction
Reconstruction
1.2
2
10a1
10-1
10-
1
6
26
31
36
41
6
26
31
36
Number of Iterations per flow step
Number of Iterations per flow step
(c) GRFs
(d) GRFt
Figure 15: Reconstruction error achieved when inverting GNF-M and GRF on 100 samples from the Arithmetic Circuit datasct. The reconstruction error is plotted function of the number of iterations used to invert each flow step, for different values of the slep-size,
25

---

Published in Transactions on Machine Learning Research (04/2023)
10
10a
Reconstruction errer
Reconstruction error
10*
WW
10a
10a
20
0.1
10
0.7
0.6
0.9
10-3
0.8
10-3
1.0
1.0
1.2
10
10
6
11
26
31
36
41
46
50
11
21
28
31
36
41
Number of Iterstions flow
Number of iterations per flow
(a) GNF-M5
(b) GNF-MA
10a
10F
10a
10a
Reconstruction
Reconstruction
1.2
1.2
10
10
10
10
10"5
10a6
6
11
26
31
36
46
50
6
11
26
31
36
Number of Iterations per flow step
Number of Iterations per flow step
(c) GRFs
(d) GRF,
Figure 16: Reconstruction when inverting GNF-M and GRF samples from the Protein dataset.
10a
10a
0.7
Reconstructin error
Reconstruction error
0.9
10a
10a
1.0
1.2
0.2
2
0.4
1
0.6
10-3
0.8
10a3
1.0
10
10
6
11
16
21
26
31
36
41
46
50
1
6
11
16
21
26
31
36
41
46
50
Number of iterations flow
Number of iterations, flow step
(a) GNF-Ms
(b) GNF-M
10a
10a
10a
Reconstructon
Reconstruction
10-1
10-1
-
1
10-9
10-9
1
6
11
26
31
36
41
46
50
6
11
25
31
36
41
Number of Iterations per flow step
Number of iterations per flow step
(c) GRFs
(d) GRFz
Figure 17: Reconstruction error when inverting GNF-M and GRF samples from the EColi datasct.
26

---

Published in Transactions on Machine Learning Research (04/2023)
10a
Reconstuctio enror
10a
0.7
10a
0.7
0.9
0.9
10a
100
Reconstructon
10a
1.0
1.2
1.2
20-1
10a1
20
10
10a0
10"0
1
6
11
16
21
26
31
36
41
46
50
1
6
11
16
21
26
31
36
41
45
50
Number iterations per step
Number iterations per flow step
(a) GNF-Ms
(b) GNF-M,
10a
10a
10a
0.7
10a
0.7
0.9
Reconstruction
1.0
Reconstruction
1.0
1.2
1.2
10
10
10"3
10"3
10-3
10:5
36
6
Number of iterations per flow step
Number of iterations per flow step
(c) GRFs
(d) GRFl
Figure 18: Reconstruction when inverting GNF-M and GRF on samples from the MEHRA dataset.
10a
10a
Reconsiructoo
10
Reconsruction
10-1
10
10-3
10-3
1
10
1
6
11
16
26
31
36
41
46
50
1
6
11
16
26
31
36
41
46
50
per flow step
per flow step
(a) GNF-Ms
(b) GRFs
Figure 19: Reconstruction error when inverting GNF-M and GRF samples from the Tree dataset.
27

---

Published in Transactions on Machine Learning Research (04/2023)
B.3.2 Performance of the LipMish Activation Function
Table 9 gives the performance results of GRF on the different datasets when using either the LipSwish or Liphish activation functions. See Figure 20 for comparison of the first and second derivatives of these two activation functions. Apart from the activation function, the model architectures are the same for GRFs and GRFL dctailed in Table B function of the flow depth, when using dilferent residual block aclivalions, see Figure 21 We see that Lip typically slightly outperforms LipSwish. Figure 21 further supports this, with LipMish performing on par and often better than other activations.
We also determine the negative log-likelihoods achievod by GRF
Table 9: Density estimation and variational inference performance of GRFs and GRF, when using either the LipMish or LipSwish activation function. Lower is better in all cases, Note that the real-world Protein and MEHRA datasets were not used for the inference tasks.
Density estimation (NLL)
Inference (ELBO)
BN
Flow
LipSwish
LipMish
LipSwish
LipMish
Arithmetic
GRFs
1.270
1.248
4.21910.18
4.194
Circuit
GRF
1.10710.01
1.110
3.76610.12
3.713
GRFs
8.649
8.64210.01
1.739+4
1.738 A
Tree
GRF,
8.649
8.645
1.705+A
1.705 A
GRFs GRF,
5.230
5..265.0.01
Protein
6.03554007
-6.11140.0
GRFs 40.062+
40.059
34.986=0.04
34.964+
EColi
GRF,
40.064
40.062
34.962
34.963
GRFs
11.66510.01
11.660
MEHRA
GRF,
11.62340.04
11.612_0.03
1.0
LipSwish'(x (x)
0.5
LipMish((x)
LipSwish"
LipMish"(x)
0.0
Figure 20: Comparison of the first and second derivatives of the LipSwish CCenn et al. 2019 and the proposed LipMish activation functions.
28

---

Published in Transactions on Machine Learning Research (04/2023)
Negative log-ikkllhood
1.35
Tanh
Log-likelihood
Tanh
ELU
8.75
ELU
LipSwish
LipSwish
LipMish,
LipMish
1.25
Negative
1.20
8.65
12
16
20
24
28
8
10
12
Flow depth
Flow depth
(a) Arithmetic Circuit
(b) Tree
40.24
Negative Log- egglikelihood
Tanh
Negative Log-iikellhood
40.22
ELU
Tanh
LipSwish LipMish
40.20
ELU
-555
40.18
LipSwish
40.16
LipMish
40.14
40.12
Flow depth
Flow depth
(c) Protein
(d) EColi
Negative oggllikelhhodd
Tanh
11.10
ELU
11.05
LipSwish
11.00
Lip LipMish
10.95
10.90
3
10
12
Flow depth
(e) MEHRA
Figure 21:
Negative biglliill achieved by GRF models for varying flow depths and different residual block activation functions (tanh, ELU, LipSwish and LipMish) on each of the datasets. Lower is better.
B.3.3 Dependency Structure Induced by GRFs
We provide empirical results supporting our argument that even though information leakage may occur in a GRF between variables and their ancestors, the flow architecture still provides a strong enough inductive bias to encourage the resulting distribution to respect the provided dependence structure. If the induced dependence structure fully corresponds to the transitive closure of the BN graph 9, denoted by TC(g), then another BN graph, G', where TC(G') TC(G), should provide similar performance. We investigate this for the Tree, Protein and EColi datasets by constructing such alternative graphs. These G' graphs can be constructed by removing each edge in the original graph that does not change its transitive closure, and adding the same number of new edges that are in the transitive closure of G, but not in G itsclf. The type of edge that can be removed without changing the transitive closure, is any edge between a child and one of its parents where this parent is also an ancestor of another one of the child's parents. The above construction ensures that G' has the same number of odges as 9, as well as a matching transitive closure, while specifying a different set of independence assumptions. We also compared using G and g' to using a graph G' where all possible edges have been removed such that TC(9 TC(G). Since the Arithmetic
29

---

Published in Transactions on Machine Learning Research (04/2023)
Circuit and MEHRA BNs do not have any such edges that can be used to modify the dependence structure,
we did not consider them here. Figures 22 and 23 show the graphs g. 9' and gmin used for the Tree and Protein datasets, respectively. Table 10 provides the modelling performance of the small and large GRF models when encoding either g, g' and Mmin for density cstimation. The architectures of these models are the same as given in Table 6
20
1
22
23
20
1
22
2/
20
1
22
23
25
24
24
25
TD
F0
TQ
(a) g
(b) g'
(c) Fmin
Figure 22: Illustration of the BN graphs encoded into the GRFs presented in Table 10 for the Tree dataset. Red edges in (a) the true BN graph were removed and placed between different child-ancestors pairs to create (b) a different graph with the same transitive closure as (a), or removed entirely to obtain (c) a graph with the minimum number of edges that still has the same transitive closure as (a).
PIP2
PEC
Raf
PKA
Mac
PKA
Mck
PIPa
Juk
Iake
PaR
Eak
PIPS
Akt
Akt
(a) g
(b) g'
(c)
Pmin
Figure 23: Illustration of the BN graphs encoded into the GRFs presented in Table 10 for the Protein dataset.
Density estimation performance of GRF's and GRF,, when encoding either the true BN graph, G, alternative graphs g' and Ginin where TC(G) TC(g') TC(G'm.)) still holds. TC is the transitive closure of graph. For each BN, provide the number of edges that were either removed placed bet ween different child-ancestor pairs to obtain the graphs Gin. and g', respectively. Each entry corresponds to the average over the test set for five independent runs, with standard deviation given in the subscript. Lower is better in all cases, The best performance in each group is indicated with bold.
Table 10:
Density estimation (NLL)
BN
#Edges
Flow
g'
Gnin
GRFs
5..66.0.01
-4.8710.04
4.41
Protcin
9
GRF,
-6.12...0
-5.8540.13
-5.6540.09
GRFs 40.06-0.00
40.10
40.27
EColi
12
GRF
40.06
40.07
40.10
GRFs 0.64
8.66+1.000
8.65
Tree
2
GRF, 8.64..0.00
8.6610.00
8.65
30

---

Published in Transactions on Machine Learning Research (04/2023)
The results show that models encoding the original dependency structure provided the best modelling perfor- mance in all cases. This is most noticeable for the real-world Protein dataset. Even for the simple Gaussian EColi BN and for the Tree BN where only two edges had been changed, the original BN graph still provided the best performance. We suspect therefore that even though information leakage may occur, the direct dependencies encoded by the flow given the BN, still play a more important role in informing the resulting distribution. While the GRF formulation does allow dependencies corresponding the transitive closure of the encoded BN graph, we expect the forms of the dependencies on variables beyond the underlying graph to be quite constrained, making learning more sophisticated dependencies between vertices not connected in the underlying BN more difficult. Although future work should investigate this issue in more detail, the above results help to motivate our use of GRFs.
B.3.4 Addressing Posterior Collapse
Here, provide the performance results of STReN-VAFTru the full training set when employing dif- ferent combinations of posterior collapse mitigation techniques. Table II expands the results presented in Section 5.2.2 In addition to warm-up and importance weighted objectives, we also considered using a varir-variance gradient estimator for optimization which is expected to reduce the likelihood that posterior collapse occurs (Melis et al. 2022). Specifically, we employed the doubly-reparammctriie gradient (DReG) estimator of Tucker et al. (2018). As seen in Table 11 using this lower-variance gradient estimator typi- cally resulted in the model achieving better negative lvgdevidence as well as having fewer collapsed latent variables. However, using DReG requires inverting the GRF in the variational posterior (Vaitl et al. 2022) which both slows down training and results in higher memory usage (since the computational graphs of the fixed-point inversion iterat ions need to be stored to facilitate backpropagation). It is therefore not clear how worthwhile the use of DReG to improve results is, given this additional cost.
31

---

Published in Transactions on Machine Learning Research (04/2023)
Table 11: Effect of applying different combinations of posterior collapse mitigation techniques, namely:
warm-up (WU), importance weighted objectives (IWAE) and dbubly-reparaneeleriedd gradient estimates (DReG). Warm-up is applied to either all latents (WUau), or to selected subset that is prone to collapse (WUselect) The performance metries are the same as for Table
BN
Model
logp(x)
#Tnactive Units
SIReN- VAB True
10.03
2.00+0.00
SIReN-IWAL True
9.86_2004
1.00+0.00
Arithmetic
STReN-IWA T DReG
9.80 1..02
1.00
Circuit 2
STReN-VAE WU.
10.11
1.00
SIReN-IWAE WU
9.86_0.08
1.00
SIReN-IWAETrue DReG+ WU
9.88
1.00
SIReN-VAEErud + WU
9.82=0.02
0.00=0.00
BRReN-IWAETrue WUseleet
9.80 A
0.00
SIReN-IWAEThee +DReG+ TDDeeGGG- WUGG-eet
9.80
0.00=20.00
STReN-VAF V
34.99
0.00
SIIReN-IWAE True
34.98 A
0.00=0000
EColi
STReN-IWAE Trm DReG
34.98
0.00
SIReN-VAB True WU
34.99
0.00=0.00
STReN-IWA True WUall
34.99
0.00
BIReN-IWAE11a BefeN-WWABDnn-DDGGG--M +DRGG-WU.I
34.99
0.00
BIReN-VAB. True
37.7320.05
14.80=0.33
STReN-IWA ETruc
37.49 10.26
11.00
SIReN-IWAEE DDeG
37.30
6.00
Arth
SIReN-VAE, WU
37.65
11.20
SIReN-IWAE True WU
37.48 47
10.20
STReN-IWAETT... DReG+ -UUI
37.27 10.54
7.80
SIReN-VAE + WUseleet
37.71
9.80
SIReN-IWA AA WUseeret
37.42.0.24
7.00
SIReN-IWAETTe+ +DRG=++++++W WUs
37.43+0.22
1.80
SIReN-VAEThee
8.37=0.00
0.00=0.00
MEHRA
SIReN-IWAE, Irree
8.19=0.06
0.00=
STReN-IWAF E DReG
DDRG
8.08+0.02
0.00=00.0
STReN-VAE WUal
8.66 2.66
0.00
SIReN--WAETTme WUal
8.39
0.00=0.00
SIReN-IWAEThee -DAAAABBnnn--GG------
8.23=2008
0.00=0.00
32


Reviews:
[{"review_id": "6LTqZLYkH0", "reviewer": "Reviewer_YbNy", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n\n1. The paper is very well written, with a high quality review of related literature and clear introduction of background knowledges. Graphical illustration is also very helpful.\n2. The idea of introducing GRF to both the prior and variational posterior in order to model the dependency is interesting, and the success of it highly depends on the novel design of GRF which allows fast and accurate inverse computation.\n3. The experimental results are comprehensive, with multiple synthetic and real datasets included. Results do show some improvement over baselines. Although the improvements are not very significant, as the author claimed, the method has its own advantage when the training set is small.\n\nWeakness:\n\n1. I think the masking mechanism is an important component of the model, and it should be clearly introduced in the main text rather than in appendix. As an exchange, some background information may be moved to the appendix. \n2. While incorporating explicit information regarding dependency in training latent variable models is definitely an interesting problem, it would be better to introduce  more background/motivation for studying this problem. In particular, since in reality the explicit dependency is difficult to obtain (that's why so many people are studying causal inference), what are the potential application of this method? Is there any cases in real life that the model can be useful? These would let the reader better appreciate the work.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper tackle the issue of how to incorporate dependency structure (represented by a bayesian network) into the training of generative models (in particular, VAEs). The paper proposes to Graphical Residual Flow (GRF), which is a graphical normalizing flow whose inverse can be accurately computed, to encode the predefined dependency structure over the latent and observed variables. To preserve the dependency information, a masking mechanism is used to ensure that the output at each dimension is only a function of the itself and its parents. Experimental results show that VAE equipped with GRF obtains better likelihood.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "As said in my comments, I would request authors to include introduction of some missing parts of the model in the main text, as well as a section that tells the motivation/application of this model.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ewcOO4DWuY", "reviewer": "Reviewer_1tzL", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Overall I find this work interesting and would think TMLR is good venue it. However, I have some concerns I would like to see addressed before voting for acceptance. Providing a public implementation of this work is also very important in my opinion.\n\nFor now, I would attribute a \"no\" to the claims and evidence but I am quite convinced this should become a yes within the scope of a discussion with the authors and a few additional experiments.\n\n# Weaknesses:\n- I find the title misleading as there already exist NFs architectures integrating Bayesian networks.\n- The contribution mainly builds on existing work (I-resnets and GNFs) and may seem incremental.\n- The paper is lacking a good covering of independence in Bayesian networks whereas this is the key point in assuming a Bayesian Network structure.\n- Some experiments could be pushed further in order to provide better insights regarding the value of the proposed methods (see requested changes).\n- It is difficult to decipher what exactly is the key contribution here. To add structure within VAEs? To combine iresnet with graphical constraints? I think this should be clarified. \n- As the technical contribution is somehow incremental, I would expect a better evaluation of the interest of incorporating structure into VAEs. To me, the current paper is not making this point unarguable although I am convinced this is indeed a good idea in many cases.\n\n# Strengths:\n- Building a better alternative of SCCNF and GNF is an interesting research direction as many deep probabilistic models are missing a point by not exploiting domain knowledge about independence between the random variables modelled. In particular, embedding the Graphical structure directly in the neuron's connectivity, similarly to MAF and MADE is a nice to have.\n- The paper reads well.\n- The methods works well compared to GNF and SCCNF, this is a really important point.\n- A lot of effort has been made to create a method that is stable numerically. The experiments and discussion about the posterior collapse are interesting.\n- I also really liked the idea of using the structure inversion algorithm! ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces a new normalizing flow architecture that combines Bayesian Network structures, such as in graphical normalizing flows (GNFs) and SCCNF with invertible residual networks. The new architecture is naturally more stable for inversion and perform similarly to existing NFs embedding graphical constraints. \nThe authors show the architecture is effective in scarce data settings if we can hypothesise a good Bayesian network structure.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- I would recommend to push further the experiments with various degree of mistakes in the graphical model used. A thorough analysis would provide valuable insights as when we can hope to gain by embedding graphical constraints which, in many cases, will not correspond to the \"true\" independencies. \n- Cover better the topic of independencies in BNs.\n- Figure 3 should be cleaned.\n- Provide an experiments where a VAE has really a good reason of being there, e.g., if the observed signal is an image. I would be happy to iterate a bit with the authors to find something that is really nice here. In my opinion, this is an important aspects that could really improve the impact of this publication.\n- Improve the discussion on failure points of the approach. For instance, learning with such constraints on the factorisation can lead to very complicated 1d distributions that hard to model with neural nets whereas a simpler architecture would work well. \n- For the experiment with the VAE, also provide the numbers obtained with a simpler NFs (e.g. a 3-layer MAF or UMNN-MAF)\n- Explain how you compute the MI for Figure 5.\n- Plot the non-standard LipMIsh activation function.\n- A discussion of strategies to apply when only partial knowledge of a good graph is known would be interesting.", "broader_impact_concerns": "I do not see any broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "1ocYBv4jCo", "reviewer": "Reviewer_RTpd", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "*Strengths*:\n+ The idea of incorporating a structure into (generally speaking) latent-variable models is interesting and promising.\n+ The paper proposes a new flow-based model, dubbed Graphical Residual Flows (GRFs), which takes into account a BN-based structure.\n+ The paper proposes a new VAE model, dubbed SiReN (structured invertible residual network) VAEs, which use a BN to form the latent space.\n+ The experiments introduce some evidence for using the proposed methods.\n\n*Weaknesses*:\n- WQ1: The paper heavily builds upon two papers: (Wehenkel & Louppe, 2021) and (Weilbach et al., 2020). In such a situation, it is very important to highlight all differences and new ideas. I miss an example of how to formulate masks to introduce a BN. At the moment, it is quite hard to see the difference between previous papers and the proposed method. Figure 1 tries to do that but I miss a more in-depth presentation and discussion.\n- WQ2: The experiments are interesting, however, I miss a good baseline. It would be greatly beneficial to run vanilla residual flows and report their performance. It would be more convincing if the flows with a BN outperforms this baseline.\n- WQ3: In Table 3, why does SIReN-VAE perform sometimes worse (especially for the full training data) than SIReN-VAE-FC or VAE?\n\n*Open questions*:\n- OQ1: Would it be possible to extend the proposed idea to invertible DenseNet flows? (Perugachi-Diaz, Y., Tomczak, J., & Bhulai, S. (2021). Invertible densenets with concatenated lipswish. Advances in Neural Information Processing Systems, 34, 17246-17257.)\n- OQ2: Would it make sense to do a two-stage training of SIReN-VAEs, more along (Dai, B., & Wipf, D. (2019). Diagnosing and enhancing VAE models. arXiv preprint arXiv:1903.05789). I can imagine that learning the variational posterior together with the GRF-based prior may not be beneficial from the optimization perspective. First training with a \"naive\" prior like standard Gaussian, and then turning on a more elaborated prior may result in even better results.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper focuses on incorporating a structure into random variables in a form of a Bayesian network (BN) to flow-based models and Variational Auto-Encoders (VAEs). For flow-based models, the idea is to mask weight matrices in such a way that the BN is retained. For VAEs, the latent space if factorized according to a given BN. In a series of experiments, the proposed idea seems to be preferable over other methods, especially for higher dimensional problems.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "R1: A better presentation and discussion of the proposed method (i.e., how to mask). See WQ1.\nR2: Additional baseline (e.g., residual flows). See WQ2.", "broader_impact_concerns": "No concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper already went through three revisions during the discussion with the reviewers. In these revisions, the authors expanded and clarified the content of the paper, added additional experiments, and addressed all reviewers' concerns. I don't think the paper needs an additional revision, so I'm happy to recommend acceptance as is.

To the authors: please don't forget to add the link to the code in the camera-ready version of the paper.

Decision: Accept (poster)