Embedded-model flows: Combining the inductive biases of model-free deep learning and explicit probabilistic modeling
Authors: Gianluigi Silvestri
Emily Fertig
Dave Moore
Luca Ambrogioni
Venue: ICLR 2022 Poster (2022)
Keywords: Normalizing Flows
Probabilistic model
Probabilistic programming
Generative modeling
Variational Inference

Abstract: Normalizing flows have shown great success as general-purpose density estimators. However, many real world applications require the use of domain-specific knowledge, which normalizing flows cannot readily incorporate. We propose embedded-model flows (EMF), which alternate general-purpose transformations with structured layers that embed domain-specific inductive biases. These layers are automatically constructed by converting user-specified differentiable probabilistic models into equivalent bijective transformations. We also introduce gated structured layers, which allow bypassing the parts of the models that fail to capture the statistics of the data. We demonstrate that EMFs can be used to induce desirable properties such as multimodality and continuity. Furthermore, we show that EMFs enable a high performance form of variational inference where the structure of the prior model is embedded in the variational architecture. In our experiments, we show that this approach outperforms a large number of alternative methods in common structured inference problems.

Published as a conference paper at ICLR 2022
EMBEDDED-MODEL FLOWS: COMBINING THE INDUC- TIVE BIASES OF MODEL-FREE DEEP LEARNING AND EXPLICIT PROBABILISTIC MODELING
Gianluigi Silvestri
Emily Fertig, Dave Moore Google Research San Francisco, CA, USA {emi lyaf, davmre com
OnePlanet Research Center
imee-the Netherlands
Wageningen, Netherlands
Maanligg... silvestriigg mec. n.l
Luca Ambrogioni
Department of Artificial Intelligence
Donders Institute for Brain, Cognition and Behaviour Radboud University
Nijmegen, Netherlands
ammorogginnnidonnerr. ru, ru.
ABSTRACT
Normalizing flows have shown great success as purppse density estima- tors. However, many real world applications require the use of domain-specific knowledge, which normalizing flows cannot readily incorporate. We propose embedded-model flows (EMF), which alternate general-purpon transformations with structured layers that embed domain-specific inductive biases. These layers are automatically constructed by converting user-specified differentiable proba- bilistic models into equivalent bijective transformations, We also introduce gated structured layers, which allow bypassing the parts of the models that fail to capture the statistics of the data. We demonstrate that EMFs can be used induce desirable properties such as multimodality and continuity. Furthermore, we show that EMFs enable high performance form of variational inference where the structure of the prior model is embedded in the variational architecture. In our experiments, we show that this approach outperforms large number of alternative methods common structured inference problems.
INTRODUCTION
Normalizing flows have emerged in recent years strong framework for high-dimensional density estimation, a core task in modern machine learning (Papamakarios et al., 2021] Kobyzev et al.) 2020: Kingma Dhariwal, 2018). As with other deep generative architectures such GANs and VAES, normalizing flows are commonly designed with the flexibility to model very general class of distributions. This works well for large naturalistic datasets, such as those common in computer vision. However, general- purpose flows are less appealing when datasets are small or when we possess strong priori knowledge regarding the relationships between variables.
On the other hand, differentiable (or 'deep') probabilistic programming is a powerful and general framework for explicit stochastic modeling that can express strong assumptions about the data- generating process (Tran et al. 2017,201 Bingham et al., 2019 Dillon et al. 2017: Piponi et al.) 2020 Kucukelbir et al.) 2017 Ambrogioni et al. 2021a). For example, the user may know that the data represents process evolving over time in space, is subject to known physical laws, or that there is an unobserved common cause or other shared factor behind set of observations. Such structured prior information would otherwise be required. For example, in the field of astrophysics. differentiable probabilistic programs have been successfully used to study strong gravitational lensing (Chianese al.) 2020) and
enable the user to form effective inferences using much less data than

---

Published as a conference paper at ICLR 2022
Model
Model
mn 2 d8t te selly
A (o (
MAF
MAP
MAF
Figure I: Didactic diagram a hypothetical embedded- modell architecture with model-free masked autoregressive flow (MAF) layers and two gated layers corresponding two different probabilistic models.
dark matter substructures (Coogan al.) 2020 Varma et al. 2020). All of these applications entail the translation of sophisticated physical theories into highly parameterized and structured probabilistic programs. However, it remains the case that "all models wrong" (Box 1976), and it is rare that modeler can be fully confident the strong assumptions (e.g.. Gaussianity, statistical independence) expressed by typical parametric probability models.
In this paper we bridge these two worlds, providing an automatic technique to convert differentiable probabilistic programs into equivalent normalizing flow layers with domain-specific inductive biases. We call these architectures embeulee-mood flows (EMF). EMFs can be use to integrate the structure of "wrong but useful" models within generic deep learning architectures capable of correcting the deviation of the model from the data. Furthermore, in the context of Bayesian inference, we show that the EMF architecture can be used define powerful class of approximate posterior distributions that embed the structure of the prior.
2 PRELIMINARIES
Normalizing flows define a random variable as an invertible differentiable transformation fo (x)) of a base variable (0) ~ (o)) In these expressions, 6 are parameters that control the form of the transformation and consequently the distribution of y. The transformed (log- deegsiyy can then be given explicitly using the change of variables formula from probability theory,
log pby) log (f (z)) - log |det (3))),
(1)
where (3) is the Jacobian matrix of fa. The base distribution po is often taken to be standard normal (0) NO0, I)). In typical applications, the functional form of S is specified as the composition series of simpler invertible transformations:
y ()) 100
(m 1) (0)
(2)
times
We denote the resulting sequence of transformed variables (0) x(1)
.y. Each of these transformations (i.e. flow layers) are designed to have tractable formulas for inversion and log-Jacobian. The parameters d are usually trained by stochastic gradient descent to maximize the log-likelihood (given observed data) evidence lower bound (in the variational inference setting).
(m--1)
2.1 PROBABILISTIC PROGRAMS AND DIRECTED GRAPHICAL MODELS
Probabilistic programming allows a joint probability distribution over a set random variables to be specified intuitively program that includes random sampling steps (van de Meent al. 2018).
2

---

Published as a conference paper at ICLR 2022
Computations on this distribution, including the joint log-density function, structured inference algo- rithms, and the bijective transformations described in this paper, may program transformations. Recently frameworks have emerged for 'deep' probabilistic programming, which exploit the automatic differentiation and hardware acceleration provided by deep learning frameworks; examples include Pyro (Bingham al. 2019), PyMC (Salvatier al., (006). Edward (Tran et al., 2017). and TensorFlow Probability (Dillon et al., 2017 Piponi et al.] 2020). These allow for straightforward implementation of gradient-based inference and parameter learning, and enabling methods such as ours that integrate probabilistic programs with standard "deep learning' components
may then be derived automatically as
such as flows.
Probabilistic programs that contain only deterministic control flow may equivalently be viewed as directed graphical models, in which each random variable 2; is graph node connected by incoming edges to some subset of the previous variables æzj. This leads to the following general expression for the joint density of a set of vector-valued variables 2im = {m,)
p( i Φ) = (ro (o()))
II py 0, (x<=
Here (z<zz; Φ) denotes the 'link function' associated with each node that takes as input the model parameters & and the values of all parent variables (excepting to where there are no parents), and outputs the parameters of its respective density function.
As practical matter, flow architectures generally assume layers of fixed size, so the experiments in this paper focus on programs of fixed structure and we use graphical-most notation for simplicity. However, the methods we describe do not require access to an explicit graph structure and could be straightforwardly applied to programs with stochastic control flow.
CONVERTING PROBABILISTIC PROGRAMS INTO NORMALIZING FLOWS
In this section, we will show that large class of differentiable probabilistic programs can be converted into equivalent normalizing flow layers. More precisely, we will construct an invertible function F that maps spherical normal random inputs to the joint distribution of the probabilistic program:
x (x( p(x(");))
(z) NO I))
This is a special case of the so called Rosenblati transform (Rosenblat) 1952). We interpret this transform flow layer and extend it by including trainable gate variables that can be used to "discomnect" a set of variables from its parents so to correct model miss-specification. In our applications, the input to the bijective transformation may be arbitrary real-valued vectors, for example, the output of a previous flow layer, which likely will not be normally distributed. In such cases the transformed values will not be samples from the original probabilistic program, but we expect the transformation to capture some of its properties, such as multimodality and the dependence structure of its random variables.
3.1 INDIVIDUAL RANDOM VARIABLES AS BIECTIVE TRANSFORMATIONS
We first consider the case of a univariate random variable (1) P (2) which
express as continuous bijective transformation f of standard normal variable (0) via slight extension of inverse transform sampling. The probability integral transform theorem states that the cumulative distribution function (CDF) of Ps. Co(x) J"" Po (y)dy, with P defined over a generic interval b, is uniformly distributed, u Cx) (0, 1) (Angus) 1994). If the CDF is
(a, b) and invertible, i.e., strictly increasing within its domain, it follows that can sample x C ps given only uniform random variable (d( 1). We obtain u through a second application of the probability integral transform theorem to the standard normal CDF Po.1. Our univariate "flow' is therefore given by
(1) = fo(((0) (2 (z)
(z()) N' 1))
(5)
Closed-form expressions for fo may be available in some (see below). In the general univariate setting where the inverse distribution function C=1(u)
(u) is not available in closed form, it may
3

---

Published as a conference paper at ICLR 2022
be evaluated efficiently using numerical root-finding methods such the method of bisection (see Appendix A.6)). with derivatives obtained via the inverse function theorem and the implicit reparameterization trick (Figurnov et al. 2018). However, this is not needed for maximum likelihood training of the parameters Φ since eq. (1) only involves f and its derivatives.
Example: Gaussian variables. The inverse CDF of Gaussian variable x;()) NN,, o²) can be expressed as scale-shift transformation of the inverse CDF of a standard normal variable: (w) p + (.). We can obtain the forward flow transformation by composing this function with the CDF of a standard Gaussian: Jn.- (x(())
$ T ( oo (xx")) = + ox. This is the famous reparamcterization formula used to train VAEs and other variational methods (Kingma & Welling 2013). The inverse function is f (x:)) = (xx(1) 11/o while the log-Jacobian is equal to o.
is not generally straightforward evaluate invert the CDF of multivariate distribution. However, the most popular multivariate distributions, such as the multivariate Gaussian, can be expressed as transformations of univariate standard normal variables, and so may be treated using the methods of this section. More generally, the next section presents simple approach to construct structured bijections (i.e., flows) from multivariate distributions expressed as probabilistic programs.
3.2 FROM DIFFERENTIABL PROBABILISTIC PROGRAMS TO STRUCTURED LAYERS
A probabilistic program samples a set of variables sequentially, where (as in eq. (3) the distribution of each variable is conditioned on the syrmplssy--sapped parent variables. Using the results of
Section 3.1 we can represent this conditional sample as an invertible transformation of a standard normal input. Feeding the result back into the model defines a sequence of local transformations, which together form multivariate flow layer (algorithm D. More specifically, with autoregressive flows, the forward transformation of the j-th input requires us to have computed the preceding outputs (6+1) This may be expressed recursively as follows: tz
(++1) ,10 (ae) (41))
a i:n
where (()) (:: x ()+1)) denotes the invertible transformation associated to the conditional distribution of the j-th variable given the values of its parents. The recursive expression terminates since the parenting graph has an acyclic-directed structure.
(++1) are all given, so all transformations may be evaluated in In the reverse direction, the variables x Lin
parallel:
(i) - 1, (00 (1, 2 ((+2))
This is generalization of the inversion formula used in autoregressive flows (Kingma et al. 2016 Papamakarios et al. 70. The Jacobian of the transformation is block triangular (up to a permutation of the variables) and, consequently, the log determinant is just the sum of the log determinants of the block diagonal entries:
log det J (xx ((
Σ log det J7d (1t))
(a 2 (tn))]
(8)
where 1j. (ar)itt)) 2
(++1) is the Jacobian of the local inverse transformation f/100 (x0
(1+1) (++1))
x
We denote this multivariate invertible transformation a standard Gaussian distribution, then the transformed variable follows the distribution determined by the input probabilistic program. This is not going to be the case when the layer is placed in the middle of larger flow architecture. However, it will still preserve the structure of the probabilistic program with its potentially complex chain of conditional dependencies.
structured layer. If the input æ i follows
(t)
Gated structured layers Most user-specified models offer a simplified and imperfect description of the data. In order allow the flow skip the problematic parts of the model, we consider gated layers. In a gated layer, we take each local bijective transformation Ii to be a convex combination of the original transformation fe and the identity mapping:
(10
(1) (1) (++1) x200 2/0 1.
(n an (0+1)) + (1 ->)7)aaf
4

---

Published as a conference paper at ICLR 2022
(2) ) = Input variables sorted Algorithm 1 Gated forward transformation: Φb: Model parameters; {at respecting parenting structure; { FE=1: Gate parameters, n Number of variables.
procedure GATEDFORW.RDSSMMPIIRRRRS. (2) )==))) for j in Range(n) do
)i-1) State log det J| 0
(8) {pr 2j parents.idx
{pi l= indices of parents of j-th variable. my: number of parents of j-th variable.
(pth- the is not empty then
$ a(it1) j.
(6) (6)
(1+1) y
zy fitl) t 2 A
)a)) (1)
log det J| t log det. J| + log det [ (1+1)
)k=1) (1- )2))))
else
(6+1)
t (() (6)
j x,)
oj: parameters of the j-th root distribution.
j [++)) -- 2 zj
(8) 1)
+ (1 2, )ar
log det J| + logl det J/| + log det |J j.gg (z) (2t + (1 11)
(1+1) in 1R=11 log J
return {*t
This gating technique is conceptually similar the method used in highway networks (Srivastaval et al. 2015). recurrent highway networks (Zilly et al. (017)) and, more recently, highway flows Ambrogioni al. 2021b). Here the gates X, (0, 1) are capable of decoupling each node from its parents, bypassing part of the original structure of the program. As before, the local inverses 96 (x (++1), x (++1) A/0) can in general be computed by numeric root search, but the Gaussian case admits closed form in both directions:
x(i+1) Ap 1+ A(o - 1)
9u.o (z()))) (1 A(o (1) + Ap=
(x(+1)) .)
(10)
Analogues of eq. 6 eq. 7 and eq. (8) then define the gated joint transformation, its inverse and log Jacobian determinant, respectively. Note that the gating affects only the diagonal of the Jacobian, so the overall block-triangular structure unchanged.
3.3 AUTOMATION
All the techniques described in this section can be fully automated in modern deep probabilistic programming frameworks such as TensorFlow Probability. A user-specified probabilistic program in the form given by Eq. iss recursively converted into bijective transformation shown in the pseudo-code in Fig. In TensorFlow Probability, the bijector abstraction uses pre-implemented analytic inverse and Jacobian formulas and can be adapted to revert to root finding methods otherwise.
3.4 EMBEDDED-MODER FLOWS
EMF is normalizing flow architecture that contains one or more (gated) structured layers. EMFs can be used to combine the inductive biases of explicit models with those of regular normalizing EMF both in parallel or
flow architectures. Several probabilistic programs can be embedded sequentially. In the parallel case, a single flow layer is comprised by two or more (gated) structured layers acting on non-overlapping subsets of variables. The use of several programs embedded in single architecture allows the model to select the most appropriate structure during training by changing the gating parameters and the downstream/upsttraan transformations. This allows the to start with "wrong but useful model" and then learn the deviation from the distribution of the collected data using flexible generic normalizing flow transformations, A visualization of a hypothetical EMF architecture is given in Fig.
5

---

Published as a conference paper at ICLR 2022
4 RELATED WORK
Embedded-model flows may be seen as generalization of autoregressive flows (Kingma et al. 2016 Papamakarios al., 2017) in the special case where the probabilistic program consists of Gaussian autoregression parameterized by deep link networks, although generic autoregressive architectures often permute the variable order after each layer so that the overall architecture does not reflect any specific graphical structure. Graphical normalizing flows (Wehenkel & Louppe 2 generalize autoregressive structure arbitrary DAG structures; however, this approach only constrains the conditional independence structure and does not embed the forward pass of user-designed probabilistic program.
In the last few years, the use of tailored model-based normalizing flow architectures has gained substantial popularity. Perhaps the most successful example comes from physies, where gauge- equivariant normalizing flows used to sample from lattice field theories in way that guarantees preservation of the intricate symmetries fundamental fields (Albergo et al., 2019) Kanwar et al., 2020 Albergo et al. 2021 Nicoli et al. (021). Similarly, structured flows have been used model complex molecules (Satorras et al. 202 1). turbulence in fluids (Bezgin & Adams (021), classical
dynamical systems (Rezende al. 2019 Li al.) (0200) and multivariate timeseries (Rasul al.
2020). These approaches are tailored to specific applications and their design requires substantial
machine learning and domain expertise.
Normalizing flows have been applied to variational inference since their introduction by Rezende & Mohamed (015). Recognizing that structured posteriors can be challenging for generic architectures. number of recent approaches attempt to exploit the model structure. Structured conditional continuous normalizing flows (Weilbach et al. 2020) use minimally faithful inversion to constrain the posterior conditional independence structure, rather than embedding the forward pass of the prior model. Our architecture can be seen as a flexible extension of previous variational models that also embed the prior structure, such as stochastic structured variational inference (Hoffman & Blei 2015), automatic structured variational inference (Ambrogioni et al., 2021a) and cascading flows (Ambrogioni et al., (2021b). It is also closely related to the non con-eente transformations applied by Gorinova et al. (2020) to automatically reparameterize a probabilistic program, although that work focused on improving inference geometry and did not directly exploit the bijective aspect of these reparameterizations.
APPLICATIONS
Model-embedding allows users of all levels of expertise to insert domain knowledge into normalizing flow architectures. We will illustrate this with some examples, starting from simple uncoupled multi-modal models and moving we will discuss the application of EMF to automatic structured variational inference, where we use EMF architectures that embed the prior distribution. Details about the datasets used, the models' hyperparameters and the training procedures can be found in Appendix [..e A.3 and A.4 respectively. Additional experiments with an hierarchical structure can be found in appendix A.7.3 Number of parameters and sampling time for all the models are reported in appendix A.3 All the code used in the experiments is available at wtthttts://// //tp::///githbb com/ ott
more sophisticated structured models for time series. Finally,
5.1 MULTIMODALITY
Multimodal target densities are notoriously hard to learn for traditional normalizing flow architectures (Cornish et al.) 2020).. In contrast, straightforward to model multimodal distributions using explicit mixture models: the simplest example is perhaps model where each variable follows an independent mixture of Gaussian distributions:
Pp.... (x)
-II (an
(11)

---

Published as a conference paper at ICLR 2022
The CDF of mixture of distributions is given by the mixture of the individual CDFs. Therefore, the inverse transformation associated with a mixture of Gaussians model is
p.p. (x) = =0.1
Σ Ps p. (x)
(12)
In itself, the probabilistic program in Eq. 12 can only model very specific family of multivariate mixture distributions with Gaussian components organized in an axis-aligned grid. However, the model becomes much more flexible when used structured layer in EMF architecture since the generic upstream layers can induce statistical coupling between the input variables, thereby loosening the independence structure of the original model. The resulting approach is conceptually similar to architectures based on monotonic transformations such as neural spline flows (Durkan et al. 2019a b). which are arguably more efficient as they have an analytic inversion formula. However, this application shows that a specific inductive bias such as multimodality can be effectively incorporated into a flow architecture by users with only a basic understanding of probabilistic modeling.
We apply an EMF with an independent mixture of Gaussians to two 21 toy problems common in the flow literature, namely the "eight Gaussians" and the "checkerboard", and to the MNIST dataset. As baselines, we use a Masked Autoregressive Flow (MAF) model with two autoregressive layers and standard Gaussian as a base density, a "large" version of MAF (MAF-L) with three autoregressive layers, which has much more flexibility compared to the other models in terms of trainable parameters, and a six steps Neural Splines Flows (NFS, with Rational Quadratic splines) with coupling layers as proposed in (Durkan et al. (011b)). We then combine the EMF with the MAF baseline by adding an EMF on top of the two autoregressive layers (EMF-T, for top) and in between the two layers, after the latent variables permutation (EMF-M, for middle). We also combine NSF with the structured layer, both with the EMF-T and EMF-M settings. Each mixture has 100 components, with trainable weights, means and standard deviations. The results are reported in table I while densities are shown in appendix 1 The use of the multimodal structured layer with MAF always outperforms the autoregressive models. The improvement not solely due to the increased parameterization both methods outperform the more complex baseline MAF-L. NSF achieves superior results, as it is an architecture designed to model multimodality. The combination of the structured layer with NSF achieves the overall best performances. However, this comes at the cost of additional sampling time, caused by the numeric inversion of the mixture of Gaussians transformation.
Table Results in negative log probability on one million test samples for the 2d toy problems. We report mean and standard error of the mean over five different runs. 8G stands for eight gaussians while CKB for checkerboard. The resulting negative log probability for MNIST is computed on the test set.
FMF-T
FMF-M
NSF-EME-T
NSF-EMF-M
MAF
MAF4J
NSF
SCT
2,881 0.000
2.897 + 0.002
2.837 + 0.003
2.829 = 0.000
3.341 0.009
3.032 0.018
2,832 0.004
CKB
3.570 +: 0.000
3.503 + 0.000
3.480 + 0.001
3.494 0.000
3.795 + 0.000
3.614 4: 0.006
3.461 +: 0.002
MNIST
390.64155 0.504
1024.065 4 1.285
958.960 :: 0.695
961.078 0.316
1310.943 : 0.676
1291.870 +: 0.951
960. 000 0.838
5.2 TIMESERIES MODELING
Timeseries data is usually characterized by strong local correlations that can often be described with terms such as continuity and smoothness. In the discrete models considered here, these properties can be obtained by discretizing stochastic differential equations. One of the simplest timeseries model with continuous path is the Wiener process (t) = w()., with w(t) being white noise input with variance o². By Maruuaaaa discretization with At 1, we obtain the following autoregressive model
It+1 ~ N( 0²)
(13)
which gives us the bijective function f++1. (1:)1iT;) = Jg + 0T1+1. We use four stochastic differential equation models time series datasets, discretized with the Euler-Maruyama method: the models are Brownian motion (BR), Ornstein- -hiiebbbeh (OU) process, Lorenz system (LZ), and Van der Pol oscillator (VDP). We use EMF with continuity structure. The baselines are MAF, MAF-L and NSF models like in the previous section, and an MAF in which the base distribution follows the continuity structure (B-MAF), as discrete-time version of the model proposed in (Deng
7

---

Published as a conference paper at ICLR 2022
et al. 2020). Note that no variables permutation is used for B-MAF. We then GEMF-T and combination NSF with the structured layer (NSF-GEMF-T), with continuity structure. The results reported in Table 2 The baselines combined with continuity structure greatly outperforms the other models (even the more complex MAF-L) for all of the datasets. Results obtained with the smoothness structure reported in appendix A.7.4
Table 2: Results in negative log probability on one million test samples for the time-series toy problems, and on a test of 10000 datapoints. We report mean and standard error the mean over five different runs.
GEME-T(e)
NSF-GIFMP-TTC)
MAF
MAF-1.
NSF
B-MAF
-26.221
0.0103
0.0099
0.0585
-25.905
OU
24.086 + 0.0015
27:274 + 0.0858
24.137
0.0028
24.162 + 0.0029
28.425 = 0.0400
24.169 + 0.0034
LZ
-211 301
0.2253
224.981
0.2152
-194,240
0.3240
-201.311
0.9729
203.270
0.4950
-218.351 A 0.0960
562.081
0.0446
568.099 0.1478
5117119
0.2567
558.032
0.2753
533.506 0.7265
-657.420 0.1056
5.3
VARIATIONAL INFERENCE
Prior model
architecture
A/t :
IAF
(1 )o
Figure 2: Diagram of gated prior-embedding architecture with sample of transformed variables for a Lorentz dynamics smoothing problem.
Normalizing flows are often used as surrogate posterior distributions for variational inference in Bayesian models, where the target posterior may contain dependencies arising both from the prior structure and from conditioning on observed data via collider (or "explaining away") phenomena. Generic flow architectures must leam the prior structure from scratch, which wastes parameters and computation and may lead the optimization into suboptimal local minima. Embedding the prior model into the flow itself gated structured layer avoids this difficulty and should allow the remaining layers to focus their capacity on modeling the posterior.
We deploy a gated EMF architecture with two-layers of inverse autoregressive flow (IAF) and final structured layer embedding the prior model. This is visualized in 2 for inference in a Lorentz dynamical system. In addition, train plain IAF, automatic differentiation Mean Field (MF) and Multivariate Normal (MVN) surrogate posteriors from Kucukelbir et al. (017)) and ASVI from Ambrogioni al. (2021a). We experiment on both time series and hierarchical models. The time series models are Brownian motion (BR). Lorenz system (LZ) and Van der Pol oscillator (VDP), with Bernoulli (classification) emissions. Furthermore, we either observe all the emissions (smoothing setting) or we omit central time window (bridge setting). As hierarchical models we use the Eight Schools (ES) dataset and a Gaussian binary tree model with different depths and link functions, in which only the last node is observed (Ambrogioni et al. b021 The results are shown in Table 3. Results on additional experiments can be found in Appendix A.7.5) GEMF-T outperforms the baselines all the problems (on-par with some baselines on the most trivial problems, Appendix
A.7.5). showing the benefits of embedding the prior program for highly structured problems, and the capability of such architecture model collider dependencies.
8

---

Published as a conference paper at ICLR 2022
Table 3: Results in negative ELBO. We report mean and standard error of the mean over ten different runs. S and B correspond respectively to smoothing and bridge, while stands classification. As an example, BRB-c is the Brownian motion with the Fidge-classification settings. For the binary tree experiments, Lin and Tanh correspond to the link function (Lin is linear link), and the following number is the depth of the tree.
GEMF-T
MF
MVN
ASVI
IAF
BRS-c
15.882 + 1.405
23.761 + 1.262
16.148 + 1.394
15.894 + 1.403
15.950 + 1.393
BRB-c
11.880 + 0.990
20.174 0.883
12.098 + 0.985
11.884 0.987
11.947 + 0.985
175-cc
8.317 = 1.033
60.458 + 0.511
43.024 + 0.621
26.296 + 2.008
27.604 + 0.717
LZB-c
5.819 + 0.492
53.418 + 0.394
36.063 + 0.449
19.318 + 1.722
20.778 + 0.542
VDPS-c
68.385 + 2.599
162.336 1.425
183.020 1.676
70.884 2.247
88.284 +: L.437
VDPB-c
43.314 + 2.039
138.103 + 1.766
157.736 2.146
49.415 + 1.819
63.914 + 1.502
ES
36.140 + 0.004
36.793 + 0.039
36.494 + 0.014
36.793 + 0.039
36.169 + 0.007
Lin-8
2.596 + 0.213
108. 869.869
0.467
13.834 + 0.271
26.232
0.340
3.440 0.246
Tanh-8
1.626 + 0.159
144.668 + 0.349
44.230 + 0.258
14.241 + 0.650
4.127 0.220
DISCUSSION
We introduced EMF, technique to combine domain- inductive biases with normalizing flow architectures. Our method automatically converts differentiable probabilistic programs into bijective transformations, allowing by choosing appropriate inductive biases, EMF can improve over generic normalizing flows on a range of different domains, with only a negligible increase in complexity, and we achieved high performance on a series of structured variational inference problems. EMF is a powerful generic tool which can find several applications both in probabilistic inference and more applied domains. Limitations: I) While we can map very large family of probabilistic models to normalizing flow architectures, the approach is still limited by the topological constraints of bijective differentiable transformations (Cornish et al. (020)). This forces the support of the program to be homeomorphic to the range, making impossible to map models with non-connected support sets into normalizing flows. II) While the root finding methods used for inverting univariate densities are reliable and relatively fast, they do have higher computational demands than closed form inversion formulas. IIII While it is possible model discrete variables using Eq. 6 this does not result in invertible and differentiable transformation that can be trained using the standard normalizing flow approach. IV) The probabilistic program must be executed during sampling. This result in significant sampling slow-downs when using complex autoregressive programs. Note that, except in the case of variational inference, this does not lead to training slow-downs as the inverse transform is always trivially parallelizable. Future work: Embedded-model flows are a generic tool that can find application in several domains in which there is structure in the data. theory, any number of structured layers can be added to EMF architectures, and combining different structures in the same model may help model complex dependencies in the data. Further empirical work is needed to asses the capacity of trained EMFs to select the correct (or the least wrong) model. While we only considered the use of structured layers normalizing flows, they may also be used any kind of deep net in order to embed domain knowledge in tasks such as classification, regression and reinforcement learning.
to easily embed their knowledge into their models. We showed how,
ACKNOWLEDGEMMNTS
OnePlanet Research Center aknowledges the support of the Province of Gelderland.
REFERENCES
M. S. Albergo, G. Kanwar, and P.E.. Shanahan. Flow-based generative models for markov chain monte carlo in lattice field theory. Physical Review D, 100(3):034515, 2019.

---

Published as a conference paper at ICLR 2022
M. S. Albergo, D. Boyda, D. C. Hackett, G. Kanwar, K. Cranmer, S. Racanière, D. J. Rezende, and E. Shanaham. Introduction to normalizing flows for lattice field theory. arXiv preprint arXiv:2101.08176, 2021.
L. Ambrogioni, K. Lin, E. Fertig, S. Vikram, M. Hinne, D. Moore, and M. Gerven. Automatic structured variational inference. International Conference on Artificial Intelligence and Statistics, 2021a.
L. Ambrogioni, G. Silvestri, and M. van Gerven. Automatic variational inference with cascading flows. International Conference on Machine Learning, 2021b.
J. E. Angus. The probability integral transform and related results. SIAM review, 36(4):652-654, 1994.
D. A. Bezgin and N. A. Adams. Normalizing flows novel pdf turbulence model. arXiv preprint arXiv:2101003590, 2021.
E. Bingham, J. P Chen, M. Jankowiak, F. Obermeyer, N. Pradhan, T. Karaletsos, R. Singh, P. Szerlip,
Horsfall, and N. D. Goodman. Pyro: Deep universal probabilistic programming. The Journal of Machine Learning Research, 20:973-978, 2019.
G. E. P. Box. Science and statistics. Journal of the American Statistical Association, 71(356): 791-799, 1976.
R. Chandrupatla. new hybrid quadratic/bisection algorithm for finding the zero of nonlinear function without using derivatives. Advances in Engineering Software, 28(3):145-149, 1997.
M. Chianese, A. Coogan, P. Hofma, S. Otten, and C. Weniger. Differentiable strong lensing: Uniting gravity and neural nets through differentiable probabilistic programming. Monthly Notices of the Royal Astronomical Society, 496( 3:331-333, 2020.
A. Coogan, K. Karchev, and C. Weniger. Targeted likelihood-free inference of dark matter substructure in strongly-lensed galaxies. arXiv preprint arXiv:2010.07032, 2020.
R. Comish, A. Caterini, G. Deligiannidis, and A. Doucet. Relaxing bijectivity constraints with continuously indexed normalising flows. International Conference on Machine Learning, 2020.
R. Deng, B. Chang, M. A. Brubaker, G. Mori, and A. Lehrmann. Modeling continuous stochastic processes with dynamic normalizing flows. arXiv preprint arXiv:2002. 105 16, 2020.
J. V. Dillon, I. Langmore, D. Tran, E. Brevdo, S. Vasudevan, D. Moore, B. Patton, A. Alemi, M. Hoffman, and R. A. Saurous. Tensorflow distributions. arXiv preprint arXiv:1711.10604, 2017.
C. Durkan, A. Bekasov, I. Murray, and G. Papamakarios. Cubic-spline flows. arXiv preprint 10w::9906.02115, 2019a.
C. Durkan, A. Bekasov, I. Murray, and G. Papamakarios. Neural spline flows. Advances in Neural Information Processing Systems, 2019b.
M. Figurnov, S. Mohamed, and A. Mnih. Implicit reparameterizzation gradients. Advances Neural Information Processing Systems, 2018.
M. Gorinova, D. Moore, and M. Hoffman. Automatic reparameterisation of probabilistic programs. International Conference on Machine Learning, 2020.
W. Grathwohl, R. T. Q. Chen, J. Bettencourt, Sutskever, and D. Duvenaud. Ffjord: Free-form continuous dynamics for scalable reversible generative models. International Conference Learning Representations, 2019.
M. Hoffman and D. Blei. Stochastic structured variational inference. International Conference on Artificial Intelligence and Statistics, 2015.
G. Kanwar, M. S. Albergo, D. Boyda, K. Cranmer, D. C. Hackett, S. Racaniere, D. Rezende, and P. E. Shanahan. Equivariant flow-based sampling for lattice gauge theory. Physical Review Letters, 121(12) 2020.
10

---

Published as a conference paper at ICLR 2022
D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. International Conference Learning Representations, 2014.
D. P. Kingma and P. Dhariwal. Glow: Generative flow with invertible Ixl convolutions. Advances in Neural Information Processing Systems, 2018
D. P. Kingma and M. Welling. Auto-encoding variational bayes. International Conference on Learning Representations, 2013.
D. P. Kingma, T. Salimans, Jozefowicz, Chen, Sutskever, and M. Welling. Improved variational inference with inverse autoregressive flow. Advances in Neural Information Processing Systems, 2016.
I. Kobyzev, S. Prince, and M. Brubaker. Normalizing flows: An introduction and review of current methods. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020.
A. Kucukelbir, D. Tran, Ranganath, A. Gelman, and M. Blei. Automatic differentiation variational inference. The Journal of Machine Learning Research, 18:430-474, 2017.
S. Li, C. Dong, L.. Zhang, and L. Wang. Neural canonical transformation with symplectic flows. Physical Review X, 10(2):021020, 2020.
K. A. Nicoli, C. J. Anders, L. Funcke, T. Hartung, K. Jansen, P. Kessel, S. Nakajima, and P. Stornati. Estimation of thermodynamic observables in lattice field theories with deep generative models. Physical Review Letters, 126(3)033001,, 2021.
G. Papamakarios, T. Pavlakou, and I. Murray, Masked autoregressive flow for density estimation. Advances in Neural Information Processing Systems, 2017.
G. Papamakarios, E. Nalisnick, D. J. Rezende, S. Mohamed, and B. Lakshminarayanan. Normalizing flows for probabilistic modeling and inference. The Journal of Machine Learning Research, 22: 1-60, 2021.
D. Piponi, D. Moore, and J. V. Dillon. Joint distributions for tensorflow probability. arXiv preprint arXiv:2001.11819, 2020.
K. Rasul, A. Sheikh, I. Schuster, U. Bergmann, and R. Vollgraf. Multivariate probabilistic time series forecasting via conditioned normalizing flows. arXiv preprint arXiv:2002.06103. 2020.
D. Rezende and S. Mohamed. Variational inference with normalizing flows. International Conference on Machine Learning. 2015.
D. J. Rezende, S. Racanière, I. Higgins, and P. Toth. Equivariant hamiltonian flows. arXiv preprint 1rX9v:1909.133 13739, 2019,
Murray Rosenblatt. Remarks on multivariate transformation. The Annals of Mathematical Statistics, 23(3):470-477) 1952.
John Salvatier, Thomas V Wiecki, and Christopher Fonnesbeck. Probabilistic programming python using pymc3. PeerJ Computer Science, 2:e55, 2016.
V. Garcia Satorras, Emiel H., Fabian B. F., I. Posner, and M. Welling. E(n) equivariant normalizing flows for molecule generation in 3d. arXiv preprint arXiv:2105.09016, 2021.
P. P.O. J. Scherer. Computational Physics. Springer, 2010.
R. K. Srivastava, K. Greff, and Schmidhuber. Highway networks. arXiv preprint arX1v:1505.00387. 2015.
D. Tran, A. Kucukelbir, A. B. Dieng, M. Rudolph, D. Liang, and D. M. Blei. Edward: A library for probabilistic modeling, inference, and criticism. arXiv preprint arXiv:6610.09787 2016.
D. Tran, M. D. Hoffman, R. A. Saurous, E. Brevdo, K. Murphy, and D. M Blei. Deep probabilistic programming. International Conference on Learning Representations, 2017.
11

---

Published as a conference paper at ICLR 2022
Jan Willem van de Meent, Brooks Paige, Hongseok Yang. and Frank Wood. An introduction to probabilistic programming. arXiv preprint 1xkiv:8809.105560 2018.
S. Varma, M. Fairbairn, and J. Figueroa. Dark matter subhalos, strong lensing and machine learning. arXiv preprint arXiv:2005.0535: 2020.
A. Wehenkel and G. Louppe. Graphical normalizing flows. International Conference on Artificial Intelligence and Statistics, 2021.
C. Weilbach, B. Beronov, F. Wood, and W. Harvey. Structured conditional continuous normalizing flows for efficient amortized inference in graphical models. International Conference on Artificial Intelligence and Statistics, 2020.
J. Georg Zilly, Rupesh K. S., J. Koutmik, and J. Schmidhuber. Recurrent highway networks. Interna- tional Conference on Machine Learning, 2017.
12

---

Published as a conference paper at ICLR 2022
A APPENDIX
A.1 DATASETS
Here we provide a detailed description of the datasets used in the experiments.
A.1.1 2D TOY PROBLEMS
The implementation of the 2D toy datasets is taken from the publicly available implementation of FFJORD (Grathwohl al. 2019).
A.1.2
MNIST
We use as training set 50000 data points, while the remaining 10000 are used as validation set. The images are dequantized following the same procedure as in (Papamakarios al. 2017).
A.1.3 STOCHASTIC DIFFERENTIAL EQUATIONS
For the generative time series experiments we generate data using four stochastic differential equations (SDEs) models, discretized with the Euler- Maruyaaa method. The SDEs are: Geometric Brownian motion, ConstennUUhlenbed process, Lorenz system and Van der Pol oscillator. the following, W, corresponds to Wiener process, and the series length 7 = 30 with time step unless specified.
Geometric Brownian motion: the dynamics evolve as:
i = pr,dtt + aw,dW,
where μ and a are constants. In practice, we generate sequences as simple Brownian motion, and then take the exponential of the whole sequence. The Brownian motion sequence is generated as:
Zo (H, Iy ~ NN Vt E [ 1]
where the mean and the standard deviation a = 0.1. Ornstein-Uhlenbeck process: This process has the following dynamics:
ft -0xdd + adW,
with 0 0 and 0. The sequences are generated as:
30 N σo)
N
with 0, 00 = 5. = 0.5 and 0 = 0.8.
Lorenz system: the Lorenz dynamics evolve follows:
a ΦVy .zt)
8 = Z₁ (p 21)
where Φ = 10. p0 28 and 3
8 In the discrete case we have:
3.
Fo. yo. 2u 10)
Vt E [1....,T - -
N(xx-1 + -(0(y1-1--1-11) 2T-1)), V)*
31 - s(Ir-1(p 2--1) 1)-1),, v5+0)
2 M( 8(11-181-11 32- v
with standard deviation 0.1 and step size 0.02.
13

---

Published as a conference paper at ICLR 2022
Van der Pol oscillator: The system evolves
I = y + W
ý - p(l x² by W
In the discrete case, with T 120 and 0.05, we have:
Fo: Mi (0, 1)
vt E ,.,..., ,T' -1]
( + Vt-1 v
yt ((h-11 + s4(1 - (1-1)
Here f and σ 0.1
A.1.4 SMOOTHING AND BRIDGE DATA
The data for the variational inference experiments is generated with either Brownian motion, Lorenz system or the Van der Pol oscillator. The first follows the same dynamics of the geometric Brownia motion described above, but without taking the exponential, while the others are the same as the Lorenz system and Van der Pol oscillator described above. The true time series remains unobserved, and we only have access to noisy emissions. In the regression case, the emissions follow Gaussian distribution:
N(r,, o.))
where de is the emission noise, and is 0.15 for the Brownian motion, d, 1 for the Lorenz system, and oe = 0.5 for the Van der Pol oscillator. the classification case, the emissions follow a Bernoulli distribution:
Bernoulli(kxa))
with A gain parameter, the Van der Pol oscillator. In the smoothing problem, we observe emission for all the time steps, while in the bridge problem we observe emissions only in the first and last 10 time points (first and last 40 for the Van der Pol oscillator). For the Lorenz system amd Van der Pol oscillator, the emissions are observed only for the variables in both smoothing and bridge.
in the Brownian Motion, A: = in the Lorenz system and k: 1 for
A.1.5 HIERARCHICAL MODELS
IRIS dataset: We use the IRIS dataset from the UCI Machine learning Repository. The dataset is composed by three classes of iris plant, each described by four attributes: sepal length, sepal width, petal length and petal width. There are 50 datapoints for each class. In the experiments, we use samples of n 10 datapoints.
Digits dataset: We use the Digits dataset from the UCI Machine learning Repository. The dataset composed by 10 classes 8 x 8 images of handwritten digits, from 0 to In the experiments, use samples of n 20 datapoints. The images are dequantized following the same procedure as in
(Papamakarios al. 2017).
Eight Schools: The Eight Schools (ES) model considers the effectiveness of coaching programs on standardized test score conducted in parallel eight schools. It is specified follows:
# ~ NO 100)
log r ~ (5, 0 ~(m,77
where μ represents the prior average treatment effect, 7 controls how much variance there is between schools, = 1, 1,...,& is the school index, and y/ and a; observed.
14

---

Published as a conference paper at ICLR 2022
Gaussian Binary tree: The Gaussian Binary tree is reverse tree of D layers, in which the variables at a given layer d, 2z., are sampled from a Gaussian distribution where the mean is function two parent variables md-1 d-1 the previous layer: "I1 "3,2
zy (link((m)) 1.2 ",,
All the variables at the O-th layers are sampled from Gaussian with mean 0, and variance 1. The last node of the tree is observed, and the inference problem consists in computing the posterior of the nodes at the previous layers. We use a linear coupling function link(x, sr y) = x and a nonlinear one link(x, tanh(x) tan) with trees of depth D 4 and D = 8.
A.2 STRUCTURES INDUCED WITH EMF
In this section we describe in detail the structures embedded with EMF and the bijective transforma- tions they correspond to. In the following, we denote the output vector from the normalizing flow, and & the vector after the transformation induced by EMF. When the probabílistic program embedded by EMF has only Gaussian distributions N ((4, a²), the bijective transformation is in the form:
f(x) = 1t Ox
(x) (x p
while the gated case, with À (0,1) 1) the transformation becomes:
gx,, A) = \f + xx\o= - 1) 1)
x Aμ
g (x, A) 1 X(o 1))
In both time series and Gaussian hierarchical generative models, the embedded structures have only Gaussian distributions, so all the gated and non-gated bijective transformation will have the form described above. What changes in the structure is the way we define the means and variances of such structures.
Continuity:
for the first variable of the sequence, we have μ 0, a dss while for the remaining is trainable variable
1], we have - and transformed with the softplus function, shared among all the latent variables.
T-1 1 variables AA E .l....,7 T ,
Smoothness: the first two variables of the sequence have μ 0 and o = 1. The remaining variables have μ 2z, - and a.. with trainable variable like the case above.
Gaussian hierarchical model: for the first variable of the sequence, corresponding to the mean of the remaining variables, we have μ = 0 and a = 1, while for the remaining variables, we set μ = Zo and 1.
A.3 FLOW MODELS
In all the experiments, we use MAF for generative models and IAF for variational inference, both with autoregressive layers (three for MAF-L) and a standard Gaussian as base density. Each autoregressive layer has two hidden layers with 512 units each. We use ReLU nonlinearity for all the models but for IAF, MAF and MAF-L without EMF or GEMF, for which we empirically found Tanh activation to be more stable. The features permutation always happen between the two autoregressive layers. In the case of EMF-M and GEMF-M, the permutation happens before the structured layer. We found empirically that training the gates for GEMF is slow as the variables lie on an unbounded space before the Sigmoid activation. possible solution scaling the variables (by 100 in our experiments) before applying Sigmoid, which speeds-up the training for the gates, leading to better results.
A.4 TRAINING DETAILS
In this section we describe the training procedure for the experiments. We train all the models with Adam optimizer (Kingma & Ba 2014).
15

---

Published as a conference paper at ICLR 2022
2D toy experiments: the models trained for 500 thousand iterations, with learning rate le-4 and cosine annealing. The mixtures of Gaussians are initialized with means evenly spaced between -4 and for EMF-T and between -10 and 10 for EMF-M, while the standard deviations initialized to 1 For EMF-T and for EMF-M.
MNIST: the models trained with early stopping until there is no improvement in the validation loss for 100 epochs, with learning rate le-4. The mixtures Gaussians are initialized with means evenly spaced between -15 and 15 for EMF-T and between -20 and 20 for EMF-M, while the standard deviations are initialized to 3 For EMF-T and for EMF-M.
Generative Hierarchical:
fit the models for 100 thousand iterations for the IRIS dataset and for 400 thousand iterations for the Digits dataset, with learning rate lc-4 and cosine annealing. The annealing schedule is over 500 thousands iterations.
Generative SDEs: the models are trained for 50 thousand iterations on BR and OU, while for 400 thousand iterations for LZ and VDP, with learning rate le-4 and cosine annealing. The annealing schedule is over 500 thousand iterations.
Variational inference: we fit all the surrogate posteriors for 100000 iterations with full-batch gradient descent, using 50-samples Monte Carlo estimate of the ELBO, and learning rate 1e-3 for all the methods but IAF, EMF-T and GEMF-T, for which we use 5e-5. For GEMF, the gates are initialized to be close to the prior, whit a value of 0.999 after Sigmoid activation.
A.5 NUMBER OF PARAMETERS AND SAMPLE TIME
We report the number of trainable parameters and sample time in seconds for each model used in the generative experiments (tables 4
performed GPU Nvidia Quadro RTX 6000 We do not report parameters and inference time for the Variational Inference experiments, the number of parameter changes only with gates, in which there is one additional parameter per latent variable, and the inference time difference between models with and without EMF layer similar the sampling time in the generative case.
Note that the samples
Table Number of trainable parameters per model the generative time series experiments.
GEMF-T(c)
GEMF-T(s)
NSF-GEMF-T(c)
MAF
MAP-L
B-MAF
NSF
BR/OU
618647
618647
291589
618616
927924
618617
291558
LZ
803209
803209
861651
803176
1204764
803179
861618
VDP
1264698
1264698
2286890
1264576
1896864
1264578
2286768
Table 5; Number of trainable parameters per model for the generative hierarchical experiments.
GEMF-T
GEMF-M
MAF
MAF-L
IRIS
649386
649386
649376
974064
DIGITS
4463636
4463636
4463616
6695424
Table 6: Number of trainable parameters per model for the 2D toy and MNIST experiments.
EMF-T
EMF-M
NSF-EMF-T
NSF-EMF-M
MAF
MAF-L
NSF
8 Caussians/Checkertuun
533088
533088
26130
26130
532488
798732
25530
MNIST
3173120
3173120
7690512
7690512
2937920
4406880
7455312
16

---

Published as a conference paper at ICLR 2022
Table 7: Sample time in seconds for the generative time series experiments. We report mean and standard deviation over ten samples with batch 100.
GEMF-T(e)
GEMF-T()
NSF-GEMP-T(c)
NSF
0.658 0.048
0.757
0.012
0.704 0.053
0.987 + 0.049
0.154 0.010
OU
0.655
0.038
0.807 0.072
0.338
0.040
0.481 0.034
0.664 0.041
0.977 1. 0.033
0.184 1: 0.006
1Z
1.511 = 0.097
1.636 0.077
0.536
0.008
1,283 0.039
1.912 0.083
1. 1.834 + 0.067
0.348 = 0.014
VDP
4.419 + 0.170
4.749 + 0.147
1.063 0.071
3.560 0.118
5.337 + 0.234
5.635 + 0.186
0,378 = 0.011
Table 8: Sample time in seconds for the generative hierarchical experiments. We report mean and standard deviation over ten samples with batch 100.
GEME-T
GEMF-M
MAF
MAF-L
IRIS
0.667 + 0.046
0.741 + 0.037
0.595 + 0.024
0.794 0.016
DIGITS
18.040 0.335
17.921 + 0.266
17.395 0.232
26.211 + 0.314
Table 9: Sample time seconds for the 2D toy and MNIST experiments. report mean and standard deviation ten samples with batch 100. We use batch 10 for MNIST the numeric inversion needs a bigger amount of GPU memory.
EMF-T
EMF-M
NSPEME-T
NSF-EMF-M
MAF
MAF-L
NSF
Gaussians
0.236
0.036
Checkerboard
1.406 0.179
1.379 à: 035
1.516 4: 0.152
1.546 à: 0.178
0.036 4: 0.004
0.049 0.004
0.075 0.006
MNIST
13.809 1.009
13.901 0.844
2.745 0.807
3.791 0.518
10,679 0.258
16:272 + 0.495
0.094 0.009
17

---

Published as a conference paper at ICLR 2022
A.6 NUMERICAL ROOT FINDING METHODS
There are cases in which closed form inversion for the transformation induced by EMF is not available, such for the Mixture of Gaussian case. In those cases, we can still compute the inverse function and log-lacobian determinants by using numerical root finding methods. In this work, we use either the secant method or the Chandrupatla's method (Chandrupatla, 1997; Scherer 2010), as the implementation is available in the probabilisticc programming framework Tensorflow Probability,
A.6.1 SECANT METHOD
The secant method is a root- coot-fnding algorithm that uses a succession of roots of secant lines to better approximate root of function. The secant method can be thought of as finie-differrenc approximation of Newton's 's method. The secant method starts with two initial values zo and 2 which should be chosen lie close the root, and then uses the following recurrent relation:
In = In-1 f((n-1)
En In
f(zn 22)
In In-2f ( 1) 2n (In-2)
(1n-1) (2
A.6.2 CHANDRUPATLA'S 'S METHOD
This root-finding algorithm guaranteed to converge if root lies within the given bounds. At each step, it performs either bisection or inverse quadratic interpolation. The specific procedure can be found in (Chandrupatla) 1997 Scherer 2010).
18

---

Published as a conference paper at ICLR 2022
Ground Truth
EMF-T
EMF-M
MAF
NSF
NSF-EMF-T
NSF-EMF-M
MAF-L
Figure 3: Comparison of densities learned by different models the 8 Gaussians dataset.
Ground Truth
EMF-T
EMF-M
MAF
NSF
NSF-EMP-T
NSF-EMF-M
MAF-L
Figure 4: Comparison of densities learned by different models on the Checkerboard dataset.
A.7 ADDITIONAL RESULTS
A.7.1 DENSITY PLOTS FOR 2D TOY EXPERIMENTS
We include density plots for the 2D toy experiments in figures and
19

---

Published as a conference paper at ICLR 2022
HAF
EMF-T
7
7
4
<
4
7
/
6
6a
7
3
3
3
NSF
NSI LMMIT
S
1
9
5
9
5
4
3
7
Figure 5: Sampled MNIST digits for different models.
A.7.2 MNIST SAMPLES
We include samples for models trained on MNIST in figure 5
20

---

Published as a conference paper at ICLR 2022
A.7.3 HIERARCHICAL MODELS
Hierarchical models are central to both Bayesian and frequentist statistics and can be used model data with nested sub-populations. For example, a Gaussian hierarchical model with m populations each with datapoints can be expressed as
m
a-1
p(m, x) = III
Vmmm: 0, o²)
II (Z) j=1
(14)
k=1
where my denotes the empirical population mean. For the leaf variables, this results in the following structured layer: fzk, (Zy4: 1Ty) 0T}, + jk where Tjk is the latent variable of the leaf node zi while 77, is the latent of its root node. This extremely simple layer can used couple variables that are known to be hierarchically related and can be straightforwardly generalized to deeper and more complex bierarchical couplings. Again, the downstream and upstream layers the EMF architecture can model additional couplings and non-Gaussianity while keeping the hierarchical structure. To test the effectiveness of EMF with Gaussian hierarchical structure, we train the flows on hierarchical generative modeling problem where each network outputs a whole dataset subdivided into several classes. Specifically, we use two classification datasets, the IRIS dataset, in which three flower classes are described by four numerical features, and the Digits dataset, composed of 8 8 images of handwritten digits from to 9. We modify the datasets by using vectors ã = [i. di..... d.,), where the d;, i [I...., mi datapoints randomly sampled from the same class, and ju is the feature-wise mean of such datapoints. We then train the flows with vectors datapoint d, can be inferred from the mean and the other - 1 datapoints. We use the same MAF and MAF-L baselines as for the 2d toy problems, and the gated version of EMF-T and EMF-M (GEMF-T, GEMF-M). The results are reported in table I0 The EMF architectures perform better than their MAF equivalents but worse than the much more highly parameterized MAF-L architectures. This is not surprising given the extreme simplicity of the hierarchical layer.
[e, dj ................................ dn-1],
the last
Table 10:
Results in negative log probability on one hundred thousand test samples for the IRIS dataset and over ten thousand test samples for the Digits dataset. We report mean and standard error of the mean over five different runs.
GEMF-T
GEMF-M
MAF
MAF-I
IRIS
-9.165 + 0.046
-8095 + 0.054
-7.213 0.066
-11.446 + 0.3883
DIGITS
1260.849 1.0764
1310.619
0.7419
1307.927 + 0.5284
1181.283 + 0.8151
A.7.4 SMOOTHING STRUCTURE FOR TIME SERIES
We can induce smoothness using layer obtained by discretizing the second-order equation iit) (t) + (t)., which models a physical Brownian motion. We show the results obtained in table
Table 11: Results in negative log probability on one million test samples for the time-series toy problems, and on a test set of datapoints. We report mean and standard error of the mean over five different runs.
GEMF-T(s)
BR
-26.125 0.0015
OU
24.131 + 0.0030
1.7.
-189.647 0.3899
VDP
-518.464 2.5147
A.7.5 VI EXPERIMENTS
As additional variational inference experiments, we show the results on the time series with Gaussian (regression) emissions and shallow version of the Gaussian binary tree model with depth 4. The
21

---

Published as a conference paper at ICLR 2022
results are shown in tables 12 results are reported in table 13
We also compute the forward KL divergence for each model. Such
The EMF bijective transformation can be also used in combination with models which do not normalizing flows. We combine EMF with the mean field and multivariate normal approximations from (Kucukelbir al. 2017), with results in table 14] We also report results of the gated version of EMF.
Table 12: Results in negative ELBO. We report mean and standard error of the mean over ten different runs
GEME-T
MF
MVN
ASVI
IAF
BRS-
-3.341 + 0.977
0.147 +: 1.002
-3.126 +: 0.977
-3.354 +: 0.974
-3.304 +: 0.974
BRB-r
-3.122 + 0.967
1.932 + 0.951
-2.866 +: 0.967
-3.142 + 0.966
-3.086 +: 0.961
46.981 +: 0.478
1492.572 #: 255.358
1485.622
251, 251.296
1257.942 +: 330.467
1837.364 +: 901. 901.002
LZB-r
110.500 +: 40.958
756.338 +: 146.012
748.298 +: 146.063
490.696 +: 166.080
598. 402.4002200 +: 153.412
VDPS-r
93.520 + 2.729
174.445 +: 2.846
194.592 = 2.181
93.696 +: 3.283
117.028 +: 2. 696
VDPB-r
68.030 + 1.315
150.184 # 1.445
171.263 = 1.359
72-101 +: 3.076
92.739 + 1.249
Lin-4
1.513 + 0.576
6.136 0.573
1.542 + 0.571
3.268 + 0.561
1.526 :: 0.581
Tanh-4
0.035 + 0.119
6.640 0.157
0.075 + 0.124
2.866 + 0.113
-0.026 0.118
Table 13: Results in forward KL divergence. We report mean and standard error of the mean over ten different runs.
BRS-r
37.362 1.049
32,333 + 2.051
37.608 + 1.122
37.346 1.0011
37.319 + 1.101
BRS-c
28.539
0.920
-32.910 :: 9.829
28.174 0.958
28.571
0.913
28. 158 0.896
BRB-1
32,103
0.762
11.608 +: 6.311
31,800 0.921
32.149
0.748
32,26 26.266 0,769
BRB-c
26.928
1.940
-76.227 = 18.008
26.582 2.004
26.825
1.946
26.810 1,906
LZS-r
245.458
2.404
-1.e"** =
-7.6e : 195
-3.6e
1.3e
1.0e: 4: 6.4e
LZS-c
245.337
+ 1.577
-1.77
1.96
-1.1e + 1 1.4c
33.
7.9
-1 1.2e + 7.1
LZB-r 148.488
51.783
-1.2 Fi = 2.9e
-6.9 4 1.
-4.2e
2.0
-2.1e
1.1e
LZB-c
-18
#1.5e
33.5 4 16.9e+
-7.7e 4 13.3ee
VDPS-r
577.901
+ 4.153
-1997.811 812 # 121.368
390.120 120 = 28.331
569 293 #: 3,782
-12.726
104.995
569.202
2.83e
959.169
342.340
562.509 7.984
VDPB-r
564.488 d: 3.558
2.66e
04 2.04c + 04
236.209 à: 138.579
582.442 J: 7.922
-3285.368 d: 2764.594
VDPB.c
5.002
05
+
1749.587
575.022
565.238
5.630
+
ES
-13.032 + 0.910
-13.541 + 1.204
-13.687 1.294
-13.533
1.201
-12.876 + 0.937
Lin-4
5.780 + 0.495
126.1966 + 8-401
5.738 0.518
-1.0200 +: 2.165
5.763 + 0.533
Lin8
89.049
2.772
1387.313 + 201 201.479
17.042 3.848
004475
25.415
8K. 506 + 2,030
Tanb-1
17.388 1111
-21.730 13.715
17.404 + 1.081
12. 12.108 3,309
17.316 1,150
Taub-8
311.528 + 3.827
22-11.216 d: 301.753.
255. 9.701
239.432 17.222
30H,161 +: 3.220
22

---

Published as a conference paper at ICLR 2022
Table 14: Results in variational inference for additional models: non gated EMF-T, and the combination of Mean Field and Multivariate Normal with EMF-T and GEMF-T.
EMP-T
MF-EMN MF-EMF-T
MVX-EMF-T
MF-GEMP-T
MVN-GEMF-I
BRS-r
-ELBO
33.293 + 0.974
13.906 + 224
-3.123 0.968
3.087 0.967
-3077 D.966 32.351 0.822
FKL
37.483
6,429 6.068
32,458
0.775
0.779
BRS-c
-E.80
15.804
1.406
18.602 1: 1.723
15.917
15,960 1.399
18.921 103
FKI
28.554
#: 0.907
27.117 + 1.373
28.562 = 0. 0.906
28,500 =: 0.907
28 396
0.938
BRB-r
-EB8O
-3.096 +: 0.971
13.905 224
-3.123 0.968
-3.150 0.967
-3.077 0.966
PKI
66.429
32.458 0.776
32.351 0.822
BRB-c
ELBO
11.878 1. 0.990
13.538 1.251
11.912 0.985
11.940 0.988
11.930 0.988
FKL
27 002 1: 1,885
25. 007 1: 120
20.880 1.014
26.674 2.041
26.845 1,924
L2S-
ELBBO
46.005
0.450
319.063
1255.087
330.588
52,865 0.604
1254.725 320.517 +
L25-c
-E.80
8 20020012 +: 1.031
56 t: 45-445
23.896 =: 1.342
10.844 1.104
23.540 :: 1.881
FKI.
245 214 1,583
176.426 18.073
3.46 04 9.4e
192.223 16.425
3.76 + 04 1.0c
12BB
187.388 87.731
487.722 1698.100
57.100
487.488 166. -5.8e +
FKL
192:460 1: 11.929
1.8e + 04
-5.8 5.88 04 2.8e 2.8e-+t0
-1 + 01 1:
LZB-c
-ELBO
5,800 1: 0,479
39. 506 1: 15.5500 506
17.014 =: 1.592
51.010 1: 18.540
16.732 1.557
FKI.
249.858
1,379
-2455.551 561.6111 2165.963
1.44 04 dt: 9.2e US
114,907 46.043
-4,4e + 9.2e
2.741
2.725
FXL
577.918
4.143
367 880 6.758
578.639 4.191
577.257 1: 4.164
576.078 1: 1.129
VDPS-
-FI B0
68.395 + 2,509
60.348 + 636
69.594 2.640
68.873 2.574
69.600 2.502 568-492
PKI.
569.
1.954
368.876 1.962
367.706
508.800
VDPB-r
EEBBO
66 03.036 1: 1,312
75.147 1.377
67.914 1.387
1.314
69. 19.166 1,289
FKL
564.53 516 3,620
557, 027442225 1: 3.521
562.322 2: 3.857
562.396 4: 3.603
563-441 4: 3.847
VDPB-c
EEB8O
43.314 1: 2.043
43.783 0.0066
40.467 2,083
43.65 43.680 2,070
44. 443 2.222.00 2.065
PKI.
574.330 4.057
574.061 4.254
574.561
073-262-4 4.676
ES
ELBO
36. 1: 0.004
36.794 J: 0,040
38.532 =: 0.016
36.798 0.041
36.512 0.019
FKL
- -13.013 1. 0.906
13.5525 = 1.200
13.666 1: 1,296
-13,582 1: 1.219
133.792 1: 1.311
Lin-4
EEI BLB
1.512 0.573
5,981 0.520
1.513 0.574
3.371 0.650
1,533 0.572
0.518
0.499
Lin-8
-8.80
2 2,600 +: 0.208
109.561 2: 3.608
5.600 = 0.260
26.396 2:: 0.329
4.178 it 0. 195
FKI.
89.005
2,776
22.826 7.270
87.689 3.056
-2.279 + 26.556
88. 143 2,838
Tanh-4
-0.337
6:221
00,025
2.876
120
17.436
6.913
1.069
12.561 3.256
Tank-8
EEBBO
1,866 0.119
19. 26. 1: 1.790
5.338 = 0.124
14.134 1: 0.784
4.257 1: 0.191
FKI
311.594 3.618
295.877 4.500
285.645
7.422
280.296 3: 9.743
303-473 4: 5.529
23

---

Published as a conference paper at ICLR 2022
Figure 6: Surrogate posterior for Lorenz Smoothing.
146-1
Figure 7: Surrogate posterior for Lorenz Bridge.
A.7.6 SURROGATE POSTERIORS
We show and compare the obtained surrogate posteriors using IAF, EMF-T and GEMF-T, together with the ground truth and the observations (figures
24

---

Published as a conference paper at ICLR 2022
LMI-1
CEMF-T
Figure 8: Surrogate posterior for Van der Pol Smoothing.
25

---

Published as a conference paper at ICLR 2022
LMI-1
CEMF-T
Figure 9: Surrogate posterior for Van der Pol Bridge.
26

---

Published as a conference paper at ICLR 2022
Goussians
Checkerboard
43-
15
40-
1.5
33-
30-
2003
2000
3002
1000
3000
3000
2003
3000
RODE
Figure 10: Training losses on 2D toy data. Each plotted value the average loss of 100 iterations.
MNIST
CEMP-T
CEMF-M
1800
NSF EMF-T NSF
MAF
1600
NSF
1000
800
250
500
750
1000
1250
1500
1750
Training losses on MNIST. Some models trained for a smaller amount of epochs as they would overfit quicker on the validation set.
Figure 11:
A.7.7 TRAINING LOSSES
We show a comparison the training losses for the different models, for the 2D data (figure I0
MMIST (figure 11 the IRIS and digits datsests (figure 13) and for the time series data (figure 12
27

---

Published as a conference paper at ICLR 2022
Brownian mokin
Enstte--Uhembbee process
10
GEVI-TIC
70-
-10
-15-
24-
320
202
300
400
500
500
200
300
400
500
Lorenz
Van der oscillator
MAL
MAF-
M41-4
VEF
8 M61
-30
-33
-250
Figure 12: Training losses on time series toy data. Each plotted value the average loss of 100 iterations.
Digits
350
NAT-
75
60-
25-
Figure 13: Training losses on Iris (left) and Digits (right) datasets. Each plotted value is the average loss of 100 iterations.
28

---

Published as a conference paper at ICLR 2022
Dnessendo CMPTH
Eomba neoo MP
Figure 14: Samples for Brownian motion.
price
Dradethatee COPETIO
Deensheeee 2911
Deable Jirbed
Figure 15: Samples for Omstein-Uhlenbeck process.
A.7.8 SAMPLES
provide comparison between training data and the samples from the trained models. Figures 14 15 and show samples for the Brownian motion, Castein-UUllebbed process and Lorenz system respectively.
29

---

Published as a conference paper at ICLR 2022
olaan 100
Lana
122
Figure 16: Samples for Lorenz system.
Figure 17: Samples for Van der Pol oscillator.
30


Reviews:
[{"review_id": "yv8mVFSTWEB", "reviewer": "Reviewer_ihM2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I do not count this as a weakness of the paper (but it should be changed): It seems that the paper is using the ICLR 2021 (last year) format because the page headers say “Under review as a conference paper at ICLR 2021”.\n### Strengths\n* Incorporating inductive biases into flow architectures is a very interesting and important problem to deal with.\n* The method presented in section 3.1 of converting a distribution into a uniform distribution and the uniform distribution into another one sounds very interesting and applicable in other settings.\n* The presented model outperforms the baselines in the presented experiments.\n\n### Weaknesses\n* The paper claims outperforming the state-of-the-art. I do not think that this conclusion can be drawn from the presented experiments:\n* The paper compares the suggested architecture with two flow models, Inverse Autoregressive Flows (IAFs) and Masked Autoregressive Flows (MAFs), which were published in 2016 and 2017, respectively. Both baseline models are dated and do not constitute the current state of the art. In a more recent survey on normalizing flows ([1], p. 13), MAFs did not achieve state-of-the-art performance on any of the covered datasets.\n* The datasets that the paper uses for evaluation are not common for assessing the quality of normalizing flow models. None of the datasets is used in [1]. In particular, neither the IAF paper nor the MAF paper use these datasets for evaluation. I recommend evaluating the presented model on some (simple) image datasets, like MNIST and CIFAR10, which are very commonly used in the normalizing flow literature.\n* The conclusion states “We showed how, by choosing appropriate inductive biases, EMF can improve over generic normalizing flows on a range of different domains, with only a negligible increase in complexity …”\nThis conclusion should only be drawn if it can be supported by some concrete numbers. To the best of my knowledge, the paper does not mention anywhere how the complexity of the proposed EMF compares to the complexity of the baseline model. Indeed, presenting training and sampling times of the proposed and the baselines models would strengthen the contribution.\nClarity\n* I have really tried to understand the methodology in the main section 3.2, but the section is still not clear to me. I understand the univariate case in section 3.1, but I cannot make sense of the main section 3.2.\nIn Figure 2, I do not understand what expressions like model_generator, gen.send and d.as_bijector mean. Moreover, the function forward_and_log_det_jacobian seems to be called within itself, but with only one argument instead of the 3 used in the definition. (The same applies to inverse_and_log_det_jacobian.)\nWhat is the input and the output of the structured layers? What do the epsilon variables in e.g. equation (5) mean? Shouldn’t the epsilons only come from the flow latent space and not be present in intermediate layers? \nA toy example where each step of the proposed layers is explained for some easy problem would be really helpful. This could either be put into the appendix or replace Figure 1, which I did not find very illuminating.\n\n**Minor and typos**\n1. x_0 is not present on the left hand side of equation (2), so it should not be on the right hand side either.\n2. On p. 3, in the sentence “The probability integral transform theorem states …”: The upper limit of the integral should be x and not y.\n3. p. 2: “We call these architectures embedded-model flow s (EMF).” -> *flows*\n\n[1] Normalizing Flows: An Introduction and Review of Current Methods, Kobyzev, Prince, Brubaker.\n", "summary_of_the_paper": "This paper proposes a new type of normalizing flow for incorporating inductive biases into the model architecture. To this end, the authors introduce a so-called “structure layer”, aiming to transform a spherical Gaussian variable into a pre-defined probabilistic program. A modified version, called “gated structured layer” is proposed in order to skip problematic parts of the model. The model is evaluated on a variety of different datasets and outperforms the masked autoregressive flow baseline.", "summary_of_the_review": "Incorporating inductive biases into normalizing flow architectures is a very relevant and important problem. Some of the techniques presented in the paper seem interesting, but I really struggled to understand the main methodology part in the way it was presented. The paper claims to outperform the state-of-the-art normalizing flows. For the reasons stated above, I do not agree with this. The paper would benefit from further experiments on more common datasets and from a comparison with more recent state-of-the-art flow models.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "n5EtbXrg4Zy", "reviewer": "Reviewer_2pw8", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Novelty\nAlthough it is well-known that a probabilistic program (with deterministic control flow and continuous random variables) is reparameterisable with samples from a unit Gaussian, I believe that using this insight to construct a normalising flow layer is a novel and interesting idea. The \"gated\" layers introduced add further novelty (while being demonstrably useful). I am therefore confident that this paper makes sufficiently novel contributions.\n\n### Strengths\n- Incorporating the known structure/properties of complex distributions into usually black-box normalising flow architectures is a well-studied problem, which I suspect is important for many real-world applications. This paper presents a general solution applicable whenever a probabilistic model for the distribution can be written, even in the common case where it is only approximately correct (or e.g. only the prior is known but we are modelling a posterior).\n- Experiments demonstrate the method on various distribution-modelling tasks, as well as its use for variational inference. Results are generally positive for the proposed method.\n- The paper is mostly well-written.\n\n### Weaknesses\n- The experiments are all fairly small-scale with in the order of 10s of random variables. It would be interesting to see larger scale applications but I don't think this is necessary for publication.\n- The method has some limitations (correct me if I'm wrong): (1) All variables must be continuous. (2) The probabilistic program must sample the same random variables on every execution. (3) The probabilistic program must be executed whenever the normalising flow is run (which could be a problem if the probabilistic program is slow to execute). There are not critical issues (and are shared by much related work), but are probably worth mentioning in the discussion.\n\n### Minor\n- In Section 5.2, it is misleading to say that \"GEMF-T outperforms the baselines on all the problems\", since the baselines obtain the best performance on two of the results in Table 5 in the appendix.\n- Confusing equation: my understanding from Section 3.2 is that you set $x_{t+1} := f_{t+1, \\sigma}(\\epsilon_{t+1}; x_t)$. Why do you replace $\\epsilon$ with $x_{t+1}$ in $f_{t+1, \\sigma}(x_{t+1}; x_t) = x_t + \\sigma x_{t+1}$ on the line below equation 13?", "summary_of_the_paper": "The authors describe how to translate a probabilistic program into a normalising flow layer which maps samples from a unit Gaussian into samples from the probabilistic program. They propose a \"gating\" mechanism for this layer so that it can learn to interpolate between this transformation and the identity transform. They then show how this layer can be combined with generic normalizing flow layers, and in this way add an inductive bias to a normalising flow without sacrificing its expressivity. In the experiments, they provide various examples using probabilistic programs in this way to construct normalising flows with helpful inductive biases.", "summary_of_the_review": "I'm recommending acceptance because this paper makes a novel, yet simple and generic, way to impose inductive biases on normalising flows. It is validated by experiments in various domains.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "PNBVbFDbOvX", "reviewer": "Reviewer_zyZ8", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths:**\n- Proposes a generic method to compile probabilistic programs into invertible functions given an ordering of the variables from the probabilistic program.\n\n- Introduces a gating layer that allows the model to select whether to use the \"prior\" or not.  This is essentially a neural architecture search idea where the model is free to choose the prior or not.\n\n- Shows better performance on structured VAE problems (particularly for the Lorenz system).\n\n- Paper will provide code to actually do the compilation automatically from a probabilistic program. This could be generally useful to the community.\n\n**Weaknesses:**\n- There seems to be a disconnect between the probabilistic program (i.e., the domain knowledge) and flows.  As soon as the flow parts (i.e., MAF) modify the input to the EMF layer, the EMF layer no longer represents the probabilistic program, but rather than some distorted version of the probabilistic program. The gated part allows the model to completely ignore the prior if better. Thus, a key question remains: Why not just use a more state-of-the-art flow which can handle more arbitrary distributions? Why go through the trouble of carefully specifying a probabilistic model?\n\n- Limited novelty in compilation approach: The idea of using conditional CDF functions is generally known as the \"Rosenblatt transform\".  Thus the construction of an invertible transformation to or from a Gaussian distribution via conditional CDF functions is not particularly novel.\n[1] M. Rosenblatt, \"Remarks on a multivariate transformation,\" Annals of Mathematical Statistics, vol. 23, no.3, pp. 470-472, 1952.\n\n- The empirical results seem relatively weak (especially because the baselines seem weak).  It is unclear that the added work of incorporating domain knowledge significantly improves performance compared to just using more state-of-the-art flows. More details below.\n\n    - Why not just use a more state-of-the-art normalizing flow (Neural Spline Flow, Residual Flow, Flow++, etc.)?  It is unclear what is the real advantage of EMF layers.  It seems that they add some modeling power but this extra modeling power could be added in different ways.\n\n    - The multimodal part can be easily captured by Neural Spline Flows or more recent flow models than MAF as they have complex non-linear transformations that can model multi-modality.  Thus, it is unclear that the \"multimodality\" introduced by the EMF is caused by the \"prior knowledge\" about multimodality or merely because MAF is a very limited model and EMF is more complex. This would be significantly more convincing if this \"prior knowledge\" could be incorporated into models that already can model multimodality.  Overall, the experimental setup seems unfair as EMF introduces complex non-linearities that are unavailable to MAF (even with more parameters).\n\n    - In the hierarchical experiment, again, it seems that even though the experimental setup favors hierarchical distributions, the MAF with more parameters does better.  Again, it's not clear the added benefit of EMF layers since they are not directly comparable to MAF (which has natural limitations).\n\n    - In all these experiments, it seems that EMF should be the clear winner as the EMF incorporates domain knowledge that the baselines do not (i.e., it should have an unfair advantage because it uses external domain knowledge).  Yet, except for the Lorenz experiments, other simple baselines seem quite close.\n\n- Limitation: The autoregressive transformation can be quite slow to compute in one direction.  This is why coupling layers have become quite useful in comparison to autoregressive invertible transformations.  The construction seems to inherit this computational issue with an autoregressive structure---i.e., if you want to use structured layers, you will be limited computationally in one direction.\n\n**Other comments or questions**\n- The gated architecture is useful for model power as the model can choose not to use the prior specification (i.e., the \"bad\" model).  However, this seems to be in partial opposition to the idea of incorporating prior knowledge---i.e., the goal of incorporating prior knowledge is to improve performance by biasing the model towards more appropriate solutions.  Thus, can these domain-specific or prior-specific models just be considered \"hints\" to help the algorithm find the best fitting model?\n\n- Could the interpretability of the model be better because of these structured layers?  This could be an added benefit but it is unclear if that is true if the input and output of the structured layer can be arbitrarily manipulated by unstructured flow layers.\n\n- Why was the dataset augmented in the hierarchical experiment?  Shouldn't the hierarchical EMF layer be able to learn the hierarchy automatically from the data?\n\n- The Lorenz system seems to show the most improvement.  The BR and OU time series only show a marginal improvement of negative log likelihood.  Why is that?\n\n- Regarding time series experiment, while the base distribution of B-MAF follows the continuity equation, why not also make the masked flow layers obey this ordering of the variables (as I believe MAF generally assumes a random order)? This would be a simple and straightforward way to incorporate the ordering of variables in the flow part of MAF.\n", "summary_of_the_paper": "This paper first proposes to take a probabilistic program and compile it into a sequence of invertible transformations. Applying this invertible transformation to a standard normal distribution produces a sample from the probabilistic program.\nThis \"compiled\" invertible transformation is then used as a layer with learnable parameters, called a structured layer, in a larger flow model which can also have unstructured layers.\nThe paper argues that this enables the ability to incorporate domain knowledge via these structured layers while enabling deviation from this domain knowledge via the unstructured flow layers.\nThe paper performs experiments on multimodal, hierarchical, time series and variational inference data.\n", "summary_of_the_review": "**Update since author response**:\nI thank the authors for their very thoughtful response.  I greatly appreciated the new experiments and more discussion on why the structured layers are simple and have fixed parameters for essentially simple inductive biases.  Also, I appreciate the new NSF experiments.  Overall, I felt these responses answered most of my major concerns.   I have updated my score accordingly.  I would encourage the authors to incorporate some of this discussion in the paper if accepted.\n\n----\nOverall, I think the idea of the paper seems reasonable (i.e., to compile known domain structures into flows). However, the compilation technique using conditional CDFs is relatively well-known as the Rosenblatt transform, and it is unclear why a practitioner would not just want to use a more state-of-the-art flow instead of spending time to construct a probabilistic model.  The empirical results are unconvincing that a SOTA flow (even with similar number parameters) wouldn't outperform the current approach.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "0EwztI1MmTK", "reviewer": "Reviewer_g1yH", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The strengths and weaknesses are pretty much discussed in the paper. Specifically,\n\nThe advantage of the method lies in the ability to control the flow model by injecting inductive bias. The whole model can be considered as a mixture of freely trainable MAF layers and fixed probabilistic programs, connected by the gated layers. The idea is novel and inspiring to me.  The whole idea looks natural, it could be considered as an augmentation of existing flow models that work well for small-sized data.\n\nSeveral weaknesses that I've noticed:\n1. The paper overall is easy to follow, however, I find the algorithm detail hard to read. Especially Figure 2, I encountered some difficulties in understanding the algorithm details as python. For instance, how as_bijector(gated=True) is implemented?\n2. There are some typos in Section 3.1, when $C_{\\phi}$ is introduced, it should be $\\int_a^x$ right? and $a$ is undefined.\n3. The gated layer requires root finding as mentioned, so exactly how much overhead is introduced? does this algorithm scalable to high dimensional data?\n4. I noticed other methods such as Glow (Kingma et al.) not mentioned here, could authors explain why?", "summary_of_the_paper": "This paper tries to bridge the concept of the normalizing flow model with the structured layers that encode the domain knowledge (also called inductive bias here). The general idea is to first notice that many existing probablistic programs (such as univariable RV) can be converted to flow under some special $f_{\\phi}$.  Furthermore, the authors proposed a gated layer to allow the model to switch between user-specified model and the learned MAF. Extensive experiments in toy multimodality distributions, hierarchical gaussian, timeseries models and variational inferences are conducted to compare the proposed method with other baselines.\n\n", "summary_of_the_review": "Overall, I think this paper advances the research in this area, although more careful ablations are required to ensure the empirical results are indeed promising. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a method for incorporating inductive biases into the model architecture of normalizing flows through a suitable probabilistic program. All reviewers agree the paper makes an interesting contribution to the growing normalizing flow literature. The paper is well written and the idea is novel. Additionally, the experimental results are promising and the additional experiments and baselines added during the rebuttal further strengthen the paper. I recommend acceptance.
Decision: Accept (Poster)