Localised Generative Flows
Authors: Rob Cornish
Anthony Caterini
George Deligiannidis
Arnaud Doucet
Venue:  (2020)
Keywords: Deep generative models
normalizing flows
variational inference

Abstract: We argue that flow-based density models based on continuous bijections are limited in their ability to learn target distributions with complicated topologies, and propose localised generative flows (LGFs) to address this problem. LGFs are composed of stacked continuous mixtures of bijections, which enables each bijection to learn a local region of the target rather than its entirety. Our method is a generalisation of existing flow-based methods, which can be used without modification as the basis for an LGF model. Unlike normalising flows, LGFs do not permit exact computation of log likelihoods, but we propose a simple variational scheme that performs well in practice. We show empirically that LGFs yield improved performance across a variety of common density estimation tasks.

Under review as a conference paper at ICLR 2020
LOCALISED GENERATIVE FLOWs
Anonymous authors
Paper under double-blind review
ABSTRACT
We argue that flow-based density models based on continuous bijections are lim- ited in their ability to learn target distributions with complicated topologies, and propose localised generative flows (LGFs) to address this problem. LGFss are posed of stacked continuous mixtures of bijections, which enables each bijection learn local region of the target rather than its entirety. Our method a gener- alisation of existing flow-based methods, which can be used without modification as the basis for an LGF model. Unlike normalising flows, LGFs do not permit exact computation of log likelihoods, but we propose a simple variational scheme that performs well in practice. We show empirically that LGFs yield improved performance across variety of density estimation tasks.
INTRODUCTION
Flow-based generative models, often referred to as normalising flows, have become popular methods for density estimation because of their flexibility, expressiveness, and tractable likelihoods. Given the problem of learning an unknown target density p3 on a data space X, normalising flows model px as the marginal of X obtained by the generative process
7 ~ pz, X (Z),
where pz is prior density on a space 2, and X 2 is a bijection Assuming sufficient regularity, it follows that X has density p. (z) = (z(((z)) det Dg(z)) (see e.g. Billingsley (2008)). The parameters of can be learned via maximum likelihood given i.i.d. samples from pX.
To be effective, normalising flow model must specify an expressive family of bijections with tractable Jacobians. Affine coupling layers (Dinh et al., 2016). autoregressive transformations (Germain et al. 2015; Papamakarios et al. (017). ODE-based transformations (Grathwohl et al.) 2018), and invertible ResNet blocks (Behrmann et al. 2019) are all examples of such bijections that can be composed to produce complicated flows. These models have demonstrated significant promise in their ability to model complex datasets (Papamakarios et al., 2017) and to synthesise novel data points (Kingma & Dhariwal 2018).
However, in all these cases, g is continuous in z. We believe this is a significant limitation of these models since it imposes a global constraint on g which must learn to match the topology of Z, which is usually quite simple, the topology of X' which expect be very complicated. We gue that this constraint makes maximum likelihood estimation extremely difficult in general, leading training instabilities and erroneous regions of high likelihood in the learned density landscape.
To address this problem we introduce localised generative flows (L.GFs), which generalise equation T by replacing the single bijection g with stacked continuous mixtures of bijections (G(-: w)Jaeu for an index set U. Intuitively, LGFs allow each G(:;u) to focus on modelling only a local component of the target that may have much simpler topology than the full density. LGFss do not stipulate the form of G, and indeed any standard choice of can be used as the basis of its definition. We pay a price for these benefits in that we can no longer compute the likelihood of our model exactly and must instead resort to a variational approximation, with our training objective replaced by the evidence lower bound (ELBO). However, in practice we find this not a significant limitation, as the bijective structure of LGFs permits learning high-quality variational distribution suitable for large-scale training. We show empirically that LGFs outperform their counterpart normalising flows across a variety of density estimation tasks.
We assume throughout that X, Z C RRd, and that all densities with respect the Lebesgue measure.

---

Under review as a conference paper at ICLR 2020
0.48
0.88
1.0
1.0
0.315
0.315
0.20
0.70
0.0
0.0
0.18
0.186
-0.5
-0.5
0.00
0.30
0.000
0.000
-2
Figure 1: Density models learned after 300 epochs by standard ten-layer MAF (left) and by five- layer LGF-MAF (right). Both models have roughly 80,000 parameters and use standard Gaussian prior. Samples from the target distribution shown in black. Details of the experimental setup given in Appendix D.2
LIMITATIONS OF NORMALISING FLOWS
Consider normalising flow model defined by family of bijections yo parameterised by 0. Suppose we are in the typical case that each Yo is continuous in æ. Intuitively, this seems to pose a problem when p'x and pz supported on domains with different topologies, since continuous bijections (with continuous inverses) necessarily preserve topology. This is made precise in Proposition below; defer the proof to Appendix A. In practice, suggestive pathologies along these lines readily apparent in simple 2-D experiments as shown in Figure T] In this case, the density model (a masked autoregressive flow (MAF) (Papamakarios al. 2017)) is unable continuously transform the support of the prior (a standard 2-D Gaussian) into the support of p'x which is the union of two disjoint rectangles and hence clearly has a different topology.
(Limitations of Normalising Flows). Suppose each ge homeomorphism and det (z) * 0 for all and If the prior pz has support that is not homeomorphic to Pxx then no choice of 0 will yield a normalising flow model that matches p'x exactly.
Proposition
In other words, once we pz, we immediately restrict the set of target distributions that we may possibly express using our model. Provided each de is continuous, this occurs no matter how com- plicated we make our parameterisation it holds even we allow all continuous bijections. We believe this fact significantly limits the ability of normalising flows to learn any target distribution whose support has a complicated topology.
Furthermore, note that the standard maximum likelihood objective is asymptotically equivalent to arg min (KL(p
(2)
where (x) Pz (x))) det Dgo (z) DKL denotes the Kullback-Leibler divergence. We conjecture that the problem identified in Proposition I is further exacerbated by this objective. Note that equation 2 infinite unless supp supp but strictly positive unless suppr = supp p'- Thus equation 2 encourages the support of px to approximate that of pi from above as closely as possible, but any overshoot however small carries an immediate infinite penalty. This seems problematic for gradient-based methods of solving equation - especially when, light of Proposition we expect an optimal 0 to be pathological, since go encouraged to approximate some function that is not continuous bijection. In this context, for near-optimal 6, it seems plau- sible that supp p might be very sensitive to perturbations of 9, leading landscape when near the optimum. We conjecture this makes the landscape around the optimal difficult to navigate for gradient-based methods, leading increasing fluctuations and instabilities
highly-vayiing loss
2 As usually defined, the support density p technically refers the on which p is strictly positive. However, our statements are also approximately true when supp is interpreted as the region of p which is not smaller than some threshold. This is relevant in practice since. even if both are highly concentrated on some small region, it is common to assume that r' and pz have full support, in which case the supports of px and pz would he trivially homeomorphie.
2

---

Under review as a conference paper at ICLR 2020
Figure 2: Three subsequent epochs of training for the models shown in Figure The support of the standard MAF (above) fluctuates significantly, while LGF-MAF (below) is stable.
of the KL in equation that degrade the quality of the final go produced. Figure illustrates this in practice: observe that, even after 300 training epochs, the support of the standard MAF is unstable.
A simple way fix these problems would be to to the structure of pX. For instance, taking pz to be mixture model has previously been found to improve the performance of normalising flows in some cases (Papamakarios et al. 2017). However, this approach requires prior knowledge of the topology of supp P' that might be difficult to obtain: e.g. solving the problem with a Gaussian mixture pz requires to know the number of connected components of supp p beforehand, and even then would require that these components are each homeomorphic to hypersphere. Ideally, we would like our model be flexible enough to learn the structure of the target on its own, with minimal explicit design choices required on our part.
more complicated pz that is better matched
An alternative approach would be to try more expressive family Ie the hope that this better conditions the optimisation problem in equation 2 Several works have considered families of ge that are (in principle) universal approximators 2018 Jaini et al. 2019). While have not performed a thorough empirical evaluation of these methods, we suspect that these models can at best mitigate the problems described above, since the assumption of continuity of go in practice holds for universal approximators also. Moreover, the method we propose below can be used in conjunction with any standard flow, so that we expect even better performance when an expressive Oe is combined with approach.
any continuous probability densities (Huang et al.)
Finally, we note that the shortcomings of the KL divergence for generative modelling are described length by Arjovsky et (2017). There the authors suggest instead using the Wasserstein distance to measure the discrepancy between PX and Pz. since under typical assumptions this will yield continuous loss function suitable for gradient- difficult to estimate in high dimensions, and its performance can be sensitive to the choice of ground metric used (Peyré et al. (019).. Our proposal here is to keep the KI. objective in equation [ and instead modify the model so that we are not required to map supp pz onto supp using a single continuous bijection. We describe our method in full now.
training. However, the Wasserstein distance is
3
LOCALISED GENERATIVE FLOWS
3.1 MODEL
The essence our idea is to replace the single g used in equation 2 with an indexed family {G(: w)eeeu such that each G(:; u) is bijection from X' to Z. Intuitively, aim is for each G(::u) u) only to push the prior onto a local region of supp p\x thereby relaxing the constraints posed by standard normalising flows described above. To do so, now define px the marginal density of X obtained via the following generative process:
7 ~ Pz, U ~ ( X (Z;U).
(3)
Here Puiz is an additional term that we must specify. In all experiments take this to be mean field Gaussian, that Pulz Gaussian(u; ((2), (z))2Ta))) for some functions 14, a
3

---

Under review as a conference paper at ICLR 2020
Ui
UL-1
Uz
Ui
UL-1
Uz
Zo
Z
Z
Zo
Z
Zs
X
(a) Generative Model
(b) Inference Model
Figure 3: Multi-layer model schematic with L layers. Given their parents, circular nodes are random, and diamond nodes are deterministic.
2 z-4 u C Rdv, where dh is the dimension of the index 2, and La, is the du, x d,, identity matrix. Other possibilities, such as the Concrete distribution (Maddison et al. 2016). might also be useful.
Informally Pthis yields the joint model Px,U,2 uu (I,, is the Dirac delta. We can marginalise out the dependence on by making the change of variable u()';u), which means dæ deet DG [ u) dæ This yields proper density for (X, U) via
2) pz Puzz ( 8(x- (z; u)), where
(x, u) f px,,,2(0, u, dz = pz (G(;; ; u)) puiz ((()))) |det DG
We then obtain density model px by integrating over u: px(xx) / pz (G u)) Pu (u (u]((() |det DG u)l du.
(4)
In other words, px (I)) is a mixture (in general, infinite) of individual normalising flows G(z;; u), each weighted by PUiz (ul ((z;
We can also stack this architecture by taking pz itself be a density of the form of equation Doing so with L layers of stacking corresponds to the marginal of X = Z obtained via:
Zo PZg
Ul PU, (-)Ze-1),
Ze G (Z)
E fl....,L);
where now each Ge(:u)= : X' + 2 a bijection for all u E U. The stochastic computation graph corresponding to this model is shown in Figure 3al In this case, the same argument yields
ue)) DG, u4)1 (5) where PZo.U1. = Pz. This approach is in keeping with the standard practice of constructing normalising flows the composition of simpler bijections, which can indeed be recovered here by taking each Zu.iz... (-12, (-12,-) to be Dirac. We have found stacking to improve significantly the overall expressiveness of models, and make extensive of it in our experiments below.
Pic,Ui (ze, u1:e) = 1.17--1.--.-111 (Ge(ze; ue), (
3.2 BENEFITS
Heuristically, we believe our model allows each G(:; u) to learn a local region of pxx thereby greatly relaxing the global constraints on existing fow-based models described above. To ensure finite KL, we no longer require the density pz (G(z; u)) |det DG u) to have support covering the entirety of supp pi for any given u; all that matters is that every region of supp px is covered for some u. Our model can thus achieve good performance with each bijection Gi:: u) faithfully mapping onto potentially very small component of px- This seems intuitively more achievable than the previous case, wherein single bijection is required to capture the entire target.
This argument is potentially clearest if is discrete. For example, even if u (u_1, u1} can take only two possible values, it immediately becomes simpler to represent the target shown in Figure (ssing a Gaussian prior: we simply require G (:; u-1) map onto one component, and G (::ua) to map onto the other. In practice. we could casily implement such a G using two separate normalising flows that are trained jointly. The discrete case is also appealing since in principle allows exact evaluation of the integral in equation 4 which becomes a summation. Unfortunately this approach also has significant drawbacks that we discuss at greater length in Appendix B
We therefore instead focus here a continuous In this case, for example, we can recover p'x from Figure by partitioning 2/ into disjoint regions U_, and Ui and having (:; u) map onto
This argument can made precise using disintegrations (Chan Pollard 1997 but since the proof mainly a matter of mcasure-theoretia formalitics we omit it. NNot that DG(r; refers the Jacobian with respect to x only.
4

---

Under review as a conference paper at ICLR 2020
the left component of px for U-1, and the right component for U4. Observe that in this scenario we do not require any given G (-: u) to map onto both components of the target, which is in keeping with our goal of localising the model of pi that is learned by method.
In practice G will invariably be continuous in both its arguments, in which case it will not be possible
to partition U disjointly in this way. Instead we will necessarily obtain some additional intermediate region 4o on which G (:; u) maps part of supp pz outside of supp P'x so that px (z) will be
strictly positive there. However, Proposition 2 below shows (proof in Appendix (A.2) that this is not a problem for LGF's we are able to use Pciz to downweight such a region, which avoids the support mismatch issue of standard normalising flows.
Proposition 2 (Benefit of Localised Generative Flows). Suppose that supp pix is open and G(: continuous for each u. Suppose further that, for each supp P'x. the set
Bg == {u E U 1P((G(r) u))l det (G; > 0}
has positive Lebesgue measure. Then there exists Puiz such that supp suppp*-
3.3 INFERENCE
Even in the single layer case (I. 1). if u is continuous, then the integral in equation 4 is in- tractable. In order to train our model, we therefore resort to variational approximation: we intro- duce an approximate posterior Q1., |X PU.. and consider the evidence lower bound (ELBO) of log px (x):
((x)= == Fo (w1 L|2) [logxx.Us. (z, un.L) log W.s xx (u1.2|1))-
is straightforward to show that f(x) < logpx(z). and that this bound is tight when Wy Ix is the exact posterior P This allows learning an approximation to px by maximising -. jointly in Px,U1. and Wi.z for dataset of n i.i.d. samples Ia ~ PX.
It can be shown (see Appendix A.3)) that the exact posterior factors as Pulz x(unr/=) = IT's Puzize (uelz((uelze) where 2L == x, and 28-1 Ge(ze; ue) for S We thus endow 4 with the same form:
qu (w1.z/>)
II qu, q0,12, (uelze).
The stochastic computation graph for this inference model is shown in Figure 3b In conjunction with equation [5] this allows writing the ELBO recursively as
(G uz)) log 2,_, (uelzz + log |det DGr(zz; ue)l log 12, (wxlzz) for & 1, with the base case (=o) = pzo (zo). Here we recover L(x) = Lr.(21.).
(=(z)) = Puu.iz
Now let / denote all the parameters of both Px.t and qu LX I reparametrised (Kingma & Welling 2013 Rezende et al., 2014) so that he(ee, zt) ~ QU,1Z, (-12g) when 6e ~ 17e for some function feg and density T). where ye does not depend on In all our experiments we give qu, the same mean field form as in equation 3, in which case this holds immediately as described e.g. by Kingma & Welling (2013). We can then obtain unbiased estimates of V VoL(x) straightforwardly using Algorithm 1| which in turn allows minimising our objective via stochastic gradient descent. Note that although this algorithm is specified in terms of single value of zg, it is trivial to obtain an unbiased estimate of Vam Σx= [(x,) for minibatch of points {ay 1 by averaging over the batch index at each layer of recursion.
Suppose each WelZ, can be suitably
Algorithm Recursive calculation of an unbiased estimator of (eLe(2))
function GRADELBO(Ze, 8
if é ===00 0 then
return Vo log Pz, (zt)
else
EE ~ ne Ue + he(E,,2e)
- f- Ge(ze: uz)
A log log (uelze-1) log det DGe(ze; ue)l log w0.z, (uelze) return GRADELBO vooi
5

---

Under review as a conference paper at ICLR 2020
3.3.1 PERFORMANCE
A major reason for the popularity of normalising flows is the tractability of their exact log likeli- hoods. In contrast, the variational scheme described here can produce at best approxination to this value, which we might expect reduces performance of the final density estimator learned. More- over, particularly when the number of layers I. is large, it might seem that the variance of gradient estimators obtained from Algorithm 1 would be impractically high.
However, in practice have not found either of these problems to be significant limitation, as experimental results in [Section 5) show. Empirically we find improved performance over standard flows even when using the ELBO as our training objective. We also find that importance sampling is sufficient for obtaining good, low-variance (if slightly negatively biased) estimates of log Px (æ) (Rezende et al., 2014] Appendix E) at test time, although we do note that the stochasticity here can lead to small artefacts like the occasional white spots visible above in our 2-D experiments.
We similarly do not find that the variance of Algorithm grows intractably when we use a large number of layers L, and in practice we are able to train models having the same depth as popular normalising flows. We conjecture that this occurs because, as the number stacked bijections in our generative process grows, the complexity required of each individual bijection to map pz to px naturally decreases. We therefore have reason to think that, as L becomes large, learning each quz!z, will become easier, so that the variance at each layer will decrease and the overall variance will remain fairly stable.
3.4 CHOICE OF INDEXED BIJETIOON FAMILY
We now consider the choice of G, for which there are many possibilities. In our experiments, which all take 2 = Rdd we focus on the simple case of
G(x; u)) (xp(s(u)) @ g(x) + t(u),
where s, t: U Z are unrestricted mappings. g X' + Z bijection, and @ denotes elementwise multiplication. In this case log | det DG w)l = log det Dy(=)) + [s(]],, where [s(u)]; is the 7th component of s(u). This has the advantage of working out-of-the-box with all pre-existing normalising flow methods for which tractable Jacobian of g is available. We provide alternative suggestions to this construction in Amppendix C
Equation also has appealing similarity with the common practice of applying affine transforma- tions between flow steps for normalisation purposes, which has been found empirically to improve stability, convergence time, and overall performance (Dinh et al., 2016 Papamakarios et al., 2017 Kingma & Dhariwal, 2018). In prior work, and have been simple parameters that are learned either directly as part of the model, updated according running batch statistics. Our approach may be understood as a generalisation of these techniques.
4 RELATED WORK
4.1 DISCRETE MIXTURE METHODS
Several density models exist that involve discrete mixtures of normalising flows. A closely-related approach to ours is RAD (Dinh et al. 2019), which special case of our model when = 1, u is discrete, and X' is partitioned disjointly. In the context of Monte Carlo estimation, Duan (0019)
proposes a similar model to RAD that does not use partitioning. Ziegler & Rush (2019) introduce a normalising flow model for sequences with additional latent variable indicating sequence length. More generally, our method may be considered an addition to the class of deep mixture models [Tang al. 2012) Van den Oord & Schrauwen 2014), with our of continuous mixing vari- ables designed to reduce the computational difficulties that arise when stacking discrete mixtures hierarchically we refer the reader to Appendix for more details on this.
4.2 METHODS COMBINING VARIATIONAL. INFERENCE, AND NORMALISING FLOWS
There is also large class of methods which use normalising flows to improve the inference proce- dure in variational inference (van den Berg et al. 2018: Kingma et al. 2016 Rezende & Mohamed, 2015). although flows not typically present in the generative process here. This approach can be described as using normalising flows to improve variational inference, which contrasts our goal of using variational inference to improve normalising flows.

---

Under review as a conference paper at ICLR 2020
However, there are indeed some methods augmenting normalising flows with variational inference, but in all cases below the variational structure is not stacked to obtain extra expressiveness. Ho et al. (2019) use variational scheme to improve upon the standard dequantisation method for deep gener- ative modelling of images (Theis et 2015); this approach is orthogonal to ours and could indeed be used alongside LGFFs. Grrisenko et al. (2019) also generalise normalising flows using variational methods, but they incorporate the extra latent noise into the model in much more restrictive way. Das al. (2019) only learn a low-dimensional prior over the noise space variationally.
4.3 STACKED VARIATIONAL METIODS
Finally, we contrast our approach with purely variational methods which are not flow-based, but still involve some type of stacking architecture. The main difference between these methods and LGFs is that the bijections we use provide us with a generative model with far more structure, which allows us to build appropriate inference models more easily. Contrast this with, for example, Rezende et al. (2014). in which the layers are independently inferred, or [Sonderyy et al.] (2016), which requires complicated parameter-sharing scheme to reliably perform inference. A single- layer instance of our model also shares some similarity with the uulyy-unsueervieed case in Maaløe et al. 2016), but the generative process there conditions the auxiliary variable on the data.
5 EXPERIMENTS
We evaluated the performance of LGFs several problems of varying difficulty, including synthetic 2-D data, several UCI datasets, and two image datasets. We describe the results of the UCI and image datasets here; Appendix D.2 contains the details of our 2-D experiments.
In each case, we compared a baseline flow to its extension as an LGF model with roughly the same number of parameters. We obtained each LGF by inserting mixing variable Up between every component layer of the baseline flow. For example, we inserted a U/ after each autoregressive layer in MAF (Papamakarios al., 2017). and after each affine coupling layer in RealNVP (Dinh) et al. 2016). In all cases we obtained pu,i 9t; I2,, and ty as described in Appendix D.1 We inserted batch normalisation (Toffe Szegedy) 2015) between flow steps for the baselines as suggested by Dinh et al. (2016), but omit it for the LGFs, since our choice of bijection family is a generalisation of batch normalisation as described in Subsection 3.4
We trained our models to maximise either the log-likelihood (for the baselines) or the ELBO (for the LGFs) using the ADAM optimiser (Kingma & Ba| (014) with default hyperparameters and no weight decay. For the UCI and image experiments we stopped training after 30 epochs of no validation improvement for the UCI experiments, and after 50 epochs for the image experiments. Both validation and test performance were evaluated using the exact log-likelihood for the bascline, and the standard importance sampling estimator of the average log- log-likelihood (Rezende et al.) 2014 Appendix E) for LGFs. For validation used 5 importance samples for the UCI datasets and 10 for the image datasets, while for testing we used 1000 importance samples in all cases. Our code is available at https:::/
com wnw.submiiiiii...mmmmm s on974/1gf
5.1 UCI DATASETS
We tested the performance of LGFs on the POWER, GAS, HEPMASS, and MINIBOONE datasets from the UCI repository (Bache & Lichman| 2013). We preprocessed these datasets identically to Papamakarios
al. (2017), and use the same rraninvaiiatiinnness splits. For all models we used a batch size of 1000 and a learning rate of 10-3 These constitute a factor of 10 increase over the values used by Papamakarios results differ slightly from Papamakarios et al. 2017), which may be as a result of this change.
al. (2017) and were chosen decrease training time. Our baseline
We focused on MAF as our baseline normalising flow because of its improved performance over alternatives such as RealNVP for given MAF model is defined by how many autoregressive layers it uses as well as the sizes of the autoregressive networks at each layer. For an LGF-MAF, we must additionally define the neural networks used in the generative and inference process, for which we use multi-layer perceptrons (MLPs). We considered variety of choices of hyperparameters for MAF and LGF-MAF. Each instance of LGF-MAF had a corresponding MAF configuration, but in order to compensate for the parameters introduced by our additional neural networks, we also considered deeper and wider MAF models. For each LGF-MAF, the dimensionality of uj was roughly a quarter of that of the data. Full hyperparameter details given in Appendix D.3 For three random seeds, we trained models using
density estimation (Papamakarios et al. 2017).
7

---

Under review as a conference paper at ICLR 2020
Table 1: Average plus/minus standard error of the best test-set log-likelihood.
POWER
GAS
HEPMASS
MINIBOONE
MAF
0.19 + 0.02
9.23 + 0.07
-18.33 + 0.10
- 100.98 + 0.03
LGF-MAF
0.48 + 0.01
12.02 0.10
-16.63 0.09
-9.93 + 0.04
every hyperparameter configuration, and then chose the best- best-perforiing model across all parameter configurations using validation performance. Taalle shows the resulting test-set log-likel likkliiihoods averaged across the different seeds. It is clear that LGF-MAFs yield improved results in this case.
5.2
IMAGE DATASETS
We also considered LGPs applied to the Fashion-MNIST (Xiao et al., 2017) and CIFAR-10 (Krizhevsky al. 2009) datasets. In both cases we applied the dequantisation scheme of (Theis et al. (2015) beforehand, and trained all models with a learning rate of 10 and batch size of 100.
We took our baseline to be a RealNVP with the same architecture used by Dinh et al. (2016) for their CIFAR-10 experiments. In particular, used 10 affine coupling layers with the corresponding alternating channelwise and checkerboard masks. Each coupling layer used a ResNet (He et al,, 2016a b) consisting of 8 residual blocks of 64 channels (denoted 8 x 64 for brevity). We also replicated their multi-scale architecture, squeezing the channel dimension after the first coupling layers, and splitting off half the dimensions after the first 6. This model had 5.94M parameters for Fashion-MNIST and 6.01M parameters for CIFAR-10. For completeness, we also consider a RealNVP model with coupler networks of size 4 64 match those used below in LGF-RealN RealNVP. This model had 2.99M and 3.05M parameters for Meshion-MNIST" and CIFAR- 10 respectively.
For the LGF-ReaINVP, we sought to maintain roughly the same depth over which we propagate gradients as in the baseline. To this end, our cou- pling networks were ResNets of size 4 x 64, and each Po,|z, and 2t.12z used ResNets of size 2 x 64. Our (se, 44) network was ResNet of size
Table 2: Average plus/minus standard error of test-set bits per dimension. RealNVP (k) refers to a RealNVP model with k residual blocks in the coupling networks.
Fashion-MNIST
CIFAR-10
RealNVP (4)
2.944 + 0.003
3.565 + 0.001
RealNVP (8)
2.946 + 0.002
3.554 + 0.001
2 x 8. We give dg the same shape as single channel of zg, and upsampled to the dimension of z by adding channels at the output of the (st, (e) ResNet. Our model had 5.99M parameters for MasHion-MNIST and 6.07M parameters for CIFAR-10.
LGF-RealNVP (4)
2.823 = 0.003
3.477 = 0.019
Table shows that LGFs consistently outperformed the baseline models. We moreover found that LGFs tend to train faster: the average epoch with best validation performance on CIFAR-10 10 was 458 for LGF-RealNNV, and 723 for RealNVP (8) Samples from all models are shown in Appendix D.4
We also found that using the ELBO instead of the log- likellood does not penalise our method. The gap between the estimated test-set log-likeu ligeliid and the average test-set ELBO was not very large for the LGF models, with relative eттоr of 8.98 10-3 3 for Rashion-MNIST and 6.88 10-3 for CIFAR-10. Moreover, the importance-smmpling-aee
log-likelihood estimator itself had very low variance when using 1000 importance samples. For each trained LGF model, we estimated the relative standard deviation using three separate samples of this estimator. We obtained an average over all trained models of 8.34 x 10 for Pashion-MNIST. and 2.07 x 10 for CIFAR-10.
6 CONCLUSION
In this paper, have proposed localised generative flows for density estimation, which gener- alise existing normalising flow methods and address the limitations of these models in expressing complicated target distributions. Our method obtains successful empirical results variety of tasks. Many extensions appear possible, and we believe localised generative flows show promise as means for improving the performance of density estimators in practice.
No RealNVP (4) run had converged after 1000 epochs, at which point we stopped training. However, by this point the of improvement had slowed significantly.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In International conference on machine learning, pp. 214-223, 2017.
K. Bache and M. Lichman. UCI machine learning repository, 2013. URL httt:://aaccteee ics. uci edu/ml
Jens Behrmann, Will Grathwohl, Ricky TQ Chen, David Duvenaud, and Joem-HHnrik Jacobsen. Invertible residual networks. In International Conference Machine Learning, pp. 573-582, 2019.
Patrick Billingsley. Probability and measure. John Wiley & Sons, 2008.
Christopher M Bishop. Putlern recognition and machine learning. Springer, 2006.
Joseph T Chang and David Pollard. Conditioning as disintegration. Statistica Neerlandica, 51()): 287-317, 1997.
Hari Prasanna Das, Pieter Abbeel, and Costas J Spanos. Dimensionality reduction flows. arXiv preprint
2019.
Laurent Dinh, David Krueger, and Yoshua Bengio. Nice: Non-linear independent components esti-
mation. arXiv preprini 140.v::4400.5//// 2014.
Laurent Dinh, Jascha Sohk-Dicks and Samy Bengio. Density estimation using real NVP. arXiv preprint arXiv: 2016.
Laurent Dinh, Jascha Sohl-Dickstein, Razvan Pascanu, and Hugo Larochelle. RAD approach to deep mixture models. arXiv preprint arXiv: 1903.07714, 2019.
Leo L Duan. Transport Monte Carlo. arXiv preprint arXiv: 2019.
Mathieu Germain, Karol Gregor, lain Murray, and Hugo Larochelle. Made: Masked autoencoder for distribution estimation, In International Conference on Machine Learning, pp. 881-889, 2015.
Will Grathwohl, Ricky TQ Chen, Jesse Betterncourt, Ilya Sutskever, and David Duvenaud. FFJORD: Free-form continuous dynamics for scalable reversible generative models. arXiv preprint arXiv:/880000136,,00 2018.
Alexey A Gritsenko, Jasper Snock, and Tim Salimans. On the relationship between normalising flows and variational- and denoising autoencoders. In DGS@ ICLR, 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016a.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In European conference on computer vision, pp. 635-645. Springer, 201
Jonathan Ho, Xi Chen, Aravind Srinivas, Yan Duan, and Pieter Abbeel. Flow++: Improving flow-
based generative models with variational dequantization and architecture design. In International Conference on Machine Learning, pp. 2722-2730, 2019.
Chin-Was Huang, David Krueger, Alexandre Lacoste, and Aaron Courville. Neural autoregressive flows. In International Conference on Machine Learning, pp. 2083-2092,
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International Conference on Machine Learning, pp. 456, 2015.
Priyank Jaini, Kira A Selby, and Yaoliang Yu. Sum-of-squares polynomial flow. In International Conference on Machine Learning, pp. 3009-3018, 2019.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 14/2.6980, 2014.

---

Under review as a conference paper at ICLR 2020
Durk P Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1x1 convolutions, In Advances in Neural Information Processing Systems, pp. 10215-10224, 2018.
Durk P Kingma and Max Welling, Auto-encoding variational Bayes. arXiv:1312.6114 2013.
arXiv preprint
Durk P Kingma, Tim Salimans, Rafal Jozefowicz, Xi Chen, Ilya Sutskever, and Max Welling. Im- proved variational inference with inverse autoregressive flow. In Advances in neural information processing systems, pp. 4753-4751, 2016.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.
Lars Maaloe, Casper Kaae Sønderby, Søren Kaae Sønderby, and Ole Winther. Auxiliary deep gen- erative models. In International Conference on Machine Learning, pp. 1445-1453, 2016.
Chris J Maddison, Andriy Mnih, and Yee Whye Teh. The concrete distribution: A continuous relaxation of discrete random variables. arXiv preprint arXiv: 10001111..00222.1 2016.
George Papamakarios, Theo Pavlakou, and lain Murray. Masked autoregressive flow for density estimation. In Advances in Neural Information Processing Systems, pp. 2338-2347, 2017.
Gabriel Peyré, Marco Cuturi, et al. Computational optimal transport. Foundations and Trends in Machine Learning, 11(5-6)3355-677, 2019.
Danilo Rezende and Shakir Mohamed. Variational inference with normalizing flows. In Interna- tional Conference on Machine Learning, pp. 1530-1538, 2015.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and ap- proximate inference in deep generative models. In International Conference on Machine Learn- ing. pp. 1278 1278-1286, 2014.
Casper Kaae Sønderby, Tapani Raiko, Lars Maaløe, Søren Kaae Sønderby, and Ole Winther. Ladder variational autoencoders. In Advances in neural information processing systems, pp. 3738-3746, 2016.
Yichuan Tang, Ruslan Salakhutdinov, and Geoffrey Hinton. Deep mixtures of factor analysers. In Proceedings of the 29th International Conference on International Conference on Machine Learning, pp. 1123-1130. Omnipress, 2012.
Lucas Theis, Aäron den Oord, and Matthias Bethge. A note the evaluation of generative models. Xiviv preprint arXiv: 151.011511,.144,, 2015.
Rianne van den Berg, Leonard Hasenclever, Jakub M Tomczak, and Max Welling. Sylvester normal- izing flows for variational inference. In UAI 2018: The Conference Uncertainty in Artificial Intelligence (UAI). pp. 393-402, 2018.
Aaron Van den Oord and Benjamin Schrauwen. Factoring variations in natural images with deep Gaussian mixture models. Advances in Neural Information Processing Systems, pp. 3518- 3526, 2014.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: novel image dataset for benchmark- ing machine learning algorithms. arXiv preprint arXiv: 170.:07777 2017.
Zachary Ziegler and Alexander Rush. Latent normalizing flows for discrete sequences. In Interna- tional Conference on Machine Learning, pp. 7673-7682, 2019.
10

---

Under review as a conference paper at ICLR 2020
A ADDITIONAL PROOFS
A.1 PROOF OF PROPOSITION ! (SEE PAGE 2
Proposition (Limitations of Normalising Flows). Suppose each is homeomorphism and det (z) * for all and If the prior pz has support that is not homeomorphic to p'x, then no choice of 0 will yield a normalising flow model that matches p'x exactly.
Proof. For any choice of let p'x (x) be the normalising flow model produced by pz and ga. i.c. (=) pz (g0(())) det (3)(
We can bound the total variation distance drv between p'x and px by
/ suppy's
dTv(Px.PX) 2
px (x()))
2x (x))d
supp p
px (x))
Observe that this strictly positive unless supp pi = supp px (up to set of probbaibby 0). But since go is a homeomorphism, this is not possible. Hence drv (px p'x) > O. so that p3 # p'x O
A.2 PROOF OF PROPOSITION (SEE PAGE 5
Proposition 2 (Benefit of Localised Generative Flows). Suppose that supp pi open and G(::u) is continuous for each u. Suppose further that, for each supp P'x the set
B₂ == {u E U pz (G(;; u))l det DG u) > 0}
has positive Lebesgue measure. Then there exists Puiz such that supp px supppx-
Proof. Take pu to any positive density. Define
Au == G(suppp:::u)
Observe that each A, has positive Lebesgue measure, since it is the continuous injective image of an open set. Thus, for each u, we can choose (2)o0(-(u) to be any positive density on A,. This allows us to define
(z
Poiz(u|2)= = Pzzu (2/w/ (u
We claim this construction gives supppx = rupprx- First suppose æ E supp For each u, observe that Pznu (G(z; u)lu) by the definition of A., and hence Pui G(:; u)) 0 since pu is positive. Consequently,
px(x) 2
Pz(G(a; (l)popzuG))p w))l det DG(x;u) du > 0
B_
since the integrand strictly positive B., which has positive measure.
Now suppose x E supprx. For each since (- is supported on Ag, we must have Pziu (G(x; u)) = 0, and hence PuIz (u) = 0. From this it follows directly that px(x) = O, i.e. x & supppx.
A.3 CORRECTNESS OF POSTERIOR FACTORISATION
We want to prove that the exact posterior factorises according to
PUL. P
II Pu. (uelze), 8=1
where Z,, X, 21. = xr and žt-1 = Ge(ze; up) for S L. convenient to introduce the index notation Us = U2+1: First note that we can always write this posterior autoregressively as
P (u1_2k=) = II
( (ueluse,
(8)
8=1

---

Under review as a conference paper at ICLR 2020
Now, consider the graph of the full generative model shown in Figure 3a It is clear that all paths from Ug to any node in the set My := {U>E, Zye} are blocked by Ze, and hence that Ze d-separates Hy from Ug [ishhop 2006). Consequently,
PUslz, (wel u>e, = Purlz, (Uslz,(uela (uelze).
Informally
we then have the following for any l
L 1}:
PU (Melussee)) PU..ZilUSe. ( (ug zeluse; (t x)d2
/ PU. (uel use: () (Zglus ddzz
/ pu.lz (ze (Ge-1,me41 o... GL,wz) ) (x)) dže
PUz|z (wel (Ge+I,ue47 .... GL,a)) (x))
Pu,iz, (uelze)
by the definition of z6, where we write Ge(:; G-,((-) remove any ambiguities when com- posing functions. We can substitute this result into equation g to obtain equation
B DISCRETE u
While a discrete u is appealing for its ability to produce exact likelihoods, it also suffers several disadvantages that describe now. First, observe that the choice of the number of discrete values taken by u has immediate implications for the number of disconnected components of supp pX that G can separate, which therefore seems require making fairly concrete assumptions (perhaps implicitly) about the topology of the target of interest. To mitigate this, we might try taking the number of values to very large, but then in turn the time required evaluate equation 4 (now a sum rather than an integral) necessarily increases. This is particularly true when using a stacked architecture, since to evaluate px (z) with L layers each having K possible u-values takes ((K*) complexity. Dinh et al. (2019) propose model that partitions X' so that only one component in each summation is nonzero for any given x, which reduces this cost to (L). However, this partitioning means that their xx is not continuous
function of which is reported to make the optimisation
problem in equation 2 difficult.
Unlike for continuous the ELBO objective is also of limited effectiveness in the discrete case. Since Pu,IZ,-, defines a discrete distribution at each layer, we would need a discrete variational approximation Qu,12, to ensure well-defined ELBO. However, the parameters of discrete dis- tribution are not ameanble to the reparametrisation trick, and hence we would expect our gradient estimates of the ELBO to have high variance. As mentioned above, compromise here might be to use the CONCRETE distribution (Maddison et al., (016) to approximate a discrete u while still apply variational methods. We leave exploring this for future work.
C OTHER CHOICES OF INDEXED BIJECTION FAMILY
Other choices of G than our suggestion in Subsection 3.4 are certainly possible. For instance, we have had preliminary experimental success on small problems by simply taking to be the identity, in which case the model is greatly simplified by not requiring any Jacobians at all. Alternatively, it is also frequently possible to modify the architecture standard choices of g to obtain an appropriate G. For instance, affine coupling layers, key component of models such RealNVP (Dinh et al.) 2016), make use of neural networks that take as input a subset of the dimensions of x. By concate- nating this input, we straightforwardly obtain family of bijections G(:; u) for each value of u. This requires more work to implement than our suggested method, but has the advantage of no longer requiring choice of and t. We have again had preliminary empirical success with this approach. We leave a more thorough exploration of these alternative possibilites for future work.
As with our arguments in Subsection 3.1 this could again he made precise using disintegrations (Chang & Pollard 1997).
12

---

Under review as a conference paper at ICLR 2020
D FURTHER EXPERIMENTAL DETAILS
D.1 LGF ARCHITECTURE
In addition the bijections from the baseline flow, LGF model of the form consider requires specifying P (0e|Z.> and b.. In all our experiments
1 (: = = Normal (1 (2x--11); op op(=e-1))) where Up and op were two separate
outputs of the same neural network
4u (- =e) Normal (1g( og (zx)²), where My and og were two separate outputs of the same neural network
Sg and te were two separate outputs of the same neural network.
D.2 2-D EXPERIMENTS
several experiments the simple 2-D datasets shown in Figure I and Figure& 4] Specifically, we compared the performance of a baseline MAF against an LGF-MAF. We used MLPs for all networks involved our model.
To gain intuition about our model,
For the dataset shown in Figure I the baseline MAF had 10 autoregressive layers consisting of 4 hidden layers with 50 hidden units. The LGF-MAF had 5 autoregressive layers consisting of hidden layers with 10 hidden units. Each was single scalar. The (s,, ty) network consisted of hidden layers of 10 hidden units, and both the (My. ap) and 7,) networks consisted of 4 hidden layers of 50 hidden units. In total the baseline MAF had 80080 parameters, while LGF-MAF had 80810 parameters. We trained both models for 300 epochs.
We used more parameters for the datasets shown in Figure 4, since these targets have more compli- cated topologies. In particular, the baseline MAF had 20 autoregressive layers, each with the same structure before. The LGF-MAF had autoregressive layers, hidden units. The other networks were the same as before, and each ue was still a scalar. In total the baseline MAF had 160160 parameters, while model had 119910 parameters. We trained all models now for 500 epochs.
with 4 hidden layers of 50
The results of these experiments shown in Figure and Figure 4
Observe that LGF-MAF consistently produces a more faithful representation the target distribution than the baseline. A failure mode of our approach exhibited in the spiral dataset, where our model still lacks the power to fully capture the topology of the target. However, we did not find it difficult to improve on this: by increasing the size of the mean/standard deviation networks to hidden layers of 50 hidden units (and keeping all other parameters fixed), we were able to obtain the result shown in [Figure 5) This model had total of 221910 parameters. For the sake the complexity of the MAF model, by the size of its autoregressive networks to 8 hidden layers of 50 hidden units (obtaining 364160 parameters total). This model diverged after approximately 160 epochs. The result after 150 epochs is shown in Figure
fair comparison, we also tried increasing
D.3 UCI EXPERIMENTS
In Tables 3 and 5 we list the choices of parameters for MAF and LGF-MAF, In all cases, we allowed the base MAF have more layers and deeper coupler networks compensate for the additional parameters added by the additional components of our model. Note that neural networks listed size Ki K2, where K, denotes the number of hidden layers and K2 denotes the size of the hidden layers. All combinations of parameters were considered; in each case, there were configurations for MAF and 8 configurations for LGF-MAF.
Table 3: Parameter configurations for POWER and GAS.
Layers
Coupler size
a dim
Ps size
8, size
MAF
5, 10. 20
100, x 200, x 400
LGE-MAF
5, 10
2 x 128
2
2 x 100, 2 x 200
2 128
13

---

Under review as a conference paper at ICLR 2020
Figure 4: Density models learned by standard 20 layer MAF (left) and by 5 layer LGF-MAR (right) for a variety of 2-D target distributions. Samples from the target are shown in black.
Figure 5: Density models learned by larger 20 layer MAF (left) and a larger 5 layer -GF- (right) for the spirals dataset.
14

---

Under review as a conference paper at ICLR 2020
Table Parameter configurations for HEPMASS
Layers
Coupler size
W dim
size
8, s,tsii size
MAF
5, 10, 20
2 x 128, 2 x 512, 2 x 1024 128
LGF-MAF
5, 10
X 128, 2 x 512
128
Table 5: Parameter configurations for MINIBOONE
Layers
Coupler size
dim
size
8, size
MAF
5. 10, 20
2 128, 2 512, 2 1024
LGF-MAF
5, 10
2 128
X 128, 2 x 512
x 128
D.4 IMAGE EXPERIMENTS
In figures to II present samples synthesised from the density models trained on Fashion- MNIST and CIFAR-10.
15

---

Under review as a conference paper at ICLR 2020
Figure 6: Synthetic samples from Fashion-MNIST generated by RealNVP (4)
Figure 7: Synthetic samples from Fashion-MNIST generated by RealNVP (8)
16

---

Under review as a conference paper at ICLR 2020
Figure 8: Synthetic samples from Fashion-MNIST generated by LGF-RealNVP (4)
Figure 9: Synthetic samples from CIFAR-10 generated by RealNVP (4)
17

---

Under review as a conference paper at ICLR 2020
Figure 10: Synthetic samples from CIFAR-10 10 generated by RealNVP (8)
Figure 11: Synthetic samples from CIFAR-10 generated by LGF-ReaINVP (4)
18


Reviews:
[{"review_id": "r1xPlCLSFS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper introduces a straight-forward way to expand the flow models by considering mixture of flow distributions. The idea is not very novel since several previous work have tried the mixture of flow such as the mentioned RAD and Deep Mixture. The paper studies some further improvements such as using the continuous auxiliary variable and stacking multiple mixture layers. \n\nThe major concerns are the following:\n\n1) The paper tries to solve a “problem” build upon intuition. The paper explains as “the normalizing flow places global constraint on the bijection”, “it need to match the topology of  X to the topology of Z ”, “continuous functions necessarily preserve topology”. What kind of topological properties are referred to here? Are all topological properties preserved under continuous function? It needs to be more accurate when using such terminologies. The intuition of the paper is weak and heuristic. The example in Figure 1 can potentially be easily solved with a two component Gaussian mixture of input Z to a vanilla flow model.\n\n2) For the mixture p(X), will the proposed method generate samples concentrated on one or some of the components? Why or why not?\n\n3) Considering there are a plenty of improvements of flow models, it is neccesary for the proposed method to compare with, at least for some methods explained in the Related Work section.\n\n4) Since the proposed methods inevitably lose the advantage of analytic density property of flow methods, it is better to show some advantage over implicit or semi-implicit methods. For example, (https://arxiv.org/abs/1805.11183) also uses a hierarchical model with continuous auxiliary variables and a marginalization similar to Eq.(4) in this paper. How does the proposed method related to or compared to these methods?\n\n5) In experiment section, in Table 1, the proposed method is worse than MAF for 2 datasets out of 4? In Table 2, what are the numbers refer to?\n\n6) On page 4, it should be p(U|Z) not p(U|X)?\n\nIn sum, I think though the paper makes contribution on exploring better flow models but the novelty is relatively weak, the discussion and comparison of related work is insufficient and the experiments are not convincing or have mistakes. I think a modification is necessary before publishing.\n\n################\n\nI have read the author's feedback.\n\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HJgjqGLwtH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose to extend flow-based density models by replacing a single bijection with a hierarchical mixture of bijections. Each component in the mixture is then required to only push the prior onto a local region; this helps improve the coverage of $\\mathcal{X}$. This is motivated by the conjecture that in many cases the topology of $\\mathcal{X}$ might be overly complicated to be effectively captured by a single bijection. Formally, this is achieved by introducing a conditional random variable $U|Z$. In doing however, the log-likelihood is rendered intractable and a variational approximation must instead be resorted to. A recursive formula for computing the ELBO is introduced in this vain. \n\nOverall, I think this is a generally interesting contribution to the normalizing-flow literature that I expect to spark further research. However, there are some rough edges to this paper. The initial motivation is well-presented and relatively easy to follow, though a diagram would serve to cement the intuition regarding the support mismatch. The issue mentioned in footnote 2 deserves further discussion. At the same time, while well-reasoned, their justifications are nonetheless largely conjectural and further theoretical or empirical evidence would be welcome, both for characterising the pathologies they aim to redress and their proposed solution.\n\nFor the experiment section, I would have liked to have seen comparisons not only to the simplest baseline but also to some of the other methods mentioned in related works. In general, the experiment section is quite short and I didn't get a very good sense of how well this method performs.\n\nThe following should be addressed:\n\n- provide more evidence for the conjectures surrounding the motivation and derivation\n- supply more varied baselines (e.g. RAD model)\n\nMinor comments:\n\n- at the top of page 4, you refer to $p_{U|X}$ several times, but I think you mean $p_{U|Z}$\n- in the paragraph after equation (2), the $\\theta$ superscript seems to be missing from the $p_X^\\theta$\n- in the first sentence of section 3.1, you refer to \"the single $g$ used in equation 2\", but equation 2 mentions no $g$\n- in the first line on page 3, you talk about some region of supp $p_X^\\theta$ being pushed out of supp $p_X^*$, shouldn't this be the other way round since the KL is infinite only if the the support of $p_x^*$ is not contained within the support of $p_X^\\theta$?", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BJlmkymy5S", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThis paper conjectures that normalizing flows are fundamentally limited due to the architecture assumption that the generative function g is continuous in x. It is argued that this constraint makes maximum likelihood estimation difficult in general. Localised generative flows are proposed as a solution and consist in modeling the generative model as a continuous mixture of bijections. Experiments suggest an improvement over MAF.\n\nDecision:\nThe observation that continuity imposes a hard constraint on the network is sound, and the proposed solution appears to show some improvement. However, in its current state, this work appears to be quite fragile both from a theoretical and experimental point of view. First, it is only conjectured that this constraint poses actual problems. Second, the experimental evaluation is weak and insufficient. It omits comparisons with more recent generative flows that have shown to be able to model discontinuous densities. For this reason, I do not recommend the paper for acceptance.\n\nFurther arguments:\n- The whole paper rests on intuition without strong theoretical backup. \n- The experiments are quite poor and results frankly oversold. It is said the method \"improves performance across a variety of common density benchmarks\". While we see improvements in Table 1 over MAF, the comparison omits all recent architectures based on Normalizing Flows, such as TAN (Olivia et al, 2018), NAF (Huang et al, 2018), B-NAF (De Cao et al, 2019) or SOS (Jaini et al, 2019). All of those methods have reported better results than those provided in Table 1. They have also been shown empirically to work for discontinuous densities. While I understand that LGF can be combined with any flow architecture, the question remains whether using a continuous mixture translates into significant improvements for those baselines as well. The experimental benchmarks also omit datasets such as BSDS300, for which the higher dimensionality is usually challenging. The same goes for Table 2 which omits recent and better results, such as Glow or FFJORD.\n- Closer to LGF, a proper experimental comparison to RAD (Dinh et al, 2019) would be appreciated.\n- The proposed architecture supposedly enables better generative models. However, this comes at the price that the density can no longer be evaluated exactly and analytically. Since normalizing flows are also typically slow for sampling, this makes the benefits of the proposed architecture quite limited. In particular, it is not clear why generative models that are good at sampling only (e.g., GANs) should not then be preferred?\n- As a result of the point above, the experimental results are reported only in terms of approximated negative log-likelihood. I do not think this is fair, since models like MAF do provide exact values. It also makes the comparison with previous methods more difficult.\n\nFurther feedback:\n- As per ICLR policy, higher standards should be applied to papers with 9 or more pages. I am confident the paper could be written within 8 pages only.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes to overcome some fundamental limitations of normalizing flows by introducing auxiliary continuous latent variables. While the problem this paper is trying to address is mathematically legitimate, there is no strong evidence that this is a relevant problem in practice. Moreover, the proposed solution is not entirely novel, converting the flow in a latent-variable model. Overall, I believe this paper will be of minor relevance to the ICLR community.
Decision: Reject