Deterministic training of generative autoencoders using invertible layers
Authors: Gianluigi Silvestri
Daan Roos
Luca Ambrogioni
Venue: ICLR 2023 notable top 25% (2023)

Abstract: In this work, we provide a deterministic alternative to the stochastic variational training of generative autoencoders. We refer to these new generative autoencoders as AutoEncoders within Flows (AEF), since the encoder and decoder are defined as affine layers of an overall invertible architecture. This results in a deterministic encoding of the data, as opposed to the stochastic encoding of VAEs. The paper introduces two related families of AEFs. The first family relies on a partition of the ambient space and is trained by exact maximum-likelihood. The second family exploits a deterministic expansion of the ambient space and is trained by maximizing the log-probability in this extended space. This latter case leaves complete freedom in the choice of encoder, decoder and prior architectures, making it a drop-in replacement for the training of existing VAEs and VAE-style models. We show that these AEFs can have strikingly higher performance than architecturally identical VAEs in terms of log-likelihood and sample quality, especially for low dimensional latent spaces. Importantly, we show that AEF samples are substantially sharper than VAE samples. 

Published as a conference paper at ICLR 2023
DETERMINISTIC TRAINING OF GENERATIVE AUTOEN- CODERS USING INVERTIBLE LAYERS
Gianluigi Silvestri"
Daan Roos UvA-Bosch Delta Lab
OnePlanct Research Center, imec-the Netherlands Donders Institute for Brain, Cognition and Behaviour gianluigi. si wil.sivvele livestrrie mec. n.l
d. f. a roos @ @uaa n1
Luca Ambrogioni
Donders Institute for Brain, Cognition and Behaviour Radboud University
rog on wolgggiiin@dddderrreeeeo @donders.......o ru.nl
ABSTRACT
this work, we provide a deterministic alternative to the stochastic variational training of generative autoencoders. We refer to these new generative autoencoders as AutoEncoders within Flows (AEF), since the encoder and decoder are defined affine layers of an overall invertible architecture. This results in deterministic encoding of the data, as opposed to the stochastic encoding of VAEs. The paper introduces two related families of AEFs. The first family relies on partition of the ambient space and is trained by exact maximun-likelhhodd The second family exploits a deterministic expansion of the ambient space and is trained by maxi- mizing the log- bog-probability in this extended space. This latter case leaves complete freedom in the choice of encoder, decoder and prior architectures, making a drop-in replacement for the training of existing VAEs and VAE-style models. We show that these AEFs can have strikingly higher performance than architecturally identical VAEs in terms of log- likeliooddo and sample quality, especially for low dimensional latent spaces. Importantly, we show that AEF samples are substantially sharper than VAE samples.
INTRODUCTION
Rezende et al. 2014) have maintained an enduring popularity in the machine learning community in spite of the impressive performance of
Variational autoencoders (VAEs) (Kingma & Welling 2014
other generative models (Goodfellow et al. 2014 Karras et al. 2020; Van Oord et al. [016]] Van den
Oord et al. 2016; Salimans et al. 2017 Dinh al. [204]] Rezende Mohamed [2015] Dinh et al.
2017 Kingma & Dhariwal 2018 Sohl-Dickste et al. 2015 Nichol & Dhariwal [2021). One key
Nichol
feature of VAES is their ability to project complex data into a semantically meaningful set of latent variables. This feature is considered particularly useful in fields such as model-based reinforcement learning, where temporally linked VAE architectures form the backbone of most sfate-of-the-art world- models (Ha & Schmidhuber 2018b a Hafner et al. 2020 Gregor et al.) 2019: Zintgraf et al. 2020 Hafner et al., 2021). Another attractive feature of VAES is that they leave ample architectural freedom generative models, with their signature encoder-decoder architectures being popular in many areas of ML beside generative modeling (Ronneberger et al.)
when compared with other likelihood-batte
2015 Vaswani et al. 2017; Radford et al.) 2021 Ramesh et al. 2021 2022). However, VAE training is complicated by the lack of a closed-form expression for the legllikllhood. with the variational gap between the surrogate loss (i.c. the ELBO) and the true log-likelihood being responsible for unstable training and, at least in non-hierarchical models, sub-optimal encoding and sample quality (Hoffman
Johnson 2016 Zhao et al.) 2017 Alemi al., 20188] Cremer et al.) 2018 Mattei & Frelisen 2018). Consequently, a large fraction of VAE research is devoted to tightening the gap between
"qqual contribution.
Work done while Radhoud University

---

Published as a conference paper at ICLR 2023
the ELBO and the true likelihood of the model. Gap reduction can be achieved both by devising
alternative lower bounds (Burda et al. 2015: Bamler et al. 2017) or more flexible parameterized
posterior distributions (Rezende & Mohamed [015]]ingna et al. 2016). Normalizing flows (NF) are deep generative models comprised of tractably invertible layers, whose log-likelihood can be computed in closed-form using the change of variable formula (Kobyzev et al.] [020) |Papamakarios ict al., (021). However, this constraint appears to be at odds with autoencoder architectures, which map all relevant information in latent space of different (often smaller) dimensionality. This is potentially problematic since naturalistic data such as images and speech waveforms are thought to live, at least approximately. in a lower dimensional manifold of the ambient space (Bengio et al., 2013: F. et al. 2016 Pope et al., 2021). It is therefore common to use hybrid VAE-flow models that deploy NFs for modeling the VAE prior and/or the variational posterior (Rezende & Mohamed 2015, Kingma et al., 2016). However, training these models is often a delicate business as changes in the encoder and the prior cause a misalignment from the posterior, increasing the gap and causing shifting-target dynamic that introduces instabilities and decreases performance. For this reason, the complex autoregressive or flow priors common in modern applications are often trained ex- ex-post after VAE training (Van Den Oord et al.) 2017 Razavi et al. 2019). In this paper we introduce a new approach for training VAE-style architectures with deterministically encoded latents. The key insight is that we can formulate an autoencoder within conventional invertible architecture by using invertible affine layers and by keeping track of the deviations between data and predictions. Importantly. this can be done while leaving complete freedom in the design of the encoder, decoder and prior, which makes our approach a drop-in replacement for the training of existing VAE and VAE-style models. The resulting models can be trained by maximum- likelhhod using the change of variables formula. We denote these new generative autoencoders as autoencoders within flows (AEF), since the autoencoder architecture is constructed inside a NF architecture.
2 PRELIMINARIES
In this section, we will outline the standard theory behind probabilistic generative modeling and non-linear dimensionality reduction. Consider a dataset comprised of data-points æ E R. We refer to R. as the ambient space. The dataset is assumed to be sampled from a D-dimensional curved manifold M embedded in the ambient space. We refer to M as the signal space. The dimensionality of the signal space reflects the true dimensionality of the signal while the dimensionality of the ambient space depends on the particularities of the measurement device (e.g. the nominal resolution of the camera).
Variational autoencoders: VAFs are deep generative models in which the density of each data- point depends on D--dimensional stochastic latent variable z RD, which parameterizes the signal space. The emission model is often assumed to be a diagonal Gaussian with parameters determined by deep architectures: p(æ zz; [[;;;; fz;; 0)) where denotes the model parameters. In this formula, the parameterized functions f [z; 0) and (2; are the outputs of a decoder architecture. The emission model is paired with prior Po(2; f) over the latents. While the marginal likelihood is intractable, it is possible to derive a lower bound (the ELBO) by introducing a parameterized approximate posterior defined by the encoder architectures gma (x; 4)) and ga (z; )) which respectively return the posterior mean and scale over the latent variables. An additional normalizing flow transformation Roost (-; v)) is often included in order to account for the non-Gaussianity of the posterior (Kingma et al. 2016). Stochastic estimates of the gradient of the ELBO can be computed by expressing samples from the posterior
differentiable deterministic
function of the random samples (Kingma & Welling 2014: Rezende et al., 2014). For our purposes, it is important to notice that the Gaussian reparameterization formula:
2(2 e; (b) gm (m; v) + (x: Oc.
defines an affine invertible layer, formally analogous to those used in RealNVPs and related NFs (Dinh et al. 2017; Papamakarios et al. (2017). In the simplified case of Gaussian residual model with variance o², the reparameterization of the ELBO leads the following surrogate objective function
2

---

Published as a conference paper at ICLR 2023
for one data-point
=Be
20 ||
(2(2,, (4:); (1)
log po (5))
Prior loss
Reconstruction error
N + log 2TT
Hlg
Posterior entrogy
While this loss has an interpretable appeal, it is important to keep in mind that it is just tractable surrogate for the log-likelihood. The gap between the ELBO and log- leelliielihood can be decomposed into an inference gap and an amortization gap (Cremer et al. (2018). Large gaps lead to highly sub-optimal training since the network is trained on loose approximation of the true objective.
Normalizing flows and affine layers: a sub-optimality in VAE training that can only be remedied using complex posterior models high variance lower bounds. NFs are alternative methods that assume the latent manifold to have the same dimensionality of the ambient space. Consider M = RV and Φ(y; 0) being an invertible differentiable mapping. Under these assumptions, the log-likelihood of the model can be expressed in closed form using the change-of- variable formula log p(z) = log Pu (y)) + log det DΦ D
The unavailability of a closed form log- ligeliieldood introduces
where D normal, the loss has the form
(y) is the Jacobi matrix of the inverse mapping. the base distribution Po is standard
1 (NF( - (x; + log det D
NF architectures are designed by composing K invertible layers Φk. Affine layers divide the variables in two blocks yo and sig one of which remain unchanged and the other is scaled and translated based on the first du (v VK+1 s(y{) @ 2 + m(y)). where is an arbitrary positive-valued function and m is an unconstrained function. When applied in the inverse direction, this layer becomes (vL-+1 m( (v²),, which "predicts away" the mean and variance of one block given the other. Using several of these layers with randomized partitions gradually removes statistical dependencies until at the last layer the resulting y variable approximately matches the uncorrelated Gaussian target Po-
3 METHOD
3.1
AUTOENCODING WITH NORMALIZING FLOWS
In this section we show how to define a deterministic generative autoencoder within a NF architecture. Given its similarity with autoencoders, we named this flow architecture as autoencoder within flows (AEF). For now, we assume & N, namely that the manifold dimensionality of the signal much lower than the ambient dimensionality. We will remove this assumption in the next sections. Let us begin by partitioning the input signal x E R two subsets x done for coupling layers in NFs. In this paper, we refer to 1:: and x P+IN as core and shell variables respectively. As we shall see, the core variables are in 1-to-1 relation with the latents while the shell variables are (approximately) "predicted away" by the latents. We define mean (shell) + RD and a scale (shell) encoder architectures analogous the mean and scale encoders of VAE except for the fact that they only take the shell variables as input. We also define an invertible core encoder n (-; RRD + R2 that takes care of encoding the core variables. Using these architectures and the core/shell partition, we encode the data (both core and shell) into the latent variables via an invertible affine transformation:
1:D and x D++1N as commonly
encoder (::(::)) 0) N N-D De.D----o
R. N-D -> RDD These two
Jm (z: D+INN
A) ;0)
9s (1 D+IN, O) O n (x::
In order to recover the data from the latent, we define decoder fD+1:N RD N-D This decoder only needs to reconstruct the shell variables since the core variables can be recovered by inverting the core encoder. As far as the scale encoding is not zero, the encoding formula defines a surjective transformation between the ambient space R and the latent space R D. Such transformation is not invertible since information concerning the shell variables be lost and,
3

---

Published as a conference paper at ICLR 2023
consequently, (=(x) # z. We can circumvent this problem by retaining the residuals of the autoencoder as additional variables:
8D+1N =
= D41:N -x+t:N
(5)
These variables retain the information lost during encoding, thereby completing the surjective encod- ing formula into an invertible transformation &b R + R N defined as &b 0--((xx '(x1:" >++1:N 2, 2 Formally, the variables 8 are parts of the 'latents' of the NF. However, conceptually they are not true latent variables as they can only account for additive white noise. We can now define a base distribution po 0) to the latents. This distribution defines generative model in the latent space RD and it is analogous to the prior in VAEs. For this reason, we will often (improperly) refer (o 0) as the 'prior'. However, it important to keep in mind that this distribution is not Bayesian prior since AEFs latents do not have straightforward Bayesian interpretation. Moreover, we define an error distribution r(;; 8) to the residuals. It is important for this distribution to have a learnable scale parameter that can be scaled down during training as the residuals are "predicted away". summary, the invertible architecture $-1 1 is trained to map the distribution of the data into factorized distribution of latent codes and residuals:
(a.:D D+1:N, 0) ~ po(2; ())();
(6)
The exact negative log- likelihood loss can be obtained by using the change of variable formula of normalizing flows (Eq.
N
CAEF(0, 02) 20
D+IN fD+IN (z(x; 0):0)
log 2TO² log po (= 0);
2
log ((5:))
log| det DΦ (z: 9)
where for the sake of simplicity we assumed (centered) Gaussian residual noise with (trainable) variance 2². It is now possible note some striking similarities between the flow just described and VAE. Eq. 4 resembles the Gaussian reparametrization trick for variational autoencoders, with the reparameterized posterior noise replaced by the core variables. Furthermore, equation eloosey resembles the ELBO in equation but without noise injection, with the reconstruction error applied only to the shell variables and the Jacobian term replacing the entropy of the posterior distribution. As usual in flows, the AEF architecture can be used as generative model simply by sampling the latent code z from the 'prior' (2; 0) and inverting the AEF transformation & (-). Since it is usually not useful to add residual noise to the generated data, can set 8D+1:N 0 instead of sampling it from the error distribution. This results in the following procedure:
(
2 9m OI
2 ~ (o(2;) 0) + D+INN = DD+1:N + II:D =
(8)
Do D2tiNN
This is visualized in Figure (b) Appendix A. and implemented in Algorithm
3.2 PARTITION STRATEGIES AND AMBIENT SPACE EXPANSION
So far, we did not specify how select core and shell variables. The most straightforward approach is to to use an arbitrarily chosen partition of the original variables. For example, we could use a random partition or, in an image, we could extract central sub-image of "core pixels" and have the image with cropped center as shell. An AEF with this kind of partition interpolates between a regular normalizing flow (for D N) and an autoencoder (for D < N). However, this approach limits the number of latents to be smaller than N and reduces compatibility with the VAE literature. Furthermore, in the case of noise corrupted data this partitioning strategy does not allow for denoising of the core variables since they do not have corresponding deviations and error distributions. All these issues can be circumvented if we appropriately expand the ambient space. In fact, the dimensionality of the ambient space is usually largely arbitrary, depending on factors such as the sampling resolution of cameras and digital microphones instead of the physical features of their respective measured signals. Consider a parameterized injective function V R - N defined as follows (x:7) (z, = h(z; where y are the transformation parameters. This function expands the ambient dimensionality but, since the transformation is injective and deterministic, it leaves the dimensionality of the signal
4

---

Published as a conference paper at ICLR 2023
manifold unchanged. Conceptually, this should be seen form of feature expansion and not architectural component of a flow. In this expanded space, we can define deviations for all the original variables and, consequently, can be done simply by using a regular AEF architecture with the original variables x as shell and (xx) core. This results in the following encoding layer:
loss with VAE-style reconstruction enror on all variables. This
== (2; 0) + (x; 0) @ n nnnnnnnhh (1((:)))00
Note that this differs from the well-known VAE reparameterized encoding formula (E. just by the fact that the input white noise is replaced by deterministic function of the data. This is the only architectural difference between VAEs and AEF in the extended space. It is not immediately obvious that the feature expansion parameters can be trained by maximizing the log- likeelioooodd However, this joint training can be fully justified as the minimization of (the limit of) KI. divergences (see Section 3.3) which results in the following objective function:
CAE
1
f(3(1, h(:: (): 1 log 2TO²
N
(10)
20
log Po (z h(a; y): 0): 8) log det D& D h(x:)): I,
which is just the log-likelihood in the expanded ambient space. We include the pseudocode for computing the objective function as negative log likelihood in Algorithm
Algorithm Negative Log Likelihood AEF on expanded ambient space: g Encoder; f: Decoder;
h: feature expansion map; n: core encoder; Po: Base distribution ("prior'); ); r: error distribution; 0: model parameters, y: feature expansion parameters; z: input image
procedure NEGATIVELOGG.IELHHOONS
log det t- 0
w + h(x:7)
9m (a;; 0) ga (2; 0) @ n nw;
log det J +- log det J| log det (w; 0)) Σ Elogg (x;
6 +- f(;;
return (log po (2; 0) + (o:r((;00) + log det J))
During generative sampling, the additional variables w are redundant and can be discarded. This results generative sampling formula that identical the one used in VAEs (see also Algorithm 3): = f(2; 0), 2 ~ Po(2; 0). Consequently, the core flow n does no longer participate in the generative sampling and instead has an auxiliary role. This is sign of its strong relation with the posterior flow in VAF, which is already implicit in Eq. 9 The marginal likelihood in the original ambient space p(x) = f p(a, w w)dw cannot be obtained closed-form from the joint p(æ, w). Note that, at least from the point of view of manifold learning and dimensionality reduction, the original likelihood p(x) itself does not have a privileged interpretation as the ambient dimensionality usually arbitrary and the 'true' likelihood of the data is degenerate and lives lower dimensional manifold. Nevertheless, p(x) is often useful for evaluation purposes (model comparison). Therefore, we provide an efficient importance sampling scheme in Appendix C All likelihood results reported in this paper have been corrected using this method so as to be comparable with the VAE baselines.
Algorithm 2 Sampling AEF with core-shell partition: Encoder; f: Decoder; n: core en- coder; Po: base distribution ('prior'); 0: model parameters
Algorithm 3 Sampling AEF with expanded space: f: Decoder; Po: base distribution
('rrior') ); 0: model parameters
procedure SAMPLE ~ (z
procedure SAMPLE ~ (2:
- f(z:0)
x D+INN
+ f(;;0)
return r
-- 3m D41N
11: +
On (2
+ cat(x 1:D D+1 D+INN)
return ø
5

---

Published as a conference paper at ICLR 2023
3.3 TRAINING THE EXPANSION PARAMETERS 7
The feature expansion map (ax 7) is, strictly speaking, not component of the flow architecture. Therefore, in spite of its intuitive appeal, it is not obvious that can train y by minimizing the joint negative log-likelihodd. However, this can be fully justified as minimization of the KI. divergence between the generative model and the empirical distribution of the data with expanded dimensionality, The idea is to define a loss functional that can be used to make two parameterized distributions approach each other, instead of just having a parameterized distribution approaching a fixed target distribution. We will start considering a stochastic dimensionality expansion and then show that the deterministic case can be obtained as a limit. We will denote the empirical sampling distribution of the dataset d(xr). Now consider the joint distribution of the empirical data and the stochastically expanded latent dimensions:
(11) where the expansion is performed using a Gaussian conditional density with fixed variance. The KL divergence between the expanded empirical distribution g(a, w) and the joint distribution of the AEF model p(ar, w) is given by:
la (2, w) (hz) Io)
DKL (g, w), p(z, w)), = -Bz,e Deree-oloogpee w)] +
(12)
where the constant e the differential entropy of gor (ar, w), which independent from both 0 and y. This happens because the differential entropy of a Gaussian variable does not depend on its mean. We can now compute the gradient with respect to y::
DKL (go w). p(a, w)) Ez, V. log,(z, w)]
(13)
which, up to a constant, is just the gradient of the stochastically augmented joint ligelikeidooo. We can now notice that, while the KI. divergence is not well-defined at the limit a -> 0, the limiting gradient is well-defined since the diverging entropy term does not depend on 7. This leads with the following limiting gradient:
lim V. (yr (I, p(r, w))
(14)
a a-0
= lim Eg,e [V, log pl h(x, 7) + a Ge)
(15)
o-0
= [, log ((, ())]
(16)
where ha, 7) + a O c, the second line, is the the reparameterization formula for the conditional Gaussian variable We can now see that, up to proportionality constant, is the gradient of the negative ligeliiel used to train the flow.
4 RELATED WORK
Improvements of VAE training: In recent years, there have been many works analyzing and improving the training procedure of VAFs. Works such as (Hoffman & Johnson] 2016). (Zhao et al. 2017) and (Alemi et al. 2018) diagnose some issues with ELBO training and propose a series of methods to improve results. (Rezende & Viola 2018) and (Dai & Wipf 2019), propose augmented objective to tackle specific optimization issues. In the same vein, (Cremer et al. 2018) and (Mattei) & Frellsen 2018) offer theoretical analysis of the factors affecting the quality of approximate inference in VAËS and introduce strategies to alleviate them. None of these works introduce an exact maximum likelihood training. popular way to improve VAEs is to use more flexible prior or posterior distribution. (Rezende & Mohamed] 2015) and (Kingma et al. 2016) use NFss as variational posteriors, showing how the aggregate posterior matches the prior more closely, while (Tomczak) & Welling 2018) use a variational mixture of posteriors as prior. MMorrow & Chiu (020) use a two-stage training procedure, in which first they train a VAE with NF prior, and then combine the decoder with Glow (Kingma & Dhariwal 2018) for improved sample quality. An alternative line of research is the use of hierarchical priors with several stochastic layers (Sønderby et al. 2016) Maaløe et al. 2019). More recent works use hierarchical VAEs with many layers, achieving state of the art log- likelihood and generating images with impressive sample quality (Vahdat & Kautz 2020: Child 2021 Hazami et al.) 2022). These works use latent dinensionality that is greater, often by orders of magnitude, than the dimensionality of the ambient space. This dimensionality expansion greatly ameliorate the problem of posterior separations diagnosed in (Dai & Wipf 2019) at

---

Published as a conference paper at ICLR 2023
the price of higher computational and memory costs and lower interpretability. Interestingly, a recent post-training analysis showed that just a few percents of the hierarchical VAE's latent dimensions are needed to encode the data (Hazami et al. 2022).
Stochastic auxiliary variables and augmented normalizing flows:
The fact that VAE encoding
can be seen a form of affine coupling was first noticed in (Dinh et al. 2014) (Appendix C). This allows one to conceptualize the one-sample reparameterization estimate of the ELBO as a likelihood in a stochastically augmented space. However, this is just an equivalent re-formulation and does not solve the inference sub-optimality problems of VAES, which in this setting can be explained by the fact that sharp posteriors correspond to singular points of non non-invertibility. This approach was recently generalized in (Huang et al. (020), where several VAE-style affine layers are stacked in an flow architecture that takes noise augmented input. The stacking procedure removes the signal/noise separation and autoencoder-like loss of VAEs since the final prediction is not compared with the original image but only with the previous layer. The resulting model very similar to other
flows (Cornish et al. 2020 Weilbach et al. 2020: Caterini et al. 2021a).
Two-steps training procedures: recent trend in generative modeling is to disentangle the tasks of learning low-dimensional latent representation and maximum-likelinn likelihood density estimation. (Dai & Wipf, 2019) proposes a two-stage procedure for training VAEs, and show its effectiveness both theoretically and empirically. (Ghosh et al.) 2020) propose to instead use explicit regularization scheme for the decoder, and employ an ex-post density estimation on the latent space to allow for sampling and ensure that the latent space is distributed according to simple distribution. Similarly, (Xiao et al., 2019) and (Böhm & Seljak| 2020) use a deterministic autoencoder to learn the latent representation of the data, and a normalizing flow to model the distribution of such latents, leading to better density estimation while avoiding over-regularization of the latent variables. More recently, (Loaiza- Ganem et al., 2022) discusses the problem of manifold overfitting, which arises when the manifold is learned but not the distribution on it, and propose two-step training procedure applicable to all the likelihood-basse
models.
NFs on manifolds: Several authors propose variations of NFs that can model data on manifold. Works such as (Gemici et al., (016)) (Mathieu & Nickel, 2020) and (Rezende al. (020)) assume that the dimensionality reducing map is already known and available. (Kim et al. 2020 Horvat & Phiste 202T) propose methods based on adding noise to the data, which are capable of learning unknown lower-dimensional manifolds. Injective flows strive for the same goal by using injective deterministic transformations to map the data to a lower dimensional base density. (Brehmer & Cranmer 2020), (Kothari al. 2021 and (Cramer et al. 2022) train injective flows with two-steps training procedure in which they alternate manifold and density training, while (Kumar et af.) (020) introduces lower bounds on the injective change of variable. (Cunningham al. (0220) combines injective flows with an additive noise model to account for deviations from the learned manifold
using stochastic inversions trained on a variational bound. In (Caterini al. 2021b the authors
train injective flows by evaluating the in-manifold likelihood using a modification of the change of variable formulas for rectangular Jacobi matrices, which is then supplemented by an additional ad-hoc reconstruction loss. (Ross& & Cresswell 202T) uses conformal embeddings to train injective flows with exact on-manifold likelioeld plus an additional ad hoc reconstruction loss.
A work that is in spirit similar to ours is Sur VAE (Nellsen et al. 2020), in the sense that it aims at bridging the gap between Normalizing Flows and VAEs by combining their strengths. Sur extend flow models by incorporating stochastic and surjective transformations in both the generative and the inference direction. The (generative direction) surjective transformations give rise to stochastic estimates of the likelihood contribution and introduce lower bound likelihood estimates. On the other hand, surjections in the inference direction allow exact probability computations for the latent variables. However, these inference surjections themselves cannot be straightforwardly used to perform generative dimensionality reduction since the stochastic forward transformation p(z|z) is intractable and depends on the data probability itself via Bayes theorem (which is exactly the quantity we want to estimate in generative modeling). In general, the contributions of our work and SAE are almost orthogonal and could be combined to obtain the benefits of both techniques, for example by incorporating surjections in our decoder architecture so as to enforce symmetries in the data.
SurVAF flows:
7

---

Published as a conference paper at ICLR 2023
5 EXPERIMENTS
We now show empirically that, at least in complex naturalistic datasets such as CelebA-HQ and ImageNet Karras et al. (2018); (Deng et al. (2009), the exact maximum likelihood training leads to drastically improved results compared to architecturally equivalent VAEs. Our aim is to show the difference in performance between exact log- log-likelihood and ELBO objective functions when the architecture is kept constant. Therefore, in VAE baselines do not use the many regularization, annealing and KL scaling terms that are somewhat common in VAE applications (Vahdat & Kautz 2020; Child 202T). However,
make sure to compare architecturally identical models with strong prior and variational posteriors, which in themselves ameliorate many of the pathologies of the ELBO (Kingma et al. 2016 Tomczak & Welling 2018). As main baseline, we use the original VAE algorithm as introduced in (Kingma & Welling (0144) with an Inverse Autoregressive Flow (IAF) posterior and a Masked Autoregressive Flow (MAF) prior. Apart from the feature expansion layer, this baseline is architecturally identical to our AEF (linear) model, which uses learnable linear features as core variables. We also test two AEF models that do not expand the ambient space, one with the center pixels and the other with the corner pixels as core variables, referred to as AEF (center) and AEF (corner) respectively. Additional experiments on denoising are reported in Appendix B while more details about the experiments and results can be found in Appendix D and The code for all the experiments is available at: https://github..omm httpttuubb.oommiiiis/////E gis
Generative modeling and manifold learning: To compare the generative performance of AEFs with VAES we test on CelebA-HO AHH resized to 64 x 64 and 32 x 32, and ImageNet resized to 32 x 32. We focus models with significantly less latent variables than observable dimensions since this is the regime that is the most problematic for traditional VAE training. We use a complex Encoder- Decoder architecture with residual blocks, similar the one used in (Child (021), and compare the performances of the AEF (with linear core variables) and its architecturally identical VAE for different latent dimensions. Table shows that AEFs significantly outperform their architecturally equivalent VAEs both in terms of bits per dimension and FID score, generating significantly sharper and more detailed samples (Fig. | To compute bits per dimension, we use importance sampling, as described in Appendix D.6 Additionally, we look at smaller scale datasets: MNIST, MaSS and KMNIST (Deng) 2012 Xiao al., (017] (0077/Clanuwatt et al., 2018). Here we use small conventional encoder and decoder architectures (see Appendix D.I). Results on MNIST are shown in Table for latent dimensionality of 2 and 32. In Appendix L we present these results for all latent dimensions and datasets, well as the results for AEF (center) and AEF (corner). Performance for all three versions of AEF is comparable, with the linear AEF performing slightly better. Overall, we observe an increase in performance for AEFs compared with VAEs. On the other hand, VAE outperforms AEF for a high number of latent dimensions. In Appendix | we investigate the importance of the posterior and prior flows on generative performance for both AEFs and VAEs. Here observe that incorporating a prior flow is very important to sample quality for both models. Additionally, we observed that, in this setting, even the AEF without flows performs better than VAE with both posterior and prior flow in terms of BPD and FID.
MNIST
CelebA (64 x 64)
CelebA (32 x 32)
ImageNet
Models
2
32
128
256
64
128
256
128
256
BPD
2.40
1.80
6.98
6.86
6.79
6.90
6.64
6.63
6.89
6.74
VAE
FID
57.5
16.2
106
92.6
90.9
87.6
67.0
63.5
179
160
BPD
2.37
1.82
6.35
6.07
5.78
6.20
5.88
5.57
6.30
6.21
AEF
FID
55.8
17.7
76.5
60.8
55.8
51.9
38.4
30.9
131
123
Table 1: Bits per dimension (BPD) and Frechet Inception Distance (FID) for AEF and VAE models trained on various datasets. The second row denotes the latent dimensionality used. Lower is better for both metrics. For MNIST we report the mean over five runs, for the other datasets we report the mean over two runs. For both VAE and AEF, BPD values are estimated using importance sampling. We refer to Appendix E for more results, and Appendix for details on the importance sampling estimators.
8

---

Published as a conference paper at ICLR 2023
CelebA-HQ (64x64)
ImageNet (32x32)
64
128
256
Figure 1: Reconstructions from an AEF (top row) and VAE (bottom row) with equivalent ar- chitectures trained on rescaled CelebA-HQ and rescaled ImageNet with 256 latent dimensions.
Samples from an AEF (top row) and VAE (bottom row) with equivalent architectures trained on rescaled CelebA-HQ (64 64) with 64, 128, and 256 latent dimensions respectively. Images were sampled with a temperature of 0.85.
Figure 2:
6 DISCUSSION AND CONCLUSIONS
In this work we showed that autoencoders, if properly constructed out of invertible layers, can be trained by maximum-likellhhn either in the original ambient space or in an appropriately expanded space. This latter approach results in an objective function that can be directly used in the training of any pre-existing VAE and VAE-like model that uses Gaussian residual noise. In our experiments, we showed that in many datasets AEFs perform strikingly better than VAEs. Importantly, the AEF images where not affected by the blurriness typical of fow-dimensional VAEs, which resulted in very remarkable difference in the quality of samples and reconstructions. This an interesting and perhaps surprising result as the AEF and VAE models were architecturally identical. Given the arguments presented in Alemi et al. 2018). we conjecture that this difference is due to the failure of the VAEs to converge to sufficiently sharp posterior, which has been proven to result in poor separation between the encoding of the training samples. We further conjecture that this failure is due the fact that the optimum is very close singular point of con-invertibility of the VAE architecture, which results in numerical instabilities due to the diverging KL term. This problem does not affect AEFs since the encoding always deterministic. In our experiments, the main exception to this trend was MNIST with 32 latent dimensions, where the VAE performed consistently better. We conjecture that this is due the relatively large ratio between the latent dimensionality and the true signal dimensionality. which results in less concentrated (and less unstable) variational posterior distributions. this regime, the extra statistical variability provided by the posterior samples of the VAE can loosen the topological constraints of the architecture, potentially leading to higher performance Cornish et al. (2000); [aterini et al.] (2021a). When compared to VAEs, the main limitation of AEFs is that they cannot straightforwardly use discrete emission models (decoders) since their flow architecture is assumed to work on continuous data. Another limitation is that, since we are not using stochastic auxiliary variables, AEFs have the same topological constraints of regular NFs (Cornish et al., [020] Caterini et al., 2021a). However, auxiliary variables can be straightforwardyy added AEFS using standard methods Caterini et al. (2021a). Finally, from the point of view of the NF literature, our work opens the door to hybrid autoencoder-floww models that can learn how to reduce the latent dimensionality by "predicting away" some dimensions, possibly at several different stages of processing. Adding manifold learning capabilities to NFs has great potential since it can avoid some of the pathologies of invertible models when used lower dimensional data, while at the same time increasing training efficiency.
7 REPRODUCIBILITY
We provide access to all the code used in our experiments here: https//gihhub..mm/gsssssAAE The readme contains instructions on how to reproduce the experiments presented in the paper from the command line, well Jupyter notebook example training AEF from scratch using the

---

Published as a conference paper at ICLR 2023
provided code. To further increase the clarity of our proposed method, have added diagrams that visually explain the sampling and inference procedure for our model in Appendix [A] as well as corresponding pseudocode (algorithms) I
The reader find additional explanations of the methods in Sections [.3 a Additional details on the performed experiments such as description of architectural details and hyperparameter used in this work can be found in Section Ablation experiments are found in Section in the Supplementary material.
ACKNOWLEDGEMENTS
OnePlanet Research Center aknowledges the support of the Province of Gelderland.
REFERENCES
Alexander Alemi, Ben Poole, Ian Fischer, Joshua Dillon, Rif A Saurous, and Kevin Murphy. Fixing broken ELBO. International Conference on Machine Learning, 2018.
Robert Bamler, Cheng Zhang, Manfred Opper, and Stephan Mandt. Perturbative black box variational inference. Advances in Neural Information Processing Systems, 2017.
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence, 35(8):1798-182) 2013.
Lukas Biewald. Experiment tracking with weights and biases, 2020. com/ Software available from wandb.com.
Vanessa Böhm and Uroš Seljak. Probabilistic auto-encoder. arXiv preprint arXiv:2006.05779, 2020.
Johann Brehmer and Kyle Cranmer. Flows for simultaneous manifold learning and density estimation. Advances in Neural Information Processing Systems, 2020.
Yuri Burda, Roger Grosse, and Ruslan Salakhutdinov. Importance weighted autoencoders. arXiv preprint 1rXiv:1509.005// 2015.
Anthony Caterini, Rob Cornish, Dino Sejdinovic, and Arnaud Doucet. Variational inference with continuously-indexed normalizing flows. Uncertainty in Artificial Intelligence, 2021a.
Anthony L Caterini, Gabriel Loaiza-Ganem. Geoff Pleiss, and John Cunningham. Rectangular flows for manifold learning. Advances in Neural Information Processing Systems, 2021b.
Rewon Child, Very deep vaes generalize autoregressive models and can outperform them on images. International Conference on Learning Representations, 2021.
Tarin Clanuwat, Mikel Bober-Irizar, Asanobu Kitamoto, Alex Lamb, Kazuaki Yamamoto, and David Ha. Deep learning for classical japanese literature. arXiv preprint arXiv: 1812.0118, 2018.
Rob Cornish, Anthony Caterini, George Deligiannidis, and Arnaud Doucet. Relaxing bijectivity constraints with continuously indexed normalising flows. International conference on machine learning, 2020.
Eike Cramer, Felix Rauh, Alexander Mitsos, Raúl Tempone, and Manuel Dahmen. Nonlinear isometric manifold learning for injective normalizing flows. arXiv preprint arXiv:2203.0392 039344 2022.
Chris Cremer, Xuechen Li, and David Duvenaud. Inference suboptimality in variational autoencoders. International Conference on Machine Learning, 2018.
Edmond Cunningham, Renos Zabounidis, Abhinav Agrawal, Ina Fiterau, and Daniel Sheldon. Normalizing flows across dimensions. arXiv preprint arXiv:2006.1300 2020.
Bin Dai and David Wipf. Diagnosing and enhancing vae models. International Conference on
Learning Representations, 2019.
10

---

Published as a conference paper at ICLR 2023
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hier- archical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248-255, 2009. doi: 10.1
Li Deng. The mnist database of handwritten digit images for machine learning research. IEEE Signal Processing Magazine, 29(6): 141-142, 2012.
Laurent Dinh, David Krueger, and Yoshua Bengio. Nice: Non-linear independent components estimation. arXiv preprint arXiv:: 16, 2014.
Laurent Dinh, Jascha Sohl- Sohl-Dickstein, and Samy Bengio. Density estimation using real nvp. Interna- tional Conference on Learning Representations, 2017.
Conor Durkan, Artur Bekasov, lain Murray, and George Papamakarios. nflows: normalizing flows in PyTorch, November 2020. URL Attps:: org. 10. 5051/ .20.lo...9999 4296287
Charles F., Sanjoy M., and Hariharan N. Testing the manifold hypothesis. Journal of the American Mathematical Society, 2944),983-1049 2016.
Mevlana C Gemici, Danilo Rezende, and Shakir Mohamed. Normalizing flows riemannian manifolds. arXiv preprint 1.Xv:1661.00334 2016.
Mathieu Germain, Karol Gregor, lain Murray, and Hugo Larochelle. Made: Masked autoencoder for distribution estimation. International Conference on Machine Learning, 2015.
Partha Ghosh, Mehdi SM Sajjadi, Antonio Vergari, Michael Black, and Bernhard Schölkopf. From variational to deterministic autoencoders. International Conference on Learning Representations, 2020.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 2014.
Karol Gregor, Danilo Jimenez Rezende, Frederic Besse, Yan Wu, Hamza Merzic, and Aaron van den Oord. Shaping belief states with generative environment models for rl. Advances in Neural Information Processing Systems, 2019.
David Ha and Jürgen Schmidhuber. Recurrent world models facilitate policy evolution. Advances in neural information processing systems, 201 8a.
David Ha and Jürgen Schmidhuber. World models. arXiv preprint 1rXiv: 2018b.
2018b.
Danijar Hafner, Timothy Lillicrap. Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. International Conference Learning Representations, 2020.
Danijar Hafner, Timothy Lillicrap, Mohammad Norouzi, and Jimmy Ba. Mastering atari with discrete world models. International Conference on Learning Representations, 2021.
Louay Hazami, Rayhane Mama, and Ragavan Thurairatnam. Efficient-vdvae: Less more. arXiv preprint arXiv:2203.13777 2022.
Matthew Hoffman and Matthew Johnson. Elbo surgery: yet another way carve up the variational evidence lower bound. In Workshop in Advances in Approximate Bayesian Inference, NIPS, 2016. 6.
Christian Horvat and Pascal Pfister. Denoising normalizing flow. Advances in Neural Information Processing Systems. 2021.
Chin-Wei Huang, Laurent D., and Aaron C. Augmented normalizing flows: Bridging the gap between generative flows and latent variable models. arXiv preprint arXiv:2002.07101, 2020.
Tero Karras, Timo Aila. Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for improved quality, stability, and variation. International Conference on Learning Representations, 2018.
11

---

Published as a conference paper at ICLR 2023
Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality stylegan. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp. 8110-8119, 2020.
Hyeongju Kim, Hyeonseung Lee, Woo Hyun Kang, Joun Yeop Lee, and Nam Soo Kim. Softflow: Probabilistic framework for normalizing flow on manifolds. Advances in Neural Information Processing Systems, 2020.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. International Conference on Learning Representations, 2015.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. International Conference on Learning Representations, 2014.
Durk P Kingma and Prafulla Dhariwal. Glow:
Generative flow with invertible 1x1 convolutions. Advances in neural information processing systems, 2018.
Durk Kingma, Tim Salimans, Rafal Jozefowicz, Xi Chen, Ilya Sutskever, and Max Welling. Improved variational inference with inverse autoregressive flow. Advances in neural information processing systems, 2016.
Ivan Kobyzev, Simon JD Prince, and Marcus A Brubaker. Normalizing flows: An introduction and review of current methods. IEEE transactions on pattern analysis and machine intelligence, 43 (11):3964-3979; 2020.
Konik Kothari, AmirEhsan Khorashadizadeh, Maarten de Hoop, and Ivan Dokmanić. Trumpets: Injective flows for inference and inverse problems. Uncertainty in Artificial Intelligence, 2021.
Abhishek Kumar, Ben Poole, and Kevin Murphy, Regularized autoencoders via relaxed injective probability flow. International Conference on Artificial Intelligence and Statistics, 2020.
Gabriel Loaiza-Ganen Brendan Leigh Ross, Jesse C Cresswell, and Anthony L. Caterini. Diagnosing and fixing manifold overfitting deep generative models. arXiv preprins arXiv:220.00717.. 2022.
Lars Maalóe, Marco Fraccaro, Valentin Liévin, and Ole Winther. Biva: A very deep hierarchy of latent variables for generative modeling. Advances in neural information processing systems, 2019.
Emile Mathieu and Maximilian Nickel. Riemannian continuous normalizing flows. Advances in Neural Information Processing Systems, 2020.
fierre-Alexandrr Mattei and Jes Frellsen. Leveraging the exact likelihood of deep latent variable models. Advances in Neural Information Processing Systems, 2018
Rogan Morrow and Wei-Chen Chiu, Variational autoencoders with normalizing flow decoders. arXiv preprint r=Xiv:2004.056/7 2020.
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. International Conference Machine Learning, 2021.
Didrik Nielsen, Priyank Jaini, Emiel Hoogeboom, Ole Winther, and Max Welling. Survae flows:
Surjections to bridge the gap between vaes and flows. Advances in Neural Information Processing Systems, 13:12285-1299 2020.
George Papamakarios, Theo Pavlakou, and lain Murray. Masked autoregressive flow for density estimation. Advances neural information processing systems, 2017.
George Papamakarios, Eric Nalisnick, Danilo Jimenez Rezende, Shakir Mohamed, and Balaji Lakshninarayanan. Normalizing flows for probabilistic modeling and inference. Journal of Machine Learning Research, 22(57): 2021.
Phillip Pope, Chen Zhu, Ahmed Abdelkader, Micah Goldblum, and Tom Goldstein. The intrinsic dimension of images and its impact on learning. International Conference on Learning Represen- fations, 2021.
12

---

Published as a conference paper at ICLR 2023
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. International Conference on Machine Learning, 2021.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to- rext-to-image generation. International Conference on Machine Learning, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.0612) 2022.
Ali Razavi, Aaron Van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with vq-vae-2. Advances in neural information processing systems, 2019.
Danilo Rezende and Shakir Mohamed. Variational inference with normalizing flows. International conference on machine learning, 2015.
Danilo Jimenez Rezende and Fabio Viola. Taming vaes. arXiv preprint arXiv:1810.005997 2018.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. International conference on machine learning, 2014.
Danilo Jimenez Rezende, George Papamakarios, Sébastien Racaniere, Michael Albergo, Gurtej Kanwar, Phiala Shanahan, and Kyle Cranmer. Normalizing flows on tori and spheres. International Conference on Machine Leurning, 2020.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer- assisted intervention, pp. 234-241. Springer, 2015.
Brendan Ross and Jesse Cresswell. Tractable density estimation on learned manifolds with conformal embedding flows. Advances in Neural Information Processing Systems, 2021.
Tim Salimans, Andrej Karpathy, Xi Chen, and Diederik P Kingma. Pixclenn++: Improving the pixelenn with discretized logistic mixture likelihood and other modifications. International Conference Learning Representations, 2017.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. International Conference on Machine Learning. 2015.
Casper Kaae Sønderby, Tapani Raiko, Lars Maalge, Søren Kaae Sønderby, and Ole Winther. Ladder variational autoencoders. Advances in neural information processing systems, 2016.
Jakub Tomezak and Max Welling. Vae with vampprior. International Conference Artificial Intelligence and Statistics, 2018.
Arash Vahdat and Jan Kautz. Nvae: A deep hierarchical variational autoencoder. Advances in Neural Information Processing Systems, 2020.
Aaron Van den Oord, Nal Kalchbrenner, Lasse Espeholt, Oriol Vinyals, Alex Graves, et al. Conditional image generation with pixelenn decoders. Advances in neural information processing systems, 2016.
Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems, 2017.
Aaron Van Oord, Nal Kalchbrenner, and Koray Kavukcuoglu. Pixel recurrent neural networks. International conference on machine learning, 2016.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 2017.
13

---

Published as a conference paper at ICLR 2023
Christian Weilbach, Boyan Beronov, Frank Wood, and William Harvey. Structured conditional continuous normalizing flows for efficient amortized inference in graphical models. International Conference Artificial Intelligence and Statistics, 2020.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: novel image dataset for benchmarking machine learning algorithms. arXiv preprini arXiv:1708.07747, 2017.
Zhisheng Xiao, Qing Yan, and Yali Amit. Generative latent flow. arXiv preprint 19XXi::905.044 10485, 2019.
Shengjia Zhao, Jiaming Song, and Stefano Ermon. Towards deeper understanding of variational autoencoding models. arXiv preprint arXiv: 2017.
Luisa Zintgraf, Kyriacos Shiarlis, Maximilian Igl, Sebastian Schulze, Yarin Gal, Katja Hofmann, and Shimon Whiteson. Varibad: A very good method for bayes-adaptive deep rl via meta-learning. International Conference on Learning Representations, 2020.
14

---

Published as a conference paper at ICLR 2023
SUPPLEMENTARY MATERIALS
A ADDITIONAL DIAGRAMS
In Figure 3 we show diagrams for inferece and sampling procedures, for AEFs with partitioning and expanded ambient space.
2- Sic
e (y (9)
af M)
n (w; 0)
1 + n 1(2112,0)
g gowix
#u + pu(es 9) on Yur
f1=:8;
ftz 6)
Sis
2- Puls d)
8 x- f(:
z- Pulsof)
(d)
Figure 3: Diagrams showcasing the inference and sampling process for (a, b) AEFs with partioning: (c. d) AEFs with expanded ambient space.
B DENOISING EXPERIMENTS
Denoising is one of the main applications of the classical autoencoder literature. The basic idea is that the signal can be compressed in low dimensional manifold while the (white) noise, being incomprehensible, is filtered out. The denoising problem shows the capacity of our training scheme to separate the signal and the noise spaces, thereby performing an adaptive non-linear filtering. We test performance on CelebA-HQ (32 x 32), and the MNIST, FashionMNIST and KMNIST datasets with different noise levels. We compare against architecturally equivalent VAFs and least squares denoising autoencoders (AE). All models were trained exclusively on noisy data. To compare performance look at the mean squared error between the Inception feature activations (the same ones as used for the FID score) for an original, noiseless sample and the reconstruction based on its noisy version. Results for CelebA-HQ and MNIST are shown in Table aable Fig. for various noise regimes, while results for MashionMNISTT and KMNIST and the other noise regimes can be found in Appendix E.2 For CelebA-HQ we that the gap performance between AEF and VAE for denoising is similar to the one in generative modeling. On the other hand, for the less complex MNIST datasets we observe that the VAE often performs better than both the AEF and the denoising AE, which suggests that the posterior uncertainty of the stochastic latent variables of the VAE may play beneficial role the denoising performance.
C IMPORTANCE SAMPLING FOR LINEAR AEF
After training, we wish to compute the probability p(x) for model comparison purposes, we would need to solve the integral p(x) p(x, w)dw, which cannot be obtained in closed form. However, we can use an importance sampling scheme:
p(x) = p(z, w)dw = f ((w)) p(a, w) du =
p(x, wk)
Σ
(17)
q(w)
K
g(wa)
15

---

Published as a conference paper at ICLR 2023
CelcbA-HQ (128)
MNIST (2)
MNIST (32)
0.05
0.1
0.2
0.5
0.75
1.0
0.5
0.75
AE
0.080
0.078
0.078
0.135
0.142
0.144
0.086
0.115
0.133
VAE
0.082
0.081
0.085
0.085
0.094
0.109
0.045
0.058
0.075
AEF
0.058
0.063
0.075
0.082
0.091
0.102
0.053
0.072
0.094
Table 2: Mean squared error between inception feature activations of original inputs and reconstruc- tions of noisy inputs. The number between parentheses denotes the latent dimensionality of the models. The second row gives the standard deviation of the noise distribution. For CelebA-HQ and MNIST we report the mean over two and five runs respectively.
(a) AEF
(b) VAE
(c) AE
Figure 4: Examples of reconstruction performance of AEF, VAE, and deterministic autoencoder (AE) trained on CelebA samples with noise level (o) of 0.1, and latent dimensionality of 128. Top: image with added noise; Middle: reconstructed image; Bottom: original image.
To do so, we first compute w h(; y), then take K samples from a normal distribution centered around wt Wi....K ((w)= .V(w: <), where is the approximate posterior scale and needs to be tuned on the validation set for each model. We perform the rest of the computations using W Wi....K
instead of and finally compute the probability in Eq. 17 To reduce the variance of the estimator,
we additionally use importance weighted sampling:
1
p(æ, wk)
log p(z) log E (01)
(18)
g(wx)
D EXPERIMENTS' DETAILS
D. ENCODER AND DECODER
For the MNIST datasets, all the encoders and decoders consist of a two-layers convolutional neural network. The encoder uses 3 kernels, 64 for the first layer and 128 for the second. Each layer is followed by ReLU activation. Finally, two linear layers map the feature maps to mean and standard deviation of the latent space respectively. Similarly, the decoder first uses linear layer increase the dimensionality of the latent samples, and then two transposed convolutional layers with respectively 128 and 64 kernels of size 4. The decoder outputs two values: the mean and the standard deviation, as set of trainable parameters constrained with softplus activation.
For the larger CelebA and ImageNet datasets we use an encoder and decoder with many more layers and residual blocks, an architecture similar to (Child (021). In particular, we use the same residual bottleneck block, but the encoder does not output activations at intermediate layers, and the decoder processes only the input coming from the previous layer, without stochastic sampling and prior computation. In other words, we reuse the residual bottleneck blocks from (Child 2021) to build a "traditional" encoder-decoder architecture, and the outputs of the encoder and decoder are the same as for the manifold learning and denoising experiments. In all the experiments, we use four residual bottleneck blocks for each feature map size.
16

---

Published as a conference paper at ICLR 2023
D.2 MASKED AUTOREGRESSIVE FLOW
For all of our NFs, we and adapt implementations from (Durkan et al. 2020). Our MAF models stack MADE autoregressive layers (Germain et al. 2015). each with residual blocks with N hidden units. We add ActNorm (Kingma & Dhariwal (018) between each autoregressive layer. IAF is simply the inverse of MAF. When MAF or IAF are used within an autoencoder architecture, whether as prior, posterior encoder flow, use K 4 and N 256.
D.3 PREPROCESSING LAYER
For all models, apply preprocessing bijective transformation like the used in Paapaaakario et al. 2017)
x logit (A + (1 - 2A)=)
(19)
where z the input image and À a parameter. We use = le 1 for the MNIST-like datasets and À 0.05 for CelebA-HQ and ImageNet. This transformation is then followed by ActNorm layer.
D.4 DATASETS AND DATA PRE-PROCESSING
In all our experiments, u ~ M(0, 1) to the image and then divide by 256. Division by 256 requires an adjustment in the log likelihood, which take into account when computing the bits per dimension.
dequantized version of the data, in which we first add uniform noise
For the denoising experiments we add gaussian white noise ((0, σ)) to the samples, with different levels of a depending the dataset: 0.25, 0.5. 0.75 and 1.0 for MNIST datasets, and 0.05, 0.1 and 0.2 for CelebA-HQ. By varying the standard deviation of the noise distribution we can increase the intensity of the noise. After adding the noise the images we clip them that the pixel values stay in their original range.
For CelebA-HQ we apply random left-right flipping whenever image is loaded into batch. For all the models we use 10% of the training dataset as validation set, apart from CelebA-HQ for which use the predefined train-val-test split.
D.5 TRAINING PROCEDURE
On the MNIST datasets we train all models for 100K iterations, and we evaluate the metrics on the iteration that achieved the best validation loss. For CelebA-HQ (32 x 32 and 64 x 64), we train instead for 1M iterations and do early stopping if the validation loss does not improve for more than 20k iterations. ImageNet models are trained for 2M iterations with early stopping set to 100K iterations with no improvement. For generative modeling on CelebA-HQ, ImageNet, and all the denoising models we use gradient clipping if the magnitude of the gradients is bigger than 200. As optimizer we choose ADAM (Kingma Ba 2015) with a learning rate - for the MNIST-like datasets, and 1e for denoising experiments, CelebA-HQ and ImageNet. We use batch size 128 for all the MNIST experiments, a batch size of 64 for CelebA resized to 32 x 32 and a batch size of 16 for CelebA-HQ resized to 64 x 64 and ImageNet.
We use Weights & Biases to track all our experiments and store models' checkpoints. (Biewald 2020
D.6 IMPORTANCE SAMPLING
To get an estimate of the marginal likelihood for both VAE and AEF (lincar) we use importance sampling with an average of 20 times 128 samples.
D.7 COMPUTE RESOURCES
We used AzureML and Google Colab Pro run all our experiments. All the manifold learning and denoising experiments are run on one GPU NVIDIA M60, while we use one GPU NVIDIA K80 for the experiments CelebA-HQ and ImageNet.
17

---

Published as a conference paper at ICLR 2023
D.8 MODELS' PARAMETERS
We report the number of parameters used in each model for the different datasets: Table 3 for the models trained MNIST-like datasets, Table 4 for CelebA-HQ resized to 32 x 32, and Table
Nr. of latent dimensions
Models
2
8
16
32
VAB
2.37M
2.42M
2.52M
2.71M
3.12M
AEF
2.37M
2.42M
2.52M
2.71M
3.12M
AEF (linear)
2.37M
2.42M
2.53M
2.73M
3.15M
Table 3: Approximate number of model parameters for all models trained on MNIST-like datasets.
Nr. of latent dimensions
Models 64
128
256
VAE
2.80M
4.07M
8.32M
AEF (linear)
2.99M
4.46M
9.11M
Table 4: Approximate number of model parameters for all models trained on CelebA-HQ and ImageNet rescaled to 32 x 32.
Nr. of latent dimensions
Models
128
256
512
VAE
4.34M
9.40M
28.0M
AEF (linear)
5.91M
12.5M
34.3M
Table 5: Approximate number of model parameters for all models trained on CelebA-HQ rescaled to 64 x 64.
18

---

Published as a conference paper at ICLR 2023
E ADDITIONAL RESULTS
E. GENERATIVE MODELING AND MANIFOLD LEARNING
In this section we provide additional results. Table 6 shows the results of all runs on CelebA-HQ and ImageNet. We show results comparing the linear and partitioned versions of AEF with VAEs on the MNIST datasets in Figure 5 and Figure 6 shows samples of VAEs and AEFs trained on MNIST, FashionMNIST and KMNIST with latent dimensionality of 32, and Figure shows the same for CelebA-HQ resized to 64 x 64 for various latent dimensionalitiis.
CelebA (64 x 64)
CelebA (32 x 32)
ImageNet
Models
64
128
256
64
128
256
128
256
6.98
6.87
6.79
6.98
6.70
6.63
6.89
6.76
BPD
6.99
6.86
6.79
6.90
6.64
6.65
6.92
6.74
VAE
106
96.3
90.9
91.9
68.4
63.5
179
164
FID
102
92.6
92.1
87.6
67.0
66.2
182
160
6.35
6.07
5.78
6.20
5.86
5.58
6.30
6.24
BPD
6.35
6.07
5.79
6.21
5.93
5.57
6.31
6.21
AEF
76.5
60.8
55.8
51.9
38.4
27.9
131
123
FID
76.2
62.6
54.9
53.3
40.2
30.9
132
124
Table 6: Bits per dimension (BPD) and Frechet Inception Distance (FID) for both runs of AEF and VAE models trained CclebA and ImageNet. The second row denotes the latent dimensionality used. Lower is better for both metrics.
Figure 5: Bits per dimension achieved by AEFs and equivalent VAE MNIST, FashionMNIST and KMNIST. We show the mean and 95% confidence interval over 5 runs.
8"
Figure 6: FID score achieved by achieved by AEFs and equivalent VAE multiple datasets. We show the mean and 95% confidence interval over 5 runs.
19

---

Published as a conference paper at ICLR 2023
7
7
7
?
9
y 7
7
7
9
7
7
7
7
7
7
?
9
9
9
7
7
9
9
3
6
7
7
9
5
5
7
7
7
5
7
7
7
7
S
5
7
7
7
9 5 4
5
7 7
7
7
7
S 5
Figure 7: Visualization of a learned two-dimensional manifold by AEF trained on MNIST with two-dimensional latent space.
VAE
AEF (linear)
AEF (center)
AEF (corner)
8
1
6 8
6 8 6
4
7 6 7
7
/
/
O
0
9
6
9
3
7 4
5
3 6
9
a
4
9
3
8
Figure 8: Samples generated by VAEs and AEFs trained on MNIST, FashionMNIST and KMNIST with 32 latent dimensions. Samples were generated with temperature of 0.85.
VAE
64
128
256
Figure 9: Samples generated by VAES and AEFs trained on CelebA-HQ resized to 64 64 with 64, 128 and 256 latent dimensions. Samples were generated with temperature of 0.85.
20

---

Published as a conference paper at ICLR 2023
E.2 DENOISING
This section presents additional results and figures comparing the denoising performance of AEFs baseline models, specifically to VAE with equivalent architecture and a least squares denoising autoencoder (AE) with equivalent encoder and decoder. Figure presents the mean squared error between inception feature activations (IFE) for increasing levels of noise MNIST, FashionMNIST and KMNIST datasets.
Figure 10: Mean squared error between inception feature activations of the original inputs, and the reconstructions based a noisy input for increasing levels of noise. Averaged over five runs. Bars indicate 95% confidence intervals. The number of latent dimensions for was set to 2 for the upper row, and 32 for the lower row.
F ABLATIONS
In this section we investigate the importance of the prior and posterior flow on the performance of both VAEs and AEFs. We train a VAE and AEF with and without prior and posterior flow on CelebA-HQ (32 x 32), and report the results in BPD and FID in Table 7 Qualitatively and by FID score we observe that for both VAE and AEF the quality of samples generated without prior flow present the model are of significantly worse quality than when there is prior flow present. Additionally, having only posterior flow decreases samples quality in both the VAE and the ABF. is interesting to note that an AEF without core encoder and prior flow still performs better than VAE with both posterior flow and prior flow in terms of BPD and FID score. Examples of samples for each ablation are presented in Figure and Figure I2
BPD
FID
VAE (no flows)
6.68
77.2
VAE (only posterior)
6.56
81.5
VAE (only prior)
6.71
74.1
VAE (both flows)
6.67
68.1
AEF (no flows)
5.90
64.4
AEF (only posterior)
5.88
108
AEF (only prior)
6.05
37.6
AEF (both flows)
5.90
35.6
Table 7: Ablation results of VAEs and AEFs with and without a posterior flow/core encoder and prior flow.
21

---

Published as a conference paper at ICLR 2023
VAR
Figure 11: Samples and reconstructions of VAEs and AEFs trained on CelebA-HQ resized 32 32 with: 1) no posterior or prior flow; 2) only posterior flow; 3) only prior flow; 4) both posterior and prior flows. All models had latent dimensionality of 128. Samples were generated with temperature of 0.85.
VAE
AEF
7
7 :
e
7 8
4
9
6
4 4
3
3
2
9
7
3
4
9
6 6
1
1 7
9
7
8
4
7
6
J 4
6
4 2
7
9
4
7
4
;
1
1
9
8
4 4
9 9
6
7 {
4
6 3
; :
9
8
4
9 6 6
3
2
7 1
7 3
6
2
9 7
9
4
9
6
9
2
9
6 5
9 7
3
4 4
9 6
6
2
9
6 5
9
7
8
4
9 6 6
8
5
9
3
3
/
9
7
3
9
6
8
5
9
3
3 /
9
7
8
4
9
4 3
8 3
9 7
8
4
9 6
4
O
7
8 6
9
7
8
4
9 6 6
(4)
6
8
2
3
8
9
7
9
4
7
8
6
4
9
7
9
4
9
Figure 12: Samples and reconstructions of VAEs and AEFs trained on MNIST with: 1) no posterior
prior flow; 2) only posterior flow: 3) only prior flow; 4) both posterior and prior flows. All models had a latent dimensionality 32. Samples were generated with a temperature of 0.85.
22


Reviews:
[{"review_id": "u1Zqshx6rd", "reviewer": "Reviewer_4VSE", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a novel scheme for training a VAE by embedding it inside a normalizing flow model on the expanded ambient space.  Specifically, the method uses an invertible network to transform the expanded ambient dimensions followed by an affine coupling layer to compute the \"latent\" variable of the embedded VAE.  To satisfy the invertibility requirement, the output of the VAE decoder is augmented with error (residual) terms.  Authors then empirically show that this flow-VAE hybrid scheme leads to qualitatively sharper images and improved log likelihood on MNIST, CelebA-HQ and ImageNet 32x32.", "summary_of_the_review": "The paper proposes a novel training scheme for VAE by embedding it inside a flow model in an expanded ambient space. The idea is well-motivated, and the empirically results show clear improvement.  I believe the proposed method is an interesting hybrid of VAEs an normalizing flows, and is valuable to the community.", "strength_and_weaknesses": "**Strength**\n* Interesting idea with clearly explained motivations.\n* Clear empirical results, both qualitative and quantitative.\n* Thorough discussion on how the proposed method relates to existing approaches.\n\n**Weakness**\n* Title is misleading -- the proposed method does not perform max-likelihood training of a VAE with an arbitrary architecture.  It instead constructs a normalizing flow model in the expanded ambient space and trains that flow model with exact likelihood instead.  See my first question below.\n* Writing is a bit verbose and could be condensed (see \"Clarity\" below).\n\n**Questions**\n* It's unclear to me how the ambient space expansion affects the likelihood.  Authors point out in Sec 3.4 that $p(x)$ does not have a particular advantage over $p(x, w)$ under the manifold hypothesis, but I believe this point is debatable. More importantly, it would be informative to know how $p(x, w)$ relates to the tightness of the ELBO. \n* I'm curious what the learned error distribution $r(\\delta; \\theta)$ looks like.  Speficially: how does actually sampling $\\delta \\sim r()$ change the samples, and what is the learned value of $\\sigma$?\n\n**Minor comments**\n* Eq. 4 uses $g_{\\sigma}$ but the text uses $g_s$\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "fVQy2Xsyy_g", "reviewer": "Reviewer_jiVP", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes to overcome the training limitations of VAEs as both manifold learners and generative models.\nEssentially, the proposed AEF architecture is one giant coupling layer.\nIt transforms some dimensions (called the \"core\") using a standard normalizing flow (similar to the use of inverse autoregressive flows for latent variables) and conditions on the other dimensions (called the \"shell\").\nThe key trick to make the whole architecture invertible is to save the residuals of the reconstructed x as additional variables (i.e., the other latent dimensions for a flow model).\nThus, the core latent space + the residuals of the shell space is the entire space and can be inverted.\nThe paper extends this to allowing an augmented space where the augmentation variables are functions of the input, and while this disables observed log likelihood computation, it avoids having to choose the shell vs core.\nFinally, the paper shows that this new approach is usually better than the equivalent VAE in terms of FID, bits per dimension, and qualitative inspection of samples and reconstructions.\n", "summary_of_the_review": "This paper proposes an simple yet useful combination of autoencoder ideas and normalizing flows to enable flexible VAE-like normalizing flows that could be used as drop-in replacements for standard VAEs. Overall, this is an excellent direction and a nice contribution. The main weakness is the lack of comparison to SurVAE flows, which seem to have similar ideas and bridge VAEs and flows more fully. Though I believe the contributions are still interesting especially by making it a drop-in replacement for VAEs, I believe the proposed technical ideas are less novel because of the prior work in SurVAE flows. A good comparison to this prior work could improve my score.\n", "strength_and_weaknesses": "*Strengths*\n- The approach can use standard VAE and AE architectures directly within the framework. Thus, it provides an elegant bridge between VAEs and flows.\n\n- The expanded AEF version does not require specifying the shell and core variables which makes it more broadly applicable as a dro-in replacement for standard VAEs.\n\n*Weaknesses*\n- How does your work compare to SurVAE Flows [Nielsen et al., 2020]? This work also bridges VAEs and flow models. Can your model be cast as a type of SurVAE Flow?  This seems to be a significant oversight in related works and may suggest another reasonable baseline.\n\nNielsen, D., Jaini, P., Hoogeboom, E., Winther, O., & Welling, M. (2020). Survae flows: Surjections to bridge the gap between vaes and flows. Advances in Neural Information Processing Systems, 33, 12685-12696.\n\n\n- (Minor) It is not clear that the sampling procedure for estimating the likelihood for the expanded AEF is good, and it may be impossible to check. While the VAE values are a true lower bound on log likelihood. The FID and qualitative samples definitely help support the overall claims so this is not necessarily a huge concern but it should be noted that the sampling procedure is merely an approximation to the likelihood and could be above or below the true likelihood. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "bhsxTPT2hlZ", "reviewer": "Reviewer_LqjZ", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose a method to train an auto-encoder in which the mapping to a latent space is invertible, making the (exact) likelihood tractable and so allowing maximum likelihood training. The method involves augmenting the data with additional variables (obtained from a learnable function of the data) and then running a single affine normalizing flow layer on the augmented space. This normalizing flow layer encodes these additional variables using an affine function that depends on the original data and sets the remaining latent variables to be the residual of the data after subtracting the prediction given the latents. The prior in the latent space is Gaussian, and a learnable prior for the residuals encourages the network to drive the residuals to zero and therefore use the other latents.\n\nIn my mental model of this, the prior for the residuals has the feel of a pixelwise Gaussian likelihood in a traditional VAE. A significant difference from standard VAEs, though, is that no noise is added by the encoder.", "summary_of_the_review": "The proposed method is interesting and demonstrates advantages over the VAEs introduced in (Kingma and Welling, 2014). The paper could be improved further with better demonstration of use-cases for this method, or demonstration of compatibility with more modern (e.g. hierarchical) VAE architectures.", "strength_and_weaknesses": "Strengths:\n - A conceptually interesting method and integration of VAEs with normalising flows\n - Clear and well-structured\n\nWeaknesses:\n - The main weakness of this paper is that the architectures experimented with are far from the state-of-the-art, coming from (Kingma and Welling, 2014). Despite having referenced \"compatibility with the VAE literature\", and specifically VAEs with higher-dimensional latent spaces than data spaces, as a motivation for the ambient space expansion, none of the VAEs in the experiments have this property. I would be interested in e.g. whether this method could be applied in some way to more modern (e.g. hierarchical) VAE architectures.\n- Based on Figure 10 and Table 2, it seems like the advantage of AEFs for denoising only holds for low latent dimensionality, and diminishes as the latent dimensionality increases. This suggests that this method is unlikely to be useful for practical denoising tasks. I would also be worried about whether the same scaling with latent dimensionality holds for other tasks (e.g. use as a generative model).\n- Another possible advantage of older VAE architectures, mentioned by the authors, is the \"interpretability\" of their latent space and \"their ability to project complex data into a semantically meaningful set of latent variables\", but the authors do not demonstrate that the interpretability of their method's latent features (or usefulness for downstream tasks) is any better than that of baselines. I mention this merely because, based on my previously listed weaknesses, the method does not appear to have significant advantages as a generative model or as a denoiser. I would see this as a much stronger paper if the authors could make a convincing case for their method's applicability to at least one use-case.\n\nMinor:\n - Acronym \"AEF\" used in paragraph afer Eq. 5 without having been previously introduced.\n\nOther minor comment:\n - It is not clear to me what properties the learned auxiliary variables ($w=h(x; \\gamma)$) would have. Some intuition about what makes a good auxiliary variable would be interesting.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "3FPWUiCONE", "reviewer": "Reviewer_Mtne", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a novel generative model class for situations when the data distribution is concentrated around a low dimensional manifold in the input space. The authors propose to enlarge the input space by the outputs of a non-linear parametrised mapping that can be thought to represent the manifold coordinates of a data point. They then define a normalising flow model on this enlarged space, which leads to a deterministic invertible encoder. The corresponding decoder/encoder pair can be learned by maximising the data likelihood. Experiments show that the proposed model outperforms VAEs with comparable architectures on datasets like CelebA and ImageNet.", "summary_of_the_review": "The paper proses a novel type of generative models under the assumption that the data distribution is concentrated around a low dimensional manifold in the input space. The derived likelihood maximisation seems to be correct and outperforming comparable VAEs learned by ELBO. It is perhaps possible to achieve a better ab-initio justification of the approach by starting from appropriate structural assumptions for the distribution $p(x,w)$ on the enlarged ambient space.", "strength_and_weaknesses": "Paper strengths:\nThe idea to enlarge the ambient space by additional variables that can be interpreted as manifold coordinates and then applying a normalising flow to these base coordinates combined with a second invertible mapping that accounts for the remaining part of input, seems to be well suited for the considered assumptions. The remaining part of the latent code can be then interpreted as independent residual noise. The price is however, that parts of the decoder are needed for the full formulation of the encoder.\n\nThe experiments show that the resulting model provides better results than VAEs with comparable encoder/decoder architectures. \n\nPaper weaknesses:\nI would expect that the model can be better justified (ab initio) by starting from appropriate assumptions about the distribution $p(x,w)$, where $w$ denotes the additional dimensions of the enlarged ambient space.\n\nIn order to learn the parameters of the feature expansion mapping by maximising likelihood, the authors propose to interpret it as a deterministic limit of a normal distribution. This and the requirement to assume distributions with learned scale for the residual part of the latent variables are in my view conceptually not fully convincing.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-25%