Diagnosing and Fixing Manifold Overfitting in Deep Generative Models
Authors: Gabriel Loaiza-Ganem
Brendan Leigh Ross
Jesse C Cresswell
Anthony L. Caterini
Venue: Accepted by TMLR ()

Abstract: Likelihood-based, or explicit, deep generative models use neural networks to construct flexible high-dimensional densities. This formulation directly contradicts the manifold hypothesis, which states that observed data lies on a low-dimensional manifold embedded in high-dimensional ambient space. In this paper we investigate the pathologies of maximum-likelihood training in the presence of this dimensionality mismatch. We formally prove that degenerate optima are achieved wherein the manifold itself is learned but not the distribution on it, a phenomenon we call manifold overfitting. We propose a class of two-step procedures consisting of a dimensionality reduction step followed by maximum-likelihood density estimation, and prove that they recover the data-generating distribution in the nonparametric regime, thus avoiding manifold overfitting. We also show that these procedures enable density estimation on the manifolds learned by implicit models, such as generative adversarial networks, hence addressing a major shortcoming of these models. Several recently proposed methods are instances of our two-step procedures; we thus unify, extend, and theoretically justify a large class of models.

Published in Transactions on Machine Learning Research (07/2022)
Diagnosing and Fixing Manifold Overfitting in Deep Generative Models
Gabriel Loaiza-Ganem Layer 6 AT
gabriel@layer6.at
Brendan Leigh Ross
brenden&layer&ee
Layer 6 AI
Jesse C. Cresswell Layer 6
jesse
Anthony L. Caterini
anthony@layer6.co at
Layer 6
Reviewed on OpenReview: https: // openreview. net/ forum? id- OnEZCVskaS
//
Abstract
Likelihood-based, or explicit, doep generative models use noural networks to construct flexible high-dimensional densities. This formulation directly contradicts the manifold hypothesis, which states that observed data lies on low-dimensional manifold embedded in high-dimensional ambient space. In this paper investigate the pathologics of maximum- likelihood training in the presence of this dimensionalityy mismatch. We formally prove that degenerate optima are achieved wherein the manifold itself is learned but not the distribution on it, a phenomenon we call manifold overfitting. We propose a class of two-step procedures consisting of a dimensionality reduction step followed by maximum- likelihood density estimation, and prove that they recover the data-generating distribution in the nonparametric regime, thus avoiding manifold overfitting. We also show that these procedures enable density estimation on the manifolds learned by implicit models, such as generative adversarial networks, hence addressing major shortcoming of these models. Several recently proposed methods are instances of our two-step procedures; we thus unify, extend, and theoretically justify a large class of models.
1
Introduction
We consider the standard setting for generative modelling, where samples [xn] C RD of high-dimensional data from some unknown distribution P* are obscrved, and the task is to estimate P Many deep generative
models (DGMs) (Bond-TTylor et al.
2021
including variational autoencoders (VAEs) (Kingma & Welling
2014 Rezende et al.] 2014 Ho et al.
2020
Kingma et al., 2021 and variants such as adversarial variational
Bayes (AVB) (Mescheder et al. [0017]
normalizing flows (NFs) Dinh et al. 2017 Kingma Dhariwal 2018
Behrmann et al. [2019] Chen al.) 2019 Durkan al. 2019 Cornish al., (2220), mneryy-baser models
[BBMs) (Du & Mordatch 2019), and continuous autoregressive models (ARMs) (Uria et al. 2013 Theis &
Bethge 2015), use neural networks to construct flexible density trained to match 1s by maximizing either
the likelihood or a lower bound of it. This modelling choice implics the model has D-dimensional support
thus directly contradicting the manifold hypothesis (Bengio et al.) 2013), which states that high- feth-dimensional data is supported on M, an unknown d-dimensional embedded submanifold of DR,, where < D.
This is indeed true of VAFs and AVB, even though both use low-dimensional latent variables, as the observational model being fully dimensional implies every point in RD is assigned strictly positive density, regardless of what latent dimension

---

Published in Transactions on Machine Learning Research (07/2022)
There is strong evidence supporting the manifold hypothesis. Theoretically, the sample complexity of kernel density estimation is known to scale exponentially with ambient dimension D when no low-dimensional structure exists (Cacoullos 1966), and with intrinsic dimension d when it does (Ozakin & Gray] 2009]. These results suggest the complexity of learning distributions scales exponentially with the intrinsic dimension of their support, and the same applies for manifold learning (Narayanan & Mitter 2010). Yet, if estimnting distributions or manifolds required exponentially many samples in D, these problems would be impossible to solve in practice. The success itself of deep-learning- based methods on these tasks thus supports the manifold hypothesis. Empirically, Pope et al. (2021) estimate d for commonly-used image datasets and find that, indeed, it is much smaller than D.
A natural question arises: how relevant is the aforementioned modelling mismatch? We answer this question by proving that when IP" is supported on M, Haniimum-ikeelhhoo training of a flexible D-dimensional density results in M itself being learned, but not P". Our result extends that of Dai & Wipf (2019) beyond VAFs to all likelihood- based models and drops the empirically unrcalistic assumption that M is homcomorphic to Rd (e.g. one can imagine the MNIST [LeCun) 1998) manifold as having 10 connected components, one per digit). This phenomenon which we call manifold overfitting has profound consequences for generative modelling. Maximum-likclihodd is indisputably one of the most important concepts in statistics, and enjoys well-studiod theoretical properties such as consistency and asymptotic efficiency under seemingly mild regularity conditions [Lhhmann & Casella 2006)- These conditions can indeed be reasonably expected to hold in the setting of "classical statistics" " under which they were first considered, where models were simpler and available data was of much lower ambient dimension than by modern standards. However, in the presence of d-dimensional manifold structure, the previously innocuous assumption that there exists ground truth D-dimensional density cannot possibly hold. Manifold overfitting thus shows that DGMs do not enjoy the supposed theoretical benefits of maximun-likeliion DGMs, because they will recover the manifold but not the distribution on it. We highlight that manifold overfitting is a problem with maximum-lkclihood itself, and thus universally affects all explicit DGMs.
which is often regarded as a principled objective for training
In order to address manifold overfitting, we propose class of two-step procedures, depicted in Fig. [ The first step, which we call gencralized autoencod- ing. reduces the dimension of the data through encoder g RDD + dd while also learning how to map back to M through a decoder G : Rd + RD. In the second step, maximum--lkkllhooo is performed on the low-dimensional representations {g( using a DGM. Intuitively, the first step removes the dimensionality mismatch in order to avoid manifold overfitting in the second step. This intuition is confirmed in a second theoretical result where we prove that, given enough capacity, our two- step procedures indeed recover 1x in the infinite data limit while retaining density evaluation. Just as man- ifold overfitting pervasively affects sitihhood-bssedd DGMs, our proposed two-step procedures address this issue in an equally broad manner. We also iden- tify DGMs that are instances of our procedure class. provides a unifying perspective and theoretical justification for all these related works.
M
g
estimation
RC
G
xx
Figure 1: Depiction of our two-step procedures. In the first step, we learn to map from M to Rd through g, and to invert this mapping through G. In the second step, we perform density estimation (green density on the right) on the dataset encoded through g. Our learned distribution on M (shades of green on the spiral) is given by pushing forward the density from the second step through G.
Our methodology thus results in novel models, and
We also show that some implicit models (Mohamed & Lakshminarayanan 2016); e.g. generative adversarial networks (GANs) (Goodfellow et al.) 20014), can be made into generalized autoencoders. Consequently, in addition to preventing manifold overfitting on explicit models, our two-step procedures enable density evaluation for implicit models, thus addressing one of their main limitations. We show that this newly obtained ability of implici models to perform density estimation can be used empirically to perform out-of-distribution
2Qur generalized autoencoders are unrelated to those Wang et
2014
2

---

Published in Transactions on Machine Learning Research (07 (07/2022)
(OOD) detection, and we obtain very promising results. To the best of our knowledge principled density istimation with implicit models was previously considered impossible.
Finally, we achieve siguificant empirical improvements in sample quality over sakimmum-likelhooooe strongly supporting our theoretical findings. We show these improvements persist even when accounting for the additional parameters of the second-step model, or when adding Gaussian noise to the data as an attempt to remove the dimensionality mismatch that causes manifold overfitting.
2 Related Work and Motivation
Manifold mismatch It has been observed in the literature that R- -s-supoorted models exhibit undesirable behaviour when the support of the target distribution has complicated topological structure. For example, Cornish et al.] (2020) show that the bi-Lipschitz constant of (opologicallymmispeeifeee NFs must go to infinity, even without dimensionality mismatch, explaining phenomena like the numerical instabilities observed by Behrmann et al. (2021). Mattei & Frellsen| (2018) observe VAEs can have unbounded likelihoods and are thus susceptible to similar instabilities. Dai & Wipf (2019) study dimensionality mismatch in VAEs and its effects on posterior collapse. These works molivate the development. of models with low-dimensional support. [Goodfelow et al. (2014) and Nowozin et al. (2016) model the data as the pushforward of a low-dimensional Gaussian through neural network, thus making it possible to properly account for the dimension of the support. However, in addition to requiring adversarial training which is more unstable than maximum-lkkllhood (Chu et al. 2020) - these models minimize the Jensen-Shannon divergence or f-divergences, respectively, in the nonporametric setting (i.e. infinite data limit with sufficient capacity), which are ill-defined due to dimensionality mismatch. Attempting to minimize Wasserstein distance has also been proposed (Arjovsky et al.] 2017 Tolstikhin et al. 2018) as way to remedy this issue, alt hough estimating this distance is hard in practice (Arora et al.) (017)] and unbiased gradient estimators are not available. In addition to having more challenging training objective than maximun-likelioodd these implicil models lose key advantage of explicit models: density evaluation. Our work aims to both properly account for the manifold hypothesis in likelihood-based DGMs while retaining density evaluation, and endow implicit. models with density evaluation.
NFs on manifolds Several recent flow-based methods properly account for the manifold structure of the data. Gemici et al.] (2016). Rezende et al. (2020), and Mathieu & Nickel (2020) construct flow models for prespecified manifolds, with the obvious disadvantage that the manifold is unknown for most data of interest. Brehmer Cranmer (2020) propose injective NFs, which model the generrtin distribution as the pushforward of a d-dimensional Gaussian through an injective function G Rd + RD_ and avoid
the change-of- variable computation through a two-step training procedure; we will see in Sec. 5 that this procedure is an instance of our methodology. Caterini et al. (2021) and Ross & Cresswell (002)) endow injective flows with tractable saangeof-vvarible computations, the former through automatic differentiation and numerical linear algebra methods, and the latter with a specific construction of injective NFs admitting closed-fone evaluation. We build a general framework encompassing a broader class of DGMs than NF's alone, giving them low-dimensional support without requiring injective transformations over Rd.
Adding noise
Denoising approaches add Gaussian noise to the data, making the D-dimensional model
appropriate at the cost of recovering a noisy version of P (Vincent et al., 2008 Vincent 2011 Alain &
Bengio 2014 Meng al. 2021 Chac et al. 2021 Horvat & Pfister 2021a b Cunningham & Fiterau 2021 b.
In particular, Horvat & Plister (202bb) show that recovering the true manifold structure in this case is only guaranteed when adding noise orthogonally to the tangent space of the manifold, which cannot be achieved in practice when the manifold itself is unknown. In the context of matching (Hyvärinen, 2005), denoising has led to empirical success (Song & Ermon 2019 Song et al.) 2021). In Scc. 3.2 wc show that adding small amounts of Gaussian noise to a distribution supported on a manifold results in highly peaked densities, which can be hard to learn. Zhang et al. (2020b) also make this observation, and propose to add the same amount of noise to the model itself. However, their method requires access to the density of the model after having added noise, which in practice requires a variational approximation and is thus only applicable to VAEs. Our first theoretical result can be seen motivation for any method based adding noise to the data (as
3

---

Published in Transactions on Machine Learning Research (07/2022)
Manifold overfitting, D 1, d
Manifold overfitting, D d
2.00
intended behaviour
manifold overfitiing
1.75
manifold overfitting
1.50
VAE
0.
1.23
1.00
pia
0.50
0.3
0.5
0.00
1,16t) (orange, dashed) for t= 5, which converges weakly to P as t + oc: and px(xx) = 0.8 -, - 1/t) (purple, dotted) for / t= 5, which converges weakly to IPt 0.86 + 0.28, while getting arbitrarily large likelihoods under IP,, ie. (x) + 00 as + oc for x E M; Gaussian VAE density (blue, solid). Right panel: Analogous phenomenon with D 2 and d = 1, with the blue density "spiking" around M in manner unlike IP" (green) while achieving large likelihoods.
Figure 2: Left panel: IP* (green); (x) = 0.3
attempting to address manifold overfitting, and our two-step procedures are applicable to all likelihood- DGMs. We empirically verify that simply adding Gaussian noise the data and fitting a maximun-likelihour DGM usual is not enough to avoid manifold overfitting in practice. Our results highlight that manifold overfitting can manifest itself empirically even when the data is close to manifold rather than exactly on one, and that naïvely adding noise does not fix it. We hope that our work will encourage further advances aiming to address manifold overfitting including ones based on adding noise.
3 Manifold Overfitting
3.1 An Illustrative Example
Consider the simple case where D 1 d= d, M- (-1,,,,, and F 0.38-1 0.78,, where 8z denotes a point mass at Suppose the data is modelled with mixture of Gaussians p(x) = X NN; m1, = + (1
A) -N(x;; m2, o²) parameterized by mixture weight X E 1J, means m1, E R, and shared variance o² E Rzo, which we will think of as a flexible density. This model can learn the correct distribution in the limit o² + O, as shown on the left panel of Fig. 2 (dashed line in orange). However, arbitrarily large likelihood values can be achieved by other densitics the one shown with a purple dotted line approximates a distribution pt on M which is not F but nonetheless has large likelihoods. The implication is simple: maximum- likelihooo estimation will not necessarily recover the data-generating distribution IP. Our choice of IPt (sec figure caption) was completely arbitrary, hence any distribution on M other than 8_1 or 6, could be recovered with likelihoods diverging to infinity. Recovering P is then coincidence which we should not expect to occur when training via maximun-likelhhodd In other words, we should expect maximum-likelinee recover the manifold (i.e. my = =1, m2 =1 and o² + 0), but not the distribution on it (i.e. À & {0.3, 0.7}). We also plot the density learned by a Gaussian VAE (see App. C.2) in blue to show this issue empirically. While this model assigns some probability outside of 1-1,1} due to limited capacity, the probabilities assigned around -1 and 1 are far off from 0.3 and 0.7, respectively; even after quantizing with the sign function, the VAE only assigns probability 0.53 to
to
The underlying issue here is that M is "too thin in DD= (it has Lebesgue measure 0), and thus p(z) can "spike to infinity" at every 2 E M. If the dimensionalitics were correctly matched this could not happen, as the requirement that p integrate to would be violated. We highlight that this issue is not only a problem with data having intrinsic dimension d = 0, and can happen whenever d < D. The right panel of Fig. 2 shows another example of this phenomenon with = and D 2, where a distribution P (green curve) is poorly approximated with a density p (blue surface) which nonetheless would achieve high likelihoods by "spiking around M Looking ahead to our experiments, the middle panel of Fig. 4 shows 2-dimensional
4

---

Published in Transactions on Machine Learning Research (07/2022)
EBM suffering from this issue, spiking around the ground truth manifold on the left panel, but not correctly recovering the distribution on it. The intuition provided by these examples is that if a flexible D-dimensional density p is trained with maximum- maximur--ikeliiood likelihood when IP* is supported on a low-dimensional manifold, it is possible to simultancously achiove large likelihoods while being close to any Pf, rather than close to P.. We refer to this phenomenon as manifold overfilling, as the density will concentrate around the manifold, but will do so in an incorrect way, recovering an arbitrary distribution on the manifold rather than the correct one. Note that the problem is not that the likelihood can be arbitrarily large (e.g. intended behaviour in Fig. 2 but that large likelihoods can be achieved while not recovering P Manifold overfitting thus calls into question the validity of maximum-likelihon as training objective in the setting where the data lies on a low-dimensional manifold.
3.2 The Manifold Overfitting Theorem
We now formalize the intuition developed so far. We assume some familiarity with measure theory (Billingseey 2008) and with smooth (Lee) 2013) and Riemannian manifolds (Lee, 2018). Nonetheless, we provide a measure theory primer in App. A where we informally review relevant concepts such as absolute continuity of measures (<), densities as Radon-Nikodym derivatives, weak convergence, properties holding almost surely with respect to a probability measure, and pushforward measures. We also use the concept of Riemannian measure (Pennec 2006), which plays an analogous role on manifolds to that of the Lebesgue measure on Euclidean spaces. We briefly review Riemannian measures in App. B.1 and refer the reader to Dieudonné 1973) for a thorough treatment [ We begin by defining a useful condition on probability distributions for the following theorems, which captures the intuition of "continuously spreading mass all around M
Definition 1 (Smoothness of Probability Measures): Let M be finite-dimensional C' manifold, and let P be a probability mcasure on M. Let g be a Ricmannian metric on M and ا the corresponding (g)
Riemannian measure. We say that IP is smooth if P < MM and it admits continuous density p M - R>0 with respect to H.M
(g)
Note that smoothness of P is independent of the choice of Ricmannian metric g (see App. B.1 We emphasize that this is weak requirement, correspouding in the Euclidean case to admilting a continuous and posilive density with respect to the Tebesgue measure, and that is not required of TP* in our first theorem below. Denoting the Lebesgue measure on RD as pD, we now state our first result.
Theorem 1 (Manifold Overfitting:: Let M c R be an analytic d-dimensional embedded submanifold of D with d < D, and pt smooth probability measure on M. Then there exists a sequence of probability measures (p)) on RD such that:
P.- + pt weakly as t co.
2. For every 2 1, P. < uD and P. admits a density pe : RD - Ryo with respect to uD such that:
(a) lim (x(x)= for every E M.
(b) lim (x))) 0 for every I & cl(M), where cl(-) denotes closure in R..
Proof sketch: We construct P. by convolving IPl with mean, DDDD covariance Gaussian noise for a sequence (o))g=, satisfying of 0 as t - oo, and then carefully verify that the stated properties of P, indeed hold. See App- B.2 for the full formal proof.
Informally, part 1 says that IP, can get arbitrarily close to pi, and part 2 says that this can be achieved with densities diverging to infinity on all M. The relevance of this statement is that large likelihoods of a model do not imply it is adequately learning the target distribution Pt, showing that maximum-likellhoon is not valid objective when data has low-dimensional manifold structure. Maximizing 1 - log p(r.), or Ex-r- [log pX)] in the nonparametrie regime, over a D-dimensional density p need not recover IP: since IP* is supported on M, it follows by Theorem 1 that not only can the objective be made arbitrarily large, but that this can be done while recovering any Pt, which need not match P.. The failure to recover 1 is caused
N
SSee especially Sec. 22 of Ch. 16. Note Riemannian measures are called Lebesgue measures this reference.
5

---

Published in Transactions on Machine Learning Research (07/2022)
by the density being able to take arbitrarily large values on all of M, thus overfitting to the manifold. When p is flexible density; as for many DGMs with universal approximation properties (Hornik] 1991 Koehler ett al. 2021), manifold overfitting becomes a key deficiency of maximuum-lkklliood
whhich we fix in Sec. 4
Note also that the proof of Theorem 1 applied to the specific case where Pt = PP" formalizes the intuition that adding small amounts of Gaussian noise to IP* results highly peaked densities, suggesting that the resulting distribution, which denoising methods aim to estimate, might be empirically difficult to learn. More generally, even if there exists ground truth D-dimensional density which allocates most of its mass around M, this density will be highly peaked. In other words, even if Theorem 1 does not technically apply in this setting, it still provides useful intuition as manifold overfitting might still happen in practice. Indeed, we empirically confirm in Sec. 6 that even if IP" is only "very close" to M, manifold overfitting remains a problem.
Differences from regular overfitting Manifold overfitting is fundamentally different from regular overfitting, At its core, regular overfitting involves memorizing observed datapoints as a direct consequence of maximizing the finite-sample objective N log=1logppzaa This memorization can happen in different ways, e.g. the empirical distribution PN - + > 6z,, could be recovered Recovering IN requires increased model capacity as N increases, as new data points have to be memorized. In contrast, manifold overfitting only requires enough capacity to concentrate mass around the manifold. Regular overfitting can happen in other ways too: a classical example (Bishop 2006) being p(x)- = (x Ip) + [N(x;; #1, ID), which achieves arbitrarily large likelihoods o² O and only requires memorizing Iл. On the other hand, manifold overfitting does not arise from memorizing datapoints, and unlike regular overfitting, can persist even when maximizing the nonparametric objective Ex-r- [log P(X)). Manifold overfitting is thus a more severe problem than regular overfitling, as it does not disappear in the infinite data regime. This property of manifold overfitting also makes detecting it more difficult: an unseen test datapoint IN+1 E M will still be assigned very high likelihood in line with the training data- under manifold overfitting, yet very low likelihood under regular overfitting Comparing train and test likelihoods is thus not valid way of detecting manifold overfitting, once again contrasting with regular overfitting, and highlighting that manifold overfitting is the more acute problem of the two.
A note on divergences Maximum-likelihodd is often thought of as minimizing the KL. divergence KL((P*) [IP) over the model distribution P.. Naïvely one might believe that this contradicts the manifold overfitting theorem, but this is not the case. In order for (L(PII[P) < oc, it is required that P < P, which does not happen when Ie is a distribution on M and F & HD. For example, KL(P'||P,) Ex-r- (X)) varies in f. In other words, minimizing the KT. divergence is not equivalent to maximizing the likelihood in the setting of dimensionality mismatch, and the manifold overfitting theorem elucidates the effect of ceniimum-ikkelhhoodd training in this setting. Similarly, other commonly considered divergences such as f-divergences cannot be meaningfully minimized. [Arjovsky et al. (2017) propose using the Wasserstein distance as it is well-defined even in the presence of support mismatch, although we highlight once again that estimating and/or /or minimizing this distance is difficult in practice.
oo, for every 2 1 even if
Non-convergence of maximum-likelhhodd The manifold overfitting theorem shows that any smooth distribution Pl on M can be recovered through mexximun-likelihood even if it does not match IP* It does not, however, guarantee that some pt will even be recovered. It is thus natural to ask whether it is possible to have a sequence of distributions achieving arbitrarily large likelihoods while not converging at all. The result below shows this to be truc: in other words, training a D-dimensional model could result in maximum-likellhood not even converging.
Corollary 1: Let M C R be an analytic d-dimensional embedded submanifold of gD with more than a single element, and d < D. Then, there exists sequence of probability measures (P,)) on RD such that:
1. (P:) does not converge weakly.
2. For every t 2 1, P. < HD and P, admits a density Pe : R + Rys with respect to aD such that:
(a) lim px = oo for every a E M. t-
4 For example, the flexible model p(x) 1/2 N , ((; with の² recovers Px-
6

---

Published in Transactions on Machine Learning Research (07 (07/2022)
(b) lim Pc(x)= 0 for every & cl(M).
Proof: Let pil and pia be two different smooth probability measures on M, which exist since M has more than single element. Let (P)) and (P?) be the corresponding sequences from Theorem 1. The sequence (P.) given by Pr = P7 if is even and P. PP otherwise, satisfies the above requirements. ]
4 Fixing Manifold Overfitting
4.1 The Two-Step Correctness Theorem
The previous section motivates the development of likelihood-bast methods which work correctly even in the presence of dimensionality mismatch. Intuitively, fixing the mismatch should be enough, which suggests (i) first reducing the dimension of the data some d-dimensional representation, and then (ii) applying maximum- likeliooo density estimation the lower-dimensional dataset. The following theorem, where pa denotes the Lebesgue measure on Rd, confirms that this intuition is correct.
Theorem 2 (Two-Step Correctness): Let M RD be Ca d-dimensional embedded submanifold of RD and let Ip* be a distribution on M. Assume there exist measurable functions G : Rd - RR and g RDD + Rd such that G(g(x)) = x, P -lmost surely. Then:
G# = IP, where h denotes the pushforward of mcasure IP through the function h.
Moreover, P is smooth, and G and g are C², then:
(a) g# F. pd-
(b) G(g(#)) =x x for every x M, and the functions g: : (MM) and G g(M) - M given by g(x) = g(x) and G(z) G(2) are diffeomorphisms and inverses of each other.
Proof: See App. B.3
We now discuss the implications of Theorem 2.
Assumptions and correctness The condition G(g(z)) = x, -l*mmtmost surely, is what one should expect obtain during the dimensionality reduction step, for example through an autoencoder (AE) (Rumelhart) et al. 1985) where (x-x=(]((((XX)) -x] minimized over G and g. provided these have enough capacity and that population-level expectations be minimized. We do highlight however that we allow for much more general class of procedures than just autoencoders, nonetheless we still refer to and G as the "encoder" Pr, justifies using first step where g reduces the dimension
and "decoder", respectively. Part 1, ( of the data, and then having a second step attempting to learn the low-dimensional distribution P.: P" a model Pz on Rd matches the encoded data distribution, i.c. Pz = IP it follows that G2P2 = P*. In other words, matching the distribution of encoded data and then decoding recovers the target distribution.
Part 2a guarantees that maximun-likeliloon can be used to learn I#I²* note that if the model Pz such that Pz < pa with density (i.e. Radon-Nikodym derivative) pz dP2z/dja, and F < pd, then both distributions are dominated by pa- Their KI. divergence can then be expressed in terms of their densities:
P'gggP"1F2 /
P2 dua. pz
P
P2 log
(1)
((M)
where pb = dg /dpa is the density of the encoded ground truth distribution. Assuming that oo, the usual decomposition of KL divergence into expected log-likelioodd and entropy applies, and it thus follows that sadttmmm--kkelihooee over pz is once again equivalent to minimizing (IL(g{P*||IPP) over Izz. In other words, learning the distribution encoded data through maximum-likelihem with a flexible density approximator such as a VAE, AVB, NF, EBM, or ARM, and then decoding the result is valid way of learning Ip* which avoids manifold overfitting.
Igcco P2 log (ogr2ddal
Density evaluation Part 2b of the two-step correctness theorem bears some resem- blance to injective NTs. However, note that the theorem does not. imply G is injec- tive: it only implies its restriction to g(M), Glg(Mys is injective (and similarly for g).
7

---

Published in Transactions on Machine Learning Research (07 (07/2022)
Fig. exemplifies how this can happen even if 9 and G are not injective. As with injective NFs, the density px of G2Pz for a model Pz on g(M) is given by the injective change-of-variale variable formula
RDD
M
g
g(M) Rd
(x pz(g(z)) det (((()))) (2)
for x E EM, where (x() E Rxd is the Jaco- bian matrix G evaluated at g(x). Practically, this observation enables density evaluation of a trained two-step model, for example for OOD detection. Implementation- wise, use the approach proposed by Caterini et al. (2021) in the context of injective NEs, which uses forward-mode automatic differentiation Baydin et al. 2018) to efficiently construct the Jacobian in 2 We highlight that, unlike Caterini et al. (2021), 1, we do not train our models through (2). Furthermore, injectivity is not enforced in G, but rather achieved at optimality of the encoder/decoder pair, and only on g(M).
Figure 3: Illustration of how and G' can biject between M (spiral) and 9(M) (linc segment) while not being fully bijective between RDD and Rd.
4.2 Generalized Autoencoders
We now explain different approaches for obtaining G and g- As previously mentioned, natural choice would be an AE minimizing Ex-P- (xxee(()((((()))) -x over G and g. However, many other choices are also valid. We call a generalized autoencoder (GAE) any procodure in which both (t) low-dimensional reprosentations Zn = ((x)) are recovered for = I,..., N, and (ii) function G is learned with the intention that G(z,) = Zn for = I.
As alternatives to an AE, some DGMs can be used as GAEs, either because they directly provide G and g or can be easily modified to do so. These methods alone might obtain G which correctly maps to M, but might not be correctly recovering IP. From the manifold overfitting theorem, this is what we should expect from likelihood- For example, the high quality of samples generated from adversarial methods (Brock et al. 2019) suggests they are indeed learning M, but issues such as mode collapse (Che et al. 2017 suggest they might not be recovering Ip* (Arbel al. 2021 Among other options (Wang al. 2020) we can use the following explicit DGMs as GAEs: (i) VAEs or (vi) AVB, using the mean of the encoder as y and the mean of the decoder G. We can also use the following implicit DGMs as GAEs: (iti) Wasserstein autoencoders (WAEs) (Tolstikhin)
models, and we argue it is not unreasonable to expect from other models as well.
ct al. 2018) or any of its follow-ups (Kolouri et al.) 2018 Patrini et al. 2020), again using the decoder G and the encoder as 9, (iv) bidirectional GANs (BiGANs) (Donahue et al.) 2017: Dumoulin et al.) 2017). taking G as the generator and as the encoder, or (v) any GAN, by fixing G as the generator and then learning g by minimizing reconstruction error Ex~r [[[G(g(X)) - X
Note that explicit construction of g can be avoided as long the representations {2, 1 are learned, which could be achieved through non-amortized models (Gershman & Goodman| 2014 Kim et al. 2018), with optimization--ase GAN inversion methods (Xia et al. 2021).
We summarize our two-step procedure class once again:
1. Learn G and {2, [= from {ax
J with a GAE.
2. Learn pz from (r with likelihood-based DGM.
The final model is then given by pushing pz forward through G. Any choice of GAE and likelihood-based DGM gives a valid instance of two-step procedure. Note that G, and g if it is also explicitly constructed, are fixed throughout the second step.
TTee density px is with respect to the Riemannian measure on M corresponding the Riemannian metric inherited from RP. This measure can be understood as the volume form on M in that integrating against them yields the same results.
8

---

Published in Transactions on Machine Learning Research (07 (07/2022) /022)
5 Towards Unifying Deep Generative Models
Making implicit models explicit As noted above, some DGMs are themselves GAEs, including some implicit models for which density evaluation is not typically available, such WAEs, BiGANs, and GANs. Ramesh & LeCun (2018) use (2) to train implicit models, but they do not train second-step DGM and thus have no mechanism to encourage trained models to satisfy the change-of-variable formula. Dieng et al. (2019) aim to provide GANs with density evaluation, but add D-dimensional Gaussian noise in order to achieve this, resulting in an adversarialllytrainee two-step correctness theorem not only fixes manifold overfitting for explicit likelihood--aase DGMs, but also enables density evaluation for these implicit models through (2) once a low-dimensional likelihood model has been trained on g(M). We highlight the relevance of training the second-step model pz for (2) to hold: even if G mapped some base distribution on R e.g. Gaussian, to F it need not be injective to achieve this, and could map distinct inputs to the same point on M (see Fig. 3 Succ a G' could be the result of training an implicit model, e.g. a GAN, which correctly learned its target distribution. Training and pz on g(M) C R2,, is still required to ensure GlgiM) is injective and (2) can be applied, even the end result of this additional training is that the target distribution remains properly learned. Endowing implicit models with density evaluation addresses a significant downside of these models, and we show in Sec. 6.3 how this newfound capability can be used for OOD detection.
explicit model, rather than truly making an implicit model explicit. The
Two-step procedures Several methods can be seen through the lens of our two-step approach, and can be interpreted as addressing manifold overfitting thanks to Theorem 2. Dai & Wipf (2019) use two-step VAE, where both the GAE and DGM are taken VAEs. Xiao et al. (2019) use standard AE along with NF. Brehmer & Cranmer 2020), and Kothari et al
2021 use AE as the GAE where G is an injective NF and its left inverse and use an NF as the DGM. Ghosh et al. (2020) use an AE with added regularizers along with a Gaussian mixture model. Rombach et al. (2022) use a VAE along with a diffusion model (Ho et al., 2020 and obtain highly competitive empirical performance, which is justified by our theoretical results.
Other methods, while not exact instances, are philosophically aligned. Razavi et al. (2019) first obtain discrete low-dimensional representations of observed data and then train an ARM on these, which is similar to discrete version of our own approach. Arbel et al. (2021) propose model which they show is equivalent to pushing forward low-dimensional EBM through G. The design of this model fits squarely into our framework, although a different training procedure is used.
(021) simultaneously optimize G, 9. and pz rather than using a two-step approach, combining in their loss a reconstruction term with a likelihood term as in ( The validity of these methods however is not guaranteed by the two-step correctness theorem, and we believe theoretical understanding of their objectives to be an interesting
The methods of Zhang et al. (2020c). ), Caterini et al. (2021). and Ross & Cresswell
direction for future work.
6 Experiments
We now experimentally validate the advantages of our proposed two-step procedures across a variety of settings. We use the nomenclature A+ to refer to the two-step model with A as its GAF and B as its DGM. All experimental details are provided in App. C including a brief summary of the losses of the individual models we consider. For all experiments on images, we set d = 20 as a hyperparameter 6 which we did not tune. We chose this value as it was close to the intrinsic dimension estimates obtained by Pope et al. (2021). b. Our code_ provides baseline implementations of all our considered GAEs and DGMs, which we hope will be useful to the community even outside of our proposed two-step methodology.
We slightly abuse notation when talking about d for given model, since here does not refer to the true intrinsic dimension anymore, but rather the dimeusion over which pz is defined (and which G maps from and g maps to), which need not match the true and unknown intrinsic dimension. htttps///gtthub... wwceaauu..oomm////aeeeeeeeeeeeee www.///aaeeriiieee labs/two. stt_wo___e_eeeeoe
9

---

Published in Transactions on Machine Learning Research (07 (0222
Ground truth
4E+EBM
1.00
100
0.75
950
0.50
0.25
0.00
0.00
0.00
25
-0.0
-0.50
-0.75
0.75
0.25
00
-1.
-2.5
0.0
-05
1.0
-1.0
0.
Figure 4: Results on simulated data: von Mises ground truth (left), EBM (middle), and AE+EBM (right).
6.1 Simulated Data
We consider a von Mises distribution on the unit circle in Fig. 4 We learn this distribution both with an EBM and a two-step AE+EBM model. While the EBM indeed concentrates mass around the circle, assigns higher density to incorrect region of it (the top, rather than the right), corroborating manifold overfitting. The AE+EBA model not only learns the manifold more accurately, it also assigns higher likelihoods to the correct part of it. We show additional results simulated data in App. D.1. where we visually confirm that the reason two-step models outperform single-step ones trained through maximum-likeliben is the data being supported on a low-dimensional manifold.
6.2 Comparisons Against Maximum-Likelibon
We now show that two-step methods empirically outperform maximum-likelilee traiming. Conve- niently, some likelihood-based DGMs recover low- dimensional representations and hence are GAEs too, providing the opportunity to compare two-step training and In particular, AVB and VAEs both maximize lower bound of the log-like loeloo,,, so can train first model as GAE, recover low-dimensiona representa- tions, and then train a second-step DGM. Any perfor- mance diffcrence compared to maximum-likllihon is then due the second-step DGM rather than the choice of GAE.
Table 1: FID scores (lower is better). Mcans + stan- dard errors across 3 runs are shown. The superscripl "4" indicates larger model, and the subscript "o" in- dicates added Gaussian noise. Unreliable FID scores are highlighted in red (see text for description).
likelihood training directly.
MODEL
MNIST
FMNIST
SVHN
CIFAR-10
AVB
219.0 4.2
235.9
4.5
356.3
289.0 3.0
AVB*
205.0 3.9
216.2
352.6 7.6
297.1
AVR/
205.2 +
223.8
353.0 7.2
305.8 + 8.7
AVB+-ARM
96.4
78.0 0.9
56.6
182.5 = 1.0
AVB+AVV
143.9 139
71.5
183.9 = 1.7
AVE+EBM
96.6
103.3
61.5 0.8
189.7 1.8
AVB+NE
51.5
55.4 0.8
181.7 0.8
AVB+VAE
2.6.
105.7
0.6
59.9 1.3
186.7 0.9
VAE
197.4
188.9 + 1.8
311.5 6.9
270.3 3.2
We show the results in Table 1 for MNIST, FMNIST (Xiao et al. 2017). SVHN (Neterr et al. 2011 and CIFAR-10 Krizhevsky 2009). We use Gaussian de- coders with learnable scalar variance for both mod- els, even for MNIST and FMNIST, as opposed to Bernoulli or other common choices (Loaiza-Ganco & Cunningham, 2019) in order properly model the data as continuous and allow for manifold over- fitting to happen. While ideally we would compare models based on log-likelihood, this is only sensible for models sharing the same dominating measure; here this is not the case as the single-step models are D-dimensional, while our two-step models are not. We thus use the FID score (Heusel et al. (017) as mcasure of how well models recover P*. Table shows training, even when adding Gaussian noise the data, thus highlighting that manifold overfitting still empirical issue even
VAE
184.0 1: 0.7
179.1 1: 0.2
300.1 1: 2.1
257.8 0.6
VAE
183.1 188
302.2
257.8 1.7
VAE-ARM ARM
69.7 +
70.9 1.0
52.9 0.3
175.2 + 1.3
AVE-AVB
11.11.0
129.6 3.1
64.0 1.3
176.7
2 2.0
VAE-EDI EBM
74.1100
78.7 2.2
63.7 3.3
181.7 -
VAE-NF
7.73
73.0 0.3
52.9 0.3
175.1
ARM
10.6
168.3
ARM!
317
136.1
AE- BE-ARM
72.01:3
76.0 - 0.3
60.1 + 3.0
186.9 1.0
EBM
81.2413
135.6 1.6
228 5.0
201 7.9
EBM
10L. 12.3
135.3 + 0.9
235.0 + 5.6
200.6 4.8
AE-EBI
7.34
83.1 +
75.2
187.4 + 3.7
that our two-step procedures consistently outperform single-step
10

---

Published in Transactions on Machine Learning Research (07/2022)
when the ground truth distribution is D-dimensional but highly peaked around manifold. We emphasize that we did not tune our two-step models, and thus the takeaway from Table should not. be about which combination of models is the best performing one, but rather how consistently two-step models outperform single-step models trained through maximum-lkclihood. We also note that some of the bascline modcls are significantly larger, e.g. the VAE+ on MNIST has approximately 824k parameters, while the VAE model has 412k, and the EAE+ only 416k. The parameter efficiency of two-step models highlights that our empirical gains are not due to increasing model capacity but rather from addressing manifold overfitting. We show in App. C.4.3 a comprehensive list of parameter counts, along with accompanying discussion.
Table 1 also shows comparisons between single and two-step models for ARMs and EBMs, which unlike AVB and VAEs, are not GAEs themselves; we thus use an AE as the GAE for these comparisons. Although FID scores did not consistently improve for these two-step models over their corresponding single-step baselines, we found the visual quality of samples was significantly better for almost all two-step models, as demonstrated in the first two columns of Fig. [ and by the additional samples shown in App. D.2 We thus highlight with red the corresponding FID scores as unreliable in Table [ We believe these failures modes of the FID metric itself, wherein the scores do not correlate with visual quality, emphasize the importance of further research on cample-bascd scalar evaluation metrics for DGMs (Borji 2022), although developing such metrics falls outside our scope. We also show comparisons using precision and recall (Kynkäänniemi et al. 2019) in App. D.4, and observe that two-step models still outperform single-step ones.
We also point out that one-step EBMs exhibited training difficulties consistent with maximun-likellhood con-convergence (App. D.3. Meanwhile, Langevin dynamics (Welling & Teh 2011] for AE+EBM exhibit better and faster convergence, yielding good samples even when not initialized from the training buffer (sce Fig. 13 in App. D.3 and AE+ARM speeds up sampling over the baseline ARM by factor of O(D/d) in both cases because there are fewer coordinates in the sample space. All of the 44 two-step models shown in Table | visually outperformed their single-step counterparts (App. D.2 empirically corroborating our thcoretical findings.
Finally, we have omitted some comparisons verified in prior work: Dai & Wipl (2019) show VAE+VAR outperforms VAE, and Xiao et al.] (2019) that AE+NF outperforms NF. We also include some preliminary experiments where we attempted to improve upon GAN's gencrative performance high resolution images in App. D.5 We used an oplinization-based GAN inversion method, but found the reconstruction errors were too large to enable empirical improvements from adding a second-step model.
6.3 OOD Detection with Implicit Models
Having verified that, predicted by Theorem 2, two-step models outperform -ikelihood training, we now turn our attention to the other consequence of this theorem, namely endowing implicit models with density evaluation after training a second-step DGM. We demonstrate that our approach advances fully-unsupervised likelihood-based out-of-distribution detection. Nalisnick et al. (2019) discovered the counter-intuitive phenomenon that likelihood- based DGMs sometimes assign higher likelihoods to OOD data than to in-distribution data. In particular, they found models trained on FMNIST and CIFAR-10 assigned higher likelihoods to MNIST and SVHN, respectively. While there has been a significant amount of research trying to remedy and explain this situation (Choi et al.) 2018 Ren et al. 2019 Zisselman &
Tamar 2020 Zhang et al., 2020al Kirichenko et al.) 2020 Le Lan & Dinh 2020 Caterini & Loaiza- Loaiza-Ganem 2021), there is little work achieving good OOD performance using only likelihoods of models trained in a full-unsupervised way to recover pa rather than explicitly trained for OOD detection. Caterini et. al. (0221 achieve improvements in this regard, although their method remains computationally expensive and has issues scaling (e.g. no results are reported on the CIFAR-10 + SVHN task).
We train several two-step models where the GAE is either a BiGAN a WAF, which do not by themselves
allow for likelihood evaluation, and then use the resulting log-likelihoods (or lower hounds/negation energy
functions) for OOD detection. Two-step models allow us to use either the high-dimensional log px from (2)
or low- dinennsinii log as metrics for this task. We conjecture that the latter is more reliable, since (i)
the base mcasure is always Md, and (it) the encoder-lecoder is unlikely to exactly satisfy the conditions of
Theorem 2. Hence, we use log here, and show results for logpx px in App. D.6
11

---

Published in Transactions on Machine Learning Research (07/202) 2022)
V
2
O
O 7
5
<
2
O
2
4
8 9
2
2
9
0
9
2
7
7
7
3
7
2
R
3
O
8
5
2
3
O
8
5
7
S
7
9
Z
9
O
3
5
6
2
O
7
4
e
4 3
3 S
9
8
5
3
5
4
Figure 5: Uncurated samples from single-step models (first row, showing ARM EBM*, AVB and VAE) and their respective two-step counterparts (second row, showing AE+ARM, AE+EBM, AVB+NE, and AVB)AVB), for MNIST (first column), FMNIST (second column), SVHN (third column), and CIFAR-10 (fourth column).
Table 2 shows the (balanced) classification accuracy of a decision stump given only the log- likelihood; we show some corresponding histograms in App. D.6 The stump is forced to assign large likelihoods as in- distribution, so that accuracies below 50% indicate it incorreclly assigned higher likelihoods to OOD data. We correct the classification accuracy to account for datascts of different size (dctails in App. D.6), result- ing in an easily interpretable metric which can be understood as the expected classification accuracy if two samo-sized samples of in-distribution and OOD data were compared. Not only did we enable im- plicit models perform OOD detection, but we also outperformed likelihood- based single-step models in this setting. To the best of our knowledge, no other model achieves nearly 50% (balanced) accuracy on CIPAR-I--SVHH using only likelihoods. Although admittedly the problem is not yet solved, we have certainly made progress on a challenging task for fully-unsupervised methods.
Table 2: OOD (balanced) classification accuracy percentage (higher is better). Means = standard errors across 3 runs are shown. Arrows point from
in-distribution to OOD data.
MODEL
FMNIST + MNIST CIFAR-10 - SVHN
ARM
9.9 + 0.6
15.5 0.0
BIGAN+ARM
81.9 1.4
3.2 40.1
WAE+ARM
69.8 + 13.9
AVB*
96.0
23.4 0.1
BIGAN+AVE
59.5 3.1
36.4
WVE+AVB
90.7 1.7
43.5 13.51..9 1.9
EBM+
1.1
46.4
BIGAN +EBM
5.22
48.8 0.11
WAE+EBI
57.2
49.3
NF
3.24
18.6
BiGAN+NF
842
0.2
WAE+NF
95.4
46.1
VAE*
96.1
23.8
BIGAN
59.7
0.1
WAE+VAE
92.5 2.7
41.4 0.2
For completeness, we show samples from these models in App. D.2 and FID scores in App- D.4 Implicit models see less improvement in FID from adding second-step DGM than explicit models, suggesting that manifold overfitting is less dire problem for implicit models. Nonetheless, we do observe some improvements, particularly for BiGANs, hinting that our two-step methodology not only endows these models with density evaluation, but that it can also improve their generative performance. We further show in App. D.6 that. OOD improvements obtained by two-step models apply to explicit models as well.
Interestingly, whereas the VAEs used in Nalisnick al. (2019) have Bernoulli likelihoods, we find that our single-step likelihood-based Gaussian-decoder VAE and AVB models perform quite well distinguishing
12

---

Published in Transactions on Machine Learning Research (07/2022)
FMNIST from MNIST, yet still fail on the CIFAR-10 task. Studying this is of future interest but is outside the scope of this work.
7 Conclusions, Scope, and Limitations
In this paper we diagnosed manifold overfitting, a fundamental problem of maximum-likeliooe training with flexible densities when the data lives in a low-dimensional manifold. We proposed to fix manifold overfitting with a class of two-step procedures which remedy the issuc, theoretically justify a large group of existing methods, and endow implicit models with density evaluation after training a low-dimensional likelihood-basse DGM on encoded data.
Our two-step correctness thcorem remains nonetheless nonparametric result. In practice, the reconstruction error will be positive, i.c. Ex~==[]] [I)(((())((()))))))))))))))))))) - > O. Note that this can happen even when assuming infinite capacity, as M needs to be diffeomorphic to g(M) for some C function g TRD - Rd for the reconstruction error to be O, We leave a study of learnable topologies of M for future work. The density in 2 might then not be valid, cither if the reconstruction error is positive, or if pz assigns positive probability outside of (MM. However, we note that our approach least provides mechanism to encourage our trained encoder-decoder pair to invert each other, suggesting that (2) might not be too far off. We also believe that sinite-sample extension of our result, while challenging, would be relevant direction for future work. We hope our work will encourage follow-up research exploring different ways of addressing manifold overfitting, or its interaction with the score-matching objective.
Finally, we treated d as a hyperparameter, but in practice is unknown and improvements can likely be had by estimating (Levina & Bickel 2004). as overspecifying it should not fully remove manifold overfitting, and underspecifying would make learning M mathematically impossible. Still, we observed significant empirical improvements across variety of tasks and datasets, demonstrating that manifold overfitting is not just theoretical issue in DGMs, and that two-step methods are an important class of procedures to deal with it.
Broader Impact Statement
Generative modelling has numerous applications besides image generation, including but not limited to:
audio generation (van den Oord et al. 2016a Engel et al.
2017). biology (Lopez et al.) 2020). chemistry
(Gómez-Bombaral) et al. 2018), compression (Townsond et al. 2019 Ho et al. 2019; Golinski & Caterini
2021:
Yang et al.] 2022), 9, genetics (Riesselman el al.
2018),
neuroscience (Sussillo el al. 2016 Gao el al,
2016
Canza--Ganem et al. 2019), physics (Otten al.
2021
Padmanabha & Zabaras 2021), ) text generation
(Bowman et al. 2016 Devlin et al.
2019 Brown et al.
2020), text-to-image generation (Zhang et al. 2017
Ramesh et al. 2022 Saharia et al.
2022
video generation (Vondrick al. 2016, Weissenborn et al.) 2020
and weather lorecasting (Ravuri et al.
2021). While each of these applications can have positive impacts on
society, it is also possible to apply deep generative models inappropriately, or create negative socictal impacts through their use (Brundage et al.] 2018 Urbina et al., 2022). When datasets are biased, accurate generative models will inherit those biases (Steed & Caliskan 2021 Humayun et al. 2022). Inaccurate generative models may introduce new biases not reflected the data. Our paper addresses a ubiquitous problem generative modelling with maximum likelihood estimation manifold overfitting that causes models to fail learn the dist of data correctly. In this sense, correcting manifold overfitting should lead to more accurate generative models, and representations that more closely reflect the data.
Acknowledgments
We thank the anonymous reviewers whose suggestions helped improved our work. In particular, we thank anonymous reviewer Cev4, as well as Taiga Abc, both of whom pointed out the mixture of two Gaussians regular overfitting example from [Bihhop| 2006), which was lacking from a previous version of our manuscript.
We wrote our code in Python (Van Rossum & Drake 2009), and specifically relied on the following packages:
Matplotlib (Hunter 2007), TensorFlow qAbadi et al. 2015) (particularly for TensorBoard) Jupyter Notebook
Kluyver et 2016), PyTorch (Paszke et al. 2019 nflows (Durkan et al.) 2020), NumPy (Harris al.
2020 prdc (Naeem et al. 2020), pytorch-fid Seitzer 2020 and functorch (He & Zou 2021
13

---

Published in Transactions on Machine Learning Research (07 (0222)
References
Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dandelion Mané, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals, Pele Warden, Martin Watlenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. TensorFlow: Large-scale machine learning on heterogeneous systems, 2015. URL https: //www tensorflow.org. Software available from tensorflow.org.
Guillaume Alain and Yoshua Bengio. What regularized auto-encoders learn from the data-geng distribution. The Journal of Machine Learning Research, 15(1):3565 0393,,5333- 2014.
Michael Arbel, Liang Zhou, and Arthur Gretton. Generalized energy based models. TCLR, 2021.
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In International conference on machine learning, pp. 214 223. PMLR, 2017.
Sanjeev Arora, Rong Ge, Yingyu Liang, Tengyu Ma, and Yi Zhang, Generalization and equilibrium in generative adversarial nets (gans). In International Conference on Machine Learning, pp. PMLR, 2017.
Shane Barratt and Rishi Sharma. A note on the inception score. ar preprint
2018.
Atalm Günes Baydin, Barak A Pearlmutter, Alexey Andreyevich Radul, and Jeffrey Mark Siskind. Automatic differentiation in machine learning: survey. Journal of Machine Learning Research, 18:1-43, 2018.
Jens Behrmann, Will Grathwohl, Ricky TQ Chen, David Duvenaud, and Jörn-Henrik Jacobsen. Invertible residual networks. In International Conference Machine Learning, pp. 573 582. PMLR, 2019.
Jens Behrmann, Paul VIcol, Kuan-Chieh Wang, Roger Grosse, and Jörn-Henrik Jacobsen. Understanding and mitigating exploding inverses in invertible neural networks. In International Conference on Artificial Intelligence and Statistics, pp. 1802-1800. PMLR, 2021.
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Reprosentation learning: A review and new perspectives. IEEE transactions on pallern analysis and machine intelligence, 35(8):1798 1828, 2013.
Patrick Billingsley. Probubility and measure. John Wiley & Sons, 2008.
Christopher M. Bishop. Pattern Recogration and Machine Learning. Springer: New York, 2006.
S Boud-Taylor, Leach, Y Long, and CG Willcocks. Deep generative modelling: A comparative review of VAEs, GANs, normalizing flows, energy-based and autoregressive models. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021.
Ali Borji. Pros and cons of gan evaluation measures: New developments. Computer Vision and Image Understanding, 215:103329, 2022.
Samuel R Bowman, Luke Vilnis, Oriol Vinyals, Andrew M Dai, Rafal Jozefowicz, and Samy Bengio. Generating sentences from a continuous space. In 20th SIGNLL Conference on Computational Natural Language Learning, CoNLL 2016, pp. 10-21. 21. Association for Computational Linguistics (ACL), 2016.
Johann Brehmer and Kyle Cranmer. Flows for simultaneous manifold learning and density estimation. In Advances Newral Information Processing Systems, volume 33, 2020.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. TCLR, 2019.
14

---

Published in Transactions on Machine Learning Research (07/2022)
Tom Brown, Benjamin Mann, Nick Ryder, Melanic Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877 1901,, 2020.
Miles Brundage, Shahar Avin, Jack Clark, Helen Toner, Peter Eckersley, Ben Garfinkel, Allan Dafoe, Paul Scharre, Thomas Zeitzoff, Bobby Filar, et al. The malicious use of artificial intelligence: Forecasting, prevention, and mitigation. arXiv preprint are: 1802 07228, 2018.
Theophilos Cacoullos. Estimation of a multivariate densily. Annals of the Institule of Statislionl Mathemalics, 18(1):179-1535) 189, 1966.
Anthony L Caterini and Gabriel Loaiza-Gant Ganem. Eutropic Issues in Likelihood-Basee OOD Detection. ar.ivv preprint (kki:2119.110999 2021.
Authony L Caterini, Gabriel Loaiza- Ganem, Geoff Pleiss, and John P Cunningham. Rectangular flows for manifold learning. In Advances Neural Information Processing Systems, volume 34, 2021.
Minwoo Chae, Dongha Kim, Yongdai Kim, and Lizhen Lin. A likelihood approach to nonparametric estimation of singular distribution using deep generative models. arXiv preprint arXiv:2105.044 04046, 2021.
Tong Che, Yanran Li, Athul Paul Jacob, Yoshua Bengio, and Wenjie Li. Mode regularized generative adversarial networks. ICLR, 2017.
Ricky T. Q. Chen, Jens Behrmann, David K Duvenaud, and Joern-Henria Jacobsen. Residual flows for invertible generative modeling, In Advances in Newral Information Processing Systems, volume 32, 2019.
Hyunsum Choi, Eric Jang, and Alexander A Alemi. WAIC, but why? Generative ensembles for robust anomaly detection. arXiv preprint arXhv:1810.01392, 2018.
Casey Chu, Kentaro Minami, and Kenji Fukumizu. Smoothness and stability in GANs. ICLR, 2020.
Djork-Arné Clevert, Thomas Unterthiner, and Sepp Hochreiter. Fast and accurate deep network learning by exponential linear units (elus). ICLR, 2016.
Rob Cornish, Anthony Caterini, George Deligiannidis, and Arnaud Doucet. Relaxing bijectivity constraints with continuously indexed normalising flows. Tn International Conference on Machine Learning, pp- 2133 2143. PMLR, 2020.
Edmond Cunningham and Madalina Fiterau. A change of variables method for rectangular matrix-vector products. In Proceedings of The 24th International Conference Artificial Intelligence and Statistics, volume 130. PMLR, 2021.
Bin Dai and David Wipf. Diagnosing and enhancing VAE models. ICLR, 2019.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pp. 4171-4186, 2019.
Adji B Dieng, Francisco JR Ruiz, David M Blei, and Michalis K Titsias. Prescribed generative adversarial networks. arXiv preprinl ar Xix::1990 Xxi::191004432, 2019.
Jean Dieudonné. Treatise on Analysis: Volume III. Academic Press, 1973.
Laurent. Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using Real NVP. ICLR, 2017.
Jef Donahue, Philipp Krähenbühl, and Trevor Darrell. Adversarial feature learning. ICLR, 2017.
Yihun Du and Igor Mordatch. Implicit generation and modeling with energy based models. Aduances in Newral Information Processing Systems, 32:3608 3618, 2019.
15

---

Published in Transactions on Machine Learning Research (07/2022)
Vincent Dumoulin, Ishmael Belghazi, Ben Poole, Olivier Mastropietro, Alex Lamb, Martin Arjovsky, and Aaron Courville. Adversarially learned inference. TCLR, 2017.
Conor Durkan, Artur Bekasov, lain Murray, and George Papamakarios. Neural Spline Flows. In Advances in Newral Information Processing Systems, volume 32, 2019.
Conor Durkan, Artur Bekasov, lain Murray, and George Papamakarios. nllows: normalizing flows PyTorch, November 2020. URL httpp:://oo...rrt http:/10..2111eeee1 42962877
Jesse Engel, Cinjon Resnick, Adam Roberts, Sander Dieleman, Mohammad Norouzi, Douglas Eck, and Karen Simonyan. Neural audio synthesis of musical notes with wavenet autocncoders. In International Conference on Machine Learning, pp. 1068 1077. PMLR, 2017.
Yuanjun Gao, Evan W Archer, Liam Paninski, and John P Cunningham. Lincar dynamical neural population models through nonlinear embeddings. Advances in neural information processing systems, 29, 2016.
Mevlana C Gemici, Danilo Rezende, and Shakir Mohamed. Normalizing flows riemannian manifolds. arXiv preprinl ar Xix: 1. .02304, 2016.
Samuel Gershman and Noah Goodman. Amortized inference in probabilistic reasoning. In Proceedings of the annual meeting of the cognitive science sociely, volume 36, 2014.
Partha Ghosh, Mehdi SM Sajjadi, Antonio Vergari, Michael Black, and Beruhard Schölkopf. From variational to deterministic autoencoders. ICLR, 2020.
Adam Golinski and Anthony L Caterini. Lossless compression using contimuously-indexxed normalizing flows. In Neural Compression: From Information Theory to Applications Applcations-Wrrksooggg @ ICLR 2021, 2021.
Rafael GómezeBonbarelli, Jennifer N Wei, David Duvenaud, José Miguel Hernández-Lobato, Benjamín Sánchez-Lengeling, Dennis Sheberla, Jorge aguilera-Iparraguire Timothy D Hirzel, Ryan P Adams, and Alán Aspuru-Guzik. Automatic chemical design using a data-driven continuous representation of molecules. ACS central science, 4(2):268 270, 2018
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
Alfred Gray. The volume of a small geodesic ball of riemannian manifold. The Michigan Mathematical Journal, 20(4):329-344, 1974.
Arthur Gretton, Karsten M Borgwardt, Malte J Rasch, Bernhard Schölkopf, and Alexander Smola. kernel two-sample test. Journal of Machine Learning Research, 13:723-777 2012.
Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville. Improved training of wasserstein gans. In Advances in Neural Information Processing Systems, volume 30, 2017.
Charles R. Harris, K. Jarrod Millman, Stéfan J. van der Walt, Ralf Gommers, Pauli Virtanen, David Cournapeau, Eric Wieser, Julian Taylor, Sebastian Berg, Nathaniel J. Smith, Robert Kern, Matti Picus, Stephan Hoyer, Marten H. van Kerkwijk, Matthew Brett, Allan Haldane, Jaime Fernández del Río, Mark Wiebe, Pearu Peterson, Pierre Gerard-Marchantt Kevin Sheppard, Tyler Reddy, Warren Weckesser, Hameer Abbasi, Christoph Gohlke, and Travis E. Oliphant. Array programming with NumPy. Nature (85(7)25):357 362, September 2020. doi: 1111088/441866-00 2649-2. URL https:///oil http::///bo..orr/l0..1000 941586-020-26490
Horace He and Richard Zou. functorch: Jax-like composable function transforms for pytorch. https
2021.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs trained by a two time-scale update rule converge to a local nash equilibrium. In Advances in Neural Information Processing Systems, volume 30, 2017.
16

---

Published in Transactions on Machine Learning Research (07/2022)
Jonathan Ho, Evan Lohn, and Picter Abbeel. Compression with flows via local bits- coding. Advances in Newral Information Processing Systems, 32, 2019.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems, volume 33, pp. 6840 6851, 2020.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8):1735-1788) 1997.
Kurt Hornik. Approximation capabilities of multilayer feodforward networks. Neural networks, 4(2):251-257, 1991.
Christian Horvat and Jean-Pascal Pfister. Denoising normalizing flow. Advances in Neural Information Processing Systems, 34, 2021a.
Christian Horvat and Jean-Pascal Pfister. Density estimation on low-dimensional manifolds: an inflation- deflation approach. arXiv preprint arXiv:2105.12152, 2021b.
Minyoung Huh, Richard Zhang, Yun Zhu, Sylvain Paris, and Aaron Hertzmann. Transforming and projecting images into class-conditional generative networks. In European Conference Computer Vision, pp. 34-34. Springer, 2020.
Ahmed Imtiaz Humayun, Randall Balestriero, and Richard Baraniuk. Magnet: Uniform sampling from deep generative network manifolds without retraining. ICLR, 2022.
J. D. Hunter. Matplothib: A 2d graphics environment. Computing in Science & Engincering, 9(3):90-99) -5, 2007. doi:
Aapo Hyvärinen. Estimation of non-normalized statistical models by score matching. Journal of Machine Learning Research, 6(4), 2005.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 4401-4410, 2019.
Tero Karras, Milka Ailtala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Training generative adversarial networks with limited data. In Advances in Neural Information Processing Systems, 2020a.
Tero Karras, Samuli Laine, Milk Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila, Analyzing and improving the image quality of StyleGAN. In Proc. CVPR, 2020b.
Yoon Kim, Sam Wiseman, Andrew Miller, David Sontag, and Alexander Rush. Semi-amortized variational autoencoders. In International Conference on Machine Learning, pp. 2678 2687. PMLR, 2018.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. ICLR, 2015.
Diederik P Kingma and Prafulla Dhariwal. Glow: Generative Flow with Invertible 1x Convolutions. In Advances in Neural Information Processing Systems, volume 31, 2018.
Diederik P Kingma and Max Welling. Auto-encoding Variational Bayes. ICLR, 2014.
Diederik P Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models. In Advances in neural information processing systems, volume 34, 2021.
Polina Kirichenko, Pavel Izmailov, and Andrew G Wilson. Why normalizing flows fail to detect out-of- distribution data. Advances in neural information processing systems, 33:20578 20589, 2020.
17

---

Published in Transactions on Machine Learning Research (07/2022)
Thomas Kluyver, Benjamin Ragan-Kelley, Fernando Pérez, Brian Granger, Matthias Bussonnier, Jonathan Frederic, Kyle Kelley, Jessica Hamrick, Jason Grout, Sylvain Corlay, Paul Ivanov, Damián Avila, Safia Abdalla, and Carol Willing. Jupyter notebooks a publishing format for reproducible computational workflows. In F. Loizides and B. Schmidt (eds.), Positioning and Power in Academic Publishing: Players, Agents and Agendas, pp. 87 90. IOS Press, 2016.
Frederic Koehler, Viraj Mehta, and Andrej Risteski. Representational aspects of depth and conditioning in normalizing flows. In International Conference on Machine Learning, pp. 5628 5636. PMLR, 2021.
Soheil Kolouri, Phillip E Pope, Charles E Martin, and Gustavo K Rohde. Sliced wasserstein auto-encoders. ICLRR 2018.
Konik Kothari, AmirEhsan Khorashadizadeh, Maarten de Hoop, and Ivan Dokmanić. Trumpets: Injective flows for inference and inverse problems. In Proceedings of the "heeety-eeeetth Conference on Uncertainty in Artificial Intelligence, volume 161, pp. 1269-1278, 2021.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Master's thesis, University of Toronto, 2009.
Tuomas Kynkäänniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Improved precision and recall metric for assessing generative models. Advances in Neural Information Processing Systems, 32, 2019.
Charline Le Lan and Laurent Dinh. Perfect density models cannot guarantee anomaly detection. In "I Can't Believe It's Not Better!"NeurlP" 2020 workshop, 2020.
Y LeCun. The MNIST database of handwritten digits, 1998. URL http://wwwwcn.ccom/eeee
1998.
John M Lee. Smooth manifolds. In Introduction to Smooth Manifolds, pp. 31. Springer, 2013.
John M Lec. Introduction to Riemannian manifolds. Springer, 2018
Erich L Lehmann and George Casella. Theory of point estimation. Springer Science & Business Media, 2006.
Elizaveta Levina and Peter Bickel. Maximum likelihood estimation intrinsic dimension. Advances in newral information processing systems, 17, 2004.
Gabriel Loaiza-Ganem and John P Cunningham. The continuous bernoulli: fixing pervasive error in variational autoencoders. Advances in Neural Information Processing Systems, 32:13287 13297, 2019.
Gabriel Loaiza-Ganena Sean Perkins, Karen Schroeder, Mark Churchland, and John P Cunningham. Deep random splines for point process intensity estimation of neural population data. Advances in Newral Information Processing Systems, 32, 2019.
Romain Lopez, Adam Gayoso, and Nir Yosef. Enhancing scientific discoveries in molecular biology with deep generative models. Molecular Systems Biology, 16(9):99198, 2020.
Emile Mathieu and Maximilian Nickel. Riemannian continuous normalizing flows. Advances in Newral Information Processing Systems, volume 33, 2020.
Pierre-Alexandre Mattei and Jes Frellsen. Leveraging the exact likelihood of deep latent variable models. Advances in Neural Information Processing Systems, 31, 2018.
Chenlin Meng, Jiaming Song, Yang Song, Shengjia Zhao, and Stefano Ermon. Improved autoregressive modeling with distribution smoothing. ICLR, 2021.
Lars Mescheder, Sebastian Nowozin, and Andreas Geiger. Adversarial Variational Bayes:
Unifying variational autocncoders and generative adversarial networks. In International Conference on Machine Learning, pp. 2391-2400. PMLR, 2017.
Shakir Mohamed and Balaji Lakshminarayanan. Learning in implicit generative models. ar preprint arXiv:1610.0344 2016.
18

---

Published in Transactions on Machine Learning Research (07/2022)
Muhammad Ferjad Nacem, Scong Joon Oh, Youngjung Uh Yunjey Choi, and Jaejun Yoo. Reliable fidelity and diversity metrics for generative models. In International Conference on Machine Learning, pp. 1176--185, PMLR, 2020.
Eric Nalisnick, Akihiro Matsukawa, Yee Whye Tch, Dilan Gorur, and Balaji Lakshminarayanan. Do deep generative models know what they don't know? ICLR, 2019.
Hariharan Narayanan and Sanjoy Mitter. Sample complexity of testing the manifold hypothesis. Advances in nearal informalion processing systems, 23, 2010.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y. Ng. Reading digits in natural images with unsupervised feature learning. In NTPS Workshop on Deep Learning and Unsupercised Feature Learning 2011, 2011. URL http://ufldl. stanford, eetuhhouseeuuberrr//pp111111
housenumbers pdff
Sebastian Nowozin, Botond Cseke, and Ryota Tomioka. F-GAN: Training generative neural samplers using variational divergence minimization. In Proceedings of the 30th International Conference on Neural Information Processing Systems, pp. 271 279, 2016.
Sydney Otten, Sascha Caron, Wieske de Swart, Melissa van Beekveld, Lue Hendriks, Caspar van Leeuwen, Damian Podareanu, Roberto Ruiz de Austri, and Rob Verheyen. Event generation and statistical sampling for physics with deep generative models and a density information buffer. Nature communications, 12(1): 1-16, 2021.
Arkadas Ozakin and Alexander Gray. Submanifold density estimation. Advances in Neural Information Processing Systems, 22, 2009.
Govinda Anantha Padmanabha and Nicholas Zabaras. Solving inverse problems using conditional invertible neural networks. Journal of Computational Physics, 433:110194, 2021.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32:8026-800 8037, 2019.
Giorgio Patrini, Rianne van den Berg, Patrick Forre, Marcello Carioni, Samarth Bhargav, Max Welling, Tim Genewein, and Frank Nielsen. Sinkhorn autoencoders. In Uncertainty in Artificial Intelligence, pp. 733-743. PMLR, 2020.
Xavier Pennec. Intrinsic statistics on riemannian manifolds: Basic tools for geometric measurements. Journal of Mathematical Imaging and Vision, 25(1):127-15) 154, 2006.
Phillip Pope, Chen Zhu, Ahmed Abdelkader, Micah Goldblum, and Tom Goldstein. The intrinsic dimension of images and its impact on learning. ICLR, 2021.
Prajit Ramachandran, Barret Zoph, and Quoc V Le. Searching for activation functions. Xxix:1710.050) 2017.
arXiv preprint
Aditya Ramesh and Yann LeCun. Backpropagation for implicit spectral densities. arXiv preprint arxiv: 1ao:1806.0000 906.00499, 2018.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical lext-conditional image generation with clip latents. arXiv preprint arxiv:2204.000 06122, 2022.
Suman Ravuri, Karel Lene, Matthew Willson, Dmitry Kangin, Remi Lam, Piotr Mirowski, Megan Fitzsimons, Maria Athanassiadon, Shelcem Kashem, Sam Madge, et al. Skilful precipitation nowcasting using deep generalive models of radar. Nature, 597(7878)6672 677, 2021.
Ali Razavi, Aaron van den Oord, and Oriol Vinyals. Generating diverse high fideliiyy images with VQ- VAE-2. 2019.
Tn Advances in neural information processing systems, pp.
19

---

Published in Transactions on Machine Learning Research (07/2022)
Jie Ren, Peter J Liu, Emily Fertig, Jasper Snock, Ryan Poplin, Mark Depristo, Joshua Dillon, and Balaji Takshminarayanan. Likelihood ratios for out-of-distribution detection. In Advances in Newral Information Processing Systems, volume 32, pp. 14701, 2019.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in doep generative models. In International conference on machine learning, pp. 1278- 1286.
PMLR, 2014.
Danilo Jimenez Rezende, George Papamakarios, Sébastien Racaniere, Michael Albergo, Gurtej Kanwar, Phiala Shanahan, and Kyle Cranmer. Normalizing flows on tori and spheres. In International Conference on Machine Learning, pp. 8083-8092, PMLR, 2020.
Adam J Riesselman, John B Ingraham, and Debora S Marks. Deep generative models of genetic variation capture the effects of mutations. Nature methods, 15(10):816 822, 2018.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEFE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10684 10695, 2022.
Brendan Leigh Ross and Jesse C Cresswell. Tractable density estimation learned manifolds with conformal embedding flows. In Advances in Newral Information Processing Systems, volume 34, 2021.
David E Rumelhart, GeofTrey E Hinton, and Ronald Williams. Learning internal representations by error propagation. Technical report, California Univ San Diego La Jolla Inst for Cognitive Science, 1985.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Bureu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, et al. Photorcalistic text-to- image diffusion models with deep language understanding, arXiv preprinl ar Xrx 1 11487, 2022.
Mchdi SM Sajjadi, Olivier Bachem, Mario Lucic, Olivier Bousquet, and Sylvain Gelly. Assessing generative models via precision and recall. Advances in Neural Information Processing Systems, 31, 2018.
Tim Salimans, Tan Goodfellow, Wojeiech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. Advances in neural information processing systems, 29, 2016.
Maximilian Seitzer. pytorch-fid: FID Score for PyTorch. http::///gaab..cc.../aaatttttttr August 2020. Vorsion 0.2.1.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. In Proceedings of the 33rd Annuol Conference Neural Information Processing Systems, 2019.
Yang Song, Jascha Dick-sicks,,in, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In ICLR, 2021.
Ryan Steed and Aylin Caliskan. Image representations learned with unsupervised pre-training contain human-like biases. In Proccedings of the 2021 ACM conference fairness, accountability, and transparency, pp. 701-713, 2021.
David Sussillo, Rafal Jozelowicz, LF Abbolt, and Chethan Pandarinath. Lfads-latent factor analysis via dynamical systems. arXiv preprint arXiv:160s.0633 2016.
Lucas Theis and Matthias Bethge. Generative image modeling using spatial LSTMs. Advances in Neural Information Processing Systems, 28:1927-1935, 2015.
llya Tolstikhin, Olivier Bousquet, Sylvain Gelly, and Bernhard Schoelkopf. Wasserstein auto-encoders. ICLR, 2018
James Townsend, Tom Bird, and David Barber. Practical lossless compression with latent variables using bits back coding. ICIR, 2019.
20

---

Published in Transactions on Machine Learning Research (07/2022)
Fabio Urbina, Filippa Lentzos, Cédric Invernizzi, and Sean Ekins. Dual use of artificial innfffcc--ntelllgnnee-pwrrdddddd drug discovery. Nature Machine Intelligence, ((3):189-199) 191, 2022.
Benigno Uria, lain Murray, and Hugo Larochelle. Roade: The real-valued neural autoregressive density-
estimator. In In Advances in Neural Information Processing Systems 26 (NIPS 26), 2013.
Aäron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Scnior, and Koray Kavukcnoglu. Wavenet: A generative model for raw andio. arXic preprinl 160: 03499, 2016a.
Aäron den Oord, Nal Kalchbrenner, and Koray Kavukcuoglu. Pixel recurrent neural networks. In Proceedings of the 33rd Internalional Conference International Conference on Machine Learning- Volume 48, pp. 7747-1756 2016b.
Guido Van Rossum and Fred L. Drake. Python 3 Reference Manual. CreateSpace, Scotts Valley, CA, 2009. ISBN 1441412697.
Pascal Vincent. A connection between score matching and denoising autoencoders. Neural computation, 23 (7):1661 1674, 2011.
Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre- Antoine Manzagol. Extracting and composing robust features with denoising autoencoders. In Proceedings of the 25th internalional conference on Machine learning, pp. 1096-10 1103, 2008.
Carl Vondrick, Hamed Pirsiavash, and Antonio Torralba. Generating videos with scene dynamics. Advances in neural information processing systems, 29, 2016.
Wei Wang, Yan Huang, Yizhou Wang, and Liang Wang. Generalized autoencoder: neural network framework for dimensionality reduction. In Proceedings of the IEEE conference on computer vision and pollern recognilion workshops, pp. 490 497, 2014.
Yingfan Wang, Haiyang Huang, Cynthia Rudin, and Yaron Shaposhnik. Understanding how dimension reduction tools work: An empirical approach to deciphering t-SNE, UMAP, TriMAP, and PaCMAP for data visualization. arXiv preprint orXio22012.04550 2020.
Dirk Weissenborn, Oscar Täckström, and Jakob Uszkoreit. Scaling antoregressive video models. ICLR, 2020.
Max Welling and Yee W Teh. Bayesian learning via stochastic gradient langevin dynamics. In Proceedings of the 28th international conference on machine learning (ICML-11), pp. 681 688. Citeseer, 2011.
Weihao Xia, Yulun Zhang, Yujiu Yang, Jing-Hao Xue, Bolei Zhou, and Ming- Hsuan Yang. GAN inversion: A survey. Xrxiv preprint arXiv:2101.05272 2021.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-MNIST: a novel image datasct for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.07743 2017.
Zhisheng Xiao, Qing Yan, and Yali Amit. Generative latent flow. arXiv preprint arXiv: 2019.
Yibo Yang, Stephan Mandt, and Lucas Theis. An introduction neural data compression. arXiv preprint arXiv:2202.000 06533, 2022.
Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaogang Wang, Xiaolei Huang, and Dimitris N Metaxas. Stackgan: Text to photo-realistic image synthesis with stacked generative adversarial networks. In Proceedings of the IEEE international conference on computer vision, pp. 5907-5915, 2017.
Hongjie Zhang, Ang Li, Jie Guo, and Yanwen Guo. Hybrid models for open set recognition. In European Conference on Computer Vision, pp. 102-117. Springer, 2020a.
Mingtian Zhang, Peter Hayes, Thomas Bird, Raza Habib, and David Barber. Spread divergence. In International Conference on Machine Learning, pp. 11106-11166 PMLR, 2020b.
21

---

Published in Transactions on Machine Learning Research (07/2022)
Zijun Zhang, Ruixiang Zhang, Zongpeng Li, Yoshua Bengio, and Liam Paull. Perceptual generative autoen- coders. In International Conference on Machine Learning, pp. 11298-111 11306. PMT.R, 2020c.
Ev Zisselman and Aviv Tamar. Deep residual flow for out of distribution detection. In Proceedings of the IEFE/CVF Conference on Computer Vision and Pattern Recognation, pp- 13994-14003, 2020.
22

---

Published in Transactions on Machine Learning Research (07/2022)
A Informal Measure Theory Primer
Before stating Theorems and 2, and studying their implications, we provide a brief tutorial on some aspects of measure theory that are relevant to follow our discussion. This review is not meant to be comprehensive, and we prioritize intuition over formalism. Readers interested in the topic may consult textbooks such Billingsley (2008).
A.1 Probability Measures
Let us first motivate the need for measure theory in the first place and consider the question: what is a densily? Intuitively, the density px of a random variable X is a function having the property that integrating px over any set A gives back the probability that X E A. This density characterizes the distribution of X, in that it can be used to answer any probabilistic question about X. It is common knowledge that discrete random variables are not specified through density, but rather a probability mass function. Similarly, in our setting, where X might always take values in M, such density will not exist. To see this, consider the case where A = M,, so that the integral of px over M would have to be 1, which cannot happen since M has volume 0 in IP (or more formally, Lebesgue measure 0). Measure theory provides the tools necessary to properly specify any distribution, subsuming as special cases probability mass functions, densities of continuous random variables, and distributions on manifolds.
A measure μ on Rº is a function mapping subsets A C CR to Rzo, obeying the following properties: (1) ((A) 2 0 for every A, (ii) ((()= O, where 0 denotes the empty set, and (iiz) (( Ax) ( ((Ax) for any sequence of pairwise disjoint sets A2... (i.c. A, n A, = ( whenever i * j). Note that most measures of interest are only defined over a large class of subsets of RD (called o-algebras, the most notable one being the Borel o-algebra) rather than for every possible subset due to technical reasons, but we omit details in the interest of better conveying intuition. A mcasure is called a probability mcasure if it also satisfics (( = To any random variable X corresponds a probability measure μx, having the property that (x(A) is the probability that X E A for any A. Analogously to probability mass functions random variables, μx allows us to answer any probabilistic question about X. The probability measure μx is often called the distribution or law of X. Throughout our paper, Pe" is the distribution from which we observe data.
densities of continuous
Let us consider two examples to show how probability mass functions and densities continuous random variables are really just specifying distributions. Given ai....,0p OK E R., consider the probability mass function of a random variable X given by px(x) 1/K for x a1, .2,,..,, OK and otherwise. This probability mass function is simply specilying the distribution μx (x(A) = 1/K Ek= 1(as E A), where (( E A) denotes the indicator function for A, i.e. Ia E A) is if E A, and 0 otherwise. Now consider a standard Gaussian random variable X in R with density px(x) = 0(z;0,1)) Similarly to how the probability mass function from the previous example characterized a distribution, this density does well through px (A) SA A (xx;; 0, Dd).. We will see in the next section how these ideas can be extended to distributions on manifolds.
The concept of integrating a function h D.R + R with respect to a measure μ on RD is fundamental in measure theory, and can be thought of as "weighting the inputs of h according to n² In the case of the Lebesgue mcasure PD (which assigns to subscts A of RDD their "volume" (D())), integration extends the concept of Riemann integrals commonly taught in calculus courses, and in the case of random variables integration defines expectations, i.e. Ex [h(X)] hdpx. In the next section we will talk about the interplay between integration and densities.
We finish this sontion by explaining the relevant concept of a property holding almost surely with respect to a measure p. A property is said to hold p-almost surely if the set A over which it does not hold is such that p(A)= = 0. For example, if μ the distribution of a standard Gaussian random variable X in RDD, then we can say that X + holds μ almost surely, since (x()))) = 0. The assumption that G(g(z)) = x, Plmoomsst surely in Theorem 2 thus means that. Pe ((x RR.: G(g(z)) # x)) = 0.
23

---

Published in Transactions on Machine Learning Research (07/ 2202)
A.2 Absolute Continuity
So far we have seen that probability measures allow us to talk about distributions in full generality, and that probability mass functions and densities of continuous random variables can be used to specify probability measures. A distribution on a manifold M embedded in Rº can simply be thought of as a probability measure 1l such that μuMM) = 1. We would like to define densities on manifolds in an analogous way to probability mass functions and densities of continuous random variables, in such way that they allow us to characterize distributions on the manifold. Absolute continuity of measures is a concept that allows to formalize the concept of density with respect to dominating measure, and encompasses probability mass functions, densities of continuous random variables, and also allows us to define densities on manifolds. We will see that our intuitive definition of a density as function which, when integrated over a set gives back its probability, is in fact correct, just as long as we specify the measure we integrate with respect to.
Given two measures j and v, we say that μ is absolutely continuous with respect to V for every A such that v(4) = 0, also holds that ((A) O. If μ is absolutely continuous with respect to V, we also say that v dominates μt, and denote this property as μ & v. The Radon-Nikodyın theorem states that, under some mild assumptions on μ and v which hold for all the measures considered in this paper, / < implies the existence of a function h such that p(A) = SA hdv for every A. This result provides the means to formally define densities: h is called the density or Radon-Nikodym derivative of μ with respect to and is often written as du//dv.
Before explaining how this machinery allows us to talk about densities on manifolds, we first continue our examples to show that probability mass functions and densities of continuous random variables are Radon- derivatives with respect to appropriate measures. Tet us reconsider the example where (x( = 1/K for the measure v(A) = Ex 1(ax E A), which essentially just counts the number of ass in A. Clearly px < v, and so it follows that μa admits a density with respect to v. This density turns out to be px, since /x(A) SA pxdv. In other words, the probability mass function px can be thought of a Radon-Nikodym derivative, i.e. Px dux/dv. Let now go back to the continuous density example where (x(x) = N2; 0, ID) and μx is given by the Riemann integral (x
a1, 02 .2,...,,a0 ak and 0 otherwise, and (x(A) = 1/K Σ 1aa E A). Consider
SA N(x; 0, LD)dr. In this case, V == HD. and since the Lebesgue integral extends the Riemann integral, it follows that (x = =fa Pxduu, so that the density px is ually also a density in the formal sense of being a Radon-Nikodym derivative, so that px = =dux/dup
We can thus see that the formal concept of density or Radon-Nikodym derivative generalizes both probability mass functions and densities of continuous random variables as we usually think of them, allowing to specify distributions in a general way.
The concept of Radon-Nikodyı derivative also allows us to obtain densities manifolds, the only missing ingredient being dominating measure the manifold. Riemannian measures (App. B.1] play this role on manifolds, in the same way that the Lebesgue measure plays the usual role of dominating measure to define densities of contimous random variables on D
A.3 Weak Convergence
A key point in Theorem 1 is weak convergence of the sequence of probability measures (P.) to Pi. The intuitive interpretation that this statement simply means that "P, converges to pi" is correct, although formally defining convergence of a sequence of measures is still required. Weak convergence provides such a definition, and IP, is said to converge weakly to pe if the sequence of scalars Pr(A) converges to (A) for every satisfying a technical condition (for intuitive purposes, one can think this property as holding for every A). In this sense weak convergence is a very natural way of defining convergence of measures: in the limit, P. will assign the same probability to every set as Pt.
A.4 Pushforward Measures
We have seen that to a random variable X in D corresponds a distribution px- Applying a function h RD + ad to X will result in a new random variable, h(X) in Rd, and it is natural to ask what its distribution is. This distribution is called the pushforward measure of px through h, which is denoted
24

---

Published in Transactions on Machine Learning Research (07 (022)
/#/xx, and is defined as (B) = μx (B)) for every subset B of R A way to intuitively understand this concept is that if one could sample X from μx, then sampling from hg can be done by simply applying & to X. Note that here / is measure on Rd
The concept of pushforward measure relevant in Theorem as it allows us to formally reason about e.g. the distribution of encoded data, I# 9gP* Similarly, for distribution Pz corresponding to our second-step model, we can reason about the distribution obtained after decoding, i.e. GyPzz
B Proofs
B.1 Riemannian Measures
We begin with a quick review of a Riemannian measures. Let M be a d-dimensional Riemannian manifold with Riemannian metric and let (U, Φ) be chart. The local Riemannian mcasure PM. on M (with its Borel o-algebra) is given by:
(g)
-OCAnes V det (o (o(1))
a à
(M
djd
(3)
(g) for any measurable A C M. The Riemannian measure PM on M is such that:
(0) (AnU) (M
(4)
for every measurable A M and every chart (U,
If 9i and 02 are two Riemannian metrics on M, then HM positive density with respect to PM
p and PM admits a continuous and [82] Thus, mentioned in the main manuscript, smoothness of probability
measures is indeed independent of the choice of Riemannian metric.
Below we prove a lemma which we will later use, showing that much like the Lebesgue measure, Riemannian measures assign positive measure to nonempty open sets. While we are sure this is a known property, we could not find a proof and thus provide one.
Lemma 1: Let M be a d-dimensional Riemannian manifold, and p a Riemannian measure on it. Let A C M be nonempty open set M. Then 11(0) (A) >
Proof: Let (U) be chart such that Un / 0, which exists because A o 0. Clearly n A is open, and since & is diffeomorphism onto its image, it follows that @(U n A) Rd is also open and nonempty, and thus Pa( n A)) > 0. As result,
J V det (0)
))
p O (((
قेه des
ddda > O,
(5)
where the last inequality follows since the integrand is positive and the integration set has positive measure.
B.2 Manifold Overfitting Theorem
We restate the manifold overfitting theorem below for convenience:
Theorem 1 (Manifold Overfitting:: Let M c R be analytic d-dimensional embedded submanifold of D with d < D, and ptt smooth probability measure on M. Then there exists sequence of probability measures (Pr) on RDD such that:
1. P, + IPt weakly as f +
25

---

Published in Transactions on Machine Learning Research (07/2022)
2. For every 2 Pi ap and Pa admits a density pe: RP Rss with respect an such that:
(a) lim Pr(x) for every M. too
(b) lim (r(z) = for every I & cl(M), where cl(-) denotes closure in R..
100
Before proving the theorem, note that p is distribution M and Pt is distribution Rº, with their respective Borel o-algebras. Weak convergence is defined for measures on the same probability space, and so we slightly abuse notation and think of pl as a measure on R assigning to any measurable set A C R the probability Pf (AAn M), which well-defined as M is an embedded submanifold of RP. We do not differentiate between pt on M and pt on TR to avoid cumbersome notation.
Let Y be a random variable whose law IP,, and let (Ze) be a sequence of i.idd standard Gaussians in R?, independent of Y. We assume all the variables are defined the same probability space (Ω, F.P. Let X, Y + o,Zr where (or) is positive sequence converging O. Let Pt be the law of Xt-
Proof:
First we prove 1. Clearly o + 0 in probability and Y Y in distribution as t + 00. Since o,Z converges in probability to a constant, it follows that X, + Y in distribution, and thus P. - pt weakly.
Now we prove that P. < uD. Let A RD be a measurable set such that (A()) = 0. We denote the law of oi G2 and the Gaussian density Rº with mean m and covariance matrix E evaluated at Nww; m, Σ). Let B ((,,y) RD x M: + y A). By Fubini's theorem:
Pt(A) =PV + 0Z E A) = / dG. x pt (w, v) / N(w N(w;;,oflo)dup x (w,
|B
(6)
= -/ N(x - y;0, dup x (x, f /. NNxx v:0, d
(7)
AXM
- Odp (y)= 0.
(8)
Then, P. < unD proving the first part of 2. Note also that:
(x(x) = / N(x o:)
(9)
is valid density for P, with respect to μD, once again by Fubini's theorem since, for any measurable set A C R
S (x)) (())d)u(()))) = / N(x ";
dPl 1
dun(z)
(10)
S N(x y: 0,. o (' = P((A).
(11)
AXM
We now prove 2a. Since pf being smooth is independent of the choice of Riemannian measure, assume without loss of generality that the Riemannian metric M is the metric inherited from thinking of M as a submanifold of D and we can then take a continuous and positive density p with respect to the Riemannian measure PM associated with this metric.
(e)
26

---

Published in Transactions on Machine Learning Research (07/2022)
Take M and let (x) (v E M: de (z, v) S denote the geodesic ball M of radius centered
at F, where d'M is the geodesic distance. We then have:
p((x)= = / N(x v;0,o dPt 2
f N(x - -;;,0,) dP)
(12)
M
(B)(a)
| BM(
(N)) - y; oinn) du (v) (1)
J inf p p'//)N(o op (v) (13)
JBMG)) (
HM
(0) (BM
inf
p'(y')((x - 02
(14)
y'ERM (a))
(0) 2 1M (B (x))
inf (E
N(x - 0,
inf p (1/). (ERM
(15)
Since Ba (x) is compact in M for small enough or and p is continuous M and positive, it follows that. inf, y (x) pl (v/) is bounded away from as t + cc. It is then enough to show that 00,
(o) (B (z))
inf N(x o':0, + (a)
(16)
in order to prove that 2a holds. Let (0) denote an Le ball of radius r in Rd centered at 0 E R and let denote the Lebesgue measure on Rd, that Pa (0) Car where Ca > 0 is a constant depending only on d. is known that " (B,M (x)) = (B) (1 (rr)) for analytic d-dimensional Riemannian manifolds Grray 1974,, and thus:
(0) M (z)
inf
N(r - yJ; 0, of
y ('E
1
|| - 3'12
= Caff (1+ (1+
y'E (x) 0. (/2 2/2
inf
exp
207
(17)
sup (EEM((a)
|| - 1/1
C1 (/x) (1 + 0(of))
of d-D
exp
(18)
20
The first term is positive constant, and the second term converges to 1. The third term goes to infinity since d < D, which leaves only the last term. Thus, as long as the last term is bounded away from O as t - oc, we can be certain that the product of all four term goes to infinity. In particular, verifying the following equation would be enough:
sup v' (E)
1|2 v < of.
(19)
This equation holds, since for any z.y E M, it is the case that a| V'1l2 d.0) v/ g is inherited from M being a submanifold of R
Now we prove 2b for Pr. Let DERD c(M). We have:
Po(x) / N(r y: 0. 1.0,0I10 of (v) < I sup N(x - 1/:0,00 dpt dp)()) sup N(x - y/: 02 0,) (20) WEM
JM
M
inf WEM
||
y'1l3
sup y'EM (2m exp
| v'llé 20
of (2m) D/2
1
exp
207
1000 0,
(21)
where convergence follows from & cl(M) implying that infyem || v
27

---

Published in Transactions on Machine Learning Research (07/ (222))
B.3 Two-Ster Step Correctness Theorem
We restate the two-step correctness theorem below for convenience:
Theorem 2 (Two-Step Correctness): Let M C RDD be a Ca d-dimensional embedded submanifold of RD
and let IP" be a distribution on M. Assume there exist measurable functions : 12 + DR and g IR - Rd such that G(g(z)) = v; P*-almost surely. Then:
1. C#(0+IP) IP" where by P denotes the pushforward of measure IP through the function h.
2. Moreover, if P is smooth, and G and g are C22 then:
(a) f*P < pd
(b) G(g(z)) = x for every x E M, and the functions ğ: M -MM-an ((M) and &: g(M) - M given by
g(x) = g and G(z) G(z) are diffeomorphisms and inverses of each other.
Similarly to the manifold overfitting theorem, we think of IPt a distribution on RP, assigning to any Borel set A CCR. R the probability P (AnM), which once again is well-defined since M is an embedded submanifold of R.
Proof: We start with part 1. Let A= {a E RD: G(g(x)) + z}, which is a mull set under P by assumption. By applying the definition of pushforward measure twice, for any measurable set B S M:
G4(g*P) =
(F0 B)))
(22)
= =I' = A)
A)))
(23)
(24)
where we used that 9 (GG B)) S A, and thus G P*) = P Note that this derivation requires thinking of T as measure RD to ensure that A and g (G- B)) can be assigned 0 probability.
We now prove 2b. We begin by showing that G(g(z)) x for all with the product topology. Clearly RD x M is Hausdorff since both RD and M are Hausdorff (M is Hausdorff by the definition of a manifold). Let E E= {(x, x) E RD x : E MM. which is then closed in R . (since diagonals of Hausdorff spaces are closed). Consider the function H .M + +R" x M given by H(x) (G(g(z)), 2), which is clearly continuous. It follows that H H='EE) = {x E M : G(g(z)) = #} is closed in M,, and thus M (E) = {a E M : G(g(z)) # x) is open in M, and by assumption PP (M \ H (E)) = O. It follows by Lemma 1 in App. B.1 that M \ H HE) = 0, and thus G(g(z)) = for all x E M.
M.. Consider RD M endowed
We now prove that ğ is a diffeomorphism. Clearly ÿ is surjective, and since it admits left inverse (namely G), it is also injective. Then ğ is bijective, and since it is clcarly C¹ duc to g being C'l and M being an embedded submanifold of RD,, only remains to show that its inverse is also C². Since G(g(z)) = x for every x E M, it follows that G(g(M)) = M, and thus & is well-defined (i.e. the image of its domain is indeed contained in its codomain). Clearly G is a left inverse g, and by bijectivity of g, it follows G is its inverse. Finally, Ō is also C'l since G is C²,, so that ğ is indeed diffeomorphism.
Now, we prove 2a. Let K C Rd be such that (a(K)= O. We need to show that 9# Px (K) = 0 in order to complete the proof. We have that:
g*
-P= PP
P" (1-(((())
(25)
Let g be a Riemannian metric on M. Since F.+ < < by assumption, enough to show that ( (K)) M) = O. Let {Ua}a be an open (in M) cover of 9 (K)n n.. Since M is second countable by definition, by Lindelöf's lemma there exists countable subcover (Va)seN- Since M is diffeomorphism onto its image,
28

---

Published in Transactions on Machine Learning Research (07 (07/2022)
(Va, 2lvg) is chart for every B E N. We have:
(U+
nM) = MM (k) C
1 BEN
(26)
SEN
2 Σ
HM (K)
(27)
SEN
det
8
dua 0,
(28)
BEN slv, (K)nMMV)))))
0glivs
dgliv
where the final equality follows from glv, (K) (M) S K for every and pa(K) O.
C Experimental Details
C.1 Model Losses
Throughout this section we use L to denote the loss of different models. We use notation that assumnes all of these are first-step models, ie. datapoints denoted In, but highlight that when trained second-step models, the datapoints actually correspond to Similarly, whenever a loss includes D, this should be understood as d for second-step models. The description of these losses here is meant only for reference, and we recommend that any reader unfamiliar with these see the relevant. citations in the main manuscript. Unlike our main manuscript, thesure- notation is not needed to describe these models, and we thus drop it.
A.E As mentioned in the main manuscript, we train autoencoders with squared reconstruction error:
G(g,G)= N Σ
[G(g(z,)) - Fn 113.
(29)
ARMs
The loss of autoregressive models is given by the negative log- lieliikelioood:
c(p)
1 N
Σ
logg(En,1) + Σ pl #nmm-1)
(30)
where Fn,m denotes the m coordinate of Fn-
AVB Adversarial variational Bayes is highly related to VAEs (sce description below), except the approximate posterior is defined implicitly, so that a sample U from q(-lz) can be obtained as = ğ(x, e), where ~ pe()), which is often taken a standard Gaussian of dimension d, and ğ RR x Rd - Fdd Since g(-lar) cannot be evaluated, the ELBO used to train VAEs becomes intractable, and thus a discriminator T RRD x Rd + R is introduced, and for fixed q(-1z), trained to minimize:
N ((T)=>>
(Eurg( (log (T U))] EU- (logg ( U)))))
(31)
where s(-) denotes the sigmoid function. Denoting the optimal 7 as T*, the rest of the model components are trained through:
N E Eungt (In,
L(G,ox) =
(2n, U'
(32)
N
29

---

Published in Transactions on Machine Learning Research (07 (07/2022)
where PUz) depends on G and ox in identical way as in VAEs (see below). Analogously to VAEs, this training procedure maximizes lower bound on the log-likelihood, which is tight when the approximate posterior matches the true one. Finally, 2n can either be taken as:
In (U- [U] Beupe (+) (1 }] Or
in = 0).
(33)
We use the former, and approximate the expectation through a Monte Carlo average. Note that both options define g through ÿ such a way that 2n = g(xn). Finally, line with Goodfellow et al. (2014), we found that using the "log trick" to avoid saturation in the adversarial loss further improved performance.
BiGAN Bidirectional GANs model the data as X = =(),, where 7 pz, and Pz is taken as d- dimensional standard Gaussian. Note that this pz is different from pz in the main manuscript (which corresponds to the densily of the second-step model), hence why we use different notation. BiGANs also aim to recover the žn corresponding to each In, and so also use an encoder g, in addition to a discriminator T DRD P Rd R. All the components are trained through the following objective:
N
(:T)= EZ- (
Σ ((xn)) 1=1
(34)
N
which is minimized with respect to g and G, but maximized with respect to T. We highlight that this objective is slightly different than the originally proposed BiGAN objective, as we use the Wasserstein loss In practice we penalize the gradient of T
[Arjovsyy et al. 2017) instead of the original as is often done for the Wasserstein objective (Gulrajani et al. 2017). We also found that adding a square reconstruction error as an additional regularization term helped improve performance.
EBM Energy-based models energy function E: E R, which implicitly defines a density R
B(x)
p
(35)
Eix' Bld" (x)
These models attempt minimize the negative log- likelihood:
((F) N
Σ log p(x.),
(36)
which is seemingly intractable due to the integral in 35 However, when parameterixing E with 0 as neural network Eg, gradients of this loss can be obtained thanks to the following identity:
- log px(,)) (xn)) Ex-ro (x)),
(37)
where we have also made the dependence of p on explicit. While it might seem that the expectation in (37 is just as intractable as the integral (35), in practice approximate samples from pa are obtained through Langevin dynamics and are used approximate this expectation.
NFs Normalizing flows use a bijective neural network h RDD - RDD along with base density Pu on DR,, often taken as standard Gaussian, and model the data X = =hU),, where U ~ pv- Thanks to the change-of-variable formula, the density of the model can be evaluated:
p(x) Pu (h
(h- (x)) det Jn (x)..
(38)
and flows can thus be trained via maximun-likelihood.
L(h) Σ Pu (
+ log det Jn- (xn)))
(39)
In practice h constructed in such a way that not only ensures it is bijective, but also ensures that log det. Jn-1 (æ.)) can be efficiently computed.
30

---

Published in Transactions on Machine Learning Research (07/2022)
VAEs Variational autoencoders define the generative process for the data U pu: XIU p(- Typically, pu is a standard d-dimensional Gaussian (although learnable prior can also be used), and in our case, p(-lu) is given by a Gaussian:
p(zuu) (2; (())).)(()I))
(40)
where ox R + I is neural network. Maximum-likelihoon is intractable since the latent variables un corresponding to In are unobserved, so instead an approximate posterior g(u[2) is introduced. We take to be Gaussian:
duk=) = (N diag(of
(41)
where of : RD - RR is a neural network, and diag(of denotes a diagonal matrix whose nonzero elements are given by (z). An objective called the negative ELBO then minimized:
N (LL(g(I Bur-ot [logp(zn(U)))
Clg,G,ou, N
(42)
The ELBO can be shown to be a lower bound to the log-likelihood, which becomes light as the approximate posterior matches the true posterior. Note that 2n corresponds to the mean of the unobserved latent un
= Ea = g(xn).
(3)
We highlight once again that the notation we use here corresponds to VAEs when used as first-step models. When used as second-step models, as previously mentioned, the observed datapoint &n becomes on, but in this case the encoder and decoder functions do not correspond to g and G anymore. Similarly, for second-step models, the unobscrved variables u, bocome "irrelovant" in terms of the main contents of our paper, and are not related to 2n in the same way in first-step models. For second-step models, we keep the latent dimension d still.
WAEs Wasserstein autoencoders, similarly to BiGANs, model the data as X GZZ), where Z pz, which is taken as d-dimensional standard Gaussian, and use a discriminator T Rd - IR. The WAE objective is given by:
Glg.GiT) 1 Σ (IG(g(Z,)) 2, + Alog(1 - s(T(((x.))))) + AEz log (T())). N
Σ
(44)
where s(-) denotes the sigmoid function, À > 0 is hyperparameter, and the objective is minimized with respect to and G, and maximized with respect to T. Just as in AVB, we found that using the "log trick" of Goodfellow et al. (2044) in the adversarial loss further improved performance.
C.2 VAE from Fig.
We generated N = 1000 samples from Fx 0 times. The Gaussian VAE had d 1, 1, D = 1, and both the encoder and decoder have a single hidden layer with 25 units and ReLU activations. We use the Adam optimizer (Kingma & Ba 2015) with learning rate 0.001 and train for 200 epochs. We use gradient norm clipping with a value of 10.
+ 0.76,, resulting in a dataset containing a total of 693
C.3 Simulated Data
For the ground truth, we use von Mises distribution with parameter x = 1, and transform to Cartesian coordinates to obtain a distribution on the unit circle in RDD = R2_ We generate N = 1000 samples from this distribution. For the EBM model, we use an energy function with two hidden layers 25 units each and Swish activations (Ramachandran et al. 2017). We use the Adam optimizer with learning rate 0.01, and gradient norm clipping with value of 1. We train for 100 epochs. We follow Du & Mordatch (2019) for the training of the FBM, and use 0.1 for the objective regularization value, iterate Langevin dynamics for 60
31

---

Published in Transactions on Machine Learning Research (07 (07/2022)
iterations at every training step, step size of 10 within Langevin dynamics, sample new images with probability 0.05 in the bulfer, use Gaussian noise with standard deviation 0.005 in Langevin dynamics, and truncate gradients to (-0.03, 0.03) in Langevin dynamics. For the AE+EBM model, we use an AE with d d=1 1 and two hidden layers of 20 units each with ELU activations (Clevert et al. 2016). We use the Adam optimizer with learning rate 0.001 and train for 200 epochs. We use gradient norm clipping with a value of 10. For the EBM of this model, we use an energy function with two hidden layers of 15 units cach, and all the other parameters are identical to the single step EBM. We observed some variability with respect to the seed for both the EBM and the AE+EBM models; the manuscript shows the best performing versions.
For the additional results of Sec. D.1 the ground truth is given by a Gaussian whose first coordinate has mcan O and variance 2, while the second coordinate has mean 1 and variance 1, and they have a covariance of 0.5. The VAEs are identical to those from Fig. 2 except their input and output dimensions change accordingly.
C.4 Comparisons Against Maximum-Likelihood and OOD Detection with Implicit Models
For all experiments, we use the Adam optimizer, typically with learning rate 0.001. For all experiments we also clip gradient entries larger than 10 during optimization. We also set = 20 in all experiments.
C.4.1 Single and First-Step Models
For all single and first-step models, unless specified otherwise, pre-process the data by scaling it, ie. dividing by the maximum absolute value entry. All convolutions have a kernel size of 3 and stride 1. For all versions with added Gaussian noise, we tried standard deviation values o E {1, 0.1, 0.01, 0.001, 0.0001} and kept the best performing one (a = 0.1, as measured by FID) unless otherwise specified.
AEs For MNIST and FMNIST, we use MLPs for the encoder and decoder, with ReLU activations. The encoder and decoder have each single hidden layer with 256 units. For SVHN and CIFAR-10, use convolutional networks. The encoder and decoder have 4 convolutional layers with (32, 32, 16, 16) and (16, 16, 32) channels, respectively, followed by a flattening operation and a fully-connected layer. The convolutional networks also use ReLU activations, and have kernel size 3 and stride 1. We perform early stopping on reconstruction error with a patience of 10 epochs, for a maximum of 100 epochs.
ARMs We use an updated version of RNADE (Uria et al.] 2013), where we use an LSTM (Hochreiter & Schmidhuber 1997] to improve performance. More specifically, every pixel is processed sequentially through the LSTM, and a given pixel is modelled with a mixture of Gaussians whose paramcters are given by transforming the hidden state obtained from all the previous pixels through a linear layer. The dimension of a pixel is given by the number of channels, so that MNIST and FMNIST use mixtures of 1-dimensional Gaussians, whereas SVHN and CIFAR-10 use mixtures of 3-dimensional Gaussians. We also tried a continuous version of the PixcICNN model (van den Oord et al. 2016b), where we replaced the discrete distribution over pixels with mixture of Gaussians, but found this model highly unstable which is once again consistent. with manifold overfitting and thus opted for the LSTM-based model. We used 10 components for the Gaussian mixtures, and used an LSTM with 2 layers and hidden states of size 256. We train for maximum of 100 epochs, and use early stopping on log-likeling lig-iikel with a patience of 10. We also use cosine annealing on the learning rate. For the version with added Gaussian noise, we used a = 1.0. We observed some instabilities in training these single step models, particularly when not adding noise, where the final model was much worse than average (over 100 difference in FID score). We treated these runs as failed runs and excluded them from the averages and standard errors reported in our paper.
We use the exact same configuration for the encoder and decoder as in AEs, and use an MLP with 2 hidden layers of size 256 each for the discriminator, which also uses ReLU activations. We train the MLPs for a maximum of 50 epochs, and CNNs for 100 epochs, using cosine annealing on the learning rates, For the large version, AVB+, we use two hidden layers of 256 units for the encoder and docoder MLPs, and increase the encoder and decoder number of hidden channels to (64, 32, 32) and (32, 32, 64, 64), respectively, for convolutional networks. In all cases, the encoder takes in 256-dimensional Gaussian noise with covariance
AVB
32

---

Published in Transactions on Machine Learning Research (07/2022)
9 I.ID. We also tried having the decoder output per-pixel variances, but found this parameterization to be numerically unstable, which is again consistent with manifold overfitting.
BiGAN As mentioned in Sec. C.1 we used a Wasserstein (W-GAN) objective (Arjovsky et al. 2017 with gradient penalties (Gulrajani et al., 2017) where both the data and latents are interpolated between the real and generated samples. The gradient penalty weight was 10. The generator-encoder loss includes the W-GAN loss, and the reconstruction loss (joint. latent regressor from Donahue et al. (2017)) equally weighted. For both small and large versions, we use the exact same configuration for the encoder, decoder, and discriminator as for AVB. We used learning rates of 0.0001 with cosine anncaling over 200 epochs. The discriminator was trained for two steps for every step laken with the encoder/decotee
EBMs For MNIST and FMNIST, our energy functions use MLPs with two hidden layers with 256 and 128 units, respectively. For SVHN and CIFAR-10, the energy functions have 4 convolutional layers with hidden channols (64, 64, 32, 32). We use the Swish activation function and spectral normalization in all cases. We set the energy function's 's output regularization coefficient to and the learning rate to 0.0003. Otherwise, we use the same hyperparameters as on the simulated data. At the beginning of training, we scale all the data between 0 and We train for 100 epochs without early stopping, which tended to halt training too early.
We use a rational quadratic spline flow (Durkan et al. 2019 with 128 hidden units, layers, and blocks per layer. We train using early stopping on validation loss with a patience of 30 epochs, up to a maximum of 100 epochs. We use a learning rate of 0.0005, and use whitening transform at the start of training to make the data zero-mean and marginally unit-variance, whenever possible (some pixels, particularly in MNIST, were only value throughout the entire training set); note that this affine transformation does not affect the manifold structure of the data.
NFs
The settings for VAFs were largely identical to those of AVB, except we did not do early stopping and always trained for 100 epochs, in addition to not needing a discriminator. For large models a single hidden layer of 512 units was used for each of the encoder and decoder MLPs. We also tried the same decoder per-pixel variance parameterization that we attempted with AVB and obtained similar numerical instabilities, once again in line with manifold overfitting.
VAEs
WAEs We use the adversarial variant rather than the maximum mean discrepancy (Gretton et al. 2012) one. We weight the adversarial loss with coefficient of 10. The settings for WAEs were identical to those of AVB, except (i) we used a patience of 30 epochs, trained for maximum of 300 epochs, (it) we used no learning rate schoduling, with a discriminator learning rate of 2.5 x 10-4 and an encoder-decoder learning rate of 5 x 10-4, and (iti) we used only convolutional encoders and deeooder, with (64, 64, 32, 32) and (32, 32, 64, 64) hidden channels, respectively. For large models the number of hidden channels was increased to (96, 96, 48, 48) and (48, 48, 96, 96) for the encoder and decoder, respectively.
C.4.2 Second-Stee Step Models
All second-step models, unless otherwise specified, pre-process the encoded data by standardizing it (i.e. subtracting the mean and dividing by the standard deviation).
ARMs We used the same configuration for second-step ARMs as for the first-step version, except the LSTM has a single hidden layer with hidden states of size 128.
AVB We used the same configuration for second-step AVB as did for the first-step MLP version of AVB, except that we do not do early stopping and train for 100 epochs. The latent dimension is set to d (i.e. 20).
EBMs We used the same configuration that we used for single-step EBMs, except we use a learning rate of 0.001, we regularize the energy function's output by 0.1, do not use spectral normalization, take the energy function to have two hidden layers with (64,32) units, and scale the data between -1 and 1.
33

---

Published in Transactions on Machine Learning Research (07/2022)
NFs We used the same settings for second-step NEs as we did for first-step NEs, except (i) use 64 hidden units, (ii) we do not do early stopping, training for maximum of 100 epochs, and (iit) we use learning rate of 0.001.
VAEs We used the same settings for second-step VAEs as we did for first-step VAEs. The latent dimension is also set to (i.e. 20).
C.4.3 Parameter Counts
Table 3 includes parameter counts for all the models we consider in Table _ Two-step models have either fewer paramcters than the large onc-step model versions, roughly comparable amount, except for some exceptions which we now discuss. First, when using normalizing flows second-step models, used significantly more complex models than with other two-step models. We did this for added variability in the number of parameters, not because using fewer parameters makes two-step models not outperform their single-step counterparts. Two-step models with an NF as the second-step model outperform other two-step models (see Table D. but there much more drastic improvement from single to two-step models. This difference in improvements further highlights that the main cause for empirical gains the two-step nature of parameters
our models, rather than increased number of parameters. Second, the AE+EBM models than their single-step baselines. This was by design, the architecture of the energy functions mimics that of the encoders of other larger models, except it outputs scalars and thus has fewer parameters, and hence we believe this remains a fair comparison. We also note that AE+EBM models have most of their parameters assigned to the AE, and the second-step EBM contributes only 4k additional parameters. AE+EBM models also train and sample much faster then their single-step EBM+ counterparts. Finally, we finish with the obscrvation that mcasuring capacity is difficult, and parameter counts simply provide a proxy.
Table 3: Approximate parameter counts in thousands.
MODEL
MNIST/FMNIST SVHN/CIFAR-10
AVB
750
980
AVB / AVB;
882
1725
AVB+ARM
1021
1251
AVB+AVI
913
1143
AVB-EBM
754
984
AVB NF
5756
5986
AVB-VAE
771
1001
VAE
412
703
VAE+ / VAES
824
1448
VAE+ARM
683
974
VVE+AVB
575
866
EBE
416
707
VAE+NF
5418
5709
ARM / ARM,
797
799
AR+ARM
683
974
EBM* EBM
236
99
AE+EBM
416
707
Additional Experimental Results
D.1 Simulated Data
As mentioned in the main manuscript, we carry out additional experiments where we have access to the ground truth Pa in order to further verify that our improvements from two-step models indeed come from mismatched dimensions. Fig. 0 shows the results of rumning VAE and VAE+ models when trying approximate nonstandard 2-dimensional Gaussian distribution. First, can that when setting the
34

---

Published in Transactions on Machine Learning Research (07/2022)
intrinsic dimension of the models to d = 2, the VAE and VAE+VAE models have very similar performance, with the VAE. being slightly better. Indeed, there is no reason to suppose the second-step VAF. will have an easier time learning encoded data than the first-step VAE learning the actual data. This result visually confirms that two-step models do not outperform single-step models trained with maximum likelihood when the dimensson of maximun-likellhod is correctly specified. Second, we can see that both the VAE and the VAE+ VAE models with intrinsic dimension d = 1 underperform their counterparts with d = However, while the VAE model still manages to approximate its target distribution, the VAE+VAE completely fails. This result visually confirms that two-step models significantly underperform single-step models trained with maximun-likelibe
if the data has no low-dimensional structure and the two-step model tries to enforce such structure anyway. Together, these results highlight that the reason two-step models outperform maxinun-ikellhood so strongly in the main manuscript is indeed the dimensionalily mismatch caused by not. heeding to the manifold hypothesis.
trut
VAL/d 2)
2
8
-23
-2g
-23
VAE VVB(a 3]
0.68
VAEIVAR(/ 2)
0.12
-1
-1
Figure 6: Results on simulated data: Gaussian ground truth (top left), VAE with d= (top middle), VAE with d = (top right), VAE+VAE with d = (bottom left), and VAE+ VAE with d = 2 (bottom right).
D.2 Samples
We show samples obtained by the VAE, VAE+, VAES and VAE+ARM models in Fig. In addition to the FID improvements shown in the main mamuscript, we can see very noticcable qualitative improvement obtained by the two-step models. Note that the VAE in the VAE+ARM model is the same as the single-step VAE model. Similarly, we show samples from AVB AVB+NF, AVB+EBM, and AVB+VAE in Fig. g where two-step models greatly improve visual quality. We also show samples from the ARM+, ARM and AE+ARM from the main manuscript in Fig. and for the EBM+ EBM and AE+EBM models in Fig. 10 We can see that FID score is indeed not always indicative of image quality, and that our AE+ARM and AE+EBM models significantly outperform their single-step counterparts (except AE+EBM on MNIST), Finally, the BiGAN and WAE samples shown in Fig. 11 and Fig. 12 respectively are not consistently better for two-step models, but neither BiGANs nor WAEs are trained via maximum likelihood so manifold overfitting is not necessarily implied by Theorem 1. Other two-step combinations not shown gave similar results.
35

---

Published in Transactions on Machine Learning Research (07 (07/202) /202))
J
8
5
4
9
7
7
67
7
9
0
5
7
7
2
V
AR
3
47
7
8
9
9
9
9
3
6
3
8
4
4
3
4 8 7
7
9
5
3
3
6
X
5
4
2
7
3
7
0
2
Z
8
5
/
0
6
3
2
6
7
0
8
8
5
e
53
2
Figure 7: Uncurated samples from models trained on MNIST (first row), FMNIST (second row), SVHN (third row), and CIFAR-10 (fourth row). Models are VAE (first column), VAE* (second column), VAE (third column), and VAE+ARM (fourth column).
36

---

Published in Transactions on Machine Learning Research (07/202) /202))
3
3
6
3
7
7
5
4
4
7
4
9
9
7
4
4
3
4
/
0
5
3
D
4
8
3
3
7
9
0
7
O
6
2
S
7
5
9
6
6
6
/
3
6
4
7
9
7
7
7
6
8
O
5
2
/
0
5
5
6
9
7
6
O
3
O
D
/
3
2
3
O
5
3
5
7
%
6
3
Z
6
6
7
9
3
R
2
9
5
7
2
9
&
6
3
2
6
/
(
4
8
6
7
4
9
?
0
8
7
2
6
9
0
2
8
2
Figure 8: Uncurated samples from models trained on MNIST (first row), FMNIST (second row), SVHN (third row), and CIFAR-10 (fourth row). Models are AVB; (first column), AVB+EBM (second column), AVB+NF (third column), and AVB+VAE (fourth column).
37

---

Published in Transactions on Machine Learning Research (07/202) (202))
0
0
9
8
5
24
4
2
3
O
8
4
9
5
S
4
3
6
2
9
6
8
7
7
O
3
5
5
9
7
8
4
2
0
2
4
3
4
R
8
5
5
S
3
5
4
Figure 9: Uncurated samples from models trained on MNIST. (first row), FMNIST (second row), SVHN (third row), and CIFAR-10 10 (fourth row). Models ARM (first column), ARM (second column). and AR+ARM (third column).
38

---

Published in Transactions on Machine Learning Research (07 (07/2022)
?
.
2
3
O
5
9
5
9
5
9
3
4
9
3
6
7
9
9
0
5 8 3
/
/
5
6
0
7
3
5
2
7
/
2
2
O
5
7
7
4
1
7
S
6
7
8
6
6
7
8
A 4
2
O
3
3
O
4
5
O
3
6
7
3
6
_
7
1
9
9
6
3
5
3
6 6
7
4
7
5
4
6
3
7
4
&
6
2
7
4
/
3
2
2
2
7
Figure 10: Uncurated samples with Langevin dynamics run for 60 steps initialized from training buffer on MNIST (first row), FMNIST (second row), SVHN (third row). and CIFAR-10 (fourth row). Models are + (first column), EBM, (second column), and AE EBM (third column).
39

---

Published in Transactions on Machine Learning Research (07 (07/2022)
4
4
7
4
6
8
3
3
7
5
7
7
7
6
2
7
O
9
3
9
Q
O
4
7
7
f
6
/
3
7
7
8
9
5
6
C
7
8
G
5
I
4
6
8
7
7
/
6
3
7
44
5
6
S
6
O
2
/
2
2
O
4
4
3
3
9
6
3
7
6
/
6
7
S
V
4
2
O
7
O
4
3
5
Z
f
4
C
2
5
7
3
6
5
4
3
5
7
3
0
O
3
5
2
7
7
/
/
3
7
8
5 8
Figure 11: Uncurated samples from models trained on MNIST (first row), FMNIST (sccond row), SVHN (third row), and CIFAR-10 (fourth row). Models are BiGAN (first column), BiGAN+ (second column), BiGAN+AVE (third column), and BiGAN+NF (fourth column). BiGANs are not trained via maximum-likelihodd Theorem 1 docs not imply that manifold overfitting should occur.
40

---

Published in Transactions on Machine Learning Research (07 (07/2022)
6
4
6
4
6
7
4
9
7
2
3
136
4
8
9
7
U
4
1
4
3
6
3
6
0
2
8
X C
3
9
9
2
7
4
2
5
5 O
O 1
6
7
3
3
7
7
9
3
2
7 4 /
6
/
3 3 O
8 R
7
8
9
6
6 9 S 6
6
2
7
8
9
6
5
2 6
8
3 7 6
C
6
?
S
7
O O 7 /
R
/
4 /
1 3 O
3
C
5
/
8
9
O O 3 3
9
3
2
3
S
3
S
?
3
7
O
7
7
2 /
4
2
6
7
4
/ 3
3
0
3
S
U
6
8
5 4 3
0
1
5
2
G
?
7
6
6
9
6
9
5
7
9
0
2
3
5
2
/
8
2
Figure 12: Uncurated samples from models trained on MNIST (first row), FMNIST (second row), SVHN (third row), and CIFAR-10 (fourth row). Models are WAE+ (first column), WAE+ARM (second column), WAE+NF (third column), and WAE+VAE (fourth column). WAEs are trained via maximum-likelihodd Thcorem 1 docs not imply that manifold overfitting should occur.
41

---

Published in Transactions on Machine Learning Research (07 (07/2022)
D.3 EBM Improvements
Following Du & Mordatch (0119), evaluated the single-step EBM's sample quality the basis of samples initialized from the training buffer. However, when MCMC samples were initialized from uniform noise, we observed that all samples would converge to a small collection of low-quality modes (sce Fig. 13 Morcover, at each training epoch, these modes would change, even as the loss value decreased.
The described non-convergence in the EBM's model distribution is consistent with Corollary On the other hand, when used as low-dimensional density estimator in the two-step procedure, this problem vanished: MCMC samples initialized from random noise yielded diverse images. See Fig.
13 for comparison.
3
8
7
7
2
7
7
7
4
0
0
7
0
7 3
7
7
4
R
7
3
3
5
+
2
3
5
7
8
7
2
5
C
4
7
4
5
4
7
3
5
9
5
?
5
9
7 7 )
7 4
7.
?
3
7
L
S
D
3
7
7
3
7
4
C
7
4
7
7
7 52
7
7
%
5
/
9
3
Figure 13: Uncurated samples with Langevin dynamics initialized from random noise (with no buffer) trained on MNIST (first row), FMNIST (second row), SVHN (third row), and CIFAR-10 (fourth row). Models are + with 60 steps (first column), EBM+ with 200 steps (second column), EBM+ with 500 steps (third column), and AE EBM with 60 steps, (fourth column).
42

---

Published in Transactions on Machine Learning Research (07/ (07/2022)
D.4 FID, Precision, and Recall Scores
We show in Tables 4 and precision and recall (along with FID) of all the models used in Sec. 6.2 We opt for the precision and recall scores of Kynkäänniemi et al. (2019) rather than those of Sajjadii et al. 20188) as the former aim to improve on the latter. We also tried the density and coverage metrics proposed by Nacem et al. (2020), but found these metrics to correlate with visual quality less than FID. Similarly, we also considered using the inception score (Salimans et al., (2016), but this metric known to have issues (Barratt & Sharma, 2018), and the FID is widely preferred over it, We can see in Tables andd [ that two-step models consistently outperform single-step models in recall, while either also outperforming or not underperforming in precision. Much like with FID score, some instances of AR+ARM have worse scores on both precision and recall than their corresponding single-step model. Given the superior visual quality of those two-step models, we also consider these as failure cases of the evaluation metrics themselves, which we highlight in red in Tables | and We believe that some non-highlinn hon-hiiglighttd results do not properly reflect the magnitude by which the two-step models outperformed single-step models, and encourage the reader to see the corresponding samples.
We show in Table the FID scores of models involving BiGANs and WAEs. These methods are not trained via maximum likelihood, so Theorem 1 does not apply. In contrast to the likelihood-based models from Table 1 there is no significant improvement in FID for BiGANs and WAEs from using a two-step approach, and sometimes two-step models perform worse. However, for BiGANs we observe similar visual quality in samples (see Fig. 11,, once again highlighting a failure of the FID score as a metric. We show these failures with red in Table
Table 4: FID (lower is better) and Precision, and Recall scores (higher is better). Means + standard errors across 3 runs are shown. Unreliable scores are highlighted in red.
MNIST
FMNIST
MODEL
FID
Precision
Recall
FID
Precision
Recall
AVB
219.0 4.2
0.0000 0.0000
0,0008 + 0.0007
235.9 1: 4.5
0.0006 =:2 0.0000
0,0086 - 0.0037
AVR*
205.0 + 3.9
0.0000
0.0000
0.0106
0.0089
216.2
0.0008 + 0.0002
0.0075
00.0522
AVB
205.2 1.0
0.0000 = 0.0000
0.0065 : 0.0032
223.8 5.4
0.0007 = 0.0002
0.0034
0.0009
AVB+ARM
86.4 0.9
0.0012 1: 0.0003
0.0051 1: 0.0011
78.0 0.9
0.1069 1 0.0055
0.0106
0.0011
AVB+AVB
133.3 1 0.9
0.0001 1. 0.0000
0.0093 1 0.0027
143.9 1.5
0.0151 1 0.0015
0.0093
0.0019
AVB+EBM
96.6 = 3.0
0.0006 = 0.0000
0.0021 + 0.0007
103.3 + 1.4
0.0386 = 0.0016
0.0110 = 0.0013
AVB NF
83.5 1: 2.0
0,0009 + 0.0001
0.0059 1 0.0015
1.1
0.1153 1 0.0031
0,0092 +: 0,0004
AVB VAE
106.2 1 2.5
0,0005 + 0.0000
0,0088 + 0.0005
105.7 0.6
0.0521 = 0.0035
0.0166 = 0.0007
VAE
197.4 +: 1.5
0.0000
0.0000
0.0035 0.0004
188.9 :: 1.8
0.0030
0.0006
0.0270 0.0048
VAE+
184. = 0.7
0.0000 = 0.0000
0.0036 = 0.0006
179.1 0.2
0.0025 = 0.0003
0.0009 0.0012
VAES
185.9 = 1.8
0.0000 = 0.0000
0.0070 + 0.0012
183.4 0.7
0.0027 = 0.0002
0.0095 = 0.0036
VAE+ARM
69.7 - 0.8
0.0008 + 0.0000
0.0041 + 0.0001
70.9 + 1.0
0.1485 = 0.0037
0.0129 = 0.0011
IAE
117.1 + 0.8
0,0002 1 0.0000
0.0123 1 0.0002
129.6 3.1
0.0291 1 0.0040
0.0454 + 0.00-66
VAE+EBM
74.1 1 1.0
0,0007 + 0.0001
0,0015 1 0.0006
78.7 2.2
0.1275
1 0.0052
0.0030 = 0.0002
VAR+NF
70.3 + 0.7
0.0009 + 0.0000
0.0067 + 0.0011
73.0 133.0+00 0.3
0.1403 + 0.0022
0.0116 0.0016
ARM*
98.7 10.6
0.0471 + 0.0098
0.3795 = 0.0710
72.7 2.1
0.2005 = 0.0059
0.4349 0.0143
ARM,
31.7 3.1
0.0849 + 0.0112
0.3349 + 0.0063
23.1 +
0.3508 = 0.0099
0.5603 = 0.0092
AE+ARM
72.0 = 1.3
0.0006 + 0.0001
0.0038 + 0.0003
76.0 + 0.3
0.0986 + 0.0038
0.0069 = 0.0005
EBM
84.2 + 4.3
0.4056
0.0145
0.0008 + 0.0006
135.6 1.6
0.6550 + 0.0054
0.0000
0..000
EBMË
10L. 12.3
0.3748
0.0496
0.0013 + 0.0008
135.3
0.9
0.6384 + 0.0027
0.0000
0.0000
AE+EBM
75.4 2.3
0.0007 + 0.0001
0.000S + 0.0002
83.1 + 1.9
0.0891 = 0.0046
0.0037 0.0009
43

---

Published in Transactions on Machine Learning Research (07 (07/2022)
Table 5: FID (lower is better) and Precision, and Recall scores (higher better). Means + standard errors
across 3 runs are shown. Unreliable scores are highlighted in red.
SVHN
CIFAR-10
MODEL
FID
Precision
Recall
FID
Precision
Recall
AVB
356.3 + 10.2
0.0148 + 0.0035 0.0000 + 0.0000
289.0 + 3.0
0.0602 + 0.0111 0.0000
0.0000
AVB*
352.6 + 7.6
0.0088 + 0.0018
0.0000 + 0.0000
297.1
+ 1.1
0.0902 + 0.0192
0.0000
0.0000
AVB;
353.0
0,0425 + 0.0293
0,0000 +:200 0.0000
305.8 8.7
0.1304 = 0.0460
0.0000
0.0000
AYR+ARM
56.6 + 0.6
0.6741 + 0.0090
0.0206 + 0.0011
182.5 + 1.0
0.4670 + 0.0037
0.0003
0.0001
AVB+ AVB
74.5 1: 2.5
0.5765 +: 0.0157
0.0224 +: 0.0008
183.9 1.7
0.4617 :: 0.0078
0.0006
0.0003
AVB+EBM
61. 0.8
0.6809
0.0092
0.0162 = 0.0020
189.7
0.4543
0.0094
0.0006
0.0002
AVB+NF
55.4 0.8
0.6724 0.0078
0.0217 = 0.0007
0.8
0.4632
= 0.0024
0.0009
0.0001
AVB VAE
50.9 1 1.3
0.6698
0.0105
0.0214 1 0.0010
186.7 1. 0.9
0.4517 + 0.0046
0.0006 0. 0.0001
VAE
311.5 = 6.9
0.0098 = 0.0030
0.0018 + 0.0012
270.3
+ 3.2
0.0805 = 0.0016
0.0000 = 0,0000
VAE*
300 = 2.1
0,0133 = 0.0014
0,0000) = 0.0000
257.8 + 0.6
0.1287 = 0.0183
0,0001 = O. 0.0000
VAE,
302.2 1: 2.0
0.0086 1 0.0018
0.0004 1: 0.0003
257.8
1: 1.7
0.1328 1 0.0152
0.0000 0.0000
VAE+ARM
52.9 0.3
0.7004
0.0016
0.0234 + 0.0005
1.3
0.4865 = 0.0055
0.0004 0.0001
VAE+AVB
64.0 1.3
0.6234
= 0.0110
0.0273 + 0.0006
2.0
0.5140 = 0.0123
0.0007
0.0002
VAETEBM
63.7 3.3
0.6983
0.0071
0.0163 1 0.0008
181,7
2.8
0,4849 1 0.0098
0.0002 O. 0.0001
VAE+NF
52.9 0.3
0.6902
0.0059
0.0243 1 0.0011
175.1 1: 0.9
0.4755 + 0.0095
0.0007 0.0002
ARM
168.3 = 4.1
0.1425 = 0.0086
0,0759 = 0.0031
162.6 2.2
0.6093 = 0.0066
0,0313
0.0061
ARMA
149.2 1: 10.7
0.1622 1 0.0210
0.0961 1 0.0069
136.1 4:2.
0.6585 1 0.0116
0.0993
0.0106
AE+ARM
60.1 = 3.0
0.5790
0.0275
0.0192 + 0.0014
186.9 + 1.0
0.4544 = 0.0073
0.0008
0.0002
EBM
228 + 5.0
0.0955 + 0.0367
0.0000 1 0.0000
201.4 7.9
0.6345 1 0.0310
0.0000
0.0000
EBM
235.0 : 5,6
0,0983 + 0.0183
0,0000 + 0.0000
200.6 + 4.8
0.6380 + 0.0156
0.0000 = 0.0000
AE+EBM
75.2 = 4.1
0.5739 + 0.0299
0.0196 + 0.0035
187
0.4586 = 0.0117
0.0006 0.0001
Table 6: FID scores (lower is better) for non-likelihood based GAEs and two-step models. These GAEs are not trained to maximize likelihood, so Theorem 1 does not apply. Means + standard errors across 3 runs are shown. Unreliable scores are shown in red. Samples for unreliable scores are provided in Fig. 11
MODEL
MNIST
FMNIST
SVHN
CIFAR-10
BIGAN
150.0 1.5
139.0 + 1.0
105.5
170.9 4.3
BIGAN+
135.2
11.
1.4.44..9
152.9.10.6
BIGAN+ARM
1.2.6+1.6
0.7
60.8
210.7
BIGAN AVB
149.9
1.7.5
2.6
1.0
BIGAN+EBM
4.7
108.1124
217.5 1.3
1.2.4
9..0
60.2+ 1.5
211.6 1.7
BIGAN+VAE
127.9-11.6
1
63.6 1.4
216.3 1.2
WAE
19.8
4..1
52.7
1.7.4
WAE+
1..7
4.2
179.7 0.4
WAE+ARM
0.5
46.1
73.1+18
182.3 1.7
WAE+AVB
1..
4.9
60.2
157.6 0.8
WAE+EBM
1.0
6..2
7.56
161. 4.7
WAE+NF
20.7 2.2
1.99
57.6
138
178.2 2.8
WAE VAE
16.4
50.9 0.5
72.2 1 1.9
178.3122
D.5 High Resolution Image Generation
As mentioned in the main manuscript, we attempted to use two-step methodology to improve upon a high-performing GAN model: StyleGAN2 (Karras et al. 2020b). We used the PyTorrh (aszke et al. 2019 code of Karras et al. (2020a), which implements the optimization-basee projection method of Karras et al. (2020b). That is, we did not explicilly construct y, and used this optimization-based GAN inversion method to recover {2n ) on the FFHQ dataset (Karras et al., 2019), with the intention of training low-dimensional DGMs to produce high resolution images. This method projects into the intermediate 512-dimensional space referred to as W by default (Karras et al. (220b)). We also adapted this method to the GAN's true latent. space, referred to 2, during which we decreased the initial learning rate to 0.01 from the default.
44

---

Published in Transactions on Machine Learning Research (07 (07/2022)
of 0.1. In experiments with optimization-aasee inversion into the latent spaces g(M) W and g 2, reconstructions (G(zn))m= yielded FIDs of 13.00 and 25.87, respectively. In contrast, the StyleGAN2 achieves an FID score of 5.4 by itself, which is much better than the scores achieved by the reconstructions (perfect reconstructions would achieve scores of 0).
The FID between the reconstructions and the ground truth images represents an approximate lower-bound on the FID score attainable by the two-step method, since the second step estimates the distribution of the projected latents {an)n=1- Since reconstructing the entire FFHQ dataset of 70000 images would be expensive (for instance, W-space reconstructions take about 90 seconds per image), computed the FID (again using the code of Karras et al. (2020a)) between the first 10000 images of FFHQ and their reconstructions.
We also experimented with the approach of Huh et al. (020), which inverts into Z-space, but it takes about minutes per image and was thus prohibitively expensive. Most other GAN inversion work (Xia et al. 2021) has projected images into the extended 512 18-dimensional W+ space, which describes different intermediale latent input for each layer of the generator. Since this latent space is higher in dimension than the true model manifold, did not pursue these approaches. The main obstacle to improving StyleGAN2's FID using the two-step procedure appears to be reconstruction quality, Since the goal of our experiments is to highlight the benefits of two-step procedures rather than proposing new GAN inversion methods, we did not further pursue this direction, although we hope our results will encourage research improving GAN inversion methods and exploring their benefits within two-step models.
D.6 OOD Detection
OOD Metric We now precisely describe our classification metric, which properly accounts for datasets of imbalanced size and ensures correct directionality, in that higher likelihoods are considered to be in- dist.ribttion First, using the in- and out-of-sample training likelihoods, we train a decision stump iee. a single-threshol-
classifier. Then, calling that threshold T, we count the number of in-sample test likelihoods which are greater than T, RTST, and the number of out-of-sample test likelihoods which are greater than T, no>T,. Then, calling the number of in-sample test points nı, and the number of OOD test points no, our final classification rate acc is given as:
71ST + nR (no - nOST)
acc
(45)
2n
Intuitively, we can think of this metric as simply the fraction of correctly-classifed points (i.e. acc' = (os-t(no=(es)),, but with the contributions from the OOD data re-weighted by factor of nz to ensure no
ar+no both datasets are equally weighted in the metric. Note that this metric is sometimes referred to as balanced accuracy, and can also be understood as the average between the true positive and true negative rates.
We show further OOD detection results using logpz in Table 7 and using logpx in Table S Note that, for one-step models, we record results for lx,,x, px, the log-density of the model, in place of log pz (which is not
defined).
45

---

Published in Transactions on Machine Learning Research (07 (072)022)
Table 7: OOD classification accuracy as percentage (higher better), using log Means + standard errors across runs are shown. Arrows point from in-distribution to OOD data.
MODEL
FMNIST MNIST CIFAR-10 SVHN
AVB
96.0 0.5
2.14
AVB ARM
2.4
4.2
AVB+AVI
4.2 7..4
AVB+ERM
0.1
49.0+000
AVB+NF
8.9
4.9
AVB+VAE
1..
4.1
VAE*
96.1
2.28
VAE+ARM
9.0
39.7
VAE+AVB
80.6
4.14
VAE+EBM
0..1
4.0
VAE+NF
9..
0.1
*RM
9.9 0.6
1.0
AE ARM
86.5
37.4
FBM
32.5 1.1
46.4
AE+EBM
0.2
4..4
Table 8: OOD classification accuracy as percentage (higher is better), using logpx. Means + standard errors across 3 runs are shown. Arrows point from in-distribution to OOD data.
MODEL
FMNIST + MNIST CIFAR-10 SVHN
AVB*
96.0 + 0.5
23.4 0.1
NWB+ARM
90.8 1 1.8
37.7 1 0.5
AVB AVB
75.0 2.2
43.7 1 2.0
AVB+ERM
53.3
39.1 0.9
AV
89.2
43.9 = 1.3
AVB+VAB
7.
40.2 0.2
VAE+
9.1
23.8 0.2
VAE+ARM
0.7
37.6 0.4
VAE AVB
8.4124
42.2 1 1.0
VAE+EBA EBM
6.7
42.4 0.9
VAE+NF
91.7
42.4 0.3
ARM
0.6
15.5 = 0.0
ARM
89.5 1.51022
33.8 1: 0.3
EBM
32.5 1.1
46.4 3.1
EBM
56.9 14.4
34.5 0.1
BIGAN+ARM
81.5 +: 1.4
35.7 1: 0.4
DGGA+-AB
59.6 1. 3.2
34.3 2.3
BIGAN-EBM
57.4 1.7
47.7 0.7
BIGAN-NF
83.7 + 1.2
39.2 0.3
BIGAN VAE
59.3 2.1
35.6 - 0.4
WAE+ARM
89.0 1: 0.5
38.1 +: 0.6
WAE+AVD
74.5 L 1.3
43.1 1 0.7
WAE+EBM
36.5 + 1.6
36.8 - 0.4
WAE+NF
85.7 + 2.8
40.2 - 1.8
WAE VAE
87.7 1 0.7
38.3 - 0.4
46

---

Published in Transactions on Machine Learning Research (07/202) 2022)
NF Trained on FMNIST
BIGAN+NF Trained on FMNIST
WAE+ NF Trained FMNIST
020
PANEST
Number sampltes
120
bion Boundary
Number of sampless
Decision Boundary
Number of samples
Deceson Boundary
WWIST
423-
220
200
-500
500
10:0
15:0
2003
A186
-120
-100
-90
-90
-00
-20
-100
-00
-20
-40
-20
log
10g
log
NF Trained on CIFAR-10
BiGAN + NF Trained on CIFAR-10
Trained CIFAR-10
DTA0-10
DINI-1C
1780
CIRR-10
Number samples
300
Docsion Boundary
Number samples
400
on Boundary
Number of somples
Dcciston Boundary
SWHII
600
200
220
400
500
200
1295
600
50
20
10 10
90
30
22
10 20
log,zz
log
log
Figure 14: Comparison of the distribution of log-likelihood values between in-distribution (green) and aut-of-distributte (blue) data. In both cases, the two-step models push the in-distribution likelihoods further to the right than the NF+ model alone. N.B.: The absolute value of the likelihoods in the NF+ model on its own are off by a constant factor because of the aforementioned whitening transform used to scale the data before training. However, the relative value within a single plot remains correct.
VAE Trained FMNIST
VAE+NF Trained on FMNIST
SHNSST
Nuumbrr sampleo
Deasion Bouncary
WISS
Number sempltes
MEAST
Decisinn Bourneryy
320
X00
-952
-250
500
1250
Po
log,2z
VAE Trained on CIFAR-10
NF Trained on CIFAR-10
C3ARB-10
Choa 10
Number of samplee
200
Boundary
Number sarmlees
Decsion Boundary
SW--1
400
120
-1000
1000
3000
X000
aX0
SC56
100)
300
-202
-100
-302
-140
-200
1og
log
Figure 15: Comparison of the distribution of log- likelihood values between in-distribution (green) and out-of- distribution (blue) data for VAE-based models. While the VAE* model does well FMNIST-MMIIS its performance is poor for CIFAR-10 -SHN.. The two-step model VAE+NF improves on the task.
47

---

Published in Transactions on Machine Learning Research (07/2022)
AVB Trained FMNIST
AVB+NF Trained on FMNIST
Number of samplees
Dedsion Boundary
Number semples
MNSS
necn Roundwy
600
800
300
-1000
-15h
1006
1901
-0
1og
AVB Trained on CIFAR-10
AVB+NF Trained CIFAR-10
CI-AB50
C3RA8-10
Number of camplles
300
Number semples
SW-11
Decsion Boundary
120
X000
(003
80
-00
50
40
log.z
log
Figure 16: Comparison of the distribution of log-likelihood values between in-distribution (green) and out-of- distribution (blue) data for AVB- based models. While the AVB model does well FMNIST- its performance is poor for CIFAR-10-SVVE The two-step model AVB+NE improves on the CIFAR-10-SVHN task.
48


Reviews:
[{"review_id": "1pZsseoF15", "reviewer": "Reviewer_Cev4", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths.**\n\n- The paper is very well written. It includes fairly technical mathematical statements, but they are presented in a clear way, with supporting intuition. I enjoyed reading the paper.\n- The presented observations are interesting and insightful. While in hindsight the main statement (manifold overfitting) may be fairly clear, I don't think I thought about this particular phenomenon deeply before.\n- The proposed method is very simple and practical. \n- The empirical results are promising.\n\n**Weaknesses and other thoughts.**\n- The theory in the paper focuses on the case when the data actually lies on a low-dimensional manifold. However, it is not obvious how much of a concern this scenario is in practice. In particular, in practice people often use dequantization which leads to a slightly softer distribution, which doesn't strictly lie on a manifold. The authors comment on this point and show that adding noise to the data is not sufficient to achieve the gains in FID scores from the proposed two-stage procedure. However, I don't think this point can be fully dismissed based on the presented results.\n-  Generally, it is not obvious from the results in the paper if manifold overfitting happens in generative models in practice. The main evidence is the synthetic experiment in Figure 4, and the experiments showing improved FID scores and OOD detection with the two-step procedure. It would be great to see experiments directly showing that a version of manifold overfitting happens in realistic generative models.\n- Methods similar to the proposed two-stage procedure have been considered before, at least in the context of OOD detection (but likely also in generative models more generally). For example [1-3] all consider density modeling in the embedding space, showing strong results. \n- \n\n**Classical mixture of Gaussians example**\n\nThe authors mention that the manifold overfitting is quite different from regular density model overfitting on page 5, and I generally agree. However, I wanted to mention the classical MoG example mentioned in Bishop [4], section 9.2.1, Fig. 9.7. On a finite dataset, a mixture of Gaussians with two components can already achieve arbitrarily high likelihoods: the first component needs to overfit one datapoint, while the other component should just cover the rest of the data. This way, it is possible to achieve arbitrary likelihoods with a fixed capacity model class, as the data size grows. \n\nThis example is, however, different from the manifold overfitting, as the likelihood will only be infinite for one datapoint. But I think it is still worth mentioning. \n\n**References**\n\n[1] _Deep Residual Flow for Out of Distribution Detection_\nEv Zisselman, Aviv Tamar\n\n[2] _Why Normalizing Flows Fail to Detect Out-of-Distribution Data_\nPolina Kirichenko, Pavel Izmailov, Andrew Gordon Wilson\n\n[3] _Hybrid Models for Open Set Recognition_\nHongjie Zhang, Ang Li, Jie Guo, Yanwen Guo\n\n[4] _Pattern Recognition and Machine Learning_\nChristopher Bishop\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "- The paper identifies a failure mode of maximum likelihood training, where the data distribution is concentrated on a low-dimensional manifold, but a density model is defined over the full input space. In this case, the authors argue that the density model can learn an arbitrary distribution over the data manifold.\n- The authors prove the possibility of manifold overfitting theoretically.\n- The authors propose a two-step procedure, where first the data is projected in a low-dimensional embedding space, and then a density model is learned in the embedding space. The embedding is done with a generalized auto-encoder model.\n- The authors show that the two step procedure allows for improvements in FID scores and OOD detection.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "I think this is a strong paper and I recommend acceptance. So, I don't request any changes that are _required_ for acceptance, just some suggestions for what can strengthen the paper in my opinion.\n\n- I would be excited to see an experiment showing that manifold overfitting is an issue in practice. [**not required**]\n- It would be interesting to see more ablations in the experiments. In particular, what is the effect of the parameter $d$, dimension of the learned manifold? What if you under or over-estimate it. [**not required**]\n- More controlled experiments would be interesting. In particular, what if the learned manifold in Fig 4 was different from the ground truth (shifted circle or a square)? [**not required**]\n- I would love to see experiments showing that learning a distribution in the low-dimensionality space in the two-step procedure helps with FID scores for state-of-the-art models. As far as I understand, learning a density model in the embedding space should be cheap, so you may even be able to do it on a dataset like ImageNet, with a state-of-the-art GAN model. There, an improvement in FID scores would be quite exciting. [**not required**]", "broader_impact_concerns": "No ethical concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "H0aBKR0WQR", "reviewer": "Reviewer_UUHX", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n\n* Theoretical analysis is rigorous and has minimal assumptions. The theoretical investigation and the proposed two step training can explain practical tricks widely used in many previous works on generative modeling.\n\n* Comprehensive discussion of related work.\n\n# Weaknesses\n\n* Although having a rigorous theoretical analysis is desirable, the proposed two-step training has already been widely used in the community. It is also widely accepted that combining manifold learning and likelihood-based generative modeling can lead to better sample quality (see, e.g., [1]) and better results on outlier detection (see, e.g., [2]). The experimental results are therefore not surprising and do not provide much information to the research community.\n\n* Experiments can be more rigorous: \n  1. When comparing with baselines, need to make sure the improved performance is not due to having more model parameters. I know authors have briefly described the difference in model parameters in the main text, but I think it can be more clear to include such information directly in Table 1.\n  2. Need to verify whether the performance improvement of two-step training only happens for data on low dimensional manifolds. Authors should report results on synthetic datasets in which datapoints do not have a low-dimensional submanifold structure.\n \n* Paper can be more comprehensive by including additional related work in the discussion. For example, ref. [3] discusses the difficulty of learning autoregressive generative models on low-dimensional data manifolds, and proposes to perturb data with Gaussian noise before training autoregressive models for significantly improved sample quality.\n\nReferences:\n\n[1] Razavi, A., Van den Oord, A., & Vinyals, O. (2019). Generating diverse high-fidelity images with vq-vae-2. Advances in neural information processing systems, 32.\n\n[2] Caterini, A. L., Loaiza-Ganem, G., Pleiss, G., & Cunningham, J. P. (2021). Rectangular flows for manifold learning. Advances in Neural Information Processing Systems, 34.\n\n[3] Meng, C., Song, J., Song, Y., Zhao, S., & Ermon, S. (2021). Improved autoregressive modeling with distribution smoothing. arXiv preprint arXiv:2103.15089.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Leveraging measure theory, this paper analyzes the difficulty of maximum likelihood training when the true data density lies on a low-dimensional manifold. An alternative method is thus proposed to address this difficulty, with a two-step idea where a (generalized) autoencoder is first trained to capture the low dimensional manifold before a second likelihood-based model is trained with samples in the latent space. Experiments have demonstrated that this two-step training procedure leads to higher sample quality as well as improved outlier detection results.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "1. Include model parameter counts as an additional column in Table 1.\n2. Perform experiments on synthetic datasets that do not lie on low-dimensional manifolds, and check if the results are consistent with authors' theory.", "broader_impact_concerns": "I disagree with the authors that their approach does not have any negative societal consequences. Two-step training leads to better generative models, which themselves have a lot of issues when applied inappropriately in the real world.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Dz3SMn36wt", "reviewer": "Reviewer_SQHa", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Clarifying question: \nPage 1: M \\in R^D should this be d rather than D? This sentence was confusing “unknown d-dimensional manifold M ⊂ R^D, where d < D.”\n\nThe authors say that “This modelling choice implies the model has D-dimensional support” however, many models produce samples from a latent distribution which has a dimension << D. Could the authors comment on the connection here, please.\n\nCan the authors please check for consistency of D and d notation. \n\nFigure 2: What peak values are expected for the manifold in Figure 2 Left panel? Is it that the area under the graph surrounding each data point integrates to 0.3 and 0.7 and hence he peeks are c.0.6 and c.1.4 respectively? This would be good to clarify please?\n\nFollowing the example from 3.1 it would be very helpful to be explicit that the “manifold overfitting” problem is not that the model can assign density to the correct values, but that it applies the incorrect density to the point values.\n\nQ:  “The underlying issue here is that M is “too thin in R^D” , and thus p(x) can “spike to infinity” at every x ∈ M.” Is this not true in all dimensions if density is at a point?\n\nPractically speaking, why does it matter if the model overfits to the manifold, provided that density is assigned to the correct parts of the distribution?\n\nPg6 “P* -almost surely”, not sure of the role of P* in this sentence?\n\nFigure 4: No upper limit value on the colour bar.\n\nQ: How do you pick the dimension of Pz? If this is too large the same problems will exist and if it’s too small you will not be able to capture the manifold?\n\nQ: Optimising (1) does not guarantee a good decoder? And it is possible to learn a g() that maps all samples to the same value (for example). You need a good decoder to avoid this.\n\n“The condition G(g(x)) = x, P ∗ -almost surely, is what one should expect to obtain during the dimensionality reduction step” is also contradicted by Figure 3?\n\n\nComments:\n“a DGM is performed” <— this sentence does not make sense.\n\nIt may make more sense to use more standard generative model notation (either from GAN or VAE literature).\n\nThe example in 3.1 is nice to have and it’s great to have the accompanying figure.\n\nState explicitly that m and sigma are mean and standard deviation.\n\nThis sentence was a great summary: “The underlying issue here is that M is “too thin in R^D” , and thus p(x) can “spike to infinity” at every x ∈ M.”\n\nThis is a very general framework and many existing generative models could easily benefit from this approach. Perhaps it is worth emphasising this more?\n\nResults in Figure 4 appear convincing, but please add a scale to the colour bar.\n\nVery extensive experimentation in Table 1 on a number of datasets and with many model variations. How does the dimension of Pz affect these results?\n\nTable 1 and 2 highlight in bold the best results in each sub-part of the table.\n\nIt would help to have one concrete example of the losses used to train a model.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper is concerned with learning a data distribution P* in R^D where the data exists on a manifold in R^D. The authors show that max-likelihood approaches lead to generative models assigning density to the correct values in R^D but do not assign the correct amount of density. The authors refer to this as manifold overfitting.\n\nThe authors provide theoretical explanation for this and propose a two-step solution, where by they encode (to a latent) and decode an image, but match the distributions of the latent distributions (as well as ensuring decoder(encoder(x)) = x).", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "The following changes are critical:\n\nPage 1: M \\in R^D should this be d rather than D? This sentence was confusing “unknown d-dimensional manifold M ⊂ R^D, where d < D.”\n\nThe authors say that “This modelling choice implies the model has D-dimensional support” however, many models produce samples from a latent distribution which has a dimension << D. Could the authors comment on the connection here, please.\n\nPlease provide more information about how d is selected?\n\nPlease provide examples of losses used, this will add significantly to the understanding of the paper and make it more practical for others to benefit from this work.\n\nGreat to have:\nHow well does your model perform compared to single step models when there are some under-represented classes? \nHow does the values of \"d\" in your model affect performance?\n", "broader_impact_concerns": "Generative models have been used for data augmentation and for learning representations in low data regimes. If existing models *can* place more density on lower probability (i.e. under-represented) samples this is may help a model learn representations for those samples(?). In your setup you encourage the model not to assign as much density to less likely samples, could this perpetuate bias and reduce usefulness of learning in low data regimes? Your OOD result were encouraging, but how well does your model perform compared to single step models when there are some under-represented classes? Some simple experiment on MNIST would be interesting.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The submission conveys a mathematical insight on manifold overfitting rigorously and provides/explains two-steps solutions to address this issue, backed by convincing experiments. Moreover, the authors have also addressed the reviewers concerns adequately, therefore the reviewers and I have come to to a consensus to accept this submission to TMLR.
Decision: Accept (poster)