Deep Generative Models through the Lens of the Manifold Hypothesis: A Survey and New Connections
Authors: Gabriel Loaiza-Ganem
Brendan Leigh Ross
Rasa Hosseinzadeh
Anthony L. Caterini
Jesse C. Cresswell
Venue: Accepted by TMLR ()

Abstract: In recent years there has been increased interest in understanding the interplay between deep generative models (DGMs) and the manifold hypothesis. Research in this area focuses on understanding the reasons why commonly-used DGMs succeed or fail at learning distributions supported on unknown low-dimensional manifolds, as well as developing new models explicitly designed to account for manifold-supported data. This manifold lens provides both clarity as to why some DGMs (e.g. diffusion models and some generative adversarial networks) empirically surpass others (e.g. likelihood-based models such as variational autoencoders, normalizing flows, or energy-based models) at sample generation, and guidance for devising more performant DGMs. We carry out the first survey of DGMs viewed through this lens, making two novel contributions along the way. First, we formally establish that numerical instability of likelihoods in high ambient dimensions is unavoidable when modelling data with low intrinsic dimension. We then show that DGMs on learned representations of autoencoders can be interpreted as approximately minimizing Wasserstein distance: this result, which applies to latent diffusion models, helps justify their outstanding empirical results. The manifold lens provides a rich perspective from which to understand DGMs, and we aim to make this perspective more accessible and widespread.

Published in Transactions on Machine Learning Research (09/2024)
Deep Generative Models through the Lens of the Manifold Hypothesis: A Survey and New Connections
Gabriel Loaiza-Ganem Layer 6 AT
gabriel@layer6.ai
Brendan Leigh Ross
brendon&layerre
Lager 6 AT
Rasa Hosseinzadeh Layer 6
rasa@layer6.ct
Anthony L. Caterini Lager 6 AI
anthony@layerb.co at
Jesse C. Cresswell
jesse Glayerf.a
Layer 6 AT
Reviewed on OpenReview: https: // openreview. net/ forum? id= a90Wpm$t01
Abstract
In recent years there has been increased interest in understanding the interplay between deep generative models (DGMs) and the manifold hypothesis. Research in this area focuses on understanding the reasons why commonly-used DGMs succeed or fail at learning distribu- tions supported on unknown low-dinensional manifolds, as well as developing new models explicitly designed to account for manifold-supporled data. This manifold lens provides both clarity as to why some DGMs (e.g. diffusion models and some generative adversarial models such as variational au-
networks) empirically surpass others (e.g. likelihood- loencoders, normalizing flows, or energy-based models) at sample generalion, and guidance for devising more performant DGMs. We carry out the first survey of DGMs viewed through this lens, making two novel contributions along the way. First, we formally establish that nu- merical instability of likelihoods high ambient dimensions unavoidable when modelling data with low intrinsic dimension. We then show that DGMs on learned representations of autoencoders can be interpreted as approximately minimizing Wasserstein distance: this result, which applies to latent diffusion models, helps justify their outstanding empirical results. The manifold lens provides rich perspective from which understand DGMs, and we aim to make this perspective more accessible and widespread.

---

Published in Transactions on Machine Learning Research (09/2024)
Contents
Introduction
4
2
Notation and Setup
5
2.1
Notation
6
2.2
Setup
7
Background
9
3.1
Deep Generative Models Known Manifolds
9
3.2
Manilold Learning
10
3.3
The Change-ot- Variables Formula
11
3.4
Failures of KL Divergence
12
Wasserstein Distances
13
3.6
Maximum Mean Discrepancy
4
Unaware Deep Generative Models
15
4.1
The Problem with Likelihood Based Approaches: Manifold Overfitting
15
4.1.1
The Unavoidable Numerical Instability of Boah--Dimnssonall Likelihoods
17
4.1.2
Variational Autoencoders
4.1.3
Normalizing Flows
21
4.1.4
Energy- Based Models
23
2
Generative Adversarial Networks
24
Score Matching
24
Manifold-Aware Deep Generative Models
25
Manifodd-wwarenes by Adding Noise
25
5.1.1
Denoising Score Matching
26
5.1.2
Score-Based Diffusion Models
................................
27
5.1.3
Conditional Flow Matching
................................
31
5.1.4
Noisy Normalizing Flows
32
5.1.5
Spread Divergences
34
5.2
Manifold-Awarennn through Support-Agnosti Optinization Objectives
35
5.2.1
Wasserstein Generative Adversarial Networks
35
5.2.2
Wasserstein Autoencoders
35
5.2.3
Generative Networks Based on Maximum Mean Discrepancy
36
5.2.4
Generalized Enseey-Based Models
37
5.2.5
Principal Component Flows
38

---

Published in Transactions on Machine Learning Research (09/2024)
5.3
Mawitold-Awareness through Two-Step Models
39
5.3.1
Two-Step Models Minimize Wasserstein Distance
42
5.3.2
Latent Diftusion Models
44
5.3.3
Injective Normalizing Flows
5.4
Overcoming Topological Obstacles to Manifold Learning
47
5.4.1
Neural Implicit Manifolds
48
5.4.2
Multi-Chart Manitod
49
5.4.3
Disconnected Manifolds
49
6
Discrete Deep Generative Models
50
Conclusions and Future Outlook
51
Weak Convergence Primer
68
Proofs
69
B.1 The Likelihood Instability Theorem
B.2 Formalizing the Link between Two-Step Models and Optimal Transport
72
C Experimental Details
73
3

---

Published in Transactions on Machine Learning Research (09/2024)
1 Introduction
Learning the distribution that gave risc to observed data in IR has long been a central problem in statistics and machine learning (Lehrann & Casella 2006 Murphy 2012). In the deep learning era, there has been a tremendous amount of research aimed at leveraging neural networks to solve this task, bringing forth deep generative models (DGMs). This effort has paid off, with state-of-the-art approaches such as diffusion models (Sohl-Dickstcin et al. 2015 Ho et al.) 2020; Song et al., (021b) and their latent variants (Rombach et al., 2022) achieving remarkable empirical success (Ramesh et al. 2022 Nichol et al. 2022 Saharia el. al. 2022 A natural question is why these latest models outperform previous DGMs, more generally:
What makes deep generative model good?
To answer this fundamental question, line of research has emerged with the goal of understanding DGMs through their relationship with the manifold hypothesis, and using the obtained insights to drive empirical improvements. In its simplest form, this crucial hypothesis states that high-dimensional data of interest often lics on an unknown d"-dimensional submanifold M of R", with d' < D. Studying the behaviour of DGMs when the underlying data-generating distribution is supported on an unknown low-dimensional submanifold of R.D has already proven fruitful: for example, it is precisely those models with the capacity to learn low-dimensional manifolds (such as diffusion models, latent or otherwise) that tend to work better in practice. Similarly, various pathologies of DGMs such as mode collapse in variational autoencoders Kingma & Welling 2014 Rezende et al.
Dai & Wipf 2019), and numerical instabilities in the score function of diffusion models (Pidstrigach] 2022 Lu et al. 2023) or in normalizing flows (Dinh et al.) 2015 2017 Cornish et al. 2020 Behrmann et al.
2014
2021 among others- can be explained as a failure to properly account for manifold structure within data. Despite the existence of various DGM review papers (Kobyzev
ct al.] 2020 Papamakarios ct al.] 2021 Bond-Taylor et al., 2022 Yang ct al.] 2023), to the best of our knowledge none takes this "manifold lens". Here, we present the first survey of DGMs from this viewpoint.
The relevance and usefulness of studying DGMs through this lens hinges
critical assumption: namely, that the manifold hypothesis actually holds. It is thus relevant to justify this hypothesis. There is plethora of arguments supporting the existence of low-dimensional structure in most high-dimensional data of interest, which we summarize below:
Intuition Informally, saying that a subset M of TDD is a low-dimensional submanifold is math- ematical way of capturing two important properties: sense of sparsity in ambient space (M has volume, or Lebesgue measure, 0 in R²), and nolion of smoothness. These are properlies that one can commonly expect of ligh-dimensional data of interest. For example, consider natural images in [o, 1] (assume they have been scaled to this range), where D corresponds to the number of pixels. Imagine sampling uniformly from [o, 1j2 until a human deems the resulting sample to be a natural image. For all intents and purposes, such an image would never be sampled since natural images are "sparse" in their ambient space. Images are also "smooth" in that, given natural image, one can always conceive of slightly deforming it in such way that the result remains natural image. One can also intuitively understand a d"=diiensional submanifold M of RD as (smooth in some way) subset of intrinsic dimension d", which can be thought of as the number of factors of variation needed to characterize a point in M.. Again, most machine learning researchers or practitioners who have worked with natural images will have the tacit understanding that far fewer dimensions than the ambient dimension, D, are nceded to describe an image: for example, images can be successfully synthesized from low-dimensional latent variables (Rombach et al., 2022 Sauer al. 2023). and they can be effectively compressed without affecting how humans perceive them (Wallace) 1992;
Townsend et al. 2019 Ruan et al.] 2021). Through this line of thinking, the manifold hypothesis has been a motivating concept in the field of machine learning from its infancy. Some of the first autoencoders (Kramer 1991) aimed to account for data with low intrinsic dimension. Early unsu- pervised algorithms such as Boltzmann machines (Ackley et al.] 1985) were conceived as ways of learning the constraints that govern complex data distributions. Indeed, the manifold hypothesis is one of the core intuitions behind why neural networks are so successful at learning low-dimensional representations in the first. place (Bengio et al. 2013
4

---

Published in Transactions on Machine Learning Research (09/2024)
Theory The manifold hypothesis helps explain the success of deep learning through more than just intuition. For example, under standard assumptions. the sample complexity of kernel density estimation in RD is exponential in D (Wand & Jones 1994). This well-known result is a manifes- tation of the curse of dimensionality, and highlights the fundamental hardness of learning arbitrary high-dimensional distributions. Yet DGMs succeed at this task, suggesting that. these standard as- sumptions are too loose and do not take into account relevant structure present in data of interest. Making the assumption that the data lies on submanifold of DR is a sensible way of incorporating more structure, and is consistent with theory: the sample complexity of kernel density estimation actually scales exponentially with intrinsic dimensionality (Ozakin & Gray 2009 Berenfeld & Hoff-
mann 2021 even if the data is concentrated around manifold rather than exactly on one (Divol, 2022]. These results suggest that the task of learning distributions when d' is much smaller than D fundamentally more tractable than when there is no low-dimensional submanifold (i.e. d' = D).
This observation not unique to density estimation: the difficulty of classification and manifold learning are also known to scale with intrinsic rather than ambient dimensson (Narayanan & Niyogi 2009 Narayanan & Mitter 2010), making them effectively impossible for nigh-dimensiooal data without additional structure. These results are sigu that manifold structure in data is the reason why deep learning manages to avoid the curse of dimensionality. The triumph of modern algorithms on these tasks thus provides strong implicit, justification for the manifold hypothesis.
Empiricism There are various works which use existing intrinsic dimension estimators (Levina & Bickel| 2004 MacKay & Ghahramani] 2005; Johnsson al.] 2014 Facco et al. 2017 Bac et al.
2021], develop their own, and apply them commonly- used image datasets (Pope et al 2021 Tempczyk et al. 2022 Zheng et al. 2022 Brown et al. 2023). These works unanimously estimate the intrinsic dimension of images to be orders of magnitude smaller than their ambient dimension, and similar studics have been carried out on physics datasets with analogous findings (Cresswell
let al. 2022). All these works provide explicit evidence supporting the manifold hypothesis.
Our goal in this survey is to present an accessible, yet. mathematically precise, view of DGMs from the perspective of the manifold hypothesis. First, [Section 2 characterizes the setup we will consider throughout, and provides a consistent set of notation and terminology with which to describe DGMs. [Section 3 lays out relevant background, covering manifold learning and divergences between probability distributions with a special focus on which ones provide an adequate minimization objective when manifolds are involved. Section 4 describes popular manifold-unaaarr DGMs ie. models which do not account for the manifold likelihood-
hypothesis and [Section 4.1.I we provide our first novel result, showing that high-dimensien based models are unavoidably bound to suffer numerical instability when the manifold hypothesis holds. [Section 5) covers manifold--ware DGMs i.e. models which do account for the manifold hypothesis. We include both popular DGMs which happen to be manifold-aware and models which were explicitly designed to account for manifold structure. Inseccion 5.3.1 we provide a new perspective on two-step models, one of the prodominant paradigms of manifold-aware DGMs: we show that, in addition to their common interpretation as jointly learning a manifold and a distribution, they minimize a (potentially regularized) upper bound which can become tight at optimality of the Wasserstein distance against the true data-generating distribution. In [Section 6 we cover discrete DGMs, before concluding and discussing directions for future rescarch in Section 7
2 Notation and Setup
In this section we present the notation and setup that we will use throughout our work. We try to deviate as little as possible from standard nolation, but we nonetheless prioritize precision and consistency across models. While knowledge of measure theory and differential geometry is needed to understand some technical details of how DGMs relate to manifolds, the core idcas, methods, and intuitions of this area do not require mathematics beyond what is typically known by machine learning researchers. Our intention in this survey is thus to remain accessible to readers with no background in these topics.
5

---

Published in Transactions on Machine Learning Research (09/2024)
Advanced topics In the interest of mathematically-incinee readers, we use grey boxes like this one throughout the manuscript to present content which does require familiarity with topics such measure theory (Billingsley 2012), topology (Munkres, 2014), or differential geometry (Lee) 2012; 2018). Providing a primer covering all the relevant material from these topics would be prohibitively lengthy and thus falls outside the scope of our work. Nonetheless, we do include short summary of weak convergence of probability measures in Appendix A] as this is a particularly important tool from measure theory allowing us lo formalize the int ion that a model learns its target distribution throughout training- even when this target is supported on low-dimensional manifold. The material within these grey boxes is self-contained and is not necessary to understand the rest of our survey.
2.1 Notation
Ambient and latent spaces We denote the D-dimensional ambient space as X, where depending on the particular model being discussed, X' could be R or [0, _jP. Many models use latent space, which we denote as 2, where Z = R.. In most cases the latent space is low-dimensional, i.c. d < D, but in some instances this need not hold. We use lower-case letters E X' and = E Z to denote points, and upper-case letters X X and Z E Z for random variables, on these respective spaces.
Encoders and decoders It will often be the case that consider an encoder and decoder ween the aforementioned spaces, which we denote as f : X - 2 and 2 + X. respectively.
Probability We use the letters p and g to denote densities; an exception being the Gaussian density, which we denote as (s; Jr, Σ) when evaluated at æ, where y and Σ correspond to its mean and covariance matrix, respectively. We write X ~ p to indicate that X distributed according to p. Since we will often need various densities, we use superindices to identify them e.g. px and p will denote densities on X and Z, respectively. In particular, we write the true data- gente-eene density X' as p.a. For space S (e.g. Z X), we denote the set of all probability distributions on S as A(S). We use p to denote the convolution between the densities p and q, i.e. if X1 and X2 ~ are independent, @ g is the densily of X1 + X2. We denote expectations with E, and use a subindex to specify what density the expectation taken with respect to, c.g. Ex=px [.].
Network parameters All DGMs leverage neural networks, in one way or another, to define a density p parameterized by the learnable parameters of the neural network(s). We will often abuse notation and use to parameterize all the generative components of the DGM; e.g. some models involve a decoder go and a prior p6 on 2, and we frequently subindex both components with 0 even if they do not share parameters. In some instances it will be necessary to distinguish between generative parameters, in which case we will explicitly write 0 = 02), using c.g. ge, and Pá, instead of go and p/. respectively. We will sometimes overload notation by using subindices to denote a sequence of model parameters (0,))21 the meaning of parameter subindices will always be clear from context. We use & for all auxiliary parameters; i.c. those that are not needed for generation. For example, an encoder So could have been learned alongside p and go, but it might not be needed to sample from the model. We use H and o' to denote optimal values of these parameters (with respect to the loss being optimized for the particular model being discussed).
Calculus We denote derivatives paadientssJacobiann with V, and use subindex to indicate which variable the differentiation is with respect to. For example, V vzfo(I) and Vzge(z) respectively denote the Jacobians of the encoder and decoder with respect to their inputs (not their parameters), evaluated at x and z; and Va logrt (x) denotes, for a given x, the gradient of the log density of the model with respect to its parameters, evaluated at 0.
Linear algebra We denote the identity matrix as 1, with correspouding subindex to indicate dimension. For example, T corresponds to the D x D identity matrix. For matrix J, we use det J, tr J, and JT
That is, do not the common overloading of notation where p(z) and p(z) refer to different. densities: in our notation these would correspond to the same density evaluated two different points.
6

---

Published in Transactions on Machine Learning Research (09/2024)
p*(A)= (2,,
f p.
+(()) d/)
(T)
Janu
dvvvlM
M
M
x2
32
12
21
2
I
(b)
(c)
Figure 1: (a) Depiction of a full-dimensional density (i.c. a density in the "usual sense") p on RR,,, with
2. The probability assigned by px a region A of IR. its integral over the region, i.e. If p (x1, dg) (b) When the density p is instead supported and M is curve), the integral evaluates to zero, and thus p is not a density in the "usual sense". (c) Formally, in order to recover the probability assigned to A by the manifold-supported density p. the density must be integrated only over M using a volume form (dvolm) on the manifold, SAnm P dvolm which in this case simply corresponds to a line integral.
d"-dimensional manifold M embedded in RD (here, dr =1
to denote its determinant, trace, and transpose, respectively. We denote the & and Eg norms in Euclidean space as || and || [2, respectively.
Formal notation In order to formally discuss probability distributions on manifolds, we need the language of measure theory. All the measures we will consider are defined over or Z, with their respective Borel r-algebras. We remind the reader that, since they are Euclidean, X' and Z are not arbitrary probability spaces. For subset. A of X, we denote its closure in X' as clx(A). We denote measures with the same letters as densities, but with uppercase blackboard style, i.e. P and Q two exceptions being the Lebesgue and Gaussian mcasures, which we denote as X and ((, E), respectively. We use a subindex on À to indicate its ambient dimension; e.g. Ap denotes the D- dimensional Lebesgue measure. We write P < Q to indicate that IP absolutely contimuous with respect to Q We use # as a subscript to denote pushforward measures; e.g. for a measurable function X and probability measure pd on 2, P4Pº is the pushforward probability measure (on X') of T2 through g. We will write the true data-generating distribution corresponding to p as TP and the model distribution corresponding to Pd as p.y. Finally, we use 4 indicate weak convergence of probability measures.
g:
2.2 Setup
The main goal of all the models p considered in this paper is to learn p. Two main assumptions, which most of the works presented throughout our survey follow, are commonly made when attempting to understand DGMs through the manifold lens:
Manifold support As mentioned in the introduction, there is substantial body of work support- ing the existence of low-dimensional structure in high-dimensional data of interest such as images. One way of mathematically expressing this structure is by taking literal interpretation of the manifold hypothesis; that is, assuming that y. is supported on a d' d'-dimensionnll submanifold M of X, where < < and both M and d' are unknown. Several aspects of this assumption war- rant additional discussion. (t) P. being manffold-supported implics that it not a full-dimensional density (i.e. with respect the D-dimensional Lebesgue measure) and it thus a density in the "usual sense". See Figure 1 for explanation. (fi) This assumption is choice about how to
7

---

Published in Transactions on Machine Learning Research (09/2024)
represent low-dimensional structure, and can be relaxed in various ways. For example, one could instead assume that the support of pz has varying intrinsic dimension (Brown et al. 2023). that it has singularities (Von Rohrscheidt & Rieck] 2023; Wang & Wang 2024], or that p. simply con- centrates most of its mass around a manifold - rather than all of it (Divol 2022 Berenfeld et all 2024). (iii) The assumption that p." is supported exactly on a manifold remains nonetheless very useful, even if we believe it be "slightly off"; it serves as first step towards understanding the interplay between DGMs and the low-dimensional structure of the data they are trained on, and will see, insights arising from this assumption explain various empirical observations. (iv) The requirement that d² > 0 simply rules out p being a probability mass function, which are better modelled with discrete DGMs. Our main focus is thus on continuous models, although we discuss their discrete counterparts in Section 6
Nonparametric regime We use the term nonparametric regime to refer to the setting where we assume access to an infinite amount of data, arbitrarily flexible models, and exact optimization. More specifically, training any of the DGMs that we will consider requires minimizing loss that depends on model parameters and which involves expectation with respect to p.; doing so is challenging for various reasons. (i) In practice, expectations with respect to p. cannot be computed, and are thus approximated through empirical averages over the dataset at hand. Writing the losses using expectations with respect to v," is thus assuming access to infinite data. (ii) Attempting to reason about optimal parameter values for any given neural network architecture quickly becomes essentially impossible in all but trivial cases. One way to circumvent this issue is to assume that all the neural networks involved, any density model p used, can represent any continuous function, continuous density, respectively. This assumption of arbitrary flexibility is of course motivated by universal approximation properties of neural networks (Hornik 1991 Koehler et al. 2021 Puthawala et al.) (2022). (iii) Stochastic gradient- brasee optimization [Robbins & Monro 1951 over mini-batches of the non-convex loss is used in practice (Kingma & | 2015), which is not guaranteed to recover a global optimum. Once again to facilitate analysis, it is convenient to assume that all the optimization problems can be solved exactly.
Overall, even though these assumptions are optimistic, they remain popular as they allow for mathematical analysis of DGMs. The nonparametric regime also provides a necessary condition for DGMs to learn y in practice; a model fails to capture p. even in this idealistic regime which as we will see happens surprisingly often for commonly-used DGMs then there no hope that the model can empirically recover p.N in realistic setting.
Formal setup Throughout our work, M will be di-dimensional embedded submanifold of R.. We previously mentioned that we will assume P. is supported on M, and that it admits density p.y. in this grey box we formalize the meaning of these statements, the latter of which can be formalized either through measure theory or differential geometry (although we will not require this formal understanding of p.N, we nonetheless include for completeness):
Formalizing manifold support Intuitively, the support of a distribution the "smallest set of probability 1". To formally capture this intuition, the support supp(P) of a distribution P on & is defined (oggachev] 2007 Section 2 of Chapter 7, page 77)
supp(P)
n C, CEC(P)
(1)
where C(P) is the collection of closed (in X) sets C' such that P(C) = 1. It immediately follows that the support of a distribution is always a closed set in its ambient space. In general, M need not be closed in X, in which case it would be impossible for PX to be supported on M. We nonetheless abuse language throughout our survey (both in the main text and in these formal boxes) and say P.X is supported on M when we actually mean that PX(M (M) 1 and that supp(PXX) (MM)

---

Published in Transactions on Machine Learning Research (09/2024)
Formalizing manifold-supported densities through measure theory The manifold M can always be equipped with Riemannian metric which it inherits from X, making M into a Riemannian manifold. Riemannian manifolds admit unique measure (defined over their Borel sets), AM. which plays an analogous role to that of Ap on R". The measure AM is called the Riemannian measure (or sometimes the Lebesgue measure) of M. We refer readers interested in Riemannian measures to the treatments by Dicudooné (1973, Section 22 of Chapter 16) and Pennec (2006). Then IX can be restricted to M, resulting the probability measure F I defined over the Borel sets of M given by
P. (BB)
for any Borel set B M, and p. can then be defined as the density (i.e. Radon-Nikodym derivative) assuming it exists of this measure with respect the corresponding Rieman- nian measure, i.c.
dPx M dAm In other words p is such that, for any Borel set of X,
(3)
P (A) S (z))dM((z))
(4)
Formalizing manifold-supported densitics through differential geometry When the Riemannian manifold M orientable, the manifold admits Riemannian volume form dvol and p.x is defined the function assuming it exists having the property that
P(N = -f Junm p. d
for every open set U of X, i.c. dvolm "plays the role" the Riemannian mcasure AM in Equation 4 The technical tool allowing us to establish a correspondence between these two views of p. is known the Riesz- Riesz-Markov-Kaan Kakutain theorem (Rudin 1987 Theorem 2.14) which is sometimes also referred to the Riesz representation theorem and should not be confused with different theorem about Hilbert spaces bearing the same name. The Riesy- Markov-Kakutan theorem allows us to assign unique measure to dvol namely XM- such that integrating continuous compactly-supported functions against them is equivalent. In this sense, Am is the natural measure to "extend" dvolm- We point out that when M is non-orientable, even though dvol not defined, integration on manifolds can still be carried out through the above crasuro-thcoreti formulation.
3 Background
In this section we cover background topics and standard tools which we make use of throughout the survey. Expert readers may wish skip to (Section 4
3.1 Deep Generative Models on Known Manifolds
This work focuses on data governed by the manifold hypothesis, wherein the dataset interest is constrained to an unknown d' or-dimensional submanifold of X. However, for some datasets, the manifold is known a priori, and the challenge lies in designing generative model which can learn densities within the manifold. Generative modelling on known manifolds is a distinct task from our focus in this survey, but it closely related, and thus briefly summarize work on this topic below.

---

Published in Transactions on Machine Learning Research (09/2024)
Gemici et al. (2016) first identified deep generative modelling on known manifolds as problem of interest and showed that the thange-of-variable formula (Section 3.3p used to train normalizing flows (Section 4.1.3)
can be generalized to account for manifold structure. However, naively applying this idea can be numerically unstable, so past work has specifically focused on developing this approach further for manifolds such as
tori, spheres, and hyperbolic spaces (Rezende et al.] 2020 Bose et al. 2020; Sorrenson et al.) 2023 Other
works have designed generative models which preserve the symmetrics of data on certain manifolds (Lie
groups) of interest in the natural sciences (Kanwar et al., 2020
Boyda et al.) 2021 Katsman et al.] 2021
Using ordinary differential equations or stochastic differential equations to model data known manifolds
has also been proven to be effective (Mathieu & Nickel 2020;
Rozen et al. 2021 De Bortoli et al.) 2022:
Ben-Hamu et al., 2022 Lou et al. 2023 Chen & Lipman 2024). with the advantage that these approaches can be defined independently of any parameterizalioo ion of the manifold. [Bonet el. al. (2024) recently proposed an approach based on optimal transport (Section 3.5p for generative modelling on known manifolds.
3.2 Manifold Learning
Learning distributions whose support is an unknown manifold M implies learning M as well, at least implic- itly. The field of manifold learning is thus closely related to the main topic of our work. The term "manifold learning" is often treated synonymously with dimensionality reduction, and refers to methods whose goal is to provide useful representation of nigh-dimensional data by transforming it into a -imee-dimeniional space, That representation may provide information about the data such as its intrinsic dimension, yield a uscful visualization in two or three dimensions, or serve a simplified starting point for rownsstream supervised learning tasks. Generally, manifold learning methods fall into three categories: spectral methods (Pearson 1901 Kruskal 1964 Beals et al., 1968 Schölkopf et al., 1998:
Roweis & Saul 2000 Tenenbaum et al. 20000 which rely on the eigenvectors and cigenvalues of some matrix related to the data; probabilistic methods
(Tipping & Bishopp 1999: van der Maaten & Hinton| 2008
Mclnnes et al.] 2018), which treat datapoints
as high-dimensional random vectors whose relevant information is containod in some low-dimensional latent
variables; and bottleneck methods (Rumelhart et al., 1988 Kramer 1991 Tishby et al.) 2000 Kingma &
Welling 2014 Alemi et al.] 2017), which rely on passing information through a low-dimensional bottleneck representation, often using neural networks. We refer the reader to the work of Ghojooh et al. (0023) for a
comprehensive review of manifold learning using this tripartite classification.
The focus on manifold learning in this work is mostly on bottleneck methods such as autoencoders (Rumelhart ct al. 1988) - and their many variants- whose core idca to train an encoder-docoder pair of neural networks to ensure reconstruction, for example through a squared & loss 2
min min Expex [|XX - go (X)( [g] 0,6
(6)
Here, the "bottleneck" refers the d-dimensional encoder output fa(X), which the decoder go uses to reconstruct X. Since x.x is supported on M, the objective in Equation 6 aims to learn the manifold in the sense that it encourages perfect reconstructions on it, or more formally, if x E M, then x = go- (fe- (z)) (z)). We highlight that perfect reconstructions need not always be achievable - even under the nonparametrie regime (Section 2.2h- - due to topological constraints. We discuss this point, which we will revisit in Section 5.4 in the next grey box.
We finish this section with the observation that, despite what the term "manifold learning" might suggest, autocncoders do not by themselves formally learn M even when perfect reconstructions are achieved. One might expect perfectly trained autoencoder to characterize M as the set of possible decoder outputs. However, the encoder may not make use of the entire latent space, leaving some points that would not be seen by the decoder during training, As a result, points = E Z (M) might be decoded outside of M, as illustrated in Figure 2 Alternatively, one might expect an autocncoder to characterize M as the set of points which are perfectly reconstructed. However, some points outside the manifold can in principle still be perfectly reconstructed, as also illustrated in Figure 2 Despite these observations, assuming perfect reconstructions. the set (M) together with the decoder ga- jointly characterize M, since gu- (f (M))
Note that autoencoders are by themselves generative models, but we nonetheless parameterize g with (which we use for generative parameters) for consistency with other decoders the rest. of the paper.
10

---

Published in Transactions on Machine Learning Research (09/2024)
X
før
fØ (M)
a
Z
Z
90*
M
Figure 2: Illustration why autoencoders, by themselves, do not characterize M even if they achieve perfect reconstructions on it. The illustrative point 2 E Z \ for (M) is such that = = (2) & M, so that the of possible decoder outputs does not match M, i.e. go- (2) # M even even though .M is contained in go- (Z) due to the assumption of perfect reconstructions. Additionally, in this example that the set of perfectly reconstructed points does not match M, i.e. {x: E X x go- (far (x)) + M even though M is contained in this set whenever I = go- (før (z)) for every x E M.
X' is perfectly reconstructed,
M; we will revisit this point in Section 5.3 to show that training a DGM on data encoded by for can characterize M.
When are perfect reconstructions achievable? Idcally, the loss in Equation 6 achieves a value of at optimality, which would directly imply that x = go- (Sa (r)). IP. -almost-surely. Under mild regularity conditions (Loaiza-Gann Ganem et al. 2022a). this in turn implies that x = ge- (fe. ge- (x)) for all x M, in which case we say that the encoder-docoder pair (fer.go) reconstructs M perfectly. When this condition satisfied, the restriction for lM is a (topological) embedding of M into Z, is evidenced by the existence of its continuous left-inverse, go-. In other words, the existence of some continuous function that embeds M into Z is necessary condition to achieve perfect reconstructions and thus to learn M..
In general, however, such an f may not exist. For example, as we will see in Section 5.3 it is sometines desirable to set d, the dimensionality of Z, to be equal to d", the dimensionality of M. This precludes the existence f for many manifolds M, such as if M is d' re-dimensional sphere, for which no embedding f . - Rd
d is possible when d = d' In cases where plausible embedding exists, even networks (Ja. go which come close to perfectly reconstructing M will incur numerical instability (Cornish et al. 2020). In some other cases, it is possible to resolve these topological issues by increasing d. For instance, dimensionality of d= 2d + +1 is enough to topologically embed any manifold of dimension in Rd Hurewicz Wallman 1948 Theorem 3).
3.3 The Change-of- ofrVabiabl Formula
It will often be the case that we have a density pd on along with a decoder g Z - X. Together, these two components implicitly define the distribution of X = g(Z), where Z ~ p², and it will often be interest to explicitly evaluate the density x of X (formally, x the pushforward density p through g). The suitable tool the change-of-variable formula, whose simplest form states that, when Z X RR D if g is diffeomorphism (i.c. continuously differentiable function with continuously differentiable inverse), then
xx)) = (2) /det Veg(z)] =p² (f(x)) det v.f(x)),
(7)
(8)
where f g- and z f(z), so that V2g(2) E D and Vzf(x) E DXD, An extension of this formula that will also be of use applies to the case where 2 Rd with d < D. When d < D, g Z X cannot. be
11

---

Published in Transactions on Machine Learning Research (09/2024)
diffoomorphism, but if it is injective it can be diffeomorphism onto its image, 9(2), in which case the density of X is now given by
(x)) (z) |det [V=((>)[Vabl) [
(9)
where again = f(r),, but now 0() + 2 is the left inverse of g (i.c. e' = f(o(z")) for all e' E 2), and V=g(=) E R2 Dxd
Several remarks about these formulas are worth making. (i) Computationally, it is often the case that Equation 8 is used, rather than Equation 7 as Equation 8 requires only a forward pass through the encoder [ {as well computing its Jacobian determinant), whereas [Equation 7 requires an additional forward computation through the decoder g; (ti) Equation 9 which is referred to as the injective
formula, reduces to [Equation 7 in the case where d D, since the determinant distributes over products of square matrices; and (2i1) when d < D, px in Equation 9 is manifold-supported density because it is only defined on a submanifold, g ((2), ) of X, much like p, which is only defined on M. We refer the reader to the work of Köthe (2023) for a review of the uses the lhlange-of-variblle formula has within DGMs.
Note that more formal way of describing the change-of-variables formula is through the language of pushforward measures, where we have measure p2 Z admitting density p" with respect to Xa along with the measurable map g: Z X. In the case of Eqqation 7 and Equation 8 px corresponds to the density of g# g2P² with respect to Ap; i.c. px = dg /ddp. In the case of Eqquaion 9. when g smooth embedding, xx is now a density with respect to the Riemannian measure on g(Z), i.e. p² 1g /d)g(2), where g(Z) is treated an embedded submanifold of X.
3.4 Failures of KL Divergence
Despite the KL divergence being widely used throughout machine learning, is most commonly used with the implicit assumption that the two involved densities are densities in the "same sense" (i.e. when they both admit the same dominating mcasure, see the grey box below). This assumption fails the manifold setting when pd is full-dimensional, since xxx manifold-supportee.. We thus find it useful to provide the formal definition of the KL divergence in the grey box below, along with discussion. In summary, the usual formula for computing KL divergence,
log p px( pd (X)
KL (p. x 11 Ex-px
(10)
is only valid when pxx and P are such that for every subset. of X that is assigned probability by r the density P.º also assigns probability 0 to A. Whenever this property does not hold, KLL ( is defined infinity. It follows that in the manifold setting, [L || dimensional, as illustrated in Figure 3(a) It also follows that (L X 1I = co even if p is supported on a d' ddddimensional manifold as long as M is not contained in the support of pd as illustrated in Figure 3(b)
(L || when pd is full-
The most common way of attempting to minimize the KL divergence between the true distribution and the model is through
KL divergence and maximum-likelihodd
max Ex-pX [logre (X)]
(11)
When the KL divergence between p.d and p is not trivially equal to infinity, it can be written as
KL 11 / log
flogg
p. (x) pp (x)
p (x)da = / p. (x) logg, (r)dx -
f p. log (x)
(12)
Exxppy [logr: (X)] - Exupx [logră (x)]
(13)
Since Ex-py [log vx (X)] does not depend on 0 this common derivation shows that as long as 00, maximum-likellhood optimization equivalent. to minimizing KL divergence.
Exapy (x))
12

---

Published in Transactions on Machine Learning Research (09/2024)
X
X
X
X P
P
X
*
P0
OB
Po
A
A
M
M
(a)
(b)
Figure 3: Illustration of why KL divergences can be infinite in the manifold setting. (a) Pe has full- dimensional support (light red region), while p is supported on a lower-dimensional manifold M (blue curve). The model P assigns probability 0 to A, iee. SA dr = 0, because the region A has zero volume in X. However, p." does not, since JANMP" p. d > 0. We conclude that KL (p.N |l P ) = cc. Mcanwhile, S BnM P. dvoly 0 because BnM- = 0, yet have fB ro dr > entailing that KIL (pz || |) Analogous example where now Pø and p. are both supported on low-dimensional manifolds. Since M is not contained in the support of p there exists set A to which py assigns probability 0 despite having positive probability under p." so that KIL (p 1 xC.
(b)
However, key step in this derivation (the first equality) is the assumption that the KL divergence bet p. and pd is not trivially infinite. As previously mentioned, in the manifold setting we will generally have KL ( = co. It follows that in this setting, maxximuu-likelihooe is not equivalent to KL divergence minimization, a point that we will later revisit.
Formally, the KL divergence between two probability measures P and Q, [L[PP] Q), is defined
dP f log de (x) (x)
/
dP dQ
dP(x). if exists
KLL
(14)
00, otherwisc
where dP//QQ denoles the Radon-Nikodynn derivative of with respect to Q. By the Radon-Nikodym theorem, dP/ dP/dQ exists if and only < Q. Finally, when both and Q are dominated by the same measure 17 i.e. IP 17 and Q < 7 with corresponding densities p and q with respect n, the KL divergence between them simplifies to
/
p(x) ((x))
KL || Q2)
1KT (l)
p(z)dn(ax))
(15)
which recovers the commonly-used expressions for KL divergence Equation 13) whenever n is cither the Lebesgue measure or the counting measure.
3.5 Wasserstein Distances
In Section 3.4 we summarized why KL(p3 || (z²) does not provide useful notion of divergence between p.a and Pb in the manifold setting, and the same is true of many other common divergences between distributions (see the discussion in Section 4.2h. Wasserstein distances, which are based on the optimal transport problem (Villani 2009: Peyré & Cuturi 2019), provide a distance between distributions that remains mcaningful even the manifold sctting. Despite the fact that accurately estimating Wasscrstoin distances is challenging (Arora al. 2017), DGMs based on minimizing these distances tend work very well in practice (e.g. Section 5.2.1 and Section 5.3.1
13

---

Published in Transactions on Machine Learning Research (09/2024)
p
X
Figure The optimal transport problem can be visualized as the minimum cost of "transporting" the density p over to the density q. Picturing p and q as piles of dirt, each dirt particle from p must be moved that it becomes part of q. Moving dirt from x to y incurs cost given by c(z, y). The joint distribution y of (X,Y) can be thought of as specifying the "transport plan": the constraint that its X- X-marginal matches ensures the starting pile of dirt is p; the constraint that its Y-marginal matches ensures the final pile of dirt is and its (Y = co-ooiiiionaal illustrated with the black arrows in the figure specifies how the dirt at æ from is (potentially stochastically) allocated to dirt from The most elficient plan possible for shifting all the dirt has an overall cost (", This analogy explains why the Wasserstein-1 distance is sometimes called the carth mover's distance.
The optimal transport problem between two densitics p and q on X' given by
(p. = inf ((x,)),)d(XX)) gET((-a)
(16)
where c: X' X > IR is called the cost function, and I[,, q) is the set of distributions on X' x X' whose marginals match and q, respectively. Intuitively, the optimal transport problem can be understood as the cost (as measured by c) of "transporting" to 9, as illustrated in Figure 4 When e is given by the f distance (i.e. c(r,y) 3) = | - y)l1) w called the Wasserstein-1 distance, and is denoted as W,, The WL metric admits the following well-known dual formulation:
(1(p, q) = sup Ex (xxxpp((())) - [xxxgh((X). NEH
(17)
where H = (h: X + RRR h is Lipschitz and Lip(h) S 1}, and Lip(h) denotes the Lipschitz constant of h. Analogously, when e is given by the squared & distance (i.e. c(z. v) = || y113) vWe is called the Wasserstein-2 distance, and is denoted as W2. The Wasserstein distances W1 pp and W2(p. pp remain meaningfully defined even in the manifold setting (formally, this is bocause they metrize weak con- vergence, which we discuss the grey box below), and thus provide sensible optimization objectives. For W1, this property can be informally understood through Equation 17] which essentially says that two distri- butions close in Wasserstein distance if no Lipschitz function discriminate between them. Intuitively, if no such function can discern between p and p then they must be "truly" close, even if one is manifold- supported and the other full-dimensional (or if both are supported non-overlapping manifolds).
First, we point out that the optimal transport problem in Equation 16 applies to arbitrary probability measures and Q on X, not only densities:
W (P, == inf lcXX, Y)],
(18)
rEl(P.0)
where 1I(P, Q = (T' E A(X x X) I x X) F(A) and l'X'' A) Q(A) for every measurable set A X}. and c: X x X + R is measurable.
If c is the (or &2) distance, then convergence in W equivalent to convergence in distribution plus convergence in first (or second) moments VVillani 2009 Theorem 6.9). In particular, this
14

---

Published in Transactions on Machine Learning Research (09/2024)
implies that if X bounded (and c either the & or l2 distance), then W metrizes weak convergence, meaning that given sequence (1 of probability measures, WTPP,, IP))) + t + 0o if and only if P2 + Ip t oo. Arjovsky et al. (2017) identified that metrizing weak convergence is desirable property in an optimization objective for training DGMs, as it ensures that "getting closer and closer" to the larget distribution is properly quantified, even in the presence of dimensionality mismatch (see Appendix for a more detailed discussion this point).
Note that the KL divergence does not metrize weak convergence. Let us illustrate why this is prob- lematic through
example by letting P.K, PX (N(0, where PX is supported on M. As a 0+, 1p.x gets closer to P., yet this is not reflected in the KL divergence, since (L(PP [P**) oc for every > 0 (this is because P. & P.x does not hold). Thus KL(P, || + do as o + 0*. This means that, despite KL(P,, {) being minimized at = O, the KT. divergence provides no learning signal, and the same holds for the reverse KL. On the other hand, W'PPP, (P.,,, + 0 as a 0+, provided that We metrizes weak convergence.
3.6 Maximum Mean Discrepancy
Similarly Wasserstein distances (Section 3.5), , the maximum mean discrepancy (MMD; Gretton et al. 20066)
provides notion of distance between probability distributions which remains mathematically meaningful in the manifold setting. The MMD between two probability densities and on X, MMD, (v. q), is given by
MMMD 4) = (EXXxx)((XXXX)) - 2E X [KXX, Y) + [XX, 1
(19)
where X", Y, Y are independent, and X' X + R is a symmetric positive semi-definite kernel (i.e. function having the property that, for any n and 2i....,IT E X, the n x n matrix K given by Kú = 2;) is symmetric positive semi-definite, which is set a hyperparameter.
The MMD has several desirable mathematical properties. (i) Under some regularity conditions which are satisfied by many commonly-used kernels, the MMD is a metric in the space of probability distributions over X. (ii) MIMI 4) can be straightforwardly estimated in an unbiased manner through Monte Carlo sampling, making it particularly amenable to gradient- based optimization. (tii) Conditions on & which make MMD; meaningfully defined in the manifold setting (formally, conditions under which MMD metrizes weak convergence) are known (Simon-Gabriel & Schölkopf 2018 Simonn Gabriel al.] 2023). Provided that X is compact (which is the case for images in [0, 1)"), these conditions hold for most commonly-used kernels, meaning that MMD can be used to compare distributions regardless of their support.
4 Manifold-Unaware Deep Generative Models
In this section we describe popular deep generative modelling frameworks which were not developed with the manifold sctting in mind, and discuss their inability to learn p in this sctting.
4.1 The Problem with Likelihood-Bart Approaches: Manifold Overfitting
Likelihood-based deep generative models are broad and popular class of models, which includes variational autoencoders (Kingma & Welling 2014 Rezende et al.] 2014), normalizing flows (Dinh et al. 2015 2017
ensrgy-based models (Xie et al. 2016 Du & Mordatch 2019), contimuous autoregressive models (Uria et al.
2013), and more Glond-Taylon et al. 2022). At a high-level, these models leverage neural networks to construct a full-dimensional density Pg The models are trained by maximixing, somelimes approximately,
the likeliooo:hoo:
max mxx (logrr (X)).
(20)
When the underlying density p. is full-dimensional, this objective is equivalent to minimizing the KL divergence between p. and pi (Fquation 13/. However, in setting of interest p. is manifold-supported,,
15

---

Published in Transactions on Machine Learning Research (09/202))
Po. (z)
P
M
I22
12
122
22
Po. 2z)
P2
M
122
42
12
21
21
Figure 5: Illustration of manifold overfitting, where the 1-dimensional p (shades of blue) along a curve M in 2-dimensional ambient space is improperly approximated. Each row shows a sequence full-dimensional densities pb, (red surfaces) having the property that their likelihood diverges to infinity each sequence approximates a different manifold-supported density p on M: the top sequence will recover bimodal distribution on M and the bottom sequence a trimodal one, despite p.a being unimodal.
all of M, yet
and as mentioned in Section 3.4 this equivalence breaks down, leading to the natural question: what happens if the likelihood is optimized when pg is full-dimensional but p. is not?
The first consequence of this dimensionality misspecification is that the log-likelihood does not admit a maximum it can be made arbitrarily large. To see this, consider sequence of full-dimensional models (r which concentrate more and more mass around M during training, as depicted in Figure 5) Tf M was full-dimensional, would be impossible to have p (a) + oc as / - xo for all x M, as doing so would However, when M is low-dimensional, it
quickly violate the requirement that the densities integrate is "infinitely thin" in R and thus the model densitics be made to diverge to infinity along the entire manifold. This phenomenon is illustrated twice in Figure 5
At first glance, the fact that the likelihood does not admit maximum might seem inconsequential, as one might hope that as long as Ex=pX [logrč: (X)] + oc as ( + 0o, then n is still being learned. However, this is not the case, and the reason once again illustrated in Figure 3 there are many ways in which the likelihood can diverge to infinity. Loaiza- Ganem al. (2022a) formalized this intuition by proving that under mild regularity conditions, for any manifol--supported density xx on M, there always exists a sequence of full-dimensional densities which simultaneously (i) becomes arbitrarily large the entire manifold, in turn maximizing likelihood, yet (ii) approximates pi² rather than the true data mata-generating density p.X. The latter condition is formalized using weak convergence in the grey box below, but can be intuitively understood as saying that samples from po, and dx become indissinguise as oo
NNote that approximating P.N does not imply that P (x) + pp as oc z because p. is full-dimensional, whereas vx sapifold--upportee...
16

---

Published in Transactions on Machine Learning Research (09/2024)
An immediate consequence of this result is that maximum-likellhoon is an ill-posed objective in the manifold setting, as it simply encourages models to concentrate mass around M with no concern for the distribution within it. Loaiza- Ganem et al. (2022a) thus call this behaviour manifold overfitting. Several consequences of manifold overfitting are worth discussing. (t) Manifold overfitting does not imply that model densities diverge to infinity on all of M; as long as these densities diverge to infinity on a subset of non-zero probability under p and do not converge Ex~pX [log Po, + oo as - co. Analogously, densities could diverge infinity on a superset of M such manifold of dimension higher than d' but lower than D (Koehler et al., 2022). Similarly, the log-likelihood can be made to diverge to infinity in such a way that the sequence of models Pe, does not learn any distribution ot (Loaiza-Ganen et al., 2022a). In other words, the behaviour of models which "maximize" likelihood in the manifold sctting can be pathological beyond pa. (x) diverging to infinity if and only if x E M. (ii) One might be hopeful that in practice these pathological scenarios are avoided through the optimization dynamics of gradient descent so that p.N is properly learned, yet this not the case (Koehler et al. 2022). (Ri)) Manifold overfitting cannot be detected by using test likelihoods: long the test data is generated from p², then it lies on M with probability 1, and thus test likelihoods are also subject to degenerate behaviour. This observation highlights that, in the manifold setting, test log- log-likelhhoods should be avoided as DGM evaluation metric, and that sample- based metrics (Heusel et al. 2017 Borji 2019 Stein et al. 2023) should be favoured instead. This unreliability of test log-likelioods is consistent with the lact that they are not always correlated with sample quality when modelling images (Theis el. al. 2016
zero on the rest of M, the log-likelihooe will still be "maximized", i.c.
The manifold overfitting result of Loaiza-Ganeen et al. [2022a) be more formally stated as saying that, under some regularity conditions and provided that d' D, for any distribution IP, on X supported on M, there exists a sequence of distributions (2)))
i= such that:
F. is full-dimensional, ie. P.. λDS for every
For every æ M, it holds that p (x) - as t oc, where po. is density of PX with respect to Åp.
For every x X' clx(M), holds that (x) + 0 as 00.
px 0.
x
88 + xc.
Loaiza-Ganem et al. (002aa) proved this result under the assumption that M is analytic. Despite their other regularity conditions being very mild, this is strong assumption. However, as now argue, the result actually holds for arbitrary smooth submanifolds M of X. Gray 11974 Theorem 3.1) proved result which immediately implies that, if M is an analytic Riemannian manifold, then
for x E M, as E + 0,
AM (B) = v(d") (1+
(21)
where AM is the Riemannian measure on M, (x) denotes geodesic ball in M of radius & centred at x, and v(d") is the volume of a d*-dimensional Euclidean ball of radius 1. Loaiza-Ganem et al. (2022a) used the assumption that M is analytic only to apply Eqqation 27 by evoking the result of Gray (1974). However, Equation 21 is known to hold for arbitrary smooth Riemannian manifolds Gaaloot et al.] 2004, Theorem 3.98). It immediately follows that the result of Loaizz--aaeee et al.
2022a) indeed holds for arbitrary smooth submanifolds M of X, even if they are not analytic.
4.1.1 The Unavoidable Numerical Instability of High-Dimensions Likelihoods
The manifold overfitting result of Loaiza-Ganem et al. (2022a) described in Section 4.1 establishes that maximun-likelihodd is an ill-posed objective for high-dimensional densities in the manifold setting. Before continuing our review existing work, we point out that their result does not rule out the possibility of somchow addressing the pathological behaviour of maximm--ikellhood. for example by adding regularizer. Here we prove that it is actually impossible to do so, by showing that for any "infinitely thin" subset M of X' (of which M is example, but here do not require M to be a manifold), any density p; supported
17

---

Published in Transactions on Machine Learning Research (09/2024)
M, and any sequence of D-dimensional models Po, which learn Pt the following holds: (i) for any A outside of M, Pē, (x) gets arbitrarily close as -> oe; and (ti) for any x M and any L > O, for large enough it holds that pbz (x') > L for some x' E X' arbitrarily close to æ. We formally state our theorem and include a technical discussion in the grey box below.
X
Technicalitics aside, our result shows that likelihoods become arbitrarily close 0 outside M, and that they become arbitrarily large on it (or arbitrarily close to it). The problem here is twofold: likelihoods are unstable not only because they become arbitrarily large around M, but also because they must change very x rapidly to approach 0 outside of it. In particular, this implies that if pø, is Lipschitz, the corresponding Lipschitz constant must blow up as t 00.
One way to interpret our result is as a "soft gencralization" " of the manifold overfitting result of Loaiza- Ganem et al. (2022a); whereas they show that for any target pz" there exists a sequence of models which approximates while exploding on M and converging to elsewhere, we show that any sequence recovering p. will exhibit similar pathological behaviour on M. Two implications of our result are worth discussing:
Numerical instability of likelihood evaluation Our theorem applics even if the models were not trained through maximum-likelihoodd, so that if the target distribution is correctly recovered through any means, density evaluation will remain numerically unstable even when training itself does not involve likelihoods and is numerically stable. To see this, simply apply our theorem with Pf = =p which immediately yields that any sequence of D-dimensional models which learn p." will do s0 with mumerically unstable likelihoods. We can gain intuition as to why this should indeed be the case through Figure 5) the only way for the rod surfaces (p²)) to recover the donsity on the blue curve (p.)) is by spiking to infinity around it, and by not assigning mass elsewhere.
Unfixability of maximum-lkcelhhod Another consequence of our result is that maximum- likelihood cannot be "fixed"; for example, any regularizer added it which ensures that p, is learned (rather than some arbitrary p²²) would not circumvent the aforementioned numerical instabilities provided it does not obviate the need compute likelihoods (or any surrogates used) during training (e.g. by cancelling out the log-likc,hhoo,, at which point it would not fit the description of regularizer anymore). Analogously, any regularizer architecture guaranteeing numerical stability of likelihoods would be such that p.a is not learned. In other words, our result ensures that learning v and having numerically stable likelihoods cannot happen simultaneously.
Theorem (Likelihood Instability of Deep Generative Models). Let M C X be a Borel set such that (M(c 0, and let FN be a probability measure on X such that P. (M) and supp(PX)) = (M). Let (p be sequence of probability measures X such that I. = Px as t - 0o and Pø. < AD, with corresponding densities p Then:
lif pa, t-xc
x (x) 0, An Ab-almost-eeeryulhe on X clx(M).
x (x') + oc as L + 00 for every x E clx(M) and every e > 0, where
sup (a),
Pe.
Bx(r) = (x' E X || æ|I2 < <s)
Proof. See Appendix B.1
We now make some relevant observations about the Likelihood Instability Theorem (i) Note that we only require the closure of M to have Lebesgue measure 0, it need not be a manifold. Our result thus applics in scttings beyond the standard manifold hypothesis, such as when M is given by union of manifolds (Brown el al.] 2023), or by a non-manifold set with singularities (Von Rohrscheidt. & Rieck 2023 Wang & Wangl 2024h. (ii) lim inf PO, (x) cannot in general be replaced by lim Pě, (x) since the limit need not exist, but as an immediate corollary, if the limit exists, then it must be 0 a--almmtt-eeryytii on X' cl.(M).. (iit) We also point out that sup,'en, P (2') cannot be in general replaced by p (I) either, despite our conclusion holding for every > O. Intuitively, this is
18

---

Published in Transactions on Machine Learning Research (09/2024)
because for any given I E M the divergence to infinity of the density might not happen at æ, but rather on a sequence converging to it: provide an illustrative example in Appendix B.1 (iv) We
do emphasize that despite not concluding that p (x) 0 outside of M nor that vo, (x) + xo in M,
our result does unequivocally ensure numerical instability of the involved densities.
4.1.2 Variational Autoencoders
Variational autoencoders (VAFs; Kingma & Welling 2014; Rezende et al., 2014) are a class of likelihood- based models. The contimuous VAËs that we consider here specify a fixed prior density p² (commonly a standard Gaussian) on Z, along with learnable conditional full-dimensional likelihood Pa x is often a parameterized Gaussian,
on X, which
X (x|2) = -N ( (z:90(2), 20 1x (z))
Pe
(22)
where 2g 2x,12 2 + 12 DxD is symmetric positive definite, often given by YID, where >> 0 is treated as a free
parameter rather than the output of a neural network. The conditional likelihood Po x12 (-12) Equation 22 can be understood a stochastic decoder, whose mean is given by the deterministic decoder go(z). Together, the prior and the conditional likelihood implicitly define the marginal likelihood over data:
Pe (x) - f
p= (=) x12 (xlz)d2..
(23)
Since computing the marginal likelihood involves an intractable integral, variational posterior density 2 on Z introduced, and the following objective, called the evidence lower bound (ELBO), is jointly maximized over and db
96
Epx (0) = Expxp
( F 1 (1x) logr' X|2 (X [2]
KL g 1 (-1x) (107)|1²) |²²]
(24)
Ex
log x (X) IKIL
11 2 X (1x)
p9
21x 2 (1x))] S Exapx Dlogr (X)),
(25)
where Po x|X denotes the true posterior density, which is implicitly defined by the prior p² and the conditional likelihood P& X|Z Note that Equation 24 is used as the optimization objective, since the true posterior density cannot be tractably evaluated. While not directly usable as an objective, [Equatin 25 shows that the ELBO lower-bounds the log- lieliiol xx [log Pe (X)J, and differs from it only by the error incurred by 4 Z|X to approximate the true posterior: this is often used as a justification for using the ELBO as an objective, as it simultancously encourages learning through maximum-likllihodd, and so that q
x matches the true
posterior. It is common to also specify 4s z|x as Gaussian,
(=|x) = (N)))=N(= d
2
(26)
where X X' Raxd is also symmetric positive definite, and often given by diagonal matrix with positive entries along its diagonal. In an analogous manner to the conditional likelihood, the variational posterior % 2|X (-|z)) in Equation 26 can be interpreted stochastic encoder, whose mcan is given by the deterministic encoder Jo(æ).
An issue which commonly affects VAËs is posterior collapse [Chen et al. 2017 Wang et al. 2021), where the learned variational posterior 2g- partially collapses to the prior 2²_ We will shortly explain posterior collapse in VAEs through the manifold lens, and thus we briefly summarize the phenomenon here: in the case where z is taken as a diagonal matrix, a subset of the diagonal entries of 5 (ar) collapses to 1 for M, and the corresponding entries of far (x) collapse to 0, matching the standard Gaussian prior p²: whereas the remaining diagonal entries of Σ (x) are extremely close to 0, essentially losing stochasticity =-
along these coordinates. In other words, VAËs tend to only "use" a subset of the coordinates of their latent
space 2 to obtain data samples and default the rest to the prior.
19

---

Published in Transactions on Machine Learning Research (09/2024)
VAEs through the lens of the manifold hypothesis Despite the autoencoder-like structure of VAEs that leverages low-dimensional representations, VA as presented above are full-dimensional models. This is a direct consequence of the choice of Pe X|2 which always assigns strictly positive density to all of X, i.e. Po x1 (x122 > O for all x E X and e Z. This in turn implics that p (a) > 0 for all z E X, so that the model density is not supported on a low-dimensional manifold. Thus, since the ELBO is maximized as a proxy for the
intuitively VAEs should be subject to manifold overfitting. Dai & Wipf (2019) formally show that this is indeed the case by proving that, subject to some regularity conditions and assuming that d 2 d", for any manifold-supportedd density p on M, there exists a sequence of VAE models parameterized by (0t, d such that: (i) E,p (ft, de) + oo as + 0o; (ii) the VAE models pé, learn rt instead of y²: and (in)) (LL Z'X (-1x) p1 21x (-Ix)) + O as t + oc for every x E M. Although this result predates the manifold overfitting result of Loaiza-Ganem et al. (2022a) from Section 4.1 it can be understood as saying that maximizing the ELBO instead of the log-likelihodd does not prevent manifold overfitting Tt is also worth
noting that while the work of LLaiza-Ganen et al. (2022a) extends the result from Dai & Wipf (2019)
non-VAE VAE models and VAE models with flexible variational posteriors (Rezende & Mohamed, 2015 Kingma
et al. 2016 van den Berg et al. 2018 Caterini et al. 2021a) since if the variational posterior is flexible enough, optimizing the ELBO becomes equivalent to maximizing the log-likelihodd in the nonparametric
Z|X regime Section 2.2) it does not immedialely imply that manifold overfitting can happen when do is Gaussian, which Dai & Wipf (209)) do prove.
Well-known training instabilities of VAEs can be understood as consequences of manifold overfitting. For example, as previously mentioned, it is common to use x=x12(=) SIp for every 2 E 2, where 7 is a free parameter (y is also often treated as non-learnable hyperparameter instead). This purposely simplistic choice made for the sake of training stability, e.g. parameterizing >-p as a diagonal matrix whose non- zero entries are given by a neural network can casily result in divergent training (Lin et al. 2019 Rybkin et al. 2021; the added flexibility of the stochastic decoder of this VAE can be understood as making it more prone to experience manifold overfitting, We also highlight that, when modelling images, the best empirically performing VAEs do not use a Gaussian conditional likelihood pa XI2 Instoad, they trcat pixcls discrete and use categorical Pe x|2 (Vahdat & Kautz 2020 Mathematically, these discrete models are not afflicted by manifold overfitting (we discuss why in Section 6), which helps explain why they outperform VAEs with continuous conditional likelihoods.
Dai & Wipf (2019) provide further insights into the interplay between VAEs and the manifold hypothesis beyond manifold overfitting. In particular, they also show that under appropriate conditions, Gaussian VAEs such as the ones presented above achieve perfect reconstructions, in the sense that encoding and then decoding any x E M recovers This result justifies using VAEs as autoencoders despite the fact that they suffer from manifold overfitting. Additionally, Dai & Wipf (2019) also show that only d' latent dimensions are needed to achieve these perfect reconstruct ions, suggesting that the posterior over the remaining d - d" latent dimensions defaults to the standard Gaussian prior p² due to the KL term in Equation 24 In other words, the manifold lens helps clucidate why postcrior collapse happens.
We now formalize the discussion on the results of Dai & Wipf (2119) which assume Gaussian VA described above, with the decoder covariance given by Σg (2) (2) YID, where y 0 is free parameter that does not depend on z. They also assume throughout that M is diffeomorphic to Rd", which is much stronger assumption than required by Loaiza-Gancm et al. (022a)).
The first result of Dai & Wipf 2019) assumes some regularity conditions, that d' < D, that d d d", and that y is learnable (i.e. it part of 0). The result then states that for any distribution TP on X supported on M, there exist a sequence of VAE models parameterized by (0e, Φ1 such that:
For every x E M, it holds that fxx (x) ++ xo and KL (
2|X (-1x)
21X (1xx) - 0 as l - co.
Po. Note that together, these two limits imply not only that Ex-px [log Pg. (X)] + oc, but also that Epx (0,, Φz) + 20 as t 00. cc.
(o.
x
20

---

Published in Transactions on Machine Learning Research (09/2024)
0. = PX as as - oc, where 8, is the distribution corresponding to the model density pe.
As previously mentioned, Dai Wipl (0199) also show that Gaussian VAEs achieve perfect recon- structions, and they link this behaviour to posterior collapse. To do so, they first consider y > as a fixed hyperparameter instead of being learnable, and write the corresponding ELBO Ep (0,0)))
with corresponding maximizers (7) and o² (7). They then prove that, if d 2 d" and under similar regularity conditions to the result above, for every > 0 there exists 7 (0, such that:
E,p (7) (1)))))))))))))))))))))))))))))) > E,p (0 (7), (1)in))
(27)
In particular this suggests that, when y is learnable, it must converge to 0 to make the ELBO diverge to infinity. An intuitive way lo understand this is as saying that. for small enough decoder variance 7, is preferable to maximize the log Pg X|2 (X ()) term in Equation 24 which in this case boils down
to & reconstruction error weighted by 1/(2y) instead of minimizing the KL term. Dai & Wipf (2019) then leverage this result to show that VAFs achieve perfect reconstructions in the sense that
lim (x) (n) (f)) z, I almoost--surlly. 7-0
(28)
Finally, since P. is supported on d'-dimensional manifold M, perfectly reconstructing point E M requires d' dimensions, and not the full d of the latent space Z: this means that d' latent dimensions are used to achieve perfect reconstructions, and thus the respective approximate posterior variances are sent to O; whereas the remaining d-d' d' dimensions in the approximate postcrior dofault standard Gaussian lo minimize the KL, term in the ELBO. This explanation of posterior collapse was suggested by Dai & Wipf (2019) and it was further formalized by Zheng et al. (2022).
4.1.3 Normalizing Flows
Normalizing flows (NFs; Dinh et al. 2015 2017 Papamakarios et al. 2017 Kingma & Dhariwal 2018
Durkan et al., 2019 Kobyzev et al. 2020 Papamakarios et al. 2021) are a class of DGMs that leverage the change-of-variable
training. NFs construct a bijective neural network 90 Z + X, where Z= X R,,, such that both go and its inverse fo are differentiable
formula to enable maximuu--lkkllhoon
4 A prior density p² is specified (often a standard Gaussian), and sampling from the model pd is achieved through X go go(Z) where Z p² (formally, p is the pushforward of p² through go). The change-of variables formula from Equuatinn provides the maximun-likelihood objective:
vhangell--varialle
max maxExx-p Ex= [logp" xd (fo(X) + log |det V=fo(X]]]
(29)
Constructing the Jacobian (z) through automatic differentiation to compute the log-likelihood and then backpropagate with respect to is computationally prohibitive. To circumvent this issue, NFs are constructed in such way that ensures that det V=fo(x) can be efficiently evaluated in closed-form (or least approximated). thus enabling gradient optimization with respect to 0.
A relevant variant of NF's are continuous NEs, where fo is defined implicitly through an ordinary differential equation (ODE) rather than explicitly constructed (Chen et al. 2018
Salman et al. 2018 Grathwohl et al.
2019). More specifically, an auxiliary neural network vo X [0.T] X used specily the ODE:
dr; uo(e,, t)df, &o E X.
(30)
'Note that in NEs, the "encoder" fo uniquely determined by the "decoder" go. Thus, the "encoder" does require auxiliary parameters é which is why parameterize it with the gencrative parameters 0. We do nonetheless highlight that moving away from this restriction by parameterizing the encoder and decoder networks separately and making them learn invert each other has been attempted (Draxler al. 2024
21

---

Published in Transactions on Machine Learning Research (09/2024)
Under standard regularity conditions this ODE has unique and smooth solution on [0, (Khalil 2002 T.e. it characterizes the trajectory (#a))ej,TT) and thus implicitly defines a mapping from the initial condition to the final point in the trajectory, namely
fo : X Z, Fn ト DT:
(31)
where again 2 = X. Furthermore, under the same conditions that guarantee a unique solution to Equa- tion 30 fo is invertible and its inverse go can be computed by solving the reverse ODE
dy: -o T t)dt,
(32)
yo zт 2,
whose solution is the reversed trajectory (Be) = (IT-
so that corresponds to the map
90 2 + X,
(33)
90 + YT-
Like standard NFs, continuous NF's are sampled from by first obtaining Yo ~ p², and then computing Y7 = (Yo) - which is now done by unmerically solving Equation 32 initialized at yo = Yo. In this case the det term in the change-of-variables formula. takes the form (Chen et al. 2018)
log det Vo.fo( = 1-" tr (V. vo(e:
(34)
thus enabling maximun-likelihood training of continuous NEs through
logn? (oo(Xo)) + {-" (V vo (V (),
max E. Xompx 0
(35)
where X, corresponds to 2, when Equation 30 is initialized at 20 Xo D.a. While the computations used to train continuous NFs through Equation 35 are significantly different than those used for standard NFs, both models are fundamentally doing the same thing: modelling the data as the distribution obtained by mapping a simple distribution p such a Gaussian through bijective function go (defined explicitly implicitly), and training the model via cisimmum-iikeihhodd
Normalizing flows through the lens of the manifold hypothesis By construction, NFs continuous or not are full-dimensional models and are thus susceptible to manifold overfitting (Section 4.1). There are however other pathologies associated with using NF's in the manifold setting. For example, Cornishh et al. (2020) show that if the supports of p² and p. are not homeomorphic P then any normalizing flow which approximates x. must have exploding bi-Lipschitz constant. In the manifold setting, the support of p.N is the d'-dimensional manifold M, which is not homeomorphic to the support of p² iee. RD.. The exploding bi-Lipschitz constant implied by the result of Cornish et al. (2020) entails that, if NFs converge to a distribution on a low-dimensional manifold (whether this is p. or some other distribution), they must do so in a numerically unstable way, which is completely consistent with our result in [Section 4.1.1 These theoretical insights are borne out in practice; for example, [Bhrrmann are numerically ion-invertible. While perhaps initially surprising, this phenomenon is neatly explained by considering NE's through the lens of the manifold hypothesis, which return to in Section 5.3.3
al. (2021) b show that trained NFs
TThe most notable of these conditions is ve being Lipschitz in (with the Lipschitz constant depending x), which will become relevant when discuss diffusion models in Section 5.1.2 Recall that. two spaces homeomorphic when there exists homeomorphism i.e. contimouus invertible function with continuous inverse between them.
22

---

Published in Transactions on Machine Learning Research (09/2024)
4.1.4 Energy-Basee Models
Energy-based models (EBMs; Xie et al.
2016 Du & Mordatch 2019) are likelihood-based DGMs which
construct density pa by specilying it up to proportionality. More specifically, a neural network Eg X - IR called the energy function, is used to define p through
p (x) xe Bo(o),,
(36)
where pá is assumed to be well-defined (i.e. its normalizing constant is presumed finite: fx - oo). While the likelihood of EBMs is unavailable due to the normalizing constant of p being intractable, the observation that
Ve log x (z) Ex [X)E VoEo(z) enables gradient optimization of the log-likelihod,, since
(37)
voEx~pp [logpe x (X)] = Ex-px [e [[l]]EExpppp Ex-px [VoE((X)),
(38)
and the expectation with respect to pg can be estimated through Markov chain Monte Carlo (MCMC)
methods such as Langevin dynamics (Welling & Tch 2011). In practice, the log-likelihood is maximized by implementing the loss
min x [X)] - Ex xx [X)],),
(39)
where 6' stopgrad(O), as provides the correct gradient with respect to
Energy-based models through the lens of the manifold hypothesis Since by construction pa (1) > 0 for all x X, EBMs are full-dimensional models and are thus susceptible to manifold overfitting (Section 4.1) EBM training and sampling are known to be difficult. A common trick to sample from trained EBM is to initialize MCMC chains not from noise, but from previous chains held in a replay buffer. The buffer is a set of MCMC samples from chains that have been advanced by the EBM throughout the entire training process (Du & Mordatch 2019 Grathwohl et al. 2020). Alternatively, a mixture of historical training checkpoints can be used for sampling (uu & Mordatch, 2019). These tricks are necessary because EBMs are prone to mode collapse, especially in high-dimensional ambient spaces (Arbel et al.) 2021 Loaiza-Ganem et al., 2022a).. This mode collapse behaviour is often blamed on Langevin dynamics and the multimodality of the larget distribution, but can be further explained by the large or unstable gradients in the density landscape of a model that has undergone manifold overfitting.
One particularly interesting exception is the normalized autoencoder proposed by Yoon et al. (2021) which employs the reconstruction error of an autoencoder to define the energy function; i.e.
Ex( = 2x - go (fo
(40)
T
where > O is a hyperparameter autoencoder-like structure, normalized autoencoders remain full-dimensional models; this is true of EBMs regardless of the choice of energy function. An interesting observation, which to the best of our knowledge has not been previously made, is that the energy function in Equation 40 lower-bounded by 0, which in turn implies that e (#) is upper-bounded, meaning that pz (x) cannot be sent to infinity by making Eg, (x) arbitrarily negative for x E M. In particular, manifold overfitting can only occur when the normalizing constant fx e Es, (x) goes to O. While is of course possible for this to happen with arbitrarily flexible networks, we hypothesize that EBMs with lower-bounded energy functions might have an inductive bias which helps them avoid manifold overfitting in practice, albeit likely at the cost. of generative quality. This might explain their empirical success at density-based -ittr---iittiibttion detection (Yoon et al. 2023
8 Much like variational autocncoders (Soction 4.1.2), despite using an
7stopgrad is a computational operator, commonly available in automatic differentiation libraries such as Pyrch Paszke et al. 2019), which leaves the forward pass unchanged (0' 0) while ignoring gradicuts when differentiating (V,0 O), (Ex)), L.o. E Ex-px [x(X)] = Ex-ey [X)),, yet VoEx~pX [Ea(X)] = Ex=px [x)) even though VoE VoLx~ag [Eo()) ]] is not in general equal Ex~px [X)).(X)).
Note that the distribution of normalized autoencoders depends the encoder fs, which is why we parameterize it with instead of auxiliary parameters de this encoder need not share parameters with the decoder ge-
23

---

Published in Transactions on Machine Learning Research (09/2024)
4.2 Generative Adversarial Networks
Generative adversarial networks (GANs; Goodfellow et al. 2014; Radford et al. 2015) use a neural network 9e Z + X' called the generator, along with a latent distribution p² (usually a standard Gaussian), to specify the model distribution pz. Similarly to normalizing flows (Section 4.1.3), samples X = =g) from a GAN are obtained by sampling Z ~ p² and transforming the result through Jo (as in NEs, ph is formally given by the pushforward of p² through go); although unlike NFs, d is not required, and rather d < D is the standard choice for GANs, so that ge need not be invertible. GANs are not likelihood-basee and in order to train 9o, they introduce binary classifier he : X - (0,1), which is trained to distinguish between real samples X ~ "pp and generated samples X ~ ~yg The generator De is trained alongside hig so as to make the classifier unable to successfully differentiate between real and generated samples:
models,
min max E Ex=pX [log hx)) + Ezugz [log (1 hh((o(2))))
(41)
Assuming that the classifier is arbitrarily flexible, it can be shown that the above objective is equivalent to minimizing the densen-Shannon divergence, JS, between the true distribution and the model,
min (p= 1!²))
(42)
where JS(p = 1 1p +1) + [1 1p + }.)) This result was first shown by Goodfellow et al. (2014) under the unstated assumption that pe and p." are densities supported on the same manifold for all 0. While this assumption is unrealistic in the manifold setting and should not be expected to hold in practice, the proof provided by Goodfellow et al. (2014) is "correct in spirit", and was later formalized and generalized by Donahue et al. (2017).
Generative adversarial networks through the lens of the manifold hypothesis The Jensen- Shannon divergence J is meaningfully defined even when ( = co. At a first glance this might suggest that optimizing the Jensen-Shannon divergence circumvents issues such as manifold overfit- ting (Section 4.1), which arise from attempting to minimize the KL divergence. However, the gradients of the Jensen-Shannan divergence DS(p." Il pp'))) with respect to model parameters @ will be 0 whenever the supports of v. and rg do not overlap (formally, the Jensen-Shannon divergence does not metrize weak convergence). This property makes gradient optimization futile whenever the support of the GAN has overlap with the underlying data manifold, which Arjovsky et al. (2017) identify cause of training instabilities for GANs. The Censen-Shanmnn divergence is particular instance of an f-divergence (Polyanskiy & Wul 2022 and there is work generalizing GANs to minimize f-divergences (Nowozin et al. 2016). Yet, AArjosky et al. (2017) also show that various other f-divergences, such as the total variation distance and KL divergence, suffer from similar pathologies as the Jensen-Shannon divergence when there is mismatch between the supports of p and p.¹. In other words, although GANs do not suffer from manifold overfitting, they can still struggle to model manifold-supported data. It is however worth highlighting that the manifold-related woes of GANs are fundamentally different than those of likelihood-basee models: the former model (whenever d < D), and the resulting problems are due only to the optimization objective; whereas the latter are fill-dimen models, and are thus misspecified. Still, GANs can remain topologically mis- specified, c.g. when M is disconnccted (Section 5.4.)), but again, this is an inherently different situation than the dimensional misspecification of likelihood-bbsed models.
proper low-dimensional
4.3 Score Matching
Score matching (Hyvärinen 2005) a method to learn full-dimennional densities p.a The main idea is to learn the (Stein) score function, V, log p.N, rather than p itself In order to achieve this, a model p is implicitly characterized by six X' + X, whose goal is to approximate the unknown true score function. The
Fisher divergence, which is sometimes referred to as the Fisher information distance (DsGuupta 2008 and which is defined as
((p q) := Ex-p V x log ((X) - V log (lgg((X) (
(3)
While in machine learning Va log 1o is often called the score function of model, in the statistics literature the score function refers to Vo lo whereas Vz logrg" called the Stein score.
24

---

Published in Transactions on Machine Learning Research (09/2024)
is leveraged for this goal. Ideally the model would be trained by minimizing F(p," pO))
min E Ex=x [1lx V logr,"(( (x)
(44)
but naively doing requires evaluating the unknown Vx log regularity conditions,
Hyvärinen (2005) showed that, under mild
F(p g) Ex~p [IV log logo 2tr lxgg + c(p),
(45)
where c(p) is a term which depends only on p. Since c(p,") ) is a constant with respect to 0, the objective in Equation 44 is thus equivalent to
min E Ex [|| (x) 2tr V (x)]
(46)
which can actually be minimized. Once a model trained, Markov chain Monte Carlo methods such as Langevin dynamics can be used to sample from it, similarly to energy-based models (Section 4.1.4p.
Score matching through the lens of the manifold hypothesis As mentioned above, score match- ing is derived under the assumption that the underlying data distribution p is full-dimensional. While score matching has been extended to known manifolds (Mardia et al.] 2016), we not aware of any work theoretically studying dimensional mispecification within score matching in an analogous manner to how Suaiza-Ganem et al. (2022a) characterize manifold overfitting (Section 4.1) within likelihood-based models. Nonetheless, we should intuitively expect score matching to fail under the manifold setting due to this di- mensional misspecification. To see why this is the case, we begin by noting that p is indeed full-dimensional since si takes inputs from all of X' rather than just M (sg) evaluated at potentially any point in X' during sampling when using procedures such as Langevin dynamies). The score functions si and V loggg are thus different types of objects the former is full-dimensional score function and the latter is a manifold- supported one. Comparing the values of dimensionally iiimensionallyymimmaatceee comparison remains equally meaningless between the corresponding score functions. Consequently, there is no reason to expect Equation 44 to succeed at matching Pg
mismatched densities is not meaningful, and the
px to v. in the presence of dimensional misspec-
ification. This issue was identified by [Song & Ermon 2019 who empirically confirm that score matching struggles in the manifold setting.
5 Manifold-Aware Deep Generative Models
many commonly-used DGMs struggle to learn distributions on unknown manifolds. There are various (not always mutually exclusive) approaches that enable manifold-awarenes,, including judiciously adding noise to the target distribution; using support-agnostic optimization objectives (c.g. those which metrize weak convergence); and two-step models, which carry out generative modelling on low-dinensional latent space and then map back to data space. We review these approaches in Sectiin 5.1 Section 5.2) and[Section 5.3] respectively. In Sececino 5.3.1 we show that (i) two-step models can be interpreted as (potentially regularized) minimizers of an upper bound of the Wasserstein distance, thus establishing a link between these different approaches for achieving maaifold-awaremeee and that (ii) the upper bound becomes tight at optimality whenever an autoencoder can achieve perfect reconstructions. Finally, in [Section 5.4 we cover methods which make an explicit attempt at properly capturing the topology of M. We take a lax interpretation of manifold-awareness throughout, and discuss not only DGMs which are formally manifold- aware, but also those which, while mathematically manifold-unaware, leverage some inductive bias towards manifold-awareness.
As covered throughout (Section 4
5.1 Banifol--Awarener by Adding Noise
When manifold-umwwareneee arises due the mismatch between the dimension of the model and that of the true distribution the case for likelihood-basee models (Section 4.1) adding noise the training data seems like natural solution; this can make the target distribution full-dimensional (e.g. by
25

---

Published in Transactions on Machine Learning Research (09/2024)
convolving the true distribution with a Gaussian) and thus hopefully avoids manifold-related problems. Indeed, dequantization - i.e. the practice of adding noise to data that was discretized so to be able to fit a continuous density model is very common (Thiis et al., 2016: Dinh et al. 2017 Ho et al., 2019 and can be further justified as a way to avoid manifold overfitting. Unfortunately. it has been shown that just adding Gaussian noise is not enough to empirically avoid manifold overfitting (Zhang et al. 2020a.
Dpaiza-Ganen
ct al. 2022a b Even though the theoretical conditions for manifold overfitting do not hold anymore, the new (noisy) target density will be extremely peaked around M (Section 4.1.1), and thus still numerically exposed to manifold-related woes. This observation is consistent with known convergence rates at which DGMs trained on noisy data recover p. (Chae et al., 2023). The lesson here is that adding noise can enable DGMs to learn unknown manifolds, but the noise has to be added carefully. Various methods doing so have been proposed, which we now review.
5.1.1 Denoising Score Matching
As previously mentioned, Song & Ermon 2019) showed that score matching (Section 4.3) struggles to model napiool--supporred data, and they thus advocate for adding noise and using denoising score matching (Vincent 2011) instead. In denoising score matching, the target distribution is not p. anymore, but rather the distribution p."- obtained by adding independent Gaussian noise N( . :0, =0,o²T)) to samples from p." where o² is a hyperpararetter. More formally, p.- := p @ N( :,,:;00,,,,0 ;0,o²Ip). Importantly, adding full- dimensional Gaussian noise ensures that p.-- is always full-dimensional, regardless of the support of pz" Score matching can then be applied to learn a network si to approximate Va logp," through Equation 46 (with p." replaced by p) However, Vincent 2011 shows that this objective is equivalent to
min E X B [Ex-pp' No No (-1xo) [Is (Xx) - Vz, log X- Dog.x..Xo (X, (X. (x.1x))11]]]
(47)
where P X=11X0 (I (zo) = Nixx; ro, o2 is the density of noisy data (denoted X.) given the (un-noised) datapoint X,- = zg. Equation 47 is much casier to optimize than the usual score matching objective (Equa- [ion 46), since there is no need to backpropagate through the trace of the Jacobian of s |Saremi & Hyvärinen
(2019) use the loss function in Equution 47 to learn an energy-based model (Section 4.1.4) on the noised-out data, but derive the loss from the perspective of empirical Bayes (Robbins 1956;; ; their approach can in principle be applied to other noising processes, but only the Gaussian case is implemented in their work.
Despite mathematically avoiding manifold-related hologge,, denoising score matching as presented above faces a tradeoff; setting o to a very small value means that the target density p.K= is closer to the actual suppored data density p. but doing so also means the target density is highly peaked around M and thus might be harder to properly learn (Section 4.1.1). As a way of being able to use small amounts of noise while still efficiently learning the resulting distribution, [Song& & Ermon (2019) propose to use various noise levels. More specifically, they consider fixed noise levels 0 < 0i < 02 < ... < OT, and modify the score function to take the noise level as input; i.e. sg X' x [01, orl + X' is now such that it aims to approximate the score function at all the corresponding noise levels: si ( ..)) & Vos, log Xot for L t= I,..,, T. This new score function is trained with a weighted sum of the corresponding denoising score matching objectives,
[R (Rx,, xo (x0 [11s (No,, 0z) - Va., logp Xx, 1x0 Xx (Xo,((Xx))))) (x),
min 8
(48)
where w(()= > O is a pre-specified weight coefficient which aims to keep the T terms in the sum at roughly equal magnitudes. The intuition behind using varying noise levels is twofold. (i) Learning the score function for larger values of a is easier, and thanks to parameter sharing (0 is the same for all noise levels), doing so is helpful for learning the score function for small values of a. (fi) Once the model is trained, different noise levels are also used within an annealed sampling scheme. sg. ( .,or) is used alongside Markov chain Monte Carlo to generate sample, which is then used to initialize another Markov chain that now uses X
sp- ,0r=1); this process is repeated until sx. (1))) is used and works much better than only using
sx ( (1))). Although this scheme produces approximate samples from ps Xo, rather than from p. as long as an is small enough, the difference is negligible in practice.
26

---

Published in Transactions on Machine Learning Research (09/2024)
5.1.2
Diffusion Models
Song et al. (2021b) proposed score-based diffusion models as an extension of denoising score matching (Section 5.1.1p p where there is a continuum of noise levels. Formally, they achieve this by constructing (X)))e(0,7]) where X, X for every e [), as an Ornstein Uhlenbeck process given by the Itō stochastic differential equation (SDE) 10
dX,- B(t) x,dt + / B((d)B.,
(49)
X6 ~ pp"
where 3: (0,7] R4 is a hyperparameter (often an alfine function, iee. 3(1) = Amin + (8max Pmin) ///T where 0 < Amin < 9maxx) and (B; (B.)TE(o.T] denotes a D-dimensional Brownian motion. Other choices of SDE are possible, but we focus on the one above which is often referred to as variance preserving SDE since it assumed in some of the theoretical results that we will shortly discuss. Under mild regularity conditions, the SDE admits a unique solution [Oksendal 2003). thus characterizing the density p of X for every E [0, T and prescribes how to progressively transform the data density p =p,"" into the noisier density pxTr. . Note that, due to the added Gaussian noise (from the Brownian motion in Equation 49), p.x is full-dimensional density for every & (0, regardless of the support of p.".
Reversing (X)) provides way to transform samples from Drrr into samples from p.
The reverse
process (Yx)eeoo.r] == (Xr-r)<e(o.r) also obeys an SDE (Anderson 1982:
Haussmann & Pardoux
1986 II
( Yi
dy, 3(T -t
Vv, logp, x (Xi) dt + ((TT-t)B,
2
(50)
Yo ~ p Xr.
The main idea of score- based diffiusion models is to leverage this reverse SDE to build a generative model. In order to achieve this, some approximations are needed. First, since pe² is not known exactly, Equation 50 is initialized at a known distribution p. pxx=, Formally, (Yi)cE(o.T) is approximated by (E))ee(o.T) where
Yi V. logp: (rxpxxr((())) (8i) dt + VB(T -))B,,
dÝ, = 3(T -t)
(51)
Yo ~ p." xx
We denote the density of Yú as pxr-e, and will shortly explain how px->> is chosen so as be close to prr The score Vg. log Xr- (Y) is also unknown, and thus must be approximated as well. Score- diffusion models leverage neural networks to construct sø X' x (0, + X' with the goal of approximating this function, i.c. sy tz, = Va logg," (x) for all æ E X' and t E (0, T). We will also soon explain how this network is trained, but for a given si² (fi)ee is approximated by (fr)eeto:)) where
Y
so (Y.,T )1) dt + VB(T tjdB,,
a². B B( t)
2
(52)
Yo ~ p.x Xm0
We denote the density of Vz as 10 Ideally, a model sample would be obtained by perfectly solving Equation 52] In practice this SDE must be discretized and a numerical solver must be used. The model distribution P is thus given by the approximate solution of Equation 52 at time T.
Reeaders unfamiliar with SDEs can understand Equation 49) through its Euler-Maruyama discretization: split Jo, 2] into 2
sub-intervals of equal length Ar.. T T/n, sample Xa.- ~ p. and set XtR41.n Xtx.n
gitu] nr. I / /B(+c) B. in for O..... n - 1, where tú = kd,,,, and where AB., = BLu. - B,, ~ (V( :0, At, are independent. This procedure characterizes Nt. at the times & for & = O,...,,, and linearly interpolating between them yields continuous stochastic process (X+.n)effo.T) the limit of which as
corresponda to the process specified by the SDE.
Note that the Brownian motions in Equation 49 and Equation 50 are not in general the same Brownian motion (they just have the same distribution), but we do differentiate between them for notational simplicity. Note also that Equation 50) differs from the corresponding equation in (Song et 2021b) since in our notation corresponds to -dt theirs.
27

---

Published in Transactions on Machine Learning Research (09/2024)
In summary, diffusion models aim to solve Equation 50, since perfectly doing would yield samples from p,,, but this is impossible and three sources of error have to be introduced to approximately solve this equation. (i) pe is unknown, and is thus approximated by px=; (ii) the true score V loogpp. also unknown, and is thus approximated by sø (z, T t); and (tii) the resulting SDE in Equation 52 must be solved numerically, inducing discretization error.
We have not yet discussed how diffusion models are trained. Before doing so, we point out that Equation 49 has the known transition kernel
X.lXo X = N (1i (m5V11 07 ,3 oi1n)
p-
(53)
Xx (-(ao)) is the conditional density of X, given that Xo = zo, and where
where p.
of =1 9(s)d=
(54)
Thanks to Equation 53 V 2t log Xo (mo) can be ovaluated, and sampling from px Xx ( (r)) is very
straightforward. Together, these points imply that denoising score matching (Section 5.1.1 provides a tractable objective for training diffusion models
min / w())Exo~nx (xxmpX)NN((X))) [1ss]] (X,,t) - Vz, log px x,Ixo (x, [x)[X dt,
(55)
where w: [0, + R4 is a weighting function (set as a hyperparameter).
Additionally, as long as 3 is such that of + as T + oc, Equation 53 also implies that p.
Xxl (-Iaa) stops
as T oc: this observation provides an avenue for approximately sampling from p.T,, namely by setting p. to a standard Gaussian.
depending on zo in the sense that it converges to NV (0,00,07)) ;0)
Finally, Song al. (2021b also show that the SDE in Equation 49 is intimately linked to the ODE
dx, 3(1) (I, V,, (0gp.((x)) dt,
(56)
Tu E X.
These equations related in that, under some regularity conditions, if the ODE is initialized zo Xo p.x, then #r will have the same distribution Xy- Equation 56 can of course not be sol because the true score function is unknown, but it can be approximated by replacing it with the learned score function, resulting in the new ODE:
3(1) dă.
(Ā, + s/x (f), ()) dt,
(57)
fo E X.
This equation allows us to interpret diffusion models as contimuus normalizing flows (Section 4.1.3): Ug- (x. in Equation 30 is given by - p(t) (x + six (I. The connection between diffusion models and continuous NFs has two relevant consequences. (i) It allows for an alternative way of sampling from them: instead of solving Eqqation 52 the ODE in Equation 57 can be reversed in time as in Equation 32 to obtain
dg. 3(T -t) (ge sxx (br. T f)) dt,
(58)
d0 E X.
12While the target conditional densities in Equation and Equation 55 respectively differ in that the mean of the latter is scaled by /1 of, the result of Vincent 2011 which justifics denoising score matching can be easily adapted this scaled setting, so that Equation 55 score function V., 2o Xi.
i.e. A (ata of Tp) and M(ms / o7 of Tp).
minimized when o (.,t) matches the true
28

---

Published in Transactions on Machine Learning Research (09/2024)
Initializing this ODE at do = Yo ~ p.- (which plays the role of p in continuous NFs) and solving it will then result in samples from pe if the score function was properly learned, and provided that Equation 57 and Equation 58 admit unique solutions which are inverses of each other. (ii) The connection to continuous NEs is also used to justify using the chango-of- formula (Equation 34) for evaluating the density Xo implicitly defined by 80
Diffusion models through the lens of the manifold hypothesis There are deep connections wetween diffusion models and manifolds. First, score-based diffusion models are linked to -ilinuum-ikklohood.. For example, Sohl-Dickstein et al. (2015) and Ho et al. (2020) formulate diffusion models not through SDEs, but through variational inference. In this formulation, the time interval [o, T] is discretized, Xo still corresponds to data, and X, for > is treated as a latent variable. Then, the (discretized) forward process from Equation 49 corresponds to fixed variational approximation to the posterior distribution (of latents given data), and the backward process from [qquation 52 provides a likelihood term (of data given latents). The resulting model, which is reminiscent of a variational autoencoder (Section 4.1.2), can be trained either by maximizing an ELBO (similar to [Eqution 24) or reweighted version of it |Song et al. (2021a) establish another connection belween score-based dilfusiin models and maximun-likelloon that p.º is a full-dimensional density (which does not hold in the manifold setting) and some other regularity conditions, the denoising score matching objective from Equation 55) with a specific choice of weighting function w, becomes equivalent to minimizing an upper bound of KL(p," |l ()²")) which becomes tight at optimality In other words, when y is full-dimensional and for a particular choice of w. diffusion models are likelihood- based models (provided that one ignores the approximation error between p."r and pxx=, and the discretization error, but intuitively both of these errors can be made small by choosing a large T and making the discretization sufficiently fine, respectively).
under the assumptiin
Naively, both of these views of diffusion models suggest that they are a likelihood-based method and thus susceptible to manifold overfitting (Section 4.1p. However, a competing intuition is that the denoising carried out through the backward SDE (Equation 50 or its approximation [Equation 52) effectively projects noisy data onto M by removing the noise (Kadkhodaic & Simoncelli 2021]. Fortunately, it is the latter intuition that turns out to be correct: note that diffusion models do not only aim to learn the true manifold- supported data density p.x = noNo, but also its noisy full-dimensional versions p.x for every t E (0, 7j. Indeed, the objective in Equation 55 recovers the score function for all t E (0, TJ, and stopping the reverse process from Equation 50 at time T - instead of T results in a sample from p.v.. Since p.a is full- dimensional for 1>>00>0000 0, we should not expect maxinun-likelihood to fail at learning it, and by continuity of the solutions of Equation 50 we should expect p:O to be properly learned, even under the manifold setting. Pidstrigach (2022) formalizes this intuition, proving that under mild assumptions, ioo and p.¹ have the same support, and that KL(P," (0)x) I [p || p. ) Since 8 is the distribution of the model under the assumptions of no discretization error and having perfectly recovered the true score function (justiffedd by the nouparametric regime assumption from Section 2.2h, the fact that KL(p,r [|pX=) + 0 as T + oc then implies that perfectly trained score- based diffusion models properly learn p.N. [Pidstrigach (2022) also shows that if the score function is well approximated, in the sense that || (z, - V. log p." le is upper- bounded (with the bound not depending on x nor (), then pyDo has the same support as p. i.e. the model (assuming no discretization error) has the same support as the data. Although this result does not guarantee that diffusion models recover their target distribution, it does ensure that they recover the correct manifold, even if there is some error in the learned score function. De Bortoli (2022) further refined these results, finding an upper bound for W, Wi(p,, .gg)) under reasonable assumptions (recall that Pi corresponds to the approximate solution of Equation 52] at time T). This upper bound depends on T, the error between the true and modelled score functions, and the step size of the discretization used to solve Equation 52) the upper bound goes to O as these quantities ities go to do, 0, and 0 al appropriate rales, respectively. All of this entails that score-based diffusion models can learn distributions on unknown manifolds.
13 is worthwhile to highlight that Kwon et al. (2022) proved similar result with Wasserstein distance (Setionn 3.5 namely that the denoising score matching objective used to train diffusion modela provides, up to scaling and constant factors, an upper bound of the W2 distance between the model and p.¹ which becomes tight at optimality. Unfortunately, Kwon et al. 2022 also assume pe is full-dimensionall and thus their result cannot be used justify the manifold-awareness of diffusion models. Note that even though p. and x are not full-dimensional densities, the KI. divergence between them meaningfully defined because they have the same support. (Section 3.4).
29

---

Published in Transactions on Machine Learning Research (09/2024)
X'
X
u T-
(1, T E)
x V u(x;0) 0)
YT-
uuYo 0) Yo
M
M
(a)
(b)
Figure 6: (a) Informal illustration of why the score function explodes. Consider Y; = x for some fixed æ outside of M as increases from 0 to T. Since diffusion models learn manifolds, Yr must be in M. Thus, when gets "infinitesimally" close to T, the diffusion must push z onto M by moving it in the direction of the drift term in Equation 50, i.e p(Y,, == 3(T - (Y, + Vg. logp= logpxr-1 (r-<(())),,, but using only an "infinitesimally" small step size. Since x not "infinitesimally" close to M, ncods to move a nonninfinitesimal"" amount in an "infinitesimal" time step: the only way in which this can happen is if the norm of the drift, and thus of the score as well, explodes to infinity. (b) In practice, the norm of the score function diverges not only for fixed points (I|>)² (x, t)ll2 + oo), but also along generated trajectories (I|s) (Y,, T - +))l2 + oo).
Pidstrigach |022 also argues that to properly learn the manifold the score Vglogpt p. xx-<(x>> (x) must explode to infinity at time T, i.c. V log (x)ll - oc as t + T- for every x outside of M.. This behaviour, which we illustrate in Figure 6(a) is easy to understand intuilively: Equation 50 results in Y/ ~ T which must be in M because, as discussed in the previous paragraph, diffusion models learn manifolds. Now, fix x outside of M. Since Y₁ is full-dimensional for every t T, it follows that x is in its support, so that x is a possible value for Y, This means that when t is "infinitesimally close" to T, the norm of the drift in Equation 50| i.e. B(T - t)]l=/2 + V, log p (x))(,, must be "infinitely large" to move x to M, since x is outside of M. In other words, the score must explode to infinity. It follows that if sy (z, ( - t) closcly matches Va log Xr '(x), it must also diverge as - T- Lu et al. (2023) formalize this intuition, showing that under some regularity conditions, not only does the score function explode when d' < D, but also that it does not when d' = D. This phenomenon justifies an often used parameterization of score functions: rather than directly taking si as a neural network (which would be continuous on the compact interval [o, 7) and would thus achiove its maximum, making it impossible for the function to diverge and approximate the true score function), sy is taken as si (x,() = si (ar, 1)/or, /r,, where &i is the neural network. Empirically, it has been observed that diffusion models produce trajectories with exploding score functions (Figure 6(b) i.c. that ||> (Y, T (9--)) t)ll2 + 00 as t + T (Kim et al. 2022), which explains the common practice of not running Equation 52 until time T, and instead stopping at time T e for some small > 0 (Vahdat et al., 2021]. The results of Pidstrigach (2022) and Lu et al. (2023) showing that the score is unbounded in the manifold setting thus strongly hint at why these trajectories have exploding scores 15
We also highlight that there is no contradiction between diffusion models being able to learn distributions unknown manifolds and the fact that they can be interpreted continuous normalizing flows (which cannot do so because of manifold overfitting, see[Section 4.1]; the former are trained through denoising score matching instead of maximun-likelihood.. We note as well that the function vg- (a, which allows us to interpret diffusion models as continuous NF's through the ODE in Equation 30 is not Lipschitz in t when the score function blows up to infinity at time 0. In turn, there is no immediate guarantee that Equation 57 has a unique solution, so that dilfusion models need not be properly defined as continuous NFs; this is of course consistent with the numerical instabilities which render their likelihoods unreliable [Section 4.1.1 and Section 4.1.3).
8(t) + sg. (z, ())
Note that the result. that [,, T 1)]l2 + 00 as 1 T for every x outside of M does not formally imply that | (Y, T - t)lla + no with probability 1, even if Y, outside of M for < T, since converges Yr E M. Nevertheless the score function exploding at fixed points shows unbounded, which is necessary for the trajectories to blow up as well.
30

---

Published in Transactions on Machine Learning Research (09/2024)
Overall, we believe this view of score-based diffusion models through the manifold setting is particularly illuiinatin, as it justifies their strong empirical performance (unlike many other popular models, diffusion models can properly learn manfold-supportedd distributions) and a popular parameterization of the score function, while also explaining its exploding behaviour at time 0.
Lastly, we highlight that the progressive noising of data employed by diffusion models (Equation 49 or discretized version) has been adopted in contexts beyond denoising score matching: Xiao al. (2022)
combine it with adversarial training (Section 4.2 and [Section 5.2.1p, and Tran et al. 2023 leverage it for manifold-aware pontimum-likellbbon training of DGMs.
Here we briefly formalize some of the conclusions of Pidstrigga (202)) discussed above. The statement that gNo and p. have the same support formalized as Р < P. and P < fxx, where P.o is the probability measure corresponding to Too. Note that this statement implies that fexo and Px must have the same sets of measure 0, and thus of measure 1 as well, and since the support of a distribution depends only the sets to which it assigns probability 1 (Equation 1), these two probability measures must have the same support. Similarly, the statement that Pø- and p have the same support is formalized as p < PX and PX < fo where foXo the probability measure corresponding p
5.1.3 Conditional Flow Matching
Contimuous normalizing flows (Section 4.1.3) as defined through Eqqation 30 were originally trained through
maximun-likelihood. As discussed [Section 5.1.2] diffusion models can be interpreted as providing an alternative training objective for continuous NFs. Conditional flow matching (CFM; Liu al. 2022 Albergo
& Londen-Eijjden 2023 Lipman et al. 2023 provides additional alternatives, the main variant of which
cover here.
Recall that diffusion models parameterize and learn a vector field (s approximating the score function) by allempting to regress against the true score,
min / ((t)E x, 11 (X,,t) - Va, log p. xxxx (x))
(59)
where we will assume w(f) = 1 to better highlight their similarities to CFM. As discussed in Section 4.3 and Section 5.1.1 directly optimizing Equation 59 cannot be done in practice because p.¹ is unknown. However, by conditioning on Xo, the equivalent yet. tractable obbjectve in Equation 55 is obtained.
Consider "true" vector field v. : X x [0,7] + X in the sense that its corresponding forward ordinary differential equation (Equation 30) maps samples from p. at time t = 0 to samples from p² at t T. Note that even when such a vector field exists, it is not unique. Similarly to diffusion models, CFM learns vector field uo by attempting to regress against v., i.c.
min I'" E Exiop I1lla t) vn (X, I)
(60)
where yxx+ now corresponds to the density of x, if dr, u. (2,, f)dt is initialized at 30 = X, p.. Again, Equation 60 cannot be directly optimized because v., and hence also pxx,, are unknown. Conditioning is once again the solution, except now conditioning is done on both Xo and X7 which allows for explicit construction of target vector field that maps Xo to Xr. There are many such vector fields, but the simplest is the vector field (X,- - Xo)/T (Tong et al.) (024).. Since this vector field is time-independent, sampling from the corresponding p.x, becomes easy as one can take X, = ((T-() - Xo + (XT)/T, where Xo ~ p. and X7 ~ p² are independently sampled. In summary, Uo is trained through
min / E "ExopxXXrne Xropp
(
(T
1) + LXT ,t)
X7 - Xo
dt.
(61)
T
T
Remarkably, when p. and 7d are both dill-dimensional, and under some other regularity conditions, the conditional optimization problem in Eqquation 61 turns out to be equivalent to the unconditional optimization
31

---

Published in Transactions on Machine Learning Research (09/2024)
problem in Equation 60 for a particular Us- Hence, under these conditions, the vector field eg- optimized under Equation 67 can still be used for generation with the reverse ODE in Equation 32
Conditional flow matching through the lens of the manifold hypothesis We begin by pointing out that Lipman et al. (2023) add small amount of noise to the data while applying CFM, which enforces full-dimennsonnality of the target density, However, if no noise is added, the known correctness guarantees for CFM break down when p." is supported on a low-dimensional manifold. Kingma & Gao (2023) proved that, surprisingly, the objectives in Equation 55 for diffusson models and Equation 61 for CFM are eqqivaaentt given appropriate hyperparameter choices. Since diffiusion models are manifold-aware, at first glance this result might seem to imply that, CFM must also learn distributions on unknown manifolds. However, this does nol follow from the result of Kingma & Gao (2023): the manifol--awareness of diffuusion models is guaranteed when the backward stochastic diffcrential equation (Equation 52) is used to sample from the model, whereas CFM always uses an ODE (Equation 32) 16 To the best of our knowledge, there is currently no formal analysis of CFM under the manifold hypothesis analogous to that of Piidstrigach (2022) or De Bortoli (2022) for diffusion models [7] Nonetheless, due to its similarities with diffusion models, we conjecture that CFM is indeed manifold-award -wware, which would be consistent with its strong empirical performance. We do however point out that if CFM successfully learns its target distribution in the manifold setting, then it must experience the numerical instabilities of likelihood evaluation that we established in [Section 4.1.1] even if these instabilities do not manifest themselves during training since likelihoods do not appear in [Equattoon 61 Finally, also highlight the work of Kapusniak et al. (2024), who point out that X, linearly interpolating between data Xo and noise XT has some undesirable properties; they thus propose modification of CFM with the goal of ensuring that the interpolations remain close to geodesics in M, which they show leads to empirical improvements.
5.1.4 Noisy Normalizing Flows
Several models have been proposed based on the idea of adding noise to the data, training a normalizing
flow (Section 4.1.3) on this noisy data, and then having a "deflation" procedure whose goal is to sample
from p. rather than its learned noisy version. One such model is the denoising NF of Horvat & Pfister
22021 ; which itself is heavily based on the theoretical results of Horvat & Pfister (2023 Motivated by
the inherent struggles of flow- base architectures
manifold-supporterd data (described in Section 4.1 and
Section 4.1.3). denoising NFs attempt to model an "inflated" version of the data distribution with added noise, and then provide conditions under which this noise can be "deflated" to recover the true on-manifold density. Similar to denoising score matching (Section 5.1.1j, the inflation step of Horvat & Pfister (2021)
consists of simply adding full dimensional Gaussian noise to the data, and building a density model for p.N = p. @ (N(-:0,0IIp) where a > 0 is a hyperparameter. On the other hand, Horvat & Pfister (2023)
discuss more to X ~ p. along the normal space of .M at X 19 While determining the normal space is only possible for
inflation step, where (D - ddimeesiooal Gaussian noise is added
known manifolds, the authors argue that when d' is much smaller than D, using full-dimensional Gaussian noise is a good approximation of (D-d") "D-dd")-dae dimensiooal noise in the normal space. We will shortly review the deflation step.
To model pKx,, Horvat & Pfister 2021 use a full-dimensional DGM ph X= Their specific choice is a D- dimensional NF, albeit with a sta-stadd learnable latent distribution p For some hyperparameter d meant to approximate d", the first d latent coordinates are themselves modelled by d-dimensional NF, and the remaining D - d latent coordinates use zero-mean Gaussian with covariance I²I matching the
In turn, the result of Kingma & Gao 2023) does ensure that if the vector field learned through CFM is converted back to a score function and used alongside the backward SDE for sampling, then the resulting model will be manifold-aware. However, this is not how CFM is sampled from in practice. Note also that any result ensuring that [qqation 58 can indeed produce samples from p. in the manifold setting would in turn guarantee the manifold-avareness of CFM.
The closest analysis we are aware of is due to Gao & Zhu (2024), who provide Wasserstein distance bound for the ODE sampler. However, this bound requires assumptions which incompatible with the manifold setting, and thus does not ensure manifll-awarrness. This stands in contrast with the analogous bound for the SDE sampler by De Bortoli 2022 which does guarantce manifold-awarencss.
16 Horvat & Pfister (2023) was released first on arXiv despite having later publication date than Horvat Pfister 2021 19 the geometric sense, not the Gaussian sense.
32

---

Published in Transactions on Machine Learning Research (09/2024)
noise level of the inflation step above. The overall likelihood for their full-dimennional model is thus
Pb x (x) - 2 (21)p /det V=Jo(a)],
(62)
where: 90 with being the D-dimensional NF, == 22) = fo(x) with 21 E R² and 22 E RD-4, and the latent density pg is given by = (2)) (22), where P3 21 is the density corresponding to the d-dimensional NF model, and ((: first d latent coordinates are inseesivsse i.e. they are meant to denoise the data while the remaining D D-d d coordinates are noise-sensitive. The model X. defined above has no explicit manifold-awareness Horvat & Pfister|P (021) thus add an injective flow construction (reviewed further in [Section 5.3.3) into the mix, defining the actual (denoisod) manifold-supportedd generative process P by first sampling Z1 ~ pg and then setting X go ((Z1, 0)), where 0 E IR and (:--) denotes concatenation so that (Z,,0) 0) E RR..
with no trainable parameters. The idea is that the
Horvat & Pfister (2021) h refer to using the mappooed-supported p rather than the full-dimensional Pa
X.
"deflating" p."
Horvat & Pfister 2021 also follow injective flows (Equation 97) in adding reconstruction term regularizer to encourage manifold-awareness of the overall objective, which can be written as
max E 2r [108rR- x- (Xx)) - B -B x [1Xx, - go ((o((x)),
(63)
where / > 0 is hyperparameter, fo E Rd corresponds to the first d coordinates of fo(X,), and once again 0 E P-d. The regularizer encourages ge to not need Zz = fo (Xx)) (i.e. the last D - d coordinates of fx)))) to perfectly reconstruct X,: this can be intuitively understood as providing an inductive bias which promotes capturing the added noise only through Z2, thus justifying the use of pa instead of pp
We now discuss the "deflation" " aspect of this approach. Horvat & Pfister (2023) prove that a trained denoising normalizing flow p recovers p." under the following conditions: (i) the noise added to X ~ pt is only added along the (D- - d' dimmesiooall normal space to the true manifold M at X, (ti) the noise parameter o is sufficiently small, and (iti) the manifold M is "sufficiently smooth and disentangled" (this condition is properly formalized by Horvat & Pfister (222 ).
Condition (i) is not satisfied by the denoising normalizing flow, although Horvat & Pfister 2021 argue is a good approximation for the (D d²) dimensional Gaussian noise in the normal space; it is also worth reiterating that it would not be possible to add noise in the normal space without knowing the true manifold exactly in the first place. Condition (iit) is also worth discussing: it is entirely unclear if natural data observed "in the wild" is sufficiently smooth and disentangled, so we may not have any guarantee of retrieving the true manifoldssupported data distribution even in the nonparametric regime. In summary, despite denoising NFs being an elegant approach, their manioold-wwareness can only be ensured under potentially strong and terr-fy-verify assumptions.
that when D is much larger than d',,(( WN(:;,,00
Postels et al. (2022) proposed a very similar approach to denoising NFs, except the latent density used to define ph is fixed as a standard Gaussian, no regularizer is used during training, and the "deflation" step is done by first sampling Xa ~ pg.
Xe and then solving
X = arg max log pp (x) - -111x - Xo.
(64)
Although Postels et al. (2022] do not rigorously justify their method, the intuition behind this "deflation" " step is sensible; since py should spike around M when a is small enough, Equation 64 can be informally interpreted as pulling X, closer to M.
Finally, Kim et al. 2020) use a continuum of noise levels in manner reminiscent. of diffusson models (Section 5.1.2). More specifically, they first construct conditional normalizing flow go : Z x [0, m + X, where Tmax 0 is a hyperparameter. For every E [0, max], the flow g)( must be diffeomorphism, whose inverse we denote as fe(-,a). They then condition the NF on the amount of added noise and train through (conditional) maximum-likelihon
max
Ex -x
[logp² (fo(Xo, + log |det V.o fo So [x, do.
(65)
33

---

Published in Transactions on Machine Learning Research (09/2024)
Since the conditional NF is trained so that X, = go go- (Z,o), where Z ~ p², is distributed according to p. the "deflation" step here simply consists of using = 0 when sampling from the model, i.e. X = go- (2
Finally, we highlight that although the three methods presented above entail fitting full-dimensional likelihood-based models to full-dimensional target densities, the target densities consist of the convolutton of manifold-supported densities with small amounts of noise. As a result, these methods are still exposed to the numerical pathologies described in Seccion 4.1.1
5.1.5 Spread Divergences
As previously mentioned, attempting to minimize KL. divergence through maxximun-llkellioon in the man- ifold setting can result in manifold overfitting (Section 4.1], and while adding small amount of noise to the data can circumvent the problem in theory, it might not in practice. Zhang et al. (2020a) propose to not only add noise to the data, but to add the same amount of noise to the model as well. They formalize this idea by introducing spread divergences. Here we will focus exclusively on the spread KT. divergence and Gaussian noise, but highlight that the same ideas can be applied to other divergences and (potentially learnable) noise distributions. Formally, for a fixed o > O, the spread KL divergence KL, || q) between probability densities p and q is given by
(L, = KKL 9z),
(66)
where Pa == p H N : :0,o²Ip) and go = @ N' ;0, 1D)), ie. the spread KL divergence is the KL divergence between noisy versions of p and q. The spread KL divergence has two important properties: KILo (p /I g) 2 O, with equality if and only if p= = q: and it is always meaningfully defined, since the noisy versions of the original distributions are always full-dimensional (Section 3.4p. Together, these properties imply that the spread KL divergence provides a mathematically sensible objective to train generative models under the manifold setting.
While the noisy model p be evaluated, and thus minimizing KL (F." || pT)) ) is not immediately trivial. Zhang et al. (2020a) propose a very similar model to a variational autoencoder &Section 4.1.2), called the d- o-VAE, where the conditional distribution of X given 2 is now a point mass at go(Z), instead of a Gaussian as in Equation 22 In other words, this model is like a Gaussian VAE, except no additional noise is added to go(Z). i.c. i.c. all the noise in this model comes from the low-limensional Z. Note that, unlike VAFs, this model is not full-dimensional. Much like VAËs are trained to minimize an upper bound of KL(P." ) (or equivalently, maximizing the lower bound to the log- liclliilihood from Equation 24/, 5-VAEs minimize an upper bound of KL, P! which as we will see ends up amounting to a simple modification to standard VAE training. Importantly, even if the supports of po and p do not overlap, their spread KT, divergence is meaningfully defined and thus provides a valid objective that enables 8-VAEs to properly learn distributions on unknown manifolds. Z|X, and are trained through More specifically, 8- VAEs use a variational posterior density 96
always has full-dimensional density, this density cannot in general
(lgg (eliiggo(Z))) go(2), 1p)] - KL (1 ((xx,) pz)]
max E 0.0
X E [E zix. (-1x) llo.
(67)
X ~
where = p.x== + N(.
Note that this objective is equivalent to that of VAËs from Equation 24. except Gaussian noise is added to the data, and the covariance of the decoder is not learnable, but rather made to match the covariance of the noise that was added to the data.
A point about &-VAEs warrants discussion. A common "chcat" " when sampling from a trained Gaussian VAE model is sample Z ~ p², and then output the decoder mean go- (Z), rather than outputting a sample from Gaussian centered at this point (ie. the noise from the stochastic decoder is ignored). The use of the spread KL. divergence elegantly justifics this common practice, since the noise (corresponding to the V(Xaigo( (Z)))) term in Equation 67) here is added due to the spread KL divergence, rather than being part of the model itself. Zhhang et al. (2020a) did not make this observation when introducing é- 5-VAEs, and to the best of our knowledge, we are the first to point this out.
Finally, Zhang et al. (2023) aim to extend the use of the spread KL divergence beyond VAEs, and propose a procedure to use it within the context of normalixing flows (Section 4.1.3). Since the spread KT. divergence
34

---

Published in Transactions on Machine Learning Research (09/2024)
remains intractable, they introduce another bound the training objective. Unlike other variational bounds such the ELBO [Equation 24] or Equation 67 the bound of Zhang et al. (2023 does not become tight at optimality, so it lacks the theoretical guarantees of 6-VAEs which ensure maniold-awarenes..
5.2
through Support-Agnostic Optimization Objectives
In /ectionn 5.1 we showed various DGMs which learn manifolds by first adding noise to p and then using a full-dimensional objective like score matching KL minimization (i.e.
likelihooooo. An alternative to these approaches is using objective that is agnostic to the supports of the underlying distributions being compared. In particular, divergences between probability distributions which metrize weak convergence, such as Wasserstein distances (Section 3.5) or maximum mean discrepancy (Section 3.6), provide such support- agnostic objectives. We now cover DGMs which are trained through these objectives.
5.2.1 Wasserstein Generative Adversarial Networks
Arjovsky et al. (0177) proposed to minimize Wasserstein distance (Section 3.5) between p and pø way to address the issues arising from attempting to train generative adversarial networks using various f-divergences outlined in /ecction 4.2 resulting in strong empirical performance (Karras et al., 2018 2019: 2020). In the GAN context considered by Arjovsky et al. 22117), where p is again given by the distribution of X = (Z), where 7 ~ p² (once more, formally, P is given by the pushforward of p² through go) and pd is fixed (e.g. standard Gaussian), this means changing the GAN objective from Equation 41 to
min max E Ex=pX [x)( Ezzp" (go
(68)
where the neural network ha : X R is no longer a classifier and is now constrained to Lipschitz. If, as with standard GANs, the network h is assumed to be arbitrarily flexible, the objective reduces minimizing the W1 distance between the true distribution and the model (Equation 17).
min W Wi pt))
(69)
provides a support-agnostic optimization objective for training p5 jovsky et al. (2017) enforce the Lipschitz constraint on h2 through weight clipping, which was later improved upon by Gulrajani et al. (2017a) and by MMyato et. al. (2018). The former regularizer encouraging [v))al to be close to 1, while the latter regularizes the spectral norm of the weight parameters of hai both obtain much better empirical performance than weight clipping.
which as mentioned in Section 3.5
Ar-
5.2.2 Wasserstein Autoencoders
As described in (Sectin 5.2.1 for Wasserstein generative adversarial networks, Arjovsky Arjovkss et al. 2017) leveraged the dual formulation of Wi (Equation 17). Tolstikhin et al. (2018) proposed Wasserstein autoencoders (WAEs), which instead leverage the delinition of W (Equation 16) which remains support-agnoslic objective for training DGMs. In particular, they proved that when P is given by the distribution of X (Z) where Z ~ p² (once again, P formally corresponds to the pushforward of p through go), the optimal transport cost can be written as
inf ( .x)) e . Exayx ( go(X,g(()))]
(70)
where 2(p." 2p²) ) is the set of conditional (on X) densities on 2 whose marginal matches p² i.e. 2(x (n) = Z : X + A(Z) q² p²}, where q² = Ex=px (-Ix). Tolstikhin et al. (2018) propose two losses to train WAEs, both inspired by this property: analoguously to variational autoencoders, a conditional z|x is parameterized (e.g. Equation 26), and the first loss is given by
distribution 9
min max ExppY 0,6
E (x))
(xXxxxxxlXXXxxo + (RZ-az (log (2)) + Rzzp' [log (1 ha (2)))) (71)
35

---

Published in Transactions on Machine Learning Research (09/2024)
where ha/ 2 - (0,1), 1), and 8 0 is a hyperparameter.. The first term in the above objective can be understood as minimizing the cost in Equation 70 with no regard for the constraint that 9. whereas the second term encourages this constraint to be satisfied through an adversarial loss (Equation 41) that aims to minimize IS(1ğ )); thus, Equation 71 does indeed aim to minimize the optimal transport cost in Equation 70 Note that even though 4% cannot be evaluated, it is trivial to obtain a sample Z from it by first sampling X ~ p. and then sampling Z|X
ZIX E =(p .p
% 21X (-IX), so that optimizing Equation 71 is tractable.
We find it relevant to highlight some differences between WAEs and other DGMs. (i) Unlike GANs (Sec- tion 4.2) and Wasserstein GANs, WAEs use an adversarial loss on the latent space rather than on ambient space X. This helps stabilize WAEs, as adversarial losses on ambient space can be notorionsly unstable. Nonetheless, properly trained Wasserstein GANs tend to empirically outperform WAF.. (ii) WAFs also very similar to variational autoencoders (Section 4.1.2) e.g. if c(z,y) = =1 y 31 the WAE and Gaussian VAE losses become extremely alike but the KL term in the VAE loss (Equation 24) encourages 9% 2 (-Ix) to match p² for every X sampled from px,, whereas WAEs only encourage this to happen average, i.e. E Ex (9p 2|X (+x)]= = o = p²
The second WAE loss is completely analogous, except it uses maximum mean discrepancy Section 3.6 instead of an adversarial loss to encourage satisfying the constraint that of p²
min E x x 0.6
E z
z1x (1x) 90 + SMMIDD & (1 p2)
(72)
for some kernel & 2 2 - R. which results in an objective with no adversarial component. The choice of which objective to use on latent space to enforce of = =p² has also been expanded in follow-up work, for example Kolouri et al. (2018) use the sliced Wasserstein distance, and Patrini et al. (2020) use relaxed (Sinkhorn) optimal transport.
Finally, we point out that Tolstikhin et al. (2018) found that using arbitrarily distributions 9 x was not key
for good empirical performance, and they thus restrict O(p:" pg ) to only contain point masses, i.e. 4s ZIX (-1z) is given by a point mass at fa().. This choice, which amounts to using deterministic rather than stochastic encoders, reduces the first. term in Equation 71 and [Equaton 72 to a reconstruction error (as measured by c), Ex Ex-px (o(X, (fx(()))
Here we simply highlight that since the distributions involved need not necossarily admit Lebosgue densities, Equation 70 must be formalized by using measures instead of densities:
W .p))
inf
Ex-rx [Bz-Qaix [e(X, go (2))]]
(73)
QaIXERPX p2)
where Q(P. = I A(Z) @ = p7} with Q² = Exwex (1*x
5.2.3 Generative Networks Based on Maximum Mean Discrepancy
Dziugaite et al. (2015) and Li et al. (015)) propose another way to train the model py corresponding to X= X=go(Z) and 2 p" (again, we point out that formally, this model corresponds to the pushforward of p² through go): by minimizing maximum mean discrepancy (Section 3.6). Although their motivation was to avoid the adversarial training involved in generative adversarial networks (Section 4.2p rather than to model manifold sanifold-supported data, the resulting objective provides a mathematically principled way of training DGMs under the manifold setting. The training objective of generative moment matching networks is simply
min MMD? (p.". pp))
(74)
for a pre-specified kernel k. Although minimizing MMD straightforward, generative moment matching networks are not known for achieving good empirical performance: this highlights that, even though manifold- awareness should be considered a necessary condition for strong empirical results, it is not sufficient.
36

---

Published in Transactions on Machine Learning Research (09/2024)
To improve the empirical performance of generative moment matching networks, Li et al. (0117) propose maximum mean discrepancy generative adversarial networks (MMT) GANs), where the main idea is to reintroduce adversarial training to learn the kernel. First, for a fixed auxiliary latent space Z' = Rd a given a kernel k Z' x Z' + R, and a neural network ha : : + Z', [i ct al. (2017) defined the kernel They then showed that, some regularity conditions hold
ki X' x x + IR as ks () == hihafar) and he injective, then max, MMM., metrizes weak convergence, thus making it a mathematically sensible objective to train DGMs. In order to enforce injectivity of h2s [i et al. (2017) leverage the fact that a function X - 2' is injective on X' if and only if admits a left inverse h : + X, iee. hit (h(x)) = a for all x E X. They thus introduce an auxiliary network hi (which need not share parameters with ho), and train MMD GANs through
min max MMD2 .p)) BB x~ 1xx--nn" tix [|XX nig (h)()))) 18]
775)
is a hyperparameter, and the second term encourages h to admit h!, as left inverse on the supports of p and pz. Similarly to Wasserstein GANs (eection 5.2.1/, the empirical performance of MMD GANs benefits from gradient regularization during training [Binkowski et al. 2018 Arbel et al. 2018).
where /
5.2.4 Generalized Energy- Energy-Based Models
Generalized energy-based models (GEBMs: Arbel et al. 2021 combine generative adversarial networks Section 4.2) with energy-based models (Section 4.1.4). GEBMs consist of a fixed prior p² supported on 2, 8 generator 90, 2 + X, and an energy lunction Egg : X + R, where we explicitly distinguish between the parameters of these components as 0 (0), As in GANSs, the prior along with the generator implicitly define the density ph, of X ge, (Z), where Z p² 20
This is not a full-dimensional density, but rather it GEBMs define an EBM on Ma, by re-ghtttting pù,
is supported on the model manifold Mo, == go, (Z)
through the use of Es, as an energy function:
(76) Despite Eo, being defined over X, the above density is only defined on Mo, and is thus also not a full- dimensional density 22 The intuition behind GEBMs is that the generator can easily learn to map to M, whereas the energy function helps correct the distribution within the learned manifold.
Po (z) xc p0, (z)e - (x)
X
In order to train GEBMs, Arbel al. (021) define the quantity
KALE (p. X 11 = sup 1- Φ Ex-px [E())]EEgg [c-Bigo, (21)-)
(77)
FEE,GER
where& & is a set of Lipschitz energy functions satisfying certain regularity conditions (which are satisfied by foed-forward neural networks). They then show that KALE(p. ph,)) is a mcaningfully defined divergence between p and p even when their supports do not perfectly overlap (i.e. it metrizes weak convergence, more details are provided in the grey box below), so that it provides a sensible objective to train the generator. As divergence, IKALE is intimately related to the KL divergence: Arbel et al. (2021) also show that if Ea,
achieves the supremum in Equation 77 then (LL || p/²) I KL(P." Pg. Therefore, the re-weighting done in Equation 76 to Pš indeed improves upon simply using p/.- Putting these properties together, Arbel
ct al. 2021 train GEBMs through
min max - - Ex-px [E,, - Eznp² [e Fez [o2, (ge, (7)) -
(78)
81 02.0
Note that in Section 4.2 we denoted P9, as P but we different notation here as GEBMs further modify p9 21 Formally Mo, need not be manifold, even if ge, is smooth, as it might have points of self-intersoction. The measure- theoretic formulation of GEBMs in the grey box below remains nonetheless valid.
More formally, the x symbol in Eqquatin 76 should be understood as proportional within Ma,, only integrating over the model manifold, L.e. px (1:) P8, (z)e (Boe(// /JM, e -Fo2
Note that none of the involved densities namely p. p.. and P0, are full-dimensional densities, so the KL divergence between them could be infinite Seccion 3.4). The inequality is thus trivially true when the support of ri and ro L.e. Ma does not match M. Nonetheless, when the generator perfectly recovers M, the inequality does justify the use of the energy function in GERMs.
37

---

Published in Transactions on Machine Learning Research (09/2024)
where the Lipschitz constraint on Eg, is enforced as in Wasserstein GANs (Section 5.2.1 and 6 E R is a free auxiliary parameter.
Arbel et al. (2021) also show that in order to sample from a GEBM as defined through Equation 76 one can first sample from the EBM p on 2 given by
p (z) x p² (z)e Esg (00, (z))
(79)
and then setting& X = ge, (Z) will produce a sample from p Note that p now a full-dimensional density in 2, and can thus be sampled through Markov chain Monte Carlo as standard EBMs. We point out that GEBMs are intimately linked to two-step models, which we discuss in Section 5.3
Finally, we highlight some related works. Che et al. 2020) proposed a similar model to GEBMs, but their model is trained as standard GAN (Equation 41 and the EBM is defined post-hoc by using the discrimi- nator h despite the similarity with GEBMs, this procedure does not endow canifold-unaware GANs with manifold-awareness. Birrell et al. (2022) constructed a class of divergences, which KALE belongs to, by extending its relationship with the KL divergence to general f-divergences; and Gu et al. (2024) leveraged these divergences, along with optimal transport (Section 3.5), to obtain a manifold-awer adversarial training objective for continuous normalizing flows (Section 4.1.3).
We now formalize the presentation of GEBMs. Here we denote p/, as probability measure, go, # IP2 instead of as a density. The model distribution Fy then defined through its Radon-Nikodym derivative with respect to go, # #P²
p (x) = dP
e Eog (x)
dgo, # PZ (I)
=
E Xx P 2 le Esy (X)]²
(80)
where the expectation is assumed to be finite. Arbel et al. (002 proved under mild conditions that: (i) KALE(PX 90, P2) 2 0 with equality if and only TPx = go, # IPZ; and that (ti) for a sequence of
generators (00,.)21 KALE(PX [19,,, 0 as t + oo if and only if go Ip? = PX as t -
5.2.5 Principal Component Flows
Principal component flows (PCEs; Cunningham et al., 2022) are a variant on standard normalizing flows (Soction 4.1.3b that seek uncover manifold structure manner analogous to principal component analysis (PCA) and related to disentanglement (Bengio al. 2013). If go 2 - X is normalizing of (..o0=)V=goo2>) are said to be the principal components of go
flow, the eigenvectors (z)...., at ==((=) and can be ordered using the corresponding eigenvalues as in standard PCA. The principal components at represent the principal axes of variation in the flow's density p In manifold-learnin context, principal axes with small eigenvalues represent off-manifold directions, while those with the highest eigenvalues represent primary directions of variation along the manifold, as illustrated in Figure 7(a) we highlight that here P.N is assumed dill-simensional and to concentrate around M, rather than being strictly manioold-supported..
We now discuss special case of PCFs introduction; for presentation of the method in more generality, see the work of Cunningham et al. (2022). PCFs involve the notion of conlour log-likelihoods, which here can be interpreted as the likelihood along the ith coordinate curve of the flow,
log p (z+) - log (V (z))
(81)
for E (1, D), where (²)))= [TP)=III_ 22 (z+) is a coordinatewise factorization of the latent prior (which is typically Gaussian), and { is the ith entry on the diagonal of (x))
The goal of training a PCF is to align the principal components of ge with its latent coordinates on the manifold Figure 7(b)). One of the key insights of Cunningham et al. (2022) is that the difference between the model's log density log po (x) and the sum of its contour log-likelhhoods measures the diagonality of
38

---

Published in Transactions on Machine Learning Research (09/2024)
X Pone
Z
X
2/ (x)
po
11 (x)
N
2
g0pcr
M
X Poper
(b)
Figure 7: The motivation behind PCFs. (a) The principal components, vi and p2(2), of a trained NF with density pg taken a point x X. These principal components are scaled according to their eigenvalues: here, (x) is the primary direction of variation along the manifold M. (b) A comparison between the PCF, rospectively. Contours are visualized by the way each NF maps the grid lines of Z. Since the contours mapped by 90per correspond
contours of two NFs: pe and Po trained as standard NF and
the principal components of pp² poppr the NF model p PCP
is formally a principal component flow.
V3002)) and hence how well the model's latent coordinates are aligned with its principal compo- nents. From this, a regularizer can be derived,
I(0) = Ex logpă (X) Σ log 2: (fo(X)) + log (V=.o(X)aa)
(82)
I(0) is a non-positive quantity such that I(0) = if and only if the latent coordinates are perfectly aligned with the flow's principal components at cach point x E X. Maximizing I(0) as a regularizer while maximizing the likelihood aligns the flow's latent coordinates with the principal manifolds of the data. We highlight that while this objective was derived to provide a useful inductive bias when manifolds are involved, it remains nonetheless based full-dimensional likelihoods, and is thus subject to the corresponding pathologies (Section 4.1 and Section 4.1.1).
where fo(X); denotes the ith coordinate of (e(X)
24
2023) take a related approach which directly penalizes off-diagonal elements of Vago (fo(o))) xf)))))(((())) This results in a potentially looser regularizer which is designed to achieve the same goal at optimality as PCFs. Both PCFs and CMEs have been shown to naturally represent M using a subset of the flow's 's latent coordinates, meaning they automatically discover an estimate d of the dimension d' of M without the practitioner having to set as hyperparameter.
Canonical manifold flows (CMFs; Flouris & Konukoglu
5.3 Manifold-Awarener through Two-Step Models
Excopt for gencrative adversarial networks, all the DGMs described in Section 4 are manifold-unaware as a direct consequence of misspecified dimensionality: the data is d' dimeennsionnl whereas the model is D- dimensional. The methods described in [Section 5.1 address this misspecificatioon by adding noise to the data, making it D-dimensional, whereas those from Section 5.2 use appropid-a losses. Another
24 Note that the meaning of fx(X) different here than
Section 5.1.4
where refers the first d coordinates of fx()).
39

---

Published in Transactions on Machine Learning Research (09/2024)
approach to enable manifold-awareneee which we cover here, is to instead reduce the ambient dimension of the data to match its intrinsic dimension before learning its distribution. Two-step approaches do this by separating the overall generative modelling process into two distinct steps. Manifold learning (step 1) typically involves some form of encoding-decoding, which uncovers a lower-dimensional representation space 2 whose dimension d ideally matches the intrinsic dimension d² of the given data. Distribution learning (step 2) is then carried out on the obtained manifold, which often takes the form of generative modelling of the d-dimensional representations learned in the previous step. Importantly, this two-step procedure aims to remove the dimensionality mismatch between the data and the model, and thus circumvent any woes caused by it, such as manifold overfitting (Section 4.1). When discussing two-step models, it will be useful to distinguish between the generative parameters of each step, and we thus write 0 = (01, 02), where 01 are the ive parameters required for the first step, and 02 those for the second one. We now outline these two steps in more detail:
Manifold learning Most two-step models use autoencoder- methods for manifold learning (eccion 3.2) as in Equation 6
min E 01.
X~
[||X - -9o, (x)
(83)
or any variant such as variational autoencoders [Section 4.1.2). that non-autoencoderr-bae manifold learning rather than generative modelling, so e.g. even if VAE is used, it is interpreted as a regularized autoencoder rather than a generative model.
although we will see in Section 5.4.1 choices are also possible. Importantly, the goal in this step is to perform
Distribution learning This step consists of learning the distribution on the manifold obtained in the previous step. In the standard setup where manifold learning is performed with an autoencoder- based model, a pre-trained encoder før and decoder ge; pair is available. The encoder defines a distribution vg- of encoded data for (X) where X ~ pxd (formally, og. is the pushforward density of p through fø-); which can be learned by instantiating a DGM pé, on 2, and training with any of the methods covered in this survey (but changing the target distribution from the D-dimensional p.N to the d-dimensional o while keeping the encoder fø- and decoder go; frozen. Then, once this low-dimensional DGM is trained, resulting in phz the distribution of the two-step model on the learned manifold can be sampled through X = go; (2), where Z PO; (i.e. p is formally given by the pushforward of poz through goz)-
Importantly, by solving the generative modelling task in Z instead of X, the support of the target distribution % is for (M) S Z, whose dimension should intuitively be given by min(d, (i, d"). If the latent dimension d is chosen properly (i.c. d = d'), we should then expect 9 to be full-dimensional within Z. This full- dimensionality stands in contrast to the case discussed in Section 4.1 where the target distribution p. is d dimensional but its corresponding ambicnt space X' is D-dimensional. Furthermore, even if is overspecified as d' < d < D, the "dimensionality gap" - i.e. the ambient dimension of the model minus that of the true manifold of the second-step model is d - d', which smaller than D d= 25 Thus, we should intuitively expect any manifold-related woes arising from dimensionality mismatch to be milder than the corresponding full-dimensional issues.
Many two-step models have been proposed in the literature, sometimes with the manifold sctting in mind, and some other times simply for tractability, as training DGMs on a low-dimensional latent space is cheaper than doing so in high-dimenssonal ambient space.
et al. (2022a) provided a the- oretical justification for all of these models, proving that under mild regularity conditions, when d = d' and Ex=,x |||X 90z (fo- (X))1 = (i.c. perfect reconstructions) then: (i) o3- is indoed full-dimensional, and thus p2 can be any DGM, even if manifold-unaware (e.g. trained through maximum-likelihood) and still learn ; and (ii) transforming samples from pē; through gaz is equivalent to sampling from p. L.e.
25rThe case where d underspecified, f.e. d < d", less interesting, as in this case M, and thus px cannot he recovered.
40

---

Published in Transactions on Machine Learning Research (09/2024)
two-step models recover p.a. This result, which discuss further in the next grey box, implies that two- step models learn M, which is an interesting observation since autoencoders by themselves need not - see 2 Figure 2 and the discussion in Section 3.2) In particular, when Po: is perfectly trained it must be supported on før (M), and since go; (M)) = M, follows that together, the support of po; and the decoder go; jointly characterize MM.
Here we briefly summarize two-step models which are straightforward combinations of an autoencoder-based model with any other generative model on latent space; two-step models warranting additional discussion are covered in Section 5.3.2 and Section 5.3.3 Dai & Wip 2019) use a VAE for manifold learning and another latent space. Xiao et al. (2119) use autoencoder and normalizing
VAE for distribution learning flow (Section 4.1.3), as do Bochm & Scljak [2022]. Ghosh et al. 20200) use an autoencoder with an added
regularization term, along with a Gaussian mixture model (which, while not deep, remains a generative model and thus fits the two-step framework). Li et al.
(0155) use an autoencoder and then train a generative
moment matching network [Section 5.2.3) on the recovered latents, and [ao et al. (2023) use a regularized
VAE along with conditional flow matching (oction 5.1.3). The improved generative performance reported
in many of these works compared their full-dimensional counterparts may be attributed to the reduction of dimension mismatch.
We also point out that generalized energy- based models (Section 5.2.4) are very similar to autocncoder-based two-step models, since GEBMs instantiate an energy-based model (Section 4.1.4) on a low-dimensional latent space, whose samples are then mapped through a decoder. GEBMs are nonetheless not autoencoder-basoe two-step models, the differences being that: GEBMs do not require an eucoder, the distribution used by GEBMs on Z (Equation 79) shares parameters with the decoder, GEBMs are trained end-to-end, and they are limited to using EBMs on the latent space.
End-to-end training Although the two-step models described above are manifold-aware, the objective in the first step does not necessarily promote representations that are conducive to distribution learning in the second step. Therefore, might expect that training these models in an end-to-end manner could further improve their performance. Albeit not always inspired by this motivation, several works have pro- posed end-to-end objectives, some in the context of variational autoencoders, and some in the context of injective normalizing flows; we discuss the former here and the latter in (ecciin 5.3.3 VAEs as described in Section 4.1.2 assume a fixed prior p² on the latent space Z. However, the prior pé, can be made trainable, in which case maximizing the ELBO (Equation 24), i.e.
X (XIZ) - KLL (9% x (11)) [bb) (1)) (12) Pe. Po. M
max E Exppx
(84)
remains valid objective (assuming P.N is full-dimensional) for end-to-end training of Pe, and Pb. Depend- ing on the choice of pg. additional computational tricks might be required to efficientlyy optimize the ELBO.
X|Z
Tomezak & Welling (2018) instantiate p3, as a Gaussian mixture model; Sonderby et al. (2016). Vahdat & Kautz] 20220) and Child 2021) use learnable hicrarchical priors; Chen et al. (2017) use normalizing flows;
Pang et al. (2020) use energy-based models; and Vahdat et al. 2202 b use diflusion models (Section 5.1.2).
When Ps, X1Z is a flexible enough full-dimensional density as in Equation 22 all these models remain suscopti- and Section 4.1.2] despite directly encouraging the encoder to learn representations whose distribution can be easily recovered by p
ble to the manifold overfitting issues discussed in /Section 4.1
Even though designing an end-to-end objective for training in a manifold-aware fashion is intuitively desirable, doing so is not always straightforward. To see why, consider two-step model whose first-step loss is given by [Equation 83, and whose second-step model is trained by minimizing I(gg.,P6,) over 0₂ for some divergence DD between probability distributions. Let us further assume that D(g -%,)) cannot be computed without evaluating 93. but that D(g Pb,)) = L(r,;; :3. + c(gg.), ), where L(pg :z.) can be computed without evaluating o and where c(gZ.) ) does not depend on p Divergences with these properties are prevalent; the KL divergence (Section 3.4 is an instance, where Llvb,;48.) - Ezzag, [log p6, (Z)) and c(ag.) = EZzngg_ [log o3. (2)), as well as the Fisher divergence (Section 4.3) which underpins score matching
41

---

Published in Transactions on Machine Learning Research (09/2024)
and thus diffusion models. In this case, the second-step model is trained by using the loss L(pb (p%2.:92)) :Z-). which is equivalent minimizing (p .P,)). Naively combining the losses of this two-step model into single loss for end-to-end training would result in the objective
min E E [||X go, (X+())) + 3C (v0.: 8,6
(85)
for some 3 > 0. This naive objective ignores c(gg); so that it is not equivalent
min Ex [||X go, (fo( + 3D (16 .b.))
(86)
In short, Equation 85 does not provide valid objective for manifold-aware end-to-end training since c(g cannot be ignored when ø is not fixed after the first step of training. Unfortunately, although Equation 86 specifies a principled objective to address the issue, it remains intractable when c(g)) cannot be computed.
More formally, a trained autoencoderr-asser two-step model is given by a distribution on 2 along with a decoder 90; Z + X, and the model distribution given by Pg- got P. The result of Ganiza et al. 2022a justifying two-step models states that, under mild regularity conditions, if d d' and Ex-EX X go: (So- (x)) 0, then:
02
O2 Ad, where Q = fø-# #P= Px is the distribution encoded data.
g 0 P.X
The first point ensures Qg admits density with respect to Ad, so that it is full-dimensional. The second point ensures that if the target distribution Q is properly learned during the second step then two-step models recover the true data-generating distribution, i.e. if P3; Qg. then P2 -PXX
5.3.1 Two-Step Models Minimize Wasserstein Distance
Before continuing our review of existing two-step models (Section 5.3), we highlight that these models can be interpreted through an optimal transport lens (Section 3.5). To the best of our knowledge, this observation has not. been made in the literature, and constitutes novel contribution of our work. Here we still consider the model py given by the two learnable components 9o, and Péz and will use the notation introduced for Wasserstein autoencoders (Section 5.2.2. Key to our insight is Equation 70 which, for the model pa considered here, can be rewritten as
Wp. po
inf 8 pIxEe(p) .2,)
Ex (Ez (e(X, go, (2))1l
(87)
This equality then implies that
( pg =
inf Ex-px (x go, (f(X))), JeF(pE vb,)
(88)
where Flpx x Pb,) is the set of functions f f: X' + Z such that if X ~ D.x, then f(X) ~ pτ.- To see that Equation 87 indeed implies Equation 88| simply note that if E Flp. pb,). then the conditional (on X = x) f(x) is in Q(x. P(,)) Equation 88 is used to justify the use
distribution on Z given by the point of deterministic encoders within WAFs: doing so minimizes an upper bound of wp .rr"). We also point out that, assuming c(x, 3) is minimal if and only if æ = y, the bound becomes tight at optimality as long perfect reconstructions achievable with deterministic autoencoder (i.c. X go; (for (X)), see see Section 3.2 and Section 5.4 for discussions of when this is possible with continuous autoencoders).
For an encoder for we let z be the distribution of fx(X) where X ~ xxxx (formally, 9% is the pushforward density of p. through fo), and note that fo E Flye vb,) is equivalent to o p2 In turn, Equation 88
42

---

Published in Transactions on Machine Learning Research (09/2024)
justifies training the model p by minimizing upper bound of its optimal transport cost through
min E Ex- lc (X(,9n, go, (fo(X))) 0,4
(89)
subject to d = P02-
The key difference between this objective and that of WAEs is that the distribution on latent space is now learnable instead of being fixed. While this distinction with WAEs might seem conceptually trivial, it enables minimizing optimal transport cost through a two step process: in the first step, 01 and Φ are trained to minimize the reconstruction error, Ex=px [c(X, (1XX, ge, (fo()))) with no regard for the constraint. This step results in a now fixed distribution T. on Z, which of course need not match p Thanks to péz being learnable and 0g not appearing in the first-step objective, this mismatch can be addressed the second step, where any objective over 02 to match Pé, and zg can used. For example, the second-step model were trained through maximum-likelihood, its objective would 2d
min KL (g0- 112)))
(90)
02
which indeed satisfies the constraint in Equation 89 at optimality. In other words, two-step models solve Equation 89 by optimizing an unconstrained version of the objective during the first step, and then ensuring the constraint is actually satisfied during the second step. Crucially, the second step does not affect the optimality of the first step because Xx>px[c([X go. (X))) does not depend on 82, so that two-step models indeed provide a valid way of solving Equation 89.
We now make some additional observations. (i) When e is given by the squared Euclidean distance, the corre- sponding loss for the first-step model is exactly that of a standard autoencoder (Equation andd Equuaion 83) When the first-step model is trained through different autoencoder-basee objective, e-g. Eqqation 24 we can simply interpret the model as regularized autoencoder as long as it encourages perfect reconstructions at optimality. (ii) Although two-step models are often trained using deterministic encoders, using arbitrarily flexible stochastic encoders would imply that (p.". pg²) ) is being minimized rather than an upper bound.
In summary, we have justified the manifold-awarenes of autoencoder- based two-step models through optimal transport. We believe that this result is not only interesting on ils own, but also hope that by establishing connection between seemingly unrelated manifold-aware model classes namely two-step models and those which are trained through support-agnostic optimization objectives (Section 5.2), such as WAEs it will enable future improvements to both.
Equation 88 is formalized as follows:
inf SEF(P? .2)
Exapx (c (X))))
(91)
where F(PPX 1= (f: Z f measurable, and faP 12). J. As technical point, note that the unconstrained version of the right hand side of this equation upon which the interpretation of two-step models Wasserstein distance minimizers is based i.e.
inf E Ex-ex [X go. (f(x))),, FEF
(92)
measurable), involves an infimumm over measurable functions (which is then minimized over 0, during the first step). In the nonparametric regime (Section 2.2) we assume that neural networks are flexible enough to approximate any continuous function arbitrarily well, but this property need not priori extend to measurable functions. Intuitively this should however not be a problem thanks to Lusin's theorem which, informally, states that in certain settings any mcasurable function can be approximated by a continuous one. is nonetheless pertinent to show that the infimum in Equation 92 can actually be replaced by a corresponding infimum over
where Ff
Note that the second-step objective could itself require additional auxiliary parameters but omit this possibility for notational simplicity.
43

---

Published in Transactions on Machine Learning Research (09/2024)
continuous functions, which we do below.
Proposition 1. Let PXX be a probability measure X, 90. : 2 X be measurable, and XxX + be measurable and such that there crists C > 0 such that
sup (x.))eXXxX
|c(x, v)1 <C
(93)
Them,
inf Exaex go. (f(x))) = inf Ex-rx (c ge, (f(X))),, fEF
(94)
SEC
where F == 1f: 2 2 is measurable) and C = {f X f is continuous).
Proof. See Appendix B.2
This result allows us to formally interpret two-step models as minimizers of an upper bound of the Wasserstein distance in the nonparametric regime when the assumption in Equation 93 holds. We point out that this is mild regularity condition, as it is always satisfied in the common case where X' is compact and is continuous (since continuous functions always achieve their supremums over compact sets).
Finally, we point out that Patrini et al. (2020) claimed that, as long as IP is non-atomic, then the inequality in Equation 91 actually an equality. This would allow us to interpret two-step models as minimizing Wasserstein distance not an upper bound even when using deterministic encoders. However, Lee et al. (2024) found an error in the proof of Patrini et al. (2020). so that only the upper bound interpretation remains valid.
5.3.2 Latent Diffusion Models
Latent diffusion models (Rombach et al. 2022 Peebles & Xie 2023 Zhang et al. 2024 are another class
of two-step models (Section 5.3). They first train a regularized autoencoder, which combines a Gaussian
variational autoencoder objective (Section 4.1.2) with various potential regularizers (Larsen et al.) 2016
Higgins et al. 2017 van den Oord et al. 2017 Once this autoencoder-basee model is trained and the corresponding low-dimensional representations obtained, a diffusion model (Section 5.1.2) s [0, 2 is trained them as the second-step model.
As previously discussed, diffusion models can learn manifolds, but their score function must diverge to infinity at the end of the backward process [Equation 52) as consequence of the mismatch between the intrinsic and ambient dimensions of the data. To test that latent diffusion models are not as sensitive to this numerical pathology, we trained a diffusion model and a latent diffusion model on the CIFAR-10 dataset (Krizhevsky & Hinton 2009), with all experimental details provided Appendix C We plot the average squared Euclidean norm of the score functions, normalized by their dimension," |"1 along generated paths in Figure & it is evident that the score function of diffusion models on latent space exhibits much better numerical behaviour than when these models are trained ambient space. This result is suggested by theory, and to the best of our knowledge, we are the first to empirically confirm it.
Currently, latent diffusion modols are amongst the best performing DGMs empirically, and the manifold lens provides convincing explanation for this: (i) they can learn manifolds; (ii) they alleviate the numerical issues of diffusion models; and (iif) they are robust to misspecification of the dimension of the latent space, in the sense that even if d' d (i.e. the latent dimension specified as larger than the true intrinsic dimension), they still learn their target distribution albeit with an exploding score function. To see this, simply note that setting d' < d results in second-step model whose target distribution qg- is still supported
Note that normalizing squared Euclidean norm by dimension is the most natural way of enabling comparisons dimensions. this consider constant with entries equal which has squared l2 norm equal dimension; consider standard Gaussian vector, whose expected squared Fuclidean also matches its dimension.
44

---

Published in Transactions on Machine Learning Research (09/2024)
Diffusion model
5000
Latent diffusion model
4000
8g (Y, T t)ll
3000
dim
1000
0-
0.002
0.004
0.006
0.008
0.010
T-t
Figure 8: Average squared norm of the learned score function on CIFAR-10 over 100 generated paths from Equation 52 normalized by dimension (i.e. dim = D = 3072 for diffusion models, and dim = = 256 for latent diffusion models); the shaded area corresponds to one standard deviation. The paths are stopped at time T e, with e 0.001.
on a manifold (M) of lower-than-ambiere dimension, which the diffusion model from the second step can still learn. Although in this case the score function would also diverge to infinity, the fact that the difference between the ambient and intrinsic dimensions for the latent model (d- - d²) remains much smaller than for model ambient space (D d²) intuitively suggests that the numerical issues should nonetheless be diminished for latent diffusion models. Indeed, despite the latent diffusion model shown in
| F'iuure 8 using
d == 256,
25256, which is likely larger than the true intrinsic dimension d of CIFAR-10 (Pope et al. numerically much better behaved score function than the diffusion model on ambient space.
202 it has a
5.3.3 Injective Normalizing Flows
Ordinary normalizing flows (Section 4.1.3) are full-dimensional density models with D-dimensional latent spaces, conflicting with the d' d'-dimensional nature of p.Y. To correct this mismatch, line of rescarch started by Kumar et al.] (2020) has proposed to shrink the NF's latent space dimensionality to d < D, allowing the model to represent densities on low-dimensional submanifold of X. Whereas standard NF architectures must be bijective, ge, now cannot be bijective because it maps from d to D dimensions. The most one can ask that go, be injective, resulting in the injective normalizing flow (INF).
Brehmer & Cranmer (2020) enforce injectivity architecturally, by constructing 96, as zero-padding opera- tion followed by D-dimensional NF, in which case the left inverse fe, is given by inverting this NF and ap- plying a projection operation; this is the same construction as the one used by denoising NFs (Section 5.1.4 Injectivity enables density evaluation through the injective thange-of-variablee formula (Equation 9),
pe (x) = vτ, (=) (det =90, (=) VV 290 (=))
955
where fer (z). Unlike standard NIs, INFss cannot be naively trained through maximum-likellhood for two main reasons: (i) the involved determinant is much more computationally challenging to compute and opti- mize than in standard NFs, and more importantly (Mi) Brehmer & Cranmer (2020) showed that doing so would result in pathological solutions. To understand why, recall that Equation 95 is only valid when x E ge, (Z). When x lies outside ge, (2), the right hand side of Equation 95 evaluates to p (ão, (#)), where de, (x) = ge, (fe, (z)) can be thought of as a projection of z onto go, (Z). Since ge, (Z) need not perfectly match the data
1ike in standard NFs, here the encoder fe, determined by the decoder 90, and it. is thus parameterized by the same parameters (t.e. 01), so auxiliary parameters are needed to parameterize
45

---

Published in Transactions on Machine Learning Research (09/2024)
g0
X
P
M
x2
I
Figure 9: Pathology of naïvely maximizing Equation 95 In this case, the true density a standard Gaussian along M {(0, x2) R²2 z2 RR, here shown in blue with darker values indicating higher density. The model can maximize likelihood by aligning go, (2) perpendicular to M, and then learning density along 90, (2) that becomes infinitely peaked at the projection áa, (z) onto M; in this example the projection will always lie at the origin for any I E R2, In the figure, we show 9o, (Z) = {(21,0) 0) E R² x e Rj with the black line, the projection with the red dot, and increasing peakedness of p with increasing opacity.
manifold M, attempting to maximize Ex-pX [log vē, (Vo, (X)) log det 290, (So, (X)) V =90 (Ja, (X)))I over 0 would thus result in maximizing the likelihood of projected data. As illustrated in Figure 9. this ob- jective can admit pathological solutions where the projections collapse onto a single point whose likelihood is sent to infinity.
To circumvent this issue, Brehmer Cranmer 2020 propose to train INFs as two-step models (Section 5.3)
where go, and fo, are trained to minimize f2 reconstruction error as in Equation 83) this training procedure
also obviates the need to optimize through the determinant in Equation 95 Brehmer & Cranmer (2020) then
instantiate ph, as d-dimensional NF, which is trained on encoded data Joz (X), where X pr Koth
et al. (2021) follow on this work by proposing a more efficient architecture for ge,- Kumar et al. (2020)
originally proposed relaxed INFs, for which the encoder fo is parameterized separately (and encouraged to invert the decoder on M through a reconstruction error), and where injectivity is instead encouraged by regularizing the singular values of = (fo(X)) for X ~~ -p.." They then take the second-step density Pé, as Gaussian mixture model. By virtue of being two-step models, all these DGMs are manifold-aware.
End-to-end training As mentioned in Section 5.3, it is intuitively desirable to find an end-to-end objec- tive to train two-step models, and INFs provide particularly interesting opportunities. To circumvent the pathological behaviour of naïve Maximuu--lkkelieen perfect reconstructions with the goal of ensuring that the model manifold matches the true data manifold:
training of INFs outlined above, several works encourage
maxExxpt [log p (fe, (X)) log det (Vege, (fe, (X)) V2g0, (fe, (X)))
0
(96)
subject to Ex-eX [||X- - ge, (Je, (X)) = 0.
In practice, the constraint can be encouraged by adding an & regularization term to the likelihood,
max E Ex-px [log pú, (fe, (X)) log det (V.ge (fe, (X)) T Vig0, (fe, (X))) - -|| ge, (fe, (x))
(97)
where 3 > 0 is a hyperparameter. Much like how normalizing flows focus on tractability of the log-det- Jacobian term, a central theme of end-to-end injective flows is tractability of the second term in Equation 97 which we will refer to as the "log-det-- J" term. One approach is to impose structural constraints on ge, to make the JT term casily computable, albeit at the cost of expressiveness, for example using confor- mal embeddings (Ross& & Cresswell 2021). A concurrent approach, pursued by Caterini al. (2021b), is approximate the gradient of the Jog-det-J" term with respect to 0 through combination of Hutchinson's 's
46

---

Published in Transactions on Machine Learning Research (09/2024)
estimator (Hutchinson 1989) and various tricks from linear algebra and automatic differentiation (Baydin et al. 2018). Despite these approximations, tractability remains an issue with this technique and it struggles to scale to datasets of higher ambient dimensionality than CIFAR-10 (Krizhevsky & Hinton 2009).
Denoising NEs (Section 5.1.4) perform single-step training of a flow with an injective component for gener- ation, but with a full-dimensional model Pe of a noised-out data density px=- = p @ NN(:: (T::;O,0TTT pD) The objective for denoising NFss (Equation 63) ends up quite similar to Equation 97 although with two important differences: (i) the expectation is over p.", rather than p.¹, and (ti) the formulation of p.* as model for p.- eliminates the need to optimize over the costly log-det-J" J term. However, the computational benefit comes at the cost of introducing a disconnect between the injective generator and the full-dimensional den- sity model. In particular, the numerical instabilities described in [Section 4.1.1 do not apply to INT's trained through [Equation 97 since no full-dimensional densities are involved, whereas they do apply denoising NEs.
Cunningham et al. (2022) and Flouris Konukoglu (2023) both propose injective variants of the flow models described in Seeciinn 5.2.5 using similar objectives to Caterini et al. (2021b). Cunningham et al. (2022) in particular use regularizer that, with certain hyperparameter setting, cancels out the log-det-J J term in the likelihood, making likelihood- optimization of injective flows more efficient.
Mcanwhile, several works parameterize the encoder fe separately, as Kumar et al. (000)) did. Zhang et al. (2020b) proposed a similar objective to LoIEuution 97 the context of variational autoencoders Section 4.1.2) Sorrenson et al. (2024b) argue that the objective in Equation 97 is subject to similar pathologies to those outlined by Brehmer & Cranmer (2020) for naive maximum-likelie likelihood training if the encoder and decoder are flexible enough. More specifically, they contend that [Equaton 97 can be made pathologically large by learning a manifold with arbitrarily large curvature rather than M. This is interesting as it highlights that despile being directly motivated to account for the low-dimensional structure of the data, INFss trained through Equation 97 can nonetheless still fail to learn manifolds. [Soreeson et al. (2024b) propose an intuitively well-motivated but theoretically ad-hoc modification to the objective, along with an improved gradient estimator over that of Caterini et al. (2021b), for end-to-end training of INFs. Their model no longer falls under the umbrella of two-step models, and hence its ability to learn the manifold unclear, but it does inherit computational benefits as circumvents calculation of the challenging log-det-J J term (Brehmer & Cranmer 2020). Finally, Nazari et al. (2023) proposed an autocncoder for manifold learning nnol generative modelling ling) which penalizes the variance of the log-det-JT J term during training, and which results in a smoother and more interpretable latent space as compared to standard autoencoders.
5.4 Overcoming Topological Obstacles to Manifold Learning
As mentioned in Section 5.3, one might want to set the latent dimension d of an auloencoder to d². Yet, as discussed in Section 3.2 when d d= d", perfect manifold learning is not always achievable through bottleneck methods for topological reasons. We illustrate this problem, which can cause downstream issues with density estimation, in Figuuee 10(a) In particular, if M has any non-trivial topological properties such as holes disconnected components, any attempt to model M as the image of decoder go will cause numerical instability in go (Cornish et al. 2020 Salmona et al. 2022). This problem was originally identified in the context of normalizing flows (Section 4.1.3), with a line of work that first appends additional dimensions to X, and then trains a normalizing flow on the augmented space (Dupont et al. 2019 Chen et al. 2020 Huang et al.] 2020). While these techniques indeed increase the stability and expressiveness of the flow itself, they still produce full-support densities that can never truly model non-trivial topological structures in data. Other works have leveraged tools from the field of topological data analysis (Chazal & Michel, 2021 Barannikov et al. 2022) to explicitly regularize autoencoders with the goal of encouraging (eM) to share topological properties with M (Moor et al.) 2020 Trofimov et al. 2023). Empirically, these methods have been shown to decrease topological mismaltch; yet, as argued in the grey box in Sectiin 3.2, completely eliminating this mismatch is theoretically impossible when the root cause of the problem is the non-existence of a topological embodding of M into 2, rather than the loss used to train the autoencoder. Alternatively, to more faithfully tackle topological issues, some works have proposed to restructure the manifold-learming step to better reflect the ways manifolds are defined in theory; we cover these approaches in detail below.
47

---

Published in Transactions on Machine Learning Research (09/2024)
go go(s((M))
X
99
3
(c(M)
f
(a)
X
(1)
M
96
M
S 11)
X2
21
(2)
Za
(2)
(b)
96
(c)
Figure Models for M {(11, xt + x3 1}, the unit circle X R.2 (a) Using a single encoder-decoder pair model the circle. The pair approaches numerical non-invertibility since the encoder Jo must map two nearby points in M (in black) to two distant latent points in Z (also in black). (b) Illustration of implicit manifolds. Here, M is characterized as the O-level set of F R + R. Specifically, F(x1, = - (xî + =²) is shown in red, and the grey plane corresponds to {x E IR3 I3 = 0}. Neural implicit manifolds use no autoencoders, and instead attempt to learn F9 so that its 0-level set, F. (10)),
matches M. (c) The problem depicted in (a) can also be circumvented by employing multiple encoder- decoder pairs, each one using its own latent space and covering a different part of M.
5.4.1 Neural Implicit Manifolds
Under some conditions, manifolds with non-trivial topologies can be defined using level sets of functions. In particular, a level set of a smooth function F RD R- represents d" "d-dimensional manifold if its Jacobian has full rank that level set (Lee 2012
We illustrate such implicitly defined manifold which cannot be characterized with single encoder-decoder pair in Figure 10(b)
Neural implicit manifold learning (Ross et al. 2023) operationalizes this fact by modelling M as the zero set of a neural network Fø, X - RD- The network Fe, is trained to align its zero set FØ ((0)) with M while being regularized to have full rank on M. The following loss is used,
min E ExpeX y Youy V~ll(. (0-(01)0---)) | (X)ll2 - Fo, (Y) + 5(n - IIV (x),1 0,
(98)
1F (wll with 0, stopgrad((1) (as in energy-basod models, see Section 4.1.4); u( 50-:51 is the uniform distribution over SD-d-1 = {y E R ||al|2 = 1), the (D-d-I)-sphere in D-d; a > 0, 8 > 0, and > 0 are hyperparameters; and (-) = max(- ., 0). In this loss, the first term ensures that Fr ({0)) contains M, the second term prevents Fø ((0)) from containing off-manifold samples, and the third term regularizes the Jacobian Fe, to have full rank on M.
where: Y is independent of X; q (v)
Importantly, Fe, here is not an encoder; it is best interpreted as defining D non-linear constraints on the data, thus leaving d degrees of freedom for the data manifold. The full-rank requirement then be
48

---

Published in Transactions on Machine Learning Research (09/2024)
interpreted as ensuring none of these constraints is redundant, thereby ensuring the learned manifold has the correct dimensionality. This constraint-based learning procedure for M makes implicit manifold learning unusual among two-step models (Section 5.3) in that it is not autoencoder-baet based. The lack of encoder in this method mcans there is no latent space, which presents a challenge for learning the distribution on the model manifold, Foy ({0)).
Ross et al. 2023 propose to learn this distribution with the constrained energy-based model, which repre- sents an EBM constrained to the learned manifold, E Fø; ({0)) + R. This is parameterized in practice by a neural network Es,- X + R, for which values are ignored outside of the learned manifold For ((0)). To sample from constrained EBMs, constrained Langevin dynamics (Brubaker et al. 2012) is used to generate samples from Fag constrained to the manifold. This allows for likelihood maximisation via Equation 38 with ordinary EBMs except constrained EBMs are manfold--uppported.
5.4.2 Multi-Chart Manifolds
Typical techniques model the manifold globally using single encoder-dccoder pair. In general, however, manifolds can consist of patchwork of many charts: mathematical objects that each serve roughly the same function as single encoder-decoder pair (for formal definition, please see Lee (2012)). Some manifolds
may thus require many encoder-decoder pairs (f() 90
(())z >== each using its latent space Z, locally describe
some subset of the manifold rather than a single one: we illustrate this fact in Figure 10(c)
Several works have taken this route to learn the data manifold. Schonsheck et al. 2019 first proposed a multi-chart latent space using multiple encoder-decoder pairs in a non-generative modelling context. For each incoming datapoint, the correct chart is selected dynamically using a prediction head for the encoder- dccoder pair with the smallest reconstruction error. Kalatzis et al. (202)) propose multi-chart flows, in which likelihoods are defined using a mixture of injective normalizing llows (Section 5.3.3), with mixture weights again computed using a prediction head. On the other hand, Sidheekh et al. (2022) propose a mixture of INF' in which chart membership for an incoming datapoint is computed using discrete latent assignments.
Modelling a manifold with multiple charts imposes drawbacks. For one, each encoder-decoder pair has its own latent space, so for given datapoint E .M, choosing the correct encoder can be a challenge. This makes it unclear how to perform tasks involving the manipulation of latent representations, such as interpolation. In many cases, there is no single correct encoder, as the images of various decoders need to overlap to correctly define topologically complex manifolds (such as Figure 10(c)). The ambiguity of choosing the correct encoder underlined by how differently cach of the aforementionod mothods attempt to do so.
5.4.3 Disconnected Manifolds
Another common source of topological complexity in the data manifold is when it consists more than one connected component. This situation occurs, for example, in datasets with multiple disjoint classos. In this context, theoretical analyses have shown that any decoder-based model will suffer from training instability (Salmona el. al. 2022) and poor sample quality (Luzi et al. 2020
One technique to improve sample quality is to avoid sampling from latent regions where the network go- is unstable. Tanielian et al. (2020) propose, in the context of generative adversarial networks (Section 4.2 and [Section 5.2.1J, to reject samples X= go- (Z), where Z ~ p², for which Vzgo- (2) has a high Frobenius norm, which they show indicates an off-manifold sample. Other work, discussed below, seeks to avoid instability entirely during training.
A few general techniques have been proposed for modelling disconnected manifolds. One way to use disconnected (or near-disconnedte)) latent distributions, which aims to match the topology of the support of P with that of M. This is typically done with a Gaussian mixture model for pd and has been proposed for multiple classes of generative model (Nalisnick et al.) 2016 Dilokthanakul et al. 2016 Jiang et al. 2017
Ben- Yosef& & Weinshall 2018 Izmailov et al. 2020).
Note that the theoretical analysis of Salmona et 2022 shows numerical instability when p is multimodal, in which rase its support M can be considered as numerically disconnected.
49

---

Published in Transactions on Machine Learning Research (09/2024)
A related approach is to use multiple decoder networks in similar manner to the aforementioned multi- chart methods from (Section 5.4.2. The model then becomes mixture p (+)= neph, (z) of generative submodels ve where "i...., T are the mixture weights (sometimes trainable). For example, Arora et al.) (2077) propose to directly train a mixture of generative adversarial networks to stabilize training, wherein the entire mixture is learned with backpropagation. Cornish et al.
(2020) use hierarchical paatimoously-indeseee mixture of normalizing flows (Section 4.1.3). Other work uses techniques based on
2017 Locatello et al. 2018). A key challenge in this line of work is to encourage different submodels to model distinct parts of the distributions. This can be done by partitioning the data beforehand, by class (Luzi et al., 2020) or through unsupervised clustering (Brown et al. 2023), and training model on each partition. A more fiexible approach to backpropagate through an ancillary classification model to encourage each submodel to generate data from distinct manifolds [Hoang et al.
(Dempster et al. 1977] to train the mixture (Banijamali et. al.
2018 Khayatkhoei et al. 2018; Ghosh et al. 2018
While all the models mentioned above can properly account for some topological features of M such as disconnectedness, we highlight that most are nonetheless manifold-unawree. For example, full-dimensional modols trained through maximum-likeliooon remain exposed to the corresponding pathologics (Section 4.1)
even when they are mixture models.
6 Discrete Deep Generative Models
Since this survey's focus is on the manifold hypothesis, all of the models presented thus far are for continuous distributions. Nonetheless, many DGMs assume that the ambient space X' is discrete. For example, images can be modelled as having pixels which take only finitely many different values, rather than continuum of them. In this case, v. and p are both probability mass functions over X, and M C X denotes the support of p.N. Formally, in this setting X' is a 0-dimensional manifold, so that even when M is a strict subset of X, it remains a O-dimensional submanifold. In other words, there can be no dimensionality mismatch for discrete data since = d' = O. In turn, this implies that mathematically, discrete Rkeelhhoddbbsee DGMs are not exposed to problems such as manifold overfitting (Section 4.1) which arise from dimensionality mismatch. This view of discrete DGMs through the manifold lens is useful, since it suggests that whenever manifold-unaware DGM admits a straightforward discrete analogue, the latter should be preferred as it will be unaffected by manifold-related woes. Indeed, as mentioned in [Section 4.1.2. discrete variational autoencoders (Gulrajani et al.) 2017b Vahdat & Kautz 2020 Vahdat et al. 2021 empirically outperform their continuous variants. Similarly, discrete incarnations of likelihood-based a autoregressive DGMs (Germain ct al. 2015 van den Oord et al. 2016: Salimans et al. 2017 Parmar et al. 2018) outperform continuous ones
(Uria el al. 2013). In contr discrete versions of diffusion models (Austin et al. 2021 Campbell el al.
2022 Meng et al. (202)) do not outperform their manifold-aware continuous counterparts (Section 5.1.2
when modelling images.
Discrete and continuous DGMs nonetheless have similarities, despite the differences outlined above. As discussed in /Section 1 a key motivation behind the manifold hypothesis is to capture the intuition that M, the support of p.a is somehow sparse within X. This intuition often remains true in the discrete case: using images as an example once again, there are 256 possible discrete images (assuming each pixel entry takes one of 256 possible values), yet the subset of natural images is vanishingly small in comparison and contains orders of magnitude fewer elements. The main idea of continuous two-step models (Section 5.3), namely to first approximate the support of p. and then learn the distribution within, remains equally sensible in the discrete case. van den Oord et al. (2017) proposed an autoencoder which recovers discrete representations over which they train a discrete DGM; this idca that has been further developed, with strong empirical results (Razav el al.] 2019 Esser et al.; 2021 Ramesh et. al.) 2021 Chang et al., 2022). We finish by pointing out that the discussion in [Section 5.3.1 applies to all these discrete two-step models, so that they can be interpreted as minimizing potentially regularized upper bound of the Wasserstein distance between p and p which becomes tight at optimality, because in the discrete case, perfect reconstructions are always achievable given enough capacity of the encoder and decoder.
50

---

Published in Transactions on Machine Learning Research (09/2024)
7 Conclusions and Future Outlook
Conclusions In this survey we have carried out a review of deep generative models through the lens of the manifold hypothesis. This viewpoint presents a mathematically elegant perspective of DGMs, and suggests that manifold -wareness is an important necessary condition for strong empirical performance. We thus encourage rescarchers who are developing new DGMs to consider manifold-awareness as a desideratum, and ask themselves: Can my deep generative model leorn distributions supported on wnknown low-dimensional manifolds? When the answer is yes, remonstrating this fact. will strengthen the work's 's motivation; and when the answer is no, this suggests that the DGM can be improved by endowing it with manifold-wwarenes either through model-specific fix, or at least by training it on latent space as a two-step model (Section 5.3) We also showed that numerical instabilities of likelihood-evallaa (Section 4.1.1) and that two-step models can be interpreted as minimizing a (potentially regularized) upper bound of the Wasserstein distance objective (Section 5.3.1).
are unavoidable in the manilold set ting
Future outlook Finally, we outline a non-exhaustive list of research directions involving deep generative models and their interplay with the manifold hypothosis. We believe these lines of inquiry are interesting, and mostly unexplored at the time of writing:
Further understanding dimensionality mismatch The effects of using a full-dimensional model when the ground truth distribution is manifold-supported are well understood for likelihood- based models (Section 4.1] and diffusion models (Section 5.1.2), yet our grasp of the interplay between DGMs and the manifold hypothesis remains incomplete. For example, a theoretical un- derstanding of score matching (Section 4.3) and conditional flow matching (Section 5.1.3) under misspecified dimension is lacking, as is the effect of using lower-bounder energy functions in energy- based models (Section 4.1.4).
Improved training of DGMs with two-step architectures Two-step models as presented in Section 5.3 are manifold-aware. Yet, as also discussod in [Section 5.3 two-step training does not encourage the encoder from the first step lo represent the data in a way conducive to distribution learning in the second step. Intuitively, this means there is room for improvement in how these models are trained, and since the end-to-end approaches described at the end of [Section 5.3 are in general manfold--unwwree, several avenues remain open. For example, despite the existence of regularizers for training autoencoders (Larsen et al. 2016 Higgins et al. 2017 Nazari et al., 2023). there is very little work explicitly designing autoencoders for two-step training. The only work we are aware of in this direction is by Hu et al.] (2023). who propose to split the first step into two sub-steps: in the first sub-step the encoder is trained along with low-capacity decoder, and in the sccond sub-step the encoder is frozen and more flexible decoder is trained. Another avenue is finding end-to-end objective to train this type of model in a manifold-aware fashion. Current end- to-end methods are manifold-unawee unaaare, despite providing a desirable inductive bias - an exception being generalized energy-based models (Section 5.2.4) which cannot be readily extended beyond using energy-based models (Section 4.1.4) as the latent distribution. We thus hypothesize that any end-to-end, or improved two-step, manifold-aware procedure which can train diffusion models in latent space while scaling to massive datasets (Schuhmann et al. 2022) is likely to improve upon current commercial versions of latent diffusion models (Section 5.3.2).
Extracting and leveraging manifold information Any manifold-aware DGM which succeeds at learning its target distribution pX must have learned its support M as well, albeit perhaps implic- itly. Extracting information about M from trained DGM is thus natural problem, as is leveraging this information for any practical use. Various works have shown that trained DGMs induce Rie-
mannian metrics over the learned manifolds (Shao et al., 2018 Arvanitidis et al. 2018 Chadebec & Allassonnière 2022 Sorrenson el. al.) 2024a), which can in turn be leveraged for interpolating between datapoints and for improved sampling procedures. Several works have also shown that DGMs can be used to estimate the intrinsic dimension of . (Tempezyk et al., 2022 Zhong et al., 2022; Horvat & Pfister 2204 Kamkari et al., (2024b] Stanczuk et al., (2024). and these quantities have already proven useful for unsupervised out-of-distribution detection and to identify memorized
51

---

Published in Transactions on Machine Learning Research (09/2024)
samples (Kamkari et al. 2024a Ross et al. 2024 Humayun al. 2024 The field of topological data analysis (Chazal & Michel 2021 aims to extract topological and geometric features of M such as intrinsic dimension from an observed dataset, conventionally without the use of DGMs. Another fruitful direction for future research will involve further exploiting DGMs for topological
data analysis.
Finite-sample convergence rates All the analyses presented here assumed the nonparametric regime (Section 2.2). As we have soen throughout our survey, this simplifying assumption enables a useful and practical understanding of DGMs through the lens of the manifold hypothesis. Yet, this assumption remains unrealistic since in practice expectations with respect to p cannot be computed; p. must thus be approximated via its empirical distribution - i.c. a mixture of (oqually weighted) point masses at the (finitely many) observed datapoints. Formally, the empirical distribution is supported on a O-dimensional submanifold of X' namely, the observed dataset - so that any flexible enough and sufficiently well optimized manifold-aware DGM should simply memorize its entire training dataset. Evidently manifold-awarrnees remains desirable property - statistical consistency under the manifold hypothesis is impossible without it but the fact that state-of- saate-offthheaar the-art DGMs do not suffer from total memorization cannot be explained while assuming the nonparametric regime. Thus, understanding what drives DGMs to generalize rather than memorize remains a relevant problem. Kadkhodaie et al. (2024) study these questions for diffusion models through the lens of the inductive biases provided through the architecture of the score network. More formal explanations of generalization are provided by statistical learning theory in the form of finito-sample convergence rales. Although these results often do not. assumne the manifold hypothesis, & recent. line of work has, obtaining in turn much faster convergence rates which depend on intrinsic rather than ambient dimension (Schreuder et al.]
2021
Huang et al. 2022 Dahal et al. 2022 Tang & Yang, Chakraborty & Bartlett 2024b & Hu
2023
Chae et al.] 2023 Chen et al.
2023
Oko et al. 2023
et al. 2024 Vardanyan et al. 2024
Tang & Yang 2024). We believe that this research direction
provides a challenging but highly promising avenue for a full theoretical understanding of DGMs.
References
David H Ackley, Geoffrey E Hinton, and Terrence J Sejnowski. A learning algorithm for Boltzmann machines. Cognilive Science, 9(1):147-169, 1985.
Michael S Albergo and Eric Vanden- Eijnden. Building normalizing flows with stochastic interpolants. In International Conference on Learning Representations, 2023.
Alexander A Alemi, Ian Fischer, Joshua V Dillon, and Kevin Murphy. Deep variational information bottle- neck. In International Conference on Learning Representations, 2017.
Brian DO Anderson. Reverse-time diffusion equation models. Stochastic Processes and their Applications, 12(3):313-326, 1982.
Michael Arbel, Danica J Sutherland, Mikolaj Binkkwski,, and Arthur Gretton. On gradient regularizers for MMD GANs. In Advances in Newral Information Processing Systems, 2018.
Michael Arbel, Liang Zhou, and Arthur Gretton. Generalized energy based models. In International Con- ference on Learning Representations, 2021.
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In International Conference on Machine Learning, 2017.
Sanjeev Arora, Rong Ge, Yingyu Liang, Tengyu Ma, and Yi Zhang. Generalization and equilibrium generative adversarial nets (GANs). In International Conference on Machine Learning, 2017.
Georgios Arvanitidis, Lars Kai Hansen, and Soren Hauberg. Latent space oddity: On the curvature of deep generative models. In International Conference on Learning Representations, 2018.
52

---

Published in Transactions on Machine Learning Research (09/2024)
Jacob Austin, Daniel D Johnson, Jonathan Ho, Daniel Tarlow, and Rianne Van Den Berg. Structured denoising diffusion models in discrete state-spaces. In Advances in Neural Information Processing Systems. 2021.
Jonathan Bac, Evgeny M Mirkes, Alexander N Gorban, Ivan Tyukin, and Andrei Zinovyev. Scikit-dimension: A python package for intrinsic dimension estimation. Entropy, 23(10), 2021.
Ershad Banijamali, Ali Ghodsi, and Pascal Poupart. Generative mixture of networks. In International Joint Conference on Newral Networks (IJONN), 2017.
Serguei Barannikov, llya Trofimov, Nikita Balabin, and Evgeny Burnaev. Representation topology diver- gence: A method for comparing neural network representations. In International Conference on Machine Learning, 2022.
Atilim Günes Baydin, Barak A Pearlmutter, Alexey Andreyevich Radul, and Jeffrey Mark Siskind. Automatic diffcrentiation in machine learning: survey. Journal of Machine Learning Research, 18:1-43, 2018.
Richard Beals, David H Krantz, and Amos Tversky. Foundations of multidimensional scaling. Psychological Review, 75(2):127, 1968.
Jens Behrmann, Paul Vicol, Kuan-Chieh Wang, Roger Grosse, and Jörn-Henrik Jacobsen. Understanding and miligating exploding inverses in invertible neural networks. In International Conference on Arlificial Intelligence and Statistics, 2021.
Heli Ben-Hammn Samuel Cohen, Joey Bose, Brandon Amos, Maximillian Nickel, Aditya Grover, Ricky TQ Chen, and Yaron Lipman. Matching normalizing flows and probability paths on manifolds. In International Conference Machine Learning, 2022.
Matan Yosef and Daphna Weinshall. Gaussian mixture generative adversarial networks for diverse datasets, and the unsupervised clustering of images. arXiv:1808. 10356, 2018.
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspec- tives. IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(8):1798 1198-119988888, 2013.
Clément Berenfeld and Mare Hoffmann. Density estimation on an unknown submanifold. Electronic Journal of Statistics, 15(1):2179 2223, 2021.
Clément Berenfeld, Paul Rosa, and Judith Rousseau. Estimating a density near an unknown manifold: A Bayesian nonparametric approach. The Annals of Statistics, 2024. To Appear.
Patrick Billingsley. Probobility and Measure. Wiley, 2012.
Mikolaj Binkowski, Danica J Sutherland, Michael Arbel, and Arthur Gretton. Demystifying MMD GANs. In International Conference on Learning Representations, 2018.
Jeremiah Birrell, Paul Dupuis, Markos A Katsoulakis, Yannis Pantazis, and Luc Rey-Bellet. Y) Divergences: Interpolating between f-divergences and integral probability metrics. The Journal of Ma- chine Learning Research, 23:1816-1885, 2022.
Vanessa M Boehm and Uros Seljak. Probabilistic autoencoder. Transactions of Machine Learning Research. 2022.
Vladimir Igorevich Bogachev. Measure Theory, volume 2. Springer, 2007.
Sam Bond- Taylor, Adam Leach, Yang Long, and Chris G Willcocks. Deep generative modelling: A compara- tive review of VAEs, GANs, normalizing flows, energy-based and autoregressive models. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44 44(11):7327-7347, 2022.
Clément Bonet, Lucas Drumetz, and Nicolas Courty. Sliced- Wasseerten distances and flows on Cartan- Hadamard manifolds. arXim:2403.06500 0666000, 2024.
53

---

Published in Transactions on Machine Learning Research (09/2024)
Ali Borji. Pros and cons of GAN evaluation measures. Computer Vision and Image Understanding, 179: 41-65, 2019.
Joey Bose, Ariella Smofsky, Renjie Liao, Prakash Panangaden, and Will Hamilton. Latent variable modelling with hyperbolic normalizing flows. In International Conference on Machine Learning, pp- 1045-1055, 2020.
Denis Boyda, Gurtej Kanwar, Sébastien Racanière, Danilo Jimenez Rezende, Michael S Albergo, Kyle Cran- mer, Daniel C Hackett, and Phiala E Shanahan. Sampling using SU(N) gauge equivariant flows. Physical Review D, 103(7):074504,, 2021.
Johann Brehmer and Kyle Cranmer. Flows for simultancous manifold learning and density estimation. In Advances in Newral Information Processing Systems, 2020.
Bradley CA Brown, Anthony L Caterini, Brendan Leigh Ross, Jesse C Cresswell, and Gabriel Loaiza Ganem. Verifying the union of manifolds hypothesis for image data. In International Conference on Learning Representations, 2023.
Marcus Brubaker, Mathieu Salzmann, and Raquel Urtasun. A family of MCMC methods on implicitly defined manifolds. In International Conference on Artificial Intelligence and Statistics, 2012.
Andrew Campbell, Joe Benton, Valentin De Bortoli, Thomas Rainforth, George Deligiannidis, and Arnand Doucet. continuous time framework for discrete denoising models. In Advances in Neurol Information Processing Systems, 2022.
Anthony L Caterini, Rob Cornish, Dino Sejdinovic, and Arnaud Doucet. Variational inference with contimoously-indexxd normalizing flows. In Uncertainty in Artificial Intelligence, 2021a.
Anthony L Caterini, Gabriel Loaiza-Ganes Geoff Pleiss, and John P Cunningham. Rectangular flows for manifold learning. In Advances in Neural Information Processing Systems, 2021b.
Clément Chadebec and Stéphanie Allassonnière. A geometric perspective on variational autoencoders. Tn Advances in Neural Information Processing Systems, 2022.
Minwoo Chae, Dongha Kim, Yongdai Kim, and Lizhen Lin. A likelihood approach to nonparametric esti- mation a singular distribution using deep generative models. Journal of Machine Learning Research, (4(77))1-422 2023.
Saptarshi Chakraborty and Peter Bartlett. A statistical analysis of Wasserstein autoencoders for intrinsically low-dimensional data. In International Conference on Learning Representations, 2024a.
Saptarshi Chakraborty and Peter T, Bartlett. On the statistical properties of generative adversarial models for low intrinsic data dimension. arXiv:2401.15801, 2024b.
Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T Freeman. MaskGIT: Masked generative image transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.
Frédéric Chazal and Bertrand Michel. An introduction to topological data analysis: Fundamental and practical aspects for data scientists. Frontiers in Artificial Intelligence, 4:667963, 2021.
Tong Che, Ruixiang Zhang, Jascha Sohl-Dickstein, Hugo Larochelle, Liam Paull, Yuan Cao, and Yoshua Bengio. Your GAN is secretly an easegy-based model and you should use discriminator driven latent sampling. In Advances in Neural Information Processing Systems, 2020.
Jianfei Chen, Cheng Lu, Biqi Chenli, Jun Zhu, and Tian Tian. VFlow: More expressive generative flows with variational data augmentation. In International Conference Machine Learning, 2020.
Minshuo Chen, Kaixuan Huang, Tuo Zhao, and Mengdi Wang. Score approximation, estimation and dis- tribution recovery of diffusion models on low-dimensional data. In International Conference on Machine Learning, 2023.
54

---

Published in Transactions on Machine Learning Research (09/2024)
Ricky TQ Chen and Yaron Lipman. Flow matching on general geometries. In International Conference Learning Represendations, 2024.
Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations, In Advances in Neural Information Processing Systems, 2018.
Xi Chen, Diederik P Kingma, Tim Salimans, Yan Duan, Prafulla Dhariwal, John Schulman, llya Sutskever, and Pieter Abbeel. Variational lossy autoencoder. In International Conference Learning Representa- lions, 2017.
Rewon Child. Very deep VAFs generalize autoregressive models and can outperform them on images. International Conference on Leurning Representations, 2021.
Rob Cornish, Anthony L Caterini, George Deligiannidis, and Arnaud Doucet. Relaxing bijectivity constraints with continuously indexed normalising flows. In International Conference Machine Learning, 2020.
Jesse C' Cresswell, Brendan Leigh Ross, Gabriel
Humberto Reyes-Gonzalee Gonzalez, Marco Letizia, and Anthony T. Caterini. CaloMan: Fast generation of calorimeter showers with density estimaatio on learned manifolds. In NearIPS Workshop on Machine Learning and the Physical Sciences, 2022.
Edmond Cunningham, Adam D Cobb, and Susmit Jha. Principal component flows. In International Con- ference on Machine Learning, 2022.
Biraj Dahal, Alexander Havrilla, Minshuo Chen, Tuo Zhao, and Wenjing Liao. On deep generative models for approximation and estimation of distributions on manifolds. In Advances in Newral Information Processing Systems, 2022.
Bin Dai and David Wipf. Diagnosing and enhancing VAE models. In International Conference on Learning Representations, 2019.
Quan Dao, Hao Phung, Binh Nguyen, and Anh Tran. Flow matching in latent space. arXiv:2307.08608, 2023.
Anirban DasGupta. Asymptotic Theory of Statistics and Probability. Springer, 2008.
Valentin De Bortoli. Convergence of denoising diffusion models under the manifold hypothesis. Transactions on Machine Learning Research, 2022.
Valentin De Bortoli, Emile Mathicu, Michael Hutchinson, James Thornton, Yee Whye Tch, and Arnaud Doucet. Riemannian score-based generative modeling. In Advances in Neural Information Processing Systems, 2022.
Arthur Dempster, Nan M Laird, and Donald B Rubin. Maximum likelihood from incomplete data via the EM algorithm. Journal of the Royal Statistical Society: Series B (Methodological), 39(1):1-22, 1977.
Jean Dieudonné. Treatise on Analysis, volume 3. Academic Press, 1973.
Nat Dilokthanakul, Pedro AM Modiano, Marta Garnelo, Matthew CH Lee, Hugh Salimbeni, Kai Arulku- maran, and Murray Shanahan. Deep unsupervised clustering with Gaussian mixture variational autoen- coders. ar Xiv: 1611 1.02648, 2016.
Laurent Dinh, David Krueger, and Yoshua Bengio. NICF: Non-linear independent components estimation. In ICLR Workshop Track, 2015.
Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using Real NVP. In International Conference on Learning Representations, 2017.
Vincent Divol. Measure estimation on manifolds: An optimal transport approach. Probability Theory and Related Fields, (83(1):581-66) 2022.
55

---

Published in Transactions on Machine Learning Research (09/2024)
Jeff Donahue, Philipp Krähenbüihl, and Trevor Darrell. Adversarial feature learning. In International Con- ference on Learning Represenlations, 2017.
Felix Draxler, Peter Sorrenson, Lea Zimmermann, Armand Rousselot, and Ullrich Köthe. Free-form flows: Make any architecture a normalizing flow. In International Conference on Artificial Intelligence and Statistics, 2024.
Yilun Du and Igor Mordatch. Implicit generation and modeling with energy based models. In Advances in Newral Information Processing Systems, 2019.
Emilien Dupont, Arnaud Doucet, and Yee Whye Tch. Augmented neural ODEs. In Advances in Neural Information Processing Systems, 2019.
Conor Durkan, Artur Bekasov, lain Murray, and George Papamakarios. Neural spline flows. In Advances in Neural Information Processing Systems, 2019.
Gintare Karolina Dziugaite, Daniel M Roy, and Zoubin Ghahramani. Training generative neural networks via maximum mean discrepancy optimization. In Uncertainty in Artificial Intelligence, 2013.
Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVE Conference on Compuler Vision and Pallern Recognition, 2021.
Elena Facco, Maria d'Er d'Errico, Alex Rodriguez, and Alessandro Laio. Estimating the intrinsic dimension of datasets by a minimal neighborhood information. Scientific Reports, 7(1):12140, 2017.
Kyriakos Flouris and Ender Konukoglu. Canonical normalizing flows for manifold learning. In Advonces in Newral Information Processing Systems, 2023.
Sylvestre Gallot, Dominique Hulin, and Jacques Lafontaine. Riemannian Geometry. Springer, 3rd edition, 2004.
Xuefeng Gao and Lingjiong Zhu Convergence analysis for general probability flow ODEs of diffusion models Wasserstein distances. 211.:2401.175555 2024.
Mevlana C Gemici, Danilo Jimenez Rezende, and Shakir Mohamed. Normalizing flows on Riemannian manifolds. In NewrIPS Workshop on Bayesian Deep Learning, 2016.
Mathieu Germain, Karol Gregor, lain Murray, and Hugo Larochelle. MADE: Masked autoencoder for distribution estimation. In International Conference on Machine Learning, 2015.
Benyamin Ghojogh, Mark Crowley, Fakhri Karray, and Ali Ghodsi. Elements of Dimensionality Reduction and Manifold Learning. Springer Nature, 2023.
Arnab Ghosh, Viveka Kulharia, Vinay P Namboodiri, Philip HS Torr, and Puneet K Dokania. Multi-agent diverse generative adversarial networks. In Proceedings of the IEEE/CVF Conference Computer Vision and Pallern Recognilion, 2018.
Partha Ghosh, Mehdi SM Sajjadi, Antonio Vergari, Michael Black, and Bernhard Schölkopf. From variational to deterministic autoencoders. In International Conference on Learning Representations, 2020.
Ian Goodfellow, Jean Pouget-Abadic, Mchdi Mirza, Bing Xu, David Warde-Farlyy, Sherjil Ozair, Aaron Courville, and Yoshua. Bengio. Generative adversarial nets. In Advances in Neural Information Processing Systems, 2014.
Will Grathwohl, Ricky TQ Chen, Jesse Bettencourt, Nya Sutskever, and David Duvenaud. FFJORD: Free-form continuous dynamics for scalable reversible generative models. In International Conference on Learning Representations, 2019.
Will Grathwohl, Kuan-Chich Wang, Joern-Henrik Jacobsen, David Duvenaud, Mohammad Norouzi, and Kevin Swersky. Your classifier secretly an energy based model and you should treat it like one. In International Conference on Learning Representations, 2020.
56

---

Published in Transactions on Machine Learning Research (09/2024)
Alfred Gray. The volume of small geodesic ball of a Riemannian manifold. Michigan Mathematical Journal, 20(4):329 344,, 1974.
Arthur Gretton, Karsten Borgwardt, Malte Rasch, Bernhard Schölkopf, and Alex Smola. kernel method for the two-sample-problen In Advances in Neural Information Processing Systems, 2006.
Hyemin Gu, Markos A Katouslakis, Luc Rey-Bellet, and Benjamin J Zhang. Combining Wasserstein-1 and W proxinals: Robust manifold learning via well-posed generative flows. arXiv:2407.1100. 2024.
Ishaan Guirajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron Courville. Improved training of Wasserstein GANs. In Advances in Neural Information Processing Systems, 2017a.
Ishaan Guirajani, Kundan Kumar, Faruk Ahmed, Adrien Ali Taiga, Francesco Visin, David Vazquez, and Aaron Courville. PixeIVAE: A latent variable model for natural images. In International Conference on Learning Representations, 2017b.
UG Haussmann and E Pardoux. Time reversal of diffusions. The Annals of Probabilily, 14(4):1188 1205, 1986.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE/CV Conference on Computer Vision and Pattern Recognation, 2016.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs trained by a two time-scale update rule converge to local Nash equilibrium. In Advances in Neural Information. Processing Systems, 2017.
Trina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mo- hamed, and Alexander Lerchner. beta-VAl: Learning basic visual concepts with a constrained variational framework. In International Conference on Learning Representations, 2017.
Jonathan Ho, Xi Chen, Aravind Srinivas, Yan Duan, and Picter Abbeel. Flow++: Improving flow-based
generative models with variational dequantization and architecture design. Tn International Conference on Machine Learning, 2019.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Aduances in Neural Information Processing Systems, 2020.
Quan Hoang, Tu Dinh Nguyen, Trung Le, and Dinh Phung, MGAN: Training generative adversarial nets with multiple generators. In International Conference on Learning Representations, 2018.
Kurt Hornik, Approximation capabilities of multilayer feedforward networks. Neural Networks, 4(2):251-20) 2577, 1991.
Christian Horvat and Jean-Pascal Pfister. Denoising normalizing flow. In Advances in Neural Information Processing Systems, 2021.
Christian Horvat and Jean-Pascal Pfister. Density estimation on low-dimensional manifolds: An inflation-
deflation approach. Journal of Machine Learaing Research, (161):113 37, 2023.
Christian Horvat and estimation in diffusion models. In International Conference on Learning Representations, 2024.
Pfister. On gauge freedom, conservativity and intrinsic dimensionality
Jerry Yao-Chieh Hu, Weimin Wu, Zhuoru Li, Zhao Song, and Han Liu. On statistical rates and provably efficient criteria of latent diffusion transformers (DiTs). arXic:2407.01077, 2024.
Tianyang Hu, Fei Chen, Haonan Wang, Jiawei Li, Wenjia Wang, Jiacheng Sun, and Zhenguo Li. Com- plexity matters: Rethinking the latent space for generative modeling. In Advances in Neural Information Processing Systems, 2023.
57

---

Published in Transactions on Machine Learning Research (09/2024)
Chin-Wei Huang, Laurent Dinh, and Aaron Courville. Augmented normalizing flows: Bridging the gap between generative flows and latent variable models. or orXim:2002.071 071001 2020.
Jian Huang, Yuling Jiao, Zhen Li, Shiao Liu, Yang Wang, and Yunfei Yang. An error analysis of generative adversarial networks for learning distributions. Journal of Machine Learning Research, 23(1):5047-5089 2022.
Ahmed Imtiaz Humayun, Ibtihel Amara, Candice Schumann, Golnoosh Farnadi, Negar Rostamzadeh, and Mohammad Havaei. On the local geometry of deep generative manifolds. In TCML, Workshop on Geomelry- Grounded Representation Learning and Generative Modeling, 2024.
Witold Hurewicz and Henry Wallman. Dimension Theory (PMS-4). Princcton University Press, 1948.
M F Hutchinson. A stochastic estimator of the trace of the influence matrix for Laplacian smoothing splines. Communications in Statistics Simulation and Computation, 18(3):1059-1)) 1076, 1989.
Aapo Hyvärinen. Estimation of non-normalized statistical models by score matching. Journal of Machine Learning Research, 6(24):695 709, 2005.
Pavel Izmailov, Polina Kirichenko, Marc Finzi, and Andrew Gordon Wilson. Scmi-supervised learning with normalizing flows. In International Conference on Machine Learning, pp. 4615-4630, 2020.
Varuna Jayasiri and Nipun Wijerathne. Annotated paper implementations, 2020. URL https://nn. labml. ai/
Zhuxi Jiang, Yin Zheng, Huachun Tan, Bangsheng Tang, and Hanning Zhou Variational deep embedding: An unsupervised and generative approach to clustering. In Proceedings of the Twenly-Sie Sixth International Joint Conference on Artificial Intelligence (IJCAI), 2017.
Kerstin Johnsson, Charlotte Soneson, and Magnus Fontes. Low bias local intrinsic dimension estimation from expected simplex skewness. IEEE Transactions on Pattern Analysis and Machine Intelligence, 37 (1):196-202, 2014.
Zahra Kadkhodaie and Eero P Simoncelli. Stochastic solutions for linear inverse problems using the prior implicit in a denoiser. In Advances in Neural Information Processing Systems, 2021.
Zahra Kadkhodaie, Florentin Guth, Fero P Simoncelli, and Stéphane Mallat. Generalization diffusson models arises from geometry-adaptive harmonic representations. In International Conference on Learning Representations, 2024.
Dimitris Kalatzis, Johan Ziruo Ye, Alison Pouplin, Jesper Wohlert, and Søren Hauberg, Density estimation smooth manifolds with normalizing flows. arXin:2106.03500, 2021.
Hamidreza Kamkari, Brendan Leigh Ross, Jesse C Cresswell, Anthony L Caterini, Rahul G Krishnan, and Gabricl Loaiza-Ganem. geometric explanation of the likelihood OOD detection paradox. In International Conference on Machine Learning, 2024a.
Hamidreza Kamkari, Brendan Leigh Ross, Rasa Hosseinzadeh, Jesse C Cresswell, and Gabriel Loaiza- Gantza A geometric view of data complexity: Efficient local intrinsic dimension estimation with diffusion models. In ICML Workshop Structured Probabilistic Inference and Generative Modeling, 2024b.
Gurtej Kanwar, Michael S Albergo, Denis Boyda, Kyle Cranmer, Daniel C Hackett, Sébastien Racaniere, Danilo Jimenez Rezende, and Phiala E. Shanahan. Equivariant. flow-based sampling for lattice gauge theory. Physical Review Letters, 125(12):121601, 2020.
Kacper Kapusniak, Peter Potaptchik, Teodora Reu, Loo Zhang, Alexander Tong, Michael Bronstein, Avishek Joey Bose, and Francesco Di Giovanni. Metric flow matching for smooth interpolations on the data manifold. arXim:2405.1,2 2024.
58

---

Published in Transactions on Machine Learning Research (09/2024)
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of GANs for improved quality, stability, and variation. In International Conference. on Learning Representalions, 2018.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generalive adversarial networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019.
Tero Karras, Samuli Laine, Mika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of StyleGAN. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020.
Isay Katsman, Aaron Lou, Derek Lim, Qingxuan Jiang, Ser Nam Lim, and Christopher M De Sa. Equivariant manifold flows. In Advances in Neural Information Processing Systems, 2021.
Hassan K Khalil. Nonlinear Systems. Prentice Hall, 2002.
Mahyar Khayatkhoei, Maneesh K Singh, and Ahmed Elgammal. Disconnected manifold learning for gener- ative adversarial networks. In Advances in Neural Information Processing Systems, 2018.
Dongjun Kim, Seungjae Shin, Kyungwoo Song, Wanmo Kang, and Il-Chul Moon. Soft truncation: A univer- sal training technique of based diffusion model for high precision score estimation. In International Conference Machine Learning, 2022.
Hyeongju Kim, Hyeonseung Lee, Woo Hyun Kang, Joun Yeop Lee, and Nam Soo Kim. Softflow: Probabilistic framework for normalizing flow on manifolds. In Advances in Neural Information Processing Systems, 2020.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations, 2015.
Diederík P Kingma and Prafulla Dhariwal. Glow:
Generative flow with invertible 1x1 convolutions. Tn Advances in Newral Information Processing Systems, 2018.
Diederik P Kingma and Ruiqi Gao. Understanding diffusion objectives as the ELBO with simple data
augmentation. In Advunces in Newral Information Processing Systems, 2023.
Diederik P Kingma and Max Welling. Auto-encoding variational Bayes In International Conference
Learning Representations, 2014.
Diederik P Kingma, Tim Salimans, Rafal Jozefowicz, Xi Chen, Ilya Sutskever, and Max Welling. Improved variational inference with inverse autoregressive flow. In Advances in Neural Information Processing Systems, 2016.
Ivan Kobyzev, Simon JD Prince, and Marcus Brubaker. Normalizing flows: An introduction and review of current methods. IEEE Transactions on Pattern Analysis and Machine Intelligence, 43(11):3964-3977 2020.
Frederic Koehler, Viraj Mehta, and Andrej Risteski. Representational aspects of depth and conditioning in normalizing flows. In International Conference on Machine Learning, 2021.
Frederic Koehler, Viraj Mehta, Chenghui Zhou, and Andrej Risteski. Variational autoencoders in the pres- ence of low-dimensional data: Landscape and implicit bias. In International Conference Learning Representations, 2022.
Soheil Kolouri, Phillip E Pope, Charles E Martin, and Gustavo Rohde. Sliced Wasscrstein auto-cncoders. In International Conference on Learning Representations, 2018.
Konik Kothari, AmirEhsan Khorashadizadeh, Maarten de Hoop, and Ivan Dokmanić. Trumpets: Injective flows for inference and inverse problems. In Uncertainty in Artificial Intelligence, 2021.
Ulrrich Köthe. A review of change of variable formulas for generative modeling. ar Xxx::2308..006 0265,, 2023.
59

---

Published in Transactions on Machine Learning Research (09/2024)
Mark A Kramer. Noulinear principal component analysis using autoassociative neural networks. AIChE Journal, (7()):233-244 1991.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from Liny images. report, University of Toronto, 2009.
Technical
Joseph B Kruskal. Multidimensional scaling by optimizing goodness of fit to nonmetric hypothesis. Psy- chometrika, 29(1):1 27, 1964.
Abhishck Kumar, Ben Poole, and Kevin Murphy. Regularized autocncoders via relaxed injective probability flow. In International Conference on Artificial Intelligence and Stalislics, 2020.
Dohyun Kwon, Ying Fan, and Kangwook Lee. Score-based generative modeling secretly minimizes the Wasserstein distance. In Advances in Neural Information Processing Systems, 2022.
Anders Boesen Lindbo Larsen, Søren Kaae Sønderby, Hugo Larochelle, and Ole Winther. Autoencoding beyond pixels using learned similarity metric. In International Conference Machine Learning, 2016.
Hyunjong Lee, Yedarm Seong, Sungdong Lee, and Joong-Ho Won. StrWAFs to invariant representations. In International Conference on Machine Learning, 2024.
John M Lec. Introduction to Smooth Manifolds. Springer, 2nd edition, 2012.
John M Lee. Introduction to Riemannian Manifolds. Springer, 2nd edition, 2018.
Erich Lehmann and George Casella. Theory of Point Estimation. Springer Science & Business Media, 2006.
Elizaveta Levina and Peter Bickel. Maximum likelihood estimation of intrinsic dimension. In Advances in Newral Information Processing Systems, 2004.
Chun-Liang Li, Wei-Cheng Chang, Yu Cheng, Yiming Yang, and Barnabás Póczos. MMD GAN: Towards deeper understanding of moment matching network. In Advances in Neural Information Processing Sys- tems, 2017.
Yujia Li, Kevin Swersky, and Richard Zemel. Generative moment matching networks. Conference on Machine Learning, 2015.
In International
Shuyu Lin, Stephen Roberts, Niki Trigoni, and Ronald Clark. Balancing reconstructionn quality and regular- isation in evidence lower bound for variational autoencoders. arXic:1909.033 03765, 2019.
Yaron Lipman, Ricky T Q Chen, Heli Ben-Hamu, Maximilian Nickel, and Matthew Le. Flow matching for generative modeling. In International Conference on Learning Representations, 2023.
Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. In International Conference on Learning Representations, 2023.
Gabriel Loaiza- Ganem, Brendan Leigh Ross, Jesse C Cresswell, and Authony Caterini. Diagnosing and fixing manifold overfitting in deep generative models. Transactions on Machine Learning Research, 2022a.
Gabriel Loaiza-Ganem, Brendan Leigh Ross, Luhuan Wu, John Patrick Cunningham, Jesse C Cresswell, and Anthony L Caterini. Denoising deep generative models. In Proceedings on Can't Believe It's Not Better! Understanding Deep Learning Through Empirical Falsification " at NeurIPS 2022 Workshops, 2022b.
Francesco Locatello, Damien Vincent, 1lya Tolstikhin, Gunnar Rätsch, Sylvain Gelly, and Bernhard Schölkopf. Competitive training of mixtures of independent deep generative models. arXiu:100 11130, 2018.
Aaron Lou, Minkai Xu, and Stefano Ermon. Scaling Riemannian diffusion models. In Advances in Neural Information Processing Systems, 2023.
60

---

Published in Transactions on Machine Learning Research (09/2024)
Yubin Lu, Zhongjian Wang, and Guillaume Bal. Mathematical analysis of singularities in the diffusion model under the submanifold assumption. (fi:23300008882 2023.
Lorenzo Luzi, Randall Balestriero, and Richard G Baraniuk. Ensembles of generative adversarial networks for disconnected data. orXiv:2006. 14600, 2020.
David JC MacKay and Zoubin Ghahramani. Comments on "Maximum likelihood estimation of intrinsic dimension' by E. Levina and P. Bickel (2004). The Inference Group Website, Cavendish Laboratory, Cambridge University, 2005.
Kanti V Mardia, John T Kent, and Arnab K Laha. Score matching estimators for directional distributions. arXiv:160, 000,0, 2016.
Emile Mathieu and Maximilian Nickel. Riemannian continuous normalizing flows. In Advances in Neural Information Processing Systems, 2020.
Leland Mclnnes, John Healy, Nathaniel Saul, and Lukas Großberger. UMAP: Uniform manifold approxima- tion and projection. Journal of Open Source Software, 3(29):861, 2018.
Chenlin Meng, Kristy Choi, Jiaming Song, and Stefano Ermon. Concrete score matching: Generalized score matching for discrete data. In Advances in Neural Information Processing Systems, 2022.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for gener- ative adversarial networks. In International Conference on Learning Representations, 2018.
Michael Moor, Max Horn, Bastian Rieck, and Karsten Borgwardt. Topological autoencoders. In International Conference on Machine Learning, 2020.
James R. Munkres. Topology. Pearson Education, 2014.
Kevin P Murphy. Machine Learning: A Probabilistic Perspective. MIT Press, 2012.
Eric Nalisnick, Lars Hertel, and Padhraic Smyth. Approximate inference for deep latent Gaussian mixtures. In NeurIPS Workshop on Bayesian Deep Learning, 2016.
Hariharan Narayanan and Sanjoy Mitter. Sample complexity of testing the manifold hypothesis. In Advances in Neural Information Processing Systems, 2010.
Hariharan Narayanan and Partha Niyogi. On the sample complexity of learning smooth cuts manifold. In Conference Learning Theory, 2009.
Philipp Nazari, Sebastian Damrich, and Fred A Hamprecht. Geometric autoencoders what you see is what you decode. In International Conference on Machine Learning, 2023.
Alexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Tlya Sutskever, and Mark Chen. GLIDE: Towards photorealistic image generation and editing with text- guided diffusion models. In International Conference on Machine Learning, 2022.
Sebastian Nowozin, Botond Cseke, and Ryota Tomioka. f-GAN: Training generative neural samplers using variational divergence minimization. In Advances in Neural Information Processing Systems, 2016.
Kazusato Oko, Shunta Akiyama, and Taiji Suzuki. Diffusion models are minimax optimal distribution cstimators. In International Conference on Machine Learning, 2023.
Bernt Oksendal. Stochastic Differential Equations, pp. 65-84- Springer Science Business Media, 2003.
Arkadas Ozakin and Alexander Gray. Submanifold density estimation. In Advonces in Neural Information Processing Systems, 2009.
Bo Pang, Tian Han, Erik Nijkamp, Song- Chun Zhu, and Ying Nian Wu. Learning latent space energy-based based prior model. In Advances in Neural Information Processing Systems, 2020.
61

---

Published in Transactions on Machine Learning Research (09/2024)
George Papamakarios, Theo Pavlakou, and lain Murray. Masked autoregressive flow for density estimation. In Advances in Neural Information Processing Systems, 2017.
George Papamakarios, Eric Nalisnick, Danilo Jimenez Rezende, Shakir Mohamed, and Balaji Lakshmi- narayanan. Normalizing flows for probabilistic modeling and inference. Journal of Machine Learning Research, 22(67):1-64, 2021.
Niki Parmar, Ashish Vaswani, Jakob Uszkorcit, Lukasz Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran. Image transformer. In International Conference on Machine Learning, 2018.
Adamn Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. PyTorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems, 2019.
Giorgio Patrini, Rianne van den Berg, Patrick Forre, Marcello Carioni, Samarth Bhargav, Max Welling, Tim Genewein, and Frank Nielsen. Sinkhorn autoencoders. In Uncertainly in Artificial Intelligence, 2020.
Karl Pearson. LIII. On lines and planes of closest fit. to systems of points space. The London, Edindburgh, and Dublin Philosophical Mogazine and Journal of Science, 2(11):559 572, 1901.
William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023.
Xavier Pennee. Intrinsic statistics Riemannian manifolds: Basic tools for geometric measurements. Jour- wal of Mathematicaa Imaging and Vision, 25(1):127 154, 2006.
Gabriel Peyré and Marco Cuturi. Computational optimal transport. Foundations and Trends in Machine Learning, 11(5-))3355-60 100-355---77--0 2019.
Jakiw Pidstrigach. Score-based generative models detect manifolds. In Advances in Neural Information Processing Systems, 2022.
Yury Polyanskiy and Yihong Wu. Information Theory: From Coding to Learning. Cambridge University Press, 2022.
Phillip Pope, Chen Zhu, Ahmed Abdelkader, Micah Goldblum, and Tom Goldstein. The intrinsic dimension of images and its impact on learning. In International Conference on Learning Representations, 2021.
Janis Postels, Martin Daneljjan, Luc Van Gool, and Federico Tombari. Maniflow: Implicitly representing manifolds with normalizing flows. In International Conference on 3D Vision, 2022.
Michael Puthawala, Matti Lassas, Ivan Dokmanic, and Maarten De Hoop. Universal joint approximation of manifolds and densities by simple injective flows. In International Conference on Machine Learning, 2022.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolu- tional generative adversarial networks. In International Conference on Learning Representations, 2015.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea. Voss, Alec Radford, Mark Chen, and Tlya Sutskever. Zero-shot text-to-image generation. In International Conference on Machine Learning, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with CLIP latents. vrXXi:2204.06122 2022.
Ali Razavi, Aäron van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with VQ-VAE-2. In Advances in Neural Information Processing Systems, 2019.
Danilo Jimenez Rezende and Shakir Mohamed, Variational inference with normalizing flows. In International Conference Machine Learning, 2015.
62

---

Published in Transactions on Machine Learning Research (09/2024)
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In International Conference Machine Learning, 2014.
Danilo Jimenez Rezende, George Papamakarios, Sébastien Racaniere, Michael Albergo, Gurtej Kanwar, Phiala Shanahan, and Kyle Cranmer. Normalizing flows tori and spheres. In International Conference Machine Learning, 2020.
Herbert Robbins. An empirical Bayes approach to statistics. In Third Berkeley Symposium on Mathematical Statisties and Probability, 1956.
Herbert Robbins and Sutton Monro. A stochastic approximation method. The Annals of Mathematical Slatistics, 22(3):400 1470000007700000 1951.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resohution image synthesis with latent diffusion models. In Proceedings of the IEEE/CV /CFF Conference on Compuler Vision and Pattern Recognation, 2022.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-Net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Intervention (MICCAT), 2015.
Brendan Leigh Ross and Jesse C Cresswell. Tractable density cstimation on learned manifolds with conformal embedding flows. In Advances in Neural Information Processing Systems, 2021.
Brendan Leigh Ross, Gabriel Loaiza-Ganem, Anthony L Caterini, and Jesse C Cresswell, Neural implicit manifold learning for topology-awar aware generative modelling. Transactions on Machine Learning Research, 2023.
Brendan Leigh Ross, Hamidreza. Kamkari, Zhaoyan Liu, Tongzi Wu, George Stein, Gabriel Loaiza-Gant Ganem, and Jesse C Cresswell. A geometric framework for understanding memorization in generative models. In ICML Workshop on Geometry- Grounded Representation Learning and Generative Modeling, 2024.
Sam T Roweis and Lawrence K Saul. Nonlinear dimensionality reduction by locally linear embedding. Science, 290(5500):2323 2226, 2000.
Noam Rozen, Aditya Grover, Maximilian Nickel, and Yaron Lipman. Moser flow: eivergence-based genera- live modeling on manifolds. In Advances in Neural Information Processing Systems, 2021.
Yangjun Ruan, Karen Ulirich, Daniel S Severo, James Townsend, Ashish Khisti, Arnaud Doucet, Alireza Makhzani, and Chris Maddison. Improving lossless compression rates via Monte Carlo bits- back coding, International Conference on Machine Learning, 2021.
Walter Rudin. Real and Complez Analysis. McGraw-Hill, Inc., 3rd edition, 1987.
David E Rumelhart, Geofrrey E Hinton, and Ronald Williams. Learning Internal Representations by Error Propogation, pp. 673-695. MIT Press, 1988.
Oleh Rybkin, Kostas Danillidis, and Sergey Levine. Simple and effective VAE training with calibrated docoders. In International Conference on Machine Learning, 2021.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Bureu Karagol Ayan, Tim Salimans, Jonathan Ho, David J Fleet, and Mohammad Norouzi. Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Information Processing Systems, 2022.
Tim Salimans, Andrej Karpathy, Xi Chen, and Diederík P Kingma. PixcICNN++: Improving the Pixel- CNN with discretized logistic mixture likelihood and other modifications. In International Conference Learning Representations, 2017.
Hadi Salman, Payman Yadollahpour, Tom Fletcher, and Kayhan Batmanghelich. Deep diffeomorphic nor- malizing flows. 18100 03256, 2018.
63

---

Published in Transactions on Machine Learning Research (09/2024)
Antoine Salmona, Valentin De Bortoli, Julie Delon, and Agnès Desolneux. Can push-forward generative models fit. multimodal distributions? In Advances in Neural Information Processing Systems, 2022.
Saeed Saremi and Aapo Hyvärinen. Neural empirical Bayes, Journal of Mochine Learning Research, 20 (181):1-23, 2019.
Axel Sauer, Tero Karras, Samuli Laine, Andreas Geiger, and Timo Aila. StyleGAN-t: Unlocking the power of GANs for fast large-scale text-to-image synthesis. In International Conference on Mochine Leorning. 2023.
René L Schilling and Franziska Kühhn Counterezamples in Measure and Integration. Cambridge University Press, 2021.
Stefan Schonsheck, Jie Chen, and Rongjie Lai. Chart auto-encoders for manifold structured data. arXiv:1912. 2019.
Nicolas Schreuder, Victor-Emmanuel Brunel, and Arnak Dalalyan. Statistical guarantees for generative models without domination. In Algorithmic Learning Theory, 2021.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade W Gordon, Ross Wightman, Mchdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa R Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev. LATON-ōb: An open large-scale dataset for training next generation image-text models. In Advances in Neural Information Processing Systems, 2022.
Beruhard Schölkopf, Alexander Smola, and Klaus-Robert Müller. Nonlinear component analysis as a kernel eigenvalue problem. Neural Computation, 10(5):1299-133) 1319,, 1998.
Hang Shao, Abhishek Kumar, and P Thomas Fletcher. The Riemannian geometry of deep generative models. In Proceedings of the IEEE/CVE Conference Computer Vision and Pattern Recognition Workshops, 2018.
Sahil Sidheekh, Chris Dock, Tushar Jain, Radu Balan, and Maneesh K Singh. VQ-Flows: Vector quantized local normalizing flows. In Uncertainty in Artificial Intelligence, 2022.
Carl-Johann Simon-Gabriel and Bernhard Schölkopf. Kernel distribution embeddings: Universal kernels, characteristic kernels and kernel metrics on distributions. Journal of Machine Learning Research, 19(44)) 1-29, 2018.
Carl-Johann Simon-Gabriel, Alessandro Barp, Bernhard Schölkopf, and Lester Mackey. Metrizing weak convergence with maximum mean discrepancies. Journal of Machine Learning Research, 24(184):1-20, 2023.
Jascha Sohl-Dickstein Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learning, 2015.
Casper Kaae Sonderby, Tapani Raiko, Lars Maaloe, Soren Kaae Senderby, and Ole Winther. Ladder varia- tional autocncoders. In Advances in Neural Information Processing Systems, 2016.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. In Aduances in Neural Information Processing Systems, 2019.
Yang Song, Conor Durkan, Tain Murray, and Stefano Ermon. Maximum likelihood training of score-based diffusion models. Advances in Neural Information Processing Systems, 2021a.
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poolc. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2021b.
64

---

Published in Transactions on Machine Learning Research (09/2024)
Peter Sorrenson, Felix Draxler, Armand Rousselot, Sander Hummerich, and Ullrich Köthe. Learning distri- butions on manifolds with free-form flows. arXiv:2312.0988 2023.
Peter Sorrenson, Daniel Behrend- Uriarte, Christoph Schnörr, and Ullrich Köthe. Learning distances from data with normalizing flows and score matching. arXiv:2407.09207, 2024a.
Peter Sorrenson, Felix Draxler, Armand Rousselot, Sander Hummerich, Lea Zimmerman, and Ullrich Köthe. Lifting architectural constraints of injective flows. In International Conference on Learning Representa- tions, 2024b.
Jan Stanczuk, Georgios Batzolis, Teo Deveney, and Carola-Bibiane Schönlieb. Diffusion models encode the intrinsic dimension of data manifolds. In International Conference on Machine Learning, 2024.
George Stein, Jesse C Cresswell, Rasa Hosseinzadeh, Yi Sui, Brendan Leigh Ross, Valentin Villecroze, Zhaoyan Liu, Anthony L Caterini, J Eric T Taylor, and Gabriel Loaiza-Ganem. Exposing flaws of gen- erative model evaluation metrics and their unfair treatment of diffusion models. In Advances in Neural Information Processing Systems, 2023.
Rong Tang and Yun Yang. Minimax rate of distribution estimation on unknown submanifolds under adver- sarial losses. The Annals of Statistics, 51(3):1282-1305 2023.
Rong Tang and Yun Yang. Adaptivity of diffusion models to manifold structures. In International Conference on Artificial Intelligence and Stalislics, 2024.
Ugo Tanielian, Thibaut Issenhuth, Elvis Dohmatob, and Jeremie Mary. Learning disconnected manifolds: A no GAN's land. In International Conference on Machine Learning, 2020.
Piotr Tempozyk, Rafal Michaluk, Lukasz Garncarek, Przemyslaw Spurek, Jacek Tabor, and Adam Golinski. LIDL: Local intrinsic dimension estimation using approximate likelihood. In International Conference on Machine Learning, 2022.
Joshua B Tenenbaum, Vin de Silva, and John Langford. A global geometric framework for nonlinear dimensionality reduction. Science, 290(5500))2319 2323, 2000.
Lucas Theis, Aäron van den Oord, and Matthias Bethge. A note on the evaluation of generative models. In International Conference on Learning Representations, 2016.
Michael E Tipping and Christopher M Bishop. Probabilistic principal component analysis. Journal of the Royal Statistical Society Series B: Statistical Methodology, 61(3):611 622, 1999.
Naftali Tishby, Fernando C Percira, and William Bialek. arXivphysics/000/4 2000.
The information bottleneck method.
Nya Tolstikhin, Olivier Bousquet, Sylvain Gelly, and Bernhard Schölkopf. Wasserstein aulo-encoders. In International Conference on Learning Representations, 2018.
Jakub Tomczak and Max Welling. VAE with VampPrior. In International Conference Artificial Intelligence and Statistics, 2018.
Alexander Tong, Kilian Fatras, Nikolay Malkin, Guillaume Huguet, Yanlei Zhang, Jarrid Rector-Brooks, Guy Wolf, and Yoshua Bengio. Improving and generalizing flow-based generative models with minibatch optimal transport. Transactions on Machine Learning Research, 2024.
James Townsend, Thomas Bird, and David Barber. Practical lossless compression with latent variables using bits back coding. In International Conference on Learning Hepresentations, 2019.
Ba-Hien Tran, Giulio Franzese, Pictro Michiardi, and Maurizio Filippone. One-lino-of-code data mollifi- cation improves optimization of fhelihood-basede generative models. In Advances in Neural Information Processing Systems, 2023.
65

---

Published in Transactions on Machine Learning Research (09/2024)
Iyy Trofimov, Daniil Cherniavski,, Eduard Tulchinskii, Nikita Balabin, Evgeny Burnaev, and Serguei Baran- nikov. Learning topology-preservve data representations. In International Conference on Learning Rep- resentations, 2023.
Benigno Uria, Tain Murray, and Hugo Larochelle. RNADE: The real-valued neural autoregressive density- estimator. In Advonces in Neural Information Processing Systems, 2013.
Arash Vahdat and Jan Kautz. NVAE: A deep hierarchical variational autoencoder. In Advances in Newral Information Processing Systems, 2020.
Arash Vahdat, Karsten Kreis, and Jan Kautz. Score-based generalive modeling in latent space. In Advances in Neural Information Processing Systems, 2021.
Rianne van den Berg, Leonard Hasenclever, Jakub M Tomezak, and Max Welling. Sylvester normalizing flows for variational inference. In Uncertainty in Artificial Intelligence, 2018.
Aäron van den Oord, Nal Kalchbrenner, Lasse Espeholt, Oriol Vinyals, Alex Graves, and Koray Kavukcuoglu. Conditional image generation with PixelCNN decoders. In Advances in Newral Information Processing Systems, 2016.
Aäron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural discrete representation learning. In Advances in Neural Information Processing Systems, 2017.
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-SNE. Journal of Machine Learning Research, 9(86):2579 2605, 2008.
Elen Vardanyan, Sona Hunanyan, Tigran Galstyan, Arshak Minasyan, and Arnak S Dalalyan. Statistically optimal generative modeling with maximum deviation from the empirical distribution. In International Conference Machine Learning, 2024.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, 2017.
Cédric Villani. Optimal Transport: Old and New. Springer Science & Business Media, 2009.
Pascal Vincent. A connection between score matching and denoising autoencoders. Neural computation, 23 (7):1661-1674, 2011.
Julius Von Rohrscheidt and Bastian Rieck. Topological singularity detection at multiple scales. In Interna- tional Conference on Machine Learning, 2023.
GK Wallace. The JPEG still picture compression standard. IEEE Transactions on Consumer Electronics, 38(1):18 34, 1992.
Matt P Wand and M Chris Jones. Kernel Smoothing. CRC Press, 1994.
Yi Wang and Zhiren Wang. CW complex hypothesis for image data. In International Conference on Machine Learning, 2024.
Yixin Wang, David Blei, and John P Cunningham. Posterior collapse and latent variable non-identifiabllity In Advances in Neural Information Processing Systems, 2021.
Max Welling and Yee W Teh. Bayesian learning via stochastic gradient langevin dynamics. In International Conference on Machine Learning, 2011.
Zhisheng Xiao, Qing Yan, and Yali Amit. Generative latent flow. arXiv: 10485, 2019.
Zhisheng Xiao, Karsten Kreis, and Arash Vahdat. Tackling the generative learning trilemma with denoising diffusion GANs. In International Conference on Learning Representations 2022.
66

---

Published in Transactions on Machine Learning Research (09/2024)
Jianwen Xie, Yang Lu, Song- Chun Zhu, and Yingnian Wu. A theory of generative ConvNet. In International Conference on Machine Learning, 2016.
Ling Yang, Zhilong Zhang, Yang Song, Shenda Hong, Runsheng Xu, Yue Zhao, Wentao Zhang, Bin Cui, and Ming-Hsuan Yang. Diffusion models: A comprehensive survey of methods and applications. ACM Computing Survey, 2023.
Sangwoong Yoon, Yung-Kyun Noh, and Frank Park. Autoencoding under normalization constraints. International Conference on Machine Learning, 2021.
Sangwoong Yoon, Young-Uk Jin, Yung-Kyun Noh, and Frank Park. Energy-based models for anomaly detection: A manifold diffusion recovery approach. In Advances in Neural Information Processing Systems, 2023.
Hengrui Zhang, Jiani Zhang, Balasubramaniam Srinivasan, Zhengyuan Shen, Xiao Qin, Christos Faloutsos, Huzefa Rangwala, and George Karypis. Mixed-type tabular data synthesis with score-based diffusion in latent space. In International Conference on Learning Representations, 2024.
Mingtian Zhang, Peter Hayes, Thomas Bird, Raza Habib, and David Barber. Spread divergence. In Inter- national Conference on Machine Learning, 2020a.
Mingtian Zhang, Yitong Sun, Chen Zhang, and Steven Medonagh. Spread flows for manifold modelling. International Conference on Artificial Intelligence and Statistics, 2023.
Zijun Zhang, Ruixiang Zhang, Zongpeng Li, Yoshua Bengio, and Liam Paull. Perceptual generalive autoen- coders. In International Conference on Machine Learning, 2020b.
Yijia Zheng, Tong He, Yixuan Qiu, and David P Wipf. Learning manifold dimensions with conditional variational autoencoders. In Advances in Neural Information Processing Systems, 2022.
67

---

Published in Transactions on Machine Learning Research (09/2024)
A Weak Convergence Primer
We now provide a brief summary of weak convergence of probability measures. We do not use a grey box around this section due to its length, despite the content being fairly technical. As in the main text, all the measures we consider here will be defined on X' along with its Borel o-algebra. Given a sequence of probability measures ( E we would like to define what mcans for the sequence to converge to some probability measure P.p. As we will see, there are several ways to define convergence of probability measures; we would like to use one which captures the intuition that P. "learns" P. in the sense that PPA converges to P.N as t- oc if and only if samples from P. become progressively harder to distinguish from those of PP as becomes larger, becoming indistinguishable in the limit. When Py represents a DGM, this is precisely the form of convergence we would hope to observe as we optimize its parameters 0z.
Strong convergence The soemingly most natural way to define convergence to say that P to P as t + oc if (B) - (B) as t + oo for every Borel set B. This type of convergence is called strong converyence. As the name suggests, this type of convergence is too strong, to the point where it does not properly capture the intended intuition of "PA, converges to P. if and only if P learns PX=
converges e.
Let us illustrate why this is the case with two examples. First, let fo be Gaussian with mean 0 and covariance matrix I/tID. Intuitively, this sequence learns a point mass at 0, do: yet it does not strongly converge to it: [0} is a Borel set, and ((0)) = 0 + as 7 co, yet 5o([0)) = 1 * O. As a second example, consider P = -a,; where (zz) is a fixed sequence converging to O with (2a [ [0) = 0. Intuitively this sequence also learns &o but similarly to the previous example, P ((0)) = 0 for every & and thus the sequence does not strongly converge to óg either.
We highlight that these are not overly-contrivn examples in the DGM setting. The first example illustrates a common scenario where a sequence of full-dimensional models P2, < Ap "learn" a distribution P. supported on a low-dimensional embedded submanifold M of X, without strongly converging to P. In this case, M is also a Borel set, and P.X (M) 0 + as + co even though (MM illustrates how a sequence of models whose supports do not overlap with that of their target distribution can "learn" it without strongly converging to it. Indeed, we need laxer definition of convergence of probability measures to properly convey the idea that a sequence of models P. "Icarns" P.
A 0. The second example
Weak convergence Weak - rather than strong - convergence provides a more appropriate notion of convergence to convey "learning". We say that P converges weakly to PN if Ex_oy [h(X)] + Ex-px [N()) as + oo for every bounded and continuous function h X R. As mentioned in [Section 2.1 we write P. + p as t + x to denote weak convergence. Intuitively, P converges weakly to P., if, as t + co, it becomes arbitrarily difficult to distinguish between samples from P. and samples from pa by using a bounded and continuous function; weak convergence matches the inluition of P.Y. "learning" p. much better than strong convergence.
There are many cquivalent definitions of weak convergence, with the standard one being the one presented above. The result establishing the equivalence of these definitions is called the PPotmaaneau Lemmal We present reduced version of this lemma below which we will use to prove the Likelihoo Instability Theorem in Apppndix B.1 - where only one of these equivalences is stated. Before stating the lemma, we define the continuity sets of a probability measure.
Definition 1 (Continuity Set). Let Pp be a probability measure on X' and B C X a Borel set. We say that B is a continuity set of 1P, if I2 (0x 0, where Ox denotes the topological boundary of B on X. Lemma (Portmantcau). Let P. be a probability measure on X, and let )m, be a sequence of probability measures on X. Then, P. 4 pxx as + co if and only if P)) + Ipx (B) as t + oo for every continuity set B of P..
Let us consider once again the example where P. is Gaussian with mcan O and covariance matrix 1/t ID. It is not difficult, to prove that PB)( + (B) as - co for every Borel set B such that 0 $ 0 B (L.e
68

---

Published in Transactions on Machine Learning Research (09/2024)
B) = 0), that as intended, P "> dy as t + co. Similarly, it is not difficult to prove the same in the example where F óz,- The fact that these sequences converge weakly but not strongly dn illustrates that weak convergence does indeed provide the right tool to talk about a sequence of models "lcarning" their target distribution.
Metrizing weak convergence Finally, we say that metric D A(X) x A(x) + R on the space of probability measures on X' metrizes weak convergence if D(P P2) - 0 as t + 0 holds if and only if P.. Pj as oo. Throughout the main manuscript, we often abuse language and use the term "metrizing weak convergence" even when D is only a divergence rather than a metric, as this is enough to ensure that minimizing D(P)) .P.) over 9 is sensible training objective for DGM P. even if P.A has low-dimensional support.
B Proofs
As in Appendix A we omit the use of a grey box despite the use of technical language.
B.1 The Likelihood Instability Theorem
We start by restating the Likelihood Instability Theorem for convenience before discussing it.
Theorem 1 (Likelihood Instability of Deep Generative Models). Let M C X be Borel set such that XM(c = 0, and let P2 be a probability measure on X such that P (M) 1 and supp(P,)) clx(M). Let ( be a sequence of probability measures on X such that P. 4 P as t - oo and P < AD, with corresponding densities p Then:
X (x) 0, ab-almostteuerguhhrr on X clx(M). inf po.
sup (a)
Po. (x') 8 as + oo for every x e clx(M) and every s 0, where
(=)=== fz' E X | x112 < E}.
As mentioned in Section 4.1.1 we begin with an example satisfying the assumptions of the Likelihood In- stability Theorem for which it does not hold that pē, (x) + oo for x cl)(M), thus highlighting that the thoorem cannot be "trivially strengthened". Consider X R2,, M {(11, E TR 0 < 01 < 1}, let PN be uniform on M, and IP be uniform on M,, where Me- = ((01, x2) E TR2 1 11 < and 1/((+1) < x2 < 1/1). Finally, take the corresponding densities as
pe. = 1(2 M.) 1(t + 1)1(x E M.), X2(M.)
(99)
where 1(-) denotes an indicator function. Figure 11 illustrates this example. Here, it holds that F. = px so that the assumptions of the Likelihood Instability Theorem are satisfied - yet pě, (x) - 0 as + 0 for every æ E X, and thus in particular for every æ E (M) as well. Nonetheless, when x E clx(M), (u)aren, P (a') + 0o as - 00 does hold for every ε 0, as concluded by the thoorem.
Before proving the Likelihood Instability Theorem we state and prove three lemmas, all of which we will rely on. We will heavily use (Continuity Sets and will leverage the Portmanteau Lemmal see Appendix A for reminder on these topics.
Lemma 2. Let P. be probability measure X, X, and E > 0. Then, there exists e' (0,E) such that (x) is a continuity set of TPX where B²(x) = (a' X || xl|2 < <c..
Proof. We proceed by contradiction: let e X' and ε > 0, and assume that PA (x) > for every s' E (0, Since can countably partition (0, as (1/n/, 1/(n 1)], it follows that uncountably
69

---

Published in Transactions on Machine Learning Research (09/2024)
22
X
P0
Pd.
2/9
Pd.
M
Be (I)
Figure 11: Visualization of the sequence of densities from Equation 99 which converge weakly to a uniform distribution on M. For x E clx(M) it always holds that pe, (x) = because the support of pò, does not overlap with (M,(M), so that pd, (z) + co - does not hold. Nonetheless, for any fixed & > 0, the support of pe. always overlaps with (x) for large enough t, and thus eu Po. (x')
many elements from (P- (x)) belong to an element of the partition. Thus, in particular there exists an integer n' and distinct numbers E2,,,,.... ew in (0, such that IP (Ox (ex (x)) > 1/n' for every 1, 2, ,....., n'. Then, because 0x BxBe;(1) n ax Be (a) = whenever + j. we have that
(oonee(00)
n
p.x
px (0xBe((x))
Σ
(100)
n
i=1
which is clearly contradiction since P. (X) 1, thus finishing the proof.
Lemma 3. Let M C X, and let Ip be probability measure X such that PN (M) 1. Then for every O, the set Mj = (a E X infzem || - =/122 < 6} open in X, and is a continuity set of Ip...
Proof. M3 is open because it can be written a union of open sets:
Ms = U Bs(x').
(101)
.'EM
Then, we have:
PK (sx = P (0x M) PP ((clx(M)) (M))( = Pp ((clx(M6) \ M2) ( M) PN (0) 0,
(102)
(103)
where (M) denotes the topological interior of M in X, and the first equality follows from PA = 1, the second one from the definition of boundary, the third one from Má being open, and the fourth one from M C Mg. Thus Mj is indeed a contimuity set of P.N.
Lemma 4. Let M c X, and let PX be a probability measure on X such that supp(P.X) = clx(M). Then P. (B. > 0 for every E clx(M) and every > 0, where (x) = fx' E X || - xl12 < <c.
Proof. Since P. is Borel measure and X' is separable, PP (supp(PP)) = 1 (Bogachev 2007 Proposi- tion 7.2.9) 30 It follows that clx(M) (Pp)(P is nonempty. Let clx(M) and 0. We proceed by contradiction, and assume that (z)()) = Since P. (clx(M)) = 1, we have that
Note that general, (P(suppPP)) need not hold, for example Schilling & Kühm 2021
Examples 6.2 and 6.3).
70

---

Published in Transactions on Machine Learning Research (09/2024)
Pp (M) \ (x)) = 1. Since (x) is open, clx(M) \ (x) is closed. Then, by definition of sup- port (Equation 1. and because supp(TPX) = clx(M), it follows that clx(M) n Ba(x) = 0. This is clearly a contradiction since x E clx(M) n (x)..
Proof of Theorem 1 We first prove that lim =no p (x) 0, Ap-almost-surely on X' c.xx Let x e X (MM( and let Ug be open neighbourhood of z such that clx(U2) n clx(M) = 0, which exists because X' is regular. Clearly Pp (0xUaa = 0 (i.e. Uz is a continuity set of PP.)) and Pp (U,) 0 since (U,) n (M) 0 and P. (clx =.
By the Portmantcau Lemma.
(U2) + Ppp (U2) as t + oo, and we have the following implications:
(U*) = lim [ (z')ddd
lim (U,) (U,)) 40000
P. (U,) 0
(104)
10 J.
= lim inf [ vt (((r)))) - 80000
(105)
Then, by Fatou's lemma,
[ inm p (z (x')dD((z) S.
(106)
Ju.
Since lim inf, po. (z) 2 0, Ap-almostteeverywhee on X, it follows that
lim inf pé, (o ddA((')) = 0, 10000
(107)
and thus lim inf (0, 0, Ap-almost-everyyweee
on Uz.
We still need to extend the result from Ap-almostt-eeeryweer Clearly (Uz is an open cover of X \ (M) Since X' is second countable and every subspace of a second countable space is second countable, it follows that X \ clx(M) is second countable. Then, by Lindelöf's lemma, (Uxxae))c))<aa)) has a countable subcover (U2, J of X clx(M). The result then holds on Un, for i 1.2...., and because any countable union of sets of measure has measure 0, it also holds Ap-almost-everyyweer on
Un to Ap-almostteverryhher on X' (M).
U = X clx( 1=1
(108)
which finishes this part of the proof.
Now, let (M) and E > 0, and we will prove that supre (w) Po, (x') + oo as + oc.
First, note that (u))" P (a') ) is increasing in e for every t. If (z is not a continuity set of P,,, by Lemma 2 we could always find e' E (0,=) such that (x) is continuity set of IP,, and if we managed to prove that (u))")B.,(*)) (²)()') so as oc, the same result would immediately follow for e. We can thus assume without loss of generality that e is such that (x) is a continuity set of IP..
Now, let M3 = (x E X inf ||ا - =1122 < 8) and let (2,((z) = Ms n (x().. From basic topology we have that (x n (=))) C Ms U Be)). Since Ms and Be(x) are continuity sets of P. by Lemma 3 and by assumption, respectively, it follows that (z)((z) is continuity set of P.,, since Ip, (QxUe,((z) S (8x:Ma) + IP (x)) 0. Similarly, (x)x) \ M3 is a continuity set of IpX because 0x (B, M&) C (x) U Ms. We then write:
Pd (Ba(z)) = IP, (Ues(z)) + Pd, (Be(x) \ Ms) = =1 [(_.(e) (r')ddd)))) + FA, (B&(x) \ Ms)
(109)
S Ao (x))
sup P (x') + P. (Be(x) \ Ms)
(110)
Z'Ela(())
SD (Uc)(x))
sup
po. (x') + X (Bx \ Ms).
(111)
xEB.(x))
71

---

Published in Transactions on Machine Learning Research (09/2024)
Since (x) is open, is M& by Lemma then (2) is open well. In turn AD(U,,0(x)) 0, and it follows that
P (Be PA (B+ Md)
_ sup
(x))
(112)
An (U),)(x))
x'ER((m)
By the Portmantcau Lemma and since (B. Ms) = 0, taking the limit as + oo of the left hand side of the above equation yields
PX (Be(z)) P. (Bx(2) Ms)
(Be())
lim 1-000
(113)
AD (U),
AD (U),)(()))
Thus, taking lim inf as + oo on both sides of Equation 112 implies that
P (Be(z)) lim inf sup 2(E,(2))
Po. (z').
(114)
AD (Ue)((z))
Since Ue, (x) C (2) whenever 8' < 8, we have that
lim An (Ue,((x)) An (Ussloo) An 8-00-
(NMM (Ms (Msn =An ((( Ma Bc(()) i
(115)
620
1000
= =A (M) n (x = 0,
(116)
where we used that (i) Ms = clx (M), which holds because clx(M) is the set of points which are
arbitrarily close to M, and that (ii) (clx = 0 by assumption. Finally, by Lomma 4 P. (B=(x)) > 0, so
that taking the limit - 0+ both sides of Equation 114 yields that lim inf, Bu Pd, (x') 0o,
which in turn implics that supaee (a) p (x') 00 as oc, finishing the proof.
B.2 Formalizing the Link between Two-Step Models and Optimal Transport
For convenience, we restate Proposition below.
Proposition 1. Let P.X be probability measure on X, 90, 2 - X be mensurable, and e: X x X - R be measurable and such that there erists C' > 0 such that
sup (x.))xXXX''
lc(x, < C.
(93)
Then,
inf Exasx go, (F(X))) = inf E Ex-px (e(X.9u go,, (f(X)))), SEC
(94)
PEF
where F = (f X - 2 f is measurable) and C = {f X + 2 is continuous).
Proof.
Since C C F, it follows that
inf Ex-ex (c go, (f(x))) < inf Ex-ex [c(X.go, (f(X))))
(117)
FEF
fec
Since both infimums are finite due the assumption from Equation 93, it is enough show that, for every 0, there exists Se E such that
inf Expex (x (f(X))) Ex-rx (c(X, (fx()))) -
(118)
feF
Let 0, and let f. e F be such that
inf Exaey le (X, 90, (f(X))) Exaex [c(X, (F,())) 6
(119)
feF
72

---

Published in Transactions on Machine Learning Research (09/2024)
It is thus enough to show that there exists fe such that
Exwex (c(X, go. (f(X)))] > Exwex (e(X, go, (fe()))) 2'
(120)
this would imply Equation 118 holds. Since P. is Borel measure and X' is Polish, PX is Radon measure. Additionally, P.A (X) < oo, X' is locally compact, Z is second countable, and fo is measurable;
it follows by Lusin's 's theorem that there exists a Borel set E C X' and continuous function fc X + 2 such that fe(x) (x) for every x E E, and P E) < E4((C)). Then, we have:
E EX (x, 90, (f(())))] Ex-ry (x, 9o, (f(())))
(121)
1 cla,go, (f))))) go (f+ (x))) dP. (x)
J c(a,go, (fe(z))) c(r,go go, (f(()))) dP (122)
xxYE
/
$ fxy,8 |c(x, 90, (f(()))) c (z. go. (f.()))) dp. (z) S JxLE
2C (z) = 2CP (x 5 2
(123)
which in turn implics Equation 120 holds, thus finishing the proof.
C
Experimental Details
Here we provide details on the experiments from Section 5.3.2 Our code is available at thub. http://ayerra--las/ddpaa dgw.maanfoldd_aume
First-step objective for latent diffusion models Following Rombach et al. 2022). we trained latent
diffusion models by first using regularized variational autoencoder (Section 4.1.2) loss Larsen et al. 2016
Higgins et al. 2017
min max E Exapx 0, of
B x (4x) (4x) [|x go, (2)1
3,KL (%
(124)
+ B2Exxpx
Bizwag 21x (1x) [log has (X) + log (1 ha (go, (Z))
where 31 > 0 and B2 > 0 are hyperparameters; p is a standard Gaussian; ds
2 = NN( 21 (x))
with x z (x) being diagonal for every E X'; and ha :' + (0,1) is binary classifier inspired by generative adversarial networks (Section 4.2/, whose objective is to distinguish between real samples X ~ Pxx and their stochastic reconstructions ge, (Z), where Z ~ 96 21X (-IX). Rather than fixing 82, we dynamically update it throughout training to ensure that the first and third terms in Equation 124 have roughly the same magnitude; when taking a gradient step, this is achieved by computing the ratio of the values of the first to third term in the previous gradient step, and setting 32 to the absolute value of this ratio.
Training objective for diffusion models We train all diffusion models, latent or not, exactly as de- scribed in [Section 5.1.2] Le. through [Eguttinnnnnnnn 55 The integral with respect to is approximated by sampling uniformly at random in [o, T] during training (one such & sampled for every element in the batch).
Hyperparameters We use the Adam optimizer (Kingma & Ba 2015) with a batch size of 128 throughout, and train all models until there is no improvement on the validation metric for 50 epochs; we keep the models with the best validation performance, For the VAE of latent diffusion models we use the squared roconstruction error Ex=px ||| - (f)( rathor Equation 124 as the validation metric, 31 10-6 a learning rate of 10-4 with cosine annealing, and take two gradient steps on o' for every gradient step on (0, For the diffusion models, both on ambient and latent space, we use Equation 55 as the validation metric, learning rate of 5 x 10-5 without cosine annealing, T = 1, Smin 0.1, Bmax 20, w(t) of, and Faler-Maruyama discretization scheme with 1000 steps to generate the paths in Figure 8)
73

---

Published in Transactions on Machine Learning Research (09/2024)
Table 1: Configuration used for neural networks. The architecture of the auxiliary network h& for the VAE mimics that of the encoder, and the decoder 90, is given by simply "reversing" the architecture of the encoder. See text for additional details.
PARAMETER AMBIENT SCORE $g
2 VAE ENCODER 9,
LATENT SCORE 662
n_ channels
64 (False, True, True)
64
256 (1,2) (True, True)
ch ch_multe mults
(1,
(1,
is_ is_attn n_ blookks
2
Architectures For a fair comparison between diffusion models on ambient and latent space, we attempt to instantiate them in such a way that their overall architectures are as similar as possible. The configurations of the architectures we used are given in [able I] which we now describe. We parameterize both the ambient and latent score networks as the output of a neural network divided by dz, as mentioned in [Section 5.1.2 (this is equivalent to the so-called "8 parameterization" of Ho al. (2020) with -e being parameterized instead of e). For the diffusion model ambient space, 2015) with residual connections (He et al.) 2016)) and an additional attention mechanism (Vaswani et al. 2017). as implemented in the labml package (Jayasiri & Wijerathne| 2020), which uses a sinusoidal positional embedding for the scalar input t. The ambient space U-Net takes 32 x 32 images, and progressively downsamples them to a shape of 1024 x x before upscaling them back to their original size. For the latent diffusion model, we attempt lo copy the aforementioned U-Net as much as possible; the VAE mimics the U-Net up until the 8 resolution, and adds convolutional layer to produce outputs of shape 4 8, that d = 256. To ensure the VAE obtains low-dimensional reprosentations through proper bottleneck, remove the U-Net skip connections between the encoder and decoder, resulting in purely residual architecture. The stochastic encoder 9s z|X of the VAE consists of single neural network which takes X and produces 2d-dimensional output the first d dimensions correspond to fø(a), and the remaining ones to the diagonal of 2 2 (z). The auxiliary network h& has the same architecture as the encoder fø, except a final linear layer is added to ensure the output is a scalar. The score network of the diffusion on latent space given another U-Net which further downsamples to a x 4 resolution before upsampling.
U-Net architecture (Ronneberger et al.
Data preprocessing Recall that raw image data is intoger-valued with possible values ranging from 0 to 255. We dequantize the data before training the diffusion model on ambient space and the VAE for the latent diffusion model, i.c. add independent uniform [0, 1] noise to every pixel (Theis et al. 2016). that the resulting data now has entries in [0, 256). We then linearly scale the data that every coordinate lies in [-1, 1]. For latent diffusion models, once the VAE is trained, we also scale its encodings to lie in [], before training the diffusion model on latent space.
74


Reviews:
[{"review_id": "0kahvb3LCu", "reviewer": "Reviewer_txjV", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper does not raise any major concerns. It is well-written, the presented results are technically sound, and the literature review is extensive. The paper has most of its value as a survey encompassing numerous generative models from the perspective of the manifold supported data (which is a ubiquitous setting in machine learning). Furthermore, part of its value comes from the novel results regarding the likelihood-based models (which remain one of the main interests of the community) and the two-step learning procedure (which is the state-of-the-art approach to generative modeling). The paper is of great interest to a major part of the community: novice researchers would find it valuable as a work encompassing main approaches while senior members of the community would benefit from a coherent picture of one of the main problems in the field.\n\nThe list of minor concerns:\n- Some parts of the paper read as a \"laundry list\". However, this might be unavoidable for a survey paper.\n- Throughout the paper, the authors refer to numerical instabilities but do not give any (formal or informal) definition. Especially, this worth clarifying in the context of VAEs and EBMs whose training indeed has issues but is not usually described as numerical instabilities.\n- Page 1. The wording \"high-dimensional likelihoods\" might be confusing for the reader.\n- Page 2. I suggest the authors avoid phrases that are excessively emotional \"one would have to wait forever\" and technically imprecise \"any machine learning researcher...\".\n- Page 15. The unfixability of maximum likelihood requires a more detailed explanation. Namely, the claim that no such regularizer exists that can fix the issues of the maximum likelihood is either too vague or too strong. Indeed, couldn't I just take a regularizer that changes the maximization objective, i.e. cancels out the likelihood and substracts the Wasserstein distance?\n- Page 22. Denoising Score Matching could be considered as a part of Score-Based Diffusion models since it is usually not used independently.\n- Page 32. \"WAEs only encourage this to happen on average\" might be confusing when speaking about measures.\n- Page 42. The noised-out data density lacks parameter $\\sigma$ in its definition.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This manuscript provides a survey of the modern deep generative models from the perspective of the manifold hypothesis. Namely, the paper assumes that the target data distribution lies on a low-dimensional manifold embedded in a high-dimensional ambient space. Furthermore, this low-dimensional manifold is unknown analytically (although, the paper quickly touches the generative models where the data-manifold is known analytically, e.g. data on a sphere) but given implicitly, i.e. can be recovered from the empirical distribution of the data.\n\nUnder the manifold perspective, the paper divides all the existing generative models into two categories: the ones that operate under the assumption that the dimensionality of the data manifold matches the dimensionality of the ambient space (manifold-unaware) and the ones that assume that the dimensionality of the manifold is much smaller than of the ambient space (manifold-aware). The paper goes rigorously over the major existing approaches (to the best of my knowledge over all practical approaches), classifies them according to the proposed dichotomy, and pinpoints potential issues.\n\nFor the manifold-unaware models, the authors consider the conventional approaches to generative modeling (Variational Autoencoders, Normalizing Flows, Energy-based models, Generative Adversarial Networks (GANs), Score Matching) and demonstrate the potential issues related to the manifold hypothesis. Moreover, the authors present one of their results - the Likelihood Instability theorem, which builds on top of the result from [1]. Intuitively, this theorem shows that, for the models with full-dimensional support, the maximization of the likelihood is numerically unstable and is incapable of recovering the ground true measure. The discussion of other models (Score Matching and GANs) is more intuitive and empirical.\n\nFor the manifold-aware models, the authors outline three major approaches to overcoming issues related to the manifold hypothesis: adding noise to the data or model outputs, using objectives that are properly defined for the manifold-supported data, learning the models by the two-step procedure (learning the manifold at the first step and then learning a generative model under the assumption of the full-dimensional support). Besides careful consideration and analysis of different design choices, the authors propose a novel result (based on the results from [2]) stating that the two-step learning procedure with the learning of an autoencoder at the first stage and a generative model in the latent space minimizes an upper bound on the Wasserstein distance.\n\nFinally, the authors conclude with a short discussion of how discrete-space generative models relate to the manifold hypothesis and an outline of major challenges left in the field.\n\n[1] Gabriel Loaiza-Ganem, Brendan Leigh Ross, Jesse C Cresswell, and Anthony L Caterini. Diagnosing and fixing manifold overfitting in deep generative models. Transactions on Machine Learning Research, 2022a.  \n[2] Ilya Tolstikhin, Olivier Bousquet, Sylvain Gelly, and Bernhard Schölkopf. Wasserstein auto-encoders. In International Conference on Learning Representations, 2018.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I would suggest to rework the parts of the paper that raised my concerns. However, it is not strictly necessary for the acceptance of the manuscript.", "broader_impact_concerns": "This paper does not raise any broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "yS2hOc1qvX", "reviewer": "Reviewer_Ydqc", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n\nThe paper is well-organised and the various DGMs considered are presented with full clarity, with appropriate references. The arguments made for and against each model to learn distributions on a lower dimensional manifold are mostly sound, presented at both an intuitive level and also at a more rigorous level in grey boxes. Indeed, the capability of generative models to learn distributions under the manifold hypothesis helps to explain the success of certain models over others and is thus important to understand the interplay between the two.\nOverall, I can see the work being a useful reference for an overview of DGMs and how \"aware\" they are about the manifold hypothesis. The insights that the paper provide could be useful for designing new generative model architecture in the future.\n\n**Weaknesses:**\n\nWhile most of the arguments presented in the paper were satisfactory, there were a few parts that were unclear to me and would be improved with some additional explanations.\n- The reasoning provided for why score matching is not manifold-aware is not fully satisfactory compared to explanations provided for other models. When the authors state \"we should intuitively expect score matching to fail ... due to $p_*^X$ not being a full-dimensional density\", could you please explain why that should be the case?\n- The explanation for the exploding gradients in diffusion models under the manifold hypothesis was also confusing to me (pages 25-26). As the samples $Y_t$ approach the data manifold, shouldn't we expect the gradient to decrease with the time step? I am just not sure if the argument made here makes sense unless we are taking adaptive time steps that decrease as $t \\rightarrow T$.\n- I am not sure if I understand the logic behind the statement \"if CFM successfully learns its target distribution in the manifold setting, then it must experience the numerical instabilities of likelihood estimation\" (page 28).\n- I don't immediately see how the loss (68) is derived from (67).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This is mostly a review paper on generative models, with the focus on understanding the consequence of the manifold hypothesis on various deep generative models (DGMs). It provides an overview of methods that are compatible with the manifold hypothesis (manifold aware) and those that are not (manifold unaware). Along the way, the authors provide some new perspectives on these models, including a rigorous understanding on why DGMs based on maximum likelihood fail under the manifold hypothesis (Theorem 1) and an understanding of two-step models (i.e., models that treat the manifold learning and distribution learning steps separately) in terms of minimising an upper bound on the Wasserstein distance between the true and approximate distributions.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Can you please define $\\mathtt{stopgrad}$ in page 19 (also in page 44)?\n- In Equation (84), the $\\theta$ that is minimised should be $\\theta_1$. It would also be helpful to explicitly state somewhere that $q_\\phi^Z$ is the pushforward measure $q_\\phi^Z = f_\\sharp p_*^X$.", "broader_impact_concerns": "This is mostly a review article and as far as I can see, there are no ethical implications of the work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "TlokYnAcgR", "reviewer": "Reviewer_uivD", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strength\n1. The paper is well-organized, covers various deep generative models, and provides insights into how generative models relate to manifold hypotheses.\n2. Not only that the authors summarize existing works, but they also present their own contributions: one regarding the instability of likelihood-based training of deep generative models and another regarding two-step models. I don’t think that a review paper necessarily has such contributions, but the results provide more insights into understanding the existing deep generative models’ performance.\n3. The authors considered readers from diverse backgrounds by separating advanced topics into boxes. As the advanced topics are separated, this paper becomes more accessible to people without a strong mathematical background, and those people who are familiar with mathematical details can always delve into the advanced topics in separate boxes.\n\n### Weaknesses\n1. This is a well-written paper, and I cannot find a specific weakness to criticize.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper surveys deep generative models with insights from the manifold hypothesis. Specifically, the authors describe how the existing generative model works and explain why, based on the mathematical intuitions from the manifold hypothesis, those generative models would be desirable/undesirable. The authors also present some future research directions about deep generative models and their potential interplay with the manifold hypothesis.\n\nThe paper starts by introducing mathematical notations, formal setup, and preliminary materials. The preliminary materials describe the common learning objectives used for training deep generative models: KL divergence, Wasserstein distances, and maximum mean discrepancy (MMD). Particularly, it is pointed out that minimizing KL divergence is likely to fail to maximize the likelihood correctly if we consider the manifold hypothesis.\n\nThe next two sections (Section 4 and Section 5) cover deep generative models for continuous distributions. The authors divide such models into two categories based on their manifold awareness. Section 4 focuses on the manifold-unaware variants of deep generative models. The authors introduce their own results about why the likelihood-based approaches might fail in those deep generative models. Then, the authors go through the examples of existing generative models, e.g., variational autoencoders, normalizing flows, energy-based models, generative adversarial networks, and score matching, explaining how the authors’ results may affect those models. On the contrary, Section 5 concentrates on the manifold aware deep generative models. Specifically, the authors classify various methods by how each method achieves the manifold-awareness: adding noise, support-agnostic optimization objectives, and two-step aproaches that model the latent space. In each subsection, the authors introduce the deep generative models that use each approach to enable the manifold-awareness and explain them through the lens of manifold hypothesis. In particular, Section 5.3 introduces another result that helps us understand what the two-step models do; they minimize the upper bound of the Wasserstein distance under a specific assumption.\n\nSection 6  briefly describes deep generative models for discrete distribution cases. In these cases, there is no dimensionality mismatch (between the dimension of the underlying data manifold and the dimension of the manifold that the model generates), so the models will not be affected by the problems mentioned in the previous sections.\n\nFinally, the authors summarize their contributions and introduce future research directions with a focus on the manifold hypothesis.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I don’t see a specific change that is required for this paper.", "broader_impact_concerns": "I don’t see a particular broader impact concern regarding this paper.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper is well-written, and all reviewers agree that it is valuable for researchers interested in understanding how the manifold hypothesis is linked to existing deep generative models.
Decision: Accept (poster)