Denoising Normalizing Flow
Year: 2021

Abstract:
Normalizing flows (NF) are expressive as well as tractable density estimation methods whenever the support of the density is diffeomorphic to the entire dataspace. However, real-world data sets typically live on (or very close to) lowdimensional manifolds thereby challenging the applicability of standard NF on realworld problems. Here we propose a novel method called Denoising Normalizing Flow (DNF) that estimates the density on the low-dimensional manifold while learning the manifold as well. The DNF works in 3 steps. First, it inflates the manifold making it diffeomorphic to the entire data-space. Secondly, it learns an NF on the inflated manifold and finally it learns a denoising mapping similarly to denoising autoencoders. The DNF relies on a single cost function and does not require to alternate between a density estimation phase and a manifold learning phase as it is the case with other recent methods. Furthermore, we show that the DNF can learn meaningful low-dimensional representations from naturalistic images as well as generate high-quality samples.

heading: 1 Introduction
text: Given samples from the data-density p(x), key objectives in probabilistic Machine Learning are 1. estimating p(x) (density estimation), 2. generating new data points from p(x) (sampling), and 3. finding low-dimensional representations of the data (inference). The three main methods used to perform these tasks are Normalizing Flows (NFs) [34], Generative Adversarial Networks (GANs) [17], and Variational Autoencoders (VAEs) [26]. Among those methods, only the VAE does check all desired objectives by default. However, it does so by sacrificing the sample quality (compared to GANs), and only learning a lower bound on p(x) (rather than the exact value as NFs do). Finding new ways to meet these key objectives, based on either the known main methods or new ones, is an active and important research area.
How can we infer low-dimensional representations using NFs? The standard NF requires the datadensity p(x) to have a support diffeomorphic to the entire data-space RD. However, the manifold hypothesis conjectures that the data-manifoldM lies close to a d-dimensional manifold embedded in RD, d < D. Thus, unfortunately, a standard NF cannot be used to infer the latent space. Recently, Brehmer et al. proposed to overcome this limitation by first projecting the data into a d-dimensional space using the first d−components of a standard NF, and then using another NF to learn the latent distribution π(u) [10]. For their method to work they propose different learning schemes, separating the manifold learning from the density estimation.
In this paper, we propose an easy and new way to learn low-dimensional representations with NFs. Our idea is based on the theoretical work derived in [20] where it was shown that by inflating the data-manifold with Gaussian noise ε ∼ N (0, σ2ID), i.e. x̃ := x + ε, the data-density p(x) can be well approximated by learning the inflated distribution qσ(x̃). More concretely, the main result in [20] states sufficient conditions on the choice of noise qσ(x̃|x) and type of manifold such that
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
qσ(x) = p(x)qσ(x|x) holds. Here, by adding a penalty term to the usual KL-divergence used to learn qσ(x̃), we ensure that the first d-components of the corresponding flow are noise insensitive and thus encode the manifold. This penalty term is essentially the objective function of a Denoising autoencoder (DAE), and thus we call our method Denoising Normalizing Flow (DNF).
In summary, our contributions are the following:
1. We propose a new method, the Denoising Normalizing Flow, which combines two previously well-known methods (DAE and NF), and is able to • approximate p(x), • sample new data x ∼ p(x) as a non-linear transformation of u ∼ N (u; 0, Id), • infer low-dimensional latent variables u ∼ p(u|x) given x ∼ p(x).
2. We demonstrate on naturalistic data that our method learns meaningful latent representations without sacrificing the sample quality.
Notations: We adapt the notation used in [10]. To further simplify it and avoid clutter, we denote the Gram matrix of g evaluated at g−1(x) as
Gg(x) := Jg(g −1(x))TJg(g −1(x)) (1)
where Jg(g−1(x))T is the transpose of the Jacobian of g : Rd → RD, d ≤ D, evaluated at g−1(x).

heading: 2 Problem statement
text: In the following, we are going to show why standard NFs are not suited to infer low-dimensional representations of the given data. We end the section with the research question we are going to study in Section 3.
General setting: In generative modeling, it is assumed that the data x ∈M ⊂ RD is generated by a non-linear transformation g of some latent variables u ∈ U ⊂ Rd, i.e. x = g(u) where u ∼ π(u). Typically, d < D and the latent distribution π(u) is assumed to be Gaussian, π(u) = N (u; 0, Id). Hence, the latent random variable u generates x, and the data-density p(x) evaluated at x ∈ RD is given by
p(x) = ∫ U π(u)δ(x− g(u))du, (2)
where δ denotes the Dirac function, see [3]. If d = D and g is a diffeomorphism, we have for x ∈M that
p(x) = |detGg(x)|− 1 2π(g−1(x)). (3)
Then, the target density p(x) can be learned, in principle, exactly using an NF [21]. In general, an NF is an embedding mapping U ⊂ Rd toM⊂ RD, see [32] or [27] for some recent reviews. Denoting this mapping as gθ : U →M and its parameters as θ, the induced density onM is given by
pθ(x) = |detGgθ (x)|− 1 2 pu(g −1 θ (x)), (4)
where pu(u) is a known reference density (usually set to be standard Gaussian). The parameters θ are updated such that the KL-divergence between p(x) and pθ(x),
DKL(p(x)||pθ(x)) = −Ex∼p(x)[log pθ(x)] + const. (5)
is minimized. Thus, to learn θ efficiently, one needs to evaluate T1(x) := log pu(g−1θ (x)) and T2(x) := 1 2 log |detGgθ (x)| efficiently.
Topological constraints: The evaluation of T1(x) is efficient since we are free to choose the reference measure pu, and g−1θ (x) is the forward pass of a neural network constructed to be bijective [13, 21]. Therefore, the majority of the NF literature focuses on designing clever flow architectures to be able to calculate T2(x) efficiently without sacrificing the flow’s expressiveness (i.e. the size of the space of embeddings able to learn). However, so far these architectures are constructed for d = D since in this case, the Jacobian of gθ is a square matrix, and thus T2(x) becomes
1 2 log |detGgθ (x)| = log |det Jgθ (g −1 θ (x))|. (6)
Hence, calculating the Gram determinant of gθ efficiently amounts to calculating the determinant of the Jacobian of gθ efficiently. Popular choices are to construct gθ such that Jgθ is lower triangular, as in this case, the determinant is simply the product of diagonal elements. Unfortunately, for d < D, Jgθ is not a square matrix, and the full Gram determinant needs to be calculated. This makes NFs unsuitable for finding low-dimensional representations u of high-dimensional data points for large d as the computational complexity to calculate detGgθ is O(d2D) +O(d3). Research question: As mentioned in [10], finding ways to design gθ such that T2(x) can be efficiently calculated for d < D is an interesting research question. Here, we address it from a different angle.
LetM be a d−dimensional manifold embedded in RD through g. Can we construct a mapping gθ : Rd →M to
1. generate x ∼ p(x) in 2 steps: (a) generate u ∼ N (u; 0, Id) and (b) set x = gθ(u)? 2. infer u ∈ Rd such that x = gθ(u)? 3. approximate detGg(x) efficiently?

heading: 3 Denoising Normalizing Flow
text: We answer the research question based on the theoretical work developed in [20]. First, we briefly review this work, and then introduce the DNF.
Preliminaries: In Section 2, we discussed why classical NFs are not suited to infer low-dimensional representations of the given data. Also, if one is only interested in the value of p(x), standard NF cannot be used. Intuitively, a standard NF fψ with parameters ψ simply squeezes or expands a volume element where the net change is given by its Jacobian determinant. A volume element of a d−dimensional manifold is d−dimensional and thus has D−dimensional Lebesgue measure 0. Thus, we are asking fψ to expand a d−dimensional volume to a D−dimensional one which will lead to a degeneration of |detGfψ (x)| and manifest in numerical instabilities. Therefore, [20] inflated the manifold by adding Gaussian noise to the data points.1 This inflated manifold is D−dimensional and thus a usual flow can be used to learn the corresponding density. Their main result states sufficient conditions on the choice of noise and type of manifoldM, such that the learned inflated distribution can be deflated, and p(x) is exactly retrieved. More precisely, given a random variable x ∼ p(x) with probability measure PX and taking values in a d−dimensional manifoldM embedded in RD, if we add some noise ε to it, the resulting new random variable x̃ = x+ ε has the following density:
qσ(x̃) = ∫ M qσ(x̃|x)dPX(x). (7)
In [20], it was shown that if (a) the noise is only added in the (D− d)−dimensional normal space Nx in x, (b) the noise magnitude σ is sufficiently small, and (c) the manifoldM is sufficiently smooth and disentangled, the resulting inflated distribution evaluated at x̃ = x takes the following product form: qσ(x) = p(x)qσ(x|x), (8) where qσ(x|x) is the normalization constant of the noise density. More concretely, (a) and (b) need to ensure that x is almost surely uniquely determined by x̃ as the orthogonal projection of x̃ onM. For this projection to be well-defined, a sufficient condition is that the manifolds reach number2 is finite [6]. A manifold where almost every point x̃ ∼ q(x̃) in the inflated set has a unique projection on M was called Q−normally reachable in [20], where Q denotes the collection of noise distributions qσ(x̃|x). Their main Theorem proves that for anyQ−normally reachable manifold equation (8) holds. Therefore, if qσ(x̃) can be learned exactly using a standard NF, the on-manifold density p(x) can be retrieved exactly. It was also shown that for the case where d D (as it is generally assumed for high-resolution images), full Gaussian noise is an excellent approximation for a Gaussian in the normal space.
1Adding noise to circumvent the aforementioned degeneracy problem was also proposed in [24]. 2Informally, this reach condition ensures that a manifold is learnable through samples.
Main idea: We use a standard flow fψ to learn qσ(x̃) such that the corresponding density has the product form of equation (8), and the first d−components u of the flow’s output f−1ψ (x̃) are noiseinsensitive whereas the remaining (D − d)−components v remain noise-sensitive. Thus, intuitively, we want the first d−components to denoise the inflated data. DNF: Let fψ : RD → RD be a standard flow with reference measure pz(z). We denote the first d−components of the flows output as u, and the remaining ones as v, i.e. (u, v)T = f−1ψ (x̃). More formally,
u = u(x̃) = Proju(f −1 ψ (x̃)), v = v(x̃) = Projv(f −1 ψ (x̃)) (9)
with Proju(z) = (z1, . . . , zd) and Projv(z) = (zd+1, . . . , zD) for z ∈ RD. As reference measure pz , we choose pz((u, v)T ) = pu(u)pv(v) with pv(v) modelling the noise-sensitive part, and pu(u) the noise-insensitive part. In particular , if qσ(x̃|x) is a (D− d)−dimensional Gaussian distribution with covariance σ2ID−d, we set pv(v) = N (v; 0, σ2ID−d). For pu(u) to model the noise-insensitive part, we want the image fψ(u, 0) to be in the manifold. Therefore, we embed u back in RD by padding the missing coordinates with 0,
Pad(u) = (u, 0, . . . , 0︸ ︷︷ ︸ (D−d)-times )T , (10)
such that the operation Pad(Proju((u, v))) = (u, 0) T ignores the noise-sensitive part v in the latent space. This operator allows us to define a denoising function rψ(x̃) as
rψ(x̃) := (fψ ◦ Pad ◦ Proju ◦ f −1 ψ )(x̃) (11)
and we regularize ψ by minimizing
C(ψ) := Ex∼p(x)Ex̃∼qσ(x̃|x)||x− rψ(x̃)|| 2 (12)
where || · || denotes the L2 norm. We have not specified the reference measure pu(u). To facilitate the disentanglement of noiseinsensitivity and noise-sensitivity in the u and v variables, we transform u with yet another flow hφ with paramters φ and reference measure pu′ (e.g. standard Normal). Now, our sampling procedure looks as follows: 1. sample u′ ∼ pu′(u′), 2. apply hφ to obtain a sample u from pu(u), i.e. u = hφ(u′), and 3. set x = fψ(Pad(u)). Denoting θ = (ψ, φ), our model to learn qσ(x̃) is
qθ(x̃) =|detGfψ (x̃)|− 1 2 pu(u(x̃))pv(v(x̃))
=|detGfψ (x̃)|− 1 2 |detGhφ(u(x̃))|− 1 2 pu′(h −1 φ (u(x̃)))pv(v(x̃)). (13)
Note that the reconstruction loss, equation (12), is essentially the objective function for the Denoising Autoencoder introduced in [1]. Therefore, we call our method Denoising Normalizing Flow (DNF) and it is trained on
LDNF(θ) :=DKL(qσ(x̃)||qθ(x̃)) + λC(ψ) (14)
where λ > 0 is the penalty hyperparameter and is trading the density estimation with the manifold learning. A graphical description of the DNF model is given in Figure 1 (a), and an algorithmic description in the DNF Algorithm below.3
Answer to research question: If LDNF(θ) = 0, the reconstruction error expressed in equation (12) is 0. Thus, the generative story of the DNF is exactly the one described in point 1. of our research question with gθ := fψ ◦ Pad ◦ hφ. The inverse, h−1φ ◦ Proju ◦ f −1 ψ , can be used to infer u
′ s.t. point 2. holds. Finally, to show the third claim, we additionally assume that U (which is the domain of the manifold generating function g) is diffeomorphic to Rd, s.t. without loss of generality we set π(u) = N (u; 0, Id). Then, we exploit equation (8) and calculate |detGg(x)| efficiently with the help of fψ and hφ, see Proposition 1 and its proof in the supplementary.
3We show the algorithm for the general scenario where the manifold is unknown and thus noise cannot be added to the normal space. We also ignore terms independent of θ in the calculation of LDNF, and denote this loss function as L′DNF.
DNF Algorithm: Training of Denoising Normalizing Flow for qσ(x̃|x) = N (x̃;x, σ2ID). For simplicity, we show a stochastic gradient descent with a constant learning rate. Alternative optimization methods and learning rate schedules can be easily adapted. Require: Manifold dimension d, Learning rate α, penalty parameter λ, inflation variance σ2, batch size n, number of epochs E. Initialize: Parameters ψ and φ for flows fψ and hφ.
while θ = (ψ, φ) has not converged do for e = 1 to E do
for i = 1 to n do Sample: xi ∼ p(x) # sample data Inflate: x̃i = xi + εi, where εi ∼ N (0, σ2ID) # add noise (ui, vi)← f−1ψ (x̃i) # project on u ∈ Rd and v ∈ RD−d x̂i ← fψ(ui, 0) # reconstruct x u′i ← h−1φ (ui) # transform u end for L′DNF ← 1n ∑n i log pu′(u ′ i)− log |det Jhφ(u′i)|+ log(pv(vi))− log |det Jfψ (ui, vi)| +λ||xi − x̂i||2 # calculate logqθ(x̃) and add reconstruction error θ ← θ − α∇θL′DNF # update model parameters
end for end while
Proposition 1 LetM be a d−dimensional manifold embedded in RD through g : Rd →M. Let x ∼ p(x) be generated by g(u), where u ∼ N (u; 0, Id), i.e. x = g(u). Assume that we can learn the inflated distribution qσ(x̃) using an NF and that for x̃ = x it holds that qσ(x) = p(x)qσ(x|x). If LDNF(θ) = 0, then
|detGg(x)| = |detGfψ (x)||detGhφ(x)|. (15)

heading: 4 Related work
text: Autoencoders: An autoencoder learns to compress the data x using an encoder f−1ψ , and then to reconstruct x using a decoder gθ. As the AE is only trained on the reconstruction error ||x − gφ(f −1 ψ (x))||2, it can’t be used to generate new data or estimate p(x). However, if the input is corrupted by Gaussian noise ε, an optimal AE that can reconstruct the uncorrupted input depends on the gradient of the data-loglikelihood [1]. This can be exploited to estimate p(x) [7]. A variational autoencoder (VAE) [26] is a stochastic version of an AE. More concretely, a lower bound on the data log-likelihood, known as ELBO or free energy, is maximized to learn parameters φ and θ such that the true conditional distributions p(x|z) and p(z|x) are approximated by pθ(x|z) = N (x; gθ(z), ID) and qψ(z|x) = N (x; fψ(x), ID), respectively. If one is not interested in p(x) but still wants to generate new data using an AE, regularization terms can be exploited [28, 37]. Alternatively, an NF can be used to normalize the learned latent variables u = fψ(x). Thus, after training a usual AE, the latent density associated with the latent variable u is learned using a standard NF. This probabilistic autoencoder (PAE) was recently introduced in [9].
NFs based models: Recently, a few attempts have been made to overcome the topological constraint of NFs in terms of density estimation [10, 11, 20, 31, 34, 35], sampling [4, 10, 12, 24], and inference [5, 10]. If the manifold is known, i.e. its atlas is given, a usual flow in Rd is sufficient to learn the data-density p(x) as we can use these charts to push-back p(x) to Rd. This was first done in [16] for a manifold consisting of a single chart g. However, if g is not known, only recently some methods were proposed to learn it [4, 10]. We review these methods in the following, highlight their differences to the DNF and illustrate them in Figure 1.
Pseudo invertible encoder (PIE): The idea of [4] is to define the manifold as a level set of a usual NF f . For that, they propose to treat the first d−latent variables u differently than the remaining D − d variables v, by using different reference measures for u and v, respectively, i.e. f−1(x) = z = (u, v)T ∈ Rd × RD−d and pz(z) = pu(u)pv(v). The gist is very similar to the DNF. However, there is no incentive for f to encode the manifold in u which manifests in poor sample quality (see [10]). In addition, this approach does not work for data living exactly on a
low-dimensional manifold as the Jacobian determinant of f degenerates. In Figure 1, we depict a version of the PIE model where u is additionally transformed with a flow h. Note that a PIE model is an DNF with λ = σ = 0.
Manifold flow (M−flow): The idea of [10] is to learn the embedding g directly by first mapping the data into the latent space using a usual NF f , projecting it to the first d−coordinates u, and finally transform these variables using yet another flow h in Rd with reference measure pu′ , see Figure 1 for a depiction of theirM−flow. The resulting density,
pM(x) = pu′(h −1(u(x)))|detGh(u(x))|− 1 2 |detGg(u(x))|− 1 2 , (16)
is indeed a density defined only on the manifold. As noted in [10], calculating the Gram determinant of g may be very slow for large d. Therefore, they propose to train the parameters of g using the reconstruction error ||x− g−1(g(x))||22 only, and the parameters of h using the usual DKL objective for NFs. They propose several training strategies to ensure that g encodes the manifold (manifold phase), and h learns the density (density phase). One strategy alternates between the manifold- and density phase (alternate training) for every training epoch. Another strategy first trains g on the reconstruction error and then learns the density through h (sequential training). Note that a PAE is also trained using a sequential training scheme, as opposed to the DNF which combines the manifoldand density learning phase into a single cost function, equation (14). Another difference to the DNF is that the Gram determinant of f is neither used to train theM−flow nor to estimate p(x).

heading: 5 Results
text: In this chapter, we confirm experimentally our claims made at the end of Section 1, formalized as a research question, and answered at the end of Section 3. First, we show that the DNF circumvents the degeneracy problem of a standard NF for manifold-supported densities. Afterward, we show that the DNF can learn meaningful latent representations for naturalistic images, without sacrificing the sample quality compared to theM−flow, PAE, and VAE (see Section 4). Our method depends on two hyperparameters, the noise magnitude σ2 and the penalty coefficient λ. σ2 needs to be sufficiently small such that equation (8) approximately holds (see [20] for more details on the choice of σ). If λ = 0, only the density is learned, and if λ 1 the manifold learning dominates.
We closely follow the experimental setting of [10]. For theM−flow and DNF, we use the same network architectures and training protocols. These architectures are based on affine-coupling layers [13], neural splines [14], and trainable permutations [25]. The PAE uses convolutional neural networks for the encoder and decoder, respectively. For images, we use a recently developed variant of the VAE, the InfoMax-VAE [33], which uses insights from information theory to learn meaningful latent representations. We refer to the supplementary for more training details. 4
5.1 Density estimation
We consider a 1−dimensional manifold embedded in R2, a thin spiral. We draw the latent variable u from an exponential distribution with scale 0.3, i.e. π(u) ∝ exp(−0.3u), and generate the spiral through g(u) = α √ u
3 (cos(α √ u), sin(α √ u))T where α = 4π/3 (upper half
of Figure 2 top left).
We train a standard NF (upper right),M−flow (middle left), DNF (middle right), PAE (lower left), VAE (lower right) with similar architectures on 100 epochs with a batch size of 100. As we can see in Figure 2, the Jacobian determinant of the standard NF degenerates, and the learned density collapses into single points. Surprisingly, theM−flow fails to learn p(x) as well, no matter which training schedule we use (we display the result of the sequential training). The PAE learns an inflated version of p(x), however, it does not have a deflation procedure. The standard VAE simply equates p(x) with the Gaussian prior on the latent variables (we take a point estimate of the ELBO to approximate p(x)).
For the DNF, we use Gaussian noise with σ2 = 0.01 and λ = 1. Note that only the DNF is trained on the inflated density qσ(x̃), and it encodes the noisy part in v(x̃), i.e. for points close to the manifold v(x̃) should be close to 0. Therefore, we set p(x) to 0 whenever v(x) > σ/3 and otherwise we approximate p(x) according to equation (8) with qθ(x)/qσ(x|x) where qσ(x|x) = 1/ √ 2πσ2. The fact that theM−flow fails to learn p(x) and the DNF succeeds, suggests the regulatory importance of the Gram determinant of fψ (which is only used from the DNF, and reflects the main difference between those two methods next to the training scheme). When setting σ2 = 0, the DNF degenerates similar to the standard NF (more details in the supplementary). This illustrates the importance of the inflation step for a density supported on a low-dimensional manifold.
All densities are evaluated on a 100 × 100 grid. For that, theM−flow calculates the Gram determinant of the learned generator g (see equation (16) and Figure 1 (c)) for each point. Like this, evaluating the density on a batch consisting of 500 points takes about 188 seconds on a GPU5. On the same device and for the same batch size, the DNF needs only 1 second to evaluate the density. This illustrates the drawback of needing to calculate the full Gram determinant (see Section 2).
4Our main code is available at https://github.com/chrvt/denoising-normalizing-flow and is based on the originalM−flow implementation made public by the authors of [10] under the MIT license. For the InfoMax-VAE and PAE on images, we use the official implementations made public by the authors of [33] and [9] under the Apache License 2.0 and General Public License v3.0, respectively.
5The Gram matrix can be calculated using automatic differentiation.
To further validate the learned on-manifold density, for each model we compare the induced latent density with the ground truth exponential distribution π(u) (second half of Figure 2). For that, we first generate 1000 latent points u ∼ π(u), then evaluate the model likelihood at x = g(u), and finally multiply the likelihood with the square root of the Gram determinant of the generating mapping g, √ detGg(u) ∝ (1 + (αu)2)/(αu)2. This must lead to the correct latent density π(u) if the data-density p(x) = |detGg(x)|− 1 2π(g−1(x)) is learned correctly as p(x) is uniquely determined by the pair (g, π). The standard NF, M−flow and VAE are out of scale. The PAE follows the exponential course of π(u), however, only the DNF learns the correct course and scale. This indicates that equation (15) can be used to approximate detGg(u) well.
In the supplementary, we conduct more density estimation experiments and compare the DNF with the inflation-deflation method [20]. For a von Mises distribution on a circle, and a mixture of von Mises distributions on a sphere, the DNF learns the density almost exactly showing the correctness of equation (15). Also in the supplementary, we use the DNF for probabilistic inference following the protocol used in [10].

heading: 5.2 StyleGAN image manifold
text: One drawback of both theM−flow and DNF is that the true manifold dimension d must be known beforehand. For real-world datasets, d is unknown. Therefore, [10] uses a StyleGAN2 model [23] trained on the FFHQ dataset [22] to generate an d−dimensional manifold by only varying the first d latent variables while keeping the remaining fixed.
d = 2: We want to show that the DNF learns meaningful latent representations. For that, we first train an DNF on 104 images using 100 epochs with σ2 = 0.1 and λ = 1000. Then, we generate an image grid by varying the 2−dimensional latent variables u′ and mapping them to the data space using the learned mapping gθ. In Figure 3, we see on the top left the original grid obtained by the StyleGAN2 model, and on the top right the grid generated by the DNF. A smooth mapping from latent to data space is learned. The PAE (lower left), does not learn such a smooth mapping which may be due to the lack of learning p(x), in contrast to the InfoMax-VAE (lower right). However, the latter lacks variability which shows that the learned posterior p(z|x) does not match the prior N (z; 0, Id). To further evaluate the quality of generated test samples, we display the Fréchet inception distance (FID score) 6 [19, 30] in Table 1 along with the mean reconstruction error which measures the manifold learning. We slightly outperform theM−flow in terms of FID score, and significantly in terms of the mean reconstruction error. The latter is surprising as theM−flow is directly trained on the reconstruction error. The PAE and InfoMax-VAE perform worse compared to theM−flow and DNF indicating a suboptimal choice for the latent dimension for these models.
6The FID is the Wasserstein-2 distance between two Gaussians. The mean values and covariance matrices are obtained as sample estimates of the original and model data, respectively. However, instead of using the pixel values, one uses the outcomes of the next to the last layer of the Inception v3 ([36]) image classifier trained on the corresponding data set.
d = 64: We train the models on 2 · 104 images for 200 epochs. In Figure 4, we show in the first 5 columns samples from the original dataset (top),M−flow (second row), DNF (third row), InfoMaxVAE (fourth row), and PAE (last row). In the remaining 5 columns, we show how smoothly these models interpolate linearly in the latent space. For that, we linearly interpolate between two training images in latent space, and display the corresponding trajectory in image space. We compare the FID and mean reconstruction error of the models in Table 1. Similar to the d = 2 case, the DNF outperforms theM−flow, InfoMax-VAE, and PAE in terms of FID and mean reconstruction error.

heading: 6 Summary and discussion
text: Our model is based on the theoretical work established by [20] leading to a natural combination of NFs and DAE - the DNF. In contrast to similar methods, an DNF is trained on a single objective function combining manifold- and density learning. To learn a density supported on a low-dimensional manifold with an NF, one needs to compute the flow’s Gram determinant. The DNF circumvents this necessity and can be used to approximate it. We have pigeonholed the DNF into the literature, and compared its performance on naturalistic images with related methods (M− flow, PAE, VAE). Among those methods, we have seen that the DNF generates images with the highest quality (in terms of FID), and reconstructs a given input with the lowest L2 distance.
It is well known that adding noise in the input can increase the generalisation performance in supervised learning tasks [2, 8, 29]. However, typically this comes with the price of lower sample quality or worse density estimation. The DNF has the potential to avoid this trade-off, and proves
experimentally that adding noise to images can lead to better performance in terms of sampling quality and manifold learning.
Extensions: The theoretical foundation of the DNF is equation (8) which requires the manifold to be sufficiently smooth, and the noise to be added in the manifold’s normal space. Although for the special case where d D standard Gaussian noise is an excellent approximation for a Gaussian in the normal space, more research is needed on how to sample in a manifold’s normal space in order to improve the performance of the DNF. Essentially, the DNF learns to encode the manifold in the u component and the normal space direction in the v component. Arguably, a pre-trained DNF could be used to improve the sampling ability in the normal space.
Another limitation of the DNF is that the manifold dimension d needs to be known. Nevertheless, if p(x) is supported on a low-dimensional manifold, to the best of our knowledge the DNF together with theM−flow are the only existing methods to learn p(x) and the manifold generating mapping based on NF. Hence, in contrast to other methods which need to know the exact manifold, the DNF andM−flow can still be used when d is estimated from samples. The latter is an active research area [5, 15, 18].
Other recently developed methods (such as PAE orM−flow) separate the manifold from the density learning. We have seen that this separation is not necessary, and have used different heuristics to evaluate the quality of learning. For the value of p(x), we visualized the learned (latent) density (Figure 2). For the quality of latent representations, we generated an image grid (Figure 3), and for the smoothness in the latent space, we generated an image path (Figure 4). We measured the sampling quality using the FID, and the manifold-learning using the mean reconstruction error (Table 1). Given these different heuristics, there is an aspiration for a unified performance criterion.
Broader Impact: To compress increasingly high-dimensional data with the least loss of information as possible is becoming increasingly important. The DNF ties in with theM−flow and shows that such compression is possible, even for naturalistic images. Similar to all other generative models, a possible negative impact of DNFs would be the generation of fake data. On a more positive note, the DNF could improve out-of-distribution detection or even increase the robustness to adversarial attacks which are both essential for reliable societal applications of Machine Learning.
Acknowledgments and Disclosure of Funding
We thank Camille Gontier, Gagan Narula, Hui-An Shen, Katharina A. Wilmers, Nicolas Zucchet, and the anonymous reviewers for helpful comments. This study has been supported by the Swiss National Science Foundation grant 31003A_175644.

Review summaries:
This paper proposes Denoising Normalizing Flows (DNFs), a method for accounting for low-dimensional manifold structure of data when using Normalizing Flows (NFs). The method builds on the recent work Horvat and Pfister ([18] in the manuscript), who showed that under some conditions, adding a small amount of Gaussian noise ("inflation") to data and then using a fully dimensional NF can recover the correct distribution on the manifold after a "deflation" step. The authors combine injective flows, mapping from low-to high dimensional space, with the inflation/deflation process, in an attempt to recover a low dimensional representation as well (as opposed to just using the inflation/deflation method); and properly accounting for the change-of-volume term during training (as opposed to the M-flow method of Brehmer and Cranmer, [9] in the manuscript).

Authors propose a novel normalizing-flow-based method for simultaneous manifold and probability-density-on-the-manifold learning. Similarly to previous methods addressing this problem, the dimensionality of the manifold learned needs to be set by the user before training and is not learned as part of the process, albeit there exist other methods for estimating it. The advantage of the proposed method is the single-step training procedure, in contrast to the multi-step training procedure of the other competitive method in this domain. I consider it a good contribution towards building a normalizing flow based representation learning toolkit.

This paper is motivated by the desire to circumvent a troublesome assumption in Normalizing Flows (NFs) on the function $f$ that produces the pushforward measure --- that $f$ must be strictly invertible. This is difficult for generative models, since the dimension of the latent space $d$ is usually much smaller than that of the target data space $D$, i.e. d << D.  To address this, the authors train a transformation $f:\mathbb{R}^D \rightarrow \mathbb{R}^D$ as usually done in a NF approach, but separate the latent variable $z = f^{-1}(x)$ into two orthogonal components $u, v$, consisting of the first $d$ and remaining $D-d$ coordinates of $z$. Intuitively, $u$ should contain the "manifold" information of $x$, and $v$ should capture any "off-manifold" noise. This is encouraged by adding a reconstruction loss to the $u$ component of the latent space. To allow for greater complexity of the latent variable $u$, a second NF transformation $h$ is applied to u, i.e. $u = h(\tilde{u})$. In addition, the authors add noise robustness by adding Gaussian noise to $x$, i.e. $\tilde(x) = x + \epsilon$, and reconstruct $x$ --- much like with Denoising Autoencoders, hence the name.