Denoising Normalizing Flow
Authors: Christian Horvat
Jean-Pascal Pfister
Venue: NeurIPS 2021 Poster (2021)
Keywords: Normalizing Flow
Manifold Learning
Density Estimation

Abstract: Normalizing flows (NF) are expressive as well as tractable density estimation methods whenever the support of the density is diffeomorphic to the entire data-space. However, real-world data sets typically live on (or very close to) low-dimensional manifolds thereby challenging the applicability of standard NF on real-world problems. Here we propose a novel method - called Denoising Normalizing Flow (DNF) - that estimates the density on the low-dimensional manifold while learning the manifold as well. The DNF works in 3 steps. First, it inflates the manifold - making it diffeomorphic to the entire data-space. Secondly, it learns an NF on the inflated manifold and finally it learns a denoising mapping - similarly to denoising autoencoders. The DNF relies on a single cost function and does not require to alternate between a density estimation phase and a manifold learning phase - as it is the case with other recent methods. Furthermore, we show that the DNF can learn meaningful low-dimensional representations from naturalistic images as well as generate high-quality samples.

Denoising Normalizing Flow
Christian Horvat Department of Physiology University of Bern Bern, Switzerland christian.hov htw.iiann.oraatuuiiio ch
Jean-Pascal Pfister Department of Physiology University of Bern Bern, Switzerland Seanpascl..pfisteeeeeeeeeeeeeeee wfwsterrmibeeebctth
Abstract
Normalizing flows (NF) are expressive as well as tractable density estimation methods whenever the support of the density is diffeomorphic the entire data- space. However, real-world data sets typically live on (or very close to) low- dimensional manifolds thereby challenging the applicability of standard NF on real- world problems. Here we propose a novel method called Denoising Normalizing Flow (DNF) that estimates the density the low-dimensional manifold while learning the manifold as well. The DNF works in 3 steps. First, it inflates the manifold making diffeomorphic the entire data-space. Secondly, learns an NF on the inflated manifold and finally it learns a denoising mapping similarly denoising autoencoders. The DNF relies on a single cost function and does not require to alternate between density estimation phase and manifold learning phase as it is the case with other recent methods. Furthermore, we show that the DNF can learn meaningful low-dimensional representations from naturalistic images as well as generate high-quality samples.
Introduction
Given samples from the data-density p(x),, key objectives in probabilistic Machine Learning are 1. estimating p(x) (density estimation). 2. generating new data points from p(r) (sampling). and 3. finding low-dimensional representations of the data (inference). The three main methods used to perform these tasks are Normalizing Flows (NFs) (34), Generative Adversarial Networks (GANs) [17,, and Variational Autoencoders (VAEs) 26 Among those methods, only the VAE does check all desired objectives by default. However, it does so by sacrificing the sample quality (compared to GANs), and only learning a lower bound on p(z) (rather than the exact value NFs do). Finding new ways to meet these key objectives, based on either the known main methods or new ones, is an active and important research area.
How can we infer low-dimensional representations using NFs? The standard NF requires the data- density p(x) to have support diffeomorphic to the entire data-space RD, However, the manifold hypothesis conjectures that the data-manifold M lies close to d-dimensional manifold embedded in RD.d D. Thus, unfortunately, standard NF cannot be used infer the latent space. Recently, Brehmer et al. proposed to overcome this limitation by first projecting the data into d-dimensional space using the first d components of a standard NF. and then using another NF to learn the latent distribution IT (u) [10]. For their method to work they propose different learning schemes, separating the manifold learning from the density estimation.
In this paper, we propose an easy and new way to learn www-dimensional representations with NFs. Our idea is based on the theoretical work derived in (0) where it was shown that by inflating the data-manifold with Gaussian noise ~ (00, o² ID). L.e. := x + e, the data-density p(z) can be well approximated by learning the inflated distribution go (7). More concretely, the main result in 20 states sufficient conditions on the choice of noise qa ( (r) and type of manifold such that
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

qo (z) p( holds. Here, by adding penalty term to the usual KL-divergence used to learn 9,- (3), we ensure that the first d-components of the corresponding flow are noise insensitive and thus encode the manifold. This penalty term is essentially the objective function of Denoising autoencoder (DAE), and thus we call our method Denoising Normalizing Flow (DNF).
In summary, our contributions are the following:
1. We propose new method, the Denoising Normalizing Flow, which combines two previously well-known methods (DAE and NF), and is able to
approximate p(z).
sample new data z ~ p(z) as a non-linear transformation ~ Nuu 0, Ia),
infer www-dimensional latent variables u ~ p(u(z) given p(xx)
2. We demonstrate on naturalistic data that our method learns meaningful latent representations without sacrificing the sample quality.
Notations:
We adapt the notation used in [10]. To further simplify it and avoid clutter, we denote the Gram matrix of g evaluated at g "(x)
Gz(z)= (z) وو (z))
(1)
where J is the transpose of the Jacobian of g: R RRD. d D, evaluated (x:).
2 Problem statement
In the following, we are going to show why standard NFs are not suited to infer low-dimensional representations of the given data. We end the section with the research question we are going study in Section
General setting: In generative modeling, it is assumed that the data M C R is generated by
non-linear transformation g of some latent variables u E U C Rd, i.. x = g(u) where u ~ su)) Typically, d < D and the latent distribution (u) is assumed be Gaussian, (u)) = Nuu; 0, Ia).
Hence, the latent random variable u generates v, and the data-densig p(z) evaluated at x E R is given by
p(x) = t(u)o(= - g(u))du,
(2)
where ō denotes the Dirac function, see 13] If d 1) and g is diffeomorphism, we have for M that
p(z) = det (z)
(3)
Then, the target density p(x) can be learned, in principle, exactly using an NF (21] In general, an NF is an embedding mapping U C Rd to M C DR see [32] or (27] for some recent reviews. Denoting
this mapping as ge L > M and its parameters as 0, the induced density on M is given by
(x = det G (x)1 2 pa (x)),
(4)
where Pu (w) is known reference density (usually set to be standard Gaussian). The parameters are updated such that the KL-divergence between p(x) and po (x),
(xL(p())llpoz)) Ex)= pe (z)] + const.
(5)
minimized. Thus, to learn efficiently, one needs to evaluate (xx)) := log Pu (x)) and T2(x) = log det Gy. (z) efficiently.
Topological constraints: The evaluation of Ti (x:) is efficient since we are free to choose the reference measure Pa. and 9, -1 (x) is the forward pass of a neural network constructed to be bijective [3] 21 Therefore, the majority of the NF literature focuses on designing clever flow architectures to be able to calculate T2 (ar) efficiently without sacrificing the flow's expressiveness (i.e. the size of the space of embeddings able learn). However, so far these architectures are constructed for d since in this case, the Jacobian of ge is square matrix, and thus T2(x) becomes
log det Go (x) = log det Jgp (g) (z))
(6)

---

Hence, calculating the Gram determinant of ge efficiently amounts to calculating the determinant of the Jacobian of go efficiently. Popular choices are construct go such that Jgg is lower triangular, as in this case, the determinant is simply the product of diagonal elements. Unfortunately, for d < D, Jgu is not square matrix, and the full Gram determinant needs to calculated. This makes NFs unsuitable for finding low-dimensional representations u of high-dimenssonal data points for large d as the computational complexity to calculate det Gg is (dd D) + (dd))
Research question: As mentioned in 10]. finding ways to design 9o such that (z(2) can be efficiently calculated for d < D is an interesting research question. Here, we address from different angle.
Let M be d dimensional manifold embedded in RD through g. Can construct mapping go Rd + M to
1. generate x p(x) in 2 steps: (a) generate u NNu;; 0, Ia) and (b) set x = go()
2. infer u E Rd such that x = =
3. approximate det Gz(z) efficiently?
3 Denoising Normalizing Flow
We answer the research question based on the theoretical work developed in (20 First, we briefly review this work, and then introduce the DNF.
Preliminaries: In Section 2 we discussed why classical NFs are not suited to infer low-dimensions representations of the given data. Also, if one only interested in the value of p(z:), standard NF cannot be used. Intuitively, a standard NF fy with parameters 16 simply squeezes or expands a volume element where the net change given by its Jacobian determinant. A volume element of a d- di-diensionnal manifold is d- dimmensiional and thus has D D dimensional Lebesgue measure 0. Thus, we are asking f to expand a d dimensionnl volume to D -D-dimensional one which will lead to a degeneration of det GJ (z)l and manifest in numerical instabilities. Therefore, 120] inflated the manifold by adding Gaussian noise to the data points ! This inflated manifold is D- dimensional and thus a usual flow can be used to learn the corresponding density. Their main result states sufficient conditions on the choice of noise and type of manifold M, such that the learned inflated distribution can be deflated, and p(z)) is exactly retrieved.
More precisely, given random variable x ~ p(x) with probability measure x and taking values in d dimiennsion manifold M embedded in IR_, if we add some noise to it, the resulting new random variable ã x + has the following density:
yo (.) SM 20 (=(o)))
In 200 it was shown that if (a) the noise only added in the (D - dimennsial normal space N in x, (b) the noise magnitude a is sufficiently small, and (c) the manifold M is sufficiently smooth and disentangled, the resulting inflated distribution evaluated at ă x takes the following product form:
(z) = p(z)
(8)
where go (x/xx) x)) is the normalization constant of the noise density. More concretely, (a) and (b) need to ensure that x is almost surely uniquely determined by ă as the orthogonal projection of ă on m.M. For this projection to be well-defined, sufficient condition is that the manifolds reach numbe il finite [6. A manifold where almost every point ~ (()) in the inflated set has a unique projection on
M was called Qnnomaalyy reachable in [20] where Q denotes the collection of noise distributions go [Z]z). Their main Theorem proves that for any Q Q-normally reachable manifold equation (8) holds. Therefore, if (ō) can be learned exactly using standard NF, the on-manifold density p(x) can be retrieved exactly.
was also shown that for the case where < D (as it is generally assumed for high-resolution images), full Gaussian noise is an excellent approximation for a Gaussian in the normal space.
I AAding noise to circumvent the aforementioned degeneracy problem was also proposed in Informally, this reach condition ensures that manifold learnable through samples.
3

---

Main idea: We use a standard flow fg to learn qo (?) such that the corresponding density has the product form of equation (8), and the first d componett of the flow's s output (a) (ã) are noise- insensitive whereas the remaining (D d) componnettt v remain noise-sensitive Thus, intuitively, we want the first d component to denoise the inflated data.
DNF: Let J RRD D be standard flow with reference measure P=(2). We denote the first d- comoononssts of the flows output u, and the remaining ones v, i.e. (u, v) f (T). More formally,
u = u()) = Proj., (g5 (8)), == v(ā) = Proj, (z)
with Proj.. (21...., zd) and (2) (zd-1:.... 2D) for E R.D. As reference measure P=. we choose Pz ((u, w)² = Pu (u)pp..()) with (v) modelling the soise-sensitive part, and (u(u) the noise-insensitive part. In particular, if go (T)ar) is (D d) dimensional Gaussian distribution with covariance o² ID-d, we set (v) [u; 0, 0
For (u( to model the noise-insensitive part, we want the image (w 0) to be in the manifold. Therefore, we embed u back in Rd by padding the missing coordinates with 0,
(a)(u)= = (u, 0 0.....99% 0 )²:
(10)
d---immmss
such that the operation Pad(Proj,,(uu) o))) (u, 0) ignores the soise-sensitive part in the latent
space. This operator allows us define a denoising function r+) as
(#) == (fv o Pad Proj.. o (2)
(11)
and we regularize & by minimizing
C(v) == Benp(e) Eang, (2)2)l vi (7)
(12)
where| || denotes the L2 norm.
We have not specified the reference measure (u)).. To facilitate the disentanglement of noise- insensitivity and noise-sensitivity in the and variables, we transform u with yet another flow ha with paramters & and reference measure Pur (e.g, standard Normal).
Now, our sampling procedure looks as follows: 1. sample u' (u)). . 2. apply he obtain sample u from (u), i.e. u = ho()',, and 3. set x = fo (Pad(u)). Denoting 0 = (tb, o), our model to learn do (f)
go ( =| det G (7) (7) p (M)))))
=I det G,, (a) $ det Gn, -x (t," (u(F)))o(((z))
(13)
Note that the reconstruction loss, equation [2]. is essentially the objective function for the Denoising Autoencoder introduced in [1] Therefore, we call our method Denoising Normalizing Flow (DNF) and it is trained on
(0) : = (B)ll) + AC(1)
(14)
where À > 0 is the penalty hyperparameter and is trading the density estimation with the manifold learning. A graphical description of the DNF model is given in Figure [ (a), and an algorithmic description in the DNF Algorithm below
LDNF (0) O, the reconstruction error expressed in equation (2) is O. Thus, the generative story of the DNF is exactly the one described in point 1. of our research question with go == S Pad o hgg The inverse, 1-1 Proj.. s can be used to infer u' s.t. point 2. holds. Finally, to show the third claim, we additionally assume that 2 (which is the domain of the manifold generating function g) is diffeomorphic R.d s.t. without loss of generality we set (u) = (u 0, Ia). Then, we exploit equation (8) and calculate det (2)) efficiently with the help of f and hss see Proposition and its proof the supplementary.
Answer to research question:
We show the algorithm for the general scenario where the manifold is unknown and thus noise cannot be added to the normal space. We also ignore terms independent of 0 in the calculation of CoNE. and denote this loss function LINT.

---

DNF Algorithm: Training of Denoising Normalizing Flow for (=) N For sim- plicity, we show a stochastic gradient descent with a constant learning rate. Alternative optimization methods and learning rate schedules can be easily adapted.
Require:
Manifold dimension d, Learning rate penalty parameter X, inflation variance o², batch size n, number of epochs
Initialize: Parameters & and Φ for flows fo and hgg
while 0- (v);)) has not converged do for = to E do
for 1 to n do
Sample: 24 ~ p(x)
# sample data # add noise
Inflate: 3; 24 + Et, where E, ~ of((0,21D))
(ui, 02) r/ (i:)
# project on u E Rd and DRd # reconstruct transform
8 + fu(u,
u', x h/ (u;)
end for
DNF n Ex
Pre Pie () - log det Jn, (u':)] + (os( - log det Jr, (uz, uz)l
+1|12; 22
# calculate loggo (ū) and add reconstruction error update model parameters
0 - 0 a V CDDNE
end for end while
Proposition 1 Let M be a d- didimensional manifold embedded in RD through g Rd M. Let I ~ p(x) be generated by g(u), where u ~ NNu;; 0, La). i.e. x g(u). Assume that we can learn the inflated distribution qa (3) using an NF and that for 3t = x holds that go (x(x) = (x))))())))) If (f) = 0, then
|det G₂ |dct GG (2) det Gna (z)l.
(15)
Related work
Autoencoders: An autoencoder learns to compress the data x using encoder fū' and then to reconstruct using decoder ge- As the AE is only trained the reconstruction error || go(f% (x)))12 (x))?, it can't be used to generate new data estimate p(x). However, if the input is corrupted by Gaussian noise an optimal AE that can reconstruct the uncorrupted input depends on the gradient of the lollkeeoiide [.. This can be exploited to estimate p(x) 77] variational autoencoder (VAE) (26 is stochastic version of an AE. More concretely, lower bound on the data log-likelihood known as ELBO or free energy, is maximized to learn parameters & and such that the true conditional distributions p(z|2) and p(2(z)
approximated by pa (al 2) - NN(x; -Nxx:go(2) (z)) ID) and qu (2|zz) = (z; (I), ID). respectively. If one is not interested in p(x) but still wants to generate new data using AE, regularization terms can be exploited [28 37 Alternatively, an NF can be used to normalize the learned latent variables u = f (z). Thus, after training a usual AE, the latent density associated with the latent variable is learned using standard NE. This probabilistic autoencoder (PAE) was recently introduced [9]
NFss based models: Recently, few attempts have been made to overcome the topological constraint of NFs in terms of density estimation [10 [1 20 31 R4 35), sampling 4 IO 12 24], and inference [0] If the manifold is known, i.e. its atlas is given, usual flow in Rd is sufficient to learn the data-density p(x) as we can use these charts to push-back p(x) to ıdd. This was first done in [16] for a manifold consisting of single chart g. However, if g is not known, only recently some methods were proposed to learn it [ [O] We review these methods in the following, highlight their differences to the DNF and illustrate them in Figure
Pseudo invertible encoder (PIE): The idea of is to define the manifold as a level set of a usual NF f. For that, they propose to treat the first d-latent variables u differently than the remaining D variables v, by using different reference measures for u and v, respectively, i.e. (x) (u, w) E Rdd x RD-d and (=) = Pa (w)p.(r). The gist is very similar to the DNF. However, there is no incentive for f to encode the manifold in u which manifests in poor
sample quality (see [10). In addition, this approach does not work for data living exactly on
5

---

Loss:
(a)
5 1.1
f
(u,0)
Proj Pv
DNF:
Inflate (x(3)))
Pod
of m pe
~ (()))
+ Pe ola (A)]I
① o Pod.
(b)
f
PIE:
2 2~ p(x)
(u, b)
h
Daa(p(a)) (1cp
a
ps
(c)
M-flow:
-~ p(x)
f
(u,
Proj
h
of Per
Camiitas Ex-ptsillo (x))1
Pad
Landay Eenpte plg const.
P Pad
Figure 1:
Schematic overview of different NF-based methods (DNF, PIE, M--llow) (adopted from [10). Left: Graphical model. Red solid lines NFs, red dashed lines the corresponding generator. Black dashed lines describe the projection or padding operation, respectively. as described in the main text. Right: The losses used train the models.
low-dimensional manifold as the Jacobian determinant of f degenerates. In Figure | we depict version of the PIE model where u is additionally transformed with a flow h. Note that PIE model is an DNF with À σ
Manifold flow ffow)ww): The idea of 10 is to learn the embedding g directly by first mapping the data into the latent space using usual NF projecting it to the first c-coordinatess u, and finally transform these variables using yet another flow h in IR with reference measure Put, see Figure for depiction of their M M-flow. The resulting density,
(M(x) = Pur (h (h-hh-h(uu (u(x))) det Gn(u(z))
det det (u(u))))
(16)
is indeed density defined only on the manifold. As noted in [10]. calculating the Gram determinant of g may be very slow for large d. Therefore, they propose to train the parameters reconstruction error || g (((x)) only, and the parameters of h using the usual DEL objective for NFs. They propose several training strategies to ensure that g encodes the manifold (manifold phase), and h learns the density (density phase). One strategy alternates between the manifold- and density phase (alternate training) for every training epoch. Another strategy first trains g on the reconstruction error and then learns the density through h (sequential training). Note that a PAE is also trained using a sequential training scheme, as opposed to the DNF which combines the manifold- and density learning phase into single cost function, equation (14), Another difference to the DNF that the Gram determinant
using the
neither used to train the M-flow flow nor to estimate p(z).
5 Results
In this chapter, we confirm experimentally our claims made the end of Section T formalized research question, and answered at the end of Section 3 First, we show that the DNF circumvents the degeneracy problem of standard NF for manifold-suppont supported densities. Afterward, we show that the DNF can learn meaningful latent representations for naturalistic images, without sacrificing the sample quality compared to the M-flow, PAE, and VAE (see Section Our method depends on two hyperparameters, the noise magnitude x² and the penalty coefficient X. o² needs to be sufficiently small such that equation (8) approximately holds (see 20 for more details on the choice of o). A 0, only the density is learned, and if À > 1 the manifold learning dominates.

---

We closely follow the experimental setting of [10]. For the M-floww and DNF, we use the same network architectures and training protocols. These architectures are based on cffine-coulligg layers 13, neural splines [14], and trainable permutations [25] The PAE uses convolutional neural networks for the encoder and decoder, respectively. For images, we use a recently developed variant
of the VAE, the InfoMax-VAN [33], which uses insights from information theory to learn meaningful
latent representations. We refer to the supplementary for more training details.
5.1 Density estimation
We consider a 1-dimensional manifold embedded in R², thin spiral. We draw the latent variable from exponential distribution with scale 0.3, i.e. r(u) x
original pixi
standard
exp(-0.3u), and generate the spiral through g(u) = ey" ((ins((r))))si(((/TT)) where a 4T/3 (upper half of Figure 2 top left).
We train a standard NF (upper right), fllloo (middle left), DNF (middle right). PAE (lower left), VAE (lower right) with similar architectures on 100 epochs with batch size of 100.
flow
DNF
As we can see in Figure 2 the Jacobian determinant of the standard NF degenerates, and the learned density collapses into single points. Surprisingly, the M-floww fails to learn p(x) as well, no matter which training schedule we use (we display the result of the sequential training). The PAE learns an inflated version of p(r), however, it does not have a defla- tion procedure. The standard VAE simply equates p(x) with the Gaussian prior on the latent variables (we take point estimate of the ELBO to approximate p(z)).
PAE
VAE
low
high
For the DNF, we use Gaussian noise with o² 0.01 and A == 1. Note that only the DNF is trained on the inflated den- sity qo (ø), and it encodes the noisy part in vzz). i.e. for points close to the manifold v(ā) should be close to O. Therefore, we set p(x) to whenever v(x) > a a/3 and otherwise we ap- proximate p(z) according to equation (8) with go (x))/4 (arlar) where (2(2) 1/ The fact that the M-llow fails to learn p(x) and the DNF succeeds, suggests the regulatory importance of the Gram determinant of Jo (which is only used from the DNF, and reflects the main difference between those two methods next to the training scheme). When set- ting x² = 0, the DNF degenerates similar to the standard NF (more details in the supplementary). This illustrates the importance of the inflation step for a density supported on a low-dimensional manifold.
orig lrvel mis)
standard NF
0
Jr - flow
PAR
VAIT
All densities are evaluated on a 100 x 100 grid. For that, the M-flow calculates the Gram determinant of the learned generator (see equation (16) and Figure [ (c)) for each point. Like this, evaluating the density
batch consisting
of 500 points takes about 188 seconds on a GPU On the same device and for the same batch size, the DNF needs only 1 second to evaluate the density. This illustrates the drawback of needing calculate the full Gram determinant (see Section 2).
Figure 2: Density estimation
*Our main code is available at based the original flowow implementation made public by the authors of [10] under the MIT license. For the InfoMax- VAE and PAE on images, we use the official implementations made public by the authors of (33 and (0) under the Apache License 2.0 and General Public License v3.0, respectively. TThe Gram matrix be calculated using automatic differentiation.
com/ chwvt/denoiiinni normal izing flow and is
7

---

StyleGAN d =
StyleGAN d 64
Model
FID
Mean reconstr. error
FID
Mean reconstr. error
M-fllww
4.85 + 0.14
309.32 = 6
19.95 + 0.22
1019.18 = 30.08
DNF
4.42 - 0.2
225.78 + 14.4
17.61
899.35 = 37.19
InfoMax- VAE
38.2 + 1.19
12047.92 + 105.95
58.18 + 6.67
16397.54 286.65
PAE
22.58 + 4.50
7181.21 + 27.17
54.51 7.13
7169.97 = 14.25
Table 1: FID and mean reconstruction error of the M-flow, flow, DNF, InfoMax-VAN and PAE on the StyleGAN image manifold for d = 2 and = 64. For d = 2, we train 10 models with different initializations, remove the best and worst results, and report the mean and standard deviation of the remaining 8 models. For d = 64, we train 3 models and report the mean and standard deviation.
To further validate the learned on- on-manifold density, for each model we compare the induced latent density with the ground truth exponential distribution π (u)) (second half of Figure 2 For that, we first generate 1000 latent points multiply the likelihood with the square root of the Gram determinant of the generating mapping g. det G₂(u) x (1 + (au)))/(cu)² This must lead to the correct latent density T(u) if the
(T)), then evaluate the model likelihood at (u), and finally
data-density p(x) det Gg (x)) π (z)) is learned correctly as p(z) is uniquely determined by the pair (g. The standard NF, MM-flow and VAE are out of scale. The PAE follows the exponential course of i (u),, however, only the DNF learns the correct course and scale. This indicates that equation (15) can be used to approximate G,t (u()) well.
In the supplementary, we conduct more density estimation experiments and compare the DNF with the inflation-deflation method [20]. For von Mises distribution circle, and mixture of von Mises distributions on a sphere, the DNF learns the density almost exactly showing the correctness of equation (1)). Also in the supplementary, we use the DNF for probabilistic inference following the protocol used in [TO]
5.2 StyleGAN image manifold
One drawback of both the M-flow and DNF is that the true manifold dimension d must be known beforehand. For real-world datasets, d is unknown. Therefore, [1O] uses a StyleGAN2 model 23] trained on the FFHQ dataset [22] to generate an d-dimensional manifold by only varying the first d latent variables while keeping the remaining fixed.
= 2: We want to show that the DNF learns meaningful latent representations. For that, we first train DNF on 1 images using 100 epochs with o² 0.1 and X = 1000. Then, we generate an image grid by varying the 2-dimensional latent variables u' and mapping them to the data space using the learned mapping ga- In Figure 3 we see on the top left the original grid obtained by the StyleGAN2 model, and on the top right the grid generated by the DNF. A smooth mapping from latent to data space is learned. The PAE (lower left), does not learn such a smooth mapping which may be due to the lack of learning p(z), in contrast to the InfoMax- VAE (lower right). However, the latter lacks variability which shows that the learned posterior p(2 (z) does not match the prior ( 0, I.).
To further evaluate the quality of generated test samples, display the Fréchet inception distance (FID score) 19 30 in Table along with the mean reconstruction error which measures the manifold learning. We slightly outperform the M-flow in terms of FID score, and significantly in terms of the mean reconstruction error. The latter is surprising as the M-flow is directly trained on the reconstruction error. The PAE and InfoMax-VAB perform worse compared to the M-flow and DNF indicating a suboptimal choice for the latent dimension for these models.
'The FID the Wasserstein-2 2 distance between two Gaussians. The mean values and covariance matrices are obtained as sample estimates of the original and model data, respectively. However, instead of using the pixel values, one uses the outcomes of the next to the last layer of the Inception v3 (36)) image classifier trained on the corresponding data set.
8

---

Sty lGGAN latent variable 21
DNF latent varilble
StyleGAN latent variable
DNF latent variable
PAE latent variable ui
InfoMax-VAE variable Us
PAE latent varlable ug
InfoMax-VAE variable un
Figure 3: Image grids, as described in the main text. StyleGAN images (top left) are used train an DNF (top right), PAE (bottom left), and InfoMax-VAL (bottom right).
d = 64: We train the models on 2 10 images for 200 epochs. In Figure 4 show in the first columns samples from the original dataset (top), M-llow (second row), DNF (third row), InfoMax- VAE (fourth row), and PAE (last row). In the remaining columns, we show how smoothly these models interpolate linearly in the latent space. For that, we linearly interpolate between two training images in latent space, and display the corresponding trajectory image space. We compare the FID and mean reconstruction eтrоr of the models in Table Similar to the d = 2 case, the DNF outperforms the M-flow, InfoMax VAE, and PAE terms of FID and mean reconstruction error.
Summary and discussion
Our model is based on the theoretical work established by (20 leading to natural combination of NFs and DAE the DNF. In contrast to similar methods, an DNF is trained on a single objective function combining manifold- and density learning. To learn density supported on low-dimensional manifold with an NF, one needs to compute the flow's Gram determinant. The DNF circumvents this necessity and can be used approximate it. We have pigeonholed the DNF into the literature, and compared its performance on naturalistic images with related methods (M- flow, PAE, VAE). Among those methods, we have that the DNF generates images with the highest quality (in terms of FID), and reconstructs a given input with the lowest L2 distance.
It is well known that adding noise in the input can increase the generalisation performance in supervised learning tasks 29). However, typically this comes with the price of lower sample quality or worse density estimation. The DNF has the potential to avoid this trade-off, and proves

---

Samples
Latent interpolation
Original
DNF
InfoMax-
Figure 4: First columns: Samples from StyleGAN, M-wwl DNF, InfoMax-VAE. and PAE (in this order). Last 5 columns: Latent interpolation as described in the main text.
experimentally that adding noise to images can lead to better performance in terms of sampling quality and manifold learning.
The theoretical foundation of the DNF equation ( which requires the manifold to be sufficiently smooth, and the noise to be added in the manifold's ' normal space. Although for the special case where d << standard Gaussian noise an excellent approximation for a Gaussian in the normal space, more research is needed on how to sample in manifold's normal space in order to improve the performance of the DNF. Essentially, the DNF learns to encode the manifold in the u component and the normal space direction in the component. Arguably, pre-trained DNF could be used to improve the sampling ability in the normal space.
Extensions:
Another limitation of the DNF is that the manifold dimension d needs to be known. Nevertheless, if p(x) is supported on a -ineeimnsio manifold, to the best of our knowledge the DNF together with the M-flow are the only existing methods to learn p(z) and the manifold generating mapping based on NF, Hence, in contrast to other methods which need to know the exact manifold, the DNF and M flow can still be used when d is estimated from samples. The latter is an active research area
Other recently developed methods (such as PAE or M-flow) separate the manifold from the density learning. We have seen that this separation is not necessary, and have used different heuristics to evaluate the quality of learning. For the value of p(z), we visualized the learned (latent) density (Figure 2 For the quality of latent representations, we generated an image grid (Figure B) and for the smoothness in the latent space, we generated an image path (Figure 4). We measured the sampling quality using the FID, and the manifold-learning using the mean reconstruction error (Table D. Given these different heuristics, there is an aspiration for unified performance criterion.
To compress increasingly high-dimensional data with the least loss of information as possible is becoming increasingly important. The DNF ties in with the M-flow and shows that such compression is possible, even for naturalistic images. Similar all other generative models, possible negative impact of DNFs would be the generation of fake data. On a more positive note, the DNF could improve -fi-t--distiibuttinn detection or even increase the robustness to adversarial attacks which are both essential for reliable societal applications of Machine Learning.
Broader Impact:
Acknowledgments and Disclosure of Funding
We thank Camille Gontier, Gagan Narula, Hui-An Shen, Katharina A. Wilmers, Nicolas Zucchet, and the anonymous reviewers for helpful comments. This study has been supported by the Swiss National Science Foundation grant 31003A_11700 175644.
10

---

References
[1] Guillaume Alain and Yoshua Bengio. What regularized auto-cncoders learn from the data- generating distribution. Journal of Machine Learning Research, 15:3743-3773, 2014.
[2] Guozhong An. The effects of adding noise during backpropagation training on generalization performance. Neural Computation, 8(3):4336774 1996.
[3] Chi Au and Judy Tam. Transforming variables using the dirac generalized function. The American Staristician, 53(():270--23) 1999.
[4] Jan Jetze Beitler, Ivan Sosnovik, and Arnold Smeulders. Pie: "seudo-invertible encoder. 2018.
[5] Artur Bekasov and lain Murray. Ordering dimensions with nested dropout normalizing flows. arXiv:2006.08777, 2020.
[6] Clément Berenfeld and Mare Hoffmann. Density estimation unknown submanifold. arXiv:1910.08477, 2019.
[7] Siavash A Bigdeli, Geng Lin, Tiziano Portenier, L Andrea Dunbar, and Matthias Zwicker. Learning generative models using denoising density estimators. arXiv:2001.0277 2020.
[8] Chris Bishop. Training with noise is equivalent to tikhonov regularization, Neural Computa- tion, 7(1):108-11 1995.
[9] Vanessa Böhm and Uroš Seljak. Probabilistic auto-encoder. arXiv:2006.05479.220 2020.
[10] Johann Brehmer and Kyle Cranmer. Flows for simultaneous manifold learning and density estimation. Advances in Neural Information Processing Systems, 33, 2020.
[11] Rob Cornish, Anthony Caterini, George Deligiannidis, and Arnaud Doucet. Relaxing bijectivity constraints with continuously indexed normalising flows. In International Conference on Machine Learning, pages 2133-2143, 2020.
[12] Edmond Cunningham, Renos Zabounidis, Abhinav Agrawal, Ina Fiterau, and Daniel Sheldon. Normalizing flows across dimensions. arXiv:2006.1300 2020.
[13] Laurent Dinh, Jascha Sohl-Dicken Dickstein, and Samy Bengio. Density estimation using real nvp. arXiv1160..08803, 2016.
[14] Conor Durkan, Artur Bekasov, lain Murray, and George Papamakarios. Neural spline flows. Advances in Neural Information Processing Systems, pages 751 7511-7522, 2019.
[15] Elena Facco, Maria d' Errico, Alex Rodriguez, and Alessandro Laio. Estimating the intrinsic dimension of datasets by a minimal neighborhood information. Scientific reports, 7(1):1-8, 2017.
[16] Mevlana C Gemici, Danilo Rezende, and Shakir Mohamed. Normalizing flows on riemannian manifolds. 1rXiv:1611.023000 2016.
[17] Ian J Goodfellow, Jean Pouget-Abaul Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. 47Xiv:1406.2661, 2014.
[18] Matthias Hein and Jean-Yves Audibert. Intrinsic dimensionality estimation of submanifolds in rd. In Proceedings of the 22nd international conference Machine learning, pages 289-296. 2005.
[19] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler. and Sepp Hochreiter. Gans trained by two time-scale update rule converge local nash equilibrium. In Proceedings of the 31st International Conference on Neural Information Processing Systems, pages 6629- 6640, 2017.
[20] Christian Horvat and Lean-Pascal Pfister. Density estimation on low-dimensional manifolds: an inflation-deflation approach. arXiv:2105.121522 2021.

---

[21] Chin-Waa Huang, David Krueger, Alexandre Lacoste, and Aaron Courville. Neural autoregres- sive flows. International Conference on Machine Learning, pages 2078-2085 2018.
[22] T Karras, S Laine, and 'T Aila. A style-based generator architecture for generative adversarial networks. IEEE Transactions on Puttern Analysis and Machine Intelligence, 2020.
[23] Tero Karras, Samuli Laine, Mika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of stylegan. 2020 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 8107-8116, 8116, 2020.
[24] Hyeongju Kim, Hyeonseung Lee, Woo Hyun Kang, Joun Yeop Lee, and Nam Soo Kim. Softflow: Probabilistic framework for normalizing flow on manifolds. Advances in Neural Information Processing Systems, 33, 2020.
[25] Diederik P Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1xl convolu- tions, arXiv:1807.03033 2018.
[26] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv:1312.6114, 2013.
[27] Ivan Kobyzev, Simon Prince, and Marcus Brubaker. Normalizing flows: An introduction and review of current methods. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020.
[28] Abhishek Kumar, Ben Poole, and Kevin Murphy. Regularized autoencoders via relaxed injective probability flow. In International Conference on Artificial Intelligence and Statistics, pages 4292-4301, 2020.
[29] Bai Li, Changyou Chen, Wenlin Wang, and Lawrence Carin. Certified adversarial robustness with additive noise. arXiv:1809.083 13, 2018.
[30] Mario Lucie, Karol Kurach, Marcin Michalski. Sylvain Gelly, and Olivier Bousquet. Are gans created equal? a large-scale study. arXiv:171.11337,p0 2017.
[31] Emile Mathieu and Maximilian Nickel. arXiv:2006. /0605, 2020.
Riemannian continuous normalizing flows.
[32] George Papamakarios, Eric Nalisnick, Danilo Jimenez Rezende, Shakir Mohamed, and Balaji Lakshminarayanan. Normalizing flows for probabilistic modeling and inference. 1rXiv:1912.02762, 2019.
[33] Ali Lotfi Rezaabad and Sriram Vishwanath. Learning representations by maximizing mutual information in variational autoencoders. In 2020 IEEE International Symposium on Information Theory (ISIT), pages 2729 2729-2734, 2020.
[34] Danilo Rezende and Shakir Mohamed. Variational inference with normalizing flows. In International Conference Machine Learning, pages 1530-1538, 2015.
[35] Danilo Jimenez Rezende. George Papamakarios, Sébastien Racaniere, Michael Albergo, Gurtej Kanwar, Phiala Shanahan, and Kyle Cranmer. Normalizing flows on tori and spheres. Interna- tional Conference on Machine Learning, pages
2020.
[36] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition (CVPR), June 2015.
[37] Wenju Xu, Shawn Keshmiri, and Guanghui Wang. Adversarially approximated autoencoder for image generation and manipulation. IEEE Transactions on Multimedia, 21(9):2387-2333 2019.
12

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes] See Section
(b) Did you describe the limitations of your work? [Yes] See Section 6
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See
Section
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes] In the supplemental
material.
(c) improve spiral display -> color code
(d) interpretation of cost function LNNF, how do you know which lambda is the best?
(e) Wasserstein distances for comparing different X? -> number for goodness of learned density for Fig.2
(f) Normal sampling: can we improve it using the DNF? Try JP idea.
(g) interplay À and o?
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed reproduce the main ex- perimental results (either in the supplemental material or as a URL)? [No] But it is planned.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] In the supplemental material.
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [Yes] See Table
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [No]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]] See footnote 4
(b) Did you mention the license of the assets? [Yes] See footnote
(c) Did you include any new assets either the supplemental material or as URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [No]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [No]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "G7dUf17xnRR", "reviewer": "Reviewer_xmqi", "summary": "This paper proposes Denoising Normalizing Flows (DNFs), a method for accounting for low-dimensional manifold structure of data when using Normalizing Flows (NFs). The method builds on the recent work Horvat and Pfister ([18] in the manuscript), who showed that under some conditions, adding a small amount of Gaussian noise (\"inflation\") to data and then using a fully dimensional NF can recover the correct distribution on the manifold after a \"deflation\" step. The authors combine injective flows, mapping from low-to high dimensional space, with the inflation/deflation process, in an attempt to recover a low dimensional representation as well (as opposed to just using the inflation/deflation method); and properly accounting for the change-of-volume term during training (as opposed to the M-flow method of Brehmer and Cranmer, [9] in the manuscript).", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "While I think the problem that this paper tackles is relevant and that the idea is nice, I found it to not be very well written and a bit hard to follow. I also have my doubts about the experimental results. I elaborate below.\n\n1. I find the presentation for the objective hard to parse. Equation 13 is introduced as \"the model\", but does not really correspond to a density (proposition 1, which enables the density interpretation, is only presented later), and the fact that this equation avoids the injective change-of-volume term is not emphasized. The notation \\tilde{u} is also confusing, as so far tilde had been used to denote added Gaussian noise. I had to read the section a couple times before I understood what the authors were trying to do. I do find this idea very interesting though, as an elegant way of avoiding having to compute the challenging log determinant by instead computing two easy-to-compute ones.\n\n2. Part of the requirements for proposition 1 is that L_{DNF}(\\theta) = 0, but this can actually never happen unless \\sigma=0, since C(\\psi) (equation 12) will in general not be 0 unless \\sigma=0. The way I understand proposition 1 is as justifying using equation (13) rather than accounting for the injective change-of-volume-term; but if this condition can never be met, it throws the usefulness of proposition 1 into question. Since C(\\psi) is arbitrarily chosen, would it not make sense to change it to E_{x \\sim p(x)}[||x - r_\\psi(x)||^2]?\n\n3. I think a more detailed discussion of the benefits of DNFs as compared to the inflation/deflation method is warranted, as this baseline also manages to account for low-dimensional manifold support while remaining tractable. For a method so heavily inspired by this previous work, empirical comparisons against it should be carried out (I am aware that the inflation/deflation paper was mostly theoretical and presented only low-dimensional experiments; but still think the comparison should be made).\n\n4. I find the results on the toy example in Figure 2 for all baselines to be suspiciously poor. I have personally trained standard NFs and M-flows on very similar 2-D examples with a 1-D manifold -- albeit not the same one -- and obtained much better (though not perfect) results. The authors do not specify which range of hyperparameters were considered for this example, and I suspect they might have not given a fair chance to the baselines by not tuning them enough.\n\n5. Similarly, Table 1 shows DNFs achieve smaller reconstruction errors than M-flows. I also find this intriguing, as DNFs incorporate reconstruction error as a regularization term in the loss; thus achieving a tradeoff between this term and the KL term; whereas M-flows train the injective flow exclusively to minimize reconstruction error. Again, without a detailed explanation for why this might be happening, or at the very least verification that it wasn't a fluke, I can only be suspicious of the results.\n\n6. I also think experiments on actual images rather than just naturalistic images would improve the paper.\n\nAdditional thoughts/questions/typos:\n\n-Line 27: I don't see the relevance of the manifold being Riemannian: why is it having a metric relevant?\n\n-Lines 36-38: the sentence is not properly formed.\n\n-Equation 1: G_g should be defined, rather than its determinant.\n\n-Lines 57-58: the statement \" in generative modeling, it is assumed that the data x \\in M is generated by a non-linear, injective transformation g...\". This statement is too strong and should be toned down; this assumption is made only in injective flows, and as a simplifying assumption, as it implies -- unrealistically -- that the support is homeomorphic to R^d.\n\n-Line 109: q_\\sigma(\\tilde{x} \\cup N_x|x) is not defined, nor is it standard notation.\n\n-Line 126 says q_\\sigma(\\tilde{x}|x) is a D-d dimensional Gaussian, and \\tilde{x} is D-dimensional (in the algorithm box, D-dimensional noise is added).\n\n-Line 171: The work of [1] is relevant when discussing overcoming topological constraints in flows.\n\n-Line 213: neuronal -> neural\n\n-Line 249: n-dimensional -> d-dimensional\n\n-Line 261: tabular 5.2 -> Table 1\n\n-Figure 2 and Table 1 have captions on top and on bottom, and it should be the other way around.\n\n-Lines 307-308: You have not showed DNFs can be used for out-of-distribution detection nor for robustness, these claims should not be made without empirical backing, or should be clarified as hypotheticals, with at least some intuition as to why they might be true.\n\n[1] Relaxing Bijectivity Constraints with Continuously Indexed Normalising Flows, Cornish et al., ICML 2020\n\n======================================================================================================\n\nEDIT 1 AFTER REBUTTAL\n\nI have read the other reviews as well as the author's rebuttal, and thank the authors for their additional clarifications. While some of my doubts about some experiments have been resolved, and I agree that my initial suggestion for $C(\\psi)$ was not good, I still have some of my original concerns: The assumption for proposition 1 to hold is that noise is added in the normal direction to $x$ rather than Gaussian noise as the authors add in practice; which effectively means there is no guarantee that the determinant can actually be approximated. Additionally, like I mentioned in my response to the author's rebuttal, adding noise just to $v$ seems like a good way of ensuring that $x$ can indeed be recovered from $\\tilde{x}$ -- which is not ensured by the Gaussian noise they use -- and thus a more natural way of adding noise and taking advantage of the flow structure than what the authors currently do. This modification to the author's approach seems to me like a more natural way to proceed, and would need to either be shown the modification working, or an argument for why it doesn't before increasing my score, which I am thus leaving unchanged for now.\n\n======================================================================================================\n\nEDIT 2 AFTER REBUTTAL\n\nI have read the additional verification performed by the authors on reconstruction errors and have decided to increase my score. While I am still concerned about the theoretical claims not applying to the proposed choice of inflation noise and that exploring other alternatives would further strengthen the paper, I am no longer skeptical of the empirical results, which I actually find interesting and intriguing. I would still strongly encourage the authors to both:\n\n1. More clearly specify that the Gaussian noise they use for inflation does not satisfy the requirements for their theorem, and that the theorem should thus be interpreted as giving intuition, rather than providing guarantees for the proposed empirical method.\n\n2. Either try to explain why $\\mathcal{M}$-flows achieve worse reconstruction error than the proposed method, or highlight this as intriguing behaviour. I still believe these results are a bit puzzling as $\\mathcal{M}$-flows directly minimize reconstruction while the proposed method only includes a similar term as a component of its loss function.\n\n======================================================================================================", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations and potential negative societal consequences are adequately discussed.", "review_text": "", "extra_scores": {"ethical_concerns": "None.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "CSFIFDFqUUK", "reviewer": "Reviewer_8Mvk", "summary": "This paper is motivated by the desire to circumvent a troublesome assumption in Normalizing Flows (NFs) on the function $f$ that produces the pushforward measure --- that $f$ must be strictly invertible. This is difficult for generative models, since the dimension of the latent space $d$ is usually much smaller than that of the target data space $D$, i.e. d << D.\n\nTo address this, the authors train a transformation $f:\\mathbb{R}^D \\rightarrow \\mathbb{R}^D$ as usually done in a NF approach, but separate the latent variable $z = f^{-1}(x)$ into two orthogonal components $u, v$, consisting of the first $d$ and remaining $D-d$ coordinates of $z$. Intuitively, $u$ should contain the \"manifold\" information of $x$, and $v$ should capture any \"off-manifold\" noise. This is encouraged by adding a reconstruction loss to the $u$ component of the latent space. To allow for greater complexity of the latent variable $u$, a second NF transformation $h$ is applied to u, i.e. $u = h(\\tilde{u})$. In addition, the authors add noise robustness by adding Gaussian noise to $x$, i.e. $\\tilde(x) = x + \\epsilon$, and reconstruct $x$ --- much like with Denoising Autoencoders, hence the name.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "There are two main aspects to the denoising NF method:\n1. The latent variable $z$ is decomposed into two parts $(u, v)$, where one part ($u$) is encouraged to encode the signal, and the other part ($v$) is encouraged to encode the noise.\n2. Noise is added to the original training sample $x$, and the proposed method is trained to denoise the noisy $\\tilde{x}$, like in Denoising Autoencoders.\n\nThe authors disclose that part 1) has been attempted in part by a prior paper, but the addition of a \"reconstruction loss\" as motivated by part 2) is crucial, in my opinion, and makes the contribution in this paper unique.\n\nMoreover, the method is well-motivated --- there is currently no good way to apply NFs to generative models, even though NFs are a promising candidate for deep generative modeling. Moreover, the technique is straightforward and technically sound. \n\nHowever, I do have three issues with the paper as it stands:\n\n1. I find the experiments to be somewhat lacking. The manifold assumption, while powerful, does not usually hold exactly in real-world data. Both experiments in the main text (synthetic spiral, StyleGAN image manifold) involve data that lie strictly on a manifold. The supplemental material includes an experiment with the CelebA faces dataset. However, the proposed method is only compared to one other method (M-flow [1]). I would like to see how DNFs perform against GANs and VAEs on more realistic datasets.\n2. The connection to Denoising Autoencoders is nice, but it seems that Experiment C.1 (in the Supplementary Material) suggests that the noise parameter $\\sigma^2$ is not that crucial to the performance of the model. It would be nice to see further discussion of the importance of the Denoising aspect of the method. \n3. I found the paper well written overall, but a crucial detail seems to be incorrect. In Eq. 14, it appears that the loss $\\mathcal{L}_{DNF}(\\theta)$ is the (weighted) sum of the reconstruction term $C(\\psi)$ and a KL divergence term between $q_\\sigma$ and $q_\\theta$. However, in the algorithm, the KL divergence term is replaced by a negative log probability term $-\\log q_\\theta$. Is Eq. 14 simply a typo, or am I misunderstanding something? (Furthermore, the $p_v(v_i)$ term in the negative log probability seems to be missing a $\\log$.) To my knowledge, it is not possible to evaluate the KL divergence between $q_\\sigma$ and $q_\\theta$, as $q_\\sigma(\\tilde(x)) = q_\\sigma(\\tilde(x) | x) p(x)$, and $p(x)$ is not known.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "As a highly theoretical work, there are no clear societal impacts of this paper. I am, however, concerned with the applicability of this work in datasets where the manifold assumption does not hold. As mentioned, I think that an exploration of the performance of DNFs in more realistic datasets will help explore this question.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "10"}, "extra_text": {}}, {"review_id": "9-wnkKeEPu3", "reviewer": "Reviewer_TzDc", "summary": "Authors propose a novel normalizing-flow-based method for simultaneous manifold and probability-density-on-the-manifold learning. Similarly to previous methods addressing this problem, the dimensionality of the manifold learned needs to be set by the user before training and is not learned as part of the process, albeit there exist other methods for estimating it. The advantage of the proposed method is the single-step training procedure, in contrast to the multi-step training procedure of the other competitive method in this domain. I consider it a good contribution towards building a normalizing flow based representation learning toolkit.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Good paper: the core idea seems to be well developed and explained, decent related work section, good mathematical notation. My main criticism is that I would encourage the authors to expand the empirical evaluation section.\n\n*Originality:*  \n- To the best of my knowledge the method is novel, although based to some extent on an existing insight.\n- While it might be true that “the inflation approach” for learning manifold density with normalizing flows was discussed in particular depth in [18], authors should acknowledge that the idea of applying it in exactly this context was present in the literature earlier than [18] --- one earlier work to point out is [22], but it’s possible there exist works earlier than this one. I think it’s worth acknowledging it properly for the sake of proper credit assignment in the community.  \n- Apart from this detail, I find the related work section quite well developed and Fig 1 does a good job explaining how the work differs from some of the other most related methods.  \n- One other interesting work to cite in this context, especially regarding the detection of the manifold dimensionality, is arxiv:2006.08777.   \n\n*Quality:*  \n- I believe the work is technically sound, the core idea seems to be well developed and explained, and most of the conclusions regarding the empirical evaluation of the proposed method are substantiated.\n- I haven’t caught any apparent errors but due to the heavy reviewing workload I could not commit enough time to carefully check the mathematical arguments, especially the ones in [18], which seems to not have been published in any proceedings before and hence I assume did not undergo peer review.  \n- [22] learn to utilize NFs to sample from the manifold, but it’s not useful for representation learning, due to the lack of ability to learn latent representations of dimensionality lower than the data space dimensionality. Both [9] and the proposed method overcome those difficulties and showcase their utility for representation learning. [9] also showcases its utility for probabilistic inference and I’m convinced this work could also evaluate itself in such a setting which would enrich the paper, and currently in my opinion somewhat modest empirical evaluation section (modest compared to [9] which is quite rich so that might not be an entirely fair point of comparison). However, given that the key metric the authors are comparing on is sample quality, it might be worthwhile to compare the proposed method also to Softflow --- keeping the architecture fixed as much as possible across models, it would provide an interesting comparison of what is the sample-quality price to be paid for the representation learning properties of the proposed method.  \n- One of the key goals of the proposed method is to learn the on-manifold density, but the performance in that aspect of the task is barely evaluated empirically --- the only part of the evaluation that touches upon that aspect is the limited visual comparison offered in Figure 2. In all honesty, it is not immediately clear to me what would make for a good protocol to empirically evaluate this aspect of the method, so I am sympathetic to the fact it’s not studied better in the current version of the paper. However, I would encourage authors to think about this.  \nOne vague idea that comes to my mind is: for some synthetic toy problem, make use of the knowledge of the underlying manifold to define a hand-crafted diffeomorphic transform between the learned manifold and the true manifold (since the learned one obviously will not be perfect), and then compute e.g. KL divergence between those two d-dimensional probability densities. Is this a viable direction?  \n- L117-118: I’m wondering about the effect of using the full Gaussian noise on the on-manifold density in cases when the on-manifold density changes quickly, e.g. the on-manifold density oscillates with a high frequency. Could authors comment on this please?  \n\n\n*Clarity:*  \n- Yes, the submission is mostly clearly & well written. There were very few points when I had to reread or pause due to the quality of writing.\n- Formatting wise, I find the “Research question” and “Question 1” right next to each other a bit too much, but overall I appreciate the use of paragraph captions.\n- Is “the characteristic function” \\mathcal{X} in L235-236 intended to mean an indicator function? This isn't immediately clear to me.  \n- Fig. 2 goes outside of allowed margins.  \n- Formatting of the citation: ([1]) -> [1], ([1], [2]) -> [1, 2]   \n\n*Significance:*  \n- I consider this paper a good contribution towards building a normalizing flow based representation learning toolkit, and I think it might find downstream applications in the future, especially given its simplified training procedure.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Authors did a fair job discussing the limitations of their method.\n\nSocietal impact: N/A\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes an NF with simultaneous manifold and density-on-manifold learning.  Inflation noise is added to make p(x) smooth and the reconstruction regularizer makes the first d variables independent from noise and hence extracts information of clean data. 

The proposed method is well-motivated and shows good performance.  We expect that the authors revise the paper according to the discussion made with reviewers.
Decision: Accept (Poster)