Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: Authors propose a novel normalizing-flow-based method for simultaneous manifold and probability-density-on-the-manifold learning. Similarly to previous methods addressing this problem, the dimensionality of the manifold learned needs to be set by the user before training and is not learned as part of the process, albeit there exist other methods for estimating it. The advantage of the proposed method is the single-step training procedure, in contrast to the multi-step training procedure of the other competitive method in this domain. I consider it a good contribution towards building a normalizing flow based representation learning toolkit.
main_review: Good paper: the core idea seems to be well developed and explained, decent related work section, good mathematical notation. My main criticism is that I would encourage the authors to expand the empirical evaluation section.

*Originality:*  
- To the best of my knowledge the method is novel, although based to some extent on an existing insight.
- While it might be true that “the inflation approach” for learning manifold density with normalizing flows was discussed in particular depth in [18], authors should acknowledge that the idea of applying it in exactly this context was present in the literature earlier than [18] --- one earlier work to point out is [22], but it’s possible there exist works earlier than this one. I think it’s worth acknowledging it properly for the sake of proper credit assignment in the community.  
- Apart from this detail, I find the related work section quite well developed and Fig 1 does a good job explaining how the work differs from some of the other most related methods.  
- One other interesting work to cite in this context, especially regarding the detection of the manifold dimensionality, is arxiv:2006.08777.   

*Quality:*  
- I believe the work is technically sound, the core idea seems to be well developed and explained, and most of the conclusions regarding the empirical evaluation of the proposed method are substantiated.
- I haven’t caught any apparent errors but due to the heavy reviewing workload I could not commit enough time to carefully check the mathematical arguments, especially the ones in [18], which seems to not have been published in any proceedings before and hence I assume did not undergo peer review.  
- [22] learn to utilize NFs to sample from the manifold, but it’s not useful for representation learning, due to the lack of ability to learn latent representations of dimensionality lower than the data space dimensionality. Both [9] and the proposed method overcome those difficulties and showcase their utility for representation learning. [9] also showcases its utility for probabilistic inference and I’m convinced this work could also evaluate itself in such a setting which would enrich the paper, and currently in my opinion somewhat modest empirical evaluation section (modest compared to [9] which is quite rich so that might not be an entirely fair point of comparison). However, given that the key metric the authors are comparing on is sample quality, it might be worthwhile to compare the proposed method also to Softflow --- keeping the architecture fixed as much as possible across models, it would provide an interesting comparison of what is the sample-quality price to be paid for the representation learning properties of the proposed method.  
- One of the key goals of the proposed method is to learn the on-manifold density, but the performance in that aspect of the task is barely evaluated empirically --- the only part of the evaluation that touches upon that aspect is the limited visual comparison offered in Figure 2. In all honesty, it is not immediately clear to me what would make for a good protocol to empirically evaluate this aspect of the method, so I am sympathetic to the fact it’s not studied better in the current version of the paper. However, I would encourage authors to think about this.  
One vague idea that comes to my mind is: for some synthetic toy problem, make use of the knowledge of the underlying manifold to define a hand-crafted diffeomorphic transform between the learned manifold and the true manifold (since the learned one obviously will not be perfect), and then compute e.g. KL divergence between those two d-dimensional probability densities. Is this a viable direction?  
- L117-118: I’m wondering about the effect of using the full Gaussian noise on the on-manifold density in cases when the on-manifold density changes quickly, e.g. the on-manifold density oscillates with a high frequency. Could authors comment on this please?  


*Clarity:*  
- Yes, the submission is mostly clearly & well written. There were very few points when I had to reread or pause due to the quality of writing.
- Formatting wise, I find the “Research question” and “Question 1” right next to each other a bit too much, but overall I appreciate the use of paragraph captions.
- Is “the characteristic function” \mathcal{X} in L235-236 intended to mean an indicator function? This isn't immediately clear to me.  
- Fig. 2 goes outside of allowed margins.  
- Formatting of the citation: ([1]) -> [1], ([1], [2]) -> [1, 2]   

*Significance:*  
- I consider this paper a good contribution towards building a normalizing flow based representation learning toolkit, and I think it might find downstream applications in the future, especially given its simplified training procedure.

limitations_and_societal_impact: Authors did a fair job discussing the limitations of their method.

Societal impact: N/A
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper is motivated by the desire to circumvent a troublesome assumption in Normalizing Flows (NFs) on the function $f$ that produces the pushforward measure --- that $f$ must be strictly invertible. This is difficult for generative models, since the dimension of the latent space $d$ is usually much smaller than that of the target data space $D$, i.e. d << D.

To address this, the authors train a transformation $f:\mathbb{R}^D \rightarrow \mathbb{R}^D$ as usually done in a NF approach, but separate the latent variable $z = f^{-1}(x)$ into two orthogonal components $u, v$, consisting of the first $d$ and remaining $D-d$ coordinates of $z$. Intuitively, $u$ should contain the "manifold" information of $x$, and $v$ should capture any "off-manifold" noise. This is encouraged by adding a reconstruction loss to the $u$ component of the latent space. To allow for greater complexity of the latent variable $u$, a second NF transformation $h$ is applied to u, i.e. $u = h(\tilde{u})$. In addition, the authors add noise robustness by adding Gaussian noise to $x$, i.e. $\tilde(x) = x + \epsilon$, and reconstruct $x$ --- much like with Denoising Autoencoders, hence the name.

main_review: There are two main aspects to the denoising NF method:
1. The latent variable $z$ is decomposed into two parts $(u, v)$, where one part ($u$) is encouraged to encode the signal, and the other part ($v$) is encouraged to encode the noise.
2. Noise is added to the original training sample $x$, and the proposed method is trained to denoise the noisy $\tilde{x}$, like in Denoising Autoencoders.

The authors disclose that part 1) has been attempted in part by a prior paper, but the addition of a "reconstruction loss" as motivated by part 2) is crucial, in my opinion, and makes the contribution in this paper unique.

Moreover, the method is well-motivated --- there is currently no good way to apply NFs to generative models, even though NFs are a promising candidate for deep generative modeling. Moreover, the technique is straightforward and technically sound. 

However, I do have three issues with the paper as it stands:

1. I find the experiments to be somewhat lacking. The manifold assumption, while powerful, does not usually hold exactly in real-world data. Both experiments in the main text (synthetic spiral, StyleGAN image manifold) involve data that lie strictly on a manifold. The supplemental material includes an experiment with the CelebA faces dataset. However, the proposed method is only compared to one other method (M-flow [1]). I would like to see how DNFs perform against GANs and VAEs on more realistic datasets.
2. The connection to Denoising Autoencoders is nice, but it seems that Experiment C.1 (in the Supplementary Material) suggests that the noise parameter $\sigma^2$ is not that crucial to the performance of the model. It would be nice to see further discussion of the importance of the Denoising aspect of the method. 
3. I found the paper well written overall, but a crucial detail seems to be incorrect. In Eq. 14, it appears that the loss $\mathcal{L}_{DNF}(\theta)$ is the (weighted) sum of the reconstruction term $C(\psi)$ and a KL divergence term between $q_\sigma$ and $q_\theta$. However, in the algorithm, the KL divergence term is replaced by a negative log probability term $-\log q_\theta$. Is Eq. 14 simply a typo, or am I misunderstanding something? (Furthermore, the $p_v(v_i)$ term in the negative log probability seems to be missing a $\log$.) To my knowledge, it is not possible to evaluate the KL divergence between $q_\sigma$ and $q_\theta$, as $q_\sigma(\tilde(x)) = q_\sigma(\tilde(x) | x) p(x)$, and $p(x)$ is not known.
limitations_and_societal_impact: As a highly theoretical work, there are no clear societal impacts of this paper. I am, however, concerned with the applicability of this work in datasets where the manifold assumption does not hold. As mentioned, I think that an exploration of the performance of DNFs in more realistic datasets will help explore this question.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes Denoising Normalizing Flows (DNFs), a method for accounting for low-dimensional manifold structure of data when using Normalizing Flows (NFs). The method builds on the recent work Horvat and Pfister ([18] in the manuscript), who showed that under some conditions, adding a small amount of Gaussian noise ("inflation") to data and then using a fully dimensional NF can recover the correct distribution on the manifold after a "deflation" step. The authors combine injective flows, mapping from low-to high dimensional space, with the inflation/deflation process, in an attempt to recover a low dimensional representation as well (as opposed to just using the inflation/deflation method); and properly accounting for the change-of-volume term during training (as opposed to the M-flow method of Brehmer and Cranmer, [9] in the manuscript).
main_review: While I think the problem that this paper tackles is relevant and that the idea is nice, I found it to not be very well written and a bit hard to follow. I also have my doubts about the experimental results. I elaborate below.

1. I find the presentation for the objective hard to parse. Equation 13 is introduced as "the model", but does not really correspond to a density (proposition 1, which enables the density interpretation, is only presented later), and the fact that this equation avoids the injective change-of-volume term is not emphasized. The notation \tilde{u} is also confusing, as so far tilde had been used to denote added Gaussian noise. I had to read the section a couple times before I understood what the authors were trying to do. I do find this idea very interesting though, as an elegant way of avoiding having to compute the challenging log determinant by instead computing two easy-to-compute ones.

2. Part of the requirements for proposition 1 is that L_{DNF}(\theta) = 0, but this can actually never happen unless \sigma=0, since C(\psi) (equation 12) will in general not be 0 unless \sigma=0. The way I understand proposition 1 is as justifying using equation (13) rather than accounting for the injective change-of-volume-term; but if this condition can never be met, it throws the usefulness of proposition 1 into question. Since C(\psi) is arbitrarily chosen, would it not make sense to change it to E_{x \sim p(x)}[||x - r_\psi(x)||^2]?

3. I think a more detailed discussion of the benefits of DNFs as compared to the inflation/deflation method is warranted, as this baseline also manages to account for low-dimensional manifold support while remaining tractable. For a method so heavily inspired by this previous work, empirical comparisons against it should be carried out (I am aware that the inflation/deflation paper was mostly theoretical and presented only low-dimensional experiments; but still think the comparison should be made).

4. I find the results on the toy example in Figure 2 for all baselines to be suspiciously poor. I have personally trained standard NFs and M-flows on very similar 2-D examples with a 1-D manifold -- albeit not the same one -- and obtained much better (though not perfect) results. The authors do not specify which range of hyperparameters were considered for this example, and I suspect they might have not given a fair chance to the baselines by not tuning them enough.

5. Similarly, Table 1 shows DNFs achieve smaller reconstruction errors than M-flows. I also find this intriguing, as DNFs incorporate reconstruction error as a regularization term in the loss; thus achieving a tradeoff between this term and the KL term; whereas M-flows train the injective flow exclusively to minimize reconstruction error. Again, without a detailed explanation for why this might be happening, or at the very least verification that it wasn't a fluke, I can only be suspicious of the results.

6. I also think experiments on actual images rather than just naturalistic images would improve the paper.

Additional thoughts/questions/typos:

-Line 27: I don't see the relevance of the manifold being Riemannian: why is it having a metric relevant?

-Lines 36-38: the sentence is not properly formed.

-Equation 1: G_g should be defined, rather than its determinant.

-Lines 57-58: the statement " in generative modeling, it is assumed that the data x \in M is generated by a non-linear, injective transformation g...". This statement is too strong and should be toned down; this assumption is made only in injective flows, and as a simplifying assumption, as it implies -- unrealistically -- that the support is homeomorphic to R^d.

-Line 109: q_\sigma(\tilde{x} \cup N_x|x) is not defined, nor is it standard notation.

-Line 126 says q_\sigma(\tilde{x}|x) is a D-d dimensional Gaussian, and \tilde{x} is D-dimensional (in the algorithm box, D-dimensional noise is added).

-Line 171: The work of [1] is relevant when discussing overcoming topological constraints in flows.

-Line 213: neuronal -> neural

-Line 249: n-dimensional -> d-dimensional

-Line 261: tabular 5.2 -> Table 1

-Figure 2 and Table 1 have captions on top and on bottom, and it should be the other way around.

-Lines 307-308: You have not showed DNFs can be used for out-of-distribution detection nor for robustness, these claims should not be made without empirical backing, or should be clarified as hypotheticals, with at least some intuition as to why they might be true.

[1] Relaxing Bijectivity Constraints with Continuously Indexed Normalising Flows, Cornish et al., ICML 2020

======================================================================================================

EDIT 1 AFTER REBUTTAL

I have read the other reviews as well as the author's rebuttal, and thank the authors for their additional clarifications. While some of my doubts about some experiments have been resolved, and I agree that my initial suggestion for $C(\psi)$ was not good, I still have some of my original concerns: The assumption for proposition 1 to hold is that noise is added in the normal direction to $x$ rather than Gaussian noise as the authors add in practice; which effectively means there is no guarantee that the determinant can actually be approximated. Additionally, like I mentioned in my response to the author's rebuttal, adding noise just to $v$ seems like a good way of ensuring that $x$ can indeed be recovered from $\tilde{x}$ -- which is not ensured by the Gaussian noise they use -- and thus a more natural way of adding noise and taking advantage of the flow structure than what the authors currently do. This modification to the author's approach seems to me like a more natural way to proceed, and would need to either be shown the modification working, or an argument for why it doesn't before increasing my score, which I am thus leaving unchanged for now.

======================================================================================================

EDIT 2 AFTER REBUTTAL

I have read the additional verification performed by the authors on reconstruction errors and have decided to increase my score. While I am still concerned about the theoretical claims not applying to the proposed choice of inflation noise and that exploring other alternatives would further strengthen the paper, I am no longer skeptical of the empirical results, which I actually find interesting and intriguing. I would still strongly encourage the authors to both:

1. More clearly specify that the Gaussian noise they use for inflation does not satisfy the requirements for their theorem, and that the theorem should thus be interpreted as giving intuition, rather than providing guarantees for the proposed empirical method.

2. Either try to explain why $\mathcal{M}$-flows achieve worse reconstruction error than the proposed method, or highlight this as intriguing behaviour. I still believe these results are a bit puzzling as $\mathcal{M}$-flows directly minimize reconstruction while the proposed method only includes a similar term as a component of its loss function.

======================================================================================================
limitations_and_societal_impact: Limitations and potential negative societal consequences are adequately discussed.
review_title: null

Meta-review: The paper proposes an NF with simultaneous manifold and density-on-manifold learning.  Inflation noise is added to make p(x) smooth and the reconstruction regularizer makes the first d variables independent from noise and hence extracts information of clean data. 

The proposed method is well-motivated and shows good performance.  We expect that the authors revise the paper according to the discussion made with reviewers.
Decision: Accept (Poster)