Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: This work presents a novel theoretical development to tackle the problem of estimating normalising flows (NF) for data with support on complex manifolds. Motivated by the fact that NF are diffeomorphic  transformations from a simple space, ideally Euclidean, the author address the problem of modeling data which distribution is defined on more complex and unknow manifolds.  
The idea consists in inflating the data manifold with suitable noise (normal noise) in order to make it diffeomorphic to a simpler space where a NF can be estimated. The data density can be therefore approximated from the inflated space through an opportune deflation operation.

The main contribution of the work consists in formalising the conditions on the inflating noise to obtain the results of Theorem 1. In practice, Proposition 1 shows that the noise can be drawn from a Gaussian with variance depending from the curvature of the manifold. The experimental section demonstrates the theory on simple synthetic cases on S2.

The idea proposed in the paper is interesting, and of potentially utility. The derivation is however unclear in some aspects, especially concerning the definition of Q-separation. In particular, Definition 1 implies null measure for the set of manifold points spanning the normal subspace. However, to my understanding, this definition is not sufficient to guarantee the unicity of the mapping between normal space and manifold point. For example, any finite collection of manifold points would still have zero measure. In this case however, the unique correspondence between normal space and manifold is broken. Back to Example 2, this non-unicity can be achieved by increasing the noise along the normal direction to cross the origin (0,0). In this case each point of the inflated space has two generators, making the isomorphic mapping  impossible. I can understand that this aspect is related to the choice of the noise discussed in Section 3.3, but this issue still undermines the conclusion of Theorem 1. I would encourage the authors to clarify this point.

There are  further technical aspects deserving clarification. In Example 2, the definition of the set A(\tilde{x}) should not contemplate the empty set. Rather, A(\tilde{x}) should be the point x’ if \tilde{x} is not (0,0). Moreover, in Proof A.1, equation (13), the relation between the q_n and \hat{q}_n in the two first integrals, and the reason why they can be exchanged, is not clear. 

Finally, some questions arise concerning the practical feasibility of the proposed theory. Computing the curvature of the data manifold may be extremely challenging in presence of complex data, especially in the low sample-size regime. In this case, the estimation of the noise parameter in formula (9) may be practically impossible.
review_title: Interesting framework to train NF for manifold-supported data. Questions about theoretical development and practical utility.

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper tackles the problem of recovering a probability distribution, which is supported in a low-dimensional manifold. When the dimension of that manifold is full, the problem can be solved by the Normalizing Flow method. 

Since the dimension is not full, it is suggested to add a gaussian noise to the data points (this is equivalent to a certain convolution of the initial distribution function, ie equation 3). Then full dimensional manifold is recovered and subsequently deconvolution is applied (equation 7).

Certain mathematical aspects of the narrative are vague. Eg \tilde{X} is defined as X x Noise, but then in equations 4 and 5, it becomes a subset of the plane. 

The notion of Q-normally separated is never satisfied in practice. It is only slightly stronger than "all the normal spaces have no intersections at all". The deconvolution made by equation 7 is rarely is a good solution.

Also, the experimental part is purely dealing with synthetic examples. This could be because the deconvolution by equation 7 is not the best way to recover the signal. 

It is claimed that the first goal of synthetic experiments is to confirm the scaling factor in equation (7) numerically. But experiments deal with highly specific synthetic data - a curve (ie, 1D manifold) on the plane. It is doubtful that such a simple experimental framework can justify eq (7). It is not computationally difficult to check the quality of deconvolution in a style of eq (7) for higher-dimensional data.

review_title: Mathematical part is vague, no experiments

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper proposes a method for estimating the probability deinsity distribution on a low-dimensional manifold embedded in a high-dimensional space using Normalizing Flow (NF). The problem is that the universality of NF is limited because low-dimensional manifolds are not diffeomorphic with respect to high-dimensional Euclidean space. The proposal of this study is to make NF applicable by inflating low-dimensional manifolds with Gaussian noise. Then, after the transformation is obtained, the probability distribution on the original low-dimensional manifold can be obtained by deflation.

There is a detailed discussion about the addition of Gaussian noise. Theoretically, any Gaussian noise $ e $ at coordinate x can be decomposed into projections on tangent space $ T_x $ and normal space $ N_x $ as $ e = e_t + e_n $. Only normal noise $ e_n $ is ideal to add. However, it is not realistic to find this at each coordinate x. This paper argues that Gaussian noise $ e $ is a good approximation of $ e_n $ when the manifold dimension $ d $ is sufficiently smaller than the higher-dimensional Euclidean space dimension $ D $. This also argues that the noise variance $ \ sigma $ at that time should be set according to the inverse of the density. 

In computer experiments, simple simulation results using the von mises distribution on a circle and a sphere are shown.

The gist of the manuscript is well-written, and the issues it deals with are also important and interesting.
Theoretically, interesting discussions are being developed, but discussions and experimental results are somewhat weak regarding the merits of practical application.

I have some questions.
1. I couldn't understand the description about scalability well, "our method scales to high dimensions because it is based on one NF and does not require to compute its inverse." Isn't the scalability the same as normal NF because the proposed method basically uses normal NF?

2. In this research, Authors propose to add noise to the data sampled from low-dimensional manifolds in a pseudo manner, but in actual measurement, noise has been included in data naturally without adding it in a pseudo manner. Is it necessary to add pseudo noise even when applying it to real world data?

3. The denoising auto-encoder is famous as a manifold learning method that adds pseudo noise. Is there any relation to this? Also, I would like to know if there are any advantages of the proposed method over the denoising auto-encoder.

4. VAE is a well-known method for finding the mapping of low-dimensional space to the normal distribution. Is there any relation to this? Also, I would like to know if there are any advantages of the proposed method over VAE.

5. In relation to the above, it may be good to have a comparison experiment with denoising auto-encoder and VAE.

6. The lack of experimental results with actual data makes that the paper is unconvincing. Especially for the description of scalability, it is better to prove it experimentally.

7. This is a pure question, is it possible to know the dimensions of the manifold through this technique?
review_title: Interesting but not enough comparison with related works

final_score: rating: 7: Good paper, accept
review_content: In this paper, the authors address main limitations of Normalizing Flows (NFs) method for estimation of density functions on manifolds. Since NFs requires the support of density function to cover the whole Euclidean space, they propose to add noise (inflate) to apply NF. The paper is nicely written with clear introduction of basic concepts very useful for non-expert readers. They provide theoretical guarantees on the variance and type of added noise that make the method work and illustrate with synthetic experiments.
My only concern is about the applicability and usefulness on real data and specific machine learning. I have found some issues and also have some questions. See below:
Major issues:
-	How this method can be used for a real problem with real datasets. Could you please provide some example or give some guidance about the potential of this method for the machine learning community?
-	It is shown that when D>>d, Gaussian noise is an excellent approximation of noise restricted in the normal direction and the experiments seem to confirm this. What should be the criterion to say that D is large enough to make this approximation useful? Can you give some way to test it in practice?
-	Example 2: a graphical description of this example would help, for example, showing that variance of noise can help to make the Q-normally separated condition to be met.
-	Page 5: I didn’t understand the following sentence “A potential future avenue is to \sigma^2A and this reach number”. Could you please elaborate it?

Minor issues
-	In Example 1: “an z” -> “a z”
-	References have missing information. For example, for Cornish et al, Gemici et al and Rezende et al papers, there is not information about the source. Did they works published in conferences, journals or ArXiV?
-	Page 12: Please correct the number of table in “Table ??” 

review_title: Nice theoretical work

Meta-review: The paper provides an interesting set of theoretical ideas to improve the estimation of normalizing flows on datasets that fail to be fully dimensional. Although the method is appealing, I believe the paper falls a bit short of acceptance at the conference. Too many practical issues are left out, as discussed by reviewers, and the method seems promising but not fully connected to the rest of the literature on estimating low-dimensional distributions living in high dimensional spaces. We encourage the authors to use the feedback contained in this round of reviews to improve their work.
Decision: Reject