Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes an alternative way to train a continuous normalizing flow (CNF), inspired by the Dacorogna-Moser transport. The proposed training does not require numerical integration; instead, the likelihood of the flow can be directly evaluated by computing the divergence of the mass flow rate. The idea is tested on boundaryless manifolds, such as the flat torus for toy data and implicit surfaces. 
main_review: 
*Reason for rating*: I think this work is quite novel and neat and complements the current flow literature well so I vote for acceptance. I would be happy to increase my rating if the author can address some of my concerns about scalability and the limitations that I mentioned below. 


### Significance and novelty

The proposed method is very neat, and largely simplifies the computational cost of CNF during training time. The likelihood of the model can be computed exactly in on a single forward evaluation (modulo the potential need of trace estimation), as opposed to numerically integrating the underlying ODE like FFJORD. The proposed flow is also provably universal (despite taking a simpler form, as per eq. 11, rather than having a free form velocity field). 

### Scope

The proposed method is mainly designed for modeling data on manifolds, which is empirically tested on a few toy and real datasets. More specifically, the manifolds need to be compact, boundaryless, and connected for the proposed method to work, which slightly limits its applicability and at the same time also leaves room for innovation. 

### Theoretical soundness

I only check for the main development in section 3 (so I cannot speak for section 4), which looks correct to me, except for the conditions in the statement of theorem 3. I am not quite sure about "For $\lambda\geq1$ and small $\epsilon>0$". Does the choice of $\epsilon$ depend on $\mu$ or $\lambda$? Or does it need to be asymptotically small? Also, why does $\lambda$ need to be greater than 1 if it's only a regularizer? It might help a bit to discuss the necessity of these conditions.

 
### Limitation

- (Regularization vs reparameterization) Most normalizing flows induces a valid probability density by design (such as regular CNFs which leverage the Lipschitzness of regular neural network architecture, residual flows that reparameterize the weights by spectral normalization, autoregressive and coupling blocks with masks over weight parameters, etc). However, the Moser flow needs to regularize the "negative part" of the density. It has the following disadvantages.
    1. Moser flow is not guaranteed to induce a valid probability density.
    2. It requires tuning an additional hyperparameter for regularization, $\lambda$. Is it a potential cause for instability during training? I actually think some negative results on this might help people better understand the work and think about how to improve upon it. How is it chosen as a hyperparameter (i.e. based on what criteria)? The chosen values (2 and 100) seem a bit drastic, with no discussion provided.
    3. The regularization seems to work at least empirically across the experiments conducted in this paper, but I suspect it might not scale well to higher dimensional problems, especially if $\eta$ is uniform. 

- Section 5.2: It's obvious that the proposed method is faster to train than FFJORD since it does not require integration (which the authors show in Fig 3). But given that the proposed flow has a simple time conditioning mechanism, as opposed to the free form conditioning that's allowed when using FFJORD, I think it'll be fairer and more meaningful to also compare the increase in likelihood in runtime and/or in #iterations. Also, since FFJORD in that experiment only uses a time-independent vector field, the expressivity of the baseline is also weakened.

### Additional questions

- Is the trace operator computed exactly or estimated? I imagine it is done exactly given the low dimensionality of the problems, but it might be better to mention the potential problem of scaling up and the need for approximation.
- Is it possible to generalize the flat torus for 2D toy data to accommodate higher dimensional problems lying in the Euclidean space? such as dequantized image data that live in a [0, 1]^D hypercube.
- typo? line 83 "from" instead of "form"?




limitations_and_societal_impact: See above
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This paper presents Moser Flow a new class of Continuous Normalizing Flows for Riemannian Manifolds. The main approach exploits results in differential geometry (Moser 1965) to construct a vector field that satisfies the normalizing equation given by the pullback of the volume form under $\Phi$. The approach is both sensible and provides direct advantages over Riemmanian CNF's as training does not require solving the ODE using a solver. The authors also consider Moser Flows for Euclidean Submanifolds under the Induced metric and show that Moser Flows can learn arbitrary distributions on manifolds of interest (---i.e. compact, boundryless etc...). Finally, the experiments conducted are both interesting in their variety and illuminating in that they show the effectiveness of the proposed method. Overall, this paper represents a non-trivial step forward---manifold or otherwise---for Continuous Normalizing Flow research.
main_review: Overall, I really enjoyed reading this paper and I think for the most part it is exceptionally well written. The main ideas are clear but maybe the authors could do a bit more to contrast the difference between Riemannian CNF by Mathieu and Nickel 2020. I would also like to highlight, what I believe are the principal strengths of the approach. First, unlike conventional Riemannian CNF's----and CNF's more broadly---one does not need to solve the ODE using off the shelf solvers that may introduce additional sources of error. This i believe is a significant win. To this end I think the paper would be strengthened by considering similar datasets/experiment pipelines of Euclidean CNF's (beyond 2D toy data and Figure 3) to see if Moser Flows can handle conventional non-manifold specific generative modellng datasets which are significantly larger, both in dimensionality and size, than the ones considered here. 

In terms of key weaknesses, I don't find that this paper has any major drawbacks. One could criticize that it leans too heavily on the original Moser's results (e.g. even the interpolant is borrowed) but I think this is folly. The adaptation of the method to Riemannian CNF's brings about a lot of technical novelty which should not be understated. In terms of experiments, I wish the authors also compared with concurrent NF on manifold baselines like Neural Ordinary Manifold Differential Equations by Lou et. al 2020 and Falorsi, L. and Forré 2020. These methods have subtle but important differences than Riemannian CNF by Mathieu and Nickel 2020. In terms of presentation, the supplementary material could be strengthened. Right now there is no explicit/formal proof of Theorem 2 and 3. I do concede that the main arguments for Theorem 3 are present in the main text, but this is not a formal proof. 

Questions/Minor Comments:
- The divergence computation in Riemannian CNF's relied on the Hutchinson Trace Estimator, was this also used for Moser Flow?
- This maybe my misunderstanding, but I feel like there is a missing step in the proof of Lemma 2 as the final equality in RHS of the equation below line 31 does not equal the first equality of the line below. Unless, $\sum_{j=1}^k \langle \frac{\partial u}{\partial n_i}, n_i \rangle$ (Note that there is a typo with subscript $i$ in the summation over $k$,it should be over $j$) is $0$.


 
limitations_and_societal_impact: N/A
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: The authors define a continuous normalizing flow on an arbitrary Riemannian manifold through the divergence of a neural network using the Moser theorem. They motivate their approach and show how this flow can be represented and learned. By applying their method to a toy distributions and a real world earth and climate dataset, they demonstrate the effectiveness of the method and that they can outperform competing procedures. Furthermore, they show that their method is much faster and more efficient than FFJORD when learning a complex two dimensional density.
main_review: The authors use a very creative approach to define a continuous normalizing flow through the divergence of a neural network. Usually, they are constructed as the solution of an ODE and solving it can be time consuming. Here, the authors use the Moser theorem and use that the divergence of a vector field can be the difference of two distributions, i.e the base distribution and a distribution approximating the target. The power of the method is that it can directly operate on an arbitrary Riemannian manifold without the requirement of an invertible map from $\mathrm{R}^d$ to the manifold. So far, only a small number of methods exist which are able to do that. 
Implementing and learning the flow, however, requires some heuristic tricks and since I am not an expert on the Moser theorem and its implications in this context, I am not sure how valid they are.
Anyway, the method seems to be very efficient and outperforms competing approaches on small dimensional toy and real world datasets.
limitations_and_societal_impact: Currently, the method cannot be applied to high dimensional manifolds. Although this is unfortunate, competing methods struggle with this problem as well.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper tackles the problem of learning flow-based generative models. The proposed approach, a new instance of continuous normalizing-flow methods, models the density as a difference between the prior and a divergence term, parameterized by a neural network.

The claims are that:
* MF constitutes a universal density approximator,
* MF is more efficient to train than alternative CNFs; this is demonstrated empirically and the intuition is that MF does not require invoking or backpropagating through an ODE during training (note however that solving an ODE is required for sampling),
* MF improves upon alternative CNFs in terms of density estimation, sample quality, and training complexity,
* MF demonstrates for the first time the use of flow models for sampling from general curved surfaces.
main_review: Learning flow-based generative models is an active research area. The current paper designs a new model that aims to improve upon recently published continuous flow generative models (Riemannian CNFs, FFJORD). From this perspective, the approach is well-timed, theoretically motivated, and relevant to the community. I am not familiar with a similar approach, but note that I am not familiar with all the literature on continuous flow generative models.

*Experiments*:

My main concerns (questions) regard the empirical claims:
* Figure 2: The experiment is a great illustration of the proposed method, demonstrating that it can model complex (multi-modal, discontinuous) datasets. That said, prior methods, e.g. FFJORD, have also demonstrated that they can model these distributions (by warping an isotropic Gaussian), so this result is not specific to the current method.
* The claims made regarding Figure 3 are that 1) the proposed method is computationally more efficient than FFJORD and 2) MF captures high-frequency details better. Unfortunately, the claims are illustrated on a single image, without a numerical evaluation. Have you considered performing this evaluation across several images (e.g. the MNIST, Omniglot datasets) and reporting aggregated results (similar to the FFJORD paper)?
* Figure 5 illustrates the approach on the Stanford Bunny surface. Unfortunately, it’s hard to evaluate the difficulty of the problem given no baseline approaches. Since this is a major claim in the abstract, could you discuss why none of the previous approaches can be applied to this setting for comparison?
* Table 1: The most significant result is that the proposed approach outperforms prior methods from “Riemannian Continuous Normalizing Flows”. Since the prior results are extracted from the cited paper, and the methods there chosen to have approximately the same number of parameters, I think the results would be stronger if the authors discussed whether there is a significant difference in parameter size.
limitations_and_societal_impact: The authors discuss the potential negative societal impact in the Discussion and limitations section.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper introduces a new type of continuous normalizing flow (CNF) for manifold-valued data. Unlike in previous works, this Moser Flow (MF) models a density as a base density minus the divergence of a neural network. The authors prove that under certain assumptions this defines a probability distribution over the manifold. This approach has a different trade-off between inference and generation from other approaches: while the model density can be evaluated very cheaply, sampling from this model requires an ODE solver. The approach is qualitatively demonstrated on toy datasets on tori and the Stanford bunny, and quantitatively evaluated on a few datasets on 2-spheres.
main_review: Modeling probability densities on manifolds is an important problem and Moser Flows present a new approach to solving it. Compared to existing methods, the MF setup presents a substantially different trade-off between the efficiency of likelihood evaluations and the efficiency of data generation. They may be thus be very relevant, in particular in applications where likelihood evaluations are more important than sampling.

Maybe the most important contributions of the paper are proposing this method, proving that it defines a probability distribution over the manifold, and proving that it presents a universal density estimator. These theoretical aspects seem correct to me, though I have not checked all the proofs. In some parts (for instance the proof of theorem 3) I would have appreciated more details and sometimes more precision, perhaps by providing an extended version of the argument in the supplementary material. 

My main gripe with the method part of the paper is a certain lack of discussion. In particular:
1. What happens if the trained model still has $\bar{\mu}(x) < 0$ somewhere? Can the model still be useful to define a probability density? What are the limitations of the proposed regularization scheme (for instance when going to higher dimensions)? 
2. How do Moser Flows scale? The authors briefly allude to the poor scaling in one sentence in the discussion, but this really deserves more space. Would the approach for instance work well in low-dimensional manifolds embedded in a high-dimensional space (like the hypothesized image manifold)?
3. A more in-depth comparison of MF to established flows on manifolds would really improve this paper.
4. Some part of the paper focuses on general Riemannian manifolds, while later the paper focuses on the case of submanifolds of a Euclidean space. How useful are Moser flows in the case when manifolds are not defined as submanifolds of some Euclidean space?
5. What if the manifold is not known a priori but you only have access to samples from it? Is there a way to extend this method such that an atlas for the manifold is learned as well? 
I understand that space is at a premium at NeurIPS, but perhaps the author can make room to discuss some of these aspects by e.g. moving the proof of Thm. 3 into the appendix?

In the empirical part of the paper, the authors demonstrate that the idea works, at least in low-dimensional examples. The Stanford bunny results are cool to see. The quantitative evaluation is pretty thin, more metrics and comparisons to other baselines would have been better. I would also have liked to see an empirical verification that the model densities form a properly normalized probability density over the manifolds / that $\bar{\mu}(x)$ is positive everywhere. It is nice to see the speed-up in inference time in action, but it would have been even better to also include results on the sampling speed.

The paper is well written and clearly structured. Perhaps the presentation could have been further improved by hiding away more technical details in the supplementary materials and using the space for more detailed discussions and explanations (see above). Unlike many ML papers, it has properly sized labels in the figures (except, inexplicably, for the bottom right panel in Fig 3). The extensive supplementary materials and the provided code (clean, but not very documented) are good.

All in all, the authors present a novel solution to an interesting, relevant problem. The theory part of the paper is solid. While the empirical demonstration is thin, I do think that this paper has a lot of merit even without the usual benchmark table with N baselines in which the "Ours" entry has all the bold numbers. However, I do think that it currently lacks in-depth discussion of the properties of Moser Flows relative to previous approaches as well as on scaling it to higher-dimensional problems. With an extended discussion, I think this could make a good NeurIPS paper.

Minor comments and typos:
- Eq. (5): should this be a capital $\Phi$?
- Sec. 3.2: would be nice to explicitly describe sampling from the MF here somewhere
- line 140: "Stokes' theorem" with apostrophe, I believe
- around Eq. 14: more discussion of the choice of $\lambda$ would be great
- line 175: typo: superfluous "is"
- line 182: is the $x \in \mathcal{M}$ condition necessary here?
- Fig. 2: it would be great if the colour maps were scaled such that the empirical densities from the samples (based on histograms) and the density plot match
- Thm. 4: the statement "we conclude this section by proving" sounds a bit weird because there is no proof here (it is in the supplementary materials)
- line 220/221 read weird, typo?
- Fig. 3: how do FFJORD and MF compare in terms of log likelihood?
- Fig. 3: is this not a weird example to put on a torus b/c of the discontinuity at the identified edges of the images?
- Sec. 5.4: typo in the title, should be "Curved surfaces"
- Sec. 6: how does MF compare to these methods? This was superficially discussed before, but it would be great to re-visit in more detail here

limitations_and_societal_impact: I agree with the author's assessment that this work shares the same fundamental risks of all generative models. I do think that for the scope of this paper this acknowledgement is enough.
review_title: null

Meta-review: The following is a summary of the pros and cons that resulted from the reviews and the discussion period.

pros:
* novel and nontrivial contributions to the flow literature. (R6hn, oTmD, STbS, VH2c, 96Cx)
   - reduces computational cost to train CNFs by no longer needing to solve an ODE. 
   - likelihood computation does not require evaluating an ODE, constrast to regular CNFs.
   - Probably universal density estimator under suitable assumptions.
* interesting variety of experiments that show the effectiveness of the proposed method. (oTmD)
* for the most part well written (oTmD, 96Cx)

cons:
* limitations: 
    - scalability limitations to higher dimensional problems. --> authors respond that higher dimensions are indeed challenging and promis to expand on the precise challenges in an updated version of the paper. (R6hn, 96Cx)
    - regularization of the negative part of the density in moser flows introduces an additional hyperparameter for tuning. worries that the regularization parameter might lead to unstable training of chosen wrongly. --> authors added experiments to show the behavior of the model under different ranges of this hyperparameter. (R6hn)
   - can the negative density also lead to moser flow not producing a valid density? --> authors rebut this in their response. (R6hn, 96Cx)
* confusion around theorem 3, exact or approxiamte computation of traces etc. --> solved by author response.(R6hn)
* comparison with other normalizing flow methods on manifolds would strengthen the paper. --> addressed in discussion. (oTmD, 96Cx)
* introducing an explicit proof of theorem 2 and 3 would improve the paper (arguments for proof can now be found in main paper). --> addressed in discussion. (oTmD, 96Cx)
* Clarifications needed about applicability of method. Can Moser flows still be used for manifolds that are not submanifolds of a Euclidian space? What if the manifold is not known?  --> discussed in author response. (96Cx)
* improvements needed in terms of evaluation: 
   - figure 2: prior methods can also model this distribution, so the success of the proposed method over previous methods is unclear. (VH2c)
   - quantitative evaluation would make the claims stronger of being more computationally efficient, capturing densities better (e.g. Fig 3). --> addressed in author response.  (VH2c, 96Cx)
   -  comparison is needed with FFJORD in terms of both likelihood and runtime/#iterations. --> the authors include a link in their response with these results. (R6hn)
   - figure 5: why can't previous approaches be used to model this problem? --> addressed in author response. (VH2c)
   - table 1: how do the number of parameters of this method compare to previous work? --> addressed in author response. (VH2c) 

Authors and reviewers engaged into active discussions during the discussion period. Most of the concerns raised by the reviewers were addressed, leading to 3/5 reviewers raising their scores and a consensus to accept this submission. Considering the nontrivial contribution of this work and the future research it could lead to, the recommended decision is accept.
Decision: Accept (Oral)