Moser Flow: Divergence-based Generative Modeling on Manifolds
Authors: Noam Rozen
Aditya Grover
Maximilian Nickel
Yaron Lipman
Venue: NeurIPS 2021 Oral (2021)
Keywords: generative models
manifolds
normalizing flows

Abstract: We are interested in learning generative models for complex geometries described via manifolds, such as spheres, tori, and other implicit surfaces. 
Current extensions of existing (Euclidean) generative models are restricted to specific geometries and typically suffer from high computational costs. We introduce Moser Flow (MF), a new class of generative models within the family of continuous normalizing flows (CNF). MF also produces a CNF via a solution to the change-of-variable formula, however differently from other CNF methods, its model (learned) density is parameterized as the source (prior) density minus the divergence of a neural network (NN). The divergence is a local, linear differential operator, easy to approximate and calculate on manifolds. Therefore, 
unlike other CNFs, MF does not require invoking or backpropagating through an ODE solver during training. Furthermore, representing the model density explicitly as the divergence of a NN rather than as a solution of an ODE facilitates learning high fidelity densities. Theoretically, we prove that MF constitutes a universal density approximator under suitable assumptions. Empirically, we demonstrate for the first time the use of flow models for sampling from general curved surfaces and achieve significant improvements in density estimation, sample quality, and training complexity over existing CNFs on challenging synthetic geometries and real-world benchmarks from the earth and climate sciences. 

Moser Flow: Divergence-based Generative Modeling on Manifolds
Noam Rozen'
Aditya Grover
Maximilian Nickel?
Yaron Lipman" 1.2 *CLA
Weizmann Institute of Science
Facebook Al Research
Abstract
We are interested in learning generative models for complex geometries described via manifolds, such spheres, tori, and other implicit surfaces. Current extensions of existing (Euclidean) generative models are restricted to specific geometries and typically suffer from high computational costs. We introduce Moser Flow (MF), a new class of generative models within the family of continuous normalizing flows (CNF). MF also produces CNF via solution to the change-of-variible formula, however differently from other CNF methods, its model (learned) density is parameterized as the source (prior) density minus the divergence of a neural network (NN). The divergence is a local, linear differential operator, easy to approximate and calculate on manifolds. Therefore, unlike other CNFs, MF does not require invoking or backpropagating through an ODE solver during training. Furthermore, representing the model density explicitly as the divergence of NN rather than as solution of an ODE facilitates learning high fidelity densities. Theoretically, we prove that MF constitutes a universal density approximator under suitable assumptions. Empirically, we demonstrate for the first time the use of flow models for sampling from general curved surfaces and achieve significant improvements in density estimation, sample quality, and training complexity over existing CNFs on challenging synthetic geometries and real-world benchmarks from the earth and climate sciences.
Introduction
The major successes of deep generative models in recent years are primarily in domains involving Euclidean data, such as images (Dhariwal and Nichol (021), text (Brown et al. 2020), and video (Ku- mar et al. 2019). However, many kinds of scientific data in the real world lie in non-Euclidean spaces specified as manifolds. Examples include planetary-senn scale data for earth and climate sciences (Mathieu
and Nickel 2020), protein interactions and brain imaging data for life sciences (Gerber al.| 2010 Chen et al. 2012), as well as 3D shapes in computer graphics (Hoppe et al.] 1992; Kazhdan et al. 2006). Existing (Euclidean) generative models cannot be effectively applied in these scenarios as they would tend to assign some probability mass to areas outside the natural geometry of these domains.
An effective way to impose geometric domain constraints for deep generative modeling is to design normalizing flows that operate in the desired manifold space. A normalizing flow maps a prior (source) distribution to target distribution via the change of variables formula (Rezende and Mohamed 2015; Dinh et al., 2016) Papamakarios et al. 2019). Early work in this direction proposed invertible architectures for learning probability distributions directly
the specific manifolds defined over spheres and tori (Rezende et al. (2020). Recently, Mathieu and Nickel (2020) proposed to extend continuous normalizing flows (CNF) (Cen et al.) (018)) for generative modeling over Riemannian manifolds wherein the flows are defined via vector fields on manifolss and computed as the solution to associated ordinary differential equation (ODE). CNFs have the advantage that the neural
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

network architectures parameterizing the flow need not be restricted via invertibility constraints. However, as we show our experiments, existing CNFs such as FFJORD (Grathwohl et al. 2018) and Riemannian CNFs (Mathieu and Nickel 2020) can be slow to converge and the generated samples can be inferior in capturing the details of high fidelity data densities. Moreover, it a real challenge to apply Riemannian CNFs to complex geometries such as general curved surfaces.
To address these challenges, we propose Moser Flows (MF), a new class of deep generative models within the CNF family. An MF models the desired target density as the source density minus the divergence of an (unrestricted) neural network. The divergence operator, easy to approximate and calculate on manifolds. By drawing on classic results in differential geometry by Moser (1655) and Dacorogna and Moser (1990), we can show that this parameterization induces a CNF solution to the Dhange-of-variable parameterize the model density using the divergence, unlike other CNF methods, we do not require to explicitly solve the ODE for maximum likelihond training. At test-time, we use the ODE solver for generation. We derive extensions to MFs for Euclidean submanifolds that efficiently parameterize vector fields projected to the desired manifold domain. Theoretically, we prove that Moser Flows are universal generative models over Euclidean submanifolds. That is, given a Euclidean submanifold M and target continuous positíve probability density over M,, MFs can push arbitrary positive source density / over M to densities j that are arbitrarily close to p.
local, linear differential
formula specified via an ODE, Since MFs directly
We evaluate Moser Flows on wide range of challenging real and synthetic problems defined over many different domains. On synthetic problems, we demonstrate improvements in convergence speed for attaining a desired level of details in generation quality. We then experiment with two kinds of complex geometries. First, we show significant improvements of 49% on average over Riemannian CNFs (Mathieu and Nickel 2020) for density estimation as well as high-fidelity generation on 4 earth and climate science datasets corresponding to global locations of volcano eruptions, earthquakes, floods, and wildfires spherical geometries. Next and last, we go beyond spherical geometries to demonstrate for the first time, generative models on general curved surfaces.
2 Preliminaries
Riemannian manifolds. We consider orientable, compact, boundaryless, connected dimensional Riemannian manifold M with metric g. We denote points in the manifold by x, E M. At every point E .,, T&M is an n-dimensional tangent plane to M. The metric g prescribes an inner product on each tangent space: for v, u E TrM, their inner product w.r.t. g is denoted by (v, g) X(M) is the space of smooth (tangent) vector fields to M; that is. if u E 3(M) then u(x)= E T,M, for all
M, and if written in local coordinates it consists of smooth functions. We denote by dV the Riemannian volume form, defined by the metric g over the manifold M. In particular, ((A) = - dV is the volume of the set A C ...
We consider probability measures over M that are represented by strictly positive continuous density functions μ1 .M- + R,,, where μ by convention represents the target (unknown) distribution and v represents the source (prior) distribution. μu, v are probability densities in the sense their integral w.r.t. the Riemannian volume form is one, i.e., IM pdV 1 = SM vdV. It is convenient to consider the volume forms that correspond to the probability measures, namely j = μdV and & = vdV. Volume forms are differential n-forms that can be integrated over subdomains of M, for example, (A) = SA P is the probability of the event A C M.
Continuous Normalizing Flows (CNF) manifolds operate by transforming a simple source distribution through a map & into a highly complex and multimodal target distribution. A manifold CNF, Φ MM- + M,, is orientation preserving diffeomorphism from the manifold to itself (Mathieu
and Nickel 2020 Lou et al. 2020 Falorsi and Forré 2020). A smooth map & : .M + M can be used to pull-back the target j according to the formula:
( (1) va) = (lars)) (D
(vI)..... (D.)
(v,)),)),
M is the differential where Ul,..., Un E T are arbitrary tangent vectors, DΦ; T.M + T of T namely a linear map between the tangent spaces to M at the points a and (22), respectively. By pulling-bace back j according to & and asking it to equal to the prior density v, get the manifold version of the standard normalizing equation:
D = 4' f..
(2)
2

---

If the normalizing equation holds, then for an event A C M we have that
(AA) - / 1- / 1 / A((A) f (==p((((A)
Therefore, given a random variable 2 distributed according to v, then a = ((2) is distributed according jr, and Φ is the generator.
One way to construct a CNF & is by solving an ordinary differential equation (ODE) (Chen et al. 2018 Mathieu and Nickel 2020). Given time-dependent vector field E x(M) with f E [0, 1], one-parameter lamily of diffeomorhissms (CNFs) 2, [0,1] x M + M is defined by
d Φb 02 (.t), dt
(3)
where this ODE is initialized with the identity transformation, i.e., for all x M we initialize (o(x) = x. The CNF then defined by ((x) - 4 (x)).
Example: Euclidean CNF. Let us show how the above notions boil down to standard Euclidean CNF for the choice of M R", and the standard Euclidean metric; we denote (2) ..... 2²) E R".
The Riemannian volume form in this case is dz = dz' Adz² --d2d Furthermore, fi(=) = u
and (z) (2)dz. The pull-back formula (equation) coordinates (see e.g.. Proposition 14.20
in Lee 2013 is
t' (²)) (=)(=)) del(DD
where D is the matrix of partials of & point z, (D+= Oz we get the Euclidean normalizing equation:
ao'
(z). Plugging this in equation
rxz) ((((2)) 2
(4)
3 Moser Flow
Moser 1965) and Dacorogna and Moser (1990) suggested method for solving the normalizing
equation, that is equation 2 Their method explicitly constructs a vector field vt, and the flow defines via equation 3 is guaranteed to solve equation 2 We start by introducing the method, adapted to our needs, followed by its application to generative modeling. We will use notations introduced
above.
3.1 Solving the normalizing equation
Moser's approach to solving equation 2 starts by interpolating the source and target distributions. That is, choosing an interpolant ae [0, 1 M + Rz0. such that agg = v, a1 μ4, and JM a,dV for all t E [0, 1]. Then, time-dependent vector field et E x(M) is defined so that for each time t [0, 1]. the flow T; defined by equation satisfies the continuous normalization equation:
4 b âo.
(5)
where a; a,dV is the volume form corresponding the density da. Clearly, plugging ( 1 in the above equation provides a solution to equation @ with + = Φ As turns out, considering the continuous normalization equation simplifies matters and the sought after vector field v, is constructed as follows. First, solve the partial differential equation (PDE) over the manifold M
div(ug = di
(6)
where u, E X(M) is an unknown time-dependent vector field, and div is the Riemannian generaliza- tion to the Euclidean divergence operator, divg V. This manifold divergence operator is defindd by replacing the directional derivative of the Euclidean space with its Riemannian version, namely the covariant derivative,
div(u) =
Σ (V. u, Eig
3

---

where {e is an orthonormal frame according to the Riemannian metric and Vgu is the Riemannian covariant derivative. Note that here we assume that M is boundaryless, otherwise we need u, to be also tangent to the boundary of M. Second, define
M4
Ut
(8)
Theorem in Moser
(965) implies:
Theorem (Moser). The diffeomorphism & D1, defined by the ODE in equation and vector field Ur in equation & solves the normalization equation, i.e., equation? 2
1
The proof of this theorem in our case is provided in the supplementary for completeness. A simple choice for the interpolant Ox that we use in this paper was suggested Dacorogna and Moser [1900]:
(1- -1-tj t)R tj..
(9)
d The time derivative of this interpolant, i.c., 7 at μ does not depend on t. Therefore the vector field can be chosen to be constant over time, it u. and the PDE in equation 6 takes the form
V
time
div(u)
p.
(10)
and consequently e, takes the form
Ut
U/ (1 t)v + ty
(11) Figure 1: ID example of Moser Flow: source density y in green, target μ in blue. The vector field o (black) is guaranteed to push to interpolated density a at time t, i... (1 (1-t)u+ ty..
Figure 1 shows one dimensional illustration of Moser Flow.
3.2 Generative model utilizing Moser Flow
We next utilize MF to define our generative model. Our model (learned) density f is motivated from equation 10 and is defined by
j === v - div(u),
(12)
where u E (MM is the degree of freedom of the model. We model this degree of freedom, u, with deep neural network, more specifically, Multi-Layer Perceptron (MLP). We denote 0 R the learnable parameters of u. We start by noting that, by construction, j has a unit integral over M:
Lemma 1. If M has no boundary, or ulam E I(0M), then JM jdV =.
This lemma is proved in the supplementary and direct consequence of Stokes' Theorem. If p O over M then, together with the fact that I'M .dV = 1 (Lemma 1), is a probability density over M. Consequently, Theorem[[implic that j is realized by CNF defined via Uy:
Corollary 1. ī 0 over M then is a probability distribution over M, and is generated by the flow &- 41, where 4. the solution to the ODE in equation with the vector field ot E (MM) defined in equation 71
Since j > 0 is an open constraint and is not directly implementable, we replace it with the closed constraint j 2 e, where > is small hyper-parameter. We define
1 (x) max {e, p(x);; a_ (x) min {e, ((x))
As can be readily verified:
P4. 0, and p p4 PT_.
(13)
We ready to formulate the loss for training the generative model. Consider unknown target distribution μ. provided to us as a set of i.i.d. observations X' = { C M. Our goal is to maximize the likelihood of the data X while making sure f 2 e. We therefore consider the following loss:
((0) = EE, log p++ (x) + 2 / p- (x)dV
(14)
4

---

where À is a papee-parameter. The first term in the loss is approximated by the empirical mean computed with the observations X. i.e.
Ex log (+)x)
Σ log p+ (zz).
This term is merely the negative log likelihood of the observations.
The second term in the loss penalizes the deviation of ji from satisfying ji e. According to Theorem I this measures the deviation of ji from being density function and realizing a CNF. One point that needs be verified is that the combination of these two terms does not push the minimum away from the target density This can be verified with the help of the generalized Kullback-Leibler (KL) divergence providing a distance measure between arbitrary positive functions f.g M Roo:
D(S.)) -mm flog dV
S.x gdV.
sav
(15)
Using the generalized KL,, we compute the distance between the positive part of our model density, i.e., f4, and the target density:
( O-JJA
11
J.n udV + + P+dV
((,, = Eu log
f++
E. log Ep log 14+ + S/v p_
where in the second equality used Lemma 1 The term Ep log μ is the negative entropy of the unknown target distribution The loss in equation 14 equuls D(1, (++) E2 log + (A 1) J' dV. Therefore, if À > 1, and minze.M [uxx) existence of such minimal positive value), then the unique minimum of the loss in equation 14]is the target density, i.e., J = μr. Indeed, the minimal value of this loss is -IE, log μ and is achieved by setting jı = μr. Uniqueness follows by considering an arbitrary minimizer jt. Since such a minimizer satisfies (1, p+) 0 and SM PV 0, necessarily j μ. We proved:
(we use the compactness of M to infer
Theorem 2. For x 2 1 and sufficiently small 0, the unique minimizer of the loss in equation 14 is p=
Variation of the loss. Lemma 1 and equation 13 imply that SM P. F- Therefore, an equivalent loss to the one presented in equation 14
(()) E, log F+ (x) + X_ S pd + d4 I Pv
(16)
with A_ + 2 1. Empirically found this loss favorable in some cases (i.e., with x 0).
Integral approximation. The integral JM PV in the losses in equation and 14 is approxi- mated by considering set y {リ)) of i.i.d. samples according some distribution " over M and taking Monte Carlo estimate
JN pv iー (w:) 71;))
(Ui)
(17)
SM dV is approximated similarly. In this paper opted for the simple choice of taking 7) be the uniform distribution over M.
4 Generative modeling over Euclidean submanifolds
In this section, we adapt the Moser Flow (MF) generative model to submanifolds of Euclidean spaces. That is we consider orientable, compact, boundaryless, connected n-dimensional submanifold M RRd where d. Examples include implicit surfaces and manifolds (i.e., preimage of a regular value of smooth function), well as triangulated surfaces and manifold simplicial complexes. We
5

---

denote points in Id (and therefore in M) with a, E R As the Riemannian metric of M we take the induced metric from R", that is given arbitrary tangent vectors v, u M. the metric is defined by (v, 14,, g = (t, u), where the latter is the Euclidean inner product. We denote by π IR .M the
closest point projection on M, i.e., (2)) = minyeM || yll, with |lyll = (y, y) the Euclidean norm in IR.. Lastly, we denote by PR E Rdxd the orthogonal projection matrix on the tangent space M in practice if we denote by N E Rd the matrix with orthonormal columns spanning NaM = (T,M)) (i.e., the normal space to .M at x)) then, Px = - NN
We parametrize the vector field u required for our MF model (in equation 12) by defining vector field u E (RR)) such that u M E ( We define
u(z)= Pa (=(()))
(18)
where vo Rd + Rd is MLP with Softplus activation (8 = 100) and learnable parameters 0 RP. By construction, for x E M, u(x) E M.
To realize the generative model, we need to compute the divergence div(u for x .M with respect to the Riemannian manifold M and metric g. The vector field u in equation [8] is constant
along normal directions to the manifold at (since x(x) is constant in normal directions). If
n E M, then in particular
d u( + (n) 0.
(19)
d=
We call vector fields that satisfy equation 19 iffiitessimll constant in the normal direction. As we show next, such vector fields u e X(M) have the useful property that their divergence along the manifold M coincides with their Euclidean divergence in the ambient space Rd:
Lemma 2. Ifu E X(RR)), ulM E x(M) is infinitesimally constant in normal directions of M, then for a E M, div(u(x)) = div e uu (x)), where div g denotes the standard Euclidean divergence.
This lemma simply means we can compute the Euclidean divergence of u in our implementation. Given a set of observed data X' {= C M C Rdd and a set of uniform i.i.d. samples y = {y, 1 C M over M, our loss (equation 14) takes the form
X Σ (e j=1
((0)- m Σ log max (e, v(x;) - div)u( +
min {e, v(y,) - div divruu)))))
where X_ = A_ (MM). We note the volume constant can be ignored by considering an un-normalized
source density V ((M)² 1, see supplementary for details. The loss in equation 16 is implemented similarly, namely, we add the empirical approximation of X JM
We conclude this section by stating that the MF generative model over Euclidean submanifolds (defined with equations 12 and 18) is universal. That is, MFs can generate, arbitrarily well, any continuous target density μ on a submanifold manifold M C CRd.
Theorem 3. Given an orientable, compact, boundaryless, connected, differentiable n-dimensional submanifold M C Rd. n < and target continuous probability density p. : M - Ezo. there exists for each >> 0 an MLP and a choice of weights O so that jı defined by equations and 18 satisfies
max |p(x) -[(x)) <e zEM
5 Experiments
In all experiments, we modeled a manifold vector field as a multi-layer perceptron (MLP) Uo E X(M), with parameters 0. All models were trained using Adam optimizer (Kingma and Ba 2014). and in all neural networks the activation is Softplus with 3 = 100. Unless stated otherwise, we set X O. We used an exact calculation of the divergence duv)( We experimented with two kinds of manifolds.
Flat Torus. To test our method on Euclidean 2D data, we used M as the flat torus, that is the unit square [-1, with opposite edges identified. This defines a manifold with no boundary which lo- cally isometric to the Euclidean plane. Due to this local isometry the Riemannian divergence on the flat

---

input data
samples
density
input data
samples
density
Figure 2: Moser Flow trained on 2D datasets. We show generated samples and learned density P...
torus is equivalent to the Euclidean divergence, div divg. To make us well defined smooth vector field in M we use periodic positional encoding, namely ug (x) = vo(T(x)), where Wy is a standard MLP and R²2 R44k is defined as 7(x) = (cos(w)]rz)) ,, sin (w) πæ) ..., cos(w, wa:), sin mi), i, and & is a hyper-paramete that is application dependent. Since any periodic function can be approximated by polynomial acting on е even for h = this a universal model for continuous functions the torus. As described by Tancik et al. (2020), adding extra features can help with learning higher frequencies in the data. To solve an ODE on the torus we simply solve for the periodic function and wrap the result back to 1|1,
where
Implicit surfaces. We experiment with surfaces as submanifolds of R3. We represent a surface as the zero level set of a Signed Distance Function (SDF) f RR3 - R. We experimented with two surfaces. First, the sphere, represented with the SDF f() ||> - and second, the Stanford Bunny surface, representing general curved surface and represented with an SDF learned with (Gropp al., (020)) from point cloud data. To model vector fields on an implicit surface follow the general equation
18 where for SDFs
( f(=), and P. I (fxx)
In the supplementary, detail how to replace the global projection TTx)) with local one, for cases the SDF is not exact.
5.1 Toy distributions
First, we consider collection of challenging toy 2D datasets explored in prior works (Chen et al. 2020; Huang al. 2021). We scale samples lie in the flat torus [-1, and use i: 1 for the positional encoding. Figure 2 depicts the imput data samples, the generated samples after training, and the learned distribution j.. the top six datasets, the MI.P architecture used for Moser Flows consists of 3 hidden layers with 256 units each, whereas in the bottom two we used 4 hidden layers with 256 neurons due to the higher complexity of these distributions. We set X_
5.2 Choice hyper-parameter
We test the effect of the hyper-parameter À > on the learned density. Figure shows, for different values of X, our density estimation p1+, the push-forward density $_v, and their absolute difference. To evaluate d.v from the vector field vr, we solve an ODE advised in Grathwohl et al. (0118.

---

L.V
p,V
D. v
P2.V p+
X=1
A= 2
A= 100
Figure 3: As À is increased, the closer #+ is to the generated density 2, column titled 7م - 4 v shows the absolute pointwise difference between the two: note that some of the errors in the 1p+ v column are due to numerical inaccuracies in the ODE solver used to calculate v.v.
density lk
samples lk
density 5k
samples 5k
negative log-like hood
Moser Flow
running
Reration time
FFJORD
running time
Figure Comparing learned density and generated samples with MF and FFIORD at different times (in k-sec); top right shows NLL scores for both MF and FFJORD at different times; bottom right shows time per iteration (in log-scale, sec) as function total running time (in sec); FFJORD iterations take longer training progresses; A second example of the same experiment on different image provided the supplementary. Flickr image (license CC BY 2.0): Bird by Flickr user "lakeworth" hhttps://vit flickr.... www.photos/laaaeeeeoooooooooaoao httphocce//aaeeorrr////677799999
As expected, higher values of À lead to closer modeled density A4 and P.v. This is due to the fact that a higher value of À leads to a lower value of JM f_, meaning zi is a better representation of probability density. Nonetheless, even for X 1 the learned and generated density are rather consistent.
5.3 Time evaluations
To compare our method to Euclidean CNF methods, we compare Moser Flow with FEJORD (Grath- wohl al. 2018) on the flat torus. We consider challenging density with high frequencies obtained via 800x600 images (Figure 4p. We generate a new batch every iteration by sampling each pixel location with probability which proportional to the pixel intensity. The architectures of both vs and the vector field defined in FFJORD are the same, namely an MLP with 4 hidden layers of 256 neurons each. To capture the higher frequencies in the image we use a positional encoding with k = 8 for both methods. We used a batch size of 10k. We used learning rate of le-5 for Moser Flow and le-4 for FFJORD. We used A_ the results. Note that Moser Flow captures -regufrreeency details better than FFJORD. This is also expressed in the graph on top right, showing how the NLL decreases faster MF than FFJORD. Furthermore, as can be inspected in the per iteration time graph on the bottom-right, MF per iteration time does not increase during training, and roughly 1-2 order of magnitudes faster than FFJORD iteration.
Learning was stopped after seconds. Figure 4ecure4presentt presents

---

Volcano
Earthquake
Flood
Fire
Figure 5: Moser Flow trained earth sciences data gathered by Mathieu and Nickel (2200). The learned density is colored green-blue (blue indicates larger values); Blue and red dots represent training and testing datapoints, respectively. See Table for matching quantitative results.
Volcano
Farthquake
Flood
Fire
Mixture vMF
-0.31 20.07
0.50
1.09
-0.23
Stereographic
20.64
0.43
0.99
-0.40
Riemannian
00.97 10.15
0.19
0.90
-0.66
Moser Flow (MF)
22.02
-0.09
0.62
-1.03
Data size
6124
4877
12810
Table 1: Negative log-likelihood scores of the earth sciences datasets.
5.4
Earth and climate science data
We evaluate model the earth and climate datasets gathered in Mathieu and Nickel (2020). The projection operator π in this case is simply π(xx) = T We parameterize dy as an MLP with 6 hidden layers of 512 neurons each. We used full batches for the NLL loss and batches of size 150k for our integral approximation. We trained for 30k epochs, with learning rate of le-4. We used A_ 100. The quantitative NLL results are reported in Table and qualitative visualizations in Note that we produce NLL scores smaller than the runner-up method by a large margin.
5.5 Curved surfaces
We trained an SDF f for the Stanford Bunny surface M using the method in Gropp et al. (020) To generate uniform (2/) and data (14) samples over M we first extract mesh M' from using the Marching Cubes algorithm (Lorensen and Cline 1987) setting its resolution to 100x 100x 100. Then, to randomly choose a point uniformly from M we first randomly choose face of the mesh with probability proportional to its area, and then randomly choose a point uniformly within that face. For target f we used clamped manifold harmonics to create a sequence of densities with increased complexity. To that end, we first computed the k-th eigenfunction of the Laplace-Beltramí operator over M' (we provide details on this computation in the supplementary), for the frequencies (eigenvalues) & E {10, 50, 500). Next, their negative values, and normalized to get discrete probability densities over the faces of M'. Then, to sample point, first choose a face at random based on this probability, and then random point uniformly within that face. We take 500k i.i.d. samples of this distribution as our dataset. We take Ug to be MLP with 6 hidden layers of dimension 512. We use batch size of 10k for both the NLL loss and for the integral approximation; we ran for 1000 epochs with learning rate of le-4. We used X_ X 1. Figure 3 depict the results. Note that Moser Flow able to learn the surface densities for all three frequencies.
sampled the eigenfunctions at the faces" centers, clamped
Related Work
In the following, we discuss related work on normalizing flows for manifold-valued data. On a high level, such methods can be divided into Parametric versus Riemannian methods. Parametric methods consist of normalizing flow in the Fuclidean space RR,, pushed-forward onto the manifold through an invertible map vi R. M. However, globally represent the manifold, v needs be homeomorphism implying that M and R are topologically equivalent, limiting the scope of that

---

Frequency A: = 10
Frequency k = 50
Frequency k: == 500
Figure 6: Moser Flow trained curved surface (Stanford Bunny). We show three different target distribution with increasing frequencies, where for each frequency we depict (clockwise from top-left): target density, data samples, generated samples, and learned density.
approach. Existing methods in this class are often based the exponential map expx TTM E R" - M of a manifold. This leads to the so called wrapped distributions. This approach has been taken, for instance, by Falorsi al. (2019) and Bose al. (2020) parametrize probability distributions on Lie groups and hyperbolic space. However, Parametric methods based on the exponential map often lead numerical and computational challenges. For instance, in compact manifolds (e.g., spheres or the SO(3) group) computing the density of wrapped distributions requires an infinite summation. On the hyperboloid, on the other hand, the exponential map numerically not well-behaved far away from the origin (Dooley and Wildberger 1993; Al-Mohy and Higham 2010).
In contrast Parametric methods, Riemannian methods operate directly the manifold itself and, as such, avoid numerical instabilities that arise from the mapping onto the manifold. Early work in this class of models proposed transformations along geodesics on the hypersphere by evaluating the exponential map at the gradient of a scalar manifold function (Seil [2011). Rezende et al. (2020)
introduced discrete Riemannian flows for hyperspheres and tori based on Möbius transformations
and spherical splines. Mathieu and Nickel (2020) introduced continuous flows on general Riemannian
manifolds (RCNF). In contrast to discrete flows (e.g., Bose al. 2020 Rezende al. 2020), such time-conting continuoss flows alleviate the previous topological constraints by parametrizing the flow as the solution to an ODE over the manifold (Grathwohl et al. (018). Concurrently to RCNF, Louu al.) (2020) and Falorsi and Forré (2020) proposed related extensions of neural ODEs to smooth manifolds. Moser Flow also generates a CNF, however by limiting the flow space (albeit, not the generated distributions) it allows expressing the learned distribution as the divergence of vector field.
7 Discussion and limitations
We introduced Moser Flow, generative model in the family of CNFs that represents the target density using the divergence operator applied to vector valued neural network. The main benefits of MF stems from the simplicity and locality of the divergence operator. MFs cireumvent the need to solve an ODE in the training process, and are thus applicable on broad class of manifolds. Theoretically. we prove MF is a universal generative model, able (approximately) generate arbitrary positive target densities from arbitrary positive prior densities. Empirically, we show MF enjoys favorable computational speed in comparison to previous CNF models, improves density estimation on spherical data compared to previous work by large margin, and for the first time facilitate training CNF over a general curved surface.
One important future work direction, and current limitation, is scaling of MF to higher dimensions. This challenge can be roughly broken three parts: First, the model probabilities j should be computed/approxiiatee in log-scale, as probabilities are expected to decrease exponentially with the dimension. Second, the variance of the numerical approximations of the integral [ dVV will increase significantly in high dimensions and needs to be controlled. Third, the divergence term, div (u), is too costly to be computed exactly in high dimensions and needs to be approximated, similarly to other CNF approaches. Finally, our work suggests novel generative model, and similarly to other generative models can potentially used for generation of fake data and amplify harmful biases in the dataset. Mitigating such harms is an active and important area of ongoing research.
10

---

Acknowledgments
NR is supported by the European Research Council (ERC Consolidator Grant, "LiftMatch" 771136),
the Israel Science Foundation (Grant No. 1830/17), and Carolito Stiftung (WAIC).
References
AI-Mohy, A. H. and Higham, N. J. (2010). A New Scaling and Squaring Algorithm for the Matrix Exponential. SIAM Journal on Matrix Analysis and Applications, 31(3):970-955
Bose, A. J., Smofsky, A., Liao, R., Panangaden, P., and Hamilton, W. L.. (2020). Latent Variable Modelling with Hyperbolic Normalizing Flows. arXiv:2002.06336 (cs, stat].
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P. Sastry, G., Askell, A., et al. (2020). Language models are few-shot learners. arXiv preprint 2XXv:2005.14165
Chen, M., Tu, B., and Lu, B. (2012). Triangulated manifold meshing method preserving molecular surface topology. Journal of Molecular Graphics and Modelling, 38:411-418.
Chen, T., Rubanova, Y., Bettencourt, J., and Duvenaud, D. (2018). Neural ordinary differential equations. arXiv preprint 180.0366.
Chen, R. T. Q.. Behrmann, J., Duvenaud, D., and Jacobsen, J.-H. (2020). Residual flows for invertible generative modeling.
Dacorogna, B. and Moser, J. (1990). On partial differential equation involving the jacobian determinant. Annales de l'Institut Henri Poincare (C) Non Linear Analysis, volume 7, pages 1-26. Elsevier.
Dhariwal, P. and Nichol, (2021). Diffusion models beat gans on image synthesis. arXiv preprint arXiv:2105.05233.
Dinh, L., Sohl-Dicksteni J., and Bengio, S. (2016). Density estimation using real nvp. arXiv preprint 1605::1605.
Dooley, A. and Wildberger, N. (1993). Harmonic analysis and the global exponential map for compact Lie groups Functional Analysis and Applications, 27(1):21-21
Falorsi, L., de Haan, P., Davidson, T. R., and Forré, P. (2019). Reparameterizing Distributions on Lie Groups. arXiv: /903.0298 [es, math, stat).
Falorsi, L. and Forré, P. (2020). Neural Ordinary Differential Equations on Manifolds. arXiv:2006.00 2266::0006.06663 [cs, stat].
Gerber, S., Tasdizen, T., Fletcher, P. T., Joshi, S., Whitaker, R., Initiative, A. D. N., et al. (2010). Manifold modeling for brain population analysis. Medical image analysis, 14(5):643 653.
Grathwohl, W., Chen, R. T. Q., Bettencourt, J., Sutskever, I., and Duvenaud, D. (2018). Ffjord: Free-form continuous dynamics for scalable reversible generative models.
Gropp. A., Yariy, L., Haim, N., Atzmon, M., and Lipman, Y. (2020). Implicit geometric regularization for learning shapes.
Hoppe, H., DeRose, T., Duchamp, T., McDonald, and Stuetzle, W. (1992). Surface reconstruction from unorganized points. In Proceedings of the 19th annual conference on computer graphics and interactive techniques, pages 71-78.
Huang, C.W., Chen, R. T. Q., Tsirigotis, C., and Courville, A. (2021). Convex potential flows:
Universal probability distributions with optimal transport and convex optimization.
Kazhdan, M., Bolitho, M., and Hoppe, H. (2006). Poisson surface reconstruction. In Proceedings of the fourth Eurographics symposium on Geometry processing, volume 7.

---

Kingma, D. P. and Ba, J. (2014), Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980
Kumar, M., Babaeizadeh, M., Erhan, D., Finn, C., Levine, S., Dinh, L., and Kingma, D. (2019). Videoflow: A flow-based generative model for video. arXiv preprint arXiv: arXiv: 1903.01434,, 2(5).
Lee, J. M. (2013). Smooth manifolds. In Introduction to Smooth Manifolds, pages 1-31. Springer.
Lorensen, W. E. and Cline, H. E. (1987). Marching cubes: A high resolution 3d surface construction algorithm. ACM siggraph computer graphics, 2144)1163119.
Lou, A., Lim, D., Katsman, I. Huang, L., Jiang, Q., Lim, S.-N., and De Sa, C. (2020). Neural manifold ordinary differential equations.
Mathieu, E. and Nickel, M. (2020). Riemannian continuous normalizing flows. arXiv preprint arXiv:2006.10640
Moser, J. (1965). On the volume elements manifold. Transactions the American Mathematical Sociity, 120(2):86-299.
Papamakarios, G., Nalisnick, E.,, Rezende, D. J., Mohamed, S., and Lakshminarayanan, B. (2019). Normalizing flows for probabilistic modeling and inference. arXiv preprint 1/Xiv:19/2.02762.
Rezende, D. and Mohamed, (2015). Variational inference with normalizing flows. International Conference on Machine Learning, pages 1530-1538. PMLR.
Rezende, D. J., Papamakarios, G., Racaniere, S., Albergo, M., Kanwar, G., Shanahan, P., and Cranmer, K. (2020). Normalizing flows on tori and spheres. In International Conference on Machine Learning, pages 8083-8092. PMLR.
Sei, T. (2011). A Jacobian Inequality for Gradient Maps the Sphere and Its Application to Directional Statistics. Communications in Statistics Theory and Methods, 42(14):2525-2 2542..
Tancik, M., Srinivasan, P. P., Mildenhall, B., Fridovich-Keil, S., Raghavan, N., Singhal, U., Ra- mamoorthi, R., Barron, J. T., and Ng, R. (2020). Fourier features let networks learn high frequency functions in low dimensional domains.
12


Reviews:
[{"review_id": "uDh2xjItd-g", "reviewer": "Reviewer_VH2c", "summary": "The paper tackles the problem of learning flow-based generative models. The proposed approach, a new instance of continuous normalizing-flow methods, models the density as a difference between the prior and a divergence term, parameterized by a neural network.\n\nThe claims are that:\n* MF constitutes a universal density approximator,\n* MF is more efficient to train than alternative CNFs; this is demonstrated empirically and the intuition is that MF does not require invoking or backpropagating through an ODE during training (note however that solving an ODE is required for sampling),\n* MF improves upon alternative CNFs in terms of density estimation, sample quality, and training complexity,\n* MF demonstrates for the first time the use of flow models for sampling from general curved surfaces.", "questions": "", "limitations": "", "rating": 7, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Learning flow-based generative models is an active research area. The current paper designs a new model that aims to improve upon recently published continuous flow generative models (Riemannian CNFs, FFJORD). From this perspective, the approach is well-timed, theoretically motivated, and relevant to the community. I am not familiar with a similar approach, but note that I am not familiar with all the literature on continuous flow generative models.\n\n*Experiments*:\n\nMy main concerns (questions) regard the empirical claims:\n* Figure 2: The experiment is a great illustration of the proposed method, demonstrating that it can model complex (multi-modal, discontinuous) datasets. That said, prior methods, e.g. FFJORD, have also demonstrated that they can model these distributions (by warping an isotropic Gaussian), so this result is not specific to the current method.\n* The claims made regarding Figure 3 are that 1) the proposed method is computationally more efficient than FFJORD and 2) MF captures high-frequency details better. Unfortunately, the claims are illustrated on a single image, without a numerical evaluation. Have you considered performing this evaluation across several images (e.g. the MNIST, Omniglot datasets) and reporting aggregated results (similar to the FFJORD paper)?\n* Figure 5 illustrates the approach on the Stanford Bunny surface. Unfortunately, it’s hard to evaluate the difficulty of the problem given no baseline approaches. Since this is a major claim in the abstract, could you discuss why none of the previous approaches can be applied to this setting for comparison?\n* Table 1: The most significant result is that the proposed approach outperforms prior methods from “Riemannian Continuous Normalizing Flows”. Since the prior results are extracted from the cited paper, and the methods there chosen to have approximately the same number of parameters, I think the results would be stronger if the authors discussed whether there is a significant difference in parameter size.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors discuss the potential negative societal impact in the Discussion and limitations section.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4-6"}, "extra_text": {}}, {"review_id": "tXrQt1nly-x", "reviewer": "Reviewer_R6hn", "summary": "This paper proposes an alternative way to train a continuous normalizing flow (CNF), inspired by the Dacorogna-Moser transport. The proposed training does not require numerical integration; instead, the likelihood of the flow can be directly evaluated by computing the divergence of the mass flow rate. The idea is tested on boundaryless manifolds, such as the flat torus for toy data and implicit surfaces. ", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\n*Reason for rating*: I think this work is quite novel and neat and complements the current flow literature well so I vote for acceptance. I would be happy to increase my rating if the author can address some of my concerns about scalability and the limitations that I mentioned below. \n\n\n### Significance and novelty\n\nThe proposed method is very neat, and largely simplifies the computational cost of CNF during training time. The likelihood of the model can be computed exactly in on a single forward evaluation (modulo the potential need of trace estimation), as opposed to numerically integrating the underlying ODE like FFJORD. The proposed flow is also provably universal (despite taking a simpler form, as per eq. 11, rather than having a free form velocity field). \n\n### Scope\n\nThe proposed method is mainly designed for modeling data on manifolds, which is empirically tested on a few toy and real datasets. More specifically, the manifolds need to be compact, boundaryless, and connected for the proposed method to work, which slightly limits its applicability and at the same time also leaves room for innovation. \n\n### Theoretical soundness\n\nI only check for the main development in section 3 (so I cannot speak for section 4), which looks correct to me, except for the conditions in the statement of theorem 3. I am not quite sure about \"For $\\lambda\\geq1$ and small $\\epsilon>0$\". Does the choice of $\\epsilon$ depend on $\\mu$ or $\\lambda$? Or does it need to be asymptotically small? Also, why does $\\lambda$ need to be greater than 1 if it's only a regularizer? It might help a bit to discuss the necessity of these conditions.\n\n \n### Limitation\n\n- (Regularization vs reparameterization) Most normalizing flows induces a valid probability density by design (such as regular CNFs which leverage the Lipschitzness of regular neural network architecture, residual flows that reparameterize the weights by spectral normalization, autoregressive and coupling blocks with masks over weight parameters, etc). However, the Moser flow needs to regularize the \"negative part\" of the density. It has the following disadvantages.\n    1. Moser flow is not guaranteed to induce a valid probability density.\n    2. It requires tuning an additional hyperparameter for regularization, $\\lambda$. Is it a potential cause for instability during training? I actually think some negative results on this might help people better understand the work and think about how to improve upon it. How is it chosen as a hyperparameter (i.e. based on what criteria)? The chosen values (2 and 100) seem a bit drastic, with no discussion provided.\n    3. The regularization seems to work at least empirically across the experiments conducted in this paper, but I suspect it might not scale well to higher dimensional problems, especially if $\\eta$ is uniform. \n\n- Section 5.2: It's obvious that the proposed method is faster to train than FFJORD since it does not require integration (which the authors show in Fig 3). But given that the proposed flow has a simple time conditioning mechanism, as opposed to the free form conditioning that's allowed when using FFJORD, I think it'll be fairer and more meaningful to also compare the increase in likelihood in runtime and/or in #iterations. Also, since FFJORD in that experiment only uses a time-independent vector field, the expressivity of the baseline is also weakened.\n\n### Additional questions\n\n- Is the trace operator computed exactly or estimated? I imagine it is done exactly given the low dimensionality of the problems, but it might be better to mention the potential problem of scaling up and the need for approximation.\n- Is it possible to generalize the flat torus for 2D toy data to accommodate higher dimensional problems lying in the Euclidean space? such as dequantized image data that live in a [0, 1]^D hypercube.\n- typo? line 83 \"from\" instead of \"form\"?\n\n\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "See above", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "oxVer3jJRB", "reviewer": "Reviewer_oTmD", "summary": "This paper presents Moser Flow a new class of Continuous Normalizing Flows for Riemannian Manifolds. The main approach exploits results in differential geometry (Moser 1965) to construct a vector field that satisfies the normalizing equation given by the pullback of the volume form under $\\Phi$. The approach is both sensible and provides direct advantages over Riemmanian CNF's as training does not require solving the ODE using a solver. The authors also consider Moser Flows for Euclidean Submanifolds under the Induced metric and show that Moser Flows can learn arbitrary distributions on manifolds of interest (---i.e. compact, boundryless etc...). Finally, the experiments conducted are both interesting in their variety and illuminating in that they show the effectiveness of the proposed method. Overall, this paper represents a non-trivial step forward---manifold or otherwise---for Continuous Normalizing Flow research.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, I really enjoyed reading this paper and I think for the most part it is exceptionally well written. The main ideas are clear but maybe the authors could do a bit more to contrast the difference between Riemannian CNF by Mathieu and Nickel 2020. I would also like to highlight, what I believe are the principal strengths of the approach. First, unlike conventional Riemannian CNF's----and CNF's more broadly---one does not need to solve the ODE using off the shelf solvers that may introduce additional sources of error. This i believe is a significant win. To this end I think the paper would be strengthened by considering similar datasets/experiment pipelines of Euclidean CNF's (beyond 2D toy data and Figure 3) to see if Moser Flows can handle conventional non-manifold specific generative modellng datasets which are significantly larger, both in dimensionality and size, than the ones considered here. \n\nIn terms of key weaknesses, I don't find that this paper has any major drawbacks. One could criticize that it leans too heavily on the original Moser's results (e.g. even the interpolant is borrowed) but I think this is folly. The adaptation of the method to Riemannian CNF's brings about a lot of technical novelty which should not be understated. In terms of experiments, I wish the authors also compared with concurrent NF on manifold baselines like Neural Ordinary Manifold Differential Equations by Lou et. al 2020 and Falorsi, L. and Forré 2020. These methods have subtle but important differences than Riemannian CNF by Mathieu and Nickel 2020. In terms of presentation, the supplementary material could be strengthened. Right now there is no explicit/formal proof of Theorem 2 and 3. I do concede that the main arguments for Theorem 3 are present in the main text, but this is not a formal proof. \n\nQuestions/Minor Comments:\n- The divergence computation in Riemannian CNF's relied on the Hutchinson Trace Estimator, was this also used for Moser Flow?\n- This maybe my misunderstanding, but I feel like there is a missing step in the proof of Lemma 2 as the final equality in RHS of the equation below line 31 does not equal the first equality of the line below. Unless, $\\sum_{j=1}^k \\langle \\frac{\\partial u}{\\partial n_i}, n_i \\rangle$ (Note that there is a typo with subscript $i$ in the summation over $k$,it should be over $j$) is $0$.\n\n\n ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "N/A", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "SgeIteisz5a", "reviewer": "Reviewer_96Cx", "summary": "The paper introduces a new type of continuous normalizing flow (CNF) for manifold-valued data. Unlike in previous works, this Moser Flow (MF) models a density as a base density minus the divergence of a neural network. The authors prove that under certain assumptions this defines a probability distribution over the manifold. This approach has a different trade-off between inference and generation from other approaches: while the model density can be evaluated very cheaply, sampling from this model requires an ODE solver. The approach is qualitatively demonstrated on toy datasets on tori and the Stanford bunny, and quantitatively evaluated on a few datasets on 2-spheres.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Modeling probability densities on manifolds is an important problem and Moser Flows present a new approach to solving it. Compared to existing methods, the MF setup presents a substantially different trade-off between the efficiency of likelihood evaluations and the efficiency of data generation. They may be thus be very relevant, in particular in applications where likelihood evaluations are more important than sampling.\n\nMaybe the most important contributions of the paper are proposing this method, proving that it defines a probability distribution over the manifold, and proving that it presents a universal density estimator. These theoretical aspects seem correct to me, though I have not checked all the proofs. In some parts (for instance the proof of theorem 3) I would have appreciated more details and sometimes more precision, perhaps by providing an extended version of the argument in the supplementary material. \n\nMy main gripe with the method part of the paper is a certain lack of discussion. In particular:\n1. What happens if the trained model still has $\\bar{\\mu}(x) < 0$ somewhere? Can the model still be useful to define a probability density? What are the limitations of the proposed regularization scheme (for instance when going to higher dimensions)? \n2. How do Moser Flows scale? The authors briefly allude to the poor scaling in one sentence in the discussion, but this really deserves more space. Would the approach for instance work well in low-dimensional manifolds embedded in a high-dimensional space (like the hypothesized image manifold)?\n3. A more in-depth comparison of MF to established flows on manifolds would really improve this paper.\n4. Some part of the paper focuses on general Riemannian manifolds, while later the paper focuses on the case of submanifolds of a Euclidean space. How useful are Moser flows in the case when manifolds are not defined as submanifolds of some Euclidean space?\n5. What if the manifold is not known a priori but you only have access to samples from it? Is there a way to extend this method such that an atlas for the manifold is learned as well? \nI understand that space is at a premium at NeurIPS, but perhaps the author can make room to discuss some of these aspects by e.g. moving the proof of Thm. 3 into the appendix?\n\nIn the empirical part of the paper, the authors demonstrate that the idea works, at least in low-dimensional examples. The Stanford bunny results are cool to see. The quantitative evaluation is pretty thin, more metrics and comparisons to other baselines would have been better. I would also have liked to see an empirical verification that the model densities form a properly normalized probability density over the manifolds / that $\\bar{\\mu}(x)$ is positive everywhere. It is nice to see the speed-up in inference time in action, but it would have been even better to also include results on the sampling speed.\n\nThe paper is well written and clearly structured. Perhaps the presentation could have been further improved by hiding away more technical details in the supplementary materials and using the space for more detailed discussions and explanations (see above). Unlike many ML papers, it has properly sized labels in the figures (except, inexplicably, for the bottom right panel in Fig 3). The extensive supplementary materials and the provided code (clean, but not very documented) are good.\n\nAll in all, the authors present a novel solution to an interesting, relevant problem. The theory part of the paper is solid. While the empirical demonstration is thin, I do think that this paper has a lot of merit even without the usual benchmark table with N baselines in which the \"Ours\" entry has all the bold numbers. However, I do think that it currently lacks in-depth discussion of the properties of Moser Flows relative to previous approaches as well as on scaling it to higher-dimensional problems. With an extended discussion, I think this could make a good NeurIPS paper.\n\nMinor comments and typos:\n- Eq. (5): should this be a capital $\\Phi$?\n- Sec. 3.2: would be nice to explicitly describe sampling from the MF here somewhere\n- line 140: \"Stokes' theorem\" with apostrophe, I believe\n- around Eq. 14: more discussion of the choice of $\\lambda$ would be great\n- line 175: typo: superfluous \"is\"\n- line 182: is the $x \\in \\mathcal{M}$ condition necessary here?\n- Fig. 2: it would be great if the colour maps were scaled such that the empirical densities from the samples (based on histograms) and the density plot match\n- Thm. 4: the statement \"we conclude this section by proving\" sounds a bit weird because there is no proof here (it is in the supplementary materials)\n- line 220/221 read weird, typo?\n- Fig. 3: how do FFJORD and MF compare in terms of log likelihood?\n- Fig. 3: is this not a weird example to put on a torus b/c of the discontinuity at the identified edges of the images?\n- Sec. 5.4: typo in the title, should be \"Curved surfaces\"\n- Sec. 6: how does MF compare to these methods? This was superficially discussed before, but it would be great to re-visit in more detail here\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I agree with the author's assessment that this work shares the same fundamental risks of all generative models. I do think that for the scope of this paper this acknowledgement is enough.   ", "review_text": "", "extra_scores": {"ethical_concerns": "I have no ethical concerns with this paper.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "AvKtDWF3vD5", "reviewer": "Reviewer_STbS", "summary": "The authors define a continuous normalizing flow on an arbitrary Riemannian manifold through the divergence of a neural network using the Moser theorem. They motivate their approach and show how this flow can be represented and learned. By applying their method to a toy distributions and a real world earth and climate dataset, they demonstrate the effectiveness of the method and that they can outperform competing procedures. Furthermore, they show that their method is much faster and more efficient than FFJORD when learning a complex two dimensional density.", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The authors use a very creative approach to define a continuous normalizing flow through the divergence of a neural network. Usually, they are constructed as the solution of an ODE and solving it can be time consuming. Here, the authors use the Moser theorem and use that the divergence of a vector field can be the difference of two distributions, i.e the base distribution and a distribution approximating the target. The power of the method is that it can directly operate on an arbitrary Riemannian manifold without the requirement of an invertible map from $\\mathrm{R}^d$ to the manifold. So far, only a small number of methods exist which are able to do that. \nImplementing and learning the flow, however, requires some heuristic tricks and since I am not an expert on the Moser theorem and its implications in this context, I am not sure how valid they are.\nAnyway, the method seems to be very efficient and outperforms competing approaches on small dimensional toy and real world datasets.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Currently, the method cannot be applied to high dimensional manifolds. Although this is unfortunate, competing methods struggle with this problem as well.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The following is a summary of the pros and cons that resulted from the reviews and the discussion period.

pros:
* novel and nontrivial contributions to the flow literature. (R6hn, oTmD, STbS, VH2c, 96Cx)
   - reduces computational cost to train CNFs by no longer needing to solve an ODE. 
   - likelihood computation does not require evaluating an ODE, constrast to regular CNFs.
   - Probably universal density estimator under suitable assumptions.
* interesting variety of experiments that show the effectiveness of the proposed method. (oTmD)
* for the most part well written (oTmD, 96Cx)

cons:
* limitations: 
    - scalability limitations to higher dimensional problems. --> authors respond that higher dimensions are indeed challenging and promis to expand on the precise challenges in an updated version of the paper. (R6hn, 96Cx)
    - regularization of the negative part of the density in moser flows introduces an additional hyperparameter for tuning. worries that the regularization parameter might lead to unstable training of chosen wrongly. --> authors added experiments to show the behavior of the model under different ranges of this hyperparameter. (R6hn)
   - can the negative density also lead to moser flow not producing a valid density? --> authors rebut this in their response. (R6hn, 96Cx)
* confusion around theorem 3, exact or approxiamte computation of traces etc. --> solved by author response.(R6hn)
* comparison with other normalizing flow methods on manifolds would strengthen the paper. --> addressed in discussion. (oTmD, 96Cx)
* introducing an explicit proof of theorem 2 and 3 would improve the paper (arguments for proof can now be found in main paper). --> addressed in discussion. (oTmD, 96Cx)
* Clarifications needed about applicability of method. Can Moser flows still be used for manifolds that are not submanifolds of a Euclidian space? What if the manifold is not known?  --> discussed in author response. (96Cx)
* improvements needed in terms of evaluation: 
   - figure 2: prior methods can also model this distribution, so the success of the proposed method over previous methods is unclear. (VH2c)
   - quantitative evaluation would make the claims stronger of being more computationally efficient, capturing densities better (e.g. Fig 3). --> addressed in author response.  (VH2c, 96Cx)
   -  comparison is needed with FFJORD in terms of both likelihood and runtime/#iterations. --> the authors include a link in their response with these results. (R6hn)
   - figure 5: why can't previous approaches be used to model this problem? --> addressed in author response. (VH2c)
   - table 1: how do the number of parameters of this method compare to previous work? --> addressed in author response. (VH2c) 

Authors and reviewers engaged into active discussions during the discussion period. Most of the concerns raised by the reviewers were addressed, leading to 3/5 reviewers raising their scores and a consensus to accept this submission. Considering the nontrivial contribution of this work and the future research it could lead to, the recommended decision is accept.
Decision: Accept (Oral)