Neural Conservation Laws: A Divergence-Free Perspective
Authors: Jack Richter-Powell
Yaron Lipman
Ricky T. Q. Chen
Venue: NeurIPS 2022 Accept (2022)
Keywords: density estimation
automatic differentiation
generative modeling
optimal transport maps
structured deep learning

Abstract: We investigate the parameterization of deep neural networks that by design satisfy the continuity equation, a fundamental conservation law. This is enabled by the observation that any solution of the continuity equation can be represented as a divergence-free vector field. We hence propose building divergence-free neural networks through the concept of differential forms, and with the aid of automatic differentiation, realize two practical constructions. As a result, we can parameterize pairs of densities and vector fields that always satisfy the continuity equation by construction, foregoing the need for extra penalty methods or expensive numerical simulation. Furthermore, we prove these models are universal and so can be used to represent any divergence-free vector field. Finally, we experimentally validate our approaches by computing neural network-based solutions to fluid equations, solving for the Hodge decomposition, and learning dynamical optimal transport maps.

Neural Conservation Laws: A Divergence-Free Perspective
Jack Richter-Powell Vector Institute www.rreer-oooeeleeeeeeooooo
Yaron Lipman
Ricky Q. Chen
Meta AI y ylipmandmeta.com com
Meta Al
rtgichen@meta.com com
Abstract
We investigate the parameterization of deep neural networks that by design satisfy the continuity equation, fundamental conservation law. This is enabled by the observation that any solution of the continuity equation can be represented divergence-fiee frree vector field. We hence propose building divergence- free neural networks through the concept differential forms, and with the aid automatic differentiation, realize two practical constructions. As result, parameterize pairs of densities and vector fields that always exactly satisfy the continuity equation, foregoing the need for extra penalty methods expensive numerical simulation. Furthermore, we prove these models are universal and so can be used to represent any divergence-free vector field. Finally, we experimentally validate our approaches by computing neural network- based solutions to fluid equations, solving for the Hodge decomposition, and learning dynamical optimal transport maps.
Introduction
Modern successes in deep learning are often attributed to the expressiveness of black-box neural networks. These models are known to be universal function approximators [Hornik al., 1989] but this flexibility comes at a cost. In contrast to other parametric function approximators such as Finite Elements [Schroeder and Lube, 2017], it is hard bake exact constraints into neural networks. Existing approaches often resort to penalty methods to approximately satisfy constraints- these increase the cost of training and can produce inaccuracies in downstream applications when the constraints are not exactly satisfied. For the same reason, theoretical analysis of soft-constrained models also becomes more difficult. On the other hand, enforcing hard constraints on the architecture can be challenging, and even once enforced, it is often unclear whether the model remains sufficiently expressive within the constrained function class.
In this work, we discuss an approach to directly bake in two constraints into deep neural networks: (i) having divergence of zero, and (ii) exactly satisfying the continuity equation. One of our key insights is that the former directly leads into the latter, so the first portion of the paper focuses on divergence-free vector fields. These represent special class of vector fields which have widespread use in the physical sciences. In computational fluid dynamics, divergence Wavier-Stokes equations. In R we know that the curl of a vector field has divergence of zero, which has seen many uses in graphics simulations (e.g., Eisenberger et al. [2018]). Perhaps less well-known is that lurking behind this fact is the generalization of divergence and curl through differential forms [Cartan, 1899], and the powerful identity d² O. We first explore this generalization, then discuss two constructions derived from it for parmaterizing divergence- free vector fields. While this approach has been partially discussed previously [Barbarosie, 201 I, Kelliher, 2021], it is not extensively known and to the best of our knowledge has not been explored by the machine learning community.
model incompressible fluid interactions formalized by the Euler frreeeeeeffeeeeeeee free vector fields are used
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

A Jb
J
div(A) cq. (4)
eq. (7)
b
A
A2
Figure 1: Divergence-free vector fields v Rd + F can be constructed from an antisymmetric matrix field A E2 Rdxd an arbitrary vector field b: F R2 J, represents the Jacobian matrix of b, and A1 and Ag are the first and second rows of A. Color denotes divergence.
Figure 1-to transform sufficiently smooth neural networks into divergence-free neural networks, made efficient by automatic differentiation and recent advances vectorized and composable transformations [Bradbury et al., 2018, Horace He, 2021]. Furthermore, both approaches theoretically represent any divergence- free vector field.
Concretely, we derive two approaches-visualizzee
We combine these new modeling tools with the observation that solutions of the continuity equation- a partial differential equation describing the evolution of density under a a.low -can be characterized jointly as a divergence-
vector field. As result, we can parameterize neural networks that, by design, always satisfy the continuity equation, which we coin Neural Conservation Laws (NCL). While prior works either resorted to penalizing errors [Raissi al., 2019] numerically simulating the density given flow [Chen et al., 2018), baking this constraint directly into the model allows us to forego extra penalty terms and expensive numerical simulations.
2 Constructing divergence-free vector fields
We will use the notation differential forms in R" for deriving the divergence-free and universality properties of our vector field constructions. We provide concise overview of differential forms in Appendix A. For a more extensive introduction see e.g., Do Carmo [1998], Morita [2001]. Without this formalism, it is difficult show either of these two properties; however, readers who wish to skip the derivations can jump to the matrix formulations in equation and equation 7.
Let us denote A (RR" as the space of differential k-forms, d (RR" A as the exterior derivative, and * AAA((R)) - A²"*(R²") as the Hodge star operator that maps each k-form to an (nor))form. Identifying a vector field v IRd - Ied as a 1-form, v = Σim vidz;, we can express the divergence div(v) the composition d *
To parameterize divergence-free vector field v + R", we note that by the fundamental property of the exterior derivative, taking arbitrary (n-2) μ An=?(R") we have that
0= de = d(du)
and since is its own inverse up to sign, it follows that
*dp
is divergence free. We write the parameterization v = +dp explicitly in coordinates. Since a basis for A --222(R)) ((R") can be chosen be (dz, dzz), we can write μ Ex,j= Hij * (dz, A dr,).. where Myj -ij Now a direct calculation shows that up a constant sign (see Appendix-A.1)
Opi
dp = Σ
dr.-
(3)
dz
1=1
This formula suggestive simple matrix formulation: If we let A R - Rax be the anti- symmetric matrix-valued function where Aij Mig then the divergence-free vector field v *dj can be written as taking row-wise divergence of A, i.e.
div(A1)
(4)
div(An)

---

However, this requires parameterizing O(n²) functions. A more compact representation, which starts from vector field, can also be derived. The idea behind this second construction to model j instead as 14 = Ev, where (RR)) Putting this together with equation 2 we get that
t = *dōv
(5)
divergence-free vector field. provide equation 5 in matrix formulation we first write v An (IR) in the (n-1)-fome basis, Le., -i=1 Vi * dr,, Then a direct calculation provides up to a constant sign
1
dv
ovj
8v
(dz, A dr,)
(6)
2
dz, dz,
So, given arbitrary vector field RR" construct A as
R and denoting Jo the Jacobian of b, can
A = Jb - J/
where Jn denotes the Jacobian of b. To summarize, we have two constructions for divergence- vector fields v:
Matrix-field: (equations 2 and 4) is represented using anti-symmetric matrix field A. Vector-field: (equations 5 and 4+7) is represented using vector field 6.
As we show in the next section, these two approaches are maximally expressive (i.e., universal), meaning they can approximate arbitrary smooth divergence-free vector fields. However, empirically these two constructions exhibit different practical trade-offs. The matrix-fiee construction has a computational advantage as it requires one less Jacobian computation, though it requires more components- Onn²) vs. ( represent than the vector-field construction. This generally isn't concern as all components parallel. However, the vector-field construction can make it easy to bake in additional constraints; an example of this is discussed in Section 7.1, where non-ncgativvty constraint is imposed for modeling continuous-ting probability density functions. In Figure 2 we show wallelock times for evaluating the divergence-free vector field based on our two constructions. Both exhibit quadratic scaling (in function evaluations) with the number of dimensions due to the row-wise divergence in equation 4, while the vector-field construction has an extra Jacobian computation.
matrix-field vector-field
Time (ms)
Dimension
be computed in
Figure 2: Compute times.
2.1 Universality
Both the matrix-field and vector-field representations are universal, i.e. they can model arbitrary divergence-free vector fields. The main tool in the proof is the Hodge decomposition theorem [Morita, 2001, Berger, 2003]. For simplicity we will be working on periodic piece of IR" namely the torus T" [-M, M\ ~ where ~ means identifying opposite edges of the n-dimensiooall cube, and M > 0 is arbitrary. Vector fields with compact support can always be encapsulated in T with M sufficiently large. As T" is locally identical to R", all the previous definitions and constructions hold.
Theorem 2.1. The matrix and vector-field representations are universal in T, possibly only missing constani vector field.
A formal proof of this result is in Appendix B.I.
3 Neural Conservation Laws
now discuss key aspect of our work, which is parameterizing solutions scalar conservation laws. Conservation laws are focal point of mathematical physics and have seen applications in machine learning, with the most well known examples being conserved scalar quantities often referred to as density, energy, or mass. Formally, conservation law can be expressed as first-order PDE

---

written in divergence form as 8e + div(j) = O, where j is known as the flux, and the divergence is 8t, taken over spatial variables. In the case of the continuity equation, there is velocity field u which describes the flow and the flux is equal to the density times the velocity field:
ap Ot
div(pu) = =0
(8)
where IR - RR and u R + TR. One can then interpret the equation to mean that u transports p().-) to p(t,,) continuously without teleporting, creating or destroying mass. Such an interpretation plays a key role in physics simulations as well as the dynamic formulation of optimal transport [Benamou and Brenier, 2000]. In machine learning, the continuity equation appears in continuous normalizing flows [Chen et al., 2018]-which also have been used to approximate solutions to dynamical optimal transport. [Finlay al., 2020, Tong et al.. 2020, Onken et al., 2021] These however, only model the velocity u and rely on numerical simulation to solve for the density P. which can be costly and time-consuming.
Instead, we observe that equation can be expressed as a divergence-free vector field by augmenting the spatial dimensions with the time variable, resulting in vector field that takes as input (t, and outputs (p, pu). Then equation 8 is equivalent to
Op Ot
div(u) div
P pru)
div(pu)
(9)
where the divergence operator is now taken with respect to the joint system (t, i.e. 8 We thus propose modeling solutions of conservation laws by parameterizing the divergence- field v. Specifically, we parameterize divergence-free vector field v and set i = P and 12:n-1 1 = pu, allowing us to recover the vector field as u (2:=11 assuming p * O. This allows to enforce the continuity equation at architecture level. Compared to simulation-based modeling approaches, we completely forego such computationally expensive simulation procedures. Code for our experiments are available at https: //gi thub. httm://accbookreeeercchheeeeeeee cow.arrrraaaeealaaaossoeaaaaaaaa conservation....coommcmc. law.
di -i-1 dz. vector
4 Related Works
Baking in constraints in deep learning Existing approaches to enforcing constraints in deep neural networks can induce constraints on the derivatives, such as convexity [Amos et al., 2017] or Lipschitz continuity [Miyato et al., 2018]. More complicated formulations involve solving numerical problems such as using solutions of convex optimization problems [Amos and Kolter, 2017]. solutions of fixed-point iterations [Bai et al., 2019], and solutions of ordinary differential equations [Chen et al., 2018]. These models can help provide more efficient approaches alternative algorithms for handling downstream applications such as constructing flexible density models [Chen et al., 2019, Lu et al., 2021] approximating optimal transport paths [Tong al., 2020, Makkuva al., 2020, Onken et al., 2021]. However, in many cases, there is a need to solve a numerical problem in which the solution may only be approximated up to some numerical accuracy; for instance, the need to compute the density flowing through a vector field under the continuity equation [Chen et al., 201 8J.
Applications of differential forms Differential forms and more generally, differential geometry, have been previously applied in manifold lcarning-aa sec e.g. Arvanitidis et al. [2021] and Bronstein et al. [2017] for an in-depth overview, Most of the applications thus far have been restricted to 2 or 3 dimensions-eitte using identities like div o curl = 0 in 3D for fluid simulations [Rao et al., 2020], or for learning geometric invariances in 2D images 3D space [Gerken al., 2021, Li et al., 2021].
Conservation Laws Machine Learning [Sturm and Wexler, 2022] previously explored discrete analogs of conservation laws by conserving mass via a balancing operation in the last layer of a neural network. [Müller, 2022] utilizes a wonderful application of Noether's 's theorem to model conservation laws by enforcing symmetrics in a Lagrangian represented by neural network.
5 Neural approximations to PDE solutions
As demonstration of our method, we apply it to neural-based PDE simulations of fluid dynamics. First, we apply it to modelling inviscid fluid flow in the open ball IB C R3 with free slip boundary

---

conditions, then to a 2d example on the flat Torus T², but with more complex initial conditions. While these are toy examples, they demonstrate the value of our method in comparison to existing approaches-namaee that we can exactly satisfy the continuity equation and preserve exact mass.
The Euler equations of incompressible flow The incompressible Euler equations [Feynman et al., 1989] form an idealized model of inviscid fluid flow, governed by the system of partial differential equations
ae div(pu) O,
On + ot
vu
F2. div(u)
(10)
in three unknowns: the fluid velocity u(t, x) E R. the pressure p(t, x), and the fluid density p(t, x). While the fluid velocity and density are usually given at = 0, the initial pressure is not required. Typically, on a bounded domain f2 S R", these are supplemented by the free-slip boundary condition and initial conditions
u n on an v(0, = uo and p(0, = Po on
(11)
The density P plays critical role since in addition to being a conserved quantity, it influences the dynamics of the fluid evolution over time. In numerical simulations, satisfying the continuity equation as closely possible is desirable since the equations in (10) are coupled. Error in the density feeds into error in the velocity and then back into the density over time. In the finite element literature, great deal of effort has been put towards developing conservative schemes that preserve mass (or energy in the more general compressible case)-see see Guermond and Quartapelle [2000] and the introduction of Almgren et al. [1998] for an overview. But since the application of physies informed neural networks (PINNs) to fluid problems is much newer, conservative constraints have only been incorporated as penalty terms into the loss [Mao al., 2020, Jin et al., 2021].
5.1 Physics informed neural networks
Physics Informed Neural Networks (PINNs; Raissi et al. [2019, 2017]) have recently received renewed attention as an application of deep neural networks. While using neural networks as approximate solutions PDE had been previously explored (e.g Lagaris al. [1998]), modern advances in automatic differentiation algorithms have made the application to much more complex problems feasible [Raissi et al., 2019]. The "physics" in the name is derived from the incorporation of physical terms into the loss function, which consist of adding the squared residual norm of PDE. For example, to train a neural network = [p,p, u] to satisfy the Euler equations, the standard choice of loss to fit to is
Lr
=
1u + V.ut
Ep
Luv divi())
L, = ||u(0,,) :)- -oo())l + -lp(0,))-P(())1
Leom =
|1
Bo 0i
div(pu)
L = || nl Tiotal = 7 [T.r L,, Jates I.cont Lo]
where 7= (7F; 77, Idiv "oons, Ta) denotes suitable coefficients (hyperparameters) The loss term LG ensures fluid does not pass through boundaries, when they are present. Similar approaches were taken in [Mao et al., 2020] and [Jagtap et al., 2020] for related equations. While schemes of this nature are very easy to implement, they have the drawback that since PDE terms are only penalized and not strictly enforced, one cannot make guarantees as to the properties of the solution.
To showcase the ability of our method model conservation laws, we will parameterize the density and vector field as v = [p, pu], as detailed in Section 3. This means we can omit the term Leont as described in Section 5.1 from the training loss. The divergence penalty, Laiv remains when modeling incompressible fluids, since u is not necessarily itself divergence-fres - it is = [p, which is divergence free. In order to stablize training, we can modify the loss terms LF, La, Ls to avoid division by p. This is detailed in Appendix B.2.
5.2 Incompressible variable density inside the 3D unit ball
We first construct a simple example within bounded domain, specifically, we will consider the Euler equations inside (0,, 1) C R with the initial conditions
p(0, x) 3/2 ||=||
v(0, x) = (-2, x0 1,1/2)
(12)
'The convective derivative appearing in equation 10, Vau(z)= = limm--50 < (le+hhu(s)) -((=) = [uu[(u) is
also often written (V u)u.
5

---

= 0.00
t 0.25
= 0.50
f = 0.00
? 0.25
: 0.50
(a) Incompressible baseline
(b) Neural Conservation Law (Ours)
Figure 3: Exactly satisfying the continuity equation yields much better performance than only satisfying the incompressible (divergence-free) condition of the vector field, even though both approaches obtain very small loss values. 2D slices of the learned solution of and are shown.
As baseline, we compare against a parameterization that uses the divergence- vector field only for the incompressible constraint. This equivalent the curl approach described in Rao al. [2020]; another neural network is used to parameterize p. In contrast, our generalized formulation of divergence-free fields allow us model the continuity equation exactly, as in this case, [p, pul is 4 dimensional. We parameterize our method using = [p, pul, as detailed in Section 3, with the vector-field approach from Section 2.
The results for training both networks are shown Figure 3. Surprisingly, even though both architectures signficantly improve the training loss after comparable number of iterations (see Appendix C), the curl approach fails to advect the density accurately, whereas our NCL approach evolves the density as it should given the initial conditions.
5.3 Incompressible variable density flow on T
As a more involved experiment, we model the Euler equations on the flat 2-Torus T This is parameterized by multilayer perceptron pre-composed with the periodic embedding [0, 1]2 T
((x, y) = [20s(2 sin (2Tx), cos (2rr), sin())y)
The continuity equation is enforced exactly, using the method detailed in Section 3, the Leum term omitted from Laoual The initial velocity and density are
po(x, y) (21 23)² +1
vo v) [e e /2]
where = 0(x, y, 1). Here we used the matrix-field parameterization referred to in Section 2, for the simple reason that performed better empirically. We also include the harmonic component from Hodge decomposition, which is just constant vector field, to ensure have full universality. Full methodology for the experiment detailed Appendix The results of evolving the flow for time t [o, 1/a] are shown in Figure 4, well comparison with reference finite element solution generated by the conservative splitting method from [Guermond and Quartapelle, 2000]. In general, model fits the initial conditions well and results in fluid-like movement of the density, however small approximation errors can accumulate from simulating over longer time horizon. To highlight this, we also plotted the best result were able achieve with standard PINN. Although the PINN made progress minimizing the loss, it was unable to properly model the dynamics of the fluid evolution. While this failure likely be corrected by tuning the constant (see Section 5.1), were unable to find a combination that fit the equation correctly. By contrast, our NCL model worked with the first configuration we tried.
6 Learning the Hodge decomposition
Using our divergence-free models, we can also learn the Hodge decomposition vector field itself. Informally, the Hodge decomposition states that any vector field can be expressed as the combination

---

Training Loss
100
PINN
PINN
NCL
10f
50000
100000
150000
200000 250000 s0n000 350000 KoRoon
NCL
Iteration
(b) Convergence of training loss
107
PINN NCL
--pp
FEM
s0000 30000g 150n00,
200000 250000 3s0000 koonch
200000
1=0.
1=0.1
1=02
/=0.3
Iteration
(a) Leamned density function
(c) Comparison FEM solution
Figure 4: While both PINN and NCL models minimize the training loss effectively and fit the initial conditions, the PINN fails to learn the dynamies of the advected density. When compared gold standard FEM solution, our NCL model nicely exhibits linear convergence to the solution.
of a gradient field (i.e. gradient of a scalar function) and a divergence-free field. There are many scenarios where neural network modelling vector field should be gradient field (theoretically), such as score matching [Song and Ermon, 2019], Regularization by Denoising [Hurault et al., 2022]. amortized optimization [Xue al., 2020], and meta-learned optimization [Andrychowicz al., 2016]. Many works, however, have reported that parameterizing such a field as the gradient of potential hinders performance, and such they opt use an unconstrained vector field. Learning the Hodge decomposition can then be used to remove any extra divergence-free i.e. rotational, components of the unconstrained model.
As a demonstration of the ability of our model to scale in dimension, we construct a synthetic experiment where have access to the ground truth Hodge decomposition. This provides benchmark to validate the accuracy of our method. To do so, we construct the target field 6 as
G = Vw+
(13)
where is a fixed divergence-fres free neural network, and w is fixed scalar network. We use the same embedding from Section 5.3, but extended T", for E (25, 50, 100). We then parameterize another divergence-fiee free model which has a different architecture than the target 17. Inspired by the Hodge decomposition, we propose using the following objective for training:
{(0) = 1|Jo-e [Jo-vo 1Il
(14)
where Jn-oy denotes the Jacobian of é ug. In the notation of differential forms, this corresponds to minimizing [Iō(ū (g)][F, [r, which when zero implies eg 5 + 8μ, where μ is an arbitrary n-form.
However, since is divergence-free, it must hold ôμ. = Vw, i.e. the left-over portion of i is exactly Vw, because the Hodge dccomposition is an orthogonal direct sum [Warner, 1983].
For evaluation, we plot the squared Lg norm to the ground truth 7) against wall clock time. While this demonstration is admittedly artificial, it serves to show that our model scales ef- ficiently to moderately high dimensions significantly beyond those achievable using grid based discretizations of divergence- free fields (see Bhatia et al. [2013] for an overview of such methods). The results are shown in Figure 5. While larger dimensions are more costly, we are still able to recover the ground truth in dimensions up to 100.
n=25
n=50
0.10
n-100
Clork Time
Figure 5: Lg to ground truth.

---

7 Dynamical optimal transport
The calculation of optimal transport (OT) maps between two densities of interest Po and ii serves as a powerful tool for computing distances between distributions. Moreover, the inclusion of OT into machine learning has inspired multiple works on building efficient or stable generative models [Arjovsky et al., 2017, Tong et al., 2020, Makkuva et al., 2020, Huang et al., 2020, Finlay al., 2020, Rout et al.. 2021, Rozen et al., 2021]. In particular, we are interested in the dynamical formulation of Benamou and Brenier [2000], which is a specific instance where the transport map is characterized through a vector field. Specifically, we wish to solve the optimization problem
[o [lu(t, ((t,x)) drdt min Pau
(15)
subject to the terminal constraints, p(O,-) -) po(-) and p(1,-) - pi (+), as well as remaining non- negative p 2 on the domain and satisfying the continuity equation de div(pu) O.
In many existing cases, the models for estimating either p and equation 15 are completely decoupled from the generative model or transport map [Arjovsky et al., 2017, Makkuva al., 2020], in some cases would be computed through numerical simulation and equation 15 estimated through samples [Chen et al., 2018, Finlay al., 2020, Tong al., 2020]. These approaches may not be ideal as they require an auxiliary process to satisfy or approximate the continuity equation.
In contrast, our NCL approach allows us to have access to pair of and that always satisfies the continuity equation, without the need for any numerical simulation. As such, we only need to match the terminal conditions, and will automatically have access to vector field that transports mass from p(0.-) -) to p(1,-). In order to find an optimal vector field, we can then optimize u with respect to the objective in equation 15.
7.1 Parameterizing non-negative densities with subharmonic functions
In this setting, it is important that our density is non-negan non-negative for the entire domain and within the time interval between 0 and 1. Instead of adding another loss function to satisfy this condition, show how we can easily bake this condition into our model through the vector-field parameterization.
Recall for the vector-field parameterization of the continuity equation, we have vector field (y) with y [t. x²), for [0, 1), x E R". This results in the parameterization
p = div
0h. Ür.
for i = 1,..., n n 1.
(16)
dt
We now make the adjustment that ab = O. This condition does not reduce the expressivity of the formulation, as p can still represent any density function. Interestingly, this modification does imply that p bb, with 2 being the Laplacian operator, so that satisfying the boundary conditions is equivalent to solving a time-dependent Poisson's equation v (t. = Pt-
St
Furthermore, the class of subharmonic functions is exactly the set of functions that satisfies (0- 2 0. In one dimension, this corresponds to the class of convex functions but is striet generalization of convex functions in higher dimensions. With this in mind, we propose using the parameterization
(1(t,)) = Σ wn (t)) bx (t)) where 6(2) += (log([l=])3) + Es (1=11/)))
(17)
with E1 being the exponential integral function, and Wj. ag RR+ and by R are free functions of t. This a weighted sum of generalized linear models, where the choice of nonlinear function ó is chosen because is the exact solution to the Poisson's equation for a standard normal distribution when n 2, i.e. 2h (0,, /), while for 2 2 this remains a subharmonic function. As such, this can be seen as a generalization of mixture model, which are often referenced be universal density approximators, albeit requiring many mixture components [Goodfellow et al., 2016].
8

---

(a) Circles
Pinwheel
(b) Pinwheel 8Gaussians
Figure 6: Learned approximation to the dynamical optimal transport map. (top) Density p. (mid) Transformed samples in orange, with samples from Py and P1 in blue. (bottom) Learned vector field.
0.32
Minimax
Estimated WC
0.20
Minimax
NCL
Estimated we
6:22 8:30 8:18
Minimax
NCL
Ealimated
0.55
2.0
6:26
0.54
0:22
0.77
202
300
0.20
1400
Wallclock me (5)
Wall Walkk time
Vil Iolock time (S)
(a) Circles ++ Pinwheel
(b) Pinwheel 8Gaussians
(c) 8Gaussians + Circles
Figure 7: Neural Conservation Laws approximating dynamical optimal transport converges in
minutes. The alternative minimax formulation tends to underestimate the Wasserstein distance. Dashed line is an estimate from discrete OT algorithm. Shaded regions denote standard deviation.
7.2 Experiments
We experiment with pairs of 2D densities (visualized in Figure 6) and fit a pair of (p, u) to equation 15 while satisfying boundary conditions. Specifically, we train with the loss
L'' S.y, |lu(t, p(t,)) dædt
min AE 2l)(0.x)) po(=)]] + AE 2l((1. (1)())) p.u
(18)
where P. is mixture between p; and uniform density over sufficiently large area, for 1. and À a hyperparameter. We use batch size 256 and set K=128 (from equation 17). Qualitative results displayed in Figure 6, which include interpolations between pairs of densities.
Furthermore, we can use the trained models to estimate the Wasserstein distance, by transforming samples from Po through the learned vector field to time (=1. We do this for 5000 samples and compare the estimated value to (i) one estimated by that of neural network trained through minimax optimization formulation [Makkuva et al., 2020], and (ii) one estimated by that of a discrete OT solver [Bonneel et al., 2011] based on kernel density approximation and interfaced through the pot library [Flamary al., 2021].
Estimated squared Wasserstein distance function of wallclock time are shown in Figure 7. We see that our estimated values roughly agree with the discrete OT algorithm. However, the baseline minimax approach consistently underestimates the optimal transport distance. This an issue with the minimax formulation not being able to cover all of the modes, and has been commonly observed the literature (e.g. Salimans et al. [2016], Che et al. [2017]. Srivastava al. (2017]). Moreover, in order to stabilize minimax optimization in practice Makkuva et al. [2020] made use of careful optimization tuning such as reducing momentum [Radford et al., 2016]. contrast, our NCL approach is simple optimization problem.

---

8 Conclusion
We proposed two constructions for building divergence-fres neural networks from an original uncon- strained smooth neural network, where either original matrix field or vector field is parameterized. We found both to useful in practice: the matrix field approach is generally more flexible, while the vector field approach allows the addition of extra constraints such non non-negativity constraint. We combined these methods with the insight that the continuity equation can reformulated as a divergence-free vector field, and this resulted in method for building deep neural networks that constrained to always satisfy the continuity equation, which we refer to as Neural Conservation Laws.
Currently, these models difficult to scale due to extensive use of automatic differentiation for computing divergence of neural networks. It may be possible to combine our approach with parame- terizations that provide cheap divergence computations [Chen and Duvenaud, 2019]. Nevertheless, the development efficient batching and automatic differentiation tools an active research area, so expect these tools to improve development progresses.
Acknowledgements
We acknowledge the Python community [Van Rossum and Drake Jr, 1995, Oliphant, 2007] and the core set of tools that enabled this work, including PyTTrrch [Paszke et al., 2019], functorch [Horace He, 2021], I, torchdiffeq [Chen, 2018], JAX [Bradbury et al., 2018], Flax [Heek et al., 2020], Hydra [Yadan, 2019], Jupyter [Kluyver et al., 2016], Matplodli [Hunter, 2007], numpy [Oliphant, 2006, Van Der Walt al., 201 IJ. and SciPy [Jones et al., 2014]. Jack Richter-Povee would also like to thank David Duvenaud and the Vector Institute for graciously supporting them over the last year.
References
A. S. Almgren, J. B. Bell, P. Colella, L. H. Howell, and M. L. Welcome. A Conservative Adaptive Projection Method for the Variable Density Incompressible Ntvier-Stokes Equations. Journal of Computational Physics, 142(1):1 May 1998. ISSN 00219991. doi: 10. 1006/jeph. 1tt.0///ee....9..8880... 1998.5890. URL https: htttps://iinkeie //ink///n///n/nn/bbb elsevier http:///ctrceeeeaa////////////1/
B. Amos and J. Z. Kolter. Optnet: Diffcrentiable optimization as layer in neural networks. In International Conference on Machine Learning, pages 136-145. PMLR, 2017.
B. Amos, L. Xu, and J. Z. Kolter. Input convex neural networks. In International Conference on Machine Learning, pages 146-155. PMLR, 2017.
M. Andrychowicz, M. Denil, S. Gomez, M. W. Hoffman, D. Pfau, T. Schaul, B. Shillingford, and N. De Freitas. Learning to learn by gradient descent by gradient descent. Advances in neural information processing systems, 29, 2016.
M. Arjovsky, S. Chintala, and I.. Bottou. Wasserstein generative adversarial networks. In International conference on machine learning, pages 214-223. PMLR, 2017.
G. Arvanitidis, L. K. Hansen, and S. Hauberg. Latent Space Oddity: the Curvature of Deep Generative Models, Dec. 2021. URL http: httpp::/aaxxww.rr//////.. attpaa/s.../...///./............ Number: arXiv:1710. 11379 1nXiv:1710.1 1179 [stat].
S. Bai, J. Z. Kolter, and V. Koltun. Deep equilibrium models. Advances in Neural Information Processing Systems, 32, 2019.
C. Barbarosie. Representation of divergence-free vector fields. Quarterly applied mathematics, 69 (2):309-316, 2011.
J.-D. Benamou and Brenier. A computational fluid mechanics solution to the monge-kantorovinn mass transfer problem. Numerische Mathematik, 84(3):375-393, 2000.
M. Berger. A panoramic view of riemannian geometry. 2003.
10

---

H. Bhatia, G. Norgard, V. Pascucci, and P.-T. Bremer. The helmholtz-hodge decomposition survey, IEEE Transactions on Visualization and Computer Graphics, 19(8):1386-144, 2013. 1001100109/TVCG.2022...
N. Bonneel, M. Van De Panne, S. Paris, and W. Heidrich. Displacement interpolation using lagrangian mass transport. In Proceedings of the 2011 SIGGRAPH Asia conference, pages 1-12, 2011.
J. Bradbury, R. Frostig. P. Hawkins, M. J. Johnson, C. Leary, D. Maclaurin, G. Necula, A. Paszke, VanderPlas, Wanderman-Minn and Q. Zhang. JAX: composable transformations of Python+NumPy programs, 2018. URL http: /ggghbit..b....//g com/ monn/oogle/jaa
M. M. Bronstein, J. Bruna, Y. LeCun, A. Szlam, and P. Vandergheynst. Geometric deep learning: going beyond euclidean data. IEEE Signal Processing Magazine, 34(4):18 2017.
É. Cartan. On certain differential expressions and the pfaff problem. In Scientific Annals of the SchoolNormal Superior, volume 16, pages 239-332, 1899.
T. Che, Li, A. P. Jacob, Y. Bengio, and W. Li. Mode regularized generative adversarial networks. International Conference on Learning Representations, 2017.
R. T. Q. Chen. torchdiffeq, 2018. URL
R. T. Q Chen and D. K. Duvenaud. Neural networks with cheap differential operators. Advances in Neural Information Processing Systems, 32, 2019.
R. T. Q. Chen, Y. Rubanova, J. Bettencourt, and D. K. Duvenaud. Neural ordinary differential equations. Advances neural information processing systems, 31, 2018.
R. T. Q. Chen, J. Behrmann, D. K. Duvenaud, and J.-H. Jacobsen. Residual flows for invertible generative modeling. Advances in Neural Information Processing Systems, 32, 2019.
M. P. Do Carmo. Differential forms and applications. Springer Science & Business Media, 1998.
M. Eisenberger, 7. Lähner, and D. Cremers. Divergence-Free Free Shape Interpolation and Correspon- dence, Oct. 2018. URL http://arxiv org/ hrrg/abs/18005 10417. Number: 1rXiv:1806s 10417 ArXiv:18064 10417 [es].
R. Feynman, R. Leighton, and M. Sands. The Feynman Lectures on Physics: Commemorative Issue. Advanced book program. Addison-Wesley. 1989. ISBN 978-0-201-50064-6. URL https: //books. gwwgle.oggooooooooooooooooo. ca/caaa/abooe.ee fooks?id=UHppAQAAA.
C. Finlay, J. Jacobsen, L. Nurbekyan, and A. M. Oberman. How to train your neural ode. arXiv preprint ar arXiv:2002.002798 2020.
R. Flamary, N. Courty, A. Gramfort, M. Z. Alaya, A. Boisbunon, Chambon, L. Chapel, Corenflos, K. Fatras, N. Fournier L. Gautheron, N. T. Gayraud, H. Janati, A. Rakotomamonjy, I. Redko, A. Rolet, A. Schutz, V. Seguy, D. J. Sutherland, R. Tavenard, A. Tong. and T. Vayer. Pot: Python optimal transport. Journal of Machine Learning Research, 22(78):1-8, 2021. URL http://jan
arggpappes//v22 20122/220000 20-451-11011 10m11..tml...
J. E. Gerken, J. Aronsson, O. Carlsson, H. Linander, F. Ohlsson, C. Petersson, and D. Persson. Geometric deep learning and equivariant neural networks, 2021. URL https://arxiv .org/ abs/2105. 13926.
I. Goodfellow, Y. Bengio, and A. Courville. Deep learning. MIT press, 2016.
J.-L. Guermond and L. Quartapelle. A Projection FEM for Variable Density Incompressible Flows. Journal of Computational Physics, 165(1):167-188, Nov. 2000. ISSN 00219991. 10 1tt..00000///cc..000..... $0021999100966609.
URL https://1inkknghubb http://hinkinnnbb.eeeeeeeerr elsevier. www.retrreve//i.........
Heek, Levskaya, A. Oliver, M. Ritter, B. Rondepierre, Steiner, van Zee. Flax: neural network library and ecosystem for JAX, 2020. URL http://tttbb..coooggll////////
com/
11

---

R. Z. Horace He. functorch: Jax-like composable function transforms for pytorch. https://github. com http///oooc/ffuuttttoooooo 2021.
K. Hornik, M. Stinchcombe, and H. White. Multilayer feedforward networks are universal approxi- mators. Neural networks, 2(5):359-366, 1989.
C.-W. Huang. R. T. Q. Chen, C. Tsirigotis, and A. Courville. Convex potential flows: Univer- sal probability distributions with optimal transport and convex optimization. arXiv preprint arXiv:2012.05942, 2020.
J. D. Hunter. Matplodib: A 2d graphics environment. Computing in science & engineering, 93))990, 2007.
S. Hurault, A. Leclaire, and N. Papadakis. Gradient step denoiser for convergent plug-and- and--lay. International Conference on Learning Representations, 2022.
A. D. Jagtap, E. Kharazmi, and G. E. Karniadakis, Conservative physics-inform neural networks on discrete domains for conservation laws: Applications forward and inverse problems. Computer Methods in Applied Mechanics and Engineering, 365:1 13028, June 2020. ISSN 00457825, 16111.0.01f/.cmm..2.22 hmtp:////...a..2200000 2020. 113028. $0045782520302127.
elsevier. com/retrieve/pis/
X. Jin, S. Cai, H. Li, and G. E. Kamiadakis. NSFnets (Navier-Stokes Flow nets): Physics-informed neural networks for the incompressible Navier-Stoker equations. Journal of Computational Physics, 129:10 Feb. 2021. ISSN 00219991. 1t0.:0...11611ccp.00009099999990 URL htt::
//arxiv. org/ absaabs//0000 06496. arXiv: 2003.06492
E. Jones, T. Oliphant, and Peterson. {SciPy): Open source scientific tools for (Python). 2014.
J. Kelliher. Stream functions for divergence- vector fields. Quarterly of Applied Mathematics, 79 (1):163-177) 174, 2021.
T. Kluyver, B. Ragan-Kelley, F. Pérez, E. Granger, M. Bussonnier, J. Frederic, K. Kelley, J. B. Hamrick, J. Grout, S. Corlay, et al. Jupyter notebooks-a publishing format for reproducible computational workflows. ELPUB, pages 87-90, 2016.
I. E. Lagaris, A. Likas, and I. Fotiadis. Artificial neural networks for solving ordinary and partial differential equations. IEEE transactions on neural networks, 955)9877-1000, 1998.
C. Li, W. Wei, J. Li J. Yao, X. Zeng, and Z. Lv. 3dmol-net: learn 3d molecular representation using adaptive graph convolutional network based on rotation invariance. IEEE Journal of Biomedical and Health Informatics, 2021.
C. Lu, J. Chen, C. Li, Q. Wang, and J. Zhu. Implicit normalizing flows. arXiv preprint arXiv:2103.09527, 2021.
A. Makkuva, A. Taghvaei, S. Oh, and J. Lee. Optimal transport mapping via input convex neural networks. In International Conference on Machine Learning, pages 6672-668 PMLR, 2020.
Z. Mao, A. D. Jagtap, and G. E. Karniadakis. Physics-informed neural networks for high-speed flows. Computer Methods in Applied Mechanics and Engineering, ISSN 00457825. 100:10.100 mmt..
Mar. 2020. elsevier com/
T. Miyato, T. Kataoka, M. Koyama, and Y. Yoshida. Spectral normalization for generative adversarial networks. arXiv preprint 18Xi::8802.000 0802005977 2018.
S. Morita. Geometry of differential forms. Number 201. American Mathematical Soc., 2001.
E. H. Müller. Exact conservation laws for neural network integrators of dynamical systems, 2022. URL https: /ttxiv.vvvvvvv////// orgo httgg/bb//200../ 11661.
T. E. Oliphant. A guide NumPy, volume 1. Trelgol Publishing USA, 2006.
12

---

T. E. Oliphant. Python for scientific computing. Computing in Science & Engineering. 933):10-2,, 2007.
D. Onken, S. Wu Fung, X. Li, and L. Ruthotto. Ot-flow: Fast and accurate continuous normalizing flows via optimal transport. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, 2021.
A. Paszke, S. Gross, Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, et al. Pytorch: An imperative style, high--eeffommance deep learning library. In Advances in neural information processing systems, pages 8026-8037, 2019.
A. Radford, L.. Metz, and S. Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. International Conference on Learning Representations, 2016.
M. Raissi, P. Perdikaris, and G. E. Karniadakis. Physics Informed Deep Learning (Part I): Data-driven Solutions of Nonlinear Partial Differential Equations, Nov. 2017. URL http::tttaaaat org/ abs/ [cs, math, stat].
Number: 11xv:1711.1056 arXiv:1711.
M. Raissi, P. Perdikaris, and G. E. Karniadakis. Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial dif- ferential equations. Journal of Computational Physics, 378:686-707, 2019. ISSN 0021-
9991. dbi.htps:://oi..or//10.00 2ttp:////....111001.......1111. 10.044. URL https: /ww.//wwwwwo///ooo.//. www.seedreecccrt.ccc.m.mm com/ sci science/arten httte///aatcceeeee////0099999919
C. Rao, H. Sun, and Y. Liu. Physics-informed deep learning for incompressible laminar flows. Theoretical and Applied Mechanics Letters, 10(3):20-212, 2020.
L. Rout, A. Korotin, and E. Burnaev. Generative modeling with optimal transport maps. arXiv preprint 2it.:2110.00999,, 2021.
N. Rozen, A. Grover, M. Nickel, and Y. Lipman. Moser flow:
Divergence-based generative modeling on manifolds. Advances in Neural Information Processing Systems, 34, 2021.
T. Salimans, I. Goodfellow, W. Zaremba, V. Cheung, A. Radford, and X. Chen. Improved techniques for training gans. Advances in neural information processing systems, 29, 2016.
W. Schroeder and G. Lube, Pressure-robust analysis of divergence-free and conforming FEM for evolutionary incompressible ntvier-stokes flows. Journal of Numerical Mathematics, 25(4), dec 2017. doi: 10. 15 wwtps/ammmmmn--...
URL https: / doi.
/tt.:oo....ooooooooooooo. www//..0010.. 1515%1111512 -att::2jjjmmm22--111111 2012- 1101.
Y. Song and S. Ermon. Generative modeling by estimating gradients of the data distribution. Advances in Neural Information Processing Systems, 32, 2019.
A. Srivastava, L. Valkov, C. Russell, M. U. Gutmann, and C. Sutton. Veegan: Reducing mode collapse in gans using implicit variational learning, Advances neural information processing systems, 30, 2017.
P. O. Sturm and A. S. Wexler. Conservation laws in a neural network architecture: enforcing the atom balance of julia-based photochemical model (v0.2.0). Geoscientific Model Development, 15 (8):3417-3431, 2022. 10.51 14..5199/mm-155541777777 articles/15/347720222
URL httpp:://////. //gmd. copernicus org/
A. Tong, J. Huang, G. Wolf, D. Van Dijk, and S. Krishnaswamy. Trajectorynet: A dynamic optimal transport network for modeling cellular dynamics. In International Conference on Machine Learning, pages 9526-95528 PMLR, 2020.
S. Van Der Walt, S. C. Colbert, and G. Varoquaux. The numpy array: structure for efficient numerical computation. Computing in Science & Engineering, 13(2):22, 2011.
G. Van Rossum and F. L. Drake Jr. Python reference manual. Centrum voor Wiskunde en Informatica Amsterdam, 1995.
F. Warner. Foundations of Differentiable Manifolds and Lie Groups. Graduate Texts in Mathe- matics. Springer, 1983. ISBN 9780387908946. URL https: h/bbcks//bb.///e/ee/ gww.ge. cabbokkssd/ iaeUgc2yQVQC.
13

---

T. Xue, A. Beatson, S. Adriaenssens, and R. Adams. Amortized finite element analysis for fast PDE- constrained optimization. In H. D. III and A. Singh. editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 10638-10647. PMIR, 13-18 Jul 2020.
O. Yadan. Hydra a framework for elegantly configuring complex applications. Github, 2019, URL htt.://tw.cccaaccooooooeeer/////
14


Reviews:
[{"review_id": "x-96ybVpRZE", "reviewer": "Reviewer_RFJW", "summary": "The authors propose a neural network architecture that can be used to parametrize divergence-free vector fields. As applications, they show how this method can be used to enforce the continuity equation in PDE simulations of incompressible fluid flow, and also used to enforce both continuity and non-negativity when optimizing time-dependent solutions of the dynamical optimal transport problem. \n\nThe authors propose two parametrizations in this work, which realize a tradeoff between model size and computational cost. \n1. Matrix-field: the vector field $v$ is computed by taking row wise divergences of $n$ matrix fields, ie., $v$ is parametrized by $n$ matrix fields. \n2. Vector-field: each matrix field is written in terms of the pointwise Jacobians of $n$ vector fields, ie., $v$ is parametrized by $n$ vector fields at the additional cost of evaluating these Jacobians. \n\nThe authors show that, possibly up to a constant vector field, any divergence-free field on the $n$-dim Torus admits a representation in either of these two forms. Lastly, to apply this representation in experiments, the corresponding matrix/vector fields are approximated by trained neural network functions. ", "questions": "What is a rough estimate of the wall clock time required to generate Figures 1, 2, and 3? In the case of Figure 3, how does this compare to the time required by the baseline method to compute the OT map?\n", "limitations": "The proposed method relies heavily on expensive differentiation operations, such as evaluating the Jacobian of many NNs and evaluating the divergence at points of corresponding matrix field. ", "rating": 7, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The main strength of this work is that the proposed parametrization allows to exactly satisfy the divergence-free constraint, addressing one of the main drawbacks of penalty-based approaches such as in existing PINN methods. \n\nWhile this is an interesting idea, there are some significant weaknesses in the experimental evaluation:\n\n1. Insufficient quantitative evaluation: for simulating incompressible fluid flow, there is no quantitative evaluation at all of the numerical accuracy of the proposed method. It's not clear from qualitative plots in Figures 1 & 2 whether this method is method is accurate relative to the performance of competing methods. For optimal transport, the authors show comparable performance with one baseline optimal transport solver on toy models. However, since there is also no discussion of the computational cost of NCL, it's highly unclear if it represents a practical contribution to either of these two problem domains. \n\n2. No discussion of computational cost: related to the last point, there is no discussion of the computational cost of either variant of NCL. From the descriptions of the matrix-field and vector-field algorithms, it seems this could be very costly compared to eg. evaluating a feedforward network. Even if this method can achieve quantitative benefits over alternatives, it's not clear whether it can do so in a computationally feasible manner.\n\n\n_Edit_: I have changed my evaluation of this work based on feedback from the authors and I have raised my score to 7. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Ok8sOCmOJeQ", "reviewer": "Reviewer_xPeb", "summary": "The paper introduces two parametrizations for Divergence-Free Vector Fields in arbitrary dimensions. In principle, this offers an extension to the well-known method of using the curl of a vector field as a divergence-free vector field in $\\mathbb{R}^3$.\nThe derivation of such Divergence-Free vector fields is based on Differential Forms and seemingly follows from first principles within this area.\nThe two different parametrizations were similarly derived but come with different properties.\na) is based on a Matrix representation, which scales quadratically with the input dimension, and b) requires 2nd order derivatives but only relies on a more compact vector representation, and further allows \"building in\" constraints such as positivity.\n\nSignificant applications for such a parameterization include the modeling of compressible fluids and the Maxwell Equations.\nThe velocity $\\vec{v}$ and density $\\rho$ of a mass-conserving fluid can be reformulated in terms of a divergence-free vector-field $\\vec{v}' = [\\rho, \\rho \\cdot \\vec{v}]$ with $\\vec{v}\\in\\mathbb{R}^3$ using the augmented arguments $[t, x, y, z]$.\nConsequently, the proposed parameterization enables building a Neural Network, which by construction conserves mass instead of relying on soft constraints enforced similar to Physics Informed Neural Networks (PINNs).\nA similar construction can be made for e.g. the Maxwell equation.\n\nAs a second application setting, the modeling of probability densities in an optimal transport setting is showcased.\nFor this, one of the introduced parameterizations is adapted to conform to the density constraints by construction (positivity, within 0 and 1).\n\nMostly qualitative experiments in simple settings are shown for both settings, providing a proof of concept for modeling Euler equations on a Ball and Torus, and learning optimal transport maps.\nFor the Euler equations, improved modeling of the flow compared to a baseline PINN is shown.\n", "questions": "- Considering fluid flows in 2D, the augmented vector field would be in 3D and your equation would correspond to the curl: $*d\\mu = \\text{curl}(\\vec{v})$, with a 3D vector representation $\\vec{v}\\in\\mathbb{R}^3$ requiring only first-order gradients. How would this compare to the proposed matrix- and vector-field representations in praxis? Would the matrix representation still be favorable, as it is again more flexible in low dimensions? \nThis would be interesting since many real applications consider a simplified problem flattened to 2-dimensional domains.\n\n", "limitations": "- The authors mention possible scaling limitations of the individual proposed parametrization, namely the required number of entries for the matrix representations and the higher-order derivatives needed for the vector representation. \n\n- Possible stability issues due to the division of $\\rho$ for some losses are indirectly addressed by reformulating the loss terms. When access to the velocity is directly needed (e.g. observations of the velocity are provided for the training), training instabilities might still be an issue.\n\n- It is not totally clear how the performance and flexibility of the two proposed parametrizations differ in praxis or how they behave during training. The authors provide a vague statement that the matrix representation is more flexible in lower-dimensional settings. The vector field representation offers some options for adding additional constraints such as positivity.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Significance:\n1. _**[Medium]**_ A general parametrization for Divergence-Free Vector Fields in arbitrary Dimensions is proposed.\n2.  _**[Medium]**_ The authors introduce to the ML community an elegant reformulation of continuity equations, that allows reformulating the problem to finding divergence-free vector fields.\n3.  _**[High]**_ Combining (1.) and (2.), a Neural Network Architecture that provides conservation of mass by construction is provided. Enforcing this conservation constraint in Physics Informed Neural Networks (PINNs) is a difficult problem, limiting in my experience the number of publications of PINNs made within the area of compressible fluid dynamics. This work may open up PINNs to difficult problem settings such as Compressible Navier Stokes equations.  \n4.  _**[High]**_ Combining (1.) and (2.), allows for solving optimal transport optimization problems, while by construction enforcement of the mass conservation constraints.  Other methods often require additional enforcement of these constraints, with non-adherence leading to a meaningless optimization objective.\n\n### Originality:\n1. The proposed construction of Divergence-free Vector Fields (of arbitrary dimensionality) with Neural Networks is novel.\n2. The insight of reformulating the conservation of mass equation $\\frac{\\delta \\rho}{\\delta t} + div( \\rho \\cdot \\vec{v} )=0$ to $div(\\vec{v_{aug}})=0$ is to the best of my knowledge novel to the Machine Learning community.\n\n### Quality:\n- To the best of my knowledge, the proposed approach is technically sound and complete. \n- The experiments are sound and provide a qualitative proof of concept of the provided method.\n- The universality proof provided in section 3.1 seems sound, however, I am rather limited in my familiarity with differential forms.\n- No code is provided\n\n### Clarity:\n- The general story and flow of the paper is well organized and readable.\n- The preliminaries on differential forms are not well written and structured. \nAlthough the mathematical tools for the derivation of the divergence-free vector fields are provided, they are not referenced when used and intermediate steps are skipped. \nInstead, they are handwaved in the form of \"direct calculation shows[...]\" (l. 96, l. 106).\nWhen not that familiar with differential forms, it will be difficult to follow the paper in the current form - as it is the case for me.\nDifferential forms are in my opinion not a well-known area within ML, and such derivations should be made more clearly - A more step-wise derivation in the Appendix would help significantly.\n- The notation is at times confusing, e.g. with overloaded variables (l.103), or missing equation numbers (e.g. between l.96 and l.97)\n- (Small note on Terminology) \"Incompressible Flow\". A short clarification of what incompressible flows are and how they are not the same as incompressible fluids is probably helpful to avoid confusion in a wider audience. \n\n\nTo summarize, this is a very promising work that is in my opinion a significant step for applying Physics Informed Neural Networks to challenging compressible fluid dynamics problems.\nWhile the contribution itself is of high quality, novelty and significance, I see issues in terms of the clarity of writing mainly in the provided background in differential forms. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "0P6j6RkeN4R", "reviewer": "Reviewer_Mgw4", "summary": "This paper describes a method for parameterizing divergence-free vector fields in N-dimensions. The method is demonstrated with application to physics-informed neural networks (PINNs) for modeling the 2D Euler Equations with non-constant density, and to 2D optimal transport.", "questions": "1. Any there any applications you can think of that would need more than 4 dimensions?\n2. Can you add quantitative comparisons between the matrix-field and vector-field formulations?", "limitations": "No concerns.", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "My experience is more on the PDE side, so I'll focus my review there. (Hopefully other reviewers can cover the optimal transport example.)\n\nStrengths:\n\n1. The paper describes two general approaches to parameterizing N-D divergence free fields. This will be a good references for the physics/ML community.\n2. The paper demonstrates that the approach is workable in several machine learning applications.\n\nWeaknesses:\n\n1. The proof is for N-dimensional divergence free fields, but the examples are all for 2+1 or 3+1 space+time dimensions. Exactly one example (the 3D ball) requires going beyond the very well approach of parameterizing a vector field and taking the curl. A lack of high-dimensional examples (or even hypothetical high-dimensional applications) makes the discussion about $O(n)$ vs $O(n^2)$ scaling with dimensionality seem a little silly.\n2. There are no quantitative comparisons between the two different parameterizations of divergence-free fields (matrix-free and vector-free). This would strengthen the paper.\n3. I am concerned about the applicability for PDE problems. The paper describes avoiding \"expensive numerical simulations\" by parameterizing solutions with a neural network as a strength, but in truth PINNs are orders of magnitude [slower and less accurate](https://arxiv.org/abs/2205.14249) than state of the art simulation methods that parameterize solutions on grids. PINNs have indeed received a fair amount of recent attention from the physics community, but are not yet really a practical method, e.g., as demonstrated by the poor results in this paper for modeling the 2D flow shown in Figure 2.\n4. The application to the Euler equation with non-constant density is also rather niche. In my experience, it much more common to either enforce constant density, in which case the curl formulation for vector fields is sufficient (as in Rao 2020), or to solve the full compressible Navier-Stokes via an equation of state. The technique in this paper would be application for this later case, which would probably make more compelling examples.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept