Learning Vortex Dynamics for Fluid Inference and Prediction
Authors: Yitong Deng
Hong-Xing Yu
Jiajun Wu
Bo Zhu
Venue: ICLR 2023 poster (2023)

Abstract: We propose a novel differentiable vortex particle (DVP) method to infer and predict fluid dynamics from a single video. Lying at its core is a particle-based latent space to encapsulate the hidden, Lagrangian vortical evolution underpinning the observable, Eulerian flow phenomena. Our differentiable vortex particles are coupled with a learnable, vortex-to-velocity dynamics mapping to effectively capture the complex flow features in a physically-constrained, low-dimensional space. This representation facilitates the learning of a fluid simulator tailored to the input video that can deliver robust, long-term future predictions. The value of our method is twofold: first, our learned simulator enables the inference of hidden physics quantities (e.g., velocity field) purely from visual observation; secondly, it also supports future prediction, constructing the input video's sequel along with its future dynamics evolution. We compare our method with a range of existing methods on both synthetic and real-world videos, demonstrating improved reconstruction quality, visual plausibility, and physical integrity.

Published as a conference paper at ICLR 2023
LEARNING VORTEX DYNAMICS FOR FLUID INFERENCE AND PREDICTION
Yitong Deng
Hong-Xing Yu
Jiajun Wu
Bo Zhu
Dartmouth College
Stanford University
Stanford University
Dartmouth College
Stanford University
ABSTRACT
We propose a novel differentiable vortex particle (DVP) method to infer and pre- dict fluid dynamics from a single video. Lying at its core is particle-base la- tent space to encapsulate the hidden, Lagrangian vortical evolution underpinning the observable, Eulerian flow phenomena. Our disferentiable vortex particles are coupled with a learnable, ture the complex flow features in physically conpccallyy.stttrinned.... liw-dimensional space. This representation facilitates the learning of a fluid simulator tailored to the in- put video that deliver robust, long-term future predictions. The value of our method is twofold: first, our learned simulator enables the inference of hidden physies quantities (e.g. velocity field) purely from visual observation; secondly, it also supports future prediction, constructing the input video's sequel along with its future dynamics evolution. We compare our method with a range of existing methods on both synthetic and real-world videos, demonstrating improved recon- struction quality, visual plausibility, and physical integrity
dynamics mapping to effectively cap-
INTRODUCTION
As small as thin soap films, and large as atmospheric eddies observable from outer space, fluid systems can exhibit intricate dynamic features on different mediums and scales, However, despite recent progress, it remains an open problem for scientific machine learning to effectively represent these flow features, identify the underlying dynamics system, and predict the future evolution, due to the noisy data, imperfect modeling, and unavailable, hidden physics quantities.
Here, we identify three fundamental challenges that currently hinder the success of such endeavors. First,flow flow features are difficult to represent. Traditional methods learn the fluid dynamics by storing velocity fields either using regularly-spaced grids smooth neural networks. These approaches have demonstrated promising results for fluid phenomena that are relatively damped and laminar (e.g..Chuu Chu al. 2022), but for fluid systems that can exhibit turbulent features varying scales, these methods fall short due to the problem's 's curse of dimensionality (high-resolution space and time), local non-smoothness, and hidden constraints. As result, more compact and structured representation spaces and data structures are called for.
Secondly, hidden flow dynamics is hard to learn. Fluid systems as prescribed by the Navier-Stokes equations tightly couple multiple physical quantities (i.e., velocity, pressure, and density), and yet, only the density information can be accessibly measured. Due to the system's 'ss complexity, ambiguity, and non-linearity, directly learning the underlying dynamics from the observable density space is infeasible; and successful learning usually relies on velocity or pressure supervision, a requirement that distances these methods from deployment in real-world scenarios.
Exciting recent progress has been made in hidden dynamics inference by PDE-based frameworks such as Raissi et al. (2020), which uncover the underlying physics variables solely from density observations. However, this type of methods encounter the third fundamental challenge, which is that performing future prediction is difficuls. As we will demonstrate, although strong results
Our video results, code, and data can be found at our project wehsite: httpa:://yitongdeng.
g itthbb.oovorrex__ee hatabb/oooooorrre_aaaaaaannnaaan

---

Published as a conference paper at ICLR 2023
Observed real video
Synthesized future prediction
Figure 1: Our goal is to learn vortex dynamics for fluid inference and prediction. The 3 frames on the left are observed from real video of a soap film on a circular metal rim, where the red ink is spreading. The 3 frames on the right are future prediction results produced by our method.
are obtained for interpolating inside the observation window provided by the training data, these methods cannot extrapolate into the future, profoundly limiting their usage.
In this paper, we propose the differentiable vortex particle (DVP) method, a novel fluid learning model to tackle the three aforementioned challenges in a unified framework. In particular, har- nessing the physical insights developed for the vortex methods in the computational fluid dynamics (CFD) literature, we design novel, data-driven Lagrangian vortex system to serve compact and structured latent representation the flow dynamics. We learn the complex dynamical system underneath the high- -inheseinniio image space by learning a surrogate, low- -ow-dimeniional model on the latent vortex space, and use physics-based, learnable mechanism: the vortex-to-vele toreex-to-velocity mod- ule, decode the latent-space dynamics back to the image space. Leveraging this physics-based representation, we design an end-to-end training pipeline that learns from a single video containing only density information. Our DVP method enables accurate inference of hidden physics quantities and robust long-term future predictions, as shown in Figure [
To examine the efficacy of our method, compare our method's performance on both motion in- ference and future prediction against various state-of-the-art methods along with their extensions. We conduct benchmark testing on synthetic videos generated using high-order numeric simulation schemes as well as real-world videos in the wild. Evaluation is carried out both quantitatively through exhaustive numerical analysis, and qualitatively by generating range of realistic visual effects. We compare the uncovered velocities in terms of both reconstruction quality and physical integrity, and the predicted visual results in terms both pixel-level and perceptual proximity. Re- sults indicate that our proposed method provides enhanced abilities both fronts, inferring hidden quantities at higher accuracy, and predicting future evolution with higher plausibility.
In summary, the main technical contributions of our framework align with the three challenges regarding flow representation, dynamics learning, and simulator synthesis. (1) We devise a novel representation for fluid learning, the differentiable vortex particles (DVP), drastically reduce the learning problem's 's dimensionality on complex flow fields. Motivated by the vortex methods in CFD, establish the vorticity-carrying fluid particles existing PDE-constrained optimization problem to particle trajectory (ODE) learning problem. (2) We design a novel particle-to-field paradigm for learning the Lagrangian vortex dynamics. Instead of learning the interaction among particles (e.g.. Sanchez-Gonzalez et al.) (2000), our model learns the continuous to-ee-tt-velocity induction mapping to naturally connect the vortex particle dynamics in the latent space with the fluid phenomena captured in the image space. (3) We develop an end- to-end differentiable pipeline composed of two network models synthesize data-driven simulators based on single, short RGB videos.
type of learning primitive transform the
RELATED WORK
Hidden Dynamics Inference. The problem of inferring dynamical systems based on noisy or incom-
plete observations has been addressed using a variety of techniques, including symbolic regression
(Bongard &Lip&piia Lipson| 2007 (Schmidt& & Lipson] 2009), dynamic mode decomposition (Schmid) 2010
Kutz et al. 2016)). sparse regression (Brunton al. 2016 Rudy al. (007)), Gaussian process
gression (Raissi et al. 2017 Raissi & Karniadakis 2018), and neural networks (Raissi et al. 2019
Yang et al. 2020: Jin al. 2021 Chu al. (022). Among these inspiring advancements, the "hid-
den fluid mechanics" (HFM) method proposed in Raissi et al. (2020) is particularly noteworthy, as it uncovers the continuous solutions of fluid flow using only images (the transport of smoke ink).
Data-driven Simulation. Recently, growing interests are cast on learning numerical simulators ac- cording data supervision, which has shown promise reduce computation time (Ladicky al.,
2

---

Published as a conference paper at ICLR 2023
2013
Guo et al. 2016 Wiewel et al. 2019; Pfaff et al. 2020 Sanchez- et al. 2020 Tomp-
son et al. 2017), ) increase simulation realism (Chu & Thuerey 2017] 2017 al. 2)) enable stylized
control Kim al. 2020), estimate dynamic quantities such as viscosity and energy (Chang al. 2016 Battaglia et al. 2016; Ummenhofer et al. 2019), and facilitate the training of control policies (Sanchez-Gonzalee
al., 2018; Li al. 2018). Akin to Watters al. 2017), our system takes im- ages as imputs and performs dynamics simulation on low-dimensional latent space; but our method learns purely from the input video and performs future rollout in the image space, Our method is also related to Guan et al. (2022), which infers Lagrangian fluid simulation from observed images. We propose sparse neural vortices as our representation while they use dense material points.
Vortex Methods. The underlying physical prior incorporated in our machine learning system is rooted in the family of vortex methods that are rigorously derived, analyzed, and tested in the com- putational fluid dynamics (CFD) (Leonard, 1980; Perlman 1985 [eale & Majda, 1985 Winck- elmans & Leonard. 1993
Mimeau & Mortazavi, 2021) and computer graphies (CG) (Selle et al.;
Weißmann & Pinkall 2010 Brochu et al.) 2012) communities. Xiong et al. 2020) is pioneering for combining the Discrete Vortex Method with neural networks, but its
2005 Park & Kim 2005
proposed method relies large set of ground truth velocity sequences, whereas method learns from single videos without needing the ground truth velocity.
PHYSICAL MODEL
We consider the velocity- -votticty form of the operator of both sides of the momentum equation, see Cottet et al. (2000) for details):
equations (obtained by taking the curl
Dw
Ow
ōt + u Ve w
V..Vu Vu+ + +
Dt
u v=Vx6 x Φ マタzーEs
(2)
where w denotes the vorticity, u the velocity, b the conservative body force, the kinematic viscos- ity, and &i the streamfunction. If we ignore the viscosity and stretching terms (inviscid 2D flow), we obtain Dw/ Dt = O, which directly conveys the Lagrangian conservative nature of vorticity (i.e., a particle's vorticity will not change during its advection).
If we assume the fluid domain has an open boundary. we can further obtain the vortex-to- to-clocctt induction formula, which is derived by solving Poisson's 's equation on &f using Green's 's method (also known as the Biot-Savart Law in fluid mechanics):
u(x) = K(x (x))(x")dx")
The kernel K exhibits a type-Il singularity at O and causes numerical instabilities, therefore in CFD practices, K is replaced by various mollified versions K3 improve the simulation accuracy (Beale & Majda 1985). We note that the mollified version Kg is not unique, and can be customized and tuned in different numerical schemes per human heuristics. Different types and parameters for the mollification bring about significantly different simulation results.
Takeaways. The mathematical models above provide two central physical insights guiding the design of our vortex-basee based learning framework: (1) The Lagrangian conservation of vorticity w suggests the suitability of adopting Lagrangian data structures (e.g., particles as opposed to grids) to capture the dynamics. Since the tracked variable w remains temporally invariant for each Lagrangian vortex, the evolution of the continuous flow field is embodied fully by the movement of these vortices, which significantly alleviates the difficulty in learning. (2) Equation@]preseee
an induction mapping from the vorticity w, a Lagrangian quantity carried by particles, to the velocity u, an Eulerian variable that can be queried continuously an arbitrary location æ. This lends the possibility for the Lagrangian method to be used in conjunction with Eulerian data structures (e.g., a grid) for learning from the widely available video data. Furthermore, such mapping can benefit from data-driven learning, as we can replace the human heuristics by learning a mollified kernel Ka to minimize the discrepancy between the simulated and observed flow phenomena.
4 METHOD
System Overview. Following the physics insight conveyed in Section B
we design a learning system whose workflow is illustrated in Figure 2 As shown on the top row, our system takes as input
3

---

Published as a conference paper at ICLR 2023
Analyzing The Past
Predicting The Future
Observed Image Space
Innegrator
Incegratr
Integrator
Integrator
Intermediate Velocity Space
Trajectory Module
Lagrangin
Lugrangan Inoegrator
Dymamics
Module
Module
Vortex Space
Illustration of our differentiable vortex particle (DVP) method. Given an input RGB image sequence (top row), we learn dynamical system low-dimensional vortex space (bottom row), whose motion is decoded into the motion of the dibensiinnsional image space explain the observed fluid phenomena.
Figure 2:
single RGB video that captures the vortical flow phenomena. shown on the bottom row, our method learns and outputs dynamical simulator not the image space itself, but latent space consisting of discrete vortices. Learning the latent dynamics the vortex space would only be useful and feasible if we tie it back to the image space, because it is the image space that want to perform future prediction on, and we have no ground truth values for the vortex particles to begin with. The bridge to tie the vortex space with the image space derives from Equation which supplies the core insight that there exists a learnable mapping from vortex particles to the continuous velocity field arbitrary positions. This mapping is modeled by our learned dynamics module D, which gives rise to the intermediate velocity space, as shown in the middle row of Figure
4.1 DIFFRENTIABLE VORTEX PARTICLES
We track collection V of n vortex particles, i.e., V = the 3-tuple (xx, w;, 64). where a represents the position, w the vortex strength, and the size. The number of particles n is hyperparameter which we set to 16 for all our results. Further discussions and experiments regarding the choice of n can be found in Appendix [ We also note that, since concerned with 2D inviscid incompressible flow, the size 8 of vortex does not change in time due to incompressibility. and the vortex strength w does not change in time due to Kelvin's circulation theorem (see Hald (1979) for thorough diseussion).
We define each vortex V, as
Learning Particle Trajectory. As shown in Figure 3, we learn a particle trajectory module: query function T such that V/ T(t), which predicts the configuration of all the vortices at any time I E [0. 4e] where i,; represents the end time of the input video. As described above, predicting V, boils down determining time-varying component: [xx1)...... (xn)e)- For the two time-invariant components, we introduce two trainable
time-invariant components: (1) [wi,.... w.). wx). [61, ......... 6,), and one
vectors A and Q to represent ö and w respectively, such that [wi.... Wi....,Wn! = (S2) and [81,.... 8n] = sigmoid( A) + e (e is hyperparameter we set to 0,03). The vortex size A and strength f2 are optimized fit the motion depicted by the input RGB video. For the time- varying component, we use a neural network N1(t) (t) to encode (t) = [(xI)...... (æ,))), and dr can be extracted using automatic differentiation. We note that learning the full particle trajectory, rather than the initial particle configuration, allows the aggregation of dynamics information throughout the input video for better inference and prediction. We provide further discussion on this design in Appendix
the particle velocities dNr
Trajectory Initialization. As discussed above, the trajectory T has three learnable components: A. and N1. We initialize A and as zero vectors, which gives 6, = 0.5 + e and w; = for all initialized large blobs with vortex strength, which leam to
Conceptually, these vortices alter their sizes and grow their strengths to better recreate the eddies seen the video. The initial positions ( (==)),,... (x))o) are regularly spaced points populate the entire domain. We initialize

---

Published as a conference paper at ICLR 2023
We encapsulate the motion of a continuous field by the motion of discrete particles. The blue trajec- tory is encoded by neural network Ni, corresponding to the input video; while the red trajectory is unrolled using learned dynamics module and numeric integrator, corresponding the future prediction.
Figure 3:
the 16 particles lie the grid centers of 4 grid. To do so, we simply pretrain N1 so that Ni (0) evaluates to the grid centers. The details regarding pretraining are given in Appendix A
Learning the Vortex-to- Velocity Mapping. The vortex-to-veloche mapping is performed by our dy- namics module D, which predicts the velocity u at an arbitrary query point æ given the collection of vortices [ wi, (1),,.1)...... ..., (I Wn; 8n)]. Following the physical insight conveyed in Section
D should evaluate the integration:
= f K5(w - x'' =z')wa''dda'
u (x)
(4)
which replaces the kernel K in Equation 3 by a learnable mapping Kg Ti. - TRd, with d repre- senting the spatial dimension. Rather than directly using neural network to model this Rd - R mapping, we incorporate further physical insights by amalyzing the structure of Ks. As derived in Beale & Majda (1855), the kernel K, for 2-dimensional flow exhibits the following form:
1
Kx(z) =
M (1, 2πr
(5)
where Rg (z) computes the unit direction of the cross product of and the out- out-of-plane planne unit vector; and (r, 8) is the human heuristic term that varies by choice. Hence, we opt to replace 2eF
2
M
by a I² + IR neural network function (2,66) so that:
u(x)= = / N2 - æ' &,), (x - x' (')w(x')da
(6)
Na(1 8.). (a az)ws = ((V)
Learning this induction kernel (2 8) instead of using heuristics-based kernels allows for more accurate fluid learning and prediction from input videos. We discuss more on this in Appendix B
4.2 END-TO-END TRAINING
As previously mentioned, the dynamics on the latent vortex space is bridged to the evolution of the image space through the differentiable, dynamics module D. Hence, we can optimize the vortex representation V4 T()) at time using images as supervision. First, we select m + 1 frames: (Tr...., [t+m] from the video. Then, we compute u, = D(V.). After that, (u,, T,) is fed into an integrator on the Eulerian grid to predict Ie+1. Simultaneously, (u,, V.) is fed into an integrator on the Lagrangian particles to predict D4-1- The process is then repeated, using I+41 in place of Ie and Vi41 in place of V,, to generate T, and V/42, and so on. Eventually, we would obtain [I+......, ]. which are the predicted outcome starting at time t. We optimize T and D jointly by minimizing the difference between [7,4,,..., T++m] and [T+1.....,T4t J14m] in an end-to-end fashion.
By picking different values of t in each training iteration cover [0, tg], optimize T and ' to fit the input video. There remains one more caveat - which is that the trajectories encoded with T are not enforced to be consistent with D, because each frame of V, is optimized individually. In other words, if evaluate the particle velocities [at ...,, znn] dN1 prescribed by T.
5

---

Published as a conference paper at ICLR 2023
Extrap.
UNet
Ours
HRM Extrap.
UNet
ER UNet
Ours
Figure 4: Applied to real-world videos. our DVP method can create more realistic future predictions over long periods of time compared to existing methods (and their extensions).
Velocity
Velocity Residue
Ground Truth
E-R
HFM
UNet
Ours
Figure 5: Hidden motion inference compared with existing methods on synthetic video. Our DVP method uncovers the underlying velocity field at higher accuracy.
should coincide with (D(V) (xm)l, as prescribed by D. Hence, in training, another loss is computed between dNr and (D))) ((V)(E.)) to align the vortex trajectory and the predicted velocity.
dt
Deployment. After successful training, our learned system performs two important tasks. First, us- ing our query function (t). we are able temporally interpolate for V,, which then uncovers the hidden velocity field u D(V;) at arbitrary resolutions, providing the same functionality as Raissi Karniadakis (2018), but using vorticity instead pressure as the secondary variable. Moreover, with the dynamics module D, we can perform future prediction to unroll the input video, feature unsupported by previous methods. As shown in Figure 4 since our method by nature, it can provide more realistic and robust future predictions than existing methods their extensions. Further implementation details of our method, including hyperparameters, network ar- chitectures, training schemes, and computational costs be found in Appendix
EXPERIMENTS
We evaluate our method's ability to perform motion inference and future prediction on both synthetic and real videos, comparing against existing methods.

---

Published as a conference paper at ICLR 2023
Motion
Inference
Errors
Future
Prediction Errors
18
1508
Figure 6: Error analysis synthetic video. The top row plots the inference emrors of velocity, vorticity, and compressibility. The bottom plots the future prediction errors, which consider both the dynamics error of the velocity and the perceptual error of the generated image sequence.
Baselines. For motion inference, we compare our method against Raissi Karniadakis (2018) (HFM) and Zhang et al. (2022) (E-R). We reimplement the HFM method preseribed in the paper, making only the modification that instead of using only a single concentration variable c and its corresponding variable d == 1 C, create three (c, d) pairs for each of the RGB channel for the support of colored videos. The E-R method evaluated using the published pretrained models. We further compare against an ablated version of our proposed method, termed "UNet", which es- sentially replaces the Lagrangian components of the system with UNet architecture (Ronneberger al. 2013), a classic method for learning field-to-field mappings. The UNet baseline takes two im- ages L and 1++1 and predicts a velocity field Us+1 to predict 4+2 using the same Eulerian integrator as our method. For future prediction, there do not exist previous methods that operate in comparable settings, so we extend the inference methods in a few ways to support future prediction in a logical and straightforward manner. First, since HFM offers a query function parameterized by f, we test its future prediction behavior by simply extrapolating with t tg: this is referred to "HFM extp.". Since both Raissi & Karniadakis 2018) and Zhang et al. (2022) uncover the time-varying velocity field, a UNet to learn the evolution from u, u4+1, and use this velocity update mechanism to perform future prediction. The two baselines thus obtained are referred to as "HFM+UNet" and "E-R+UNet" respectively. Our method's 's ablation "UNet" supports future prediction intrinsically.
5.1 SYNTHETIC VIDBO
The synthetic vidco for vortical flow is generated using the Discrete Vortex Method with a first- order Gaussian mollifying kernel M (Beale & Majda 1985). The high-fidelity BFECC advection scheme (Kim et al. 2005) with Runge-Kutta-3 time integration is deployed. The simulation advects background grid of size 256 x 256, with a time step dt 0.01 create 300 simulated frames. Only the first 100 frames will be disclosed to train all methods, and future predictions are tested and examined on the following 200 frames.
Motion Inference. The results for the uncovering of hidden dynamic variables are illustrated in Figure 3 Figure Shown in Figure are the velocities uncovered by all 4 methods against the ground truth, at frame 55 of the synthetic video with 100 observed frames. The velocity is visualized the forms of colors (top row) and streamlines (middle row), while the velocity residue, measured in end- and-point error (EPE), is depicted on the bottom row. It can be seen that HFM, UNet, and our method achieve agreeing results, bottom row, it can be seen that as compared to HFM and UNet, our method generates the inference velocity that best matches the unseen ground truth.
matching the ground truth values at high accuracy. On the
The inference results over the full 100 frames are depicted the top of Figure 6 the velocity with four metrics: the average end-point error (AEPE), average angular error (AAE), vorticity RMSE and compressibility RMSE. From all 4 metrics, it be seen that method outperforms the baselines consistently. The aveereee data for all four metrics are shown on the left of Table [ which deems our method favorable for all metrics used.
We evaluate
Future Prediction. In Figure 7 we visually compare the future prediction results (from frame 100 to frame 299) using method and the 4 benchmarks against the ground truth. It can be seen that the

---

Published as a conference paper at ICLR 2023
t= 1.00
t=
t= 2.32
2.98
Ground Truth
HFM Extrap.
HFM + UNet
ER UNet
UNet
Ours
Figure 7: Future prediction of our DVP method compared to baselines methods. Our method accurately predicts the unseen, future sequence that is twice as long as the seen sequence.
Time-avcraged Inference Errors
Time-averaged Prediction Errors
AEPE
AAE
Vort.
Div.
VGG
RMSE
AEPE
AAE
Vort.
Div.
E-R
0.505
1.393
8.470
2.319
+UNet
4.346
0.205
0.631
1.424
12.84
6,580
+UNet
4.258
0.205
0.720
1.062
36.73
10.41
HFM
0.100
0.212
3,949
0.202
Exip.
4.080
0.285
0.541
1.464
7.761
4.315
UNet
0.048
0.100
1.799
1.145
4.530
0.211
0.424
1.159
7,334
3,017
Ours
0.020
0.041
0.976
0.053
2.010
0.080
0.048
0.096
1.621
0.043
Table 1: Error analysis of benchmark testing on a synthetic dataset.
sequence generated by method best matches the ground truth video, capturing the vortical flow structures, while the other baselines either quickly diffuse or generate unnatural, hard-edged pat- terns. Numerical analysis confirms these visual observations, in terms both velocity and visual similarity. The velocity analysis inherits the same 4 metrics, and the visual similarity gauged using the pixel-level RMSE and the VGG feature reconstruction loss (Johnson et al., 2016). The time-averaged results of all 6 metrics are documented on the right of T and the time-varying results plotted on the bottom of Figure 6 It can be concluded from the visual and numerical evidence that our method outperforms the baselines in this case.
compare the 200 future frames
5.2
REAL VIDRO
A similar numerical analysis is carried out on a real video published on YouTube, as shown in Fig- 8 The video has 150 frames: the first 100 frames will be used for training, while the remaining 50 frames will be reserved for testing. Since the ground truth velocities for the real video are in- tractable, we will only analyze the nuture-predictio
performance. For all methods. we perform future prediction for 150 frames; among these, the first 50 frames will be compared with the original video, and the rest (100 frames) will be evaluated visually and qualitatively. Since only part of the video is fluid (within the circular rim), that only the fluid regions are considered in learning and simulation. The same boundary condition employed for all methods (except for "HFM extp." which requires advection).
pre-generate signed distance field for all methods, so
The numerical analysis for the first 50 predicted frames is documented and plotted in Table 2 and Figure g We compare method against the baselines based the VGG perceptual loss for visual plausibility, and the velocity divergence (which should in theory be for incompressible fluid) for physical integrity. It be that method prevails on all metrics used. For prediction
8

---

Published as a conference paper at ICLR 2023
t= 1.00
t=
t= 1.98
N/A
N/A
Ground Truth
HFM Extrap. HFM + UNet
ER UNet
UNet
Ours
Figure 8: Future prediction of our DVP method compared to baseline methods on real video sequence. Our method generales predicted sequence that best matches the input video within its duration, and remains visually plausible way beyond its duration.
perco sheal error time
Compress bally
VGG (avg.)
VGG (final)
Div.
10h4
(avg)
Misi
E-R
2.095
2.205
2.046
HFM+UNet
2.151
2.231
0.940
HFM Extp.
2.980
3.271
1.922
UNet
2.111
2.088
1.447
Ours
2.093
2.045
0.318
20
Figure Error plots corresponding to Figure
Table Time-averaged errors Figure
results that exceed the duration of the real video, qualitative observations can be made: our method preserves the vortical structures and generates smooth visualizations over the entire time horizon, while other methods end up yielding glitchy patterns.
We perform additional quantitative benchmark testings in Appendix B against differentiable grid- based simulator on real and synthetic videos; and in Appendix C
against baselines on another synthetic video featuring different visual and dynamical distributions.
CONCLUSION & LIMITATIONS
In this work, propose novel data-driven system to perform fluid hidden dynamics inference and future prediction from single RGB videos, leveraging a novel, vortex latent space. The success of method in synthetic and real data, both qualitatively and quantitatively, suggests the potential for embedding Lagrangian structures for fluid learning. Our method has several limitations. First, our vortex model currently limited 2D inviscid flow. Extending to 3D, viscous flow is an exciting direction, which can be enabled by allowing vortex strengths and sizes to evolve in time (Mimeau & Mortazavi, 202T). Secondly, our vortex evolution did not take into account the boundary conditions in physically-basee manner, hence it cannot accurately predict flow details around solid bound- ary. Incorporating learning-basee boundary modeling may be an interesting exploration. Thirdly, scaling our method to handle turbulence with multi-scale vortices remains to be explored. We con- sider two additional directions for future work. First, we plan to explore the numerical accuracy of neural vortex representation to improve the current vortex particle methods for scientific com- puting. Secondly, we plan to combine our differentiable simulator with neural rendering methods to synthesize visually appealing simulations from 3D videos.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGMENTS
We thank all the anonymous reviewers for their constructive feedback. This work is in part supported by ONR MURI N00014-22-1-27400 NSF RI #221 1258, #191464119977 #2106733, #2144806, #2153560, the Stanford Institute for
AI (HAI), Google, and Qualcomm.
REFERENCES
Peter Battaglia, Razvan Pascanu, Matthew Lai, Danilo Jimenez Rezende, et al. Interaction networks for learning about objects, relations and physics. Advances in neural information processing systems, 29, 2016.
J Thomas Beale and Andrew Majda. High order accurate vortex methods with explicit velocity kemelss. Journal of Computational Physics, 58(2):188-008, 1985.
Josh Bongard and Hod Lipson. Automated reverse engincering of nonlinear dynamical systems. Proceedings of the National Academy of Sciences, (04(24):9933-994)) 2007.
Tyson Brochu, Todd Keeler, and Robert Bridson. Linear-time smoke animation with vortex sheet meshes. In Proceedings of the ACM SIGGRAPPH/Eurographies Symposium Computer Anima- tion, pp. 87-95. Citeseer, 2012.
Steven L Brunton, Joshua L Proctor, and Nathan Kutz. Discovering governing equations from data by sparse identification of nonlinear dynamical systems. Proceedings of the national academy of sciences, 113(15):3922-3937 2016.
Michael B Chang, Tomer Ullman, Antonio Torralba, and Joshua B Tenenbaum. A compositional object-based approach learning physical dynamics. arXiv preprint arXiv:1612.00341. 2016.
Mengyu Chu and Nils Thuerey. Data-driven synthesis of smoke flows with enn-based feature de- scriptors, ACM Transactions on Graphics (TOG), 36(4):1-14, 2017.
Mengyu Chu, Lingjie Liu, Quan Zheng, Erik Franz, Hans-Peter Seidel, Christian Theobalt, and Rhaleb Zayer. Physics informed neural fields for smoke reconstruction with sparse data. ACM Transactions Graphics (TOG), 414):11114 2022.
Georges-Henri Cottet, Petros D Koumoutsakos, et al. Vortex methods: theory and practice, vol- ume 8. Cambridge university press Cambridge, 2000.
Ronald Fedkiw, Jos Stam, and Henrik Wann Jensen. Visual simulation of smoke. In Proceed- ings of the 28th Annual Conference on Computer Graphics and Interactive Techniques, SIG- GRAPH '01, pp. 15-22, New York, NY, USA, 2001. Association for Computing Machinery. ISBN 1581 13374X.
doi: 10.1145/383259.8888
URL
https:: htt////....o/11111111111.
3B3259 383260
Shanyan Guan, Huayu Deng, Yunbo Wang, and Xiaokang Yang. Neurofluid: Fluid dynamics grounding with particle-driven neural radiance fields. In ICML, 2022.
Xiaoxiao Guo, Wei Li, and Francesco lorio. Convolutional neural networks for steady flow ap- proximation. In Proceedings of the 22nd ACM SIGKDD international conference knowledge discovery and data mining, pp. 481 181-490, 2016.
Ole Hald. Convergence of vortex methods for euler's equations. ii. SIAM Journal on Numerical Analysis, (665):7267755, 1979. ISSN 00361429, UtLbttp:/ //www. jstor orq/ stable/ 2156630
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Yuanming Hu, Luke Anderson, Tzu-Mao Li, Qi Sun, Nathan Carr, Jonathan Ragan-Kelleg and Frédo Durand. Difftaichi: Differentiable programming for physical simulation. ICLR, 2020.
10

---

Published as a conference paper at ICLR 2023
Xiaowei Jin, Shengze Cai, Hui Li, and George Em Kariiadakis. Nsfnets (navier-stokes flow nets): Physics-informed neural networks for the incompressible navier-stokes equations. Journal of Computational Physics, 426:109951, 2021.
Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In ECCV, pp. 694-711. Springer, 2016.
ByungMoon Kim, Yingjie Liu, Ignacio Llamas, and Jaroslaw Rossignac. Flowfixer: Using bfecc for fluid simulation. Technical report, Georgia Institute of Technology. 2005.
Byungsoo Kim, Vinicius C Azevedo, Markus Gross, and Barbara Solenthaler. Lagrangian neural style transfer for fluids. ACM Transactions on Graphics (TOG), (9(4):2--1, 2020.
J Nathan Kutz, Steven I. Brunton, Bingni W Brunton, and Joshua I. Proctor. Dynamic mode decom- position: data-driven modeling of complex systems. SIAM, 2016.
L'ubor Ladický, SoHyeon Jeong, Barbara Solenthaler, Marc Pollefeys, and Markus Gross. Data- driven fluid simulations using regression forests. ACM Transactions on Graphics (TOG), 34(6): 1-9,2015 2015.
Anthony Leonard. Vortex methods for flow simulation. Journal of Computational Physics, 37(3): 289-335, 1980.
Yunzhu Li, Jiajun Wu, Russ Tedrake, Joshua B Tenenbaum, and Antonio Torralba. Learning par- ticle dynamics for manipulating rigid bodies, deformable objects, and fluids. arXiv preprint 1rXiv:1810.01566, 2018.
Chloé Mimeau and Iraj Mortazavi. review of vortex methods and their applications: From creation to recent advances. Fluids, 6(2):68, 2021.
Sang II Park and Myoung Jun Kim. Vortex fluid for gaseous phenomena. Proceedings of the ACM SIGGRAPH//Eurogrpphie symposium on Computer animation, pp. 261-2700 2005.
Mirta Perlman. On the accuracy of vortex methods. Journal of Computaatoonal Physics, 59)2)2000- 223, 1985. ISSN 0021-9991. doi: http:://hiioggoo///o////0//01110 URL https:
7/ /www. www.ssciencedireedee ott.a//a....a.aaaa..........oooa
Tobias Pfaff, Meire Fortunato, Alvaro Sanchez-Gonzalez, and Peter W Battaglia. Learning mesh- based simulation with graph networks. arXiv preprint arXiv:2010.03409, 2020.
Maziar Raissi and George Em Karniadakis. Hidden physics models: Machine learning of nonlinear partial differential equations. Journal of Computational Physics, 357:125-141, 2018.
Maziar Raissi, Paris Perdikaris, and George Em Karniadakis. Machine learning of linear differential equations using gaussian processes. Journal of Computational Physics, 348:683-693, 2017.
Maziar Raissi, Paris Perdikaris, and George E Karniadakis. Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational physics, 378:686-707, 2019.
Maziar Raissi, Alireza Yazdani, and George Em Karniadakis. Hidden fluid mechanics: Learning velocity and pressure fields from flow visualizations. Science, (61iee33667644111000-003033 2020.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks networks for biomedi- cal image segmentation, 2015.
http://awww...rraaaaa//./.../.0. 04597
Samuel H Rudy, Steven L Brunton. Joshua L Proctor, and Nathan Kutz. Data-driven discovery of partial differential equations. Science advances, 3(4ye (44):1602614, 2017.
Alvaro Sanchez-Gonzales Nicolas Heess, Jost Tobias Springenberg, Josh Merel, Martin Riedmiller, Raia Hadsell, and Peter Battaglia. Graph networks as learnable physics engines for inference and control. In International Conference on Machine Learning, pp.
PMLR, 2018.

---

Published as a conference paper at ICLR 2023
Alvaro Gonzzlzz Jonathan Godwin, Tobias Pfaff, Rex Ying, Jure Leskovee, and Peter Battaglia. Learning to simulate complex physics with graph networks. In International Confer- on Machine Learning, pp. 1459-8468 PMLR, 2020.
Peter Schmid. Dynamic mode decomposition of numerical and experimental data. Journal of fluid mechanics, 656:5-28, 2010.
Michael Schmidt and Hod Lipson. Distilling free-form natural laws from experimental data. science, 324(5923):81 2009.
Andrew Selle, Nick Rasmussen, and Ronald Fedkiw. A vortex particle method for smoke, water and explosions. In ACM SIGGRAPH, pp. 910-914, 2005.
Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein. Im- plicit neural representations with periodic activation functions. Advances in Neural Information Processing Systems, 33:7462-7473, 2020.
Jonathan Tompson, Kristofer Schlachter, Pablo Sprechmann, and Ken Perlin, Accelerating eulerian fluid simulation with convolutional networks. In International Conference Machine Learning, pp. 3424-3433. PMLR, 2017.
Benjamin Ummenhofer, Lukas Prantl, Nils Thuerey, and Vladlen Koltun. Lagrangian fluid simu- lation with continuous convolutions. In International Conference Learning Representations, 2019.
Nicholas Watters, Daniel Zoran, Theophane Weber, Peter Battaglia, Razvan Pascanu, and Andrea Tacchetti. Visual interaction networks: Learning physics simulator from video. Advances in neural information processing systems, 30, 2017.
Steffen WeiBmann and Ulrich Pinkall. Filament-based smoke with vortex shedding and variational reconnection. ACM SIGGRAPH, pp. 1-12, 2010.
Steffen Wiewel, Moritz Becher, and Nils Thuerey. Latent space physics: Towards learning the temporal evolution of fluid flow. Computer Graphics Forum, 38(2):71-82) 2019.
G.S. Winckelmans and A. Leonard. Contributions vortex particle methods for the computation of threc-dimensional incompressible unsteady flows. Journal of Computational Physics, 109(2): 247-273, 1993. ISSN 0021-9991. doi: https://oo..ogg/////0oo/ggghg119
URL https:
// www. www.sciinccc.... rect. com http:///wc..ieeeeeeerrrre///////
You Xie, Erik Franz, Mengyu Chu, and Nils Thuerey. tempogan: A temporally coherent, volumetric gan for super-resolution fluid flow. ACM Transactions on Graphics (TOG), 37(4):1-15, 2018.
Shiying Xiong, Xingzhe He, Yunjin Tong, and Bo Zhu. Neural vortex method: from finite lagrangian particles infinite dimensional eulerian dynamics. arXiv preprint 14Xi:22006.04/78 2020.
Liu Yang, Dongkun Zhang, and George Em Kamiaaakis.. Physics-informed generative adversarial networks for stochastic differential equations. SIAM Journal on Scientific Computing, 42(1): A292-A317, 2020.
Mingrui Zhang, Jianhong Wang, James Tlhomole, and Matthew D Piggott. Learning to estimate and refine fluid motion with physical dynamics. arXiv preprint arXiv:2206. 10480 2022.
12

---

Published as a conference paper at ICLR 2023
A IMPLEMENTATION DETAILS
In this section, we describe the implementation details of our proposed method.
Integrators. As described above and illustrated in Figure 2 our system embeds two differentiable integrators in the loop. The Eulerian integrator is implemented using the Back and Forth Error Compensation and Correction (BFECC) (Kim et al. 2005) method for value look-up, and the 3rd order Runge-Kutte method for time-stepping. The Lagrangian integrator is implemented using the Forward Euler method.
Network Ni. The network N1 adopts a series of 3 residue blocks with increasing widths [64, 128, 256], whose architecture is similar to He et al. (2016) but with convolution layers replaced by linear layers with sine activation functions. The frequency factor we discussed in (Sizzmann et al. (2200) is set
Network 22. The network (2 8) is structured as follows. First, the input r is scaled by the input d as F = where 1) is hyperparameter that corresponds the characteristic scale of the vortices. Then, is transformed into f as f f0.3, reparametrization that stretches the value f near O. This exploits the insight that the velocity varies more aggressively near a vortex. The value F is then fed through 4 residue blocks, which are the same as in N₁ but with shared width of 40. The output from these residue blocks is scaled by multiplying with 3. The scaled value is the output of N2(r, d), which is used for the velocity computation according to Equation 7
Training details. Both the image loss and the velocity alignment loss are MSE, and the velocity alignment loss has an extra scaling factor of 0.001. We use the Adam optimizer with 81 = 0.9, 83 = 0.999, and learning rates 0. 0.0003, 0.001, 0.005, and 0.005 for Ni. N2, Q and A respectively. We use a step learning rate scheduler and set the learning rate to decay to 0.1 of the original value at iteration 20000. We use batch size of 4, so for each iteration, 4 starting times are picked uniformly randomly among [0, .tg] for evaluation. The biding-wwnndee size m is set to 2.
Pretraining N1. We pretrain N1 for 10000 iterations with 2 objectives: (1) for all t E [0, tEl. N1 = [(=1)),..., (Z)) i] coincide with the centers of a 4 x 4 grid, (2) for all / [0, (g). dNi 0. so that these particles are initialized to be stationary. We use MSE for the positional and velocity losses, and the other training specifications are the same described above.
Computational performance. Running on a laptop with Nvidia RTX 3070 Ti and Intel Core i7- 12700H, model takes around 0.4s per training iteration, and around 40000 iterations to converge (for a 256 x 256 video with 100 frames). For inference, each advance step costs around 0.035s.
B COMPARISON WITH DIFFERENTIABLE FLUID SIMULATION
We compare our method qualitatively and quantitatively against standard, grid-based differentiable fluid simulator (referred to as Diff-Sim) on both synthetic and real videos. This baseline method is an dattedifferentiable implementation of the method proposed by Fedkiw al. (2001), which is classic, widely-adopted numerical method for simulating vortical fluids. The method is designed to solve the 2D Fuler equations for inviscid fluid, hence it can in theory recreate the inviscid fluid phe- nomena represented by any video if provided with the appropriate initial conditions and simulation parameters.
Therefore, in this experiment, we make use of its differentiable nature to optimize (1) the initial grid velocities (a 256 256 tensor), and (2) the vorticity confinement strength, which is a scalar value, with the objective of minimizing the discrepancy between the simulated results and the input video. The loss computation between the simulated image sequence and the ground truth is the same in our method. We note that the idea of optimizing initial conditions using differentiable fluid simulation to fit specific target frames has been explored in Hu et al. (2020). However, their task is notably simpler than ours, since they only require the simulated image to match target frame at the end of the simulation, while our goal is to match the underlying motion of the entire video, and dynamically unroll into the future.
Comparison on a synthetic video. We start by comparing both methods on a synthetic video with 300 frames (the first 100 observed for training, the last 200 reserved for testing), which yields visual comparison that can be found in Figure 10] We observe that our method successfully learns the dynamics represented in the video: the generated video and velocities closely resemble the ground
13

---

Published as a conference paper at ICLR 2023
t= 0.00
Observed Past
Predicted Future
t= 3.00
Differendable Fluid Simulation
Ground Truth
0.00
Inferred Past Velocity
Predicted Future Velocity
t= 3.00
Differentiable
Simulation
Ours
Ground
Figure 10: Visual comparison between differentiable grid based simulator and ours on synthetic video. The upper half displays the simulated image, while the lower half displays the underlying velocity, whose color wheel is depicted. On the bottom row of each half is the ground truth sequence, which has 300 frames. The first 100 frames available for both methods learn from, while the rest unseen during training.
Synthetic Test Errors
Real Test Errors
Weldtyy
Irage (SA persetual
10
60
80
Figure 11: Error plots of the comparison between Diff-Sim and our DVP method on synthetic dataset.
truth even the unseen frames. Diff-Sim, on the other hand, shows weak resemblance with the ground truth for the seen frames, but fails to capture the individual eddies in the video. Consequently, fails to predict the future dynamics. DiffsSim's 's lack of correspondence to the dynamics of the ground truth is also made evident in Figure 13 The result clearly suggests that method has better learned the dynamical evolution. This performance discrepancy is numerically supported by the errors documented on the left panel of Table 3 and plotted on the left panel of Figure I1 both showing that our method yields reduced image-level and velocity-level errors compared to Diff-Sim.
Comparison real video. We then the same experimental setup to perform learning real video with 139 frames (the first 93 observed for training, the last 46 reserved for testing), as depicted in Figure 12 We observe that the real video, the same behavioral patterns for both
14

---

Published as a conference paper at ICLR 2023
0.00
Predicted Future
1.5
Differentable Ruid Simulation
Ground Truth
N/A
Figure 12: Comparison between Diff-Sim and our method on real video.
Synthetic Video Errors
Real Video Errors
ΛEPE
AAE
Vort.
RMSE
VGG
VGG (avg.)
VGG (final)
Diff' fim (Grid)
0.469
0.953
24.43
0.157
26043
15076
18792
Ours
0.041
0.081
1.482
0.055
2171.4
7846.2
11081
Table Error comparison between Diff-Sim and our method on synthetic videos.
systems on the synthetic one have carried over. For the results generated by Diff-Sim (top row), we that the overall, large-scale motion (the large eddy moving towards bottom-left) is faintly identifiable. Nevertheless, all the smaller vortices are missing, and the entire image quickly diffuses the simulation proceeds. This can be attributed to the numerical diffusion issues innate to grid- based simulations, as well as the lack of embedded fluid structures. In comparison, our method well-preserves the vortical movements due its built-in structure, and produces plausible future rollout extending beyond the duration of the original video. Although both systems are unable to perfectly model the exact mechanism that governs this wal-world video (due unmodeled factors such as fluid viscosity, air friction, and 3-dimensional forces), our proposed method does a better job of retaining the vortical patterns and energetic flows thanks to its vorticity-basse Lagrangian-Eulent Eulerian design, can be observed in the middle row of Figure 12 The advantage of our system over Diff-Sim on real video is numerically supported, as shown on the right panels of compare
formulation and the
Table and Figure II Since we do not have the ground-truie velocities for real videos, the VGG perceptual loss (Johnson et al.) 2016) between the simulated sequence of both methods and the real video, which demonstrates quantitatively that our generated results better resemble the input video than those generated by the baseline.
C ADDITIONAL BENCHMARK TESTING
further illustrate our method's advantage and generalizability, we have conducted an additional set of numerical tests on another synthetic video (of 180 frames with the first 60 revealed for training), and compared our method's performance with 4 benchmarks in terms of both velocity inference guality and future prediction quality. The ground truth data is generated using a significantly different background image (sharp color tiles vs. smooth color gradients), and different velocity kernel (second-order Gaussian kernel vs. first-order Gaussian kernel) (Beale & 1985). The experimental setup otherwise the same as the one presented in the main text (in Figure with the same compared benchmarks.
As depicted in Figure 14
Majda
The comparison of the velocity inference quality can be found Figure 16 and the top panel of Figure 16 depicts the uncovered velocities of frame 40 (among the 60 input frames)
Figure 15
15

---

Published as a conference paper at ICLR 2023
Color Plot
Streamline Plot
Quiver Plot
Residue
Differentiable
Fluid
Simulation
Ours
Ground Truth
Figure 13: Comparing the quality of the velocity predicted by method and Diff-Sim. We show the predicted velocity (of frame 200) three different forms (color, streamline, quiver plots) addition residue (end-point error) compared to the ground truth velocity.
= 0.60
==
t= 1.8
Ground Truth
HFM Extrap.
HFM UNet
ER UNet
UNet
Ours
Figure 14: Future prediction results: our method compared to the baselines on synthetic video.
by all 4 methods compared to the ground truth. The top row depicts the respective velocities in colors with the color wheel supplied: the middle row depicts the velocities in streamlines; and the bottom row depicts the velocity residues compared to the ground truth, measured in end-poin error (EPE). As with the results in Figure 5) we can see that HFM, UNet, and our method can all infer the underlying velocity field at high precision, whereas E-R yields a visibly noisier approximation. As seen on the bottom row, the inference performance of UNet and Ours are very close, but our method takes the slight edge with average error (AEPE) of 0.0143, which is 33.49% less than the error of 0.0215 yielded by UNet. The advantage of our method is not unique to the specific frame selected. As plotted on the top of Figure 15 it be seen that method (red) consistently
16

---

Published as a conference paper at ICLR 2023
Motion Inference
Errors
Future
Prediction
Erross
00
36
29
Figure 15: Error analysis second synthetic video. The top plots the inference of velocity, vorticity, and compressibility. The bottom row plots the future prediction emors, which consider both dynamic error of the velocity and the pereeptual error the generated image sequence.
Velocity
Stream-
lines
Velocity Residue
Ground Truth
HFM
UNec
Ours
Figure 16: Baseline comparison of velocity inference on synthetic video. Our method recovers the underlying velocity field with the highest accuracy.
yields the lowest velocity-inie inferrece error throughout the 60 input frames, in terms of the average end-point error (AEPE), average angular error (AAE), vorticity RMSE and compressibilityy RMSE. The time-averaged emrors these metrics are documented in Table which again shows that our method yields the best estimations.
Future prediction. We also compare our method's future prediction results with the baselines. In Figure 14 we show visual comparison of all 5 methods against the ground truth. highlights our generated sequence with the ground truth, which is twice as long as
the close resemblance the sequence used for training. Compared to the baselines, our method yields the best match the ground truth video, capturing the accurate vortical flow structures. HFM+UNet, E-R+UNet, and UNet can generate reasonable future predictions up sequences start to distort in different ways, due to their lack of physical structures and constraints. The direct extrapolation of HFM yields the least plausible results, quickly degrading to noise. We compare these sequences quantitatively using the 4 velocity-based metrics, along with the 2 image- based metrics: the pixel-level RMSE and the VGG feature reconstruction loss. Four of these time- dependent errors plotted in the bottom of Figure [5] with their time-avemn aineraee counterparts documented on the right of Table 4 summary, we observe that our method outperforms the existing baselines for this video both quantitatively and qualitatively.
1.0 (for 40 frames). For 1.0. these
17

---

Published as a conference paper at ICLR 2023
Time-averaged Inference Errors
Time-averaged Prediction Errors
AEPE
AAE
Vort.
Div.
VGG
RMSE
AEPE
AAE
Vort.
Div.
E-R
0.229
0.805
4.380
1.750
+UNet
8138.8
0.178
0.272
1.115
4.694
2.504
+UNet
9389.5
0.146
0.201
0.715
10.58
3.199
HFM
0.038
0.097
3.001
0.533
Exip.
40967
0.166
0.293
1.221
4.862
3.152
UNet
0.026
0.101
1.013
0.895
7721.3
0.170
0.330
1.141
5.462
1.496
Ours
0.015
0.046
0,480
0.015
2045.0
0.097
0.057
0.173
1.547
0.013
Table 4: Time-averaged emorrs of our method compared various baselines on synthetic video.
D NUMBER OF VORTEX PARTICLES
In our proposed method, we use n vortex particles to learn fluid dynamics. However, we note that vortices are not intrinsic to fluid phenomena, but are rather imposed constructs to allow fluids to be better understood conceptually and modeled numerically. Thus, the number of vortices n is fundamentally hyperparameter that does not admit uniquely-correct value.
With this in mind, we let î denote the minimum number of particles that can be used to model the fluid system to acceptable accuracy. This natural number á surely exists since it has been proven that vortex particle methods converge to the exact solution of the 2D Euler Equations (Beale & Majda 1985 Hald 1979). We are mostly concerned with the cases where the number of deployed degrees of freedom (DoFs) is greater than the number necessary for the given fluid system. In the following section, we show that our method can spontaneously prune the redundant vortices and thus it is robust to a reasonable range of n > î. In Figure 17 we show the results of learning the same underlying motion with 4, 9, and 64 vortex particles. In Figure 18 we show the underlying velocity and vorticity fields using different numbers of vortex particles.
-. which means
Spontaneous pruning of redundant DoFs. As shown on the top row of Figure 17 the ground truth is generated with 4 vortices, so it is safe to assume that á = 4. Learning with 4 vortices (as shown on the second row) represents the case where can see that there is one-to-one correspondence between the ground-truth vortices and the learned vortices, with each learned vortex assuming the role of one individual ground-truth vortex (obtaining the same vorticity and initial position).
n. Comparing the first row with the second row, we
When we have 9 vortices (third row), there are more vortex particles than those in the ground truth. In this case, two interesting phenomena occur to spontancously prune these redundant particles: de- generation and clustering. First, some particles degenerate themselves by reducing their strengths to 0 or by moving farther away from the domain. We can observe both mechanisms taking place on the two lingering particles on the top part of the third row. They both have low strengths (evi- dent from their turquoise color) and are peripheral to the domain. Secondly, multiple particles can aggregate to emulate a single particle with greater strength. Since the velocity computation is a iistance-weiggteee summation (as Equation 7). if multiple particles coincide at the same location, they effeettivly act as one single particle with their vorticities added together. This phenomenon can be observed by comparing the lower halves of the second and third rows. Both of these mechanisms enable our system to spontaneously prune redundant vortices. In the last row, we show that our method is robust to even 64 vortices.
Figure 19 Thelps to illustrate this spontaneous pruning mechanism by showing different snapshots of the training process. Shown on the left are the vortex particles' behaviors soon after the training has begun. It is particularly noticeable that, the bottom row, the 64 particles are scattered throughout the fluid domain, and the simulated result appears quite different from the ground truth. Moving from left right, these particles become more and more clustered around the flow regions, with much fewer "freelance" particles, and the end result can approximate the ground truth much better.
Finally, we note that < f is still challenging to resolve the system would be coer-constrai Nevertheless, we empirically find that n = 16 is sufficient for all the real and synthetic videos we consider in experiments.
18

---

Published as conference paper at ICLR 2023
Observed Past
Predicted Future
Ground Truch (oortics)
Learned with Vortices
Learned
with Vortices
Learned
Vortices
Figure 17: The same underlying motion learned with different numbers of vortex particles. The ground truth has 100 frames; the first 30 frames are observed during training, and the remaining 70 frames predicted.
Velocity Field Frame
Vorticity Field Frame 80
Ground Truth
Learned with Vortices
Learned with Vortices
Learned with 64 Vortices
(orotice
Figure 18: Robustness our DVP method against different numbers of vortices used. Different numbers vortices can represent similar underlying dynamics.
E ABLATION: LEARNABLE VELOCITY KERNEL.
In traditional vortex simulation applications in Computer Graphics and Computational Fluid Dy- namics, the velocity kernel is hand-selected (typically from Gaussian kemnels of different orders) with uniform support radius (size). Such approaches designed to perform forward simulation, yet they are limiting when used for backward inference tasks, i.e., to reconstruct input videos. In our method, address this issue by learning neural kernels with learnable sizes. By leveraging data-
19

---

Published as a conference paper at ICLR 2023
Training Starts
Training Ends
Learning with 4 Vorticos
Learning with 9 Vortices
Ground Truth (4 Vortices)
Learning
with
64 Vortices
Figure 19: The training evolution when using different numbers of vortex particles.
Observed Past
Predicted Future
Without
Kemel
With Learrable Kernel
Ground
N/A
Truth
Figure 20: Ablation study: reconstruction and prediction real video with learnable velocity kemnels (our full method) and without learnable velocity kernels (ablation).
driven techniques, we can reconstruct and predict fluid flows that not only are visually pleasing, but also resemble the specific dynamical traits embodied the input video.
we present ablation study on the learnable velocity kemnels. We reconstruct and predict a real-world video using our method and an ablated version in which the learnable kernel replaced with hard-coded first-order Gaussian kernel with uniform size. The ground truth, shown on the bottom row, has 126 frames revealed for training and 62 frames hidden for testing. In middle row, we learn to fit the video with our learnalee kemel enabled. In the top row, we learn to do the same with the learnable kernel disabled. It can be observed that the middle row well- captures the characteristic smoothness of the flow, and simulates image sequence that resembles the ground truth. The ablated version (top row) can also learn the correct overall motion (clockwise rotation), but it induces various smaller eddies and wrinkles uncharacteristic of the input video.
In Figure 20
20

---

Published as a conference paper at ICLR 2023
VGG
VGG
RMSE (avg)
tovg.)
(fnal)
Ours
w/o
9698.7
11240
0.183
013
learnalle
kemels
003
Ours
7365.6
10027
0.180
Figure 21: viryingyynng losses corresponding Figure
Table 5: Time aine-averaged errors of Figure
0.00
1.80
Without Trajectory Learning
Ours
Ground Truth
0.00
1.80
Without
Trajectory Learnnng
Ours
Ground Truth
Figure 22: We compare method against its ablated version which does have the feature trajectory learning. The top depicts the simulated images, and the hottom depicts the simulated velocities. The results of both approaches are compared to the ground truth.
Extending to unseen frames, method continue retain the overall structure of the eddies, while the ablated version (without the learnable kernel) drives the pattern to disintegrate, and devel- ops various folds and wrinkles that do resemble the dynamical characteristies of the real video. We further show quantitative results plotted in Figure 21 and documented in Table 5 In summary, learning the velocity kernels allows for better reconstruction and prediction of fluid flow specific to the input video.
F ABLATION: TRAJECTORY LEARNING
In our approach, we learn the full trajectories of vortex particles for the input video. An alternative is to learn the initial condition only. However, find that the former option is more computationally tractable and effective, since it can exploit the full range of the input video at a managcable cost. To see this, suppose have 100 training frames in the video, and the goal is infer the initial
21

---

Published as a conference paper at ICLR 2023
Velocity Errors
Image Errors
AEPE
AAE
Vort.
Div.
RMSE
VGG
Ours (Ablated)
0.257
0.753
4.689
0.054
0.170
6759.4
Ours
0.043
0.131
1.180
0.014
0.081
1805.9
Table 6: Time-averaged velocity-level and image-level errors by method and its ablated version without trajectory learning.
Wiecly ADH aver
weledit over
Vertidty emor over
2'-
Oup
1.0
100
15
175
300
150
US
Comprees
Image over
Image perseptual
1.0
1/5
kune
kame
Figure 23: varevnryyngg velocity-level and image-level errors by our method and its ablated version without trajectory learning.
condition at frame-1. we directly optimize the initial condition using the last frame, we need to simulate from frame-1 all the way to frame-100, compute the loss and backpropagate. Unrolling such a long sequence for each training iteration (1) takes a long time, (2) leads to noisy gradicnts, and (3) is practically infeasible due to memory constraints. On the other hand, learning the whole trajectory allows us to address these challenges by using a smaller sliding window in time (e.g., simulating only 3 frames a time) and aggregating the dynamics information throughout the whole video. In Figure [2] we show a comparison of both methods in action, with a total of 180 simulated frames. On the top panel, we show the reconstruction and prediction results for both our full method and an ablated version where we directly learn the initial condition. Note that the ablated version can only unroll the first 13 frames (and thus is learned using only the input video's first 14 frames) due to the memory constraint (which is consistent for both candidates). In contrast, our full method can handle the 60 input frames like in the setup of Appendix C On the bottom panel, we show the velocity corresponding to the top panel. We observe that our method and its ablated version can approximate the ground truth reasonably well at the beginning of the simulation (the left three images). However, the ablated version starts to distort significantly in terms of both the advected image and the underlying velocity. This observation is in agreement with the numerical evidence, as plotted in Figure 23 and documented in Table 6 which shows that our full method consistently outperforms its ablated counterpart across all metrics. We conjecture that the underlying reasons for this performance discrepancy are threefold: first, the ablated version can only learn from the beginning section of the fluid observation, which provides limited information to correctly infer the initial condition. Secondly, only learning the initial condition more susceptible to accumulated errors than our full method. Thirdly, using a limited number of frames makes it harder to synthesize an appropriate velocity kernel. In summary, our observations suggest that learning the full trajectory is more desirable than learning the initial condition only.
22


Reviews:
[{"review_id": "skfqke_1ug", "reviewer": "Reviewer_PvPD", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "I found this paper to be very interesting. The idea of having two 'simulators' one for vorticity for physics tracking and one for RGB video tracking is clever, and the coupling between vorticity/velocity in Equation (3) is something I was unfamiliar with. Although limited to 2D, in its current formulation, the idea of being able to link a visible and hidden simulator is intriguing. The results on both synthetic and real video captures are compelling and clearly perform better than other methods compared against. ", "summary_of_the_review": "An elegant solution to a difficult problem. Compelling demonstration on real video sequences. Not often such a coupling can be obtained between visible and hidden states to regularize a solution. The hidden states have the right complexity to regularize the problem! ", "strength_and_weaknesses": "+ Elegant and well-grounded formulation\n+ Compelling results on both synthetic and real sequences\n+ Section 3 and Figures 2/3 nicely explain the key ideas behind the approach\n\n- Limited to 2D in its current version\n- Are there invariants in vortex-based formulation that can be incorporated?\n- Other limitations are discussed at the end of the paper but they should encourage future works", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "e55mH7DLY3", "reviewer": "Reviewer_jKRP", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces a learning-based method to predict vortex dynamics. It designs a vortex dynamics module to learn the evolution of the vortices and another network to transform from the vortex space to the image space velocity field. Results show that the proposed method provides a realistic and stable vortex evolution trajectory, both in synthetic and real-world videos.", "summary_of_the_review": "Although the method and the results are mostly solid, there are some clarity issues regarding the method itself which can prevent the readers from understanding the whole picture. I would recommend weak reject now, but will be happy to change once the issues are addressed.", "strength_and_weaknesses": "Strength:\n1. Instead of modeling the entire image space dynamics, the method models only the vortex information, which effectively reduces the dimension and difficulty, and also helps generalize to unseen cases.\n2. The results are very promising. They clearly outstand previous works.\n3. The experimental comparisons are thorough and complete. They support the claims well.\n\nWeakness:\n1. There should be more detailed descriptions in the method section. For example, how are the vortex positions initialized? Are they inferred from the image or overfitted to the given example? How does the model determine how many vortices to model given an image? I don't see a module to encode this information from the input image to the vortex space.\n2. The strengths and the sizes of all vortices are assumed to be the same throughout the entire prediction. Is it mostly the case in the real world? How does it introduce limitations to this model? Again, how to model the initial values using trainable vectors is unclear. Do they overfit to a certain example? If not, how are they estimated from the image?\n3. I wonder if this task can be achieved already by optimizing the parameters using differentiable fluid simulation:\nTakahashi, Tetsuya, et al. \"Differentiable fluids with solid coupling for learning and control.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 35. No. 7. 2021.\nPlease add some discussions regarding the difference between the proposed method and the differentiable simulation system.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "PWOK4LlU1z", "reviewer": "Reviewer_PYMt", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a method to learn vortex dynamics from a single fluid video and thus be able to infer and predict fluid dynamics. At the core of the method are a Lagrangian vortex particle representation and a learnable vortex-to-velocity dynamics mapping. These two components can be trained merely from the observed video in an end-to-end manner. After training, the hidden physics quantities such as velocity field would be obtained, and the prediction of future states is also supported. The proposed method is compared with existing methods on both synthetic and real videos, where the proposed method shows more accurate velocity estimation and more plausible future prediction.", "summary_of_the_review": "I am not familiar with all the literature in this field, but to the best of my knowledge, this paper looks pretty interesting to me. It cleverly combines Lagrangian particle representation and Eular representation in a unified and end-to-end trainable framework, addressing several existing challenges. The parameterization of position x and the vortex-to-velocity mapping using neural networks is reasonable. But more implementation details could be added and the evaluation can be improved by using more videos. In general, I am in favor of accepting this paper. But it is possible that I may overlook some issues given my limited knowledge in this field, so I am also interested to see other reviewers' comments.", "strength_and_weaknesses": "Strength:\n\n- The proposed method of using a Lagrangian vortex particle representation and a learnable vortex-to-velocity mapping looks pretty interesting and reasonable to me. The vortex particle representation is more compact and reduces the problem's dimension. The learnable vortex-to-velocity mapping well connects the vortex particle and the fluid phenomena, making the pipeline end-to-end trainable. The model design also integrates fluid simulation and learning-based approach in a reasonable way, combining the advantages of both fields. \n\n- The advantage of the proposed method over baselines is obvious in experiments. Quantitative and qualitative study in synthetic and real data demonstrate that the proposed method estimates more accurate velocity and predicts more plausible future sequences.\n\nWeaknesses:\n\n- The description of implementation is not detailed enough as some hyperparameters are missing. For example, the number of particles n is not provided. It is also not mentioned how $\\Delta$ and $\\Omega$ are initialized. Besides, it is suggested to also mention the efficiency of the method, like the training and inference time.\n\n- The amount of data used for evaluation seems limited. If I understand correctly, the quantitative study is conducted on only one synthetic video and one real video, which only covers limited diversity. It would be better to evaluate on more videos to strengthen the results.\n\n- The method now assumes inviscid fluid and 2-dimension, which is a somewhat constrained setting and would limit the application scope. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "1HHFwp3eE1", "reviewer": "Reviewer_W2do", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper considers the modeling of fluid dynamics from limited observations of particle densities, using the vorticity formulation of the 2D Navier-Stokes equation. Specifically, the velocity field that drives the movement of the particles are inferred from the observable particle density over a time interval. The calculated velocity field is then modeled by the vortex model of the 2D Navier-Stokes equation with a learned approximation of the kernel of the Biot-Savart Law. The trajectory module of the vortices is trained so that the predicted velocity field matches the ones derived from the training data (the density image). Further, this paper considers the prediction of the evolution of the particle density based on the learned vortex dynamics.", "summary_of_the_review": "The empirical results are impressive. However, it is not clear why the data are needed given that the underlying dynamics is already modeled by the 2D Navier Stokes equation. Is it for the purpose of estimating the viscosity parameter and the initial vortex distribution? ", "strength_and_weaknesses": "Strength: \nThis paper proposes a fluid dynamics representation with differentiable vortex particles so that the observed particle densities and be used jointly with the underlying vortex model (the vorticity formulation of the 2D Navier-Stokes equation with the Biot-Savart kernel) to learn the particle trajectory and the vorticity-to-velocity mapping, i.e. an approximated Biot-Savart kernel. The empirical results are impressive compared with previous SOTA of the domain. \n\nWeakness:\nSince we exactly know the dynamics of the vortices, why do we need data to help to learn their trajectories? Given the initial distribution of the vortices, the following dynamics can be derived by solving the 2D Navier Stokes equation, right?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster