Inferring Hybrid Neural Fluid Fields from Videos
Authors: Hong-Xing Yu
Yang Zheng
Yuan Gao
Yitong Deng
Bo Zhu
Jiajun Wu
Venue: NeurIPS 2023 poster (2023)
Keywords: neural scene representations
fluid dynamics
flow reconstruction
physics-based learning
TL;DR: Recovering fluid density and velocity from a few multiview videos. Website: https://kovenyu.com/HyFluid/

Abstract: We study recovering fluid density and velocity from sparse multiview videos. Existing neural dynamic reconstruction methods predominantly rely on optical flows; therefore, they cannot accurately estimate the density and uncover the underlying velocity due to the inherent visual ambiguities of fluid velocity, as fluids are often shapeless and lack stable visual features. The challenge is further pronounced by the turbulent nature of fluid flows, which calls for properly designed fluid velocity representations. To address these challenges, we propose hybrid neural fluid fields (HyFluid), a neural approach to jointly infer fluid density and velocity fields. Specifically, to deal with visual ambiguities of fluid velocity, we introduce a set of physics-based losses that enforce inferring a physically plausible velocity field, which is divergence-free and drives the transport of density. To deal with the turbulent nature of fluid velocity, we design a hybrid neural velocity representation that includes a base neural velocity field that captures most irrotational energy and a vortex particle-based velocity that models residual turbulent velocity. We show that our method enables recovering vortical flow details. Our approach opens up possibilities for various learning and reconstruction applications centered around 3D incompressible flow, including fluid re-simulation and editing, future prediction, and neural dynamic scene composition. Project website: https://kovenyu.com/HyFluid/

Inferring Hybrid Neural Fluid Fields from Videos
Hong-Xing Yu
Yang Zheng"
Yuan Gao'
Yitong Deng'
Bo Zhu? Jiajun Wu!
Stanford University!
Georgia Institute of Technology*
Abstract
We study recovering fluid density and velocity from sparse multiview videos. Existing neural dynamic reconstruction methods predominantly rely on optical flows; therefore, they cannot accurately estimate the density and uncover the underlying velocity due to the inherent visual ambiguities of fluid velocity, as fluids are often shapeless and lack stable visual features. The challenge is further pronounced by the turbulent nature of fluid flows, which calls for properly designed fluid velocity representations. To address these challenges, we propose hybrid neural fluid fields (HyFluid), neural approach to jointly infer fluid density and velocity fields. Specifically, to deal with visual ambiguities of fluid velocity, we introduce a set of physics-based losses that enforce inferring physically plausible velocity field, which is divergence-free and drives the transport of density. To deal with the turbulent nature of fluid velocity, we design a hybrid neural velocity representation that includes base neural velocity field that captures most irrotational energy and a vortex particle-based velocity that models residual turbulent velocity. We show that our method enables recovering vortical flow details. Our approach opens up possibilities for various learning and reconstruction applications centered around 3D incompressible flow, including fluid re-simulation and editing, future prediction, and neural dynamic scene composition. Project website: https: /kovenyu. com com/HyFluid/.
1 Introduction
Fluid is ubiquitous in our surroundings, from a small breeze in the morning to large-scale atmosphere flow that would affect the weather in the following week. Understanding and predicting fluid dynamics play a central role in climate forecasting (Bauer et al., 2015), vehicle design (Bushnell & Moore, 1991), visual special effect (Pfaff et al., 2010), etc. Yet. it remains open problem in scientific machine learning to accurately recover fluid flows from visual observations. Flow motions only be seen indirectly, and, unlike solids that have certain shapes and simple constrained motion patterns, fluid systems have intricate and complex dynamics that exhibit different features across spatial scales. These characteristics pose unique challenges in the representations and algorithms recover physically correct velocity. Recently, chysics-informe neural networks (Raissi et al., 2019) have shown promise recovering velocity fields from density fields for scientific computing. These neural methods are scalable and flexible compared to traditional grid-based methods, yet they require accurate density fields that are unavailable in uncontrolled ordinary scenes.
In this work, we focus on recovering the density' and velocity of fluids from sparsely captured multiview videos (see Fig. for illustration). We identify three key challenges. Firstly, the fluid velocity is ambiguous from visual observations. Unlike solid objects characterized by consistent
"Equal contributions. "Density" refers to the concentration of the fluid substance such as smoke soot. Not to be confused with the physical density, i.e.,
volume.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Sparse-view videos
Novel-view appearance and velocity
Re-simulation & composition
Figure I:
We aim at inferring hybrid neural fluid representations of density and velocity from sparse multi-view videos. This allows novel view video synthesis, re-simulation, and dynamic scene composition.
shapes and stable visual features, fluids are inherently shapeless, often monochromatic, and semi- transparent. This lack of physical definition makes visual tracking implausible task, contributing to pronounced visual ambiguity, particularly in regions of laminar flow where the fluid appearance hardly changes. This challenge is beyond the capacity of general-purpose purpose neural dynamic reconstruction methods (Li et al., 2021; Du et al., 2021), as they predominantly depend on optical flow which is ineffective in this context.
Secondly, accurately depicting fluid velocity necessitates appropriate representations that respect its turbulent features. Fluid flows exhibit varying features across multiple spatial scales (Frisch & Donnelly, 1996), a phenomenon that is challenging to capture within a single neural representation due to its spectral bias (Rahaman et al., 2019). At the human scale, fluid motions express not only laminar flows but also turbulence (Pope, 2000). This results in a complicated entanglement of rotational, shearing, and smooth motions. The turbulent flows are high-frequency in both space and time. Therefore, the representation of such flow should be able to accommodate a broad spectrum of signal frequencies while also preserving the unique structure of rotational flows.
Lastly, it is inherently ambiguous to reconstruct physically plausible 3D density fields from sparse observations. Recovering fluid density and appearance from limited 2D videos is ill-posed primarily due to the intricate non-linear absorption and scattering processes associated with semi-transparent fluids (Max. 1995). Consequently, the reconstruction of under-constrained continuous density fields from sparse observations an intrinsically difficult task.
To address these challenges, we propose hybrid neural fluid fields (HyFluid), neural approach to jointly infer fluid density and velocity from sparse-view videos. At the core of HyFluid are two key technical contributions including a set of simple yet effective physics-based losses and hybrid neural velocity representation. Our physics-bane based losses leverage physical constraints from the Navier-Stokes equations to jointly learn physically plausible fluid density and velocity fields. The physics-based losses enforce inferring divergence-free velocity field that drives the transport the density field. In addition to the new losses, we propose hybrid fluid velocity representation as it diffieult for a single neural representation to capture turbulent fluid velocity. The main idea is to decompose our fluid velocity field into base and a residual turbulent velocity field. The base velocity field is represented by neural field that captures most irrotational energy, and the residual velocity field is represented by vortex particles that feature highly rotational and shearing velocities. In order to address the 2D-to-3D density ambiguity, we leverage both visual imaging signals by differentiable volume rendering and the physical fluid transport constraint to regularize the density field.
We demonstrate that HyFluid can yield high nigh-fidelity recovery of density and velocity, allowing novel view video synthesis, re-simulation, editing, future prediction, and neural dynamic scene composition (Fig. 1). In summary, our contributions are three-folded: (HyFluid), a neural approach to infer fluid density and velocity from sparse nultiview videos. (2) We show that using simple physics-basee losses and a hybrid neural velocity representation allows uncovering turbulent fluid motion from sparse observations. (3) We evaluate our method on novel-
(1) We propose hybrid neural fluid fields

---

view re- re-simllation and novel-view future prediction to benchmark joint reconstruction for real fluids, and we show that our approach delivers high-fidelity synthesis and reconstructions in comparison to state-of-the-art neural methods.
2 Preliminaries
Incompressible Navier-Stokes equations. Our motion representation and learning signals are motivated by incompressible Navier-Stokes equations. With a low flow speed, fluid motion is considered incompressible and can be well described by:
Du
du
u1 Vu = - Ap+uV + Fu+f,
Dt
ōt
v-u
(2)
Here, Eqn. 1 is known as the momentum equation, where the LHS is the material derivative for + u Vu, which represents the time rate of change of velocity u of a fluid parcel, and the RHS is the momentum induced by pressure gradient _! (where denotes the physical density of the fluid parcel), viscosity v Vu, and some external force f. We consider inviscid fluid and thus drop the viscosity term. Eqn. is known as the mass conservation equation, meaning that the mass flowing into controlled volume should be equal to the mass flowing out of the volume.
velocity, Du au Dr
Differentiable volume rendering. We integrate volume rendering for joint learning of visual appearance, density, and velocity from videos. Volume rendering is a suitable model for translucent materials and participating media like smoke and fog (Kajiya & Von Herzen, 1984). In volume rendering, the radiance L(o, d) arriving at the camera location o from direction (a.k.a. the radiance of camera ray r(() Id) with near and far bounds i, and Iy is given by
L(o.d) = S T(t))((())L((())..d
where T()) = exp(- Sr, (r(s))ds) denotes the transmittance along the ray from L, to I, Le denotes the emitting radiance, and denotes the optical density which can be considered proportional to the concentration density of fluid substance according to Beer-Lambert law. Following recent neural rendering methods (Mildenhall al., 2020), we use quadrature to discretize it (Max, 1995).
3 Hybrid Neural Fluid Fields
Our goal is to recover the fluid appearance, density, and velocity from sparse multiview RGB videos. To this end, we propose hybrid neural fluid fields (HyFluid). At the core of HyFluid are set of simple physics-based losses and a hybrid neural velocity representation. In particular, HyFluid aims to infer neural density field o (I. y, 2, t), the appearance of the fluid Le. and the underlying velocity u(x, y, 2, t) Upase (z, y, z, t) + Uyee (I. y, 2, t),, where we decompose it into a base neural velocity field Uhose and a residual vortex particle-driven turbulent velocity. along v, y, axis, respectively. In the following, we first introduce the physics-based losses, then we describe hybrid velocity representation, and finally, summarize learning signals for the joint inference. We show a conceptual illustration in Fig. 2.
[u, 2, w| denotes the velocity
Physics-basee losses. We propose physics-based losses Lphysics including density loss, a projection loss, and laminar regularization loss.
Density loss. While the neural density field can be learned through differentiable volume rendering, the underlying velocity field is hidden from the visual observations. Unlike solids that have regular shapes and invariant visual features to track, fluids are often shapeless, monochromatic. and semi- transparent. This makes implausible to visually track it. To uncover the hidden velocity from visual observations, we resort to the density transport equation in incompressible flows, De introduce a physics-informed supervision signal:
Dr 0. and
da
da
Latensty
E
+
+ v
w
(4)
2.9.2.t
Øt
dr
Oy
Oz
3

---

Density field d(x,y,,,1)
Render
Vortex particles Myort ( y,,,,, 2,t)
physics
render
Velocity field y,x,,
Base flow Masse 22
Observed video
Figure 2: Illustration of hybrid neural fluid fields (HyFFuudd which aims to jointly infer fluid density and velocity from videos using visual signals via differentiable volume rendering and physics-based losses. To facilitate learning turbulent velocity, we decompose the velocity field a base velocity and vortex particle-driven velocity. See the main text for detailed descriptions.
This loss says that the 3D velocity field (u, v, w) at any given moment & should transport the density field such that it evolves to the density field the next moment, in similar spirit scene flow (Vedula et al., 1999).
Projection loss. Another important physical property of incompressible flow is mass conservation depicted in Equ. 2, that is, the velocity field should be divergence- enforce divergence-fie
straightforward way to condition is to impose a loss on the divergence of velocity. du + By + 2u 05
Ze
However, we find that this empirically leads to degenerated solution, where only the laminar flows (i.e., smooth, direct-current flows) can be uncovered this is trivial local minimum for every point. Yet, the incompressibility of fluid flows is confined through a global dynamic system, modulated by the pressure of the fluid.
Therefore, we need a globally- aware,, physically-plausibie supervision that does not conform to trivial local minima. Motivated by the commonly used pressure projection solver in computational fluid dynamics, we propose projection loss that constrains our learned velocity to be divergence
ppoj
E
[[|u up
(5)
2.
where up project(u) are the pressure-projesn projected velocity field, using a pressure project solver. project solver implements Helmholtz decomposition by solving a global linear system for pressure, and it projects the velocity field to divergence-free manifold using the gradient of the solved pressure. Also, we notice that any divergence-free velocity field u incurs zero projection loss an identity mapping for divergence-free fields. Thus, this projection loss essentially constrains the uncovered velocity within a divergence-free subspace, and it is not subject to trivial local minima of pure direct-current flows.
Laminar regularization loss. These density and projection losses do not warrant physically-correet velocity reconstruction for laminar flows. In laminar flows, all partial derivatives of density zero, so any velocity, e.g., all-zero velocity, incurs zero Lidesity- As for Cunjects all-zero velocity fields also incur zero loss. Hence, we introduce laminar regularization loss:
Ciaminar
E 2.
[max((0, yo ||]|)]]
which is hinge loss that encourages high-density regions to have non-zero velocity, and y denotes a hyperparameter to scale the threshold according to the velocity magnitude unit. These three losses are complementary and interconnected to uncover physically plausible fluid flow.
Hybrid neural velocity representation. At the human scale, fluid flows are often turbulent. They exhibit complex dynamic features including circular and shearing flows. addition to common challenges of reconstructing non-turbulent flows, turbulent flows are even more intricate and difficult to model and capture. They are high-frequency flows both space and time. This demands that the velocity representation be able accommodate wide range of signal frequencies and the particular

---

structure of rotational flows. However, neural representations are known to have certain spectral inductive biases toward representing low-frequency signals (Rahaman et al., 2019). While this can be alleviated by position embedding (Tancik et al., 2020) or sinusoidal activation functions (Sitzmann et al., 2020), they still lack a proper structure to capture and represent highly detailed turbulent flows.
We propose a hybrid neural velocity representation to tackle this challenge. The main idea is to decompose the underlying flow field u Uhese + Uvort into a base neural velocity field Ubase and residual vorticity-driven velocity Uvert. The base neural velocity field captures the large-scale flow motion, and the vorticity-drin driven velocity is a dedicated representation of the small-scale vortex details that are hard to characterize by the base field. For the base velocity field Ubme, we extend instance neural graphics primitive (iNGP) (Müller al., 2022) representation to the temporal domain.
The residual vorticity-driven flow Hyer is complementary to the base flow by leveraging the physical structure of turbulent fluid flows. The physical model behind the vorticity-driven flow is prescribed by the curl form of the Navier-Stokes equations (see Cottet et al. (2000) for more details):
Dw
dw
+ u Vw = w. Vu+ VV V
Dt.
where w = V X u denotes the vorticity. We assume inviscid fluid and conservative body force, and thus only the vortex stretching term Vu is non-zero on the RHS. In this case, Eqn. 7 becomes the vorticity transport equation, saying that vorticity is stretched and advected by the velocity flow.
This physical model has been widely used in fluid simulation (Cottet et al., 2000), known as vortex methods. In particular, we consider the vortex particle methods (Selle et al., 2005), where we use a particle-base based representation for vorticity, which is low-dimensional, is easy to temporally evolve, and well embeds the circular physical flow structures in it. We represent a vortex particle p at some time stamp by triplet (4p, sp.ww/ of its intensity Ip. position x and vorticity wy. Our vorticity flow Usor (x, t) induced by vortex particles is represented by:
Voon (x, t) = Σ IN xp),
Np p = (x,, -0)xx//xxx x|I.
3pp K(X (xxxxxx))eee x)))a,
(8)
where K(x) = exp(- | 2 2 / (2x) 3/22) is Gaussian distribution kernel. Intuitively, every vortex particle "carries" local circular momentum, which is itself transported with the flow prescribed by Eqn. 7. Collectively, a set of vortex particles allows learning complex flow details which are hard to capture by the base velocity field.
However, naively learning the vortex particles leads to poor local minima due to the complex interdependence of and [Tp. xf, Thus, we make two simplifications. First, we assume that most energy is captured by the base flow and only use the base flow to transport vortex particles. This is achieved by first learning the base flow and then freezing the base flow to learn the residual flow. Second, we introduce a seeding strategy to disentangle learnable parameters from the particle transport: we seed overly many vortex particles on high-curl spatio-temporal locations and pre- compute the trajectory {xL]} and } for all and p by the learned base flow. Therefore, the learnable parameters are reduced to }. The redundant vortex particles are automatically suppressed by learning to have zero intensities. We leave more details of the seeding strategy in the Appendix.
Joint inference of fluid fields. It is a highly ill-posed problem to recover density and appearance from sparse visual observations, the imaging process for semi-trane transparent fluids involves complex non-linear absorption and scattering. The visual appearance of fluids depends not only on density but also lighting and fluid substance properties. Therefore, recovering turbulent, under-constrained continuous density fields from limited observations is inherently challenging.
To address this challenge, we leverage both visual imaging signals and physical fluid transport constraints by jointly learning density, appearance, and velocity. The visual signal supervision through differentiable volume rendering is given by:
Lrender
E. o.d,
[|] Trender (o, d) Trobservve (o, d)13].
where Trenter is our volume rendered values by Eqn. 3 and Lotnerve is sampled from video frames. Notice that we sample rays continuously in the viewing frustums instead of sampling through only pixel centers as NiRF-like methods do (Mildenhall et al., 2020), as we aim to learn continuous fields
5

---

and we rely on good partial derivatives from the fields. The physical transport constraint is provided by Eqn. 4, i.e. we supervise the partial derivatives of the density field. We compute all derivatives by auto-differentiation (Paszke et al., 10. Our loss function is given by:
L = Crendrr + Puensity Licensty + Pproofpup + Plaminar Glaminar.
(10)
where all the Es are loss weights.
Another inherent ambiguity is between emitting radiance and density. As in Eqn. 3, the emitting radiance is multiplied by density, which means that an unconstrained ppatiall--varying emitting radiance field admits all sorts of trivial solutions for density. Empirically, we find little differences between using learned spatially-varying color and using a constant color. Thus, we set emitting radiance be constant to remove this ambiguity.
Related Work
Fluid reconstruction. Fluid flow reconstruction from visible light measurement has been widely studied in science and engineering. Well-established methods used in controlled lab environments include active sensing techniques such as laser scanners (Hawkins et al., 2005), light path (Ji et al., 2013), and structural light (Gu et al., 2012), as well as passive marker-based methods such as particle imaging velocimetry (PIV) (Adrian & Westerweel, 2011; Elsinga al., 2006) which injects passive particles into the fluid flows. These methods allow accurate flow measurement, yet they require specialized setup for markers, lighting, and capturing devices.
Recent methods also seek to reconstruct fluid flows from casual visible light measurements without a specialized setup. Earlier works use tomography and linear imaging formation to reconstruct density grids from visual observations (Gregson et al., 2014; 4; Okabe et al., 2015) and then estimate velocity using physical priors (Gregson al., 2014; Eckert et al., 2018). This line of work has been extended to joint optimization for both velocity and density (Eckert et al., 2019) to improve reconstruction quality. Since this is a highly unconstrained problem, synthesized view supervision is proposed to regularize the reconstruction (Zang et al., 2020). Franz et al. (2021) introduce a density-based rendering formation with a global differentiable simulation framework. They use manually calibrate lighting directions and fluid source location to allow consistent differentiable simulation optimization. However, these methods ignore the visual appearance, suffering from the inherent visual ambiguity of fluid density and appearance.
Neural dynamic scene representations. Using neural networks as implicit visual scene repre- sentations have been made effective and popular. Earlier works use neural representations for geometry (Park et al., 2019; Mescheder et al., 2019) and visual appearance (Sitzmann et al., 2019). As a seminal work, neural radiance fields (NeRFs) (Mildenhall et al., 2020) incorporate volume rendering to learn implicit geometry and appearance from images. Follow-ups of NeRFs extend ap- plications from novel view synthesis to dynamic scene reconstruction (Li et al., 2021; Pumarola et al., 2021), relighting (Zhang et al., 2021; Yu et al., 2023), scene segmentation (Yu et al., 2022; Sajjadi et al., 2022), robot manipulation (Le Cleac"h h et al., 2023; Tian et al., 2023), system identification (Li al., 2023), etc.
Generic dynamic NeRFs incorporate deformation fields (Park et al., 2021 b;; Pumarola et al., 2021; Park et al., 2021a) or scene flow fields (Li et al., 2021; Du et al., 2021; Xian et al., 2021; Li et al., 2022) to represent motion. These motion representations essentially rely on stable visual features such as color gradients and edges, and thus they are not suitable for uncovering fluid flows which generally do not exhibit stable visual features. Notably, (Chu et al., 2022) is the most relevant work to ours. Chu neural fields (PINF) that incorporate physics-informed losses (Raissi et al., 2019) to reconstruct fluid flows from sparse-view videos with learned priors from synthetic data. However, PINF does not consider the inherent physical structures of real fluid flows and thus only reconstructs laminar flows. Our HyFluid allows uncovering turbulent real flows by novel physics- based losses and the hybrid neural velocity representation.
et al. (2022) propose the mhysics-infommee
Learning fluid dynamics. Learning fluid dynamics holds the promise to accelerate simulation and aid fluid reconstruction. Earlier work on learning fluid dynamics to accelerate simulation includes using convolutional networks to evolve fluid states (Tompson et al., 2017; Ummenhofer et al., 2019; Prantl et al., 2022) or learning latent simulation (Wiewel et al., 2019; Kim et al., 2019), supervised by 3D

---

NeuroFluid
NeRFlow
PINF
HyFluid (Ours)
GT
Figure 3: Visualization of novel view synthesis results on real capture. Note that ours more faithfully recovers the density distribution of the fluid without floaters missing regions.
GlobTrans
NeRFlow
PINF
HyFluid (Ours)
Ground truth
Figure 4: Visualization of re-simulation results on a real capture. Ours synthesizes reasonable re-simulation.
fluid simulation data. While these methods learn from 3D fluid data, few recent work such Guan et al. (2022) and Liu et al. (2023) learn simulators from multi-view videos. However, both of them assume to have groundtruth reconstructed initial fluid states. restricting the applications synthetic data. Thus, it still remains open how existing methods may help fluid reconstruction. An exception Deng al. (2023) that use dinndimenioona vortex particles to represent fluid velocity fields and thus allow reconstructing fluid velocity from single video, yet their formulation is restricted to 2D domain due complex vortex stretching in 3D vortex dynamics.
Experiments
Datasets. We use both real captures and synthetic simulation for evaluation. For real captures, we use the ScalarFlow dataset (Eckert al., 2019) which consists of videos of buoyancy-driven rising smoke plumes. We use the first five scenes from the real captures. For each scene, there are five video recordings. The five cameras are fxeedpoostion throughout capture and distributed evenly across 120" are centered at the rising smoke. Each video has 150 frames with 1062 x 600 resolution. These videos have been post-processed remove backgrounds. We follow Chu al. (2022) to use 120 frames for each video. For each scene, we use four videos for training, and one held-out video for testing (i.e., as the groundtruth for the novel view).
Synthetic simulation allows evaluating 3D velocity and density against the groundtruth. We use ScalarFlow synthetic dataset generation code (Eckert et al., 2019). We generate five examples with different inflow source with higher viscosity and another five examples with lower viscosity.
Evaluation metrics. For real fluid flows we do not have volumetric 3D groundtruth, we use novel view video rendering to evaluate the reconstruction quality. In particular, we use the following three tasks: novel view video synthesis, novel view re-simulation, and novel view future prediction. Re-simulation and future prediction require high-fidelity velocity reconstruction and thus provide means to evaluate velocity reconstruction. We use peak signal-noise ratio (PSNR), structural similarity index measure (SSIM), and the perceptual metric LPIPS (Zhang et al., 2018).
For synthetic data, we can evaluate the reconstruction results against the simulation groundtruth. Since the simulation groundtruth are up to scale, we use scale-invariant RMSE to measure performance. We only compute metrics where groundtruth density is greater than 0.1 to rule out empty space (which is otherwise dominant) for clearer quantitative comparison. In particular, we consider volumetric density enror (by querying density networks at the simulation grid points) to
7

---

t=0
1=7
1=14
1=21
t=0
t=7
t=14
t=21
t-0
t-7
t-14
t-21
PINF
HyFluid (Ours)
Ground truth
Figure 5:
Visualization of future prediction results on real capture. Notice how plume expands while PINF (Chu et al., 2022) only moves the plume up.
Novel view synthesis
Re-simulation
Future prediction
Models
PSNR
SSIM
LPIPS
PSNRT
SSIMf
LPIPS)
PSNR1
SSIM LPIPS
NeRFlow (Du et al., 2021)
18.01
0.7880
0.1958
17.20
0.8174
0.1658
GlobTran (Franz et al., 2021)
25.97
0.9312
0.0783
24.55
0.8988
0.1017
NeuroFluid (Guan et al., 2022)
22.41
0.8452
0.1560
PINF (Chu et al., 2022)
29.55
0.9244
0.0881
24.97
0.9109
0.1278
24.19
0.8366
0.2155 0.8448
HyFluid (ours)
31.14
0.9330
0.0966
28.37
0.9158
0.1171
26.12
0.1968
Table 1: Comparison on renderings for three tasks on real captures.
evaluate density prediction, and warp error (i.e., using velocity to advect density and comparing to GT density) to evaluate both density and velocity prediction.
Baselines. We consider three recent flow reconstruction methods: NeRFlow (Du et al., 2021), a neural dynamic reconstruction method for fluid reconstruction: GlobTran (Franz et al., 2021), grid-based fluid reconstruction method; and PINF (Chu et al., 2022), the latest neural fluid reconstruction method. We also compare to NeuroFluid (Guan et al., 2022), recent fluid dynamics learning approach. Since NeuroFluid requires having 3D density and velocity at the first frame, we use PINF reconstructed fields for it. We leave training details and compute resources in Appendix.
5.1 Novel view video synthesis
In this task, we aim to evaluate the inferred density fields by synthesizing novel view videos. We use all 120 frames from the held-out test view for all scenes and report the average numbers. As shown in Table I, our method is comparable to the state-of-the-ard method PINF (Chu et al., 2022). Notice that ours is better in PSNR which is objective metric that measures the optical similarity of the synthesized image and the groundtruth. GlobTrans (Franz et al., 2021) uses adversarial loss together with specific rendering formulation, which may contribute to better perceptual LPIPS performances. Yet, this compromises objective fidelity as reflected by lower PSNR. Qualitative results in Fig. showcase the capability of our method to accurately reconstruct density fields, which paves the way for the inference of high-fidelity physically-plauibg velocity fields.
5.2 Re-simulation
The re-simulation task accesses the ability of the models to infer the velocity fields accurately. Specifically, we use the reconstructed density from the first frame and advect the density across time the last frame using the learned velocity. We use the standard MacCormack method (Selle al., 2008) for advection. As for fluid source, we use the reconstructed density field in the bottom of the scenes. Results in Table and Fig. 4 show that our method generates reason- able results, whereas PINF (Chu al., 2022) and NeRFlow (Du et al., 2021) completely fail.
PINF
HyFluid (Ours)
Reference
Figure 6: Velocity slice on a real capture.

---

Figure 7:
Visualization of velocity field inferred by HyFluid from real 3-view capture.
Low vis.
Den. emr
em...
PSNR
SSIM
IPIPS..
PINE Ours
23.43
2.94
3.37
27.28
0.8616
0.1285
High vis.
Den. emr.
PSNRT
SSIMT
LPIPS]
PINE Ours 2.85
4.93
27.26
0.8728
0.1537
3.21
28.57
0.8670
0.1233
Table 2: Evaluation on synthetic data of different lev- els of viscosity.
PINF
HyFluid (Ours)
GT
Figure 8: Novel view re-simulation on synthetic data.
We further visualize the velocity field by slicing it in Fig. 6, which demonstrates that method effectively captures the velocity details that are missed by the baseline methods, which tend to default trivial solutions where all the velocity is upward-directed.
5.3 Future prediction
In the future prediction task, we extrapolate the fluid motion into the future based the inferred velocity fields at single frame. In particular, we follow the standard grid-based fluid simulation (Fed- kiw al., 2001) evolve the velocity fields by self-advecting them using the MacCormack method, followed by a projection step to ensure the divergence free constraint. Table shows that our method predicts future frames that are quantitatively better than PINE. Fig. illustrates that compared to PINF, the future fluid state predicted by our method maintains its original structure and continues follow natural upward trajectory, and also produces natural details. These results underline the robustness of our method in inferring complex fluid velocity.
5.4 Inference in-the-wild real capture
While ScalarFlow dataset provides real captures of plumes from a controlled environment, we are also interested in inferring fluid fields from uncontrolled in-the-wild videos. We use three cameras to capture sparse multi-view videos of lit incense, and we show visualization of the inferred density (top row) and velocity (bottom row) in Fig. While the fluid in this example is much thinner than plumes, HyFluid allows plausible inference of fluid fields.
5.5 Evaluation on synthetic data
We include synthetic examples for evaluating 3D density and velocity fields against the groundtruth. We compare PINF which has shown state-of-ther results in 3D fluid fields reconstruction. We show 3D fields evaluation results together with novel view re-simulation results in Table 2 and Fig. 8. observe that ours outperforms PINF on both 3D fields and 2D rendering, especially in preserving the vortical structures and intricate shape details. This is validated by larger performance margins in low-viscosity examples, since there is more turbulence that is hard to capture by PINF.
5.6 Additional applications
The inherent properties neural representation our method allow for its applicability a variety of additional contexts, such editing and scene composition. As seen in Fig, 1, the results obtained

---

Reconstructed
Edited
Naive ++ +rrject
Full
GT
Figure Visualization of our turbulence editing by intensifying the learned vortex particles on a real capture.
Figure 10: Ablation study by re-simulation.
from our method can be effortlessly integrated into dynamic NeRF scenes (Sara Fridovich- Keil and Giacomo Meanti et al., 2023) imported into professional-gne grade graphics software.
Our hybrid neural velocity representation naturally supports turbulence editing in re-simulation. In particular, multiply vortex particle intensity by factor of 4 and showcase visualization of edited re-simulation Fig. 9. This shows that the high-quality velocity estimation from our hybrid neural velocity representation allows easily synthesizing vortical flow details.
5.7 Ablation study
We show qualitative results in Fig. 10 evaluate the proposed physics-based losses, the vortex particle-driven velocity representation, and the joint training supervision. Starting from "Naive" where we only use the rendering loss for recovering density and the density loss for recovering velocity, we gradually add back laminar loss ("+minainar)") projection loss ("+ "), and vortex particle-driven residual velocity ("Full"). From Fig. 10 we observe that our physical losses allow better velocity recovery. In particular, the projection loss enables recovering physically correct velocity that reconstructs the plume shape re-simulation. Vortex particle-driven velocity allows capturing more details and thus more consistent re-simulation.
6 Conclusion
In this work, we study recovering fluid density and velocity from sparse multi-view videos. We propose hybrid neural fluid fields (HyFluid) which features physics-based losses address inherent visual ambiguities and hybrid neural velocity representation for capturing the complex turbulent fluid flow, We show that our simple designs can already lead to physically plausible estimation of fluid fields that supports applications such as re- re-simulation, editing, and dynamic scene composition.
Limitations. Our method assumes inviscid fluid and only considers gas but not liquid. Liquid features free surface that may require further physical modeling and constraints.
Acknowledgments. This work was in part supported by the Toyota Research Institute (TRI, NSF RI #221 1258, ONR MURI 100014-22-1-221 the Stanford Institute for Centan-Centered AI (HAI), and Google. B. Zhu acknowledges NSF IIS #2313075, IIS #2106733, and CAREER #2144806.
References
Ronald J Adrian and Jerry Westerweel. Particle image velocimetry. Cambridge university press, 2011. 6
Steven F Ashby and Robert D Falgout. parallel multigrid preconditioned conjugate gradient algorithm for groundwater simulations. Nuclear science engineering, 1996.
Peter Bauer, Alan Thorpe, and Gilbert Brunet. The quiet revolution of numerical weather prediction. Nature, 2015. 1
Dennis Bushnell and KJ Moore. Drag reduction in nature. Annual review fluid mechanics, 1991.
Mengyu Chu. Lingjie Liu, Quan Zheng, Erik Franz. Mlans-Peted Seidel, Christian Theobalt, and Rhaleb Zayer. Physics informed neural fields for smoke reconstruction with sparse data. ACM TOG, 2022. 6, 7,8
Georges-Hend Henri Cottet, Petros D Koumoutsakos, et al. Vortex methods: theory and practice, volume 8, Cambridge university press, 2000.
10

---

Yitong Deng, Hong-Xing Yu. Jiajun Wu, and Bo Zhu. Learning vortex dynamics for fluid inference and prediction. In ICLR, 2023.
Yilun Du, Yinan Zhang, Hong-Xing Yu, Joshua B Tenenbaum, and Jiajun Wu. Neural radiance flow for 4d view synthesis and video processing. In ICCV, 2021. 2, 6, 8
M-L Eckert, Wolfgang Heiddich,, and Nils Thuerey. Coupled fluid density and motion from single views. CGF, 2018. 6
Marie-Lena Eckert, Kiwon Um, and Nils Thuerey. Scalarflow: large-scale volumetric data of real-world
scalar transport flows for computer animation and machine learning. ACM TOG, 2019. 6, 7
Gerrit E Elsinga, Fulvio Scarano, Bernhard Wieneke, and Bas van Oudheusden. Tomographic particle image velocimetry. Experiments in fluids, 2006. 6
Ronald Fedkiw, Jos Stam, and Henrik Wann Jensen. Visual simulation of smoke. SIGGRAPH. 2001. 14
Erik Franz, Barbara Solenthaler, and Nils Thuerey. Global transport for fluid reconstruction with learned sulffsuupriision. In CVPR, 2021. 6, 8
Uriel Frisch and Russell Donnelly. Turbulence: the legacy of an kolmogorov. Physics Today, 1996. 2
James Gregson, Ivo Ihrke, Nils Thuerey, and Wolfgang Heidrich. From capture to simulation: connecting forward and inverse problems in fluids. ACM TOG, 2014.
Jinwei Gu. Shree Nayar, Eitan Grinspun, Peter N Belhumeur. Ravi Ramamoorthi. Compressive structured light for recovering inhomogeneous participating media. IEEE TPAMI, 2012. 6
Shanyan Guan, Huayu Deng, Yunbo Wang, and Xiaokang Yang. Neurofluid: Fluid dynamics grounding with
particle-driven neural radiance fields. In ICML, 2022. 8,8
Tim Hawkins, Per Einarsson, and Paul Debevec. Acquisition of time-varying participating media. ACM TOG, 2005. 6
Yuanming Hu, Tzu-Mao Li, Luke Anderson, Jonathan Ragan-Kelley, and Frédo Durand. Taichi: a language for high-performance computation spatially sparse data structures. ACM TOG, 2019. 14
Yu Ji, Jinwei Yc, and Jingyi Yu. Reconstructing gas flows using light-path approximation. In CVPR, 2013. 6
James T Kajiya and Brian P Von Herzen. Ray tracing volume densities. SIGGRAPH, 1984. 3
Byungsoo Kim, Vinicius C Azevedo, Nils Thuerey, Thcodore Kim, Markus Gross, and Barbara Solenthaler. Deep fluids: A generative network for parameterized fluid simulations. In CGF, 2019.
Simon Le Cleac'h, Hong-Xing Yu, Michelle Guo, Taylor Howell, Ruohan Gao, Jiajun Wu, Zachary Manchester, and Mac Schwager. Differentiable physics simulation of augmenes-augmentte augmented neural objects. IEEE RA-L.
2023. 6
Tianye Li, Mira Slaveheva, Michael Zollhoefer, Simon Green, Christoph Lassner, Changil Kim, Tanner Schmidt, Steven Lovegrove, Michael Goesele, Richard Newcombe, al. Neural 3d video synthesis from multi-view video. In CVPR, 2022. 6
Xuan Li, Yi-Ling Qiao. Peter Yichen Chen, Krishna Murthy Jatavallabhula, Ming Lin, Chenfanfu Jiang. and Chuang Gan. Pac-nerf: Physics augmented continuum neural radiance fields for geometry-agnostee system identification. In JCLR, 2023. 6
Zhengqi Li, Simon Niklaus, Noah Snavely, and Oliver Wang. Neural scene flow fields for space-time time view synthesis of dynamic scenes. In CVPR, 2021. 2, 6
Jinxian Liu, Ye Chen, Binghing Ni, Jiyao Mao, and Zhenbo Yu. Inferring fluid dynamics via inverse rendering. arXiv:2304.0444 04446. 2023. 7
Nelson Max. Optical models for direct volume rendering. IEEE TVCG, 1995. 2,
Lars Mescheder, Michael Occhsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. Occupancy networks: Learning 3d reconstruction in function space. In CVPR, 2019. 6
Ben Mildenhall, Pratul P Srinivasan. Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. arXiv:2003, 0889344 2020. 3, 5, 6
11

---

Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with multiresolution hash encoding, ACM TOG, 2022. 5, 14
Makoto Okabe, Yoshinori Dohashi, Ken Anjyo, and Rikio Onai. Fluid volume modeling from sparse multi-view images by appearance transfer. ACM TOG, 2015.
Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In CVPR, 2019. 6
Keunhong Park, Utkarsh Sinha, Jonathan T Barron, Sofien Bouaziz, Dan B Goldman, Steven M Seitz, and Ricardo Braaila Nerfics: Deformable neural radiance fields. In CVPR, 2021a.
Keunhong Park. Utkarsh Sinha, Peter Hedman, Jonathan T Barron, Sofien Bouaziz, Dan B Goldman, Ricardo Martin-Brualla, and Steven Seitz. Hypernerf: A diaeee-dimensional representation for topologically varying neural radiance fields. arXiv:2106.1322 2021b. 6
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin. Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. In NeurlPS Workshops, 2017. 6
Tobias Pfaff, Nils Thuerey, Jonathan Cohen, Sarah Tariq, and Markus Gross. Scalable fluid simulation using anisotropic turbulence particles. In SIGGRAPH Asia, 2010.
Stephen Pope. Turbulent flows. Cambridge university press, 2000. 2
Lukas Prantl, Benjamin Ummenhofer, Vladlen Koltun, and Nils Thuerey. Guaranteed conservation of momentum for learning particle-based fluid dynamics. In NeurlPS, 2022. 6
Albert Pumarola, Enric Corona, Gerard Rons-Moll, and Francese Moreno-Nogus D-nerf: Neural radiance fields for dynamic scenes. In CVPR, 2021. 6
Nasim Rahaman, Aristide Baratin, Devansh Arpit, Felix Draxler, Min Lin, Fred Hamprecht, Yoshua Bengio, and Aaron Courville. On the spectral bias of neural networks. In ICML, 2019. 2.
Maziar Raissi, Paris Perdikaris, and George E Karniadakis. Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational physics, 2019. 1,
Mehdi SM Sajjadi, Daniel Duckworth, Aravindh Mahendran, Sjoerd van Steenkiste, Filip Pavetic, Mario Lucic, Leonidas Guibas, Klaus Greff, and Thomas Kipf. Object scene representation transformer. NeurlPS, 2022.
Sara Fridovich-Kan and Giacomo Meanti, Frederik Rahbæk Warburg, Benjamin Recht, and Angjoo Kanazawa. K-planes: Explicit radiance fields space, time, and appearance. CVPR, 2023.
Andrew Selle, Nick Rasmussen, and Ronald Fedkiw, A vortex particle method for smoke, water and explosions. In SIGGRAPH, 2005.
Andrew Selle, Ronald Fedkiw, Byungmoon Kim, Yingjic Liu, and Jarek Rossignac. An unconditionally stable maccormack method. Journal of Scientific Computing, 2008. 8
Vincent Sitzmann, Michael Zollhöfer, and Gordon Wetzstein. Scene representation networks: Continuous 3d-structure-awaan neural scene representations. In NewrIPS, 2019.
Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein. Implicit neural representations with periodic activation functions. NewrlPS, 2020. 5
Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Fridovich-Kan Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. In NewrIPS, 2020. 5
Stephen Tian, Yancheng Cai, Hong-Xing Yu, Sergey Zakharov, Katherine Liu, Adrien Gaidon, Yunzhu Li, and
Jiajun Wu. Multi-object manipulation via object-centric neural scattering functions. In CVPR, 2023. 6
Jonathan Tompson, Kristofer Schlachter, Pablo Sprechmann, and Ken Perlin. Accelcrating culerian fluid
simulation with convolutional networks. In ICML, 2017. 6
Benjamin Ummenhofer, Lukas Prantl, Nils Thuerey, and Vladlen Koltun. Lagrangian fluid simulation with
continuous convolutions. In ICLR, 2019. 6
12

---

Sundar Vedula. Simon Baker, Peter Rander, Robert Collins, and Takeo Kanade. Three-dimensional scene flow. In ICCV, 1999. 4
Steffen Wiewel, Moritz Becher, and Nils Thuerey. Latent space physics: Towards learning the temporal evolution of fluid flow. In CGF, 2019. 6
Wenqi Xian, Jia-Bin Huang, Johannes Kopf, and Changil Kim. Space-time neural irradiance fields for free- viewpoint video. In CVPR, 2021. 6
Long-Xing Yu, Leonidas J Guibas, and Jiajun Wu. Unsupervised discovery of object radiance fields. ICLR, 2022. 6
Hong-Xing Yu, Michelle Guo, Alireza Fathi, Yen-Yu Yu Chang, Eric Ryan Chan, Ruohan Gao, Thomas Funkhouser, and Jiajun Wu. Learning object-centric neural scattering functions for free-viewpoint relighting and scene composition. TMLR, 2023.
Guangming Zang, Ramzi Idoughi, Congli Wang, Anthony Bennelt, Jianguo Du. Scott Skeen, William L Roberts, Peter Wonka, and Wolfgang Heidrich. Tomofluid: Reconstructing dynamic fluid from sparse view videos. In CVPR, 2020.
Zhang, Fujun Luan. Qianqian Wang. Kavita Bala, Noah Snavely. Physg: Inverse rendering with spherical gaussians for physics-based material editing and relighting. In CVPR, 2021. 6
Richard Zhang. Phillip Isola, Alexei Efros, Eli Shechtman, Oliver Wang. The unreasonable effectiveness of deep features as perceptual metric. In CVPR, 2018. 7
13

---

A Appendix: Implementation details
Model architecture. For both the density field (x,, y, 2, 1) and the base velocity field [25 (2; ys 2, we use a 4D extension of iNGP (Müller et al., 2022). Specifically, we add temporal dimension to the original static iNGP. For the spatial dimensions, we use a base resolution 16 and the finest resolution 256. For the temporal dimension we set the finest resolution to 128 which is comparable the number of video frames (120 frames), as higher resolutions create hash encoding features that are never directly supervised and can lead to unstable gradients.
Training and hyperparameters. Our training can be divided into three stages. In the first stage, pre-train our density network (2, y, 2, t) and the constant learnable appearance (radiance) L. using initial estimate of both quantities. In the second stage,
the rendering loss only, which forms jointly train the density ( y, 2, t), appearance Le. and the base velocity Upuse (r, y. 2, t) using the full loss. Finally, we jointly train the density (2, y, 2, t), appearance L,, and the vortex particles }. We use an Adam optimizer with a learning rate 0.01. In the first stage, we train the density and radiance for 200, 000 iterations. In the second stage, we jointly train the model for 50, 000 iterations. In the third stage, we do training for 5, 000 iterations. We empirically set the loss weights to Siender 10, 000, Sacesity- 0.001, Spooj 1, Praminar coefficient = 0.2.
10. For the laminar loss, we the
Projection loss. For the projection routine, we multi-grid preconditioned conjugate gradient (MGPCG) solver (Ashby & Falgout, 1996). We implement it using Taichi (Hu et al., 2019) language. We use three levels of mulliigrid, with the spatial resolution 1283. Note that use the same resolution for our re-simulation and future prediction experiments for all compared methods.
Seeding strategy. In our experiments, we seed 50 vortex particles for each scene. To place initial vortex particles, we take the insight from vortex confinement methods (Fedkiw et al., 2001). We densely sample continuous spatio-tempora locations and compute curl values from the trained base velocity network, and then we place vortex particles locations with the highest curl values.
Compute usage. We train our model on a single A100 GPU for around hours in total. The first stage takes half an hour, the second stage takes around 8 hours (this is significantly slower than the first stage as it requires computing derivatives for the density loss), and the third stage takes half an hour.
14


Reviews:
[{"review_id": "UFoFczvvm2", "reviewer": "Reviewer_5YF8", "summary": "In this paper, the authors propose a method (HyFluid) to infer the fluid density and velocities from multiview videos. To deal with the visual ambiguities of fluid, physics-losses are introduced to try to enforce physics plausible velocities. A neural velocity field and a vortex particle-based velocity are introduced to capture the irrotational energy and model residual turbulent velocity respectively. Th experiments show that the proposed method has the ability to recover the vortical flow details to some extent.", "questions": "- Does the MGPCG applied to every simulation step to project the velocity during the training? How is the runtime cost for computing the projected velocity?\n- Why to encourage high-density regions to have non-zero velocity (line 137)? What is the physics intuition here?\n- As the visual appearance of fluid highly depends on the rendering process, does the results very sensitive to the parameters of the renderer?\n", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is overall well written and organized. \nThe proposed density loss and projection loss are well derived from the fluid mechanics.\nThe experiments show that the proposed method can perform better than the baselines.\n", "weaknesses": "The authors claim that the method can give physics plausible estimation of fluid fields. The physics intuitions for the density loss and projection are clear according to the incompressible condition. However, I don't see obvious physics intuition for the laminar loss.\n\nFor the problem that visual appearance of fluids depends on lighting and fluid substance properties, there is no further exploration in this direction. The problem becomes more important when the fluid is liquid.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time and comments! Please see our response below.\n\n- **Physical intuition on laminar loss**: \nLaminar flow does not manifest local density change, and thus inferring it from pure visual observations is challenging. We introduce this regularization term to account for the fact that even in constant-density fluid regions, there can still be laminar flow. Since this also depends on prior knowledge of the fluid to reconstruct, the laminar loss takes a flexible form: the hyper-parameter $\\gamma$ models the prior belief of having laminar flow, e.g., when $\\gamma=0$ it allows zero velocity even in high-density regions.\n\n- **Further exploration on visual appearance**:\nFollowing your suggestion, we allow learning spatially and temporally varying appearances. Please see the results and discussion in the \"Spatially varying appearance\" in the global response above.\n\n- **Rendering parameter**:\nWe use volumetric rendering in our formulation (Eq. (3) in the main paper). The parameters include {near plane $t_n$, far plane $t_f$, number of samples for numerial integration $N$}, where $t_n$ and $t_f$ are determined by centering it at the scene center (which is the geometric center of camera principal rays) and empirically scale it. We find that as long as there are enough samples (in our case, more than 64), the numerical integration is stable and thus the results are not sensitive to these parameters.\n\n- **On MGPCG**:\nWe use MGPCG in every training step to compute the projection loss. We implement MGPCG using Taichi [Hu2019] that allows GPU acceleration. We use three levels with resolution 128^3. On average, this costs around 100ms per step. Thanks to our efficient implementation, the whole training takes only ~9 hours on a A100 GPU, as noted in our supplementary material.\n\n**Reference**:\n\n[Hu2019] Hu, Y., Li, T. M., Anderson, L., Ragan-Kelley, J., & Durand, F. (2019). Taichi: a language for high-performance computation on spatially sparse data structures. ACM Transactions on Graphics (TOG), 38(6), 1-16."}, {"review_id": "SlG2N0FqZG", "reviewer": "Reviewer_qUcR", "summary": "This paper works on reconstructing fluid density and velocity from multi-view videos. The main idea is to inject visual clues with NeRF. They propose some physics-based regularization terms to deal with visual ambiguity that video can not reflect the inner fluid states. ", "questions": "For details, please refer to the weaknesses section. ", "limitations": "yes", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "Good motivation: Visual ambiguity indeed is the critical problem that needs to be solved in this setting. \n\nNovel framework: Introducing physical losses is a not-easy but natural point. The introduced losses are general to inspire other related researchers.\n\n\n", "weaknesses": "\n1. Setting is not new. Indeed, reconstructing fluid velocity and density from videos has been studied in NeuroFluid (ICML 2022). I didn't find the introduction or comparison. Why?\n\n2. The proposed method is not convincing to me. \n\n   -  The proposed physics-based losses can only play a regularization role in the training. To solve the visual ambiguity, the framework must introduce direct supervision of the fluid velocity. Nevertheless, ambiguity still exists. \n\n   - And, the key question is how to get the initial state of the fluid, which decides the performance of this work. The authors **should** clarity this point.\n\n3. Experiments are incomplete, lacking key results. \n\n    - Experiments just show the render results and velocity results. How about density? The authors claim that they can recover fluid density but not shown in the experiments. \n\n    - How did you evaluate the render results? Directly render fluid from the view angle used in the training stage? Please claim them before your experiments.\n    - Only evaluate the stoke with a similar environment. Can you show more examples, with more complex shapes and materials?\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time and comments! Please see our response below.\n\n- **Relation to NeuroFluid**: \nWe clarify that our setting is different from NeuroFluid. NeuroFluid focuses on learning fluid dynamics from a large amount of data. Therefore, they train and evaluate the model on synthetic data, yet the generated data distribution can be very different from real data to generalize. Moreover, they assume known initial state and no inflow source, which do not hold in some real scenes such as smoke plume scenes we used. In contrast, our goal is to reconstruct plausible fluid fields from real sparse multiview videos without assuming additional training data, so as to facilitate applications on real fluid videos such as novel view synthesis, re-simulation, future prediction, and turbulence editing.\n\n- **Comparison to NeuroFluid**:\nWe show a comparison to NeuroFluid in the \"Comparison to NeuroFluid\" in the global response above. Please note that since we use a real dataset for evaluation (as specified in L201 in our main paper), we do not have groundtruth initial states that NeuroFluid requires as input. Therefore, we use the previous state-of-the-art method PINF [Chu2022] to reconstruct the first frame for the initial state of NeuroFluid. We will add reference and discussion to our paper.\n\n- **Physics-based losses for visual ambiguity and supervision from velocity**:\nWe clarify that we aim at reconstructing plausible fluid velocity field from real videos to allow re-simulation and future prediction. We do not assume training data of groundtruth velocity as it is very scarce in real scenes. Therefore, we propose physics-based losses to regularize the recovery of fluid velocity such that they are physically plausible for the downstream applications.\n\n- **Initial state of fluid**:\nDifferent from NeuroFluid which requires initial states and learns fluid dynamics, we aim at reconstructing the fluid fields. Thus, the \"initial state\" is our model output rather than input.\n\n- **Evaluating 3D density**:\nWe clarify that all our experiments are done on **real videos** which do not have groundtruth 3D density and velocity fields. Therefore, we indirectly evaluate them by downstream applications including novel view synthesis, re-simulation, and future prediction. Following your suggestion, we additionally evaluate our method on synthetic examples which provide groundtruth 3D density. Please refer to the \"Evaluation on synthetic data\" in the global response above to see the results and discussion.\n\n- **Evaluating rendering results**:\nWe clarify that we evaluate rendering results in a hold-out novel view that is unseen during training (L205-L206 in the main paper). In particular, each example in the ScalarFlow real dataset and synthetic dataset has 5 views. We take 4 views for training, and 1 view for testing. We use this training-testing split for all our experiments including novel view video synthesis, novel view re-simulation, and novel view future prediction.\n\n- **Experiments on different examples**:\nPlease note that the additional synthetic data in \"Evaluation on different viscosity levels\" in the global response demonstrate different material properties (e.g., they are more viscid than real smokes), shapes and inflows. We believe these additional examples provide more diverse evaluations.\n\n**Reference**:\n\n[Chu2022] Chu, M., Liu, L., Zheng, Q., Franz, E., Seidel, H. P., Theobalt, C., & Zayer, R. (2022). Physics informed neural fields for smoke reconstruction with sparse data. ACM Transactions on Graphics (TOG), 41(4), 1-14."}, {"review_id": "R3WIvsfyfZ", "reviewer": "Reviewer_3dGa", "summary": "This paper presents an innovative neural dynamic reconstruction method that achieves good results in recovering fluid density and velocity fields through the introduction of physical constraints and a hybrid neural velocity representation. Despite some weaknesses and issues, this method has significant implications for the in-depth study and resolution of fluid dynamics problems. Future work could focus on refining the method and validating and applying it to a wider range of fluid scenarios.", "questions": "Further investigation is needed to determine whether the method can handle complex fluid scenarios such as non-Newtonian or multiphase fluids in practical applications.", "limitations": "The paper does not mention the computational resource requirements, such as computation time and memory consumption, which are important factors for the feasibility of practical applications.", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "This method proposes a new approach to neural dynamic reconstruction that can simultaneously recover fluid density and velocity fields, overcoming the challenges posed by the visual ambiguities of fluid velocity in existing methods.\nPhysics-based losses are introduced to enforce a divergence-free velocity field, driving the transport of density and enhancing the accuracy of velocity estimation.\nA hybrid neural velocity representation is designed, incorporating a base neural velocity field that captures most irrotational energy and a vortex particle-based velocity that models residual turbulent velocity. This enables the method to effectively recover vortical flow details.", "weaknesses": "The paper lacks detailed presentation of experimental results and quantitative evaluations, thus lacking sufficient validation of the method's performance.\nThere is no comparison with other methods in addressing the visual ambiguity of fluid velocity, making it difficult to assess the method's advantages and disadvantages comprehensively.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time and comments. Please see our response below.\n\n- **Detailed presentation of experiment results**:\nWe clarify that we aim at reconstructing plausible fluid velocity field from real videos to allow re-simulation and future prediction. For real videos, it is very challenging to collect groundtruth 3D density and velocity. \nTherefore we evaluate on the applications including novel view video synthesis, novel view re-simulation, and novel view future prediction. For each of them, we show detailed quantitative results in Table 1 (three metrics for each task) and qualitative results in Figure 3, 4, 7 in our main paper. In addition, we show qualitative results on turbulence editing and velocity recovery in Figure 5 and Figure 6. These downstream applications reflect that our approach can reconstruct plausible real fluid fields.\n\n- **Additional quantitative evaluation**:\nIn addition to the experiments on real fluid videos, we include new experiments on synthetic fluid scenes. These scenes provide 3D groundtruth density and velocity, allowing quantitative evaluations on them. Please refer to the \"Evaluation on synthetic data\" and \"Evaluation on different viscosity levels\" in the global response for results and discussion.\n\n- **No comparison with other methods in addressing the visual ambiguity of fluid velocity**:\nWe respectively disagree with this. We clarify that we have comparisons to PINF [Chu2022] and NeRFlow [Du2021], both of which aim to address visual ambiguity of velocity/flow estimation from real videos, and they both showcase plausible reconstruction on fluid scenes in their results. In particular, PINF [Chu2022] is the state-of-the-art fluid reconstruction method which tries to address visual ambiguity by physics-informed losses similar to Physics-informed Neural Networks (PINN) [Raissi2019]. PINF shows extensive results in synthetic scenes and a few real examples. NeRFlow approaches this by a set of temporal consistency losses and showcase fluid reconstruction in their \"milk pouring\" scene. Our comparison to them in Table 1 and Figure 3, 4, 5, 7 clearly demonstrate that our approach achieves better results than these existing methods.\n\n- **Additional comparison with other methods**:\nIn addition to both PINF and NeRFlow, we compare to NeuroFluid [Guan2022], and GlobTrans [Franz2021]. NeuroFluid learns fluid dynamics to address velocity ambiguity. GlobTrans aims for reconstructing fluid fields using advection constraints and regularization terms to solve visual ambiguity of fluid velocity. Please refer to the \"Comparison to NeuroFluid\" and \"Comparison to GlobTrans\" in the global response for results and discussion.\n\n- **Computation time and memory consumption**:\nAs we noted in L215 in our main paper, we leave computation resource usage in our supplementary material. As in L29 in our supplementary material, we train our model on a single A100 GPU (the GPU memory usage is around 30GB) for around 9 hours in total.\n\n**Reference**:\n\n[Chu2022] Chu, M., Liu, L., Zheng, Q., Franz, E., Seidel, H. P., Theobalt, C., & Zayer, R. (2022). Physics informed neural fields for smoke reconstruction with sparse data. ACM Transactions on Graphics (TOG), 41(4), 1-14.\n\n[Du2021] Du, Y., Zhang, Y., Yu, H. X., Tenenbaum, J. B., & Wu, J. (2021, October). Neural radiance flow for 4d view synthesis and video processing. In 2021 IEEE/CVF International Conference on Computer Vision (ICCV) (pp. 14304-14314). IEEE Computer Society.\n\n[Raissi2019] Raissi, M., Perdikaris, P., & Karniadakis, G. E. (2019). Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational physics, 378, 686-707.\n\n[Guan2022] Guan, S., Deng, H., Wang, Y., & Yang, X. (2022, June). Neurofluid: Fluid dynamics grounding with particle-driven neural radiance fields. In International Conference on Machine Learning (pp. 7919-7929). PMLR.\n\n[Franz2021] Franz, E., Solenthaler, B., & Thuerey, N. (2021). Global transport for fluid reconstruction with learned self-supervision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 1632-1642)."}, {"review_id": "KLuVVH49VF", "reviewer": "Reviewer_4cV8", "summary": "The paper presents a neural reconstruction method for individual fluid flows which is re-trained for each new scene. It combines the established iNPGs for representing densities and velocities with vorticity transporting partices in a hybrid flow representation called HyFluid. A set of appearance- and physics-based losses is used to train the iNPGs and optimize vorticity and lighting in 3 steps. With this approach it is possible to recover 3D density and velocity from sparse views and enable novel-view rendering, re-simulation, and predictions of states over time.\n", "questions": "- Since the approach is trained on ScalarFlow [2] it would be interesting to see how the method compares to the ScalarFlow reconstructions. ScalarFlow is not using neural networks, and as such a good not-learned baseline as a single scene reconstruction method using physical priors (advection, pressure projection). Re-simulation and future prediction might also be possible there since ScalarFlow is using a fluid solver (MantaFlow) internally. Can the authors explain whether they haven't compared to the ScalarFlow 3D data despite using the 2D input images?\n- It would also be interesting to compare to GlobTrans [3] as this method it is trained/optimized as a re-simulation method. Have the authors looked into this?\n- The paper uses use an unseen GT view to evaluate the metrics. Have the authors tried/considered reconstructing a synthetic case to be able to make a 3D comparison? This might also show the efficacy of the flow reconstruction more clearly, since the visual appearance could be matched perfectly.\n- The velocity flickers in the supplemental video, there is no smooth evolution. Is there any coupling of the velocity over time (like self-advection)?\n- In the videos the velocity fields are rendered to 2D. Can the authors explain how is that done? Wouldn't a slice provide more insight in the actual motions? Averaging might blur the small-scale vorticities.\n\nOverall, I think the paper targets an interesting direction and provides an interesting approach for a tough problem. I still somewhat on the edge regarding this paper, but I'm open to readjusting/raising my score after the rebuttal.\n\nReferences:\n[1] Mengyu Chu, Lingjie Liu, Quan Zheng, Erik Franz, Hans-Peter Seidel, Christian Theobalt, and Rhaleb Zayer. 318 Physics informed neural fields for smoke reconstruction with sparse data. ACM Transactions on Graphics, 2022.\n[2] Marie-Lena Eckert, Kiwon Um, and Nils Thuerey. Scalarflow: a large-scale volumetric data set of real-world 330 scalar transport flows for computer animation and machine learning. ACM Transactions on Graphics (TOG), 2019.\n[3] Erik Franz, Barbara Solenthaler, and Nils Thuerey. Global transport for fluid reconstruction with learned 337 self-supervision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021.\n", "limitations": "Limitations are briefly discussed, I don't see anything significant missing here.\n", "rating": 7, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "I see the following strong points in this submission:\n- The method is build on physical priors to recover a physically meaningful velocity field.\n- The loss to the projected velocity to ensure divergence freeness is novel as far as I can tell.\n- Using particles to transport voriticity seems novel, at least in that context. \n- Provides better results then SOTA neural methods regarding re-simulation and prediction.\n- The results look good, the motion of the density is coherent and also suited for re-simulation.\n- The reconstruction works with simple, sparse images.\n", "weaknesses": "The treatment for vorticity seems novel, but from the provided ablations I cannot see a substantial impact:\n- In Fig. 8 there seems to be a slight improvement.\n- In the videos, the ablation study results of \"w/o vort.\" and \"full\" look very similar.\n- The difference could be more visible in the flow field, and but this is not demonstrated.\n- In the videos, the novel view synthesis results of PINF [1] and HyFlow are almost identical.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful comments and constructive suggestions! Please see our response below.\n\n- **Comparison to ScalarFlow and GlobTrans**:\nThank you for your suggestion! Since both ScalarFlow and GlobTrans are optimization-based methods, we compare to GlobTrans which is a later work and it has shown better results than ScalarFlow. Please refer to the \"Comparison to GlobTrans\" in the global response above for results and discussion.\n\n- **Synthetic data for 3D evaluation**:\nFollowing your suggestion, we additionally include synthetic examples using ScalarFlow synthetic dataset generation code. Please refer to the \"Evaluation on synthetic data\" and \"Evaluation on different viscosity levels\" in the global response above for results and discussion.\n\n- **Self-advection of velocity**:\nWe do not include a physical loss for self-advection of velocity like $\\mathcal{L}=D\\mathbb{u}/Dt-\\mathbb{f}$ (where $\\mathbb{f}$ denotes external force) as we empirically found that it often leads to oversmoothed velocity fields. This may be due to that the material derivative term for velocity $D\\mathbb{u}/Dt=\\partial \\mathbb{u}/\\partial t+\\mathbb{u}\\cdot\\nabla \\mathbb{u}$ admits local trivial solutions where the velocity is spatiotemporally constant. In optimization-based methods such as GlobTrans, this local trivial solution is solved by a global optimization. However, in neural continuous reconstruction, this is not straightforward to address and we leave it as future exploration. Actually, we suspect that this is the reason that PINF (which uses a velocity advection loss) reconstructs only laminar flows for real videos.\n\n- **Render velocity field as slice**:\nWe rendered velocity field to 2D by projecting every 3D velocity vector to the camera plane and then use volumetric rendering to integrate them. This indeed smoothes the visualization. Following your suggestion, we additionally include slice rendering in Figure R6 in the global response PDF. From the comparison we can see that our velocity field recovers more vortical details than PINF. We will include this figure in our revised paper."}, {"review_id": "imj5lHpqm8", "reviewer": "Reviewer_MRuw", "summary": "The paper proposes a novel approach for recovering the density and velocity fields of inviscid fluids from sparse multiview videos. The model has two main contributions:\n- First, it incorporates physics-based losses to enforce the inference of a physically plausible velocity field that is divergence-free and drives the transport of density. This helps to deal with the visual ambiguities of fluid velocity. \n- Second, the model provides a hybrid neural velocity representation, which consists of a base neural velocity field capturing most irrotational energy and a vortex particle-based velocity modeling residual turbulent velocity. This representation enables the recovery of vortical flow details.", "questions": "1. In the paper, a grid-based representation is used for density and velocity. When implementing $L_{density}$ and $L_{project}$, how exactly are these calculations performed? Are the loss functions computed for all grid positions, or is interpolation used to compute the losses at sampled points in space?\n\n2. What is the difference in performance between decomposing the velocity field into a base neural velocity field and a vortex particle-based velocity, versus solely using high-frequency position embedding?\n\n3. In the case of using physics-based loss functions in HyFluid, how does the predicted velocity field of HyFluid compare to the ground truth (GT) velocity field quantitatively? Can HyFluid be compared to other methods in terms of velocity field quantitatively?\n\n4. The paper does not provide experimental results and discussions regarding the constraint of constant radiance. What is the capability of HyFluid with real-world scenes that exhibit spatially-varying radiance? \n\n5. Can you provide more quantitative/ qualitative results of other baselines?", "limitations": "Yes", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Originality: This paper primarily tackles the visual ambiguities in inverse rendering techniques for fluid data, focusing on resolving visual ambiguities. The integration of physics-based losses into the volume rendering framework is a rational approach. The results in Figure 8 illustrate the effectiveness of the newly proposed learning constraints in significantly improving the accuracy of the reconstruction. Moreover, the introduction of a hybrid representation of velocity fields and particle-based vortex flow showcases originality in the methodology.\n\nSignificance: The proposed model makes a significant contribution to the visual understanding of fluids, particularly smoke, fog, and gas.\n", "weaknesses": "For methodology: \n\n1. One contribution of this paper is to incorporate new forms of physical constraints in the framework of inverse rendering. However, the general idea is not entirely novel as previous work by Chu et al. (2022) has also presented a similar (albeit different) approach, which somewhat weakens the technical novelty of this paper. For example, using the density transport equation from incompressible fluid or NS equation as a loss function has been employed in other related papers. Please refer to the work from Baieri et al. (2023) and Li et al. (2023). \n\n- [Baieri et al., 2023] Fluid Dynamics Network: Topology-Agnostic 4D Reconstruction via Fluid Dynamics Priors. Arxiv, 2023. \n- [Li et al., 2023] PAC-NeRF: Physics Augmented Continuum Neural Radiance Fields for Geometry-Agnostic System Identification. ICLR, 2023. \n\n2. The constraint imposed on emitting radiance in HyFluid does present limitations in its applicability to real-world scenarios. The requirement of a constant emitting radiance may hinder accurate recovery in situations where spatially-varying lighting exists in the scene.\n\nLacking references:\n\n3. Some closely related work seems to be missing, such as NeuroFluid (Guan et al., 2022) and PAC-NeRF (Li et al., 2023), both of which also focus on visual physical inference through inverse rendering. It is important to acknowledge these works as they contribute to the existing body of literature in this field and provide valuable insights and techniques for comparison and benchmarking purposes.\n\n- [Guan et al., 2022] NeuroFluid: Fluid Dynamics Grounding with Particle-Driven Neural Radiance Fields. ICML, 2022. \n\nFor experiments: \n\n4. Is the proposed method limited to handling only inviscid fluids? It would be beneficial to evaluate the proposed method in a broader range of fluid scenarios, including different types of flows (e.g., laminar, turbulent) and varying fluid properties (e.g., viscosity, density). This would demonstrate the generalization ability of HyFluid under diverse conditions.\n\n5. Considering that the constraint of constant radiance may not be practical in complex real-world scenes, I highly recommend that the authors compare the reconstruction results obtained with and without the constraint. \n\n6. The proposed model is only compared with two existing models, which is not sufficient. To make the results more convincing, the authors could incorporate more advanced neural rendering techniques designed specifically for dynamic scenes. Additionally, given that the dataset comprises synthetic fluid simulation data, it would be beneficial for the authors to provide quantitative results regarding the reconstruction of the velocity field in comparison to the ground truth on these simulated scenes. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful comments and constructive suggestions! Please see our response below.\n\nFor methodology:\n1. **Technical novelty**:\n[Chu et.al.] is indeed most relevant. Our approach incorporates novel losses including projection loss and laminar loss as well as a hybrid representation to capture turbulent velocity fields. Our technical novelty leads to better velocity reconstruction compared to [Chu et.al.] as can be seen in novel view re-simulation and velocity visualization (especially the videos in supplementary material). [Baieri et.al.] and [Li et.al.] focus on general dynamic object and they do not consider complex fluid dynamics such as turbulence. We will add these references and discussion to our paper.\n\n2. **Spatially-varying appearance modeling**:\nFollowing your suggestion, we allow learning spatially and temporally varying appearances. Please see the results and discussion in the \"Spatiotemporally varying appearance\" in the global response above.\n\nFor reference:\n\n3. **References**:\nThank you for the note. We will add these references to our paper. We will also add a clarification that our setting is different from NeuroFluid. NeuroFluid focuses on learning fluid dynamics from a large amount of data. Therefore, they train and evaluate the model on synthetic data. Moreover, they assume known initial state and no inflow source. In contrast, our goal is to reconstruct plausible fluid fields from real sparse multiview videos without assuming any training data.\n\nFor experiments:\n\n4. **Different viscosity level**:\nFollowing your suggestion, we include additional synthetic data experiments which have different viscosity levels. Please refer to the \"Evaluation on different viscosity levels\" in the global response above for results and discussion.\n\n5. **Spatially-varying appearance modeling**: Please see response to 2. above.\n\n6. **Comparison to other existing methods**:\nWe include NeuroFluid and GlobTrans [Franz2021] as additional compared methods. Please refer to the \"Comparison to GlobTrans\" and \"Comparison to NeuroFluid\" in the global response above.\n\n7. **Quantitative results regarding velocity field reconstruction**:\nWe clarify that our goal is to reconstruct plausible fluid fields from real videos, and thus in our experiments in the main paper **we only use real data** (as specified in L201) which does not provide groundtruth 3D fields but only multi-view videos. To evaluate 3D fields, we include experiments on synthetic examples in the \"Evaluation on synthetic data\" in the global response.\n\nFor questions:\n\n1. **How to compute losses**: \nFor L_density, in each training step, we randomly select one timestamp (i.e., one frame) and we sample continuous points in 3D space. For these points, we compute the first-order derivatives by auto-gradient computation provided by PyTorch. For L_project, we do not use sampling as our MGPCG solver requires a regular grid. Thus, we use regular grid of 128^3 to solve for projection. These projected velocity vectors at the regular grid points are then used to supervise the velocity network outputs at those exact regular grid points.\n\n2. **Differences compared to solely using high-frequency position embedding**:\nPlease note that the state-of-the-art neural fluid reconstruction method PINF [Chu2022] uses an advanced high-frequency position embedding [Sitzmann2020] for velocity field. Our comparison to PINF shows that our approach reconstructs better fluid fields with richer vortical details (in particular, please see the re-simulation video in supplementary material).\n\n3. **Comparison to GT velocity**:\nPlease see our response in 7. above.\n\n4. **HyFluid on real scenes**:\nAs clarified above, all our experiments use real data which inevitably has spatially-varying lighting due to global illumination. Please also see our response in 2. above for modeling spatially-varying appearance.\n\n5. **More results of other baselines**:\nPlease refer to our response in 6. above.\n\n**Reference**:\n\n[Chu2022] Chu, M., Liu, L., Zheng, Q., Franz, E., Seidel, H. P., Theobalt, C., & Zayer, R. (2022). Physics informed neural fields for smoke reconstruction with sparse data. ACM Transactions on Graphics (TOG), 41(4), 1-14.\n\n[Sitzmann2020] Sitzmann, V., Martel, J., Bergman, A., Lindell, D., & Wetzstein, G. (2020). Implicit neural representations with periodic activation functions. Advances in neural information processing systems, 33, 7462-7473."}]

Author rebuttal:
We thank all reviewers for their time and feedback. We clarify that since our goal is to reconstruct plausible fluid fields from real videos, **all experiments in our main paper are on real captured data, as specified in L201**. Please find our summary of major changes and response to some common questions below. We will incorporate these changes to our revised paper.

**Summary of major changes per reviewers' suggestions**:

1. [*qUcR*, *3dGa*, *4cV8*, *MRuw*] We add evaluations on synthetic data that provides groundtruth 3D fields.
2. [*qUcR*, *3dGa*, *4cV8*, *MRuw*] We add evaluations on different viscosity levels using synthetic data.
3. [*5YF8*, *MRuw*] We add ablation study on using spatiotemporally-varying appearance. 
4. [*3dGa*, *4cV8*, *MRuw*] We add discussion and comparison to GlobTrans which is a SOTA non-learning-based fluid reconstruction method.
5. [*qUcR*, *3dGa*, *MRuw*] We add discussion and comparison to NeuroFluid which is a recent fluid dynamics learning method.
6. [*4cV8*] We improve velocity visualization using slice rendering.
7. [*MRuw*] We add more references and discussion to recent related work.


**[*qUcR*, *3dGa*, *4cV8*, *MRuw*] Evaluation on synthetic data**:

We include synthetic examples for evaluating 3D density and velocity fields. We use ScalarFlow synthetic dataset generation code [Eckert2019]. We generate five examples with different inflow source (randomized inflow area and density distribution) with higher viscosity and another five examples with lower viscosity. Since numerical viscosity is unavoidable, we simply use different simulation domain resolution to synthesize fluids with different viscosity levels. For the low viscosity group, we use 100x178x100. For the high viscosity group, we use 80x142x80.

We compare to the state-of-the-art neural fluid reconstruction method PINF [Chu2022] which has shown competitive results in 3D fluid fields reconstruction. Since the simulation groundtruth are up to a scale, we use scale-invariant RMSE to measure the performance. We only compute metrics where groundtruth density is greater than $0.1$ to rule out empty space (which is otherwise dominant) for clearer quantitative comparison. In particular, we consider volumetric density error (by querying density networks at the simulation grid points) to evaluate density prediction, and warp error (i.e., using velocity to advect density and comparing to GT density) to evaluate both density and velocity prediction. We also report novel view re-simulation results. From the table below, we see that ours outperforms PINF on both 3D fields and 2D rendering, similar to our main paper's observations. We show qualitative examples in Figure R1 and R2 in the PDF.

We show quantitative results in Table 1 below. We observe that ours outperforms PINF in all metrics. This is consistent with our observations on real data in our main paper.

Table 1: Evaluation on **synthetic** data.
|| Density error$\downarrow$| Warp error$\downarrow$ |PSNR$\uparrow$|SSIM$\uparrow$|LPIPS$\downarrow$|
|-|-|-|-|-|-|
| PINF | 4.95   | 4.88| 25.34 | 0.8641 | 0.1845
| Ours | **2.89**|  **3.29** |**27.93** |**0.8643**|**0.1259**|


**[*qUcR*, *3dGa*, *4cV8*, *MRuw*] Evaluation on different viscosity levels**

In addition to overall evaluation above, we also include separate evalution on the different viscosity levels. We show the results in Table R1 in the PDF. We see that ours outperforms the SOTA method PINF on all metrics except for high-viscosity SSIM, likely due to that the high-viscosity velocity fields are dominated by laminar flows which PINF tends to recover.


**[*qUcR*, *3dGa*, *MRuw*] Comparison to NeuroFluid**:

NeuroFluid [Guan2022] is a recent method on learning fluid dynamics. We use NeuroFluid official code and their released pretrained transition model, and train it on ScalarFlow real dataset following their instructions. NeuroFluid assumes known initial state while in real data there is no groundtruth initial state; thus, we use the SOTA fluid reconstruction method PINF [Chu2022] to reconstruct the first frame as the initial state. We include a comparison to NeuroFluid in Table R2 and Figure R3 in the PDF.

From the results we observe that NeuroFluid does not produce meaningful novel view synthesis, since it does not target at real fluid reconstruction.


**[*3dGa*, *4cV8*, *MRuw*] Comparison to GlobTrans**:

GlobTrans [Franz2021] is a global grid optimization-based method specifically designed for fluid re-simulation and reconstruction. We use the official code release and show an comparison on Table R2 and Figure R4 in the PDF.

From the results we observe that ours allows much better reconstruction fidelity reflected by higher PSNR and SSIM, yet GlobTrans yields better LPIPS. We note that GlobTrans assumes known lighting and uses a more sophisticated shading model. In contrast, ours does not assume known lighting and thus can be more general.


**[*5YF8*, *MRuw*] Spatiotemporally varying appearance**:

We add an ablation on predicting spatiotemporally-varying color to account for spatially-varying lighting. We show the novel view video synthesis (which mainly evaluates appearances) results in Table R3 and Figure R5 in the PDF.

We observe that this does not lead to significant differences on the ScalarFlow real dataset. This may be due to that the fluid is homogeneous in material and that the capturing environment is controlled. For more complex scenes with complex lighting, using spatially-varying color may help further.


**Reference**:

[Chu2022] Physics informed neural fields for smoke reconstruction with sparse data. TOG2022

[Eckert2019] ScalarFlow: a large-scale volumetric data set of real-world scalar transport flows for computer animation and machine learning. TOG2019

[Guan2022] Neurofluid: Fluid dynamics grounding with particle-driven neural radiance fields. ICML2022

[Franz2021] Global transport for fluid reconstruction with learned self-supervision. CVPR2021

Meta-review: At the conclusion of the discussion period the reviewers gave this paper the following scores: 6,5,5,7,5, i.e. they all recommended acceptance.  The work compares visually with the PINF method of Chu et al., (2022) in Figure 5 and compares quantitatively to it in Table 1 as well as with the NeRFlow (Du et al., 2021) results.  However, after the discussion period it was noted that an Arxiv paper from the 10th April, 2023 entitled “Inferring Fluid Dynamics via Inverse Rendering” (Liu et al. 2023) [1], also submitted to ICLR 2023 and available on OpenReview has a number of similarities. To be specific, the overall framework of the method in this submission is quite similar to Liu et al. (2023), i.e. the integration of a fluid simulator with a differentiable renderer to construct a fully-differentiable system. Both use the learned/ optimized velocity to advect fluid density and both use a NeRF-like approach to render frame by taking the advected density field as input. It was also noted that an important contribution and element of novelty in this submission consists of inferring fluid fields from videos which was also a goal and contribution of Liu et al. (2023); however, other prior work has also examined the ScalarFlow smoke video dataset of Eckert et al., (2019), which mitigates this issue. However, some methodological details are similar or the same, such as the fact that both use an Eulerian simulator to alleviate visual ambiguities and divergence-free loss to keep fluid volume, which is likely discussed for the first time in Liu et al., (2023).  This point should really be mentioned in the submission.  Some experimental settings are also the same as in Liu et al. (2023). For example, the “scene composition” experiments in the submission is extremely similar to the “scene editing” experiments in Liu et al. (2023). This setup is used to prove that the general approach has a wide range of application scenarios. The contributions of Liu et al., (2023) relative to the contributions provided by this manuscript need to clarified. If this paper is modified to appropriately cite Liu et al., (2023) and place the contributions of the submission in appropriate context, then this paper is suitable for acceptance. 

[1] Liu J, et al. Inferring Fluid Dynamics via Inverse Rendering. arXiv preprint arXiv:2304.04446, 2023.
Decision: Accept (poster)