Title: Inferring Hybrid Neural Fluid Fields from Videos
Venue: NeurIPS 2023
Accepted: yes
Keywords: neural scene representations

fluid dynamics

flow reconstruction

physics-based learning
Abstract: We study recovering fluid density and velocity from sparse multiview videos. Existing neural dynamic reconstruction methods predominantly rely on optical flows; therefore, they cannot accurately estimate the density and uncover the underlying velocity due to the inherent visual ambiguities of fluid velocity, as fluids are often shapeless and lack stable visual features. The challenge is further pronounced by the turbulent nature of fluid flows, which calls for properly designed fluid velocity representations. To address these challenges, we propose hybrid neural fluid fields (HyFluid), a neural approach to jointly infer fluid density and velocity fields. Specifically, to deal with visual ambiguities of fluid velocity, we introduce a set of physics-based losses that enforce inferring a physically plausible velocity field, which is divergence-free and drives the transport of density. To deal with the turbulent nature of fluid velocity, we design a hybrid neural velocity representation that includes a base neural velocity field that captures most irrotational energy and a vortex particle-based velocity that models residual turbulent velocity. We show that our method enables recovering vortical flow details. Our approach opens up possibilities for various learning and reconstruction applications centered around 3D incompressible flow, including fluid re-simulation and editing, future prediction, and neural dynamic scene composition. Project website: https://kovenyu.com/HyFluid/

Inferring Hybrid Neural Fluid Fields from Videos

Abstract

We study recovering fluid density and velocity from sparse multiview videos. Existing neural dynamic reconstruction methods predominantly rely on optical flows; therefore, they cannot accurately estimate the density and uncover the underlying velocity due to the inherent visual ambiguities of fluid velocity, as fluids are often shapeless and lack stable visual features. The challenge is further pronounced by the turbulent nature of fluid flows, which calls for properly designed fluid velocity representations. To address these challenges, we propose hybrid neural fluid fields (HyFluid), a neural approach to jointly infer fluid density and velocity fields. Specifically, to deal with visual ambiguities of fluid velocity, we introduce a set of physics-based losses that enforce inferring a physically plausible velocity field, which is divergence-free and drives the transport of density. To deal with the turbulent nature of fluid velocity, we design a hybrid neural velocity representation that includes a base neural velocity field that captures most irrotational energy and a vortex particle-based velocity that models residual turbulent velocity. We show that our method enables recovering vortical flow details. Our approach opens up possibilities for various learning and reconstruction applications centered around 3D incompressible flow, including fluid re-simulation and editing, future prediction, and neural dynamic scene composition. Project website: https://kovenyu.com/HyFluid/ .

1 Introduction

Fluid is ubiquitous in our surroundings, from a small breeze in the morning to large-scale atmosphere flow that would affect the weather in the following week. Understanding and predicting fluid dynamics play a central role in climate forecasting (Bauer et al., 2015), vehicle design (Bushnell & Moore, 1991), visual special effect (Pfaff et al., 2010), etc. Yet, it remains an open problem in scientific machine learning to accurately recover fluid flows from visual observations. Flow motions can only be seen indirectly, and, unlike solids that have certain shapes and simple constrained motion patterns, fluid systems have intricate and complex dynamics that exhibit different features across spatial scales. These characteristics pose unique challenges in the representations and algorithms to recover physically correct velocity. Recently, physics-informed neural networks (Raissi et al., 2019) have shown promise in recovering velocity fields from density fields for scientific computing. These neural methods are scalable and flexible compared to traditional grid-based methods, yet they require accurate density fields that are unavailable in uncontrolled ordinary scenes.

In this work, we focus on recovering the density † and velocity of fluids from sparsely captured multiview videos (see Fig. 1 for illustration). We identify three key challenges. Firstly, the fluid velocity is ambiguous from visual observations. Unlike solid objects characterized by consistent

Sparse-view videos

shapes and stable visual features, fluids are inherently shapeless, often monochromatic, and semitransparent. This lack of physical definition makes visual tracking an implausible task, contributing to pronounced visual ambiguity, particularly in regions of laminar flow where the fluid appearance hardly changes. This challenge is beyond the capacity of general-purpose neural dynamic reconstruction methods (Li et al., 2021; Du et al., 2021), as they predominantly depend on optical flow which is ineffective in this context.

Secondly, accurately depicting fluid velocity necessitates appropriate representations that respect its turbulent features. Fluid flows exhibit varying features across multiple spatial scales (Frisch & Donnelly, 1996), a phenomenon that is challenging to capture within a single neural representation due to its spectral bias (Rahaman et al., 2019). At the human scale, fluid motions express not only laminar flows but also turbulence (Pope, 2000). This results in a complicated entanglement of rotational, shearing, and smooth motions. The turbulent flows are high-frequency in both space and time. Therefore, the representation of such flow should be able to accommodate a broad spectrum of signal frequencies while also preserving the unique structure of rotational flows.

Lastly, it is inherently ambiguous to reconstruct physically plausible 3D density fields from sparse observations. Recovering fluid density and appearance from limited 2D videos is ill-posed primarily due to the intricate non-linear absorption and scattering processes associated with semi-transparent fluids (Max, 1995). Consequently, the reconstruction of under-constrained continuous density fields from sparse observations is an intrinsically difficult task.

To address these challenges, we propose hybrid neural fluid fields (HyFluid), a neural approach to jointly infer fluid density and velocity from sparse-view videos. At the core of HyFluid are two key technical contributions including a set of simple yet effective physics-based losses and a hybrid neural velocity representation. Our physics-based losses leverage physical constraints from the Navier-Stokes equations to jointly learn physically plausible fluid density and velocity fields. The physics-based losses enforce inferring a divergence-free velocity field that drives the transport of the density field. In addition to the new losses, we propose a hybrid fluid velocity representation as it is difficult for a single neural representation to capture turbulent fluid velocity. The main idea is to decompose our fluid velocity field into a base and a residual turbulent velocity field. The base velocity field is represented by a neural field that captures most irrotational energy, and the residual velocity field is represented by vortex particles that feature highly rotational and shearing velocities. In order to address the 2D-to-3D density ambiguity, we leverage both visual imaging signals by differentiable volume rendering and the physical fluid transport constraint to regularize the density field.

We demonstrate that HyFluid can yield high-fidelity recovery of density and velocity, allowing novel view video synthesis, re-simulation, editing, future prediction, and neural dynamic scene composition (Fig. 1). In summary, our contributions are three-folded: (1) We propose hybrid neural fluid fields (HyFluid), a neural approach to infer fluid density and velocity from sparse multiview videos. (2) We show that using simple physics-based losses and a hybrid neural velocity representation allows uncovering turbulent fluid motion from sparse observations. (3) We evaluate our method on novel-

view re-simulation and novel-view future prediction to benchmark joint reconstruction for real fluids, and we show that our approach delivers high-fidelity synthesis and reconstructions in comparison to state-of-the-art neural methods.

2 Preliminaries

Incompressible Navier-Stokes equations. Our motion representation and learning signals are motivated by incompressible Navier-Stokes equations. With a low flow speed, fluid motion is considered incompressible and can be well described by:

Here, Eqn. 1 is known as the momentum equation, where the LHS is the material derivative for velocity, D u Dt = ∂ u ∂t + u · ∇ u , which represents the time rate of change of velocity u of a fluid parcel, and the RHS is the momentum induced by pressure gradient -1 ρ ∇ p (where ρ denotes the physical density of the fluid parcel), viscosity ν ∇· ∇ u , and some external force f . We consider inviscid fluid and thus drop the viscosity term. Eqn. 2 is known as the mass conservation equation, meaning that the mass flowing into a controlled volume should be equal to the mass flowing out of the volume.

Differentiable volume rendering. We integrate volume rendering for joint learning of visual appearance, density, and velocity from videos. Volume rendering is a suitable model for translucent materials and participating media like smoke and fog (Kajiya & Von Herzen, 1984). In volume rendering, the radiance L ( o , d ) arriving at the camera location o from direction d (a.k.a. the radiance of a camera ray r ( t ) = o -t d ) with near and far bounds t n and t f is given by

where T ( t ) = exp( -∫ t t n σ ( r ( s )) ds ) denotes the transmittance along the ray from t n to t , L e denotes the emitting radiance, and σ denotes the optical density which can be considered proportional to the concentration density of fluid substance according to Beer-Lambert law. Following recent neural rendering methods (Mildenhall et al., 2020), we use quadrature to discretize it (Max, 1995).

3 Hybrid Neural Fluid Fields

Our goal is to recover the fluid appearance, density, and velocity from sparse multiview RGB videos. To this end, we propose hybrid neural fluid fields (HyFluid). At the core of HyFluid are a set of simple physics-based losses and a hybrid neural velocity representation. In particular, HyFluid aims to infer a neural density field σ ( x, y, z, t ) , the appearance of the fluid L e, and the underlying velocity u ( x, y, z, t ) = u base ( x, y, z, t ) + u vort ( x, y, z, t ) , where we decompose it into a base neural velocity field u base and a residual vortex particle-driven turbulent velocity. u = [ u, v, w ] denotes the velocity along x, y, z axis, respectively. In the following, we first introduce the physics-based losses, then we describe our hybrid velocity representation, and finally, we summarize our learning signals for the joint inference. We show a conceptual illustration in Fig. 2.

Physics-based losses. We propose physics-based losses L physics including a density loss, a projection loss, and a laminar regularization loss.

Density loss. While the neural density field can be learned through differentiable volume rendering, the underlying velocity field is hidden from the visual observations. Unlike solids that have regular shapes and invariant visual features to track, fluids are often shapeless, monochromatic, and semitransparent. This makes it implausible to visually track it. To uncover the hidden velocity from visual observations, we resort to the density transport equation in incompressible flows, Dσ Dt = 0 , and introduce a physics-informed supervision signal:

Vortex particles

𝒖

!"#$

(𝑥, 𝑦, 𝑧, 𝑡)

Base flow

𝒖

%&'(

(𝑥, 𝑦, 𝑧, 𝑡)

Density field

𝑑(𝑥, 𝑦, 𝑧, 𝑡)

…

…

Render

…

…

ℒ

!"#$"!

Observed video

Velocity field

𝒖(𝑥, 𝑦, 𝑧, 𝑡)

ℒ

%&'()*(

↑

+

This loss says that the 3D velocity field ( u, v, w ) at any given moment t should transport the density field σ such that it evolves to the density field at the next moment, in a similar spirit to scene flow (Vedula et al., 1999).

Projection loss. Another important physical property of incompressible flow is mass conservation depicted in Eqn. 2, that is, the velocity field should be divergence-free. A straightforward way to enforce divergence-free condition is to impose a loss on the divergence of velocity, ∂u ∂x + ∂v ∂y + ∂w ∂z . However, we find that this empirically leads to a degenerated solution, where only the laminar flows (i.e., smooth, direct-current flows) can be uncovered as this is a trivial local minimum for every point. Yet, the incompressibility of fluid flows is confined through a global dynamic system, modulated by the pressure of the fluid.

Therefore, we need a globally-aware, physically-plausible supervision that does not conform to trivial local minima. Motivated by the commonly used pressure projection solver in computational fluid dynamics, we propose a projection loss that constrains our learned velocity to be divergence-free:

where u p = project ( u ) are the pressure-projected velocity field, using a pressure project solver. A project solver implements a Helmholtz decomposition by solving a global linear system for pressure, and it projects the velocity field u to a divergence-free manifold using the gradient of the solved pressure. Also, we notice that any divergence-free velocity field u incurs zero projection loss as it is an identity mapping for divergence-free fields. Thus, this projection loss essentially constrains the uncovered velocity within a divergence-free subspace, and it is not subject to trivial local minima of pure direct-current flows.

Laminar regularization loss. These density and projection losses do not warrant physically-correct velocity reconstruction for laminar flows. In laminar flows, all partial derivatives of density are zero, so any velocity, e.g., all-zero velocity, incurs zero L density. As for L project, all-zero velocity fields also incur a zero loss. Hence, we introduce a laminar regularization loss:

which is a hinge loss that encourages high-density regions to have non-zero velocity, and γ denotes a hyperparameter to scale the threshold according to the velocity magnitude unit. These three losses are complementary and interconnected to uncover a physically plausible fluid flow.

Hybrid neural velocity representation. At the human scale, fluid flows are often turbulent. They exhibit complex dynamic features including circular and shearing flows. In addition to the common challenges of reconstructing non-turbulent flows, turbulent flows are even more intricate and difficult to model and capture. They are high-frequency flows in both space and time. This demands that the velocity representation be able to accommodate a wide range of signal frequencies and the particular

structure of rotational flows. However, neural representations are known to have certain spectral inductive biases toward representing low-frequency signals (Rahaman et al., 2019). While this can be alleviated by position embedding (Tancik et al., 2020) or sinusoidal activation functions (Sitzmann et al., 2020), they still lack a proper structure to capture and represent highly detailed turbulent flows.

We propose a hybrid neural velocity representation to tackle this challenge. The main idea is to decompose the underlying flow field u = u base + u vort into a base neural velocity field u base and a residual vorticity-driven velocity u vort. The base neural velocity field captures the large-scale flow motion, and the vorticity-driven velocity is a dedicated representation of the small-scale vortex details that are hard to characterize by the base field. For the base velocity field u base, we extend instance neural graphics primitive (iNGP) (M¨ uller et al., 2022) representation to the temporal domain.

The residual vorticity-driven flow u vort is complementary to the base flow by leveraging the physical structure of turbulent fluid flows. The physical model behind the vorticity-driven flow is prescribed by the curl form of the Navier-Stokes equations (see Cottet et al. (2000) for more details):

where ω = ∇× u denotes the vorticity. We assume inviscid fluid and conservative body force, and thus only the vortex stretching term ω · ∇ u is non-zero on the RHS. In this case, Eqn. 7 becomes the vorticity transport equation, saying that vorticity is stretched and advected by the velocity flow.

This physical model has been widely used in fluid simulation (Cottet et al., 2000), known as vortex methods. In particular, we consider the vortex particle methods (Selle et al., 2005), where we use a particle-based representation for vorticity, which is low-dimensional, is easy to temporally evolve, and well embeds the circular physical flow structures in it. We represent a vortex particle p at some time stamp t by a triplet { I p , x t p , ω t p } of its intensity I p , position x t p , and vorticity ω t p . Our vorticity-driven flow u vort ( x , t ) induced by vortex particles is represented by:

where K ( x ) = exp( -∥ x ∥ 2 / 2 r 2 ) / ( r 3 (2 π ) 3 / 2 ) is a Gaussian distribution kernel. Intuitively, every vortex particle 'carries' a local circular momentum, which is itself transported with the flow prescribed by Eqn. 7. Collectively, a set of vortex particles allows learning complex flow details which are hard to capture by the base velocity field.

However, naively learning the vortex particles leads to poor local minima due to the complex interdependence of u and { I p , x t p , ω t p } . Thus, we make two simplifications. First, we assume that most energy is captured by the base flow and only use the base flow to transport vortex particles. This is achieved by first learning the base flow and then freezing the base flow to learn the residual flow. Second, we introduce a seeding strategy to disentangle learnable parameters from the particle transport: we seed overly many vortex particles on high-curl spatio-temporal locations and precompute the trajectory { x t p } and { ω t p } for all t and p by the learned base flow. Therefore, the learnable parameters are reduced to { I p } . The redundant vortex particles are automatically suppressed by learning to have zero intensities. We leave more details of the seeding strategy in the Appendix.

Joint inference of fluid fields. It is a highly ill-posed problem to recover density and appearance from sparse visual observations, as the imaging process for semi-transparent fluids involves complex non-linear absorption and scattering. The visual appearance of fluids depends not only on density but also lighting and fluid substance properties. Therefore, recovering turbulent, under-constrained continuous density fields from limited observations is inherently challenging.

To address this challenge, we leverage both visual imaging signals and physical fluid transport constraints by jointly learning density, appearance, and velocity. The visual signal supervision through differentiable volume rendering is given by:

where L render is our volume rendered values by Eqn. 3 and L observe is sampled from video frames. Notice that we sample rays continuously in the viewing frustums instead of sampling through only pixel centers as NeRF-like methods do (Mildenhall et al., 2020), as we aim to learn continuous fields

and we rely on good partial derivatives from the fields. The physical transport constraint is provided by Eqn. 4, i.e., we supervise the partial derivatives of the density field. We compute all derivatives by auto-differentiation (Paszke et al., 2017). Our loss function is given by:

where all the β s are loss weights.

Another inherent ambiguity is between emitting radiance and density. As in Eqn. 3, the emitting radiance is multiplied by density, which means that an unconstrained spatially-varying emitting radiance field admits all sorts of trivial solutions for density. Empirically, we find little differences between using learned spatially-varying color and using a constant color. Thus, we set emitting radiance to be constant to remove this ambiguity.

4 Related Work

Fluid reconstruction. Fluid flow reconstruction from visible light measurement has been widely studied in science and engineering. Well-established methods used in controlled lab environments include active sensing techniques such as laser scanners (Hawkins et al., 2005), light path (Ji et al., 2013), and structural light (Gu et al., 2012), as well as passive marker-based methods such as particle imaging velocimetry (PIV) (Adrian & Westerweel, 2011; Elsinga et al., 2006) which injects passive particles into the fluid flows. These methods allow accurate flow measurement, yet they require specialized setup for markers, lighting, and capturing devices.

Recent methods also seek to reconstruct fluid flows from casual visible light measurements without a specialized setup. Earlier works use tomography and linear imaging formation to reconstruct density grids from visual observations (Gregson et al., 2014; Okabe et al., 2015) and then estimate velocity using physical priors (Gregson et al., 2014; Eckert et al., 2018). This line of work has been extended to joint optimization for both velocity and density (Eckert et al., 2019) to improve reconstruction quality. Since this is a highly unconstrained problem, synthesized view supervision is proposed to regularize the reconstruction (Zang et al., 2020). Franz et al. (2021) introduce a density-based rendering formation with a global differentiable simulation framework. They use manually calibrate lighting directions and fluid source location to allow consistent differentiable simulation optimization. However, these methods ignore the visual appearance, suffering from the inherent visual ambiguity of fluid density and appearance.

Neural dynamic scene representations. Using neural networks as implicit visual scene representations have been made effective and popular. Earlier works use neural representations for geometry (Park et al., 2019; Mescheder et al., 2019) and visual appearance (Sitzmann et al., 2019). As a seminal work, neural radiance fields (NeRFs) (Mildenhall et al., 2020) incorporate volume rendering to learn implicit geometry and appearance from images. Follow-ups of NeRFs extend applications from novel view synthesis to dynamic scene reconstruction (Li et al., 2021; Pumarola et al., 2021), relighting (Zhang et al., 2021; Yu et al., 2023), scene segmentation (Yu et al., 2022; Sajjadi et al., 2022), robot manipulation (Le Cleac'h et al., 2023; Tian et al., 2023), system identification (Li et al., 2023), etc.

Generic dynamic NeRFs incorporate deformation fields (Park et al., 2021b; Pumarola et al., 2021; Park et al., 2021a) or scene flow fields (Li et al., 2021; Du et al., 2021; Xian et al., 2021; Li et al., 2022) to represent motion. These motion representations essentially rely on stable visual features such as color gradients and edges, and thus they are not suitable for uncovering fluid flows which generally do not exhibit stable visual features. Notably, (Chu et al., 2022) is the most relevant work to ours. Chu et al. (2022) propose the physics-informed neural fields (PINF) that incorporate physics-informed losses (Raissi et al., 2019) to reconstruct fluid flows from sparse-view videos with learned priors from synthetic data. However, PINF does not consider the inherent physical structures of real fluid flows and thus only reconstructs laminar flows. Our HyFluid allows uncovering turbulent real flows by novel physics-based losses and the hybrid neural velocity representation.

Learning fluid dynamics. Learning fluid dynamics holds the promise to accelerate simulation and aid fluid reconstruction. Earlier work on learning fluid dynamics to accelerate simulation includes using convolutional networks to evolve fluid states (Tompson et al., 2017; Ummenhofer et al., 2019; Prantl et al., 2022) or learning latent simulation (Wiewel et al., 2019; Kim et al., 2019), supervised by 3D

HyFluid (Ours)

GT

PINF

NeRFlow

NeuroFluid

PINF

Ground truth

NeRFlow

HyFluid (Ours)

GlobTrans

fluid simulation data. While these methods learn from 3D fluid data, a few recent work such as Guan et al. (2022) and Liu et al. (2023) learn simulators from multi-view videos. However, both of them assume to have groundtruth reconstructed initial fluid states, restricting the applications to synthetic data. Thus, it still remains open how existing methods may help fluid reconstruction. An exception is Deng et al. (2023) that use low-dimensional vortex particles to represent fluid velocity fields and thus allow reconstructing fluid velocity from a single video, yet their formulation is restricted to 2D domain due to complex vortex stretching in 3D vortex dynamics.

5 Experiments

Datasets. We use both real captures and synthetic simulation for evaluation. For real captures, we use the ScalarFlow dataset (Eckert et al., 2019) which consists of videos of buoyancy-driven rising smoke plumes. We use the first five scenes from the real captures. For each scene, there are five video recordings. The five cameras are fixed-position throughout capture and distributed evenly across a 120 ◦ arc centered at the rising smoke. Each video has 150 frames with 1062 × 600 resolution. These videos have been post-processed to remove backgrounds. We follow Chu et al. (2022) to use 120 frames for each video. For each scene, we use four videos for training, and one held-out video for testing (i.e., as the groundtruth for the novel view).

Synthetic simulation allows evaluating 3D velocity and density against the groundtruth. We use ScalarFlow synthetic dataset generation code (Eckert et al., 2019). We generate five examples with different inflow source with higher viscosity and another five examples with lower viscosity.

Evaluation metrics. For real fluid flows we do not have volumetric 3D groundtruth, so we use novel view video rendering to evaluate the reconstruction quality. In particular, we use the following three tasks: novel view video synthesis, novel view re-simulation, and novel view future prediction. Re-simulation and future prediction require high-fidelity velocity reconstruction and thus provide a means to evaluate velocity reconstruction. We use peak signal-noise ratio (PSNR), structural similarity index measure (SSIM), and the perceptual metric LPIPS (Zhang et al., 2018).

For synthetic data, we can evaluate the reconstruction results against the simulation groundtruth. Since the simulation groundtruth are up to a scale, we use scale-invariant RMSE to measure the performance. We only compute metrics where groundtruth density is greater than 0 . 1 to rule out empty space (which is otherwise dominant) for clearer quantitative comparison. In particular, we consider volumetric density error (by querying density networks at the simulation grid points) to

PINF

HyFluid (Ours)

Ground truth

t=0

t=7

t=14

t=21

t=0

t=7

t=14

t=21

t=0

t=7

t=14

t=21

口

evaluate density prediction, and warp error (i.e., using velocity to advect density and comparing to GT density) to evaluate both density and velocity prediction.

Baselines. We consider three recent flow reconstruction methods: NeRFlow (Du et al., 2021), a neural dynamic reconstruction method for fluid reconstruction; GlobTran (Franz et al., 2021), a grid-based fluid reconstruction method; and PINF (Chu et al., 2022), the latest neural fluid reconstruction method. We also compare to NeuroFluid (Guan et al., 2022), a recent fluid dynamics learning approach. Since NeuroFluid requires having 3D density and velocity at the first frame, we use PINF reconstructed fields for it. We leave training details and compute resources in Appendix.

5.1 Novel view video synthesis

In this task, we aim to evaluate the inferred density fields by synthesizing novel view videos. We use all 120 frames from the held-out test view for all scenes and report the average numbers. As shown in Table 1, our method is comparable to the state-of-the-art method PINF (Chu et al., 2022). Notice that ours is better in PSNR which is an objective metric that measures the optical similarity of the synthesized image and the groundtruth. GlobTrans (Franz et al., 2021) uses adversarial loss together with a specific rendering formulation, which may contribute to better perceptual LPIPS performances. Yet, this compromises objective fidelity as reflected by lower PSNR. Qualitative results in Fig. 3 showcase the capability of our method to accurately reconstruct density fields, which paves the way for the inference of high-fidelity physically-plausible velocity fields.

5.2 Re-simulation

The re-simulation task accesses the ability of the models to infer the velocity fields accurately. Specifically, we use the reconstructed density from the first frame and advect the density across time to the last frame using the learned velocity. Weuse the standard MacCormack method (Selle et al., 2008) for advection. As for fluid source, we use the reconstructed density field in the bottom of the scenes. Results in Table 1 and Fig. 4 show that our method generates reasonable results, whereas PINF (Chu et al., 2022) and NeRFlow (Du et al., 2021) completely fail.

HyFluid (Ours)

PINF

Reference

HyFluid (Ours)

GT

PINF

We further visualize the velocity field by slicing it in Fig. 6, which demonstrates that our method effectively captures the velocity details that are missed by the baseline methods, which tend to default to trivial solutions where all the velocity is upward-directed.

5.3 Future prediction

In the future prediction task, we extrapolate the fluid motion into the future based on the inferred velocity fields at a single frame. In particular, we follow the standard grid-based fluid simulation (Fedkiw et al., 2001) to evolve the velocity fields by self-advecting them using the MacCormack method, followed by a projection step to ensure the divergence-free constraint. Table 1 shows that our method predicts future frames that are quantitatively better than PINF. Fig. 5 illustrates that compared to PINF, the future fluid state predicted by our method maintains its original structure and continues to follow a natural upward trajectory, and also produces natural details. These results underline the robustness of our method in inferring complex fluid velocity.

5.4 Inference on in-the-wild real capture

While ScalarFlow dataset provides real captures of plumes from a controlled environment, we are also interested in inferring fluid fields from uncontrolled in-the-wild videos. We use three cameras to capture sparse multi-view videos of a lit incense, and we show a visualization of the inferred density (top row) and velocity (bottom row) in Fig. 7. While the fluid in this example is much thinner than plumes, HyFluid allows plausible inference of fluid fields.

5.5 Evaluation on synthetic data

We include synthetic examples for evaluating 3D density and velocity fields against the groundtruth. We compare to PINF which has shown state-of-the-art results in 3D fluid fields reconstruction. We show 3D fields evaluation results together with novel view re-simulation results in Table 2 and Fig. 8. We observe that ours outperforms PINF on both 3D fields and 2D rendering, especially in preserving the vortical structures and intricate shape details. This is validated by larger performance margins in low-viscosity examples, since there is more turbulence that is hard to capture by PINF.

5.6 Additional applications

The inherent properties of neural representation in our method allow for its applicability in a variety of additional contexts, such as editing and scene composition. As seen in Fig. 1, the results obtained

Reconstructed

Edited

Naive

+ℒ

+ℒ

project

Full

GT

from our method can be effortlessly integrated into dynamic NeRF scenes (Sara Fridovich-Keil and Giacomo Meanti et al., 2023) or imported into professional-grade graphics software.

Our hybrid neural velocity representation naturally supports turbulence editing in re-simulation. In particular, we multiply our vortex particle intensity by a factor of 4 and showcase a visualization of edited re-simulation in Fig. 9. This shows that the high-quality velocity estimation from our hybrid neural velocity representation allows easily synthesizing vortical flow details.

5.7 Ablation study

We show qualitative results in Fig. 10 to evaluate the proposed physics-based losses, the vortex particle-driven velocity representation, and the joint training supervision. Starting from 'Naive' where we only use the rendering loss for recovering density and the density loss for recovering velocity, we gradually add back laminar loss (' + L laminar)'), projection loss (' + L project'), and vortex particle-driven residual velocity ('Full'). From Fig. 10 we observe that our physical losses allow better velocity recovery. In particular, the projection loss enables recovering physically correct velocity that reconstructs the plume shape in re-simulation. Vortex particle-driven velocity allows capturing more details and thus more consistent re-simulation.

6 Conclusion

In this work, we study recovering fluid density and velocity from sparse multi-view videos. We propose hybrid neural fluid fields (HyFluid) which features physics-based losses to address inherent visual ambiguities and a hybrid neural velocity representation for capturing the complex turbulent fluid flow. We show that our simple designs can already lead to physically plausible estimation of fluid fields that supports applications such as re-simulation, editing, and dynamic scene composition.

Limitations. Our method assumes inviscid fluid and only considers gas but not liquid. Liquid features free surface that may require further physical modeling and constraints.

Acknowledgments. This work was in part supported by the Toyota Research Institute (TRI), NSF RI #2211258, ONR MURI N00014-22-1-2740, the Stanford Institute for Human-Centered AI (HAI), and Google. B. Zhu acknowledges NSF IIS #2313075, IIS #2106733, and CAREER #2144806.

References

Ronald J Adrian and Jerry Westerweel. Particle image velocimetry . Cambridge university press, 2011. 6

Steven F Ashby and Robert D Falgout. A parallel multigrid preconditioned conjugate gradient algorithm for groundwater flow simulations. Nuclear science and engineering , 1996. 14

Peter Bauer, Alan Thorpe, and Gilbert Brunet. The quiet revolution of numerical weather prediction. Nature , 2015. 1

Dennis M Bushnell and KJ Moore. Drag reduction in nature. Annual review of fluid mechanics , 1991. 1

Mengyu Chu, Lingjie Liu, Quan Zheng, Erik Franz, Hans-Peter Seidel, Christian Theobalt, and Rhaleb Zayer. Physics informed neural fields for smoke reconstruction with sparse data. ACM TOG , 2022. 6, 7, 8

Georges-Henri Cottet, Petros D Koumoutsakos, et al. Vortex methods: theory and practice , volume 8. Cambridge university press, 2000. 5

Yitong Deng, Hong-Xing Yu, Jiajun Wu, and Bo Zhu. Learning vortex dynamics for fluid inference and prediction. In ICLR , 2023. 7

Yilun Du, Yinan Zhang, Hong-Xing Yu, Joshua B Tenenbaum, and Jiajun Wu. Neural radiance flow for 4d view synthesis and video processing. In ICCV , 2021. 2, 6, 8

M-L Eckert, Wolfgang Heidrich, and Nils Thuerey. Coupled fluid density and motion from single views. CGF , 2018. 6

Marie-Lena Eckert, Kiwon Um, and Nils Thuerey. Scalarflow: a large-scale volumetric data set of real-world scalar transport flows for computer animation and machine learning. ACM TOG , 2019. 6, 7

Gerrit E Elsinga, Fulvio Scarano, Bernhard Wieneke, and Bas W van Oudheusden. Tomographic particle image velocimetry. Experiments in fluids , 2006. 6

Ronald Fedkiw, Jos Stam, and Henrik Wann Jensen. Visual simulation of smoke. In SIGGRAPH , 2001. 9, 14

Erik Franz, Barbara Solenthaler, and Nils Thuerey. Global transport for fluid reconstruction with learned self-supervision. In CVPR , 2021. 6, 8

Uriel Frisch and Russell J Donnelly. Turbulence: the legacy of an kolmogorov. Physics Today , 1996. 2

James Gregson, Ivo Ihrke, Nils Thuerey, and Wolfgang Heidrich. From capture to simulation: connecting forward and inverse problems in fluids. ACM TOG , 2014. 6

Jinwei Gu, Shree K Nayar, Eitan Grinspun, Peter N Belhumeur, and Ravi Ramamoorthi. Compressive structured light for recovering inhomogeneous participating media. IEEE TPAMI , 2012. 6

Shanyan Guan, Huayu Deng, Yunbo Wang, and Xiaokang Yang. Neurofluid: Fluid dynamics grounding with particle-driven neural radiance fields. In ICML , 2022. 7, 8

Tim Hawkins, Per Einarsson, and Paul Debevec. Acquisition of time-varying participating media. ACM TOG , 2005. 6

Yuanming Hu, Tzu-Mao Li, Luke Anderson, Jonathan Ragan-Kelley, and Fr´ edo Durand. Taichi: a language for high-performance computation on spatially sparse data structures. ACM TOG , 2019. 14

Yu Ji, Jinwei Ye, and Jingyi Yu. Reconstructing gas flows using light-path approximation. In CVPR , 2013. 6

James T Kajiya and Brian P Von Herzen. Ray tracing volume densities. SIGGRAPH , 1984. 3

Byungsoo Kim, Vinicius C Azevedo, Nils Thuerey, Theodore Kim, Markus Gross, and Barbara Solenthaler. Deep fluids: A generative network for parameterized fluid simulations. In CGF , 2019. 6

Simon Le Cleac'h, Hong-Xing Yu, Michelle Guo, Taylor Howell, Ruohan Gao, Jiajun Wu, Zachary Manchester, and Mac Schwager. Differentiable physics simulation of dynamics-augmented neural objects. IEEE RA-L , 2023. 6

Tianye Li, Mira Slavcheva, Michael Zollhoefer, Simon Green, Christoph Lassner, Changil Kim, Tanner Schmidt, Steven Lovegrove, Michael Goesele, Richard Newcombe, et al. Neural 3d video synthesis from multi-view video. In CVPR , 2022. 6

Xuan Li, Yi-Ling Qiao, Peter Yichen Chen, Krishna Murthy Jatavallabhula, Ming Lin, Chenfanfu Jiang, and Chuang Gan. Pac-nerf: Physics augmented continuum neural radiance fields for geometry-agnostic system identification. In ICLR , 2023. 6

Zhengqi Li, Simon Niklaus, Noah Snavely, and Oliver Wang. Neural scene flow fields for space-time view synthesis of dynamic scenes. In CVPR , 2021. 2, 6

Jinxian Liu, Ye Chen, Bingbing Ni, Jiyao Mao, and Zhenbo Yu. Inferring fluid dynamics via inverse rendering. arXiv:2304.04446 , 2023. 7

Nelson Max. Optical models for direct volume rendering. IEEE TVCG , 1995. 2, 3

Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. Occupancy networks: Learning 3d reconstruction in function space. In CVPR , 2019. 6

Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. arXiv:2003.08934 , 2020. 3, 5, 6

Thomas M¨ uller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with a multiresolution hash encoding. ACM TOG , 2022. 5, 14

Makoto Okabe, Yoshinori Dobashi, Ken Anjyo, and Rikio Onai. Fluid volume modeling from sparse multi-view images by appearance transfer. ACM TOG , 2015. 6

Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In CVPR , 2019. 6

Keunhong Park, Utkarsh Sinha, Jonathan T Barron, Sofien Bouaziz, Dan B Goldman, Steven M Seitz, and Ricardo Martin-Brualla. Nerfies: Deformable neural radiance fields. In CVPR , 2021a. 6

Keunhong Park, Utkarsh Sinha, Peter Hedman, Jonathan T Barron, Sofien Bouaziz, Dan B Goldman, Ricardo Martin-Brualla, and Steven M Seitz. Hypernerf: A higher-dimensional representation for topologically varying neural radiance fields. arXiv:2106.13228 , 2021b. 6

Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. In NeurIPS Workshops , 2017. 6

Tobias Pfaff, Nils Thuerey, Jonathan Cohen, Sarah Tariq, and Markus Gross. Scalable fluid simulation using anisotropic turbulence particles. In SIGGRAPH Asia , 2010. 1

Stephen B Pope. Turbulent flows . Cambridge university press, 2000. 2

Lukas Prantl, Benjamin Ummenhofer, Vladlen Koltun, and Nils Thuerey. Guaranteed conservation of momentum for learning particle-based fluid dynamics. In NeurIPS , 2022. 6

Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francesc Moreno-Noguer. D-nerf: Neural radiance fields for dynamic scenes. In CVPR , 2021. 6

Nasim Rahaman, Aristide Baratin, Devansh Arpit, Felix Draxler, Min Lin, Fred Hamprecht, Yoshua Bengio, and Aaron Courville. On the spectral bias of neural networks. In ICML , 2019. 2, 5

Maziar Raissi, Paris Perdikaris, and George E Karniadakis. Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational physics , 2019. 1, 6

Mehdi SM Sajjadi, Daniel Duckworth, Aravindh Mahendran, Sjoerd van Steenkiste, Filip Pavetic, Mario Lucic, Leonidas J Guibas, Klaus Greff, and Thomas Kipf. Object scene representation transformer. NeurIPS , 2022. 6

Sara Fridovich-Keil and Giacomo Meanti, Frederik Rahbæk Warburg, Benjamin Recht, and Angjoo Kanazawa. K-planes: Explicit radiance fields in space, time, and appearance. In CVPR , 2023. 10

Andrew Selle, Nick Rasmussen, and Ronald Fedkiw. A vortex particle method for smoke, water and explosions. In SIGGRAPH , 2005. 5

Andrew Selle, Ronald Fedkiw, Byungmoon Kim, Yingjie Liu, and Jarek Rossignac. An unconditionally stable maccormack method. Journal of Scientific Computing , 2008. 8

Vincent Sitzmann, Michael Zollh¨ ofer, and Gordon Wetzstein. Scene representation networks: Continuous 3d-structure-aware neural scene representations. In NeurIPS , 2019. 6

Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein. Implicit neural representations with periodic activation functions. NeurIPS , 2020. 5

Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Fridovich-Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. In NeurIPS , 2020. 5

Stephen Tian, Yancheng Cai, Hong-Xing Yu, Sergey Zakharov, Katherine Liu, Adrien Gaidon, Yunzhu Li, and Jiajun Wu. Multi-object manipulation via object-centric neural scattering functions. In CVPR , 2023. 6

Jonathan Tompson, Kristofer Schlachter, Pablo Sprechmann, and Ken Perlin. Accelerating eulerian fluid simulation with convolutional networks. In ICML , 2017. 6

Benjamin Ummenhofer, Lukas Prantl, Nils Thuerey, and Vladlen Koltun. Lagrangian fluid simulation with continuous convolutions. In ICLR , 2019. 6

Sundar Vedula, Simon Baker, Peter Rander, Robert Collins, and Takeo Kanade. Three-dimensional scene flow. In ICCV , 1999. 4

Steffen Wiewel, Moritz Becher, and Nils Thuerey. Latent space physics: Towards learning the temporal evolution of fluid flow. In CGF , 2019. 6

Wenqi Xian, Jia-Bin Huang, Johannes Kopf, and Changil Kim. Space-time neural irradiance fields for freeviewpoint video. In CVPR , 2021. 6

Hong-Xing Yu, Leonidas J Guibas, and Jiajun Wu. Unsupervised discovery of object radiance fields. ICLR , 2022. 6

Hong-Xing Yu, Michelle Guo, Alireza Fathi, Yen-Yu Chang, Eric Ryan Chan, Ruohan Gao, Thomas Funkhouser, and Jiajun Wu. Learning object-centric neural scattering functions for free-viewpoint relighting and scene composition. TMLR , 2023. 6

Guangming Zang, Ramzi Idoughi, Congli Wang, Anthony Bennett, Jianguo Du, Scott Skeen, William L Roberts, Peter Wonka, and Wolfgang Heidrich. Tomofluid: Reconstructing dynamic fluid from sparse view videos. In CVPR , 2020. 6

Kai Zhang, Fujun Luan, Qianqian Wang, Kavita Bala, and Noah Snavely. Physg: Inverse rendering with spherical gaussians for physics-based material editing and relighting. In CVPR , 2021. 6

Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In CVPR , 2018. 7

A Appendix: Implementation details

Model architecture. For both the density field σ ( x, y, z, t ) and the base velocity field u base ( x, y, z, t ) we use a 4D extension of iNGP (M¨ uller et al., 2022). Specifically, we add a temporal dimension to the original static iNGP. For the spatial dimensions, we use a base resolution 16 and the finest resolution 256 . For the temporal dimension we set the finest resolution to 128 which is comparable to the number of video frames ( 120 frames), as higher resolutions create hash encoding features that are never directly supervised and can lead to unstable gradients.

Training and hyperparameters. Our training can be divided into three stages. In the first stage, we pre-train our density network σ ( x, y, z, t ) and the constant learnable appearance (radiance) L e using the rendering loss only, which forms an initial estimate of both quantities. In the second stage, we jointly train the density σ ( x, y, z, t ) , appearance L e , and the base velocity u base ( x, y, z, t ) using the full loss. Finally, we jointly train the density σ ( x, y, z, t ) , appearance L e , and the vortex particles { I p } . We use an Adam optimizer with a learning rate 0 . 01 . In the first stage, we train the density and radiance for 200 , 000 iterations. In the second stage, we jointly train the model for 50 , 000 iterations. In the third stage, we do training for 5 , 000 iterations. We empirically set the loss weights to β render = 10 , 000 , β density = 0 . 001 , β proj = 1 , β laminar = 10 . For the laminar loss, we set the coefficient γ = 0 . 2 .

Projection loss. For the projection routine, we use a multi-grid preconditioned conjugate gradient (MGPCG) solver (Ashby & Falgout, 1996). We implement it using Taichi (Hu et al., 2019) language. We use three levels of multi-grid, with the spatial resolution 128 3 . Note that we use the same resolution for our re-simulation and future prediction experiments for all compared methods.

Seeding strategy. In our experiments, we seed 50 vortex particles for each scene. To place initial vortex particles, we take the insight from vortex confinement methods (Fedkiw et al., 2001). We densely sample continuous spatio-temporal locations and compute curl values from the trained base velocity network, and then we place vortex particles at locations with the highest curl values.

Compute usage. We train our model on a single A100 GPU for around 9 hours in total. The first stage takes half an hour, the second stage takes around 8 hours (this is significantly slower than the first stage as it requires computing derivatives for the density loss), and the third stage takes half an hour.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a novel approach for recovering the density and velocity fields of inviscid fluids from sparse multiview videos. The model has two main contributions:\n- First, it incorporates physics-based losses to enforce the inference of a physically plausible velocity field that is divergence-free and drives the transport of density. This helps to deal with the visual ambiguities of fluid velocity. \n- Second, the model provides a hybrid neural velocity representation, which consists of a base neural velocity field capturing most irrotational energy and a vortex particle-based velocity modeling residual turbulent velocity. This representation enables the recovery of vortical flow details.\n\nSTRENGTHS:\nOriginality: This paper primarily tackles the visual ambiguities in inverse rendering techniques for fluid data, focusing on resolving visual ambiguities. The integration of physics-based losses into the volume rendering framework is a rational approach. The results in Figure 8 illustrate the effectiveness of the newly proposed learning constraints in significantly improving the accuracy of the reconstruction. Moreover, the introduction of a hybrid representation of velocity fields and particle-based vortex flow showcases originality in the methodology.\n\nSignificance: The proposed model makes a significant contribution to the visual understanding of fluids, particularly smoke, fog, and gas.\n\nWEAKNESSES:\nFor methodology: \n\n1. One contribution of this paper is to incorporate new forms of physical constraints in the framework of inverse rendering. However, the general idea is not entirely novel as previous work by Chu et al. (2022) has also presented a similar (albeit different) approach, which somewhat weakens the technical novelty of this paper. For example, using the density transport equation from incompressible fluid or NS equation as a loss function has been employed in other related papers. Please refer to the work from Baieri et al. (2023) and Li et al. (2023). \n\n- [Baieri et al., 2023] Fluid Dynamics Network: Topology-Agnostic 4D Reconstruction via Fluid Dynamics Priors. Arxiv, 2023. \n- [Li et al., 2023] PAC-NeRF: Physics Augmented Continuum Neural Radiance Fields for Geometry-Agnostic System Identification. ICLR, 2023. \n\n2. The constraint imposed on emitting radiance in HyFluid does present limitations in its applicability to real-world scenarios. The requirement of a constant emitting radiance may hinder accurate recovery in situations where spatially-varying lighting exists in the scene.\n\nLacking references:\n\n3. Some closely related work seems to be missing, such as NeuroFluid (Guan et al., 2022) and PAC-NeRF (Li et al., 2023), both of which also focus on visual physical inference through inverse rendering. It is important to acknowledge these works as they contribute to the existing body of literature in this field and provide valuable insights and techniques for comparison and benchmarking purposes.\n\n- [Guan et al., 2022] NeuroFluid: Fluid Dynamics Grounding with Particle-Driven Neural Radiance Fields. ICML, 2022. \n\nFor experiments: \n\n4. Is the proposed method limited to handling only inviscid fluids? It would be beneficial to evaluate the proposed method in a broader range of fluid scenarios, including different types of flows (e.g., laminar, turbulent) and varying fluid properties (e.g., viscosity, density). This would demonstrate the generalization ability of HyFluid under diverse conditions.\n\n5. Considering that the constraint of constant radiance may not be practical in complex real-world scenes, I highly recommend that the authors compare the reconstruction results obtained with and without the constraint. \n\n6. The proposed model is only compared with two existing models, which is not sufficient. To make the results more convincing, the authors could incorporate more advanced neural rendering techniques designed specifically for dynamic scenes. Additionally, given that the dataset comprises synthetic fluid simulation data, it would be beneficial for the authors to provide quantitative results regarding the reconstruction of the velocity field in comparison to the ground truth on these simulated scenes.\n\nQUESTIONS:\n1. In the paper, a grid-based representation is used for density and velocity. When implementing $L_{density}$ and $L_{project}$, how exactly are these calculations performed? Are the loss functions computed for all grid positions, or is interpolation used to compute the losses at sampled points in space?\n\n2. What is the difference in performance between decomposing the velocity field into a base neural velocity field and a vortex particle-based velocity, versus solely using high-frequency position embedding?\n\n3. In the case of using physics-based loss functions in HyFluid, how does the predicted velocity field of HyFluid compare to the ground truth (GT) velocity field quantitatively? Can HyFluid be compared to other methods in terms of velocity field quantitatively?\n\n4. The paper does not provide experimental results and discussions regarding the constraint of constant radiance. What is the capability of HyFluid with real-world scenes that exhibit spatially-varying radiance? \n\n5. Can you provide more quantitative/ qualitative results of other baselines?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a neural reconstruction method for individual fluid flows which is re-trained for each new scene. It combines the established iNPGs for representing densities and velocities with vorticity transporting partices in a hybrid flow representation called HyFluid. A set of appearance- and physics-based losses is used to train the iNPGs and optimize vorticity and lighting in 3 steps. With this approach it is possible to recover 3D density and velocity from sparse views and enable novel-view rendering, re-simulation, and predictions of states over time.\n\nSTRENGTHS:\nI see the following strong points in this submission:\n- The method is build on physical priors to recover a physically meaningful velocity field.\n- The loss to the projected velocity to ensure divergence freeness is novel as far as I can tell.\n- Using particles to transport voriticity seems novel, at least in that context. \n- Provides better results then SOTA neural methods regarding re-simulation and prediction.\n- The results look good, the motion of the density is coherent and also suited for re-simulation.\n- The reconstruction works with simple, sparse images.\n\nWEAKNESSES:\nThe treatment for vorticity seems novel, but from the provided ablations I cannot see a substantial impact:\n- In Fig. 8 there seems to be a slight improvement.\n- In the videos, the ablation study results of \"w/o vort.\" and \"full\" look very similar.\n- The difference could be more visible in the flow field, and but this is not demonstrated.\n- In the videos, the novel view synthesis results of PINF [1] and HyFlow are almost identical.\n\nQUESTIONS:\n- Since the approach is trained on ScalarFlow [2] it would be interesting to see how the method compares to the ScalarFlow reconstructions. ScalarFlow is not using neural networks, and as such a good not-learned baseline as a single scene reconstruction method using physical priors (advection, pressure projection). Re-simulation and future prediction might also be possible there since ScalarFlow is using a fluid solver (MantaFlow) internally. Can the authors explain whether they haven't compared to the ScalarFlow 3D data despite using the 2D input images?\n- It would also be interesting to compare to GlobTrans [3] as this method it is trained/optimized as a re-simulation method. Have the authors looked into this?\n- The paper uses use an unseen GT view to evaluate the metrics. Have the authors tried/considered reconstructing a synthetic case to be able to make a 3D comparison? This might also show the efficacy of the flow reconstruction more clearly, since the visual appearance could be matched perfectly.\n- The velocity flickers in the supplemental video, there is no smooth evolution. Is there any coupling of the velocity over time (like self-advection)?\n- In the videos the velocity fields are rendered to 2D. Can the authors explain how is that done? Wouldn't a slice provide more insight in the actual motions? Averaging might blur the small-scale vorticities.\n\nOverall, I think the paper targets an interesting direction and provides an interesting approach for a tough problem. I still somewhat on the edge regarding this paper, but I'm open to readjusting/raising my score after the rebuttal.\n\nReferences:\n[1] Mengyu Chu, Lingjie Liu, Quan Zheng, Erik Franz, Hans-Peter Seidel, Christian Theobalt, and Rhaleb Zayer. 318 Physics informed neural fields for smoke reconstruction with sparse data. ACM Transactions on Graphics, 2022.\n[2] Marie-Lena Eckert, Kiwon Um, and Nils Thuerey. Scalarflow: a large-scale volumetric data set of real-world 330 scalar transport flows for computer animation and machine learning. ACM Transactions on Graphics (TOG), 2019.\n[3] Erik Franz, Barbara Solenthaler, and Nils Thuerey. Global transport for fluid reconstruction with learned 337 self-supervision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021.\n\nLIMITATIONS:\nLimitations are briefly discussed, I don't see anything significant missing here.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents an innovative neural dynamic reconstruction method that achieves good results in recovering fluid density and velocity fields through the introduction of physical constraints and a hybrid neural velocity representation. Despite some weaknesses and issues, this method has significant implications for the in-depth study and resolution of fluid dynamics problems. Future work could focus on refining the method and validating and applying it to a wider range of fluid scenarios.\n\nSTRENGTHS:\nThis method proposes a new approach to neural dynamic reconstruction that can simultaneously recover fluid density and velocity fields, overcoming the challenges posed by the visual ambiguities of fluid velocity in existing methods.\nPhysics-based losses are introduced to enforce a divergence-free velocity field, driving the transport of density and enhancing the accuracy of velocity estimation.\nA hybrid neural velocity representation is designed, incorporating a base neural velocity field that captures most irrotational energy and a vortex particle-based velocity that models residual turbulent velocity. This enables the method to effectively recover vortical flow details.\n\nWEAKNESSES:\nThe paper lacks detailed presentation of experimental results and quantitative evaluations, thus lacking sufficient validation of the method's performance.\nThere is no comparison with other methods in addressing the visual ambiguity of fluid velocity, making it difficult to assess the method's advantages and disadvantages comprehensively.\n\nQUESTIONS:\nFurther investigation is needed to determine whether the method can handle complex fluid scenarios such as non-Newtonian or multiphase fluids in practical applications.\n\nLIMITATIONS:\nThe paper does not mention the computational resource requirements, such as computation time and memory consumption, which are important factors for the feasibility of practical applications.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}

comments: SUMMARY:
This paper works on reconstructing fluid density and velocity from multi-view videos. The main idea is to inject visual clues with NeRF. They propose some physics-based regularization terms to deal with visual ambiguity that video can not reflect the inner fluid states.

STRENGTHS:
Good motivation: Visual ambiguity indeed is the critical problem that needs to be solved in this setting. 

Novel framework: Introducing physical losses is a not-easy but natural point. The introduced losses are general to inspire other related researchers.

WEAKNESSES:
1. Setting is not new. Indeed, reconstructing fluid velocity and density from videos has been studied in NeuroFluid (ICML 2022). I didn't find the introduction or comparison. Why?

2. The proposed method is not convincing to me. 

   -  The proposed physics-based losses can only play a regularization role in the training. To solve the visual ambiguity, the framework must introduce direct supervision of the fluid velocity. Nevertheless, ambiguity still exists. 

   - And, the key question is how to get the initial state of the fluid, which decides the performance of this work. The authors **should** clarity this point.

3. Experiments are incomplete, lacking key results. 

    - Experiments just show the render results and velocity results. How about density? The authors claim that they can recover fluid density but not shown in the experiments. 

    - How did you evaluate the render results? Directly render fluid from the view angle used in the training stage? Please claim them before your experiments.
    - Only evaluate the stoke with a similar environment. Can you show more examples, with more complex shapes and materials?

QUESTIONS:
For details, please refer to the weaknesses section.

LIMITATIONS:
yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-08

comments: SUMMARY:
In this paper, the authors propose a method (HyFluid) to infer the fluid density and velocities from multiview videos. To deal with the visual ambiguities of fluid, physics-losses are introduced to try to enforce physics plausible velocities. A neural velocity field and a vortex particle-based velocity are introduced to capture the irrotational energy and model residual turbulent velocity respectively. Th experiments show that the proposed method has the ability to recover the vortical flow details to some extent.

STRENGTHS:
The paper is overall well written and organized. 
The proposed density loss and projection loss are well derived from the fluid mechanics.
The experiments show that the proposed method can perform better than the baselines.

WEAKNESSES:
The authors claim that the method can give physics plausible estimation of fluid fields. The physics intuitions for the density loss and projection are clear according to the incompressible condition. However, I don't see obvious physics intuition for the laminar loss.

For the problem that visual appearance of fluids depends on lighting and fluid substance properties, there is no further exploration in this direction. The problem becomes more important when the fluid is liquid.

QUESTIONS:
- Does the MGPCG applied to every simulation step to project the velocity during the training? How is the runtime cost for computing the projected velocity?
- Why to encourage high-density regions to have non-zero velocity (line 137)? What is the physics intuition here?
- As the visual appearance of fluid highly depends on the rendering process, does the results very sensitive to the parameters of the renderer?

LIMITATIONS:
yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-09