SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Hong-Xing Yu $^{1*}$  Yang Zheng $^{1*}$  Yuan Gao $^{1}$  Yitong Deng $^{1}$  Bo Zhu $^{2}$  Jiajun Wu $^{1}$
We study recovering fluid density and velocity from sparse multiview videos. Existing neural dynamic reconstruction methods predominantly rely on optical flows; therefore, they cannot accurately estimate the density and uncover the underlying velocity due to the inherent visual ambiguities of fluid velocity, as fluids are often shapeless and lack stable visual features. The challenge is further pronounced by the turbulent nature of fluid flows, which calls for properly designed fluid velocity representations. To address these challenges, we propose hybrid neural fluid fields (HyFluid), a neural approach to jointly infer fluid density and velocity fields. Specifically, to deal with visual ambiguities of fluid velocity, we introduce a set of physics-based losses that enforce inferring a physically plausible velocity field, which is divergence-free and drives the transport of density. To deal with the turbulent nature of fluid velocity, we design a hybrid neural velocity representation that includes a base neural velocity field that captures most irrotational energy and a vortex particle-based velocity that models residual turbulent velocity. We show that our method enables recovering vortical flow details. Our approach opens up possibilities for various learning and reconstruction applications centered around 3D incompressible flow, including fluid re-simulation and editing, future prediction, and neural dynamic scene composition. Project website: https://kovenyu.com/HyFluid/.
Fluid is ubiquitous in our surroundings, from a small breeze in the morning to large-scale atmosphere flow that would affect the weather in the following week. Understanding and predicting fluid dynamics play a central role in climate forecasting (Bauer et al., 2015), vehicle design (Bushnell & Moore, 1991), visual special effect (Pfaff et al., 2010), etc. Yet, it remains an open problem in scientific machine learning to accurately recover fluid flows from visual observations. Flow motions can only be seen indirectly, and, unlike solids that have certain shapes and simple constrained motion patterns, fluid systems have intricate and complex dynamics that exhibit different features across spatial scales. These characteristics pose unique challenges in the representations and algorithms to recover physically correct velocity. Recently, physics-informed neural networks (Raissi et al., 2019) have shown promise in recovering velocity fields from density fields for scientific computing. These neural methods are scalable and flexible compared to traditional grid-based methods, yet they require accurate density fields that are unavailable in uncontrolled ordinary scenes.
In this work, we focus on recovering the density $^{\dagger}$  and velocity of fluids from sparsely captured multiview videos (see Fig. 1 for illustration). We identify three key challenges. Firstly, the fluid velocity is ambiguous from visual observations. Unlike solid objects characterized by consistent
$^{\dagger}$  "Density" refers to the concentration of the fluid substance such as smoke soot. Not to be confused with the physical density, i.e., mass over volume.
Figure 1: We aim at inferring hybrid neural fluid representations of density and velocity from sparse multi-view videos. This allows novel view video synthesis, re-simulation, and dynamic scene composition.
shapes and stable visual features, fluids are inherently shapeless, often monochromatic, and semitransparent. This lack of physical definition makes visual tracking an implausible task, contributing to pronounced visual ambiguity, particularly in regions of laminar flow where the fluid appearance hardly changes. This challenge is beyond the capacity of general-purpose neural dynamic reconstruction methods (Li et al., 2021; Du et al., 2021), as they predominantly depend on optical flow which is ineffective in this context.
Secondly, accurately depicting fluid velocity necessitates appropriate representations that respect its turbulent features. Fluid flows exhibit varying features across multiple spatial scales (Frisch & Donnelly, 1996), a phenomenon that is challenging to capture within a single neural representation due to its spectral bias (Rahaman et al., 2019). At the human scale, fluid motions express not only laminar flows but also turbulence (Pope, 2000). This results in a complicated entanglement of rotational, shearing, and smooth motions. The turbulent flows are high-frequency in both space and time. Therefore, the representation of such flow should be able to accommodate a broad spectrum of signal frequencies while also preserving the unique structure of rotational flows.
Lastly, it is inherently ambiguous to reconstruct physically plausible 3D density fields from sparse observations. Recovering fluid density and appearance from limited 2D videos is ill-posed primarily due to the intricate non-linear absorption and scattering processes associated with semi-transparent fluids (Max, 1995). Consequently, the reconstruction of under-constrained continuous density fields from sparse observations is an intrinsically difficult task.
To address these challenges, we propose hybrid neural fluid fields (HyFluid), a neural approach to jointly infer fluid density and velocity from sparse-view videos. At the core of HyFluid are two key technical contributions including a set of simple yet effective physics-based losses and a hybrid neural velocity representation. Our physics-based losses leverage physical constraints from the Navier-Stokes equations to jointly learn physically plausible fluid density and velocity fields. The physics-based losses enforce inferring a divergence-free velocity field that drives the transport of the density field. In addition to the new losses, we propose a hybrid fluid velocity representation as it is difficult for a single neural representation to capture turbulent fluid velocity. The main idea is to decompose our fluid velocity field into a base and a residual turbulent velocity field. The base velocity field is represented by a neural field that captures most irrotational energy, and the residual velocity field is represented by vortex particles that feature highly rotational and shearing velocities. In order to address the 2D-to-3D density ambiguity, we leverage both visual imaging signals by differentiable volume rendering and the physical fluid transport constraint to regularize the density field.
We demonstrate that HyFluid can yield high-fidelity recovery of density and velocity, allowing novel view video synthesis, re-simulation, editing, future prediction, and neural dynamic scene composition (Fig. 1). In summary, our contributions are three-folded: (1) We propose hybrid neural fluid fields (HyFluid), a neural approach to infer fluid density and velocity from sparse multiview videos. (2) We show that using simple physics-based losses and a hybrid neural velocity representation allows uncovering turbulent fluid motion from sparse observations. (3) We evaluate our method on novel
view re-simulation and novel-view future prediction to benchmark joint reconstruction for real fluids, and we show that our approach delivers high-fidelity synthesis and reconstructions in comparison to state-of-the-art neural methods.
Incompressible Navier-Stokes equations. Our motion representation and learning signals are motivated by incompressible Navier-Stokes equations. With a low flow speed, fluid motion is considered incompressible and can be well described by:
$$
\frac {D \mathbf {u}}{D t} = \frac {\partial \mathbf {u}}{\partial t} + \mathbf {u} \cdot \nabla \mathbf {u} = - \frac {1}{\rho} \nabla p + \nu \nabla \cdot \nabla \mathbf {u} + \mathbf {f}, \tag {1}
$$
$$
\nabla \cdot \mathbf {u} = 0. \tag {2}
$$
Here, Eqn. 1 is known as the momentum equation, where the LHS is the material derivative for velocity,  $\frac{D\mathbf{u}}{Dt} = \frac{\partial\mathbf{u}}{\partial t} +\mathbf{u}\cdot \nabla \mathbf{u}$ , which represents the time rate of change of velocity  $\mathbf{u}$  of a fluid parcel, and the RHS is the momentum induced by pressure gradient  $-\frac{1}{\rho}\nabla p$  (where  $\rho$  denotes the physical density of the fluid parcel), viscosity  $\nu \nabla \cdot \nabla \mathbf{u}$ , and some external force  $\mathbf{f}$ . We consider inviscid fluid and thus drop the viscosity term. Eqn. 2 is known as the mass conservation equation, meaning that the mass flowing into a controlled volume should be equal to the mass flowing out of the volume.
Differentiable volume rendering. We integrate volume rendering for joint learning of visual appearance, density, and velocity from videos. Volume rendering is a suitable model for translucent materials and participating media like smoke and fog (Kajiya & Von Herzen, 1984). In volume rendering, the radiance  $L(\mathbf{o}, \mathbf{d})$  arriving at the camera location  $\mathbf{o}$  from direction  $\mathbf{d}$  (a.k.a. the radiance of a camera ray  $\mathbf{r}(t) = \mathbf{o} - t\mathbf{d}$ ) with near and far bounds  $t_n$  and  $t_f$  is given by
$$
L (\mathbf {o}, \mathbf {d}) = \int_ {t _ {n}} ^ {t _ {f}} T (t) \sigma (\mathbf {r} (t)) L _ {\mathrm {e}} (\mathbf {r} (t), \mathbf {d}) d t, \tag {3}
$$
where  $T(t) = \exp(-\int_{t_n}^t \sigma(\mathbf{r}(s)) ds)$  denotes the transmittance along the ray from  $t_n$  to  $t$ ,  $L_e$  denotes the emitting radiance, and  $\sigma$  denotes the optical density which can be considered proportional to the concentration density of fluid substance according to Beer-Lambert law. Following recent neural rendering methods (Mildenhall et al., 2020), we use quadrature to discretize it (Max, 1995).
3 Hybrid Neural Fluid Fields
Our goal is to recover the fluid appearance, density, and velocity from sparse multiview RGB videos. To this end, we propose hybrid neural fluid fields (HyFluid). At the core of HyFluid are a set of simple physics-based losses and a hybrid neural velocity representation. In particular, HyFluid aims to infer a neural density field  $\sigma(x,y,z,t)$ , the appearance of the fluid  $L_{\mathrm{e}}$ , and the underlying velocity  $\mathbf{u}(x,y,z,t) = \mathbf{u}_{\mathrm{base}}(x,y,z,t) + \mathbf{u}_{\mathrm{vort}}(x,y,z,t)$ , where we decompose it into a base neural velocity field  $\mathbf{u}_{\mathrm{base}}$  and a residual vortex particle-driven turbulent velocity.  $\mathbf{u} = [u,v,w]$  denotes the velocity along  $x,y,z$  axis, respectively. In the following, we first introduce the physics-based losses, then we describe our hybrid velocity representation, and finally, we summarize our learning signals for the joint inference. We show a conceptual illustration in Fig. 2.
Physics-based losses. We propose physics-based losses  $\mathcal{L}_{\mathrm{physics}}$  including a density loss, a projection loss, and a laminar regularization loss.
Density loss. While the neural density field can be learned through differentiable volume rendering, the underlying velocity field is hidden from the visual observations. Unlike solids that have regular shapes and invariant visual features to track, fluids are often shapeless, monochromatic, and semitransparent. This makes it implausible to visually track it. To uncover the hidden velocity from visual observations, we resort to the density transport equation in incompressible flows,  $\frac{D\sigma}{Dt} = 0$ , and introduce a physics-informed supervision signal:
$$
\mathcal {L} _ {\text {d e n s i t y}} = \underset {x, y, z, t} {\mathbb {E}} \left[ \frac {\partial \sigma}{\partial t} + u \cdot \frac {\partial \sigma}{\partial x} + v \cdot \frac {\partial \sigma}{\partial y} + w \cdot \frac {\partial \sigma}{\partial z} \right]. \tag {4}
$$
Figure 2: Illustration of hybrid neural fluid fields (HyFluid) which aims to jointly infer fluid density and velocity from videos using visual signals via differentiable volume rendering and physics-based losses. To facilitate learning turbulent velocity, we decompose the velocity field into a base velocity and a vortex particle-driven velocity. See the main text for detailed descriptions.
This loss says that the 3D velocity field  $(u,v,w)$  at any given moment  $t$  should transport the density field  $\sigma$  such that it evolves to the density field at the next moment, in a similar spirit to scene flow (Vedula et al., 1999).
Projection loss. Another important physical property of incompressible flow is mass conservation depicted in Eqn. 2, that is, the velocity field should be divergence-free. A straightforward way to enforce divergence-free condition is to impose a loss on the divergence of velocity,  $\frac{\partial u}{\partial x} + \frac{\partial v}{\partial y} + \frac{\partial w}{\partial z}$ . However, we find that this empirically leads to a degenerated solution, where only the laminar flows (i.e., smooth, direct-current flows) can be uncovered as this is a trivial local minimum for every point. Yet, the incompressibility of fluid flows is confined through a global dynamic system, modulated by the pressure of the fluid.
Therefore, we need a globally-aware, physically-plausible supervision that does not conform to trivial local minima. Motivated by the commonly used pressure projection solver in computational fluid dynamics, we propose a projection loss that constrains our learned velocity to be divergence-free:
$$
\mathcal {L} _ {\mathrm {p r o j}} = \underset {x, y, z, t} {\mathbb {E}} \left[ \| \mathbf {u} - \mathbf {u} _ {p} \| ^ {2} \right], \tag {5}
$$
where  $\mathbf{u}_p = \mathrm{project}(\mathbf{u})$  are the pressure-projected velocity field, using a pressure project solver. A project solver implements a Helmholtz decomposition by solving a global linear system for pressure, and it projects the velocity field  $\mathbf{u}$  to a divergence-free manifold using the gradient of the solved pressure. Also, we notice that any divergence-free velocity field  $\mathbf{u}$  incurs zero projection loss as it is an identity mapping for divergence-free fields. Thus, this projection loss essentially constrains the uncovered velocity within a divergence-free subspace, and it is not subject to trivial local minima of pure direct-current flows.
Laminar regularization loss. These density and projection losses do not warrant physically-correct velocity reconstruction for laminar flows. In laminar flows, all partial derivatives of density are zero, so any velocity, e.g., all-zero velocity, incurs zero  $\mathcal{L}_{\mathrm{density}}$ . As for  $\mathcal{L}_{\mathrm{project}}$ , all-zero velocity fields also incur a zero loss. Hence, we introduce a laminar regularization loss:
$$
\mathcal {L} _ {\text {l a m i n a r}} = \underset {x, y, z, t} {\mathbb {E}} [ \max  (0, \gamma \sigma - \| \mathbf {u} \|) ], \tag {6}
$$
which is a hinge loss that encourages high-density regions to have non-zero velocity, and  $\gamma$  denotes a hyperparameter to scale the threshold according to the velocity magnitude unit. These three losses are complementary and interconnected to uncover a physically plausible fluid flow.
Hybrid neural velocity representation. At the human scale, fluid flows are often turbulent. They exhibit complex dynamic features including circular and shearing flows. In addition to the common challenges of reconstructing non-turbulent flows, turbulent flows are even more intricate and difficult to model and capture. They are high-frequency flows in both space and time. This demands that the velocity representation be able to accommodate a wide range of signal frequencies and the particular
structure of rotational flows. However, neural representations are known to have certain spectral inductive biases toward representing low-frequency signals (Rahaman et al., 2019). While this can be alleviated by position embedding (Tancik et al., 2020) or sinusoidal activation functions (Sitzmann et al., 2020), they still lack a proper structure to capture and represent highly detailed turbulent flows.
We propose a hybrid neural velocity representation to tackle this challenge. The main idea is to decompose the underlying flow field  $\mathbf{u} = \mathbf{u}_{\mathrm{base}} + \mathbf{u}_{\mathrm{vort}}$  into a base neural velocity field  $\mathbf{u}_{\mathrm{base}}$  and a residual vorticity-driven velocity  $\mathbf{u}_{\mathrm{vort}}$ . The base neural velocity field captures the large-scale flow motion, and the vorticity-driven velocity is a dedicated representation of the small-scale vortex details that are hard to characterize by the base field. For the base velocity field  $\mathbf{u}_{\mathrm{base}}$ , we extend instance neural graphics primitive (iNGP) (Müller et al., 2022) representation to the temporal domain.
The residual vorticity-driven flow  $\mathbf{u}_{\mathrm{vort}}$  is complementary to the base flow by leveraging the physical structure of turbulent fluid flows. The physical model behind the vorticity-driven flow is prescribed by the curl form of the Navier-Stokes equations (see Cottet et al. (2000) for more details):
$$
\frac {D \boldsymbol {\omega}}{D t} = \frac {\partial \boldsymbol {\omega}}{\partial t} + \mathbf {u} \cdot \nabla \boldsymbol {\omega} = \boldsymbol {\omega} \cdot \nabla \mathbf {u} + \nu \nabla \cdot \nabla \boldsymbol {\omega} + \nabla \times \mathbf {f}, \tag {7}
$$
where  $\omega = \nabla \times \mathbf{u}$  denotes the vorticity. We assume inviscid fluid and conservative body force, and thus only the vortex stretching term  $\omega \cdot \nabla \mathbf{u}$  is non-zero on the RHS. In this case, Eqn. 7 becomes the vorticity transport equation, saying that vorticity is stretched and advected by the velocity flow.
This physical model has been widely used in fluid simulation (Cottet et al., 2000), known as vortex methods. In particular, we consider the vortex particle methods (Selle et al., 2005), where we use a particle-based representation for vorticity, which is low-dimensional, is easy to temporally evolve, and well embeds the circular physical flow structures in it. We represent a vortex particle  $p$  at some time stamp  $t$  by a triplet  $\{I_p, \mathbf{x}_p^t, \boldsymbol{\omega}_p^t\}$  of its intensity  $I_p$ , position  $\mathbf{x}_p^t$ , and vorticity  $\boldsymbol{\omega}_p^t$ . Our vorticity-driven flow  $\mathbf{u}_{\mathrm{vort}}(\mathbf{x}, t)$  induced by vortex particles is represented by:
$$
\mathbf {u} _ {\mathrm {v o r t}} (\mathbf {x}, t) = \sum_ {p} I _ {p} \left(\mathbf {N} _ {p} \times \tilde {\boldsymbol {\omega}} _ {p}\right), \quad \mathbf {N} _ {p} = \left(\mathbf {x} _ {p} ^ {t} - \mathbf {x}\right) / \left\| \mathbf {x} _ {p} ^ {t} - \mathbf {x} \right\|, \quad \tilde {\boldsymbol {\omega}} _ {p} = K \left(\mathbf {x} - \mathbf {x} _ {p} ^ {t}\right) \boldsymbol {\omega} _ {p} ^ {t}, \tag {8}
$$
where  $K(\mathbf{x}) = \exp (-\| \mathbf{x}\| ^2 /2r^2) / (r^3 (2\pi)^{3 / 2})$  is a Gaussian distribution kernel. Intuitively, every vortex particle "carries" a local circular momentum, which is itself transported with the flow prescribed by Eqn. 7. Collectively, a set of vortex particles allows learning complex flow details which are hard to capture by the base velocity field.
However, naively learning the vortex particles leads to poor local minima due to the complex interdependence of  $\mathbf{u}$  and  $\{I_p,\mathbf{x}_p^t,\boldsymbol{\omega}_p^t\}$ . Thus, we make two simplifications. First, we assume that most energy is captured by the base flow and only use the base flow to transport vortex particles. This is achieved by first learning the base flow and then freezing the base flow to learn the residual flow. Second, we introduce a seeding strategy to disentangle learnable parameters from the particle transport: we seed overly many vortex particles on high-curl spatio-temporal locations and precompute the trajectory  $\{\mathbf{x}_p^t\}$  and  $\{\boldsymbol{\omega}_p^t\}$  for all  $t$  and  $p$  by the learned base flow. Therefore, the learnable parameters are reduced to  $\{I_p\}$ . The redundant vortex particles are automatically suppressed by learning to have zero intensities. We leave more details of the seeding strategy in the Appendix.
Joint inference of fluid fields. It is a highly ill-posed problem to recover density and appearance from sparse visual observations, as the imaging process for semi-transparent fluids involves complex non-linear absorption and scattering. The visual appearance of fluids depends not only on density but also lighting and fluid substance properties. Therefore, recovering turbulent, under-constrained continuous density fields from limited observations is inherently challenging.
To address this challenge, we leverage both visual imaging signals and physical fluid transport constraints by jointly learning density, appearance, and velocity. The visual signal supervision through differentiable volume rendering is given by:
$$
\mathcal {L} _ {\text {r e n d e r}} = \underset {\mathbf {o}, \mathbf {d}, t} {\mathbb {E}} \left[ \| L _ {\text {r e n d e r}} (\mathbf {o}, \mathbf {d}) - L _ {\text {o b s e r v e}} (\mathbf {o}, \mathbf {d}) \| ^ {2} \right], \tag {9}
$$
where  $L_{\mathrm{render}}$  is our volume rendered values by Eqn. 3 and  $L_{\mathrm{observe}}$  is sampled from video frames. Notice that we sample rays continuously in the viewing frustums instead of sampling through only pixel centers as NeRF-like methods do (Mildenhall et al., 2020), as we aim to learn continuous fields
and we rely on good partial derivatives from the fields. The physical transport constraint is provided by Eqn. 4, i.e., we supervise the partial derivatives of the density field. We compute all derivatives by auto-differentiation (Paszke et al., 2017). Our loss function is given by:
$$
\mathcal {L} = \beta_ {\text {r e n d e r}} \mathcal {L} _ {\text {r e n d e r}} + \beta_ {\text {d e n s i t y}} \mathcal {L} _ {\text {d e n s i t y}} + \beta_ {\text {p r o j}} \mathcal {L} _ {\text {p r o j}} + \beta_ {\text {l a m i n a r}} \mathcal {L} _ {\text {l a m i n a r}}, \tag {10}
$$
where all the  $\beta$ s are loss weights.
Another inherent ambiguity is between emitting radiance and density. As in Eqn. 3, the emitting radiance is multiplied by density, which means that an unconstrained spatially-varying emitting radiance field admits all sorts of trivial solutions for density. Empirically, we find little differences between using learned spatially-varying color and using a constant color. Thus, we set emitting radiance to be constant to remove this ambiguity.
Fluid reconstruction. Fluid flow reconstruction from visible light measurement has been widely studied in science and engineering. Well-established methods used in controlled lab environments include active sensing techniques such as laser scanners (Hawkins et al., 2005), light path (Ji et al., 2013), and structural light (Gu et al., 2012), as well as passive marker-based methods such as particle imaging velocimetry (PIV) (Adrian & Westerweel, 2011; Elsinga et al., 2006) which injects passive particles into the fluid flows. These methods allow accurate flow measurement, yet they require specialized setup for markers, lighting, and capturing devices.
Recent methods also seek to reconstruct fluid flows from casual visible light measurements without a specialized setup. Earlier works use tomography and linear imaging formation to reconstruct density grids from visual observations (Gregson et al., 2014; Okabe et al., 2015) and then estimate velocity using physical priors (Gregson et al., 2014; Eckert et al., 2018). This line of work has been extended to joint optimization for both velocity and density (Eckert et al., 2019) to improve reconstruction quality. Since this is a highly unconstrained problem, synthesized view supervision is proposed to regularize the reconstruction (Zang et al., 2020). Franz et al. (2021) introduce a density-based rendering formation with a global differentiable simulation framework. They use manually calibrate lighting directions and fluid source location to allow consistent differentiable simulation optimization. However, these methods ignore the visual appearance, suffering from the inherent visual ambiguity of fluid density and appearance.