SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Neural dynamic scene representations. Using neural networks as implicit visual scene representations have been made effective and popular. Earlier works use neural representations for geometry (Park et al., 2019; Mescheder et al., 2019) and visual appearance (Sitzmann et al., 2019). As a seminal work, neural radiance fields (NeRFs) (Mildenhall et al., 2020) incorporate volume rendering to learn implicit geometry and appearance from images. Follow-ups of NeRFs extend applications from novel view synthesis to dynamic scene reconstruction (Li et al., 2021; Pumarola et al., 2021), relighting (Zhang et al., 2021; Yu et al., 2023), scene segmentation (Yu et al., 2022; Sajjadi et al., 2022), robot manipulation (Le Cleac'h et al., 2023; Tian et al., 2023), system identification (Li et al., 2023), etc.
Generic dynamic NeRFs incorporate deformation fields (Park et al., 2021b; Pumarola et al., 2021; Park et al., 2021a) or scene flow fields (Li et al., 2021; Du et al., 2021; Xian et al., 2021; Li et al., 2022) to represent motion. These motion representations essentially rely on stable visual features such as color gradients and edges, and thus they are not suitable for uncovering fluid flows which generally do not exhibit stable visual features. Notably, (Chu et al., 2022) is the most relevant work to ours. Chu et al. (2022) propose the physics-informed neural fields (PINF) that incorporate physics-informed losses (Raissi et al., 2019) to reconstruct fluid flows from sparse-view videos with learned priors from synthetic data. However, PINF does not consider the inherent physical structures of real fluid flows and thus only reconstructs laminar flows. Our HyFluid allows uncovering turbulent real flows by novel physics-based losses and the hybrid neural velocity representation.
Learning fluid dynamics. Learning fluid dynamics holds the promise to accelerate simulation and aid fluid reconstruction. Earlier work on learning fluid dynamics to accelerate simulation includes using convolutional networks to evolve fluid states (Tompson et al., 2017; Ummenhofer et al., 2019; Prantl et al., 2022) or learning latent simulation (Wiewel et al., 2019; Kim et al., 2019), supervised by 3D
Figure 3: Visualization of novel view synthesis results on a real capture. Note that ours more faithfully recovers the density distribution of the fluid without floaters or missing regions.
Figure 4: Visualization of re-simulation results on a real capture. Ours synthesizes reasonable re-simulation.
fluid simulation data. While these methods learn from 3D fluid data, a few recent work such as Guan et al. (2022) and Liu et al. (2023) learn simulators from multi-view videos. However, both of them assume to have groundtruth reconstructed initial fluid states, restricting the applications to synthetic data. Thus, it still remains open how existing methods may help fluid reconstruction. An exception is Deng et al. (2023) that use low-dimensional vortex particles to represent fluid velocity fields and thus allow reconstructing fluid velocity from a single video, yet their formulation is restricted to 2D domain due to complex vortex stretching in 3D vortex dynamics.
Datasets. We use both real captures and synthetic simulation for evaluation. For real captures, we use the ScalarFlow dataset (Eckert et al., 2019) which consists of videos of buoyancy-driven rising smoke plumes. We use the first five scenes from the real captures. For each scene, there are five video recordings. The five cameras are fixed-position throughout capture and distributed evenly across a  $120^{\circ}$  arc centered at the rising smoke. Each video has 150 frames with  $1062 \times 600$  resolution. These videos have been post-processed to remove backgrounds. We follow Chu et al. (2022) to use 120 frames for each video. For each scene, we use four videos for training, and one held-out video for testing (i.e., as the groundtruth for the novel view).
Synthetic simulation allows evaluating 3D velocity and density against the groundtruth. We use ScalarFlow synthetic dataset generation code (Eckert et al., 2019). We generate five examples with different inflow source with higher viscosity and another five examples with lower viscosity.
Evaluation metrics. For real fluid flows we do not have volumetric 3D groundtruth, so we use novel view video rendering to evaluate the reconstruction quality. In particular, we use the following three tasks: novel view video synthesis, novel view re-simulation, and novel view future prediction. Re-simulation and future prediction require high-fidelity velocity reconstruction and thus provide a means to evaluate velocity reconstruction. We use peak signal-noise ratio (PSNR), structural similarity index measure (SSIM), and the perceptual metric LPIPS (Zhang et al., 2018).
For synthetic data, we can evaluate the reconstruction results against the simulation groundtruth. Since the simulation groundtruth are up to a scale, we use scale-invariant RMSE to measure the performance. We only compute metrics where groundtruth density is greater than 0.1 to rule out empty space (which is otherwise dominant) for clearer quantitative comparison. In particular, we consider volumetric density error (by querying density networks at the simulation grid points) to
Figure 5: Visualization of future prediction results on a real capture. Notice how our plume expands while PINF (Chu et al., 2022) only moves the plume up.
Models	Novel view synthesis	Re-simulation	Future prediction
PSNR↑	SSIM↑	LPIPS↓	PSNR↑	SSIM↑	LPIPS↓	PSNR↑	SSIM↑	LPIPS↓
NeRFlow (Du et al., 2021)	18.01	0.7880	0.1958	17.20	0.8174	0.1658	-	-	-
GlobTran (Franz et al., 2021)	25.97	0.9312	0.0783	24.55	0.8988	0.1017	-	-	-
NeuroFluid (Guan et al., 2022)	22.41	0.8452	0.1560	-	-	-	-	-	-
PINF (Chu et al., 2022)	29.55	0.9244	0.0881	24.97	0.9109	0.1278	24.19	0.8366	0.2155
HyFluid (ours)	31.14	0.9330	0.0966	28.37	0.9158	0.1171	26.12	0.8448	0.1968
Table 1: Comparison on renderings for three tasks on real captures.
evaluate density prediction, and warp error (i.e., using velocity to advect density and comparing to GT density) to evaluate both density and velocity prediction.
Baselines. We consider three recent flow reconstruction methods: NeRFlow (Du et al., 2021), a neural dynamic reconstruction method for fluid reconstruction; GlobTran (Franz et al., 2021), a grid-based fluid reconstruction method; and PINF (Chu et al., 2022), the latest neural fluid reconstruction method. We also compare to NeuroFluid (Guan et al., 2022), a recent fluid dynamics learning approach. Since NeuroFluid requires having 3D density and velocity at the first frame, we use PINF reconstructed fields for it. We leave training details and compute resources in Appendix.
5.1 Novel view video synthesis
In this task, we aim to evaluate the inferred density fields by synthesizing novel view videos. We use all 120 frames from the held-out test view for all scenes and report the average numbers. As shown in Table 1, our method is comparable to the state-of-the-art method PINF (Chu et al., 2022). Notice that ours is better in PSNR which is an objective metric that measures the optical similarity of the synthesized image and the groundtruth. GlobTrans (Franz et al., 2021) uses adversarial loss together with a specific rendering formulation, which may contribute to better perceptual LPIPS performances. Yet, this compromises objective fidelity as reflected by lower PSNR. Qualitative results in Fig. 3 showcase the capability of our method to accurately reconstruct density fields, which paves the way for the inference of high-fidelity physically-plausible velocity fields.
5.2 Re-simulation
The re-simulation task accesses the ability of the models to infer the velocity fields accurately. Specifically, we use the reconstructed density from the first frame and advect the density across time to the last frame using the learned velocity. We use the standard MacCormack method (Selle et al., 2008) for advection. As for fluid source, we use the reconstructed density field in the bottom of the scenes. Results in Table 1 and Fig. 4 show that our method generates reasonable results, whereas PINF (Chu et al., 2022) and NeRFlow (Du et al., 2021) completely fail.
Figure 6: Velocity slice on a real capture.
Figure 7: Visualization of velocity field inferred by HyFluid from a real 3-view capture.
Low vis.	Den. err.↓	Warp err.↓	PSNR↑	SSIM↑	LPIPS↓
PINF	5.01	4.84	23.43	0.8555	0.2153
Ours	2.94	3.37	27.28	0.8616	0.1285
High vis.	Den. err.↓	Warp err.↓	PSNR↑	SSIM↑	LPIPS↓
PINF	4.91	4.93	27.26	0.8728	0.1537
Ours	2.85	3.21	28.57	0.8670	0.1233
Table 2: Evaluation on synthetic data of different levels of viscosity.
Figure 8: Novel view re-simulation on synthetic data.
We further visualize the velocity field by slicing it in Fig. 6, which demonstrates that our method effectively captures the velocity details that are missed by the baseline methods, which tend to default to trivial solutions where all the velocity is upward-directed.
5.3 Future prediction
In the future prediction task, we extrapolate the fluid motion into the future based on the inferred velocity fields at a single frame. In particular, we follow the standard grid-based fluid simulation (Fed-kiw et al., 2001) to evolve the velocity fields by self-advecting them using the MacCormack method, followed by a projection step to ensure the divergence-free constraint. Table 1 shows that our method predicts future frames that are quantitatively better than PINF. Fig. 5 illustrates that compared to PINF, the future fluid state predicted by our method maintains its original structure and continues to follow a natural upward trajectory, and also produces natural details. These results underline the robustness of our method in inferring complex fluid velocity.
5.4 Inference on in-the-wild real capture
While ScalarFlow dataset provides real captures of plumes from a controlled environment, we are also interested in inferring fluid fields from uncontrolled in-the-wild videos. We use three cameras to capture sparse multi-view videos of a lit incense, and we show a visualization of the inferred density (top row) and velocity (bottom row) in Fig. 7. While the fluid in this example is much thinner than plumes, HyFluid allows plausible inference of fluid fields.
5.5 Evaluation on synthetic data
We include synthetic examples for evaluating 3D density and velocity fields against the groundtruth. We compare to PINF which has shown state-of-the-art results in 3D fluid fields reconstruction. We show 3D fields evaluation results together with novel view re-simulation results in Table 2 and Fig. 8. We observe that ours outperforms PINF on both 3D fields and 2D rendering, especially in preserving the vortical structures and intricate shape details. This is validated by larger performance margins in low-viscosity examples, since there is more turbulence that is hard to capture by PINF.
5.6 Additional applications
The inherent properties of neural representation in our method allow for its applicability in a variety of additional contexts, such as editing and scene composition. As seen in Fig. 1, the results obtained
Figure 9: Visualization of our turbulence editing by intensifying the learned vortex particles on a real capture.
Figure 10: Ablation study by re-simulation.
from our method can be effortlessly integrated into dynamic NeRF scenes (Sara Fridovich-Keil and Giacomo Meanti et al., 2023) or imported into professional-grade graphics software.
Our hybrid neural velocity representation naturally supports turbulence editing in re-simulation. In particular, we multiply our vortex particle intensity by a factor of 4 and showcase a visualization of edited re-simulation in Fig. 9. This shows that the high-quality velocity estimation from our hybrid neural velocity representation allows easily synthesizing vortical flow details.
5.7 Ablation study
We show qualitative results in Fig. 10 to evaluate the proposed physics-based losses, the vortex particle-driven velocity representation, and the joint training supervision. Starting from "Naive" where we only use the rendering loss for recovering density and the density loss for recovering velocity, we gradually add back laminar loss ("+Llaminar"), projection loss ("+Lproject"), and vortex particle-driven residual velocity ("Full"). From Fig. 10 we observe that our physical losses allow better velocity recovery. In particular, the projection loss enables recovering physically correct velocity that reconstructs the plume shape in re-simulation. Vortex particle-driven velocity allows capturing more details and thus more consistent re-simulation.
In this work, we study recovering fluid density and velocity from sparse multi-view videos. We propose hybrid neural fluid fields (HyFluid) which features physics-based losses to address inherent visual ambiguities and a hybrid neural velocity representation for capturing the complex turbulent fluid flow. We show that our simple designs can already lead to physically plausible estimation of fluid fields that supports applications such as re-simulation, editing, and dynamic scene composition.
Limitations. Our method assumes inviscid fluid and only considers gas but not liquid. Liquid features free surface that may require further physical modeling and constraints.
Acknowledgments. This work was in part supported by the Toyota Research Institute (TRI), NSF RI #2211258, ONR MURI N00014-22-1-2740, the Stanford Institute for Human-Centered AI (HAI), and Google. B. Zhu acknowledges NSF IIS #2313075, IIS #2106733, and CAREER #2144806.
Ronald J Adrian and Jerry Westerweel. Particle image velocimetry. Cambridge university press, 2011. 6
Steven F Ashby and Robert D Falgout. A parallel multigrid preconditioned conjugate gradient algorithm for groundwater flow simulations. *Nuclear science and engineering*, 1996. 14
Peter Bauer, Alan Thorpe, and Gilbert Brunet. The quiet revolution of numerical weather prediction. Nature, 2015. 1
Dennis M Bushnell and KJ Moore. Drag reduction in nature. Annual review of fluid mechanics, 1991. 1
Mengyu Chu, Lingjie Liu, Quan Zheng, Erik Franz, Hans-Peter Seidel, Christian Theobalt, and Rhaleb Zayer. Physics informed neural fields for smoke reconstruction with sparse data. ACM TOG, 2022. 6, 7, 8
Georges-Henri Cottet, Petros D Koumoutsakos, et al. Vortex methods: theory and practice, volume 8. Cambridge university press, 2000. 5
Sundar Vedula, Simon Baker, Peter Rander, Robert Collins, and Takeo Kanade. Three-dimensional scene flow. In ICCV, 1999. 4
Steffen Wiewel, Moritz Becher, and Nils Thuerey. Latent space physics: Towards learning the temporal evolution of fluid flow. In CGF, 2019. 6
Wenqi Xian, Jia-Bin Huang, Johannes Kopf, and Changil Kim. Space-time neural irradiance fields for free-viewpoint video. In CVPR, 2021. 6
Hong-Xing Yu, Leonidas J Guibas, and Jiajun Wu. Unsupervised discovery of object radiance fields. *ICLR*, 2022. 6
Hong-Xing Yu, Michelle Guo, Alireza Fathi, Yen-Yu Chang, Eric Ryan Chan, Ruohan Gao, Thomas Funkhouser, and Jiajun Wu. Learning object-centric neural scattering functions for free-viewpoint relighting and scene composition. TMLR, 2023. 6
Guangming Zang, Ramzi Idoughi, Congli Wang, Anthony Bennett, Jianguo Du, Scott Skeen, William L Roberts, Peter Wonka, and Wolfgang Heidrich. Tomofluid: Reconstructing dynamic fluid from sparse view videos. In CVPR, 2020. 6
Kai Zhang, Fujun Luan, Qianqian Wang, Kavita Bala, and Noah Snavely. Physg: Inverse rendering with spherical gaussians for physics-based material editing and relighting. In CVPR, 2021. 6
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In CVPR, 2018. 7
A Appendix: Implementation details
Model architecture. For both the density field  $\sigma (x,y,z,t)$  and the base velocity field  $\mathbf{u}_{\mathrm{base}}(x,y,z,t)$  we use a 4D extension of iNGP (Muller et al., 2022). Specifically, we add a temporal dimension to the original static iNGP. For the spatial dimensions, we use a base resolution 16 and the finest resolution 256. For the temporal dimension we set the finest resolution to 128 which is comparable to the number of video frames (120 frames), as higher resolutions create hash encoding features that are never directly supervised and can lead to unstable gradients.
Training and hyperparameters. Our training can be divided into three stages. In the first stage, we pre-train our density network  $\sigma(x,y,z,t)$  and the constant learnable appearance (radiance)  $L_{e}$  using the rendering loss only, which forms an initial estimate of both quantities. In the second stage, we jointly train the density  $\sigma(x,y,z,t)$ , appearance  $L_{e}$ , and the base velocity  $\mathbf{u}_{\mathrm{base}}(x,y,z,t)$  using the full loss. Finally, we jointly train the density  $\sigma(x,y,z,t)$ , appearance  $L_{e}$ , and the vortex particles  $\{I_p\}$ . We use an Adam optimizer with a learning rate 0.01. In the first stage, we train the density and radiance for 200,000 iterations. In the second stage, we jointly train the model for 50,000 iterations. In the third stage, we do training for 5,000 iterations. We empirically set the loss weights to  $\beta_{\mathrm{render}} = 10,000$ ,  $\beta_{\mathrm{density}} = 0.001$ ,  $\beta_{\mathrm{proj}} = 1$ ,  $\beta_{\mathrm{laminar}} = 10$ . For the laminar loss, we set the coefficient  $\gamma = 0.2$ .
Projection loss. For the projection routine, we use a multi-grid preconditioned conjugate gradient (MGPCG) solver (Ashby & Falgout, 1996). We implement it using Taichi (Hu et al., 2019) language. We use three levels of multi-grid, with the spatial resolution  $128^3$ . Note that we use the same resolution for our re-simulation and future prediction experiments for all compared methods.
Seeding strategy. In our experiments, we seed 50 vortex particles for each scene. To place initial vortex particles, we take the insight from vortex confinement methods (Fedkiw et al., 2001). We densely sample continuous spatio-temporal locations and compute curl values from the trained base velocity network, and then we place vortex particles at locations with the highest curl values.
Compute usage. We train our model on a single A100 GPU for around 9 hours in total. The first stage takes half an hour, the second stage takes around 8 hours (this is significantly slower than the first stage as it requires computing derivatives for the density loss), and the third stage takes half an hour.