Learning to Estimate Single-View Volumetric Flow Motions without 3D Supervision
Authors: Aleksandra Franz
Barbara Solenthaler
Nils Thuerey
Venue: ICLR 2023 poster (2023)

Abstract: We address the challenging problem of jointly inferring the 3D flow and volumetric densities moving in a fluid from a monocular input video with a deep neural network. Despite the complexity of this task, we show that it is possible to train the corresponding networks without requiring any 3D ground truth for training. In the absence of ground truth data we can train our model with observations from real-world capture setups instead of relying on synthetic reconstructions. We make this unsupervised training approach possible by first generating an initial prototype volume which is then moved and transported over time without the need for volumetric supervision. Our approach relies purely on image-based losses, an adversarial discriminator network, and regularization. Our method can estimate long-term sequences in a stable manner, while achieving closely matching targets for inputs such as rising smoke plumes.

Published as a conference paper at ICLR 2023
LEARNING TO ESTIMATE SINGLE- VIEW VOLUMETRIC FLOW MOTIONS WITHOUT 3D SUPERVISION
Aleksandra Franz
Barbara Solenthaler
Technical University of Munich (TUM)
ETH Zurich
franzer wtm.mmiiiinndddd..mmmmmmmmm
TUM Institute for Advanced Study
so 1 Mollenhaaler@inf ethz. ch
Nils Thuerey
Technical University of Munich (TUM)
ni mils. thue Phtuere@@tum.do
ABSTRACT
We address the challenging problem of jointly inferring the 3D flow and volumet- ric densities moving in a fluid from a monocular input video with a deep neural network. Despite the complexity of this task, we show that it possible train the corresponding networks without requiring any 3D ground truth for training. In the absence of ground truth data we can train our model with observations from real- world capture setups instead of relying synthetic reconstructions. We make this unsupervised training approach possible by first generating an initial proto- type volume which is then moved and transported over time without the need for volumetric supervision. Our approach relies purely on image-based losses, an adversarial discriminator network, and regularization. Our method estimate long-term sequences a stable manner, while achieving closely matching targets for inputs such rising smoke plumes.
INTRODUCTION
Estimating motion is fundamental problem, and is studied for variety of settings in two and three dimensions (Ranjan Black 2017 Hur & Roth 2021]] Gregson et al., 2014). It is also highly challenging problem, since the motion u is a secondary quantity that typically can't be measured
directly and has to be recovered from changes observed in transported markers p. We focus on volumetric, momentum-driven materials like fluids, where in contrast to the single-step estimation in optical flow (OF), motion estimation typically considers multiple coupled steps to achieve a sta- ble global transport. Furthermore, in this setting the volume distribution of markers p is usually unknown and needs to be reconstructed from the observations in parallel to the motion estimation.
So far, most research is focused the reconstruction of single scenes. Classic methods an opti- mization process working with an explicit volumetric representation (Fekert et al.) 2019 Zang et al., 2020 Franz al. 2021) while some more recent approaches optimize single scenes with neural fields (Mildenhall et al. 2020 Chu et al. 2022). As such an optimization is typically extremely costly, and has to be redone for each new scene, training a neural network infer estimate of the motion in a single pass is very appealing. Similar to most direct optimizations, existing neu- ral network methods rely on multiple input views to simplify the reconstruction (Qiu et al., (02T). However, this severely limits the settings in which inputs can be captured, as a fully calibrated lab environment often the only place where such input sequences can be recorded.
The flexibility of motion estimation from single views makes them highly attractive direction, and physical priors in the form governing equations make this possible in the context of fluids (Eckert et al., 2018 Franz et al. 2021). Nonetheless, despite using strong priors, the single viewpoint makes challenging to adequately handle the otherwise fully unconstrained depth dimension. We target deep learning- baased approach where neural network learns represent the underlying motion structures, such that almost instantaneous, single-past pass motion inference is made possible without relying ground truth motion data. The latter is especially important for complex volumetric

---

Published as a conference paper at ICLR 2023
motions, as reference motions of real fluids can not be acquired directly, Instead, one has work with reconstructions or even simulated data, suffering from a mismatch between the observations and the synthetic motion data. While obtaining multiple calibrated captures for training is feasible, using additional views only for losses results in issues with the depth ambiguity of single-view inputs.
In this work, we address the challenging problem of training neural networks to infer 3D motions from monocular videos in scenarios where 3D reference data is available. To the best of our knowledge, we are the first to propose an end-to-end approach, denoted by Neural Global Transport (NGT) in the following, which
(i) yields neural network to estimate global, dense 3D velocity from a single-view image sequence without requiring any 3D ground truth as targets. Among others, this is made possible by custom 2D-to-3D UNet architecture.
(ii) We address the resulting depth-ambiguity problem using a new approach with differentiable rendering and an adversarial technique.
iiii) A single network trained with the proposed approach generalizes across range of different inputs, vastly outperforming optimization- based approaches in terms of performance.
RELATED WORK
Flow estimation is of great interest in a multitude of settings, from 2D optical flow over scene flow to the capture and physically accurate reconstruction of volumetric fluid flows. Op- erating on pair of 2D images, optical flow estimates a motion that maps one to the other (Sun) et al. 2014). In this setting, multi-scale approaches in the form of spatial pyramids are longstand- ing technique to handle displacements of different scales (Glazer 1987). More recent CNN-based methods also employ spatial pyramids (Dosovitskiy et al.) 2015 Ranjan & Black 2017) and can learn to estimate optical flow an unsupervised fashion (Ahmadi Patras 2016
Optical flow
Yu et al. 2016
Luo et al. 2021
Scene flow Scene flow (Vedula et al. 1999) bridges the gap to 3D where earlier approaches using energy minimization or variational methods(Zhang & Kambhamettu 2001 Huguett & Devernay) 2007) have been surpassed by CNNs that bring superior runtime performance while retaining state- of-the-art accuracy (1lg et al., 2018 Saaxena al. 2019) and can also be trained without the need for ground truth data (Lee et al. [2119] Wang et al.
2019). Flow estimation from single input is of
particular importance as vastly simplifies the data acquisition and several methods for monocular
scene flow have been proposed (Brickwedde et al.
2019;
Yang & Ramanan 2020 Luo et al.) (2020).
These can be un- or self-supervised (Ranjan et al. multiple frames (Hur& & Roth 22021)
2019 Hur & Roth 2020 and benefit from using
Fluid flow Fluid flows are traditionally extremely difficult to capture and methods ranging from Schlieren imaging Dalziel et al. (2000); Atcheson et al. (2008) 2009) and particle imaging
locimetry (PIV) methods Grant 1997); Elsinga et al. (2006): : Xiong et al. (2017) over laser scan- ners Hawkins et al. 22055;; Fuchs al. 2007 to structured light Gu et al. 2013 and light path Ji] let al. (2013) approaches all require specialized setups. setups. The use of multiple commodity cameras sim- plifies the acquisition (Gregson al. 2014 Eckert al. 2019) and allows for view-interpolation to create additional constrains (Zang et al. 2020). Few works have attempted to solve monocular flow estimation in the fluids setting. For single-scene optimization Eckert al. (2018) constrain the motion along the view depth, while/Franz et al. (202T) use an adversarial approach to regularize the reconstruction from unseen views. [Quu et al. (2021) have proposed network that can estimate a long-term motion from single view, but still require 3D ground truth for training.
3D reconstruction In the context of fluids it is common to also reconstruct an explicit representa- tion of the transported quantities. Such 3D reconstruction is typically addressed for clearly visible surfaces (Musialski et al., 2013 Koutsoudis et al.] 2014) where some of the algorithms that have been proposed can incorporate deformations (Zang et al. 2018 Kato et al. 2018). In this context, 2013 et al. 2018), or more Lombardi et al. (2019) Sitzmann
volumetric reconstructions make use of voxel grids (Papon et al.
recently neural network representations Sitzmann et al. 2019a)
2

---

Published as a conference paper at ICLR 2023
et al. (2019b); Mildenhall et al. (2020). Learned approaches have likewise been used recover volumetric scene motions, c.g., for moving human characters (Mescheder et al. 2019 or to separate static and dynamic parts (Chu et al. 2022). For coupling with visual observations, several differ-
entiable rendering frameworks have been proposed (Kato et al. 2020 Zhang et al. 2021). These where used to constrain unseen views of static volumes (Henzler al. 2018 2019) and to recon- struct objects from single images as SDF (Jiang et al., (220). Considering fluids, corresponding approaches where used in the context of SPH|Schenck & Fox (2018), for the initialization of water wave simulator Hu et al. (2020), to realize style transfer onto 3D density field Kim et al. (2019a
2020), and for monocular flow reconstruction (Franz et 2021).
METHOD
In this work we focus on the task of estimating a global 3D motion u of volumetric quantity over time from sequence of single view images I'. To that end, train generator network Gu to estimate the motion u' between two consecutive time steps of and gt+1, In this context, a simple, supervised method would match the output reference velocity ú +++1 given a cost function such as an L2 loss:
min ||Gu(p', ( - -
(1)
However, as ground truth velocity targets are often not available, unsupervised methods instead leverage a model for the underlying transport process. Given differentiable discretization of the transport A we can learn the motion in an unsupervised fashion by requiring that p² should match g when A is applied:
min A(p'.9u(P -
(2)
As the image sequence í represents projections of volumes that leave the formulation of equation to be highly under-constrained we impose further priors to ensure the training converges to desir- able, unimodal solutions. Assuming that an initial state p is known, we define a multi-step transport as A' u) based loss from unsupervised 3D reconstruction (Zng et al. 2020 Franz et al. 202T). p.. This loss is realized via differentiable image formation model R such that the inferred volumetric state can be compared to the observed target image f'+1, In line with R, the network Gu is likewise conditioned on the target image ft+1, which yields the combined learning objective for our approach:
u²)............... u'-1),, and the target o [+] above is replaced by an image-
min R(A(pp ( (0 - -1-1
(3)
While this formulation has the central advantage that it yields a globally coupled, end-to-end pro- cess that does not require any volumetric targets, its under-constrained nature leads to very chal- lenging environment for learning tasks: In addition to the well-studied aperture problem of optical flow [eaachheii & Barron (1995), the 3D nature of our setting introduces the problem of depth- ambiguity: single density observed in pixel of f*+1 can be satisfied by an arbitrary distribution of densities along the corresponding viewing ray direction. In contrast to the aperture problem, sim- ple regularizers such as smoothness do not suffice to address depth ambiguity. Below, we explain our solution which uses an adversarial architecture.
3.1 DEPTH AMBIGUITY
For each target color along a ray, an infinite number of distributions of density along this ray exists, e.g. larger and further away vs. smaller and closer. The existence of multiple parallel observations, such as the pixels of an image, likewise does not guarantee uniqueness. One approach to address this issue is to refer to multiple observations in the loss during training, as would be available from a calibrated multi-view capture setup. However, since the network has no way to infer the unique density distributions pertaining to these side targets from its single input view, such approach typ- ically leads to learning an suboptimally averaged solution. Instead, we address this depth-ambiguity by expanding the simple target loss
2 =
R( p++
(4)
3

---

Published as a conference paper at ICLR 2023
Density training
U
G
p
e #
gul
R
R
Gu
p
DOWN
UP
Color legend.
R
Images
Gul
Density Generator
Gu
Density
Gp
Density Prototype
Velocity
DOWN
UP
Velocity Potential
gu
Gul
Input
Velocity training
Input t+1
Figure I: Left: An overview over the complete NGT framework. We generate an initial density volume p that is advected by the velocity form sequence. Density estimates are used in addition to the single input image to guide and stabilize the velocity generation. Velocity training is done end-to-end over the whole sequence. Right: Our multi-scale velocity estimator Gu. shown for resolution scales. The inputs contain information about the current (!) and next (I + 1) time step. Each scale generates a residual velocity potential which is used to advect the inputs of step / before generating the next residual. The final velocity is divergence free due using the curl Vx.
from equation where g²+l Alp', Gu(p', f4+20)) with the adversarial loss used by Henzler et al. (2019) and Franz et al. (22T)) to restrict to plausible appearance without imposing specific solution. Specifically, we follow previous work (Franz et al. 202T) and make use of a discriminator
D with an RaLSGAN objective (Jolicoeur- Jolicceur-Martineuu 2019)
(D(
Co(ptt11) Episca
E (w) D(R(p +1 (.)) .ww) -
(5)
(D(R(P+)) .)) Bpana D(1 + +1) 27
+ Eppiee (w)
where w are randomly sampled view directions and the label / is or -1 when training D or using it loss, respectively. To focus on compact volumes and prevent unnecessary scattering of densities along the viewing direction, we add a depth-Loss Lg, assuming that the observed densities are concentrated around position c₂ along the view's depth direction, the z-axis in our case. Thus, for every position we regularize depending on the distance to center position C₂
(6) where Ps is the projection of p on the primary view direction. As we choose c, be in the center of the grid, we normalize with half the grid resolution r. To further constrain and stabilize the estimated velocity we introduce prototype volume 5 generated from a single input image (see section 3.3) as 3D target for the transported density. This again takes the form of a simple Le loss:
C. = pp (p) - ?
La +1
3.2 ESTIMATING VOLUMETRIC MOTIONS
In addition to the current state p² and the target image f++1, as indicated by equation 3 our generator network also receives it, j² and øt+1 as input. The images projected into the volume using inverse ma-mmacching denoted by R-1, before being concatenated to the density fields. This way, the inputs trivially match the target and rendering losses, are coherent between frames, and the network can compare the back- pnojeprojections and prototype volumes to estimate the 3D velocity based on the current state /². Thus, the velocity estimation becomes
n' (p' R (f') R (f pz))
The architecture of G, further follows established multi-scale approaches of optical flow and scene flow (Ranjan & Black 20. as shown figure A residual velocity u' is generated at several res- olution scales with the per-resolution network component denoted as Gut- Our Gn further utilizes
4

---

Published as a conference paper at ICLR 2023
weight sharing between scales (Luo et al. 2021), such that self-similar structures be reused. In combination, velocity generation for a single time step takes the recursive form
uf = Gur(A(pp' @ R (1') @ p., 1i-1),R-1 (( R pu+1, ('i-1) +" ui-1
(9)
where @ is concatenation along the channel dimension, - is the up-sampled velocity of the previous scale, u/ = and final velocity uj = (..(...), t denotes up-sampling using quadratic B-spline kernels, and is down-sampling using average pooling. The inputs from the current time- step & are warped such that Fur only needs to estimate the residual velocity.
Even with the constraints of equations 4 to considerably large solution spaces exist that reach the target state, e.g., unphysical solutions that create mass. For this reason we include additional constraints magnitude, smoothness and divergence. For stability enforce the CFL condition on the residual of each velocity scale with an L2 loss on velocity magnitude, while smoothness is constrained for the combined velocity each scale. To respect mass conservation the velocity field has to be divergence free, i.e. V u = 0. Instead of adding a divergence- hard constraint (Kim et al.) (209hb) that is compatible with the residual
loss like most
previous work, multi-scale approach: we define as the curl (V x) of vector potential as u V x Gu- As the curl is a linear operator, V (sF + G) sV x F V x G, with F, G being vector fields and scalar, Equation 9 and the output of Gu are treated as a vector potential.
3.3 DENSITY ESTIMATION
Our motion estimation additionally hinges on a second network 9p. whose role is to estimate individual volumetric densities ý' from single image I':
i'
G, ()).
(10)
We employ this network for the prototype density loss 7 to infer the initial distribution of markers with p² := -0 = Gp( Φ), which was previously assumed be known. As the challenges of the single view estimation are similar for density and for velocity, using a generator network in the already established GAN setting is natural choice and
reuse the losses described above: Cg = Ci + CD(pp, -1) Lg. Gp employs a custom mixed 2D/3D UNet-architecture. The encoder processes 2D information, while the skip connections are realized via R R-1
3.4 IMPLEMENTATION AND TRAINING
We implement NGT with TensorFlow in Python. The R and A operations are custom: For R, used in both the target loss (equation 4 and LD.
differentiable volume ray-marching (Franz jet al., 2021), which supports attenuation, single-scattering, and the use of background images, but limited isometrically scattering densities and fixed lighting. For the advection operator A we employ a discrete, differentiable MacCormack advection scheme. First, we train G, alone on individual frames, where we grow the resolution from 8 to 64 during training. Then we freeze Gp and train Gu with up to 5 advection-steps. We again grow the resolution of the velocity estimation network, while the estimation of pd and the advection happen on the maximum resolution, using up-scaled velocities until Gn reaches the maximum resolution. 'D is trained during both density and velocity training. We refer appendix A for more details.
EVALUATION
We first present two ablations to illustrate the importance of handling depth ambiguity, and of the prototype density volumes. We then evaluate the method in comparison to series of learned and optimization-Be based methods for a synthetic and a real-world dataset.
4.1 DEPTH AMBIGUITY
In Figure we show the effect of depth ambiguity. If there is no depth variation, a network can easily recover the object when using multiple target views in the loss (column 2), leading to low volumetric and perceptual errors (table [. the object's position is randomized (column 3),
5

---

Published as a conference paper at ICLR 2023
Reference, fix
fx, multi, -D
Var, multi, -D
var, single, -D
var, multi, D
var, single, D
reference, var
reference. fix
fix, multi DD
var. mulhi -D
var. single. -D
yar. malti, T7
var. single. T
Reference var
In the top row versions closely match the input view while the side view (bottom) shows severe degradation when increasing depth ambiguity by varying the position the objects (fix - var) or using only the input view for Lj (multi - single). Providing additional views alongside the discriminator does not yield further improvements. DD by itself can recover a plausible configuration, even if the density does not exactly match the unknown reference location.
Figure 2: Depth-am happh-ambiguitty ablation study with the shape dataset:
the network can no longer settle on clear position as the added variation in depth not evident from the single input. Thus, the network is presented with multi- ple solutions for the same input over the course the training, severely deteriorating the qual- ity and drastically increasing the FID by 2.5 times. Simply re- moving the multiple targets im- proves the FID at the cost of LPIPS, volume error, and visual quality. Adding the discriminator to the multi-target setup im- proves the perceptual metrics, but visually the result still looks blurry. Keeping but removing the multiple targets further improves the perceptual quality, sharpening the results and improving FID to 68, the cost of volumetric accuracy. the end, the multiple target views hinder the discriminator and lead to averaged solutions that are favoured by simple metrics such as RMSE, but have lower perceptual scores. While discriminator can not recover the true shape and depth location of sample, indicated by the relatively high volume error, the compact density it produces nonetheless represents a very good starting point for the motion estimation, given the single view input.
Table 1: Quantitative evaluation of the depth-ambiguity issue as seen in figure 2 Side uses 7 evenly distributed views. Lower
values are better for all metrics under consideration.
Input RMSE
Side
Version
RMSE
LPIPS
FID
RMSE
fix, multi, D
.0080
.0106
.0200
48
.260
var, multi, D
.0160
.0509
.1237
120
.636
var, single, DD
.0042
.0681
.1345
98
.69
var, multi, D
.0151
.0525
.1239
80
.642
var, single,
.0064
.0694
.1140
68
.930
4.2 PROTOTYPE VOLUMES
Using the prototype densities ğ as explicit 3D constraint in the form of Ls may at first seem counter- intuitive in the presence of the depth ambiguity. However, as the á density volumes are generated from single images, there is only single á associated with each input. This yields reduction of ambiguity for the task 9n when using the prototypes as loss. Furthermore, we provide g++1 as additional input to G.. to simplify the task of motion inference. Overall, this highlights the importance and central role of the novel 2D-to-3D UNet architecture. The addition of P to the velocity training stabilizes the inference of long sequences, evident from figure 3 and table 3 While there not much difference in the early frames, the network lacking the stabilization clearly diverges after evaluating 120 frames.
4.3 RESULTS
We evaluate our method on both synthetic smoke flows and the real-world captures from the ScalarFlow dataset (Eckert et al. 2019). As representative of simpler network architectures, i.e. without multi-scale handling and the 2D-to-3D UNet, we combine the architecture from Qiu et al. (021) with our loss formulation. This combination is denoted by (RapidGen). We further compare to the direct optimization algorithms of Eckert et al. (0099) (ScalarFlow) and Franz et al. (2021) (GlobTrans). These can achieve better target matching and long-term transport because each scene is optimized over the full trajectory individually, but this comes at the cost vastly increased (more than x x000) reconstruction times. Due the large space of solutions for the single view problem in

---

Published as a conference paper at ICLR 2023
Frame 40, 0%, mp
Frame 40, 90", mp
Frame 40, 0"
Frame 40, 90" p
Frame 120, D", /
Frame 120, 90",
P
Frame 120, 0", P
Frame 120, 90" f
Figure 3: Adding the prototype volume أ produced by 2D-to-3D UNet guidance (right side) stabilizes the inference over long evaluations and results in better target matching and a better overall density distribution, especially from unseen views. Metries be found in table 3
RapidGen
SealarFlow
GlobTrans
Reference
Figure 4: Qualitative comparison between different approaches using synthetic plume data for time- step 80. Top is the input view which all method match fairly well. Bottom is 90" side view where the shortcomings of the different approaches become visible. Due the overshoot, ScalarFlow
densities are shown with factor or 1/2.
a setting as chaotic as fluid dynamics, we evaluate the general appearance of the estimations with the perceptual metric LPIPS (Zhang et al.) 2018) and the Fréchet Inception Distance (FID) between random views of the estimated sequences and random samples of input images to obtain a measure of likeness in appearance. This allows
how realistic the overall look of the plumes is.
4.3.1 SYNTHETIC PLUMES
We first evaluate our method with synthetic data set consisting of simulated plumes of hot marker densities in front of a black background. While the flows are relatively simple compared to their real-world counterparts, the smooth density content makes it very difficult to estimate motions as the differentiable advection relies spatial gradients in the density for back-propagation. For com- pleteness, we measure the accuracy of reconstructing the input image column one of table 2 for which all methods show a low еπтоr. Our goal is to estimate plausible solutions, rather than exact match to a reference, as the ambiguity depth allows for many solutions that match the available inputs. This can be seen particularly well in the volumetric comparisons, where result has
7

---

Published as a conference paper at ICLR 2023
Table 2: Quantitative evaluation of different methods trained synthetic plumes dataset.
Input RMSE
Side RMSE
Random
U
Algorithm
LPIPS
FID
FID
RMSE
RMSE
EPE
RapidGen
.0145
.0277
.0450
105
76
1.65
.494
.851
Ours
.0156
.0243
.0444
101
.431
2.06
3.29
ScalarFlow
.166
.120
.188
135
106
1.39
.120
.104
GlobTrans
.0122
.0440
.0877
87
30
.373
.143
.137
RapidGien
Ours
ScalarFlow
Glob Trans
Reference
input view
side view 300"
Figure 5: Qualitative comparison between different approaches using ScalarFlow data for time-step 100. Our method closely matches the given input and has clearly defined shape that matches the general shape of the reference. It is only surpassed by the costly single-scene reconstruction method GlobTrans. RapidGen was adapted to be trained without 3D GT.
good match with the reference density with RMSE of 0.431, but disagrees in terms of underlying motion, leading to a high difference the velocities. The corresponding videos still show calm, swirling motion similar to the reference. A visual comparison is shown in figure 4
Comparing the unseen side targets reveals the differences between the approaches, especially in conjunction with the estimated motions, which be seen in the supplemental videos. GlobTrans performs best in terms of the perceptual metrics, reaching a low FID of 30 due to having the same large smooth structures as the reference. However, GlobTrans exhibits unnatural, excessive motion in the side view. RapidGen is closer in appearance to the reference, but lacks a clearly defined plume shape leading to a density error 3.8 times higher than ours. And even though the velocity of RapidGen matches the reference better, the videos show that it consists mainly of a static upwards motion. ScalarFlow exhibits smooth and natural motions but suffers from missing depth regular- ization, resulting in excessive amount of density. These results highlight the complexity of the single-view motion estimation and the multitude of viable solutions when given only single image for synthetic inputs, and illustrate the behavior of the different learned and optimization-bal based methods. Next, we evaluate their performance with real-worke inputs.
Table 3: Quantitative evaluation of different methods trained on the real world ScalarFlow dataset.
Lower values better for metries under consideration.
Input RMSE
Side
Random
Inference time
Algorithm
RMSE
LPIPS
FID
FID
120 steps
RapidGen QQu Ours w/o P
2021
.0227
.0405
.173
186
163
3m
.0589
.0600
.191
170
123
4m
Ours
.0229
.0333
.116
134
102
4m
ScalarFlow
(Eckrtt et al.
2019
.2544
.4488
.152
153
119
6.5h
GlobTran (Franz et al.
2021)
.0124
.0281
.085
102
77
37.5h
8

---

Published as a conference paper at ICLR 2023
RapidGien
Ours
SealarFlow
GlanTrans
from 90
from
Figure 6: Visualization of density distributions of different approaches using ScalarFlow data for time-step 100. Our method shows similar structures as GlobTrans while being slightly less detailed. The single-scale RapidGan network fails to replicate the intricate 3D structures of real world data.
4.3.2 REAL-WORDD CAPTURES
The real-world data of the ScalarFlow dataset introduces the challenges of fine structures, sensor noise, and real-world backgrounds. It also does not provide measured 3D ground truth for compar- isons, but the dataset still provides 5 calibrated views. With the center being used as input, use the remaining 4 to quantitatively evaluate the estimations from unseen views. Again, the input reconstruction in table good for all methods under consideration.
For the unseen side targets, our method with an FID of 134 now clearly outperforms RapidGen with 186, well the expensive ScalarFlow optimization (153). Only the GlobTrans optimization manages to do better in terms of this metric. We again evaluate the FID between random views of the estimated sequences and random samples of the dataset which clearly detects the lack of features in the RapidGen result. It also shows that structure-wist our method surpasses ScalarFlow, which takes ca. times longer to produce single output. While the GlobTrans optimization fares even better, it does so at the expense of a more than 560 times longer runtime. These measurements show that our trained network successfully generalizes to new inputs, and yields motion estimates that are on-par in terms of realism with classic, single-case optimization methods.
CONCLUSION
We presented Neural Global Transport (NGT), new method to estimate a realistic density and velocity sequence from new single view sequences with single pass of neural network evaluations. Our method and implementation currently have several limitations: e.g., currently only supports white smoke, while anisotropic scattering, self-illuminton, or the use of other materials would require an extension of the rendering model. Also, our transport model relies on advection without obstructions. Here, our method could be extended to support obstacles, multi-phase flows. For even better long-term stability, the method could be extended to pass back information about future time steps for improved guidance.
Nonetheless, we have demonstrated that our networks can be trained with only single view data and short time horizons while still being stable for long sequences during inference. We solved the depth ambiguity arising from the single-view setting by using an adversarial loss, and we stabilized the velocity estimation by providing estimated prototype volumes both as input and soft constraint. The resulting global trajectories of NGT are qualitatively and quantitatively competitive with single- scene optimization methods which require orders of magnitude longer runtimes.

---

Published as a conference paper at ICLR 2023
REPRODUCIBILITY STATEMENT
Our
source
code is publicly available at
Neural- G1 ooallbaa-oooo Transport and includes the data and configurations necessary re- produce all results of the paper. Further details about network architectures, training procedures and hyperparameters can also be found in the appendix.
ACKNOWLEDGMENTS
This work was supported by the Siemens/IAS Fellowship Digital Twin, and the DFG Research Unit TH 2034/2-1.
REFERENCES
Aria Ahmadi and loannis Patras. Unsupervised convolutional neural networks for motion estimation. In 2016 IEEE International Conference Image Processing (ICIP), pp. 1629-1633, 2016. doi: N0.1109//CCP
Bradley Atcheson, Ivo Ihrke, Wolfgang Heidrich, Art Tevs, Derek Bradley. Marcus Magnor, and Hans-Peter Seidel. Time-resolved capture of non-stationary gas flows. ACM Trans. Graph., 27 (5), December 2008. ISSN 0730-0301. doi: 10. 14455/1400000114000000 URL https://ddi. 2xg/10.1145/1400
1409085
Bradley Atcheson, Wolfgang Heidrich, and Ivo Ihrke. An evaluation of optical flow algorithms for background oriented schlieren imaging. Experiments in Fluids, 4663):467-476, Mar 2009. ISSN 1432-1114. doi:
00000/000038000-0577777
S. S. Beauchemin and J. L. Barron. The computation of optical flow. ACM Compul. Surv. 27(3): 433 466, sep 1995. ISSN 0360-0300.
doi: 10.1145/212044.122224 URL https://dok:
org 11......11111111 21..1144/22299.2221111
Fabian Brickwedde, Steffen Abraham, and Rudolf Mester. Mono-sf: Multi-view geometry meets
single-view depth for monocular scene flow estimation of dynamic traffic scenes. Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), October 2019.
Jumyung Chang, Vinicius C. Azevedo, and Christopher Batty, Curl-flow: Pointwise incompressible velocity interpolation forgrid-basd based fluids, 2021. URL 00867
Mengyu Chu, Lingjie Liu, Quan Zheng, Aleksandra Franz, Hans-Peter Seidel, Christian Theobalt, and Rhaleb Zayer. Physics informed neural fields for smoke reconstruction with sparse data. ACM Trans. Graph., 41),), jul 2022. ISSN 0730-0301. doi: 10.1145//522233333066. URL
https:///oi... http///....r....1111.1
3530166
S.B. Dalziel, G.O. Hughes, and B.R. Sutherland. Whole-Field Density Measurements by 'Synthetic Schlieren'. Experiments in Fluids, 28:322-335, 2000.
Alexey Dosovitskiy, Philipp Fischery, Eddy Ilg, Caner Hazirbas, Vladimir Golkov, Patrick van der Smagt, Daniel Cremers, Thomas Brox, al. Flownet: Learning optical flow with convolutional networks. In International Conference on Computer Vision (ICCV), pp. 2758-2766. IEEE, 2015.
M.-L. Eckert, W. Heidrich, and N. Thuerey. Coupled Fluid Density and Motion from Single Views. Computer Graphics Forum, 37(8):47-58. 2018. ISSN 01677055, doi: 10.1111/cgf.1355 13511. URL
hittp://dot wiley. com/ 10. 11t.1//gggf..1111
Marie-Lena Eckert, Kiwon Um, and Nils Thuerey. Scalarflow: A large-scale volumetric data set of real-world scalar transport flows for computer animation and machine learning. 38(6), nov 2019.
ISSN 0730-0301. doi: 10. 10.1145/3355089333555
URL hatpp::/ http://///ii.or/1..1111111
3355089. 3356545
Gerrit E Elsinga, Fulvio Scarano, Bernhard Wieneke, and Bas van Oudheusden. Tomographic particle image velocimetry. Experiments in fluids, 41(6)9933-947, 2006.
10

---

Published as a conference paper at ICLR 2023
Aleksandra Franz, Barbara Solenthaler, and Nils Thuerey. Global transport for fluid reconstruction with learned self-supervision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1632-1642, June 2021.
Christian Fuchs, Tongbo Chen, Michael Goesele, Holger Theisel, and Hans- Peter Seidel. Density estimation for dynamic volumes. Computers & Graphics, 31(2):205-21 2007.
Frank C. Glazer. Hierarchical motion detection. Technical report, USA, 1987.
I. Grant. Particle Image Velocimetry: a Review. Proceedings of the Institution of Mechanical Engineers, 211(():55-7,, 1997.
James Gregson, Ivo Ihrke, Nils Thuerey, and Wolfgang Heidrich. From capture to simulation: Con- necting forward and inverse problems in fluids. ACM Trans, Graph, 33(4), jul 2014. ISSN 0730- 0301. doi: 10. 00.1145/2200199222611477 26C1147
URL https: https::/ddoi.ooo 14tp://00...rr111.11141.445///50
Jinwei Gu, S.K. Nayar, E. Grinspun, P.N. Belhumeur, and R. Ramamoorthi. Compressive Structured Light for Recovering Inhomogeneous Participating Media. IEEE Trans. Puttern Analysis and Mach. Int., 35(3):555-500 2013.
Tim Hawkins, Per Einarsson, and Paul Debevec. Acquísition of time-varying participating media. ACM Trans. Gruph., 24(3):812-81) 2005.
Philipp Henzler, Niloy J. Mitra, and Tobias Ritschel. Escaping Plato's Cave: Shape From Adver- sarial Rendering. In Proceedings of the IEEE International Conference Computer Vision, pp-
2019. URL http:::://ppaacesss.. thecvf..e.c.ccc com/ content ICCV 2019/ html html/Henzler Escaping Platos Cave 3D Shape From Adversarial Rendering ICCV _eCCc__2011100 Bttps__apeee____
Phlipp Henzler, Volker Rasche, Timo Ropinski, and Tobias Ritschel. Single-image Tomography: 3D Volumes from 2D Cranial X-Rays. Computer Graphics Forum, 37(2):377-388, 2018. ISSN 1467-8659. doi: 10.1111/cgf.1331 13369. htt.ttts::/ httt:s::/oonnneeleereee htwp://wwwwwwwwl....e./oooooaa com/do1//
wi
13369
Yuanming Hu, Luke Anderson, Tzu-Mao Li, Qi Sun, Nathan Carr, Jonathan Bagan-Kelley, and Fredo Durand. Differentiable programming for physical simulation. In International Confer- Learning Representations, 2020. https:/// /ttps/////p/eeeeieeeeee net http::/wweeeeteeeeeeeeeee forum?id=
BleB5x8Fvr
Frederic Huguet and Frederie Devernay. A variational method for scene flow estimation from stereo sequences. In 2007 IEEE I Ith International Conference on Computer Vision, pp. 1-7, 2007. doi: 10.1 10.1109/CCC.22007.440 4409000..
Junhwa Hur and Stefan Roth. Self-supervised monocular scene flow estimation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), June 2020.
Junhwa Hur and Stefan Roth. Self-supervised multi-frame monocular scene flow. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2684 2694, June 2021.
Eddy 1lg, Tonmoy Saikia, Margret Keuper, and Thomas Brox. Occlusions, motion and depth bound- aries with a generic network for disparity, optical flow or scene flow estimation. In Proceedings of the European Conference Computer Vision (ECCV), September 2018.
Yu Ji, Jinwei Ye, and Jingyi Yu. Reconstructing gas flows using light-pati approximation. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), June 2013.
Yue Jiang, Dantong Ji, Zhizhong Han, and Matthias Zwicker. Sdfdiff: Differentiable rendering of signed distance fields for 3d shape optimization. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.

---

Published as a conference paper at ICLR 2023
Alexia
The relativistic discriminator: key element missing from stan- dard GAN. In International Conference on Learning Representations, 2019. URL https:
Hiroharu Kato, Yoshitaka Ushiku, and Tatsuya Harada. Neural 3D Mesh Renderer. In Pro- ceedings of the IEEE Conference Computer Vision and Pattem Recognition, pp. 3907- 3916, 2018. URL
thecff. com/ http//nnttttttteeeeeeeeeeeee _epp 2018/htm1/ Kato_ 3D @resh_CVPR___ 2018 paper.pp..... paper.htm. httml
Hiroharu Kato, Deniz Beker, Mihai Morariu, Takahiro Ando, Toru Matsuoka, Wadim Kehl, and Adrien Gaidon. Differentiable rendering: A survey, 2020. URL
2006.12057
Byungsoo Kim, Vinicius C. Azevedo, Markus Gross, and Barbara Solenthaler. Transport-based neu- ral style transfer for smoke simulations. ACM Transactions on Graphics, 38(6):1-11, nov 2019a. ISSN 07300301. doi: 10. 14/331555099.335566665 URL http: //ll//ll./// att./aalac..cccccmmmmmaa ac...a///.............
c cfm:doid=335500 3356560
Byungsoo Kim, Vinicius C Azevedo, Nils Thuerey, Theodore Kim, Markus Gross, and Barbara Solenthaler. Deep fluids: A generative network for parameterized fluid simulations. In Computer graphics forum, volume 38, pp. 59-70, 2019b.
Byungsoo Kim, Vinicius C. Azevedo, Markus Gross, and Barbara Solenthaler. Lagrangian neural style transfer for fluids. ACM Transactions on Graphics, 39(4), July 2020. ISSN 0730-0301. doi: 00,1145/3386599.33944111 URL httpps://doiii http///0ooo/..oo/oo/oo////o 1145/ 01055/33866633 3392473
Anestis Koutsoudis, Blaž Vidmar, George loannakis, Fotis Amnoouooglou, George Pavlidis, and Christodoulos Chamzas. Multi-image 3d reconstruction data evaluation. Journal of Cultural Heritage, 15(1):73-79,2)) 2014.
Seokju Lee, Sunghoo Im, Stephen Lin, and In So Kweon. Learning residual flow as dynamic motion from stereo videos. In 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 1880-186, 2019. doi: 10.1 00.1009//00044897.219998292 2977777....7.7.....
Stephen Lombardi, Tomas Simon, Jason Saragih, Gabriel Schwartz, Andreas Lehrmann, and Yaser Sheikh. Neural Volumes: Learning Dynamic Renderable Volumes from Images. ACM Transac- tions on Graphics, 38(4):1-14, 2019. ISSN 07300301. doi: 10. 10.144//330646.33322000
URL
http: http:///xxvvv.rr..eo wttp://.rrrgbbs//90000 07751
Chenxu Luo, Zhenheng Yang, Peng Wang, Yang Wang, Wei Xu, Ram Nevatia, and Alan Yuille. Every pixel counts ++: Joint learning of geometry and motion with 3d holistic understanding. IEEE Transactions on Pattern Analysis and Machine Intelligence, 42((0))2624-266. 2020. doi: N0.11109/PAMMI.219.22222 293202...
Kunming Luo, Chuan Wang, Shuaicheng Liu, Haoqiang Fan, Jue Wang, and Jian Sun. Upflow: Upsampling pyramid for unsupervised optical flow learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1055-1054, June 2021.
Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. Occupancy networks: Learning 3d reconstruction in function space. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1460-4470, 2019.
Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes neural radiance fields for view synthesis. In ECCV, 2020.
Gyeongsik Moon, Ju Yong Chang, and Kyoung Mu Lee. V2v-posenet: Voxel-to-voxel prediction network for accurate 3d hand and human pose estimation from single depth map. In Proceedings of the IEEE conference on computer vision and pattern Recognition, pp. 5079-5088, 2018.
Przemyslaw Musialski, Peter Wonka, Daniel G Aliaga, Michael Wimmer, Luc Van Gool, and Werner Purgathofer. A survey of urban reconstruction. In Computer graphics forum, volume 32(6), pp- 177-177. Wiley Online Library, 2013.
12

---

Published as a conference paper at ICLR 2023
Jeremie Papon, Alexey Abramov, Markus Schoeler, and Florentin Worgotter. Voxel cloud con- nectivity segmentation-supertt supervoxels for point clouds. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2027-2034 2013.
Sheng Qiu, Chen Li, Changbo Wang, and Hong Qin. A rapid, end-to-end, generative model for gaseous phenomena from limited views. Computer Graphics Forum, 40(6):242 257, 2021. doi: 1/egf. 1//ft/ 14270
https://doi.org/100.111
URL https:://omlinellbraayyy com.://ooo..eee...cooooo///o.o com/doi/
Anurag Ranjan and Michael J. Black. Optical flow estimation using a spatial pyramid network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017.
Anurag Ranjan, Varun Jampani, Lukas Balles, Kihwan Kim, Deqing Sun, Jonas Wulff, and Michael J. Black. Competitive collaboration: Joint unsupervised learning of depth, camera mo- tion, optical flow and motion segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
Rohan Saxena, René Schuster, Oliver Wasenmuller, and Didier Stricker. Pwoc-3d: Deep occlusion- aware end-to-end scene flow estimation. In 2019 IEEE Intelligent Vehicles Symposium (IV), pp. 324-3 331, 2019. doi: 10.1 109//19/VS..... 2019.8814146
Connor Schenck and Dieter Fox. Spnets: Differentiable fluid dynamics for deep neural networks. In Conference on Robot Learning. volume 87 of Proceedings of Machine Learning Research, 2018.
Andrew Selle, Ronald Fedkiw, MyungMoon Kim, Yingjie Liu, and Jarek Rossignac. An Uncon- ditionally Stable MacCormack Method. Journal of Scientific Computing, 35(2):350-377 2008. ISSN 1573-7691. doi: 10. (00007//0011-007-991166 s
URL http::ttp:://cot org 1ttp://0011...1111100010000
Vincent Sitzmann, Justus Thies, Felix Heide, Matthias Nießner, Gordon Wetzstein, and Michael Zollhofer. Deepvoxels: Learning persistent 3d feature embeddings. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pp. 2437-2446, 2019a.
Vincent Sitzmann, Michael Zollhöfer, and Gordon Wetzstein, Scene representation networks: Con- tinuous 3d-structure-avy aware neural scene representations. In Advances in Neural Information Pro- cessing Systems, pp. 1121-1132, 2019b.
Deqing Sun, Stefan Roth, and Michael Black. A Quantitative Analysis of Current Practices in Optical Flow Estimation and the Principles behind Them. International Journal of Computer Vision, 106(2):155-337, 2014.
Sundar Vedula, Simon Baker, Peter Rander, Robert Collins, and Takeo Kanade. Three-dimensional scene flow. In Proceedings of Computer Vision, volume 2, pp. 722-729. IEEE, 1999.
Yang Wang, Peng Wang, Zhenheng Yang, Chenxu Luo, Yi Yang, and Wei Xu. Unos: Unified unsupervised optical-flow and stereo-depth estimation by watching videos. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), June 2019.
Jinhui Xiong, Ramzi Idoughi, Andres A Aguirre-Pablo, AbduIrahman B Aljedaani, Xiong Dun, Qiang Fu, Sigurdur Thoroddsen, and Wolfgang Heidrich. Rainbow particle imaging velocime- try for dense 3d fluid velocity imaging. ACM Trans. Graph., 36(4):36, 2017.
Jingjing Xu, Xu Sun, Zhiyuan Zhang, Guangxiang Zhao, and Junyang Lin. Understanding and Improving Layer Normalization. Curran Associates Inc., Red Hook, NY, USA, 2019.
Gengshan Yang and Deva Ramanan. Upgrading optical flow to 3d scene flow through optical expan- sion. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), June 2020.
Jason J. Yu, Adam W. Harley, and Konstantinos G. Derpanis. Back to basics: Unsupervised learning of optical flow via brightness constancy and motion smoothness. In Gang Hua and Hervé Jégou (eds.), Computer Vision - ECCV 2016 Workshops, pp. 3-10, Cham, 2016. Springer International Publishing. ISBN 978 978-3-319-49409-88
13

---

Published as a conference paper at ICLR 2023
Guangming Zang, Ramzi Idoughi, Ran Tao, Gilles Lubineau, Peter Wonka, and Wolfgang Heidrich. Space-time tomography for continuously deforming objects. ACM Trans. Craph., 37(4). July 2018. ISSN 0730-0301. 7145/319751 9
doi: 10.1145/199311732211988 URL https::/dol: 10.
Guangming Zang, Ramzi Idoughi, Congli Wang, Anthony Bennett, Jianguo Du, Scott Skeen, William L. Roberts, Peter Wonka, and Wolfgang Heidrich. Tomofluid: Reconstructing dynamic fluid from sparse view videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
Cheng Zhang, Zihan Yu, and Shuang Zhao. Path-space differentiable rendering of participating media. ACM Trans. Graph., 40(4), jul 2021. ISSN 0730-0301. doi: 10.1145734500663459988. URL
011.2///22..144455555555555 3459782
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 586-595, 2018.
Ye Zhang and C. Kambhamettu. On 3d scene flow and structure estimation. In Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001, volume 2, pp. II-II, 2001. doi: 10. 109/ .910444.
14

---

Published as a conference paper at ICLR 2023
APPENDIX
A TECHNICAL DETAILS
A.1 NETWORKS
The full density generator 9p is detailed in figure the velocity generator part Guz consists only of the 6 layers which are shared between all scales (see also figure D Like Gp. it uses kernel size of and relu activation, except for the imput and output convolutions which have a kernel size of 1. Any ResBlocks that change the number of filters from their input have a kernel size convolution on their skip path. If the number of filters stays the same this convolution omitted. We use the simple Layer Normalization of Xu et al. (2019) after every convolution except the last in network.
Discriminator We reuse the discriminator of Franz et al. (022)) which is simple stack of con- volutional layers. However, we remove the crop and rotation the data augmentation allow the discriminator to see the whole domain and consider the spatial orientation of observed features.
A.2
LOSSES
CFL magnitude loss For increased stability during advection we enforce the CFL condition on the velocity by using magnitude loss that only affects velocity components larger than 1:
Can. max (u? 1,0),
(11)
where i are the x-, and z-components of u.
Smoothness loss
The first order smoothness loss is also defined per component:
Ssmooth =
Σ
3
(
Ou: Ox
2 +
Ou,
2
+
,u;
2
(12)
ay
Oz
where the spatial derivative approximated with finite differencing practice.
128
64
R
64
NN
R
NN
167
R8:32
R
16a
NN
82
R
8P
Gul
R
422
Figure 7: Our full density estimator network G,. Lower left: the velocity generator part Gui that reused for each scale. C is a convolution, RB ResBlock, NN nearest neighbour sampling. A trapezoid shape indicate stride of 2. Strided convolutions use filter size of 4, the rest 5, all use ReLU activation. The number of filters and spatial resolutions are given in the image.
15

---

Published as a conference paper at ICLR 2023
A.3 TRAINING
Our method is implemented in TensorFlow version 1.12 under python version 3.6 and trained on Nvidia GeForce GTX 1080 Ti 1IGB. For training we use the Adam optimizer. For leaning rate decay we use the following formula:
learning.rate
(13)
1 1+ (iteration - offset]decay
Data Our synthetic dataset consists 30 randomized simulations buoyant smoke plumes, sim- ulated with Mantaflow at resolution of 192x (2x higher than training resolution) for 150 time steps. We omit the first 45 frames as startup and train with the remaining 105 frames. The volumes are rendered with the same renderer we use during training to obtain the input images.
For the real world data we use the raw captures of first 30 scenes of the SalarFlow dataset as input images and omit the first 20 frames to ensure that there is enough density visible, leaving 130 frames per scene. We downsample the images by a factor of 10 to 192x108 to approximately match the projected grid resolution.
Each Network and method has been trained specifically for the corresponding dataset.
Density training Gp is trained with Lp = Cj 2e-4Lp + 1e-3C, and learning rate 2e-4 with decay of 2e-4, the decay offset by -5000 iterations. We start a grid resolution of 8x12xx 12x8 with 2 UNet levels. The resolution grows after 8k, 16k and 24k iterations by a factor of 2, adding a level of the UNet every time, thus reaching maximum grid resolution of 64x96x64 with levels. New levels are faded in over 3k iterations, starting 2k iterations after growth, by linearly interpolating between the up-sampled previous level and the current level. After fade-in only the output of the highest active level remains. The image resolution grows in conjunction with the density.
Velocity training For Tu we use L, Cj + 1e-3C; + 2e-6Cp + 1e-3Cc + 0.1 + 42 with learning rate of 4e-4 with the same decay of 2e-4, offset by -5000 iterations. The velocity training also starts at a grid resolution of 8x 128 with 2 levels of Gul and with only advection step (i.e. frames). The density estimation and advection always happen at the maximum resolution of 64x96x64 and the velocity is up-scaled accordingly before it has reached this resolu- tion. Before increasing the resolution, we extend the sequence, first to frames after 1500 iterations and then to the final 5 frames after 3k iterations. Then we grow the resolution of the velocity esti- mation after 4k, 8k and 12k iterations, each time adding a new level of Gut- The residual of a newly added velocity level is linearly faded in over 1500 iterations, starting 500 iterations after growth. A full training run of our model takes 6 days on average using a single GTX 1080 Ti GPU.
A.4 RENDERING
We use the differentiable volumetric ray-marcher of Franz et al. 2021 for our method which dis- cretizes the rendering equation
''' -[x)e-Ja pla)da dr,
R(p, L,
(14)
where L gives the lighting at point I and are camera parameters resulting in pixel rays from n to f. by stepping along the ray to solve the integrals. The single- scatterd lighting is realized by solving the rendering equation in the same way and storing intermediate step values to create the lighting volume L. A background image in can be added by attenuating it with the total density along the ray, thus the rendering becomes
Rsp I., T.,c)= = S. T.(r)e Saoee-Sr- SA pla)da dr - 1 pla)da in-
(15)
We modify the renderer to only render inside of the cell center of the outer cell-layer instead of their outer border to avoid border artifacts at the inflow, which are likely caused by the interpolation to outside the domain.
16

---

Published as a conference paper at ICLR 2023
A.5 PROJECTION INTO THE VOLUME
The operation R transfers an image or 2D tensor into the volume by evenly distributing it along the pixel rays into the volume from given view. This is achieved by inverting the ray-marching procedure. While marching along the ray the current pixel value is distributed to the 8 voxels around the current position, weighted by the distance to the voxel. The weights are the same one would use for linear interpolation at that point. Afterwards, the accumulated voxel-values are normalized with the number of contributions per voxel to obtain a smooth result. This procedure is also used for the gradient backwards pass of R.
A.6 ADVECTION
The MacCormack advection (Selle et al. 2008) is a second order transport scheme using Semi- Langrangian advection its base, which itself is implemented backwards-lookup using the velocity vectors and interpolating the transported field at the lookup location. With Ast. as the Semi-Langrangian base, MacCormac is defined
p++l = Asi(p', u')
8r = , -u²)
(16)
p 4+ = p + 0.5(p' - p
To avoid extreme values the results are limited be within the range of the interpolants encountered in the first Semi-Langrangian advection.
Boundary and inflow handling All boundaries are open, meaning that advections that transport density from outside the domain values from the nearest boundary layer. Inflow of densities thereby automatically handled by the open boundaries which works well our application but theoretically limited as initial density at the boundary layer is needed to have any inflow and the inflow can never exceed the already existing density values due to the linear interpolation in the advection.
Vector potentials and curl Generating vector potentials and defining the velocity as their curl, as we do in our velocity estimation, can lead to discontinuous velocity fields if linear interpolation is used to interpolate vector potential fields between resolutions (Chang et al. (021). Since the curl based on spatial derivatives, and the derivatives of linear interpolation are not continuous, the resulting velocity can have discontinuities which can lead to grid aligned artifacts visible in the results. To circumvent this issue we use the higher order quadratic B-splices as interpolation kernels to transfer vector potentials from coarse finer resolutions of our multi-scale network. This leads to noticeably smoother velocity, as can be seen from the advected densities in figure g
B ADDTIONAL RESULTS
B. ABLATIONS
We run additional ablations for the losses Camooth and LCH when training Gu- The corresponding metrics are in table
B.2 DENSITY PROTOTYPES
We visualize some of the density prototypes " for the ScalarFlow dataset in figure 9 These are intended rough volumetric guidance via L, and not seen as the real solution.
B.3 SCALARFLOW DATA
Additional results of our method evaluated on ScalarFlow data can be found in figures T0 and as well in the supplemental videos.
17

---

Published as a conference paper at ICLR 2023
Figure 8: Left: the grid-aligned artifacts are caused by discontinuous velocities. Right: higher order interpolation of the potentials forces the velocities to be smoother, resulting in smoother densities after advection.
Table Quantitative ablation of different regularization terms when training 9n, on the real world ScalarFlow dataset. The training diverges without any regularization (first row), leading to empty volumes. Without magnitude regularization (CCFL, second row) the training itself is stable, but the evaluation produces velocities that quickly remove all density from the volume after few time steps, again leaving empty volumes. Lower values are better for all metrics under consideration.
Input RMSE
Side RMSE
Random FID
Algorithm
Lsmooth
LCEL C2
LPIPS
FID
Ours
Ours
Ours
v
.0227
.0341
.117
136
84
Ours
V
.0589
.0670
.191
170
123
Ours
.0229
.0333
.116
134
102
18

---

Published as a conference paper at ICLR 2023
SUP
Figure 9: The density prototypes أ of ScalarFlow scene 80 used in the volumetric guidance Cp when training Gu- These are less detailed then the densities resulting from advection and show some artifacts around the border of the plume.
19

---

Published as a conference paper at ICLR 2023
91"
Figure 10: Our final model evaluated on ScalarFlow scene 80.
20

---

Published as a conference paper at ICLR 2023
990
Figure 11: Our final model evaluated on ScalarFlow scene 81.
21

---

Published as a conference paper at ICLR 2023
90"
Figure 12: Our final model evaluated on ScalarFlow scene 82.
22


Reviews:
[{"review_id": "cKaGHtWjBDa", "reviewer": "Reviewer_8YBe", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a single-view video-based volumetric reconstruction pipeline for smoke. The method assumes known background (black color) and foreground (smoke with known color). It takes the video frames as input and estimates both volumetric density and velocity. The pipeline is trained end-to-end by comparing the captured incoming frame with the rendered image that is based on the estimated volumetric density and velocity. Additional regularizations are added to tackle the inherent ambiguity in single view volumetric estimation. ", "summary_of_the_review": "The paper demonstrates a monocular video based method for smoke volumetric density and velocity reconstruction. Although the method has shown to be working for smoke with strict experiment setup (known foreground and background colors), I’m not sure if it can be extended to handle more general case (eg. smoke with real-world background, or even non smoke foreground). In addition, due to the confusions in the writing, the motivation for the design choice is not clear to me. Last but not least, the experiment is lacking in ablations. As a result, I would recommend weak rejection of the submission. ", "strength_and_weaknesses": "Strength\n\n+The usage of the transport function A() based on MacCormack advection  to render new frames adds physical inductive bias into the pipeline, so that the estimated states (density) can be propagated into the new timestamp in a physically correct way. It also helps to reduce the number of learnable parameters for the transition itself (eg. versus using GRU) and make the model more generalizable.\n\n+ The adversarial learning helps to reduce the estimation ambiguity. \n\n+ The pipeline trained end-to-end without GT volumetric density and velocity as supervisions, which is useful since GT volumetric data is hard to capture in real world.\n\nWeakness\n\n-Key assumption not explained: The rendering equation does not include the background color. If the background color changes, how to make sure this rendering equation is correct? If there is an additional assumption about the colors of the background and smoke, it should be claimed upfront since it is a very strong assumption\n\n-The writing needs improvement: \nWhat does it mean by c_z is the center along z (after Eq.6)? Without defining the start and end points along z for each ray, it is hard to understand it. \nEq.6: what is p? Is it a 3D point location? If so, how can we define L_z over one single 3D point? *\nEq.6: why multiplying 2 inside the second term eq.6 what is the intuition of the regularization?\nWhat is R^{-1} function in Eq.8\n\n-To tackle the ambiguities in the solution, the method assumes  the initial state is known. What if the initial state is not accurate? How can the subsequent estimation correct for the inaccurate initial state estimation? \n\n-Ablation study is not enough: ablations for the additional regularizations that tackle ambiguities are missing.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "W8_lZLu8l4", "reviewer": "Reviewer_UynW", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a new method to estimate the 3D flow and volumetric densities of a moving fluid from a monocular video. The proposed framework consists of two deep networks, one for predicting the volumetric densities from a single image, and one for predicting flow given the input images and predicted densities from two consecutive timestamps. Since ground truth 3D flow and densities are difficult to obtain, this paper adopts an unsupervised learning approach, which projects the inferred densities to the image space with a differentiable image formation operator, and then uses image-based losses to train the networks. To reduce the depth ambiguity, it further uses an adversarial loss to regularize the solution. Experiment results on benchmark datasets demonstrate the effectiveness and efficiency of the proposed method.", "summary_of_the_review": "Overall, this work presents an effective and efficient solution to the challenging problem of estimating 3D flow and volumetric densities from a monocular video. Leveraging unsupervised learning and adversarial losses appears to be a promising direction. However, the quality of the estimated 3D flow is not as good. To improve the results, more work needs to be done to adjust the network and losses.", "strength_and_weaknesses": "Strength:\n\n+ A novelty of this paper is that it trains deep networks to predict 3D flow and volumetric densities in an end-to-end fashion. Compared to optimization-based methods, the proposed method is much faster. \n+ Technically, the proposed method appears to be sound and effective. It adopts an adversarial loss, similar to the one used in [Franz et al., 2021], to regularize the estimated densities across different views. To some extent, this helps avoid requiring videos from multiple views as input, as prior work [Qiu et al., 2021] does. It also proposes to predict a prototype volume, which is shown to be beneficial in stabilizing the network training over time. \n\nWeaknesses:\n\n- The quality of the estimated flow is not very good. In Table 1, the errors in estimated flow u is much larger than the other methods on the synthetic dataset. Looking at the videos in the supp material, there are also noticeable artifacts in the flow motion. Perhaps, the design of network and losses makes the proposed method focus more on synthesizing the densities \\rho than predicting the flow u?\n- What are the limitations of this work? This is not discussed in the paper.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Vy_OIWUzdY", "reviewer": "Reviewer_TTZh", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a volumetric motion estimation framework specifically in the context of fluids. The framework can be trained end-to-end from only single-view image sequences. Experiments show that the proposed framework can successfully generalize across different inputs and outperforms single-scene optimization-based approaches.", "summary_of_the_review": "In general, the proposed method is technically sound to me. However, the experiments are somewhat limited. Furthermore, the clarity of the paper needs to be improved. Please see the above sections for more details.", "strength_and_weaknesses": "### Strength\n* The paper provides a reasonable explanation for the depth ambiguity problem in single-view inputs. \n* Motion estimation for fluids is a challenging task, especially from single views. In general, the proposed method is technically sound to me. \n### Weakness\n* The framework figures are complicated and hard to understand. Too many notations, and some of them are unexplained. It would be better to simplify the diagrams and add some legends.\n*  It would be better to give more background and intuitive explanations for some technical designs. For example, why use RaLSGAN objective to train the discriminator, and why would CFL condition help constrain the magnitude?\n* It would be better to visualize the learned prototype volume. \n* The ablation study is confusing. In Figure 3, there are two \"fix pos, multi, ¬D\"; one of them should probably be \"var pos, multi, ¬D.\" It is unclear from Figure 3 that adding the adversarial loss can sharpen the result. It would be better to provide quantitative results for all ablation studies. \n* The paper mentioned that using a multi-view capture setup does not help the density ambiguity issue and would result in a suboptimally averaged solution. How about using a multi-view+the proposed discriminator? Showing such a comparison could strengthen the evaluation. \n* Some minor typos. Section 3.1 - \"leadsto\",  Section 4.1 - \"spacial\".", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster