Latent Intuitive Physics: Learning to Transfer Hidden Physics from A 3D Video
Authors: Xiangming Zhu
Huayu Deng
Haochen Yuan
Yunbo Wang
Xiaokang Yang
Venue: ICLR 2024 poster (2024)
Keywords: Intuitive physics
inverse graphics
particle-based fluid simulation
neural rendering
Primary area: generative models

Abstract: We introduce latent intuitive physics, a transfer learning framework for physics simulation that can infer hidden properties of fluids from a single 3D video and simulate the observed fluid in novel scenes. Our key insight is to use latent features drawn from a learnable prior distribution conditioned on the underlying particle states to capture the invisible and complex physical properties. To achieve this, we train a parametrized prior learner given visual observations to approximate the visual posterior of inverse graphics, and both the particle states and the visual posterior are obtained from a learned neural renderer. The converged prior learner is embedded in our probabilistic physics engine, allowing us to perform novel simulations on unseen geometries, boundaries, and dynamics without knowledge of the true physical parameters. We validate our model in three ways: (i) novel scene simulation with the learned visual-world physics, (ii) future prediction of the observed fluid dynamics, and (iii) supervised particle simulation. Our model demonstrates strong performance in all three tasks.

Published as a conference paper at ICLR 2024
LATENT INTUITIVE PHYSICS: LEARNING TO TRANSFER HIDDEN PHYSICS FROM A 3D VIDEO
Xiangming Zhu" Huayu Deng" Haochen Yuan" Yunbo Wang! Xiaokang Yang MoE Key Lab of Artificial Intelligence, Al Institute. Shanghai Jiao Tong University (xmzhu76, deng_hy99, hy99, yuanhaochen, yunbow, xkyang @ itu hww.ttu.uuuuu...u.... en https://siten wttpo///...o.oo////aaaaaattttttt bo/
ABSTRACT
We introduce latent intuitive physics, a transfer learning framework for physics simulation that can infer hidden properties of fluids from a single 3D video and simulate the observed fluid in novel scenes. Our key insight is to use latent features drawn from a learnable prior distribution conditioned on the underlying particle states capture the invisible and complex physical properties. To achieve this, we train a parametrized prior learner given visual observations to approximate the visual posterior of inverse graphics, and both the particle states and the visual posterior are obtained from learned neural renderer. The converged prior learner perform novel
is embedded in probabilistic physics engine, allowing simulations on unseen geometries, boundaries, and dynamics without knowledge of the true physical parameters. We validate our model in three ways: (i) novel scene simulation with the learned visual-world physics, (ii) future prediction of the observed fluid dynamics, and (iii) supervised particle simulation. Our model demonstrates strong performance in all three tasks.
INTRODUCTION
Understanding the intricate dynamics of physical systems has been fundamental pursuit of science and engineering. Recently, deep learning- based methods have shown considerable promise in simulating complex physical systems (Battaglia et al. 2016 Mrowea et al. 2018 Schenck & Fox
2018, Li et al., 2019 Ummenhofer et al.] Sanchez-Gonzalez et al., 2020 Shao et al.) 2022
Prantl et al., 2022] Han et al.) 2022; Guan et al., 2022] Li et al. 2023). However, most previous works focus physics simulation with given accurate physical properties, which requires strong domain knowledge or highly specialized devices. Let us consider a question: Can we predict physical systems with limited knowledge of its physical properties? If not, is it possible transfer hidden physics present in readily accessible visual observations into learning-ban based physics simulators (Li
al. 2019 Ummenhofer al. 2020 Sanchez-Gonzalez al. 2020 Prantl et al. 2022
Inspired by human perception, researchers in the field of Al have proposed series of intuitive physics
methods (McCloskey al. 1983, Battaglia et al. 2013 Ehrhardt et al., (2019] Xu et al. 2019 Li
et al. 2020) to solve this problem. A typical approach is to build the so-called "inverse graphics" models of raw visual observations, which involves training ftarning-baseee physical simulators by
solving the inverse problem of rendering 3D scenes (Guan et al. 2022. Li et al. 2023). However, NeuroFluid (Guan al., (022) needs finetune the deterministic transition model response to every physics dynamics associated with new-coming physical properties. PAC-NeRF (Li et al. 2023) adopts heuristic (rather than learnable) simulators and explicitly infers physical properties (such as the viscosity of fluids) given visual observations. It requires appropriate initial guess of the optimized properties and specifying the fluid type (e.g.. Newtonian vs. Non-Newtonian fluids).
In this paper, introduce the lcarning framework of latent intuitive physics, which aims to infer hidden fluid dynamics from a 3D video, allowing for the simulation of the observed fluid in novel scenes without the need for its exact physical properties. The framework arises from intuition that we humans can imagine how a fluid with a specific physical property will move given its initial state by watching a video showcase of flowing, even though we do not explicitly estimate the exact
"quual contribution.
Corresponding author; Yunbo Wang.

---

Published as a conference paper at ICLR 2024
Particle Space Pretraining
Observation Space
Particle Space
Transfer
Linsenn Geometries
Uioasn daridas
Ursee Dynamics
Prior
Posterior
Latent
Space
p( p
Adaptation
(z
Simulation
p(x X1, z)
Figure 1: Our approach captures unobservable physical properties from image observations using parametrized latent space and adapts them for simulating novel scenes with different fluid geometries, boundary conditions, and dynamics. To achieve this, introduce variational method that connects the particle space, observation space, and latent space for intuitive physical inference.
values of physical properties. The key idea to represent the hidden physical properties in visual observations, which may be difficult to observe, using probabilistic latent states shown in Figure The latent space connects the partiele space and visual space to infer and transfer hidden physics with probabilistic modeling. Specifically, our approach includes probabilistic particle transition module p(>'|xr 2 a physical prior learner, particle-based posterior estimator, and a neural renderer, all integrated into differentiable neural network. The latent features are drawn from trainable marginal distributions p(=[z) that are learned to approximate the visual posterior distribution g(2 (=|Z) obtained from a learned neural renderer. By employing probabilistic latent features, model gains flexibiliity in modeling complex systems and is capable of capturing uncertainty in the data that a deterministic model may not be able handle. Once p(=[æ=) is converged, embed the prior learner in probabilistic fluid simulator that is pretrained in particle space containing fluids with wide range of physical properties. In this way, transfer the hidden physics from visual observations particle space to enable novel simulations of unseen fluid geometries, boundary conditions, and dynamics. In our experiments, we demonstrate the effectiveness of latent intuitive physics by comparing it to strong baselines of fluid simulation approaches in novel scene simulation, future prediction of the observed dynamics, and supervised particle simulation tasks.
The contributions of this paper can be summarized as follows:
We introduce latent intuitive physics, a learning- -baee approach for fluid simulation, which infers the hidden properties of fluids from 3D exemplars and transfers this knowledge fluid simulator.
We propose the first probabilistic particle-based fluid simulation network, which outperforms prior works in particle-bact based simulation with varying physical properties.
RELATED WORK
Learning-based simulation of particle dynamics. Recent research has introduced deep learning- based methods to accelerate the forward simulation of complex particle systems and address inverse
problems. These methods have shown success in simulating rigid bodies (Battaglia al., 2016 Han
et al. 2022), fluids (Belbute-Peres et al.) 2020) Sanchez-Gonzall
et al.
2020; Shao et al. [022]
Prantl et al. 2022). and deformable objects (Mrowca et al. [0188] Li al.
2019
et al. 2020 Lin et al., 2022). In the context of fluid simulation, DPI-Net (Li et al. 2019) proposes dynamic graphs with multi-step spatial propagation, GNS
Conzallz al., (020) uses
message-passing networks, and TIE (Shao et al. 2022) uses a Transformer-basee model to capture the spatiotemporal correlations within the particle system, Another line of work includes methods like CConv (Ummenhofer et al.) 2020) and DMCF (Prantl al. 2022), which introduces higher- dimensional continuous convolution operators to model interactions between particles. Different from the approaches for other simulation scenarios for rigid and deformable objects, these models do not assume strong geometric priors such as object-centric transformations or pre-defined mesh topologies (Pfaff et al. 202T Allen al. 2022). However, these models deterministie, assuming that all physical properties are measurable and focus on learning fluid dynamics under known physical properties. Moreover, the stochastic components that commonly exist in the real physical world are not taken into account by the deterministic models.
Intuitive physics learning with neural networks. Researchers have explored intuitive physics methods from range of perspectives. These include heuristic models GGlden & Proffu 1994
Here, use to indicate the historical states and x' to indicate the future states in the physical process.
2

---

Published as a conference paper at ICLR 2024
Trainable Components
11
12
1T
12
Prior learner
Dynamics prediction
Posterior constraint
21
22
1
2/2
Tt
I1
2222
222
11
22
2
21
2/2
21
22
(a) Pretraining
(b) Inference
(c) Transfer
(d) Simulation
iinriinngeineeeeeerrreee pipeline of latent intuitive physics. (a) Particle-space pretraining for probabilistic fluid simulation.
Figure 2: Graphical model of the
(b) Visual posterior optimization from visual observations with photometric loss. (c) Adaptation of the prior learner to the converged visual posteriors 2. (d) Novel scene simulation with the adapted prior learner. The training parts highlighted in color. We present details of these training stages in Figure
7 in the appendix.
Sanborn et al. 2013), probabilistic mental simulation models (Hegarty 2004 Bates el al. 2015). and
the cogmitive ittitive physics models (Battaglia al., 2013: Ullman et al. 2017). Recent advances in deep learning typically investigate intuitive physics from different aspects. Some approaches adopt 3D videos to downstream tasks, such as predicting multi-object dynamics (Driess al.
2023), fluid dynamic (Guan et al. 2022 Li et al. (023),, system identification (Li et al., 2023). manipulation (Simeonov et al.]
| | 2022
et al.] |l.]|2222 (0022) or reasoning physics parameters (Li al..
2020; Chen al. [2022; Le Cleac'l h et al. 2023). However, most works focus rigid body dynamics (Driess et al. 2023 Li et al. 2020; Le Cleac'h al. 2023). The most relevant work to method is NeuroFluid (Guan et al. 2022) and PAC-NeRF (Li al. 2023), both focusing on fluid dynamics modeling with visual observation. NeuroFluid directly adopts the learning- based simulator from CConv (Ummenhofer al. 2020). Unlike our approach, it is deterministic and cannot handle the stochastic components or inaccessible physical properties in complex physical scenarios. PAC-NeRF employs specific non-learnable physies simulators tailored to different types of fluids. When using PAC-NeRF to solve inverse problems, users need to make an appropriate initialization for the optimized parameters based on the category of fluid observed.
3 PROBLEM FORMULATION
We study the inverse problem of fluid simulation, which refers to learning inaccessible physical properties by leveraging visual observations. Specifically, we consider a dynamic system where we only have a single sequence of observations represented as multi-view videos {4.T It=1:7" represents visual observation received at time t from view m. We want to prediet the future states of the system when it appears in novel scenes. Let x, = (x!,....,.r) at time i, where æt (p', nf) represents the state of the 7th particle that involves the position pi and velocity uf, with 2/ being the time derivative of pł. The dynamics of particles (xI...., XT) } is jointly governed by set of physical properties, such as density, viscosity, and pressure. These properties are hidden and need to be inferred in visual observations. To bridge the gap between particle simulators and the visual world with varying set of physical properties in a unified framework, we introduce a set of latent features Z, (2] (2.,..., <NN), where 2 is the latent feature attached to each particle. As shown Figure a), the particle state transition function can thus be represented as x2 ~ p(xx,11, 2:). where ~ p( 11 2,-1). As the explicit physical properties are inaccessible, we can infer latent distribution p(z: x1:z) from q(z II.T). The final goal is to simulate novel scenes with new initial and boundary conditions based on learned physics (see Figure d)).
1m=11:M where I,"h
E X be a state of the system
4 LATENT INTUITIVE PHYSICS
In this section, introduce latent intuitive physics for fluid simulation, which enables the transfer of hidden fluid properties from visual observations to novel scenes. As shown in Figure our model consists of four network components: the probabilistic partiele transition module (0), the physical prior learner (e), the particle-based posterior estimator (E), and the neural renderer (0). The training
pipeline involves three stages shown in Figure 2 pretraining, inference, and transfer:
a) Pretrain the probabilistic fluid simulator on the particle dataset, which involves the particle transition module (x xt-1, ),), the prior learner Ps X1:-11, 1;, and the posterior de (Z,, X1:t, 1)) The prior module reasons about latent features from historical particle states.
3

---

Published as a conference paper at ICLR 2024
Train & Test
Prior Learner
Particle Transition Module
Neigharrood
X
CConv
GRU
To
Sample Point
Camara Ray
ed
Ep
View-indepandent Encoding
Train
Posterior Estimator
Rep
View-dependent Encoding
CConv
G
To
d Viewing Direction
Loss
Our model consists of four network components parametrized by 8, t) E, respectively. We present the losses for pretraining the simulator and the renderer. For schematics of other training stages (i.e. visual posterior inference and prior adaptation), please refer to Figure 7 in the appendix.
Figure 3:
Infer the visual posterior latent features from consecutive image observations of a specific fluid, which is achieved by optimizing differentiable neural renderer (Ø).
Train the prior learner py (2r X1:t-1, - to approximate the converged distribution of 2, which enables the transfer of inaccessible physical properties from the visual world to the simulator.
4.1 STAGE A: PROBABILISTIC FLUID SIMULATOR PRETRAINING
Building probabilistic model has significant ad advantages for fluid simulation: First, it allows us to predict fluid dynamics without knowing true physical parameters. Instead, it relies on inferring latent features from consecutive inputs. This is valuable because many complex physical phenomena naturally involve stochastic components, Second, when provided with visual observations, we can seamlessly integrate the probabilistic fluid simulation into our variational inference method via the latent space. Next, we introduce how to infer physical properties from particle data.
The architecture of probabilistic fluid simulator is shown in Figure 3 (Left). Particle states x itself is not feature but simply defines the particle's state in spatial space. We use the continuous convolution (CConv) (Ummenhoter et al. 2020) feature encoder to get feature representations of each particle. Inspired by traditional Smoothed Particle Hydrodynamics (SPH) methods, CCony predicts particle features by aggregating its neighbors' features in continuous and smooth manner. Static particles such as boundaries, are processed similarly but with the particle positions and normal vectors as input (see the literature by [Ummenhofer et al. (2020). Since the invisible physical properties cannot be inferred from a single state, we use a GRU to gather historical information and infer the distribution of the prior latents. The prior learner is trained along with a separate posterior estimator qe (Zt) X1:t, 1 (not used at test time). The models can be written
(I) where 2==1 and 2 are zero-initialized. The posterior estimator takes x, as input, i.e., the target of the prediction. The prior and posterior latents are sampled from distinct Gaussian distributions, with their parameters determined by predicted means and variances by two GRUs. During training, align their distributions through KI. divergence. Notably, our approach assumes time-varying and particle-dependett fluid properties, which aligns with conventional SPH methods (please refer the work by Bender & Koschier (2015)). This approach empirically achieves better performance than optimizing a global latent variable, as demonstrated in our experiments.
Prior: 2t ~ GRU (CConv(
Posterior: Z, ~ GRU, (CConv(z,11,
For the particle transition module (x, X 2t), we adopt another CConv with additional inputs of 2, drawn from the inferred latent distribution. This allows the module incorporate the previous states x;-1 and corresponding latent features for future prediction. As the stochastic physical component has been captured by Z,. we employ deterministic architecture for the particle transition module: &, & {(pt. bt) bi) (x 2t). During training. the latent posteriors z, are used as inputs of To. At test time, we use the latent priors ž, instead. The fluid simulator is trained with N
L9.v.E = E
Σ
w. willpt pilla + 3 DxL (ge (z, XL., (1:2:2111 2x-1)
Pv (2, X1a- 2 21-)))]
(2)
N
Similar to the previous work (Li et al. 2019 Prantl al. 2022 Ummenhoter al. 2020 Sanchez Gonzalez et al. 2020), we use the positions, and weight it by the neighbor count to form the reconstruction loss. Specifically, we use W:: = exp( W (f))). where N ((r)) denotes the number of neighbors for the predicted particle and e is the average neighhor count.
error between the predicted position and the ground- -ruth
4

---

Published as a conference paper at ICLR 2024
4.2 STAGE B: VISUAL POSTERIOR INFERENCE
Here we introduce how to solve the inverse problem by inferring scene-specific visual posteriors, where the visual observation governs the physical properties. In this stage, the pretrained particle transition module Ty infers the input visual posterior, which facilitates the adaptation of the prior learner in the latent space in Stage C. To this end, the particle transition module is combined with a differentiable neural renderer R.p that provides gradients backpropagated from the photometric error over sequences in observation space. Note that only visual observations are available in the following.
Neural renderer. To enable joint modeling of the state-to-state function of fluid dynamics and the state-to- siile-to-gragge co-graphics mapping function, integrate the probabilistic particle transition module with the particle-driven neural renderer (PhysNeRRF) in NeuroFluid (Guan et al. (2022) in a differentiable framework. PhysNeRF uses view-independent particle encoding ep and view-dependent particle encoding ea to estimate the volume density and the color с of each sampled point along each ray r(I) o + /d, such that a) (epep: ed: ed.d). In this way, it establishes correlations between the particle distribution and the neural radiance field. Unlike the original PhysNeRF, we exclude the position of the sampled point from the inputs to the rendering network, which enhances the relationships between the fluid particle encodings and the rendering results. The neural renderer R. pretrained on multiple visual scenes so that it can respond to various particle-batd baaiclee-baed geometries.
Initial states estimation. NeuroFluid assumes known initial particle states. However, when only visual observations are available, estimating the initial particle states x;=1 becomes necessary. These estimated initial states are used to drive the neural renderer (Rø) for generating visual predictions at the first time step and also to initiate the particle transition module (To) for simulating subsequent states. We estimate the initial particle positions using the toxel-based neural rendering technique (Liu et al. 2020 Sun et al.] 2022 Müller et al., (202)) at the first time step. During training, we maintain an occupancy cache to represent empty vs. nonempty space and randomly sample fluid particles within each voxel grid in the visual posterior inference stage (see Appendix D.2.3 for details).
Optimization. We first finetune the neural renderer Rs on current visual observation with initial state estimation x;=1. Then the parameters of To and RR are frozen and we initialize set of visual posterior latents 2, such that & To 1, 2). In practice, we attach particle-depende
Gaussian distribution (ij a*) with trainable parameters to each particle i. At each time step, we sample 2t N(e, Ø²) to form 2 = (#) ..... ). The output of the neural renderer R2 is denoted as Ö(r, t). We optimize the distributions of the visual posterior latent over the entire sequence by backpropagating the photometric error Er, [[, () - C(r, O)l. We summarize the overall training algorithm in the Alg. in the appendix.
4.3 STAGE C: PHYSICAL PRIOR ADAPTATION
The visual posteriors learned in the previous stage are specific to the estimated particles within the observed scene and cannot be directly applied to simulate novel scenes. Therefore, in this stage, we aim to adapt the hidden physical properties encoded in the visual posterior to the physical prior learner Pv. Instead of finetuning the entire particle transition model as NeuroFluid (Guan et al.) (022) does, we only finetune the prior learner module. Due to the unavailability of the ground truth supervision signal in particle space, tuning all parameters in the transition model in visual scenes might lead to overfitting problems, as the transition model may forget the pre-learned knowledge of feasible dynamics, or learn implausible particle transitions even if it can generate similar fluid geometries that are sufficient to minimize the image rendering loss. Specifically, we perform forward modeling on particle states X, by applying X, (x(x;,1, 2,), where ž, is sampled from the distribution (2, X1:- 2;-1) predicted by the prior learner. To transfer the visual posterior to Pv. we finetune the prior learner by minimizing the distance between its generated distribution and the pre-learned visual posteriors (N(j,, } ji=1:N with To and R. fixed. The volume rendering loss is still used for supervision well. The entire training objective is
Cs = =E,, r.t [|C(r,t) C(r, t)ll + 3 DKL (9(2) p (Z, X1=-11 Z.1))
(3)
With the finetuned physical prior learner Pv. we can embed it into the probabilistic fluid simulator to perform novel simulations unseen fluid geometries, boundary conditions, and dynamics with identical physical properties, which brings the simulation back to the partiele space.
5

---

Published as a conference paper at ICLR 2024
Table 1: Quantitative results of average prediction error d on unseen fluid geometries and boundary conditions given xt= We present the mean and standard deviation 10 independent samples drawn from our model (for Ours only). Geometry means unseen fluid initial positions while Boundary means unseen boundary conditions. Each physical property set trained with a single visual observation.
= 2000, v = 0.065
p = 1000, v = 0.08
= 500, v = 0.2
METHOD
GEOMETRY
BOUNDARY
GEOMETRY
BOUNDARY
GEOMETRY
BOUNDARY
CCONV
52.49
64.29
51.40
56.33
40.67
53,28
NEUROFLUID
65.01
73.55
59.79
60.46
40.88
50.73
SYS-ID
156.59
179.11
127.06
140.27
58.71
71.80
PAC-NERF
51.10
59.61
51.33
56.84
40.97
62.05
OURS
34.54+0.55
39.86+0.90
33.11+0.550
37.79=0.84
39.03++0.79
1.22
Table 2: Quantitative results of the average future prediction error d on observed scenes. We compute the mean results and standard deviations of 10 independent samples drawn from our model. Notably, NeuroFluid learns to overfit the observed scene by tuning the entire particle transition simulator.
METHOD
P = 2000, v/ = 0.065 = 1000, v 0.08
P = 500, v 0.2
CCONV
107.23
103.95
54.37
NEUROFLUID
85.
73.04
33.22
SYs-ID
37.25
36.28
42.71
PAC-NERF
44.42
42.82
50.05
OURS
32.41
32.97-10.71
41.15.00.71
5
EXPERIMENTS
5.1 EVALUATION OF VISUAL PHYSICAL INFERENCE
Settings. To evaluate latent intuitive physics for inferring and transferring unknown fluid dynamics from visual observations, we generate a single sequence using Cuboid as the fluid body (not seen during pretraining Stage A). The fluid freely falls in a default container. The sequence contains 60 time steps, where the first 50 time steps are used for visual inference, while the last 10 time steps are reserved for additional validation on future rollouts. We use Blender (Community] 2018) to generate multi-view images with 20 randomly sampled viewpoints. We assess the performance of our fluid simulator after transferring hidden properties from visual scenes through: (1) simulating novel scenes with fluid geometries that were not seen during training (Stanford Bunny, Sphere, Dam Break), (2) simulating novel scenes with fluid dynamics with previously unseen boundaries. We conduct experiments across three distinct sets of physical properties. In cach evaluation set, impose random rotation, scaling, and velocities on fluid bodies. Please refer to Appendix [ D.2 for more information visual examples, novel scenes for evaluation, and implementation details. Following Ummenhofer et al. (2020), we compute the average Euclidean distance from the ground truth particles to the closest predicted particles the evaluation metric, where the average prediction error is TXN E. E; ming, lppi - prl12. Please see Appendix D. for details.
Compared methods. We use four baseline models. CConv (Ummenhofer et al. 2020) learns particle dynamics without physical parameter inputs. NeuroFluid (Guan et al.
2022) grounds fluid
dynamics in observed scenes with particle-driven neural render. PAC-NeRF (Li et al. 2023) and System Identification (Sys-ID) estimate explicitly physical parameters in the observed scenes. PAC- NeRF employs an MPM simulator. Sys-ID utilizes CConv simulator that takes learnable physical parameters as inputs. It also employs the same neural renderer our approach. All models are trained on Cuboid and tested on novel scenes. Please refer to Appendix D.2.I for more details.
Novel scene simulation results. We evaluate the simulation results of our approach given only the true initial particle states X;=1 of the novel scenes. Table [ the average prediction error across all testing sequences. We predict each sequence 10 times with different z, drawn from the same prior learner and calculate the standard deviation of the errors. We can see that the adapted probabilistic fluid simulator from the visual posteriors significantly outperforms the baselines on novel scenes across all physical properties. Though our model is trained scenes with the default fluid boundary, shows the ability to generalize to unseen boundary conditions. Figure Seoure/showcaar showcases the qualitative results. Among the baselines, Sys-II) underperforms in novel scene simulation, it requires accurate

---

Published as a conference paper at ICLR 2024
Visual Obs
1=15
2-
t=35
445
R5
2=
c=
C=4
CConv
Neuro Fluid
PAC-
NeRF
Ours
GT
Figure 4: The first row shows the visual observation on the observed scene with physical parameters p= 2000, v 0.065. Rows 2-6 show qualitative results of simulated particles on novel scenes (Left: unseen geometries, Right: unseen boundaries). More qualitative results are provided in the appendix.
parameter inference from visual observations. In contrast, model encodes the hidden properties with higher-dimensinn latent variables, providing a stronger feature representation for the unknown properties in a physical system. PAC-NeRF employs MPM simulator, which inherently produces more accurate and stable simulation results than the learning- based simulators used by other models trained on limited data. Despite this advantage, PAC-NeRF tends to overfit the observed scenes and yields degraded performance when applied to novel scenes. More results are shown in Appendix F
Future prediction results of the observed scenes. We predict the particle dynamics of the observed scene Cuboid for 10 time steps into the future. As shown Table 2 our model performs best in most cases. NeuroFluid slightly outperforms our model on p 500, = 0.2. Since NeuroFluid jointly optimizes the entire transition model and renderer on the observed scene, it is possible to overfit the observed scene and produce plausible future prediction results. However, it fails to generalize to novel scenes as shown in Table Unlike NeuroFluid, our approach adapts the physical prior learner to visual scenes, without training probabilistic physics engine. By leveraging knowledge from the pretraining stage, the transition model less prone to overfitting on the observed scene. This not only enhances the generalization ability but also significantly reduces the training burden.
5.2 EVALUATION OF PROBABILISTIC FLUID SIMULATOR
To validate whether our approach can infer hidden physics from particle data in latent space, we evaluate the pretrained probabilistic fluid simulator (0, . a particle dataset generated with DFSPH (Bender & Koschier 2015). which simulates fluids with various physical parameters (e.g., viscosity v, density p) falling in a cubic box. Each scene contains 273-19,681 682 fluid particles and 200 time steps. To assess the simulation performance under incomplete measurement of physical parameters, the true parameters are invisible to simulators. See the Appendix C.I for more details.
We compare our probabilistic fluid simulator with four representative particle simulation approaches. based on graph neural networks, continuous convolution models, and Transformer, i.e., DPI-Net (Li et al. 2019), CConv (Ummenhofer al. 2020), DMCF (Prantl al. (022), and TIE (Shao et al. 2022). Following Ummenhofer et al. (2020), given two consecutive input states X;-1:t+ we compute the errors of the predicted particle positions w.r.t. the particles: de.r = 1 E, Ilpi+T - Pi+rlla for the next two steps (r E {1, 2))) To assess the long-term prediction ability, we also calculate the average distance d from
particle positions the predicted particles over the entire sequence. The first 10 states are given, and the models predict the following 190 states. From Table 3 our model performs best in both short-term and long-term prediction. The qualitative result of long-term prediction is shown in Figure 16 in Appendix F These results showease that our probabilistic fluid simulation method provides an effective avenue for intuitive physics learning.
7

---

Published as a conference paper at ICLR 2024
Table 3:
Quantitative comparisons of fluid simulators on the particle dataset with inaccessible physical properties. We present the prediction errors for the next two frames d and de+2, and the averaged prediction error over the whole sequence for 190 prediction time steps.
METHODS
d++1
d++2
d
DPI-NET EL AL.
2019)
0.95
2.99
90.2
CCONV HUMENHOFER ET AL.
2020
0.34
1.03
44.79
DMCF PRANIL ET 2022
0.54
1.23
39.70
TIE (HAO ET AL. OURS
2022)
0.52
1.36
41.82
0.31=0.003
0.94::0.011
38.37:t0.860
Table 4: Results of d on generalization to unseen fluid dynamics given Xt=1- Global Latent is variant our model with a scene-specific global latent.
METHODS
CCONV
NEUROFLUID
GLOBAL LATENT
OURS
OBSERVED
56.92
54.67
60.63
36.03
0.80
UNSEEN
46.83
54.50
90.51
44.25 +: 1.36
5.3 GENERALIZATION TO DYNAMICS DISCREPANCIES
To validate the generalization ability our approach across discrepancies fluid dynamics patterns, consider Specifically, we use the pretrained probabilistic fluid simulator discussed in See 5.2 and adapt the model to visual scene containing two prior learners with the same initialization and a single particle transition module to learn the hidden physics of different fluid drops separately. Table Figure 5 present both quantitative and qualitative results. Our approach showcases robust generalization ability when dealing with visual scenes containing fluid dynamics that diverge significantly from the patterns in the pretrained dataset.
complex scenario that contains mixture of two different fluids.
heterogeneous fluids interacting with each other. We
Furthermore, we explore the performance of an alternative method that employs scene-specific, time-invariant latent variables (i.e., Global Latent in Table and Figure [5. From these results, find that optimizing time-varying latent features individually for each particle is more effective than optimizing two sets of global latents, each designated for particular fluid type.
5.4 ABLATION STUDY
To verify the effectiveness of transferring the learned visual posterior 2 to the physical prior learner, we experiment with different variants of our method. The result is shown in Table 5 w/o StageB
Visual Obs
t=15
t=25
t=35
t=
t=
t=2
t=35
45
CConv
Neuro Fluid
Global Latent
Ours
Figure 5: Qualitative results on generalization unseen dynamics of heterogeneous fluids. We present simulation results on the observed scene (Left) and novel scene (Right).
8

---

Published as a conference paper at ICLR 2024
Table 5: Ablation study for each training stage in pipeline. We report mean prediction error (d) resulting from the removal of Stage B, Stage C, or both, based on 10 samples from our model. The future prediction error is reported observed scenes and the prediction errors unseen fluid geometries given x Ours Using true initial states the observed scenes for visual inference.
= 2000, v = 0.065
= 1000, V = 0.08
P p= 500, v/ = 0.2
METHODS
OBSERVED
UNSEEN
OBSERVED
UNSEEN
OBSERVED
UNSEEN
w/o STAGE
33.22
N/A
32.72
N/A
37.04
N/A
w/o STAGE B
37.55
42.43
35.75
46.98
41.71
41,40
OURS OURS!
32.41
34.54
32.97
33.11
41.15
39.03
28.99
33.34
31.87
38.84
39.70
38.41
refers to the model without transferring posterior latent distribution ? to the physical prior learner and directly finetunes it visual observations. w/o Stage refers without adapting the physical prior learner. In this case, since we only have features 2 attached to each particle, we cannot simulate novel scenes of various particle numbers. The results show that the posterior latent distribution can make training more stable by restricting the range of distribution in latent space, and the transfer learning of the prior learner enables the prediction performance gap between feeding the ground truth initial state and the estimated initial state to the probabilistic fluid simulator. We find that models utilizing estimated initial states yield comparable performance results when compared to those using ground truth initial states.
novel scenes. In addition, we further investigate
POSSIBILITIES OF REAL- WORLD EXPERIMENTS
The primary focus of this paper is to explore the fea- sibilityy of new learning scheme for intuitive physics. Using synthetic data greatly facilitate the evalu- ation of our model, as the simulation results can be directly quantified using particle states; whereas real- world scenes would necessitate more advanced fluid flow measurement techniques like particle image ve- locimetry. For similar reason, earlier attempts like NeuroFluid and PAC-NeRF are also evaluated on syn- thetic data. Nevertheless, we acknowledge that real- world validation is meaningful and challenging, and so make our best efforts to explore the possibility of imple- menting latent intuitive physics in real-world scenarios. As shown in Figure 6 we capture RGB images of dyed water a fluid tank at resolution of 1,200 x 900. To cope with the complex and noisy visual environment, we adopt NeRFREN (Guo et al. 2022) to remove the reflection and refraction and segment the fluid body by SAM (Krilllov al. 2023 The preprocessed images are then used to estimate fluid positions using our proposed initial state estimation module. We carefully discuss the pipeline in Appendix Gnd include a video the supplementary. Another notable challenge of the complete real-world experiments is to acquire high frame-rate images with synchronized cameras across muitiple viewpoints. We leave this part for future work.
Raw image
Refection removal
Background removal
Novel view rendering
Depth
Inital state estimation
estimation
Figure 6: Our pipeline and intermediate results for real-world experiments. We cap- ture images of dyed water in fluid tank and estimate the initial states. We will ex- plore dynamic scenes in future work.
7 CONCLUSION
In this paper, we presented latent intuitive physics, which learns the hidden physical properties of fluids from 3D video. The key contributions include: 1) probabilistic fluid simulator that considers the stochastic nature of complex physical processes, and 2) a variational inference learning method that can transfer the posteriors of the hidden parameters from visual observations to the fluid simulator. Accordingly, we proposed the pretraining- inference -rrnsseerrrrfrrrrrre optimization scheme for the model, which allows for easy transfer of visual-worid fluid properties to novel scene simulation with various initial states and boundary conditions. We evaluated our model on synthetic datasets (similar to existing literature) and discussed its potential in real-world experiments.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
This work was supported by the National Natural Science Foundation of China (Grant No. 62250062, 621061440 14)). the Shanghai Municipal Science and Technology Major Project (Grant No. 202ISHDZX0100)) the Fundamental Research Funds for the Central Universities, the Shanghai Sailing Program (Grant No. 212S10202133), and the CCF-Tencent Chino-Bird Open Research Fund.
REFERENCES
Kelsey R Allen, Tatiana Lopez-Guevara, Kimberly Stachenfeld, Alvaro Sanchez-Gonzalez, Pe- ter Battaglia, Jessica Hamrick, and Tobias Pfaff. Physical design using differentiable learned simulators. arXiv preprint arXiv:2202.00725. 2022.
Christopher Bates, Peter W Battaglia, lkker Yildirim, and Joshua B Tenenbaum. Humans predict liquid dynamics using probabilistic simulation. In CogSci, 2015.
Peter Battaglia, Razvan Pascanu, Matthew Lai, Danilo Jimenez Rezende, et al. Interaction networks for learning about objects, relations and physics. In NeurlPS, volume 29, 6.
Peter W Battaglia, Jessica B Hamrick, and Joshua B Tenenbaum. Simulation an engine of physical scene understanding, Proceedings of the National Academy of Sciences, 11(44)::8337-11332 2013.
Filipe De Avila Belbute-Peree Thomas Economon, and Zico Kolter. Combining differentiable pde solvers and graph neural networks for fluid flow prediction. In ICML, pp. 2402-2411, 2020.
Jan Bender and Dan Koschier. Divergence-free smoothed particle hydrodynamics. SCA, pp. 147-155, 2015.
SPlisHSPlasH Library, 2022. Jan Bender al.
URL
https://github.cr
Zhenfang Chen, Kexin Yi, Yunzhu Li, Mingyu Ding, Antonio Torralba, Joshua B Tenenbaum, and Chuang Gan. Comphy: Compositional physical reasoning of objects and events from videos. In ICLR, 2022.
Blender Online Community. Blender - a 3D modelling and rendering package. Blender Foundation, Stichting Blender Foundation, Amsterdam, 2018. UtL/htt::/ wwtp://wwwww///// http::::/wwwwbbbeeee... www.ndeee.............. org
Danny Driess, Zhiao Huang, Yunzhu Li, Russ Tedrake, and Marc Toussaint. Learning multi-object dynamics with compositional neural radiance fields. In CoRL, volume 205, pp. 1755-1768, 2023.
Sebastien Ehrhardt, Aron Monszpart, Niloy Mitra, and Andrea Vedaldi. Unsupervised intuitive physics from visual observations. In ACCV, pp. 700-716, 2019..
David I. Gilden and Dennis R Proffitt. Heuristic judgment of mass ratio in two-body collisions. (ihi:pmyysess5567788772,,,,
Perception & Psychophysies, 56:708-720, 1994.
Shanyan Guan, Huayu Deng, Yunbo Wang, and Xiaokang Yang. Neurofluid: Fluid dynamics
grounding with particle-driven neural radiance fields. JCML, pp. 7919-7929, 2022.
Yuan-Chen Guo, Di Kang, Linchao Bao, Yu He, and Song-Hai Zhang. Nerfren: Neural radiance fields with reflections. Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 18409-18418, June 2022.
Jiaqi Han, Wenbing Huang, Hengbo Ma, Jiachen Li, Josh Tenenbaum, and Chuang Gan. Learning physical dynamics with subequivariant graph neural networks. In NeurlPS, volume 35, pp. 26256- 26268, 2022.
Mary Hegarty. Mechanical reasoning by mental simulation. Trends in cognitive sciences, 8(6): 280-285, 2004.
10

---

Published as a conference paper at ICLR 2024
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, and Ross Girshick. Segment anything. arXiv:2304.0266 2023.
Simon Le Cleac'h, Hong-Xing Yu, Michelle Guo, Taylor Howell, Ruohan Gao, Jiajun Wu, Zachary Manchester, and Mac Schwager. Differentiable physics simulation of syynamcss-uugmettn neural objects. IEEE Robotics and Automation Letters, 86)):7700-2877, 2023.
Xuan Li, Yi-Ling Qiao, Peter Yichen Chen, Krishna Murthy Jatavallabhula, Ming Lin, Chenfanfu Jiang, and Chuang Gan. PAC-neRF: Physics augmented continuum neural radiance fields for geometry-agnostic system identification. In ICLR, 2023.
Yunzhu Li, Jiajun Wu, Russ Tedrake, Joshua B Tenenbaum, and Antonio Torralba. Learning particle dynamies for manipulating rigid bodies, deformable objects, and fluids. In ICLR, 2019.
Yunzhu Li, Toru Lin, Kexin Yi, Daniel Bear, Daniel L.K. Yamins, Jiajun Wu, Joshua B. Tenenbaum, and Antonio Torralba. Visual grounding of learned physical models. In ICML, 2020.
Yunzhu Li, Shuang Li, Vincent Sitzmann, Pulkit Agrawal, and Antonio Torralba. 3d neural scene representations for visuomotor control. In CoRL, pp. 112-123, 2022.
Xingyu Lin, Yufei Wang, Zixuan Huang, and David Held. Learning visible connectivity dynamics for cloth smoothing. In CoRL, pp. 256-266, 2022.
Lingjee Liu, Jiatao Gu, Kyaw Zaw Lin, Tat-Seng Chua, and Christian Theobalt. Neural sparse voxel fields. In NeurlPS, volume 33, pp. 15651-15666 2020.
Michael McCloskey, Allyson Washburn, and Linda Felch. Intuitive physics: the straight-down belief and its origin. Journal of Experimental Psychology: Learning, Memory, and Cognition, 9(4):636, 1983.
Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes neural radiance fields for view synthesis. In LCCV, URL
abs http:/iiwwrrg/bb//0........ 05934v2
Damian Mrowca, Chengxu Zhuang, Elias Wang, Nick Haber, Li F Fei-Fei, Josh Tenenbaum, and Daniel I. Yamins. Flexible neural representation for physics prediction. In NeurlPS, volume 31, 2018.
Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with multiresolution hash encoding. ACM Transactions on Graphics, -1,: 2022.
Tobias Pfaff, Meire Fortunato, Alvaro Sanchez-Gonzale and Peter Battaglia. Learning mesh-based simulation with graph networks. In International Conference on Learning Representations, 2021.
URL http:/ttts::/ppnreeeeeeeee htwpeaaeeeeieeeeeeeeeeeoeeeeee wwt.:/et/oourer fttee///oorrmm1ddoooo111.... 2id=rongyoorooooo
Lukas Prantl, Benjamin Ummenhofer. Vladlen Koltun, and Nils Thuerey. Guaranteed conservation of momentum for learning particle-based fluid dynamics. In NeurIPS, 2022.
Adam N Sanborn, Vikash K Mansinghka, and Thomas L Griffiths. Reconciling intuitive physics and newtonian mechanics for colliding objects. Psychological review, 120(2):411, 2013.
Alvaro Sanchez-Gonzale Jonathan Godwin, Tobias Pfaff, Rex Ying, Jure Leskovec, and Peter Battaglia. Learning to simulate complex physics with graph networks, In ICML, pp. 8459-8468, 2020.
Connor Schenck and Dieter Fox. Spnets: Differentiable fluid dynamics for deep neural networks. In CoRL, pp. 317-335, 2018.
Yidi Shao, Chen Change Loy, and Bo Dai. Transformer with implicit edges for particle-bast based physics simulation. In ECCV, pp. 564,566, 2022.
11

---

Published as a conference paper at ICLR 2024
Anthony Simeonov, Yilun Du, Andrea Tagliasacchi. Joshua Tenenbaum, Alberto Rodriguez, Pulkit Agrawal, and Vincent Sitzmann, Neural descriptor fields: Se equvvvrrri object representations for manipulation. In ICRA, pp. 6394-6400, 2022.
Cheng Sun, Min Sun, and Hwann-Tzong Chen. Direct voxel grid optimization: Super-fast conver- gence for radiance fields reconstruction. In CVPR, pp. 5459-5469, 2022.
Tomer D Ullman, Elizabeth Spelke, Peter Battaglia, and Joshua B Tenenbaum. Mind games: Game engines as an architecture for intuitive physics. Trends in cognitive sciences, 21(9):649 06399966666100 2017.
Benjamin Ummenhofer, Lukas Prantl, Nils Thuerey, and Vladlen Koltun. Lagrangian fluid simulation with continuous convolutions. In ICLR, 2020.
Zhenjia Xu, Jiajun Wu, Andy Zeng, Joshua B Tenenbaum, and Shuran Song. DensePhysNet. Learning dense physical object representations via multi-step dynamic interactions. In RSS, 2019.
12

---

Published as a conference paper at ICLR 2024
A MODEL DETAILS
A.1 OVERALL FRAMEWORK
Figure 7 demonstrates the main architecture of the proposed framework, and 6 the summary of all model components, including formulation, input, output, training stage, and objective. The entire model includes a probabilistic particle transition module (0), a physical prior learner (1)), a particle-based posterior module (E), and neural renderer (#). The physical prior learner (Z, X1:t-1, 2(-1) infers latent features ž; from consecutive particle data, which the transition module pe (X: X;-1, 2:) can use to predict the next state X₁. In the pre-training stage, a particle- posterior 4: (zz X1:t, 2)-1) generates posterior z₁ with the prediction target x, as input. It guides the prior learner to generate meaningful latent features that capture hidden properties essential for accurate predictions. During visual inference and transfer in Stage and the neural renderer R. generates images according to particle positions and viewing directions, which enables photometric error to be backpropagated through the whole differentiable neural network. In the visual posterior inference stage (Stage B), the visual posteriors are optimized by backpropagating image rendering errors. They serve adaptation targets for the prior learner, facilitating efficient adaptation the specific observed fluid.
Stage A: Probabilistic fluid simulator pretraining
Training
2
X
P
Frozen
min
To
X
Loss
X
qc
Physical
Zt
Ps
prior learner
Stage B: Visual posterior inforence
X
qc
Particle-based posterior
R
I
X
Particle transition model
Te
Stage C: Physical prior adaptation
Visual posterior
X
Pv
To
RNp
min KL
X
R
Neural renderer
I
The overview of the proposed framework. The prior learner infers latent features from the data with the help of particle/visual posterior. The latent features are then used the particle transition module predict the next state. The neural renderer bridges the gap from the particle space to the visual observation space.
Figure 7:
A.2 PROBABILISTIC FLUID SIMULATOR
The probabilistic fluid simulator is used to infer hidden physics and simulate various kinds of fluid dynamics. Specifically, as shown Figure 3 our probabilistic fluid simulator consists three modules: probabilistic particle transition module (6), physical prior learner (1)) and bbasele--baedd posterior module (E).
We continuous convolution (CConv) (Ummenhofer et al. 2020 the shared feature encoder for the prior learner (w) and the particle-based posterior module (E) to extract features from particle states. The CCony module predicts particle features in smooth and continuous way: (ʃ gx)
13

---

Published as a conference paper at ICLR 2024
Table 6: The overview of model components our method.
Formulation
Po(X; X(-1.2;) Particle state x and latent feature Next particle state X,
Particle
Inpul
transition
Output
module
Training stage
Stage A
Objective
Predict the next state according to particle states and latent features
Formulation
ā (2 X1:-11 ((-1)
Physical
Inpul
Particle states X1:t-1 and latent feature 7
prior
Output Training stage
Prior latent feature 2;
learner
Stage A & C
Objective
Narrow the gap with the posterior distribution
Formulation
Z: ~ [g(Zz X1:12211)
Particle-
Inpul Output Training stage
Future particle state xx, and previous states and features (x1a 1, Ze 1) Posterior latent feature Z₁
based
posterior
Stage A
Objective
Facilitate prior learner training when particle states are available
Formulation
~ q( IIT)
Input
N/A
Visual
Output Training stage
Particle-dependntt feature
posterior
Stage B
Objective
Provide adaptation targets for the prior learner in subsequent Stage
Formulation
Re ed, d)
Input
Encoding based on predicted particle positions and viewing directions Rendered image
Neural
Output
renderer
Training stage
Pretrained on multiple visual scenes
Objective
Backpropagate the photometric error
n(x)) EiEN(s a (x',x) fig (A (aa-=)), with the input feature function and g filter function. The input consists of particle positions and the corresponding feature. The normalization n(z] can be turned on with the normalization parameter. The per neighbor value a(x", x) a window function to produce smooth response of the convolution under varying particle neighborhoods. It determines the 7th particle feature by aggregating particle features given its neighbors' features. Thereby, the features of the neighbors are weighted with kernel function depending their relative position. The kernel functions themselves are discretized via regular grid with spherical mapping and contain follow the implementation of
the learnable parameters. To cope with boundary interactions, (Ummenhofer el al.) 2020). Specifically, the feature encoder is realized with two separate CCony layers: One encode the fluid particles in the neighborhood of each particle location; The other one is to handle the virtual boundary particles in the same neighborhood. The features extracted from these processes are then concafenated form the inputs for subsequent layers. The following GRUs can summarize the historical information and provide the inferred distribution of latent features about the system. In practice, we use the mean of distributions at the last time step as input of the encoder CConv to avoid excessively noisy inputs.
A.3 NEURAL RENDERER
We present detailed model architecture of PhysNeRF Guan et al. (2022). As shown in Figure 8 the network is based on fully-connected layers and similar NeRF (Mildenhall et al.) (020). Unlike original NeRF, PhysNeRF performs volume rendering according to the geometric distribution of neighboring physical particles along given camera ray. For a sampled point in a ray, ball query is conducted to identify neighboring fluid particles around the sampled point. These neighboring particles are then parameterized to obtain view-independent and view-dependent encodings ep, ea- which are used in the volume rendering subsequently. A multilayer perception (MLP) is trained to map these encodings along with view direction d to volume density a and emitted color e of each sampled point along each ray r(t) original PhysNeRF, we exclude the position of the sampled point from the inputs to the MLP network, which enhances the relationships between the fluid particle encodings and the rendering results. The
+ td, such that (c Ro(ep. ea, d). Different from
14

---

Published as a conference paper at ICLR 2024
Π(ea) + r(d)
RGB
r(ep)
256
256
256
256
256
256
236
256
128
Volume Density
[(cp)
Figure 8: Network architecture of PhysNeRR. The inputs contain view-independent particle encoding Cp. view-dependent particle encoding ca and camera viewing direction d. The outputs are volume density and RGB color c.
view- new-independent encodings Op includes 3 parts: fictitious particle center Рa, soft particle density op and radial deformation UD, calculated as:
1
Fictitious particle center:
Pc
wap
K
p'EN(r(().r.)
Soft particle density:
P Σ
i
Radial deformation:
UD
Σ K
1
I|p' -(()r
Elle'
where (rr()).r.) is the ball query neighborhood within radius of the sampled camera ray point r(() and w; = max(1 - (010111'(1__0)))) [Ip'=x(t))la وا 0). The view-dependent encoding ea includes the normalized
view direction to the fictitious particle center, which is an important reference direction for the network infer the refraction and reflection, calculated
d. = (pc - o) ||pe o
Finally, we take into account all the above physical quantities and derive the view-independent encoding and the view-dependent encoding
ep (Pe, Op. UD), ea do.
(6)
Following Mildenhall et al. 2020 Liu et al.) 2020 Sun et al. 2022), we apply positional encoding r(-)) to every input. In our experiments, and L 4 for T(ea), (d)) We optimize PhysNeRF in a coarse-to-fine manner (Mildenhall et al. 2020). For the fine MLP network, the search radius of particle encoding of ep and ea times the particle radius and we consider 20 fluid particles within this search radius. For the coarse MLP network. the search radius scale and the number of encoding neighbors are set 1.3 times the parameters of the fine network.
set the maximum encoded frequency L 10 for ((ep)
A.4 HYPERPARAMETERS
Table shows the hyperparameters used in experiments.
B TRAINING ALGORITHM
Algorithm giives detailed descriptions of the computation flow of the training process.
15

---

Published as a conference paper at ICLR 2024
Table 7: Hyperparameters of the probabilistic fluid simulator and neural renderer
Name
Symbol
Value
Stage
KI. loss scale
0.1
Distance || I12
0.5
Gaussian latent dimension
Number of CConv blocks of shared feature encoder
3
Number CConv blocks of Particle dynamics module Number of GRU layers
Neighborhood radius
4.5 x particle radius
Stage B
Image size
HxW W
400 400
Coarse sample points
N
64
Fine sample points
Nf
128
Particle encoding radius of ep and ed
X particle radius 20
Partiele encoding neighbors
Coarse encoding scale
1.3
Stage
KL. loss scale
0.01
Algorithm 1: Learning procedures the visual scene
Given:
Multi-view observation [1, pretrained Ta, prior learner Py (Stage and R.p.
2 Unknown:
Particle state x, (z.) ,..,, x and scene-specific physical properties.
3 Estimate initial state x,-1 = (+] ......... 21 as described in Sec, 4.2
Warmup Rr using rendering loss given tt= on multi-view observation on [= 1.
5 Freeze Te, R2g
// Stage B: Optimize scene specifice latent features visual posteriors
Initialize random Gaussian distribution. N(pi (ul,0 Ø*) for each particle i.
8 while not converged do for time slep 1 T do
Sample particle feature for each particle: ~ Ni o*) and produce à = (8l . 2NN).
10
11
Simulate the next step X; = To (x)
Sample camera rays R((P) and predict Ö(r, 1) described in Sec. 4.2 Update the posterior distribution ((a a²) for each particle using rendering loss C. (Sec. 4.2)
12
13
// Stage Finetune the pretrained physical prior learner p
Fix distribution \ii a²) for each particle while not converged do for time step t= .....'.. do
16
18
Sample particle feature for each particle: ž, ~ Po (Zg 81:111, Z, Z)-1).
19
Simulate the next step x; [x(x; 2,)
20
Predict Č(r, t) by volume rendering as described in Sec. 4.2
Update the prior learner Py using the loss function E.s in Eq.
Simulate novel scenes with the learned To and Pv-
16

---

Published as a conference paper at ICLR 2024
C DATASETS
C.I PARTICLE DATASETS
This dataset is used for pretraining the probabilistic fluid simulator in Stage A. Following (Ummen hofer et al., 2020; Pranti et al. 2022), we simulate the dataset with DFSPH (Bender & Koschier 2015) using the SSissHSPlasH framework This simulator generates fluid flows with low-volume compression. The particle dataset contains 600 scenes, with 540 scenes used for training and 60 scenes reserved for the test set. In each scene, the simulator randomly places fluid body of random shape in a cubic box (see Default Boundary in Figure / influence of gravity and undergoes collisions with the boundary. The initial fluid body in each scene is applied with random rotation, scaling, and initial velocity. The simulator randomly samples physical properties viscosity y and density p for fluid bodies from uniform distribution ((0001, 0.4) and U(500, 2000) respectively. The simulation lasts for 4 seconds, which consists of 200 time steps. In general, there are 273 19682 fluid particles in each scene in this dataset.
and the fluid body freely falls under the
C.2 GENERATION OF VISUAL OBSERVATIONS
The visual observations are used in Stages B & C. Under each set of physical parameters. we generate a single 3D video of fluid dynamics with Cuboid (unseen during Stage A) geometry. Each example contains 60 time steps, where the most significant dynamic changes are included. We use Blender (Community 2018) to generate multi-view visual observations. Each fluid dynamic example is captured from 20 randomly sampled viewpoints, with the cameras evenly spaced the upper hemisphere containing the object. The first 50 time steps are used for training, and the last 10 time steps used for the evaluation of future prediction.
C.3 EVALUATION SETS OF VISUAL PHYSICAL INFERENCE
Our model and the baselines are evaluated on 3 challenging novel scene simulation tasks. Figure 9 shows boundaries and fluid geometries used for evaluation.
Unseen Fluid Geometries. We use Standford Bunny, Sphere, Dam Break that are unseen during pretraining fluid bodies. The default boundary used in Stage A is used fluid container. We generate 50 sequences of 60 time steps under each set of physical parameters for evaluation. In each evaluation scene, random fluid geometry chosen and random rotation, scaling, and velocities are imposed on the fluid body.
Unseen Boundary conditions. We use Dam Break fluid and unseen boundary for this task. This evaluation set features an out-of-domain boundary with a slender pillar positioned in the center. In each scene, the Dam Break fluid collapses and strikes the pillar in the container, which during pretraining and from visual observations. Similarly, random rotation, scaling, and velocities are imposed on Dam Break fluid, and 50 sequences of 60 time steps under each set of physical parameters for evaluation.
unseen
The horizontal initial velocity of the fluid body is randomly sampled from a uniform distribution of 2(-2,2), 2). with vertical initial velocity of and scale by a factor randomly sampled from another uniform distribution of (0.88, 1.2). 9 shows the particle number of fluid bodies with no scaling applied. Table Table&Jilluste iillustrraes the features the visual observation scene and evaluation sets.
C.4 DYNAMICS DISCREPANCIES
In this task, we simulate two heterogeneous fluids with different physical properties. The generation of visual examples this task is the same as previous tasks, but with two fluids of different physical properties interacting with each other. The evaluation set contains the same heterogeneous fluids but with unseen fluid geometries as fluid bodies. The evaluation set contains sequences in total. Similarly, random rotation, scaling, and velocities imposed on the fluid bodies in scenes in the evaluation set.
17

---

Published as a conference paper at ICLR 2024
Table 8: The features of the visual observation scene and evaluation sets. Visual means the visual observation scene, Geometry means evaluation sets of unseen fluid geometries, and Boundary means evaluation sets of unseen boundary conditions. Rotation, Scaling, Velocities #Sequence
Boundary Fluid Body
Time Steps
60
Visual
Default
Cuboid
1
(50 for training) )
Standford Bumny.
Geometry Default
50
60
Sphere, Dam Break
Boundary Unseen
Dam Break
50
60
Table 9: Particle number of different fluid bodies with no scaling applied.
Dataset
Cuboid
Stanford Bunny
Sphere
Dam Break
Particle Numbers 6144
8028
8211
10368
D EXPERIMENTAL DETAILS
D.1 EVALUATION METRICS
We adopt the evaluation metric from the literature of CConv Ummenhofer et al. (2020). For all experiments but the short-term predictions in Table we compute the average Euclidean distance from the true particle positions (p:) to the positions the closest predicted particles (f:)):
ΣΣ min Ipl - P:ll2 A
d T N
where T is the prediction time horizon and N is the number of particles. In particular, for the short-term prediction experiments (n + 1 and n + 2) in Table we compute an one-to-one mapping metric, the average error of the predicted particles w.r.t. the corresponding ground truth particles:
1 N
de
Σ1
llti villa
where ji is the predicted position for particle i.
D.2
VISUAL PHYSICAL INFERENCE
D.2.1 BASELINES
Our method is compared with the four baseline models below. For NeuroFluid and system identifi- cation, we adopt PhysNeRF with the same architecture and hyperparameter in Table 7 All models are trained given the single 3D video of fluid dynamics with the Cuboid geometry and evaluated on novel scenes given ground truth initial states.
CConv (Ummenhofer et al.) 2020): Given the initial particle positions and velocities, CConv (Um- menhofer et al., (020) uses a continuous convolution network that is performed 3D space to simulate the particle transitions. However, the input to the CConv model only contains particle position and velocity such that it has a limitation that it can only perform simulation on fluid with identical physical parameters. We this CCony model which has additional input feature, pretrained on particle dataset Sec. C. and directly evaluate this model on novel scene simulations.
NeuroFluid (Guan et al., 2022): A fully differentiable method for fluid dynamics grounding that links particle-based fluid simulation with particle-driven neural rendering in an end-to-end trainable framework. This approach links particle-based fluid simulation with particle-driven neural rendering in an end-to-end trainable framework, such that the two networks can be jointly optimized to obtain reasonable particle representations between them. We adopt the CConv model pretrained on the particle dataset Sec. C.I as the initial transition model and optimize NeuroFluid on Cuboid scene in an end-to-end manner.
18

---

Published as a conference paper at ICLR 2024
Default Boundary
Unseen Boundary
Cuboid
Standford Bunny
Sphere
Dam Break
Figure 9: Visualization of used boundaries and fluid bodies in visual physical inference. The leftmost
container is the default boundary used in each scene in particle datasets and Cuboid.
PAC-NeRF et al. 2023): PAC-NeRF method that estimates physical parameters from multi-view videos with Celerian-Lagrangee representation of neural radiance field and MPM simulator. We feed the physical parameters estimated on Cuboid scene to the MPM simulator and use the given physical parameters to rollout on novel scenes.
System Identification (Sys-ID); For system identification, we train another CConv but with additional features of physical parameters for input including viscosity and density p. Sys-ID optimizes fluid properties by backpropagating the rendering loss through the trained transition network and PhysNeRF. The optimized physical parameters used model inputs simulate novel scenes.
Note that baselines and our model are optimized over the entire visual sequence.
D.2.2 IMPLEMENTATION DETAILS
We train our model and the baselines with multi-view observations on the fluid sequence of the Cuboid geometry. Before Stage B, The PhysNeRF is finetuned on visual observations for 100k steps with learning rate 3e-4 and exponential learning rate decay y = 0.1 given the estimated initial state and multi-view observation of the first frame. After that, freeze R2 and Te (pretrained Dataset) and infer the visual posterior by backpropagating the rendering loss. Then the physical prior learner Pe trained to adapt the inferred visual posterior. The visual posterior latent and physical prior learner are separately optimized for 100k steps and 50k steps in Stage B and Stage C, with learning rate of le-4 and cosine annealing scheduler.
Particle
D.2.3 INITIAL STATE ESTIMATION
We estimate the initial state of fluid particles given multi-view observation the first frame and then feed the estimated initial state to all simulators and the neural renderer. We estimate the initial particle positions using the voxel-based neural rendering technique (Liu al., 2020) [unn et al., [2022] Müller et al. 2022) and maintain an occupancy cache to represent empty vs. nonempty space. At test time, we randomly sample fluid particles within each voxel grid to generate initial particle positions. To maintain spatial consistency between the estimated initial particle positions and the particle density generated by SPlisHSPlaSH (Bender et al. 2022), we employ the fluid particle discretization tools provided in SPlisHSPlasH to calculate the fluid particle count per unit volume. When estimating initial particle positions with new fluid scenes, we adopt the voxel-based neural renderer to predict the spatial volume occupied by the fluid, enabling the subsequent sampling of fluid particles according to the prescribed fluid particle density per unit volume. Since we do not apply random initial velocities to the fluid dynamics in visual examples, the initial velocities are
19

---

Published as a conference paper at ICLR 2024
D.3 EVALUATION OF PROBABILISTIC FLUID SIMULATOR
D.3.1 BASELINES ON PARTICLE DATASET
We compare our probabilisticc fluid simulator with four representative particle simulation approaches. based on GNN, continuous convolution, and Transformer, i.e. DPI-Net (Li et al. 2019). CConv (Um-
menhofer et al. 2020), DMCF (Prantl et al. 2022). and TIE (Shao et al. 2022).
DPI-Net (Li et al. 2019): DPI-Net is particle-basee based simulation method that combines multi-step spatial propagation, a hierarchical particle structure, and dynamic interaction graphs.
CConv (Ummenhofer et al., 2020): The method employs spatial convolutions as the primary differentiable operation to establish connections between particles and their neighbors. It predicts particle features and dynamics a smooth and continuous way.
DMCF (Prantl et al., 2022): DMCF imposes hard constraint on momentum conservation by employing antisymmetrical continuous convolutional layers. It utilizes hierarchical network architecture, a resampling mechanism that ensures temporal coherence.
TIE (Shao al., 2022): A Transformer- bassed model that captures the complex semantics of particle interactions in an edge-free manner. The model adopts decentralized approach to computation, wherein the processing of pairwise particle interactions replaced with per-pan per-particle updates. The original method is conducted on the PyFlex dataset with less number of fluid particles (~ hundreds). However, on Particle Dataset (~ thousands), the experiment on TIE leads to unacceptable memory cost. Therefore, we downsample the fluid particles in each scene of the Purticle Dataset to the ratio of 1/20.
D.3.2 IMPLEMENTATION DETAILS
In Stage A, the probabilistic fluid simulator is trained to predict two future states from two inputs. The ADAM optimizer (Kingma & Bal 2015) is used with an initial learning rate of 0.001 and a batch size of 16 for 50k iterations. We follow previous works (Ummenhofer et al. 2020; Prantl et al. 2022) to set a scheduled learning rate decay where the learning rate is halved every 5k iterations, beginning at iteration 25k. The latent distribution of each particle is an 8-dimensional Gaussian with parameterized mean and standard deviation. The KL regularizer 3 is set as 0.1, shown in Table 7 The experiments are conducted 4 NVIDIA RTX 3090 GPUs. To enhance long-term prediction capability, the probabilistic fluid simulator is trained to predict 5 future states from 5 inputs for experiments in visual physical inference. Additionally, for layers in the shared feature encoder (CConv). This to ensure the learned posterior of latent distribution from visual observation lies in the space of the learned physical prior learner.
tanh
activation function
E EXPERIMENTS WITH NON-ZERO INITIAL VELOCITIES OF THE OBSERVED VISUAL SCENE
To assess the performance of our method in a more general case, specifically, learning from visual observations of fluids with non-zero initial velocities, modify the training scheme in Stage B by randomly initializing {vi=1 Ji=1:N and treating them as trainable parameters. The optimization of (vf=)hi=1:N is carried out concurrently with the optimization of the visual posteriors 2. We evaluate the results against a model trained with true non-zero initial velocities given on the observed scene. In Table 10 we compare future prediction errors on observed scenes of fluids with non-zero initial velocities. In Table IT. we compare the average simulation errors of the two models on novel scenes with new initial geometries and boundaries. We can observe that both models produce comparable results, showcasing the ability of our method to infer uncertain initial velocities of fluids by treating them as optimized variables.
20

---

Published as a conference paper at ICLR 2024
Table 10: Quantitative results of future prediction error d observed scenes of fluids with non-zero initial velocities. We compare the performance of the model trained with optimized velocity against the model with ground truth non-zero initial velocity given.
METHOD
2000, V/ = 0.065 = 1000, V/ = 0.08 p 500, V 0.2
TRAINED WITH TRUE VELOCITY
39.02
39.14
43.57
TRAINED WITH OPTIMIZED VELOCITY
39.53
40.99
45.31
Table 11: Average prediction error d on novel scenes with new initial geometries and boundaries (denoted by Geom. and Bdy.). Each model is trained with single visual observation with non-zero velocity on three physical property sets.
p == 2000, V/ = 0.065 p= 1000, v = 0.08 p= 500, v = 0.2
METHOD
GEOM.
BDY.
GEOM.
BDY.
GEOM.
TRAINED WITH TRUE VELOCITY
33.39
39.74
33.50
38.41
38.52
47.23
TRAINED WITH OPTIMIZED VELOCITY
33.25
39.55
33.72
38.75
38.34
46.90
F ADDITIONAL VISUALIZATION RESULTS
12 provide visualizations of the predicted particles of baselines and our model on the novel scenes with unseen fluid geometries. Figure 13 14 predicted particles of baselines and our model on the novel scenes with unseen boundary conditions. We can see that CConv and NeuroFluid tend to produce noisy predictions on novel scenes. PAC-NeRF produces sporadic predictions that are contrary to the continuous dynamics behavior ground truth simulations. Our model has more reasonable prediction results and can effectively respond to different physical parameters.
Figure 10
13 provide visualizations of the
21

---

Published as a conference paper at ICLR 2024
t=
t=10
F-5
t=
t=30
C=35
C=40
t=50
GT
CConv
Neuro
Fluid
PAC-
NeRF
Ours
Figure 10: Qualitative results of simulated particles from learned simulators on the unseen fluid geometry (Stanford Bunny) with physical parameter م 2000, 0.065, In this scenario. the Stanford Bunny fluid is randomly sampled with a relatively high horizontal initial velocity and initially makes contact with the side of the container. Due to its low viscosity, the fluid flows rapidly down along the container's side.
t=
t=
t=25
t=
t=35
t=40
t=
CConv
Neuro
Fluid
PAC-
NeRF
Ours
Figure 11: Qualitative results of simulated particles from learned simulators on the unseen fluid geometry (Stanford Bunny) with physical parameter م 1000, 0.08. In this scenario. the Stanford Bunny fluid apply to random rotation, presenting an inverted initial state, and is launched into the container with relatively low horizontal initial velocity in projectile motion.
22

---

Published as a conference paper at ICLR 2024
t=
t=
t=25
t=
t=35
t=40
t=
CConv
Neuro
Fluid
PAC-
NeRF
Ours
Figure 12: Qualitative results of simulated particles from learned simulators on the unseen fluid geometry (Stanford Bunny) with physical parameter 500, = 0.2. In this scenario, the Stanford Bunny fluid is randomly sampled with relatively high horizontal initial velocity and makes initial contact with the side of the container. Due to its high viscosity, the fluid slowly flows down along the container's s side.
t=
t=10
t=15
t=
t=25
t=30
t=35
t=
t=45
t=
GT
CConv
Neuro
Fluid
PAC-
NeRF
Ours
Figure 13: Qualitative results of simulated particles from learned simulators the unseen boundary with physical parameter - = 2000, v = 0.065. In umseen boundary scenarios, We use the boundary with slender pillar positioned in the center. In each scenario, the Dam Break fluid collapses and strikes the pillar in the container, which was unseen during pretraining and from visual observations.
23

---

Published as a conference paper at ICLR 2024
t=
t=10
t=15
t=
t=25
t=
c=
t=40
t=45
t=
GT
CConv
Neuro
Fluid
PAC-
NeRF
Ours
Figure 14: Qualitative results of simulated particles from learned simulators the unseen boundary with physical parameter p 1000, v = 0.08. PAC-NeRF produces discontinuous simulation predictions.
t=5
t=10
t=15
t=
t=25
t=30
t=35
t=40
t=
t=50
GT
CConv
Neuro
Fluid
PAC-
NeRF
Ours
Figure 15: Qualitative results of simulated particles from learned simulators on the unseen boundary
with physical parameter p 500, 0.2.
24

---

Published as a conference paper at ICLR 2024
CConv
Figure 16: Qualitative results of predicted long-tern term simulation on the pretrained particle dataset.
We present qualitative results of the predicted particles of the pretrained probabilistic fluid simulator in Figure T6 We can see that our method produces closer prediction results with ground truth. This indicates that our probabilistic fluid simulator shows the capability capture broad hidden physics within the latent space, leveraging spatiotemporal correlation in particle data.
G FURTHER DISCUSSIONS ON REAL-WORLD EXPERIMENTS
The primary focus of this paper is to explore the feasibility of the proposed inference-transfer transfer learning scheme for physies, we first use synthetic visual data to evaluate the proposed method. In real- world experiments, however, we need to use specialized fluid flow measurement techniques, such as Particle Image Velocimetry (PIV), to measure the performance of the model. Most inverse graphics methods not applied to real- world scenarios of fluid dynamics that undergo intense change and visual noises, including prior arts like NeuroFluid (Guan et al.) 2022) and PAC-NeRF (Li et al. 2023). The visual observations of fluid dynamics in real-world scenarios contain more visual noises, such as reflection and refraction, which makes harder to cope with.
Nevertheless, acknowledge that real-world validation is meaningful and challenging. To explore the possibility of applications latent intuitive physics in real-world scenarios with complex and noisy visual observation,
made our best efforts conduct real-world experiments. As the pipeline provided in Figure [7] we capture the dynamics of dyed water in the fluid tanks. We collect RGB images resolution of 1, 1,200 x 900 on the hemisphere the scene. As the presence of refraction and reflection phenomena (which are usually absent in other dynamics contexts) will bring extra burden for the reconstruction of fluid geometries and dynamics, we first adopt NeRFREN (Guo et 2022) to remove the inherent reflection and refraction of fluids and the container. Then we segment fluid body and remove the background using SAM (Kirillov et al.) 2023). The preprocessed images then used to estimate fluid positions using the initial state estimation module. Estimated positions are shown in Figure. I7 Please refer to the supplementary video for a vivid illustration of real-world experiments. However, outstanding challenge of the complete experiment is to acquire high frame-rate rate images with synchronized cameras across multiple viewpoints. Therefore, we have to leave this part for future work.
25

---

Published as a conference paper at ICLR 2024
Cope noises lightening [Done]
with 2D PIV equipment
NeRFREN
Final Goal: Train inverse rendering model for dynamic fluids
Raw Image
Reflection Removal
segmentation
Multiview high frame rate cameras
Estimate inital state via volume rendering [Done] Novel
Depth
Inidial stace estimation
rendering
Rondoring
h
Background Removal
Figure 17: Our pipeline and intermediate results for real-world experiments. Due the requirements of advanced data acquisition equipment, we leave the complete experiment of dynamic scenes to future work (dashed boxes).
26


Reviews:
[{"review_id": "TVPIUUd5lQ", "reviewer": "Reviewer_5z4H", "summary": "In this work, the authors model a probabilistic particle-based fluid simulator-- which they claim to be the first of its kind, to address the problem of learning inaccessible physical parameters of fluids using visual observations such as density, viscosity, pressure, and transferring these visual fluid properties to simulate new scenes with new initial and boundary conditions.", "questions": "- Is the zero initial state velocity a general assumption for all experiments performed? Does this correspond to $(z_{t=1}, \\tilde z_{t=1})$ being zero-initialized?\n- Why is there a difference in the type of posterior used in the pre-training and the inference stages, being particle-based and visual, respectively?\n- Why is the prior learning less prone to overfitting?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- The proposed simulator uses consecutive states/image observations to infer fluid dynamics without complete knowledge of the true physical parameters of the fluid.\n- The simulator showcases a direct benefit of inferring latent space components apart from its role in the data-generating process-- namely in terms of intuitive physical inference.\n- The authors also show real-world experiments on high-resolution images from dyed-water fluid tanks to estimate fluid positions, a challenging task due to reflection, refraction, and the need for high-fps multi-view images. \n- The proposed method can take into account the stochasticity of the underlying fluid dynamics unlike other competitive baselines, and not make fluid category-specific initialization assumptions.", "weaknesses": "- I am not sure about the claim of latent intuitive physics being proposed by them first, especially when similar ideas have existed in the literature on causal representation learning and other intuitive physics methods, albeit in different formats. I think this claim can be softened to say this is the first method to perform this for fluids using 3D exemplars.\n- The authors report that the initial velocities are zero for initial state estimation-- how does this tally up with the initial conditions in the particle datasets considered? Isn't a random rotation, scaling, and initial velocity applied to the initial fluid body?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EahjMRWYht", "reviewer": "Reviewer_QGTH", "summary": "In this manuscript, the authors investigated a transfer learning framework for physics simulation from 3D Video. While conventional methods require known physical states to infer fluid simulation, this study has developed a framework that operates without them, relying on latent features extracted from 3D video inputs.", "questions": "I expect the authors' future direction of real-world videos. It might be beneficial to describe the current state and the challenges more in the main text in addition to the information provided in the Appendix.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "I find this paper interesting and agree that getting inspiration from the concept of intuitive physics plays an essential role in the literature. In addition, modeling a model in terms of video-based processing can contribute to future research on understanding human brain mechanisms.", "weaknesses": "This framework includes many model processing, and it was hard for me to understand the whole structure at the first read. In particular, when I tried to understand the model architecture, I expected Figure 3 to include all the information. However,  Figure 3 lacks information about the neural renderer, and the term \"type-aware preprocessing\" is not used in the main text, leading to comprehension difficulties. Improving the connection between Figures 2 and 3 would enhance readers' understanding.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "g6iObvMMTA", "reviewer": "Reviewer_zRiM", "summary": "This paper proposes a new learning framework to extract hidden physical properties from a single 3D video to simulate the observed fluid in new scenes. It works in a probabilistic setting with a particle-based model. A parameterized prior gets the visual observations as input and tries to approximate a 'visual posterior' obtained from a learned neural renderer. Numerical results demonstrate the effectiveness of the proposed approach in novel scene simulations and the prediction of the future behavior of the observed fluid.", "questions": "I fear I'd have to invest significant extra time in reading this (but also prior) work in order to ask meaningful questions.", "limitations": "", "rating": 6, "confidence": 1, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "I have little background in the main areas of this publication. The overall method appears to be sophisticated, well-designed and sufficiently different from related work as far as I can judge.  The numerical experiments make a solid impression. They test different aspects of the proposed method against competitors, give promising results, ablate at least some design choices (of including stages B and C in the framework), and briefly discuss the extension from simulated to real-world experiments.", "weaknesses": "I found the paper very dense and extremely hard to follow for someone outside of the field (like myself). A more introductory section including a more formal (mathematical) definition of which functions with what kind of inputs and outputs are approximated by certain networks would have helped me.  Similarly, the description of the experiments and their evaluation could have been clearer including the metrics used for the evaluation (Tables 2 and 3 just say \"prediction errors\", Table 4 does not specify a metric at all, and Table 1 refers to \"average position errors on unseen fluid geometries and boundary conditions\", which I cannot understand either). Yet, such aspects might be clear to anyone more familiar with the topic. As I am neither familiar with graphics, nor with particle-based simulations, nor with learned probabilistic models, I am certainly not a good reviewer for this paper (and I informed the area chair accordingly).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hBJuqbqPnt", "reviewer": "Reviewer_eXxy", "summary": "This paper proposes to transfer the attributes of fluid from videos for further simulations in novel scenes. The properties shown in videos are encoded into a latent space, which is further used by pretrained probabilistic physics engine to rolloout the dynamics of fluid in unseen scenes. Experiments show the effectiveness of probabilistic simulation models over baselines as well as the superior performance of physical inference from visual observations.", "questions": "* In Fig 4, is it an accurate reference of the results for the phrase \"row 3-6\". It seems like the results start from row 2.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "* A probabilistic simulation model with better performance than existing models.\n* The proposed method is able to transfer the properties from videos to simulate the fluid dynamics in novel scene.\n* The proposed method obtains superior performance in most cases.", "weaknesses": "1. Since this is a probabilistic model, the predicted dynamics may be different for different sampled feature $\\mathbf{z}_t$. Is there any reference to show the variations of predicted particle states? For example, the model would predict one sequence for multiple times and calculate the standard deviation compared with the ground truth.\n2. What's the unit of measurement in Table 3? Is it millimeter or centimeter?\n3. What's the standard deviation of the prediction errors in Table 2? For example, one scene should include several sequences. The errors may vary from sequence to seqence. What's the variation of the errors?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)