PAC-NeRF: Physics Augmented Continuum Neural Radiance Fields for Geometry-Agnostic System Identification
Authors: Xuan Li
Yi-Ling Qiao
Peter Yichen Chen
Krishna Murthy Jatavallabhula
Ming Lin
Chenfanfu Jiang
Chuang Gan
Venue: ICLR 2023 notable top 25% (2023)
Keywords: System Identification
Neural Radiance Fields
Differentiable Physics
Material Point Method

Abstract: Existing approaches to system identification (estimating the physical parameters of an object) from videos assume known object geometries. This precludes their applicability in a vast majority of scenes where object geometries are complex or unknown. In this work, we aim to identify parameters characterizing a physical system from a set of multi-view videos without any assumption on object geometry or topology. To this end, we propose "Physics Augmented Continuum Neural Radiance Fields" (PAC-NeRF), to estimate both the unknown geometry and physical parameters of highly dynamic objects from multi-view videos. We design PAC-NeRF to only ever produce physically plausible states by enforcing the neural radiance field to follow the conservation laws of continuum mechanics. For this, we design a hybrid Eulerian-Lagrangian representation of the neural radiance field, i.e., we use the Eulerian grid representation for NeRF density and color fields, while advecting the neural radiance fields via Lagrangian particles. This hybrid Eulerian-Lagrangian representation seamlessly blends efficient neural rendering with the material point method (MPM) for robust differentiable physics simulation. We validate the effectiveness of our proposed framework on geometry and physical parameter estimation over a vast range of materials, including elastic bodies, plasticine, sand, Newtonian and non-Newtonian fluids, and demonstrate significant performance gain on most tasks.

Published as a conference paper at ICLR 2023
PAC-NERF:
PHYSICS AUGMENTED CONTINUUM NEURAL RADIANCE FIELDS FOR GEOMETRY-
AGNOSTIC SYSTEM IDENTIFICATION
Xuan Li Yi-Ling Qiao', Peter Yichen Chen 44, Krishna Murthy Jatavallabhula', Ming Lin?, Chenfanfu Jiang', Chuang Gan
"CC Los Angeles, University of Maryland, M CSAIL, "Columbia University, UMass Amherst, 'MIT-IBM Watson Lab xuanlil@math. www.ouullaaau..... wcw.aeaa...a..uaaaaaa...a yilingq@umd.e. edu, pycěcsail. mttiissall.ii........... .iit.ddu, "Krishna@mit.edu, lin0cs unc. edu, cffjiang@math, ucla edu, chuanggeumass.cr
ABSTRACT
Existing approaches to system identification (estimating the physical parameters an object) from videos assume known object geometries. This precludes their applicability in a vast majority of scenes where object geometries complex unknown. In this work, we aim to identify parameters characterizing a physical system from set of multi-view videos without any assumption on object geome- try or lopology. To this end, we propose "Physics Augmented Continuum Neural Radiance Fields" (PAC-NeRF)), estimate both the unknown geometry and phys- ical parameters of highly dynamic objects from multi-view videos. We design PAC-NeRF to only ever produce physically plausible states by enforcing the neu- ral radiance field to follow the conservation laws of continuum mechanics. For this, we design a hybrid Eulerian Lagrangin representation of the neural radi- ance field, i.e., we use the Eulerian grid representation for NeRF density and color fields, while advecting the neural radiance fields via Lagrangian particles. This hybrid
representation seamlessly blends efficient neural ren- dering with the material point method (MPM) for robust differentiable physics simulation. We validate the effectiveness of our proposed framework geometry and physical parameter estimation over a vast range of materials, including clastic bodies, plasticine, sand, Newtonian and newtoiatnnaan fluids, and demonstrate significant performance gain on most tasks!
INTRODUCTION
Inferring the geometric and physical properties of an object directly from visual observations is long-standing challenge in computer vision and artificial intelligence. Current machine vision sys- tems are unable disentangle the geometric structure of the scene, the dynamics of moving objects, and the mechanisms underlying the imaging process an innate cognitive process in human per- ception. For example, by merely watching someone kneading and rolling dough, we are able to disentangle the dough from background clutter, form a predictive model of its dynamies, and esti- mate physical properties, such as its consistency to be able to replicate the recipe. There exists a large body of work on inferring the geometric (extrinsic) structure of the world from multiple im- ages (c.g., structure- structure-from-t motion (Hartley & Zisserman, 2003)). This has been bolstered by recent approaches leveraging differentiable rendering pipelines (Tewari al., 2022) and neural scene repre- sentations (Xie et al., 2022), unlocking a new level of performance and visual realism. On the other hand, approaches to extract the physical (intrinsic) properties (c.g.. mass, friction, viscosity) from images are yet nascent (Jatavallabhula et al., 2020; Ma et al., 2021; Jaques et al., 2022; 2020) all assume full knowledge of the geometric structure of the scene, thereby limiting their applicability.
The key question we ask in this work is "can we recover both the geometric structure and the physi- cal properties of a wide range of objects from multi-view view video sequences"? This dispenses with all
"This work was done during an internship at the MIT-IBM Watson AI Lab
Demos available the project wehpage: https: .ooiiissssoooglloo e www/wwiw/////eeotoooo.c wwC.A///PPPAA/-/

---

Published as a conference paper at ICLR 2023
of the assumptions made by state-of-the of-the-rrttttt approaches to video-based system identification (known geometries in Ma et al. (2021) and additionally rendering configurations in Jatavallabhula et al. (2020)). Additionally, the best performing approaches to recover geometries (but not physical prop- erties) of dynamic objects in videos include variants of neural radiance fields (NeRF) (Mildenhall al., 2020), such as (Pumarola et al., 2021; Tretschk al., 2021; Park al., 2021). However, all such neural representations of dynamic scenes need to learn object dynamics from scratch, requiring significant amount of data do so, while also being uninterpretable. We instead employ differ- entiable physics simulator as a more prescriptive, data-efficient, and generalizable dynamics model; enabling parameter estimation solely from videos.
Our upprouch-Physies Augmented Continuum Neural Radiance Fields (PAC-NeRF)- is novel system identification technique that assumes nothing about the geometric structure of a system. PAC-NeRF is extremely general operating on deformable solids, granular media, plastics, and Newtoniinnnon-Newttoine fluids. PAC-NeRF brings together the best of both worlds; differentiable physics and neural radiance fields for dynamic scenes. By augmenting NeRF with differentiable continuum dynamics model, we obtain a unified model that estimates object geometries and their physical properties in single framework.
Specifically, a PAC-NeRF F is NeRF, comprising a volume density field and a color field, coupled with a velocity field that admits the continuum conservation law: Dt conjunction with a hybrid Bulerian-LLgrangeen
D' 0 (Spencer, 2004). In
formulation, this allows us to advect geometry and appearance attributes to all frames in video sequence, enabling the specification of reconstruc- tion error in the image space. This error term is minimized by gradient-based optimization, lever- aging the differentiability of the entire computation graph, and enables system identification over a wide range of physical systems, where neither the geometry nor the rendering configurations are known. Our hybrid representation considerably speeds up the original MLP-basee NeRF by efficient voxel discretization Sun et al. (2022), and also convemiently handles collisions in continuum simula- tions, following the MPM pipeline Jiang et al. (2015). The joint differentiable mudering-simullan pipeline with unified Fulerian-L Lagrangian conversion is highly optimized for high-performance computing on GPU.
In summary, we make the following contributions.
We propose PAC-NeRF- - dynamic neural radiance field that satisfies the continuum conserva- tion law (Section 3.1).
We introduce hybrid Eulerian- Lagrangia representation, seamlessly blending the Eulerian nature of NeRF with MPM's Lagrangian particle dynamics. (Section 3.3).
Our framework estimates both the geometric structure and physical parameters of a wide variety of complex systems, including elastic materials, plasticine, sand, and Newtoniaan/non-Newtoo fluids, outperforming state-of-the-art approaches by up to two orders of magnitude. (Section 5).
2 RELATED WORK
Neural radiance fields (NeRF), introduced in Mildenhall et al. (2020), are a widely adopted tech- nique encode scene geometry in a compact neural network; enabling photo-realistic rendering and depth estimation from novel views. A comprehensive survey of neural fields is available in Xie et al. (2022). In this work, we adopt the voxel representation proposed by Sun et al. (2022) as these do not require positional information and naturally fit the Eulerian stage of the Material Point Method (MPM) used in our physics prior.
For perception of dynamic scenes, Li et al. (2021) introduce forward and backward motion fields to enforce consistency in the representation space of neighboring frames. D-NeRF (Pumarola al., 2021) introduces a canonical frame with unique neural field for densities and colors, and a time- dependent backward deformation map to query the canonical frame. This representation has since been adopted in Tretschk et al. (2021) and Park et al. (2021). Chu et al. (2022) targets smoke scenes and advects the density field by the velocity field of smoke. This method does not deal with boundary conditions, so it cannot model solids and contact. Guan et al. (2022) present a combination of NeRF with intuitive fluid dynamics leveraging neural simulators; whereas we provided a principled, and interpretible simulation-and-redees framework.
2

---

Published as conference paper at ICLR 2023
Statio Vaxel NeRF
F
Ground Truth
Vosal Volume Readering
Random sampling
P2G
Loos
G2P
(R, T.
Advections
(DutherA))
J
J.G
Phyteal Pazera
Voxal Volume Rondering
Legrungian
Loas
Bulerian
(R)
Figure 1: PAC-NeRF uses both Lagrangian (particle; material-space) and Eulerian (grid; world-space) rep- resentations for an accurate yet tractable computational model of continuum materials. P2G and G2P denote particle-to-grid and grid-to-particle transforms, respectively. Renderable quantities (volume densities, colors) represented in the world space (first fraaee) using voxel NeRF (Sun al., 2022). These quantities bound to particles (by sampling scheme) whose dynamics are simulated by using a differentiable material point method (MPM) (Jiang et al., 2015). The (Eulerian) voxel representation enables efficient collision han- dling and rendering. Since the entire simulation and rendering pipeline are diffcrentiable, rendered (color) images are able optimize both the geometric and physical properties of objects. PAC-NeRF (1) accelerates NeRF rendering with Eulerian representation, (2) lends physical plausibility, interpretability, and data efficiency dynamic scenes, and (3) enables physical parameter estimation for continuum materials.
System identification of soft bodies is an extremely challenging task, owing to its high dimension- ality and presence large deformations. Neural "Sanchez-Gonzalle et al., 2020; Li et al., 2019; Xu et al., 2019) or gradient-fre free methods (Wang et al., 2015; Takahashi & Lin, 2019) struggle to achieve high accuracy on these problems, owing to their black-box nature. Recent progress dif- ferentiable physics simulation has demonstrated great promise Qiao et al. (2021); Du et al. (2021); Rojas et al. (2021); Geilinger et al. (2020); Heiden et al. (2021); Jatavallabhula et al. (2020); Ma et al. (2021), but assumes that watertight mesh of objects available. Recently, Chen et al. (2022); Qiao et al. (2022) estimate elastic object properties from videos. Our hybrid representation and simulation-renderin is general, and speeds up neural rendering by orders of magnitude.
We use the Material Point Method (MPM) due to its ability to handle topology changes and fric- tional contacts; allowing the simulation of wide range of materials, including elastic objects, sand (Klár et al., 2016), to fluids (Jiang et al., 2015) and foam (Yue et al., 2015). While MPM has previously been used for differentiable physics simulation (Hu al., 2020; Huang et al., 2021; Fang et al., 2022), a watertight, non-degenerate mesh model was assumed to be available. Our method solves the long-standing challenge of perceiving geometric and physical properties solely from videos.
3 METHOD
Problem specification: Given a set of (posed) multi-view videos of a dynamic scene, we aim to re- cover (1) an explicit geometric representation, and (2) physical properties (such Young's modulus, fluid viscosity, friction angles, etc.) of the dynamic object of interest.
Unlike existing system identification methods that operate on images, we do not require known ob- ject geometries. Our approach is general and works on wide range of material types (fluids, sand, plasticine, etc.). Our proposed approach, Physics Augmented Continuum Neural Radiance Fields PAC-NeRF, seamlessly blends neural scene representations and explicit differentiable physics en- gines for continuum materials. The core components of PAC-NeRF include a continuum NeRF, particle-grid interconverter, and Lagrangian field. We detail these in this section (see Figure 1).
3.1 CONTINUUM NEURAL RADIANCE FIELDS
Recall that a (static) NeRF comprises a view-inde view-independent volume density field (xx and a view- dependent appearance (color) field c(x,w) w) for each point E R3 and directions (9, σ) S (spherical coordinates). A dynamic (time-dependent) NeRF extends the fields above with an additional time variable / E RRt denoted a axx and c(x, w, () respectively. We use the efficient
3

---

Published as a conference paper at ICLR 2023
voxel discretization from Sun al. (2022) to specify dynamic Eulerian (world-frame) NeRF. Color images are rendered from the above time-dependent fields by sampling points along a ray, for each pixel in the resultant image. The appearance C(r, t) of pixel specified by ray direction r(s) (s E
[Smin; {mux]) is given by the volume rendering integral (Mildenhall et al., 2020)
S."
T(s, t) (r(s). c(r(s), w. ds + Cby T(s),)) (), T(s, t) exp -- S (r(()) ds
C(r,t) t)
(I)
The dynamic NeRF can be trained by enforcing the rendered pixel colors to match those in the video.
N
Lrender
Σ (IC(r,t) FER
C(r, +.Il,,
R
where is the number of frames of videos, C(r,() () is the ground truth color observation.
Additionally, we enforce that the appearance and volume density fields admit conservation laws characterized by the velocity field of the underlying physical system:
Do DI
De
=,
(3)
DL
Dé de with Dr. स + Ve being the material derivative of an arbitrary time-dependent field Φ(x, t). Here, v is velocity field, which in turn must obey momentum conservation for continuum materials
Dv - Dt
where p is the physical density field, T the internal Cauchy stress tensor, and g is the acceler- ation due to gravity. We use the differentiable Material Point Method (Hu et al., 2020) to evolve Equation (4).
3.2 PARTICLE-GRID INTERCONVERTERS
While a Lagrangian representation is ideal for advection by the material point method (MPM), an Eulerian frame is required for rendering the advected particle states to image space. We therefore employ hybrid representation to blend the best of both worlds. A key requirement is to be able to seamlessly traverse the Fulerian (grid) view to the Lagrangian (particle) view (and vice versa).
Denoting the Eulerian and Lagrangian views G and P respectively, field (t) (x(x, (x,(x)))
at time f may be interconverted as follows:
E Wi En Wsp
FP ~
Σ W.
FG
(5)
where i indices grid nodes and indices particles, and Wip is the weight of the trilinear shape function defined node and evaluated at the location of partiele p. We P2G and G2P to denote the particle-to-pe and grid-to-partich conversion processes respectively.
3.3 LAGRANGIAN FIELD
An Eulerian voxel field FG (to) is initialized over the first frame of the sequence. From this field, we use the G2P process to obtain a Lagrangian particle field F= (t(t). We advect this field using an initial guess physical parameter set and the material point method (Equation (3)) to obtain (P(() at 1 to + 5t (where öt is the duration of each simulation timestep). The advected field is then mapped to the Eulerian view using the P2G process, resulting in J (tz). which is employed for collision handling and neural rendering. The Eulerian voxel grid representation used here is at least two orders of magnitude (100x) faster in terms of image rendering time (Sun et al., 2022).
Following Sun al. (2022), the rendering density field and color field at time t are: o t) softplus(Interp(( 8)), c(x, d, t) MLP MML((Intepp/x)) é), d),
(6)
where & is scalar field and è is vector field, both are discretized denotes trilinear interpolation. Advection is performed by first advecting the grids ē and è followed by computing the interpolation functions and evaluating the MI. (or the softplus).
fixed voxel grid. Interp(-)
4

---

Published as a conference paper at ICLR 2023
The initialization of forward simulation requires generating Lagrangian representations of and č. We randomly sample 8 particles within each voxel grid and use Equation (5) to bind the density and color values onto particles. Additionally,
associate with each particle a scalar value ap --e softplus(5p) in (0, 1). A lower a value denotes a smaller contribution the radiance fields. We make particles with lower values of a softer by scaling the physical density field p (Equation (4)) and the physical stress field by factor a² We remove particle p ap < c maxp Op. where set 10-3 constant threshold.
3.4 PAC-NERF FOR GEOMETRY- GEOMETRY-AGNOSTIC AGNOSTIC SYSTEM IDENTIFICATION
Our pipeline for geometry-agnostic system identification comprises three distinct phases for com- putational tractability. We first preprocess data using video matting techniques to extract foreground objects of interest. This is followed by geometry seeding phase, where we employ a coarse-to- fine approach to recover object geometry. The extracted geometry is then used to perform system identification by rendering out future video frames based on guessed set of physical properties, computing an error term with respect to the true videos, and updating the physical properties by gradient-based optimization.
Data Preprocessing: We assume a set of static cameras with known intrinsics and extrinsics. To focus rendering computation the object of interest, et al. (2021) to remove static background objects. This also provides us with a segmentation mask of the foreground object of interest.
the video matting framework in Lin
Geometry Seeding: We first obtain a coarse geometry of the foreground object(s) of interest by employing the static voxel fields. The contents the foreground segmentation mask are rendered to produce an appearance loss term optimized using gradient descent. We employ a coarse-to-fine strategy to en- able easier optimization. We noticed that, as opposed di- rectly rendering the initial voxel radiance field, running the G2P converter followed by P2G converter ensures that ren- dering for all the frames (including the initial frame) is consis- tently based on the same group of particles (implicitly provid- ing set of consistent correspondences across time). In addi- tion to the rendering loss Equation (2), we employ a surface regularizer to regularize the geometric density field:
Without surface regularizer
With surface regularizer
Figure 2: Our surface regularizer im- proves reconstruction quality, by pro- ducing a tight-fit shape the segmen- tation mask.
Ar
Lauff Σ clamp(ap, 10 100,,10 10
This regularizer minimizes the total surface area, and as shown in Figure 2, this tends to improve the quality of reconstructed geometries by making the reconstructed point cloud more compact and fit the ground truth boundary more closely.
System Identification: Upon convergence of the geometry seeding phase, we freeze the parame- ters of the field (for the initial time step lo). To minimize scenarios where one or more variables of interest become unobservable under unknown initial conditions, we use the first 2-3 frames to estimate the initial velocity of observed particles. We then optimize for physical parameters with each subsequent frame. To mitigate convergence issues in parameter spaces with larger degrees of freedom, we found it helpful to warm start the optimizer after initial collision events, followed by an optimization over the entire sequence.
4 IMPLEMENTATION DETAILS
The architecture of voxel discretization of NeRF follows Sun et al. (2022), which stores density value and color feature within 160 voxels, only contraining extra 2-layer MLP with a hidden dimension 128 for view-dependent color fields. The dimension of color feature on each voxel is 12. Positional embedding is applied to the inputs (query position, view direction and color feature) to the shallow MLP. leading to a input dimension 39. Our differentiable MPM is using DiffTaichi (Hu et al., 2020). Both the rendering and simulation optimized for GPU. The entire training
5

---

Published as a conference paper at ICLR 2023
Droplet
Cream
Cat
Torus
Bird
Letter
Toothpaste
Playdoh
Trophy
Bird
ICLR
Figure 3: The photorealistic dataset used to evaluate system identification and geometry estimation. The dataset includes a variety of continuum materials, including Newtonian fluids (Droplet, Letter), non-N non-Newtonian fluids (Cream, Toothpaste), granular media (Trophy). deformable solids (Torus, Bird), and plasticine (Cat, Playdoh). All objects freely fall under the influence gravity, undergoing collisions. Objects rendered under complex environmental lighting conditions for photorealism.
takes ~ 1.5 hours on single Nvidia 3090 GPU. Simulatioon+rendering one frame takes ~ 18 (vs. 10mán for Chen et al. (2022)).
EXPERIMENTS
We conduct various experiments to study the efficacy of PAC-NeRF on system identification tasks and find that:
PAC-NeRF recover high-quality object geometries solely from videos.
PAC-NeRF performs significantly better on system identification tasks compared to fully learned approaches.
PAC-NeRF alleviates the assumptions that other techniques require (i.e., known object geometry), while outperforming them.
Purely pixel-based loss functions provide rich gradients that enable physical parameter estimation.
5.1 EXPERIMENT SETUP
Dataset: evaluate different system identification methods, we simulate and render wide range of objects using a photo-realstic simulation engine with varying environment lighting conditions and ground textures. Our dataset includes deformable objects, plasties, granular media, Newtonian and non-Newtonian fluids. Figure 3 demonstrates a few example scenarios. In each scene, objects freely fall under the influence of gravity, undergoing (self and external) collisions. For convenience, we assume that the collision objects such as the ground plane are known (however, this also be easily estimated from observed images). Each scene captured from uniformly sampled viewpoints, with the cameras evenly spaced the upper hemisphere containing the object. All ground truth simulation data are generated by MLS-MPM framework (Hu et al., 2018b).
Our differentiable MPM implementation supports five kinds of common materials, including elasticity, plasticine, granular media (e.g., sand), Newtonian fluids and non- Newtonian fluids.
Physical Parameters:
Elasticity: Young's modulus (E) (material stiffness). Poisson's ration (1) (ability to preserve volume under deformation).
Plasticine: Young's modulus (E), Poisson's ration (v), and yield stress (ry) (stress required to cause permanent déformmation/yelldinn).
Newtonian fluid: fluid viscosity (μ) (opposition to velocity change), bulk modulus (x) (ability to preserve volume).

---

Published as a conference paper at ICLR 2023
Newtonian
Non-Newtonian
Elasticity
Plasticine
Sand
Ground Truth
ICLR
ICLR R
PAC-NoRF
ICLR ICLR
118 Torus
Particle View
ICLR ICLD
Droplet
Lettrr
Cream
Toothpaste
Bird
Playdoh
Cat
Trophy
Our experiments cover a wide range of continuums, from Newtonian fluids and non-Newtonian fluids, to elastic/plastic solids and granular media (sand). Existing approaches (including dynamic NeRF and variants) are unable to reconstruct these highly dynamic objects. PAC-NeRF not only reconstructs them with high accuracy but also preciscly determines the underlying physical properties.
Figure 4: Qualitative results:
Guess
Optimized
Droplet
# 1.x
2.09 10 1.08
2001 10P
Leteer
1.8 10
50,85 6 1.35 10
100, 10
Cream
108
10 10,
10 == 10 .Ty
3.16
10..0x10..11m 10..
10
10a = 1==00x00=-1111
Touihpaste
10",
10. =
10",
.7y-7
10",
200
Torus
E 10 2 0.1
E 1.04 x 11. 00.222
10% 0.3
Bird
== 102 0 0.1
E 2.78 x 16 -0.273
E 3 103,
0.3
Playdoh
E= 10 0
10
8
3.84
10", 0.2722
2 10?, 0.3, 1.54 10*
Cal
10
10
0.2033 TV -3.7
10",
ry
Truphy
O9.. =
P... 36,1"
0ne 00
Table 1: PAC-NeRF estimates physical parameters very accurately.
Non-Newtonian fluid: shear modulus (μ), bulk modulus (k), yield stress (ry), and plasticity vis- cosity (1) (decayed temporary resistance to yielding).
Sand: friction angle (0frie) (proportionality constant determining slope of sand pile).
System Identification Pipeline: We first train static voxel NeRF using data from the first frame (following Sun et al. (2022)) with the Adam optimizer. The initial velocity estimator uses L-BFGS, which we experimentally find to be better than Adam for this sub-task. For all other physical pa- rameters of interest, we use the Adam optimizer.
5.2 SYSTEM IDENTIFICATION EVALUATION
Synthetic Data:
We report system identification results using PAC-NeRF over synthetic problem instances. The qualitative results are shown in Figure 4. And the quantitative results are listed in Table Each row lists the initial guess, the values obtained after optimization, and the ground truth. We see that, in each case, the optimized physical parameters closely agree with the ground truth.
Real Data: We also evaluate PAC-NeRF on real-world data (Figure 5). We built capture system comprising four synchronized Intel RealSense D455 cameras capable of streaming RGB images at resolution 640 x 480 and a rate 60 frames per second. Note that we DO NOT use any depth data recorded from these cameras in experiments, for fair evaluation. We captured deformable ball falling onto a table and manually segmented the data. Our reconstructed scene qualitatively matches the target videos. We note that, due to the limited number of views (here, 4) could

---

Published as a conference paper at ICLR 2023
Real Data
PAC-NORY
Particle-Viee View
Figure 5;
We test PAC-NeRF on set real-world multiview videos. Our reconstructed seene qualitatively matches the target videos, which validates that the rendering loss is effectively minimized.
acquire, there are slight emors in surface geometry estimation. These issues may be mitigated by increasing the number of views from which data captured.
5.3 COMPARISONS
Since there no existing approaches that estimate both geometry and material properties from videos, we make a best-effort comparison by blending combinations of state-of-the-art approaches for each sub-task. For each material type discussed above, generate 10 problem instances by varying object orientations, initial velocities, and physical parameters. The comparisons are con- ducted on this dataset. Note that our approach (and most of the baselines herein) does not require separate training phase they perform inference-ting optimization on each test sequence.
5.3.1 APPROACHES EVALUATED
Multi-view LSTM:
To evaluate amortized physical parameter inference schemes, we use a pre- trained ResNet feature extractor extract features from all views and feed them into 2-layer LSTM. This baseline uses privileged information in the form of training video sequences (while all other baselines begin with random initialization each test sequence).
To assess the impact of conservation law enforcement in PAC-NeRF, we com- pare with VEO (Chen et al., 2022) (a best-effort comparison) by implementing D-NeRF (dynamic NeRF) and integrating it with our differentiable simulator. A dynamic voxel NeRF learns both the forward and backward deformation field for each frame. The learned forward deformation field then used to provide the differentiable simulator with 3D supervision.
D-NeRF DiffSim:
We also compare with VSi state-of state-of-the-art approach for estimating physical properties with both the geometry and rendering configuration known. (By contrast, we assume neither.) We implement& a best-effort comparison where we use static voxel NeRF to recover the geometry and directly provide the rendering color configuration. Note that VSm only supports FEM simulation for continuum materials. Therefore, this baseline only runs on a small subset of our scenarios. Moreover, to run this baseline, we extract surface mesh from reconstructed point cloud from voxel NeRF and then use TetWild (Hu et al., 201 8a) to generate a tetrahedral mesh. Further, manually choose optimal camera viewing direction to facilitate optimization.
NeRF Sim:
Random: We also evaluate the performance of these approaches against random chance, by em- ploying baseline that predicts a uniformly randomly chosen value the parameter ranges used.
We implement an oracle that uses privileged 3D geometric information. The oracle knows the ground truth point clouds, and employs 3D supervision (Chamfer distance) to infer physical properties. Of the 10 problem instances for each material type, we run the oracle on 4.
Oracle:
5.3.2 ANALYSIS OF RESULTS
The performances of all approaches on the system identification tasks are listed in Table 2. For each material, we report the mean absolute error and its standard deviation over the evaluated instances. We see that compared to all other methods, PAC-NeRF achieves the best results 14 of the 17 categories of physical properties estimated.
The multi-view LSTM method is adapted from DensePhysNet (Xu et al., 2019) and learns the map- ping from videos physical parameters implicitly. Notably, this baseline requires privileged infor- mation in the form of training sequences not available to the other approaches.
The D-NeRF+Diffsim is adapted from VEO (Chen et al., 2022), where the forward deformation field used to optimize a differentiable simulation. The accuracy of this approach therefore relies on the quality of the learned forward deformation field. However, this learned deformation field (Tretschk
8

---

Published as a conference paper at ICLR 2023
D-NeRF DilSSi
NcRF + VSim
Ours
Multi-iew LSTM
Random
Oracle (x10
log (u)
9.26
6.95
Nowtonian
(k)
16.7 = 5.37
28.2 19,8
> 85.2
Not supported
72.8 2 42.5
258.9 25.2
0.86 1: 1.45
23.1 17.8
25.5
35.5 22.6
2.82 à: 1.72
logse (e)
24.1 21,9
39.4 26.9
276.7
34.2 19.9
24.0
17.1
Non- Nowtonian
Jog
44.0 + 26.3
25.1 + 22.6
262.5
61.6 + 40.4
$2.90
21.0
logaul [Y-
359.6
supported
15.0
9.75
28.7 23.3
30.9 +1111
86.1
64.3 43.7
90.0 + 37.2
0.29 0.13
24.4 14.2
25.9
52.2 21.6
2.60 0.39
logse
3/02 3.72
17.7 9.25
437.5
1516 4: 42.6
96.2 46.9
4.39 4.54
Elusticity
4,35 + 5.08
81.8 68.4
7.67
16.4
10.9 6.37
3.05 + 88
25.4
2.69
logge LE)
83.8 68.4
11.1
23.2
12.9 38.0
56.2 30.7
Plasticine
Ty y
18.9 15.7
11.2
5,40 3.49
17.3
10.36
Not supported
10.6 6.15
4.44 3.61
8:13 3.61
0,56 = 0.17
22.0 14.8
384.6
47.7 17.53
4.06 1.8T
Sund
0.21 = 0.08
1.10
14.6 11 6.25
20.1 2.06
10.6
Not supported
22.5 14.7
38.72:21.1
1.17 = 0.41
0.44 + 0.42
Table 2: System identification performance: Means and the standard derivations absolute emrors are re- ported for each metric. We compare with five baselines, including a pure vision- based method (ResNet+LSTM regressionn, D--eRFF-DfffSim (similar to VEO (Chen et al., 2022)), NeRF+ Sim (Jatavallabhula et al., 2020), random sampling from parameter distribution, and oracle point cloud plus Chamfer distance minimization. Our method obtains the best results (highlighted in boldface font) in 14/17 entries, excluding the Oracle.
et al., 2021) cannot guarantee physical correctness, unlike PAC-NeRF which is constrained by the conservation laws (Equation (3) and Equation (4)). In our experiments, we observed that this leads to very noisy deformation, and impedes system identification performance. Due to these instabilities of this approach, we only run this on one scenario per material type and report its performance.
The NeRF+ Sim (Jatavallabhula et al., 2020) baseline only supports FEM simulations for elastic materials and sensitive time integration step size. To stabilize the symplectic FEM and the con- tact model used in Sim, tens of thousands of substeps are required to simulate the entire sequence. Notwithstanding the errors induced due to geometric inaccuracies and unknown rendering configu- rations, errors accumulated from long timestepping sequences also contribute to the failure of Vim on our scenarios. PAC-NeRF, with its Eulerian- Lagerain-a representation, is more robust under larger deformations (e.g., fluids and sand) and permits larger time step sizes than FEM simulation with a symplectic integrator (used in Vim)
The Oracle baseline assumes known 3D (point cloud) geometry and computes a Chamfer distance eтror term per-timestep. We note, surprisingly, that in several scenarios, a pixel-wise color loss outperforms this 3D supervision signal. The results show the effectiveness of the differentiable rendering-simulatinn pipeline, where a 2D pixel-level supervision better optimizes the physical pa- rameters, opposed to 3D Chamfer distance metric.
CONCLUSION
We presented Physics Augmented Continuum Neural Radiance Fields (PAC-NeRF), a novel repre- sentation for system identification without geometry priors. The continuum reformulation of dy- namic NeRF naturally unifies physical simulation and rendering, enabling differentiable simulators estimate both geometry and physical properties from image space signals (such as multi-view videas). A hybrid representation is used to discretize our PAC-NeRF implementation, which lever- ages the high efficiency of Eulerian voxel-base based NeRF and the flexibility of MPM.
Despite the promising results, this work has some limitations. It assumes the availability of syn- chronized and accurately calibrated cameras to ensure good-quality reconstruction using NeRF, The scenes used in this work should also be easily amenable to video matting (background removal). Moreover, this work assumes the underlying physical phenomenon follows continuum mechanics and cannot automatically distinguish between different materials. Future work could focus ex- tending the MPM framework beyond volumetric continuum materials, such as cloth, using neural constitutive models (Li al., 2022), implicit MPM for stiff materials, and integrating other differ- entiable simulators like articulated body simulators. Additionally, incorporating interactions with rigid bodies could enable manipulation tasks (Huang et al., 2021) NeRF-represented soft-bodies.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGMENTS
This project was supported by the DARPA MCS program, MIT-IBM Watson AI Lab, NSF CAREER 2153851, CCF2153863, ECCS-20237880 and gift funding from MERL, Cisco, and Amazon.
REFERENCES
Hsiao-yu Chen, Edith Tretschk, Tuur Stuyck, Petr Kadlecek, Ladislav Kavan, Etienne Vouga, and Christoph Lassner. Virtual elastic objects. In Proceedings of the IEEE/CVE Conference on Com- puter Vision and Puttern Recognition, pp. 15827-15837, 2022. 3, 8,9
Mengyu Chu, Lingjie Liu, Quan Zheng, Erik Franz, Hans-Peter Seidel, Christian Theobalt, and Rhaleb Zayer. Physics informed neural fields for smoke reconstruction with sparse data. ACM Transactions Graphics (TOG), 41(4):1-14, 2022. 2
Tao Du, Kui Wu, Pingchuan Ma, Sebastien Wah, Andrew Spielberg, Daniela Rus, and Wojeiech Matusik. DifIPD: Differentiable projective dynamics with contact. arXiv:2101.05917 2021. 3
Yu Fang. Jiancheng Liu, Mingrui Zhang, Jiasheng Zhang, Yidong Ma, Minchen Li, Yuanming Hu, Chenfanfu Jiang, and Tiantian Liu. Complex locomotion skill learning via differentiable physics. arXiv preprint arXiv:2206.02341, 2022. 3
Moritz Geilinger, David Hahn, Jonas Zehnder, Moritz Bächer, Bernhard Thomaszewski, and Stelian Coros. ADD: Analytically differentiable dynamics for multi-hody systems with frictional contact. ACM Transactions Graphics (TOG), 39(6), 2020. 3
Shanyan Guan, Huayu Deng. Yunbo Wang. and Xiaokang Yang. Neurofluid: Fluid dynamics grounding with particle-driven neural radiance fields. In ICML, 2022. 2
Richard Hartley and Andrew Zisserman. Multiple view geometry in computer vision. Cambridge university press, 2003.
Eric Heiden, Miles Macklin, Yashraj S Narang, Dieter Fox, Animesh Garg, and Fabio Ramos. Di- SECt: A Differentiable Simulation Engine for Autonomous Robotic Cutting. In Proceedings of Robotics: Science and Systems, Virtual, July 2021. doi: 10.15607/RSS.20 NASSS.222..... 1021.XV11.067.
Yixin Hu, Qingnan Zhou, Xifeng Gao, Alec Jacobson, Denis Zorin, and Daniele Panozzo. Tetrahe- dral meshing the wild. ACM Trans. Graph., 37 2018a.
Yuanming Hu, Yu Fang. Ziheng Ge, Ziyin Qu, Yixin Zhu, Andre Pradhana, and Chenfanfu Jiang. A moving least squares material point method with displacement discontinuity and two-way rigid body coupling. ACM Transactions on Graphics (TOG), 37(4):1-14, 2018b.
Yuanming Hu, Luke Anderson, Tzu-Mao Li, Qi Sun, Nathan Carr, Jonathan Ragan-Kelley, and Frédo Durand. Difftaichi: Differentiable programming for physical simulation. ICLR, 2020. 3,
5.5
Zhiao Huang, Yuanming Hu, Tao Du, Siyuan Zhou, Hao Su, Joshua B. Tenenbaum, and Chuang Gan. Plasticinelab: soft-body manipulation benchmark with differentiable physics. In Interna- tional Conference Learning Representations, 2021. URL https: https///ppeerrieee net/ forum? xxiid=xxccdRREDD.. 3,
Miguel Jaques, Michael Burke, and Timothy Hospedales. Physios-asiimeese-grapiisss Unsuper- vised physical parameter estimation from video. ICLRR 2020.
Miguel Jaques, Martin Asenov, Michael Burke, and Timothy Hospedales. Vision-based system identification and 3d keypoint discovery using dynamics constraints. In Learning for Dynamics and Control Conference, pp. 316-329. PMLR, 2022.
Krishna Murthy Jatavallabhula, Miles Macklin, Florian Golemo, Vikram Voleti, Linda Petrini, Mar- tin Weiss, Breandan Considine, Jérôme Purent-Lévespp Kevin Xie, Kenny Erleben. et al. grad- sim: Differentiable simulation for system identification and visuomotor control. In International Conference Learning Representations, 2020. 1, 2, 3,9
10

---

Published as a conference paper at ICLR 2023
Chenfanfu Jiang, Craig Schroeder, Andrew Selle, Joseph Teran, and Alexey Stomakhin. The affine particle-in-cel method. ACM Transactions on Graphies (TOG), 34(4):1-10, 2015. 2015. 2,3
Gergely Klár, Theodore Gast, Andre Pradhana, Chuyuan Fu, Craig Schroeder, Chenfanfu Jiang, and Drucker-prager prager elastoplasticity for sand animation. ACM Transactions on Graphics (TOG), 35(4):1-12, 2016. 3, 13
Joseph Teran. Drucker-
Xuan Li, Yadi Cao, Minchen Li, Yin Yang, Craig Schroeder, and Chenfanfu Jiang. Plasticitynet: Learning simulate metal, sand, and snow for optimization time integration. In Advances in Neural Information Processing Systems, 2022.
Yunzhu Li, Jiajun Wu, Russ Tedrake, Joshua B. Tenenbaum, and Antonio Torralba. Learning par- ticle dynamics for manipulating rigid bodies, deformable objects, and fluids. In International Conference Learning Representations, 2019. 3
Zhengqi Li, Simon Niklaus, Noah Snavely, and Oliver Wang. Neural scene flow fields for space- time view synthesis of dynamic scenes. In Proceedings the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2021. 2
Shanchuan Lin, Andrey Ryabtsev, Soumyadip Sengupta, Brian L Curless, Steven M Seitz, and Ira Real-time high-resolution background matting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8762-8771, 2021.
Pingchuan Ma, Tao Du, Joshua B Tenenbaum. Wojeiech Matusik, and Chuang Gan. Risp: candering-invariann state predictor with differentiable simulation and rendering for cross-domain parameter estimation. In International Conference on Learning Representations, 2021. 1, 1,2,3
Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In ECCV, 2020.
2,
Keunhong Park, Utkarsh Sinha, Jonathan T. Barron, Sofien Bouaziz, Dan B Goldman, Steven M. Seitz, and Ricardo Martin-Brualla Nerfies: Deformable neural radiance fields. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), pp. 5865-5874, October 2021. 2
Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francesc Moreno-Noguer. D-nerf: Neural radiance fields for dynamic scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. (0318-10327, June 2021.
Yi-Ling Qiao, Alexander Gao, and Ming C. Lin. Neuphysics: Editable neural geometry and physics from monocular videos. In Conference Neural Information Processing Systems (NeurIPS), 2022. 3
Yiling Qiao, Junbang Liang, Vladlen Koltun, and Ming Lin. Differentiable simulation of soft multi- body systems. Advances Neural Information Processing Systems, 34:17123-17135, 2021.
Junior Rojas, Eftychios Sifakis, and Ladislav Kavan. Differentiable implicit soft-body physics. arXiv preprint arXiv:2102.0579,, 2021. 3
Alvaro Sanchez-Gonzale Jonathan Godwin, Tobias Pfaff, Rex Ying, Jure Leskovee, and Peter Battaglia. Learning to simulate complex physics with graph networks. In International Confer- ence on Machine Learning, pp. 8459-8468. PMLR, 2020. 3
Anthony James Merrill Spencer. Continuam mechanics. Courier Corporation, 2004. 2
Cheng Sun, Min Sun, and Awann-Tzone Chen. Direct voxel grid optimization: Super-fast vergence for radiance fields reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5459-5469, 2022. 4,5,7
Tetsuya Takahashi and Ming C Lin. Video-guided real-to-virtuse parameter transfer for viscous fluids. ACM Transactions Graphics (TOG), 38(6):1-12, 2019.

---

Published as a conference paper at ICLR 2023
Ayush Tewari, Justus Thies. Ben Mildenhall, Pratul Srinivasan, Edgar Tretschk, W Yifan, Christoph Lassner, Vincent Sitzmann, Ricardo Martin-Brualla, Stephen Lombardi, et al. Advances in neural rendering. In Computer Graphics Forum, volume 41, pp. 703-735. Wiley Online Library, 2022.
Edgar Tretschk, Ayush Tewari, Vladislav Golyanik, Michael Zollhöfer, Christoph Lassner, and Christian Theobalt. Non-rigid neural radiance fields: Reconstruction and novel view synthesis of a dynamic scene from monocular video. In JEEE International Conference on Computer Vi- sion (ICCV). IEEE, 2021. 2.8
Bin Wang, Longhua Wu, KangKang Yin, Uri M Ascher, Libin Liu, and Hui Huang. Deformation capture and modeling of soft objects. ACM Trans. Graph., 34(4):94-1, 2015.
Yiheng Xie, Towaki Takikawa, Shunsuke Saito, Or Litany, Shiqin Yan, Numair Khan, Federico Tombari, James Tompkin, Vincent Sitzmann, and Srinath Sridhar. Neural fields in visual comput- ing and beyond. Computer Graphics Forum, 2022. ISSN 1467-8659. doi: 10.1111/cggh 1/505 1,
Zhenjia Xu, Jiajun Wu, Andy Zeng, Joshua B Tenenbaum, and Shuran Song. Densephysnet: Learn- ing dense physical object representations via multi-step dynamic interactions. arXiv preprint arXiv:/906.0336 03855, 2019. 3,8 8
Yonghao Yue, Breannan Smith, Christopher Batty, Changxi Zheng, and Eitan Grinspun. Continuum foam: A material point method for deeendet flows. ACM Transactions on Graphics (TOG), 34(5):1-20, 2015. 3, 13
A APPENDIX
A.1 PHYSICAL MODELS
MPM supports all kinds of constitutive models and plasticity models. In continuum physics frame- work, the internal force is provided by Cauchy stress T, tensor field defined by deformation gradi- F. The deformation gradient tracked MPM particles to measure their distortion compared to its initial state. For plasticity, the deformation gradient is constrained within an elastic region. A return mapping applied to pull deformation gradient back onto the yield surface.
Elasticity We use reo-Hookean elasticity to model elastic objects. The Cauchy stress for this model defined by
(T(F) #(FF + (Alog(J) - p)I.
(8)
where J det(F). p, Å are Lame parameters, related to Young's modulus (E) and Poisson's 's ratio (V) by
E 2(1 2(1+
vE
#
A=
(1+ 2v)
Newtonian Fluid We use J-based fluid combined with viscosity term to model Newtonian fluid. The stress for this model is defined by
1 JT(F) = 1n (V + _o
(10)
Plasticine simulate plasticine, we use Venant-Kirchhofff (StVK) constitutive model combine with von-Mises plastic return mapping. The stress associated with the model is defined by
(T U( TTrc)))
(11)
with F UEV being the SVD decomposition of F and log( Σ) being the Hencky strain.
The von-Mises yielding condition is
87 ||
TY > 0.
(12)
24
12

---

Published as a conference paper at ICLR 2023
where é is the normalized Hencky strain. Ty is the yield stress, which is the parameter for this model. by > means the deformation gradient is outside the elastic region. The deformation gradient will be projected back onto the boundary of clastic region (yield surface) by the following return mapping:
F (E
87 S 0
Z(F)
(13)
2 V )
otherwise
Non-Newtonian Fluid
Non-Newtonian fluid also has yield stress. We use viscoplastic model (Yue et al., 2015) to model non-Newtonian fluid. We still use von-Mises criteria to define elastic region. However, the existence of viscoplastic means the deformation will not be directly projected back onto the yield surface. Define
# Tr(E2)
d
(14)
67
||||
+
2PAT
F U (Uxp
87 <
2(F)
(15)
é Tr(e)1)VT
otherwise
2p
Sand Drucker-Prage Pracerr yield criteria is used to simulate granular materials (Klár et al., 2016). The underlying constitutive model is still StVK. The yielding criteria is defined by
(d) 2p)) tr(c)
tr(c) O, or 87 ||
> 0.
(16)
2p
where = =
sin fvic
and frric is the friction angle. We the following return mapping in
33 sin fric
sand simulations:
UV F
tr(c) > 0
3(F)
8y S 0, tr(e) 50. otherwise
(17)
U Uexp (E 8y T6 V
B COMPLEX BOUNDARY CONDITIONS
Pre-known boundary conditions are trivially to be added MPM simulations, just like the ground. Here we shown examples of elastic rope falling onto two rigid cylinders in Figure 6. The initial guess, optimized values and the ground truth values of physical parameters are listed Table
bame: woc Phowea CRNERTTA
Figure 6: Examples clastic ropes falling onto cylinders. The top row the ground truth vidco. PAC-NeRF is trained on the segmented data, whose rendering outputs are visualized in the middle row. The bottom row is the reconstructed MPM particles.
C RECONSTRUCTION QUALITY COMPARISON WITH D-NERF
When evaluating the D-NeRF+Diffsim baseline, we observe that D-NeRF sensitive to sudden, large deformations (see Figure 7). As the simulation progresses, D-NeRF suffers from artificial
13

---

Published as a conference paper at ICLR 2023
Initial Guess
Optimized
Ground Truth
Rope (Short)
E'
10
0.4
E 1.12 x 10 0.22
E 10 0.3
Rupe (Long)
E
10
v
0.4
E 1.09 x 10 0.31
E
10 v 0.3
Table 3: Quantitative results of clastic rope example.
D.NeRT
PAC-NeRF
Figure 7: D-NeRF suffers from major artifacts as the object undergoes sudden large deformations. Notice how the object becomes partially absent in the middle two frames.
extreme deformation, resulting in the object breaking apart. This degrades the simulation to the extent that parts of the object even partially disappear (violating physical plausibility). This artifact due to the learned backward deformation map points to an empty area the canonical space, resulting in zero volume density. By contrast, our PAC-NeRF constrains forward deformations to follow physical conservation law and does not experience this kind of unphysical result.
D QUALITATIVE COMPARISON WITH VIM ON REAL- WARLD DATA
We also tried Vim on our real-world data. The qualitative comparison between PAC-NeRF and Vim is shown in Figure 8. As we do in the baseline comparisons, we extract surface mesh from our reconstructed point cloud from voxel NeRF and then generate tetrahedral mesh, manually pick optimal camera and set approximate rendering configuration. As we discussed in baseline comparison, explicit FEM used in Sim suffers from tiny time steps, leading to significant numerical errors in backpropagations. Combined with errors from the geometry reconstructions and the approximate rendering configurations, all these factors contribute to the failure of Vim..
Real Data
PAC-NeRF
GradSim
Figure 8: Qualitative comparison with Vim real-world data.
14


Reviews:
[{"review_id": "sDzvqm7A3_", "reviewer": "Reviewer_QRFe", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a new NeRF model, PAC-NeRF, that incorporate physical particle dynamic into the estimation to simultaneously estimate particle parameters and geometries. The particle dynamics are assumed to follow the conservation laws of continuum mechanics, and a hybrid Eulerian-Lagrangian representation is proposed to solve this dynamic system. The proposed method was trained and evaluated using synthetic data, and with the high quality inputs that satisfied the particle system assumption, it demonstrate better results than the previous dynamic-NeRF (D-NeRF) model.", "summary_of_the_review": "Please check my comments above. I like this paper and I think this work is of high quality to be accepted.", "strength_and_weaknesses": "Strength: Including particle dynamics into NeRF model is novel and it is a very useful to extend NeRF reconstruction beyond static scene/rigid object movements.\n\nWeakness: The proposed method was only evaluated using synthetic data. It would be great if the authors can include some real-world examples as the real-world observation might not strictly follow the particle dynamic system studied in this paper. Yet, I would expect to see some interesting reconstructions based on the proposed model.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "poWIP1kvAZ_", "reviewer": "Reviewer_jLz5", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents a method to recover a scene's geometry and physical parameters from a collection of images. It combines a neural radiance field method with a differentiable physics engine to jointly recover a deforming object's geometry, color, and physical parameters using continuum mechanics modeling. \n\nThe NERF model is dynamic to adapt the geometry over time and is based on the direct voxel grid optimization proposed by Sun et al. (2022) that greatly speeds up training a NERF. The differentiable physics engine is based on the MPM included in DIFFTAICHI. \n\nThe paper describes the required steps to convert from the NERF representation of the geometry to particles that move with physical constraints. The NERF and physical parameters are recovered by optimizing a photometric cost function where the physical constraints are implicitly included in the dynamic part of the NERF.    \n\nThe experiments use synthetically generated scenes with various continuum materials, including Newtonian fluids, non-Newtonian fluids, granular media, deformable solids, and plasticine.\n\n", "summary_of_the_review": "The paper tackles a very difficult problem by combining two recent methods, mixing NERF with continuum mechanics models to identify the physical parameters of a scene from monocular images. \n\nIt is well-written and sufficiently clear.\n\nThe results are convincing\n\n\n ", "strength_and_weaknesses": "Strengths\n\n- The combination of NERF and differentiable continuum mechanics to identify a system's physical parameters and geometry is new. \n- The method is well constructed, and each step is properly justified. The change from Eulerian to Lagrangian is a very clever way to combine the physical constraints and the NERF model. \n- The results are impressive, even though it only uses synthetically generated scenes. The method is even outperforming the oracle baseline, where the geometry is known. \n\nWeaknesses\n- The paper does not include results with real data, which leaves some questions unanswered. (see my comments in the next section)\n- The dataset includes only gravity as an external force and a ground plane whose position is known. Other more complex scenarios are not dealt with in the paper. \n- The camera poses must be accurate and vary significantly to obtain good results with a NERF method. In practice, this introduces some constraints on the scene, such as a rigid background with a sufficient amount of texture to run structure-from-motion. These constraints are a bit ignored in the paper.  ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "jnwxc6ni00o", "reviewer": "Reviewer_EVdM", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This article presents a novel system that brings differentiable physics and neural radiance fields for dynamic scenes by augmenting a NeRF with a differentiable continuum dynamics model. A hybrid Eulerian-Lagrangian formulation is defined in this system to permit advecting geometry and appearance. This framework outperforms previous approaches validated by a wide variety of systems.", "summary_of_the_review": "I have listed my questions in Weakness. It is necessary to explain the results of droplets and the limitation of the proposed method. Overall, it is a nice conference paper.", "strength_and_weaknesses": "Strength:\n(1) It is interesting to combine NeRF with physical constraints.\n(2) This paper is well written.\n(3) In most cases, the results are encouraging.\n\nSummary of Weakness:   \n(1) Euler method and advection equation are both very fundamental and usually applied in highly restricted mathematical or physical models, which are not defined directly so that the theoretical level of this work satisfies a more general case.   \n(2) There are some limitations as mentioned before, which are not listed in the article clearly.    \n(3) Part of the experimental results is difficult to understand, disobeying universal physics phenomenon.   \n(4)The authors haven’t listed the source of their ground truth data.\n\nMain Questions:   \nThough part of the results shown in this article reflects stable properties of the framework, the authors may not be familiar with the fluid dynamics field. Here are some following questions:   \n1) This article is aimed to estimate both the unknown geometry and physical parameters of highly dynamic objects from multi-view videos. It is noticed that a droplet is regarded as Newtonian fluid, but this property would be destroyed when velocity exceeds a boundary value. Additionally, in the general case, the descent of a droplet drop to the ground may not satisfy the two assumptions.\n2) As for fluid dynamics fields, what is the source of the listed ground truth data (In house codes, CFD software, or open-source codes)? It cannot be made sure whether this ground truth data is convincing or not.   \n3) It is conspicuous that the method presented in this article requires the properties defined as continuum and conservation. Hence, listing the limitations clearly is necessary.\n\nThings to improve the paper that did not impact the score:   \n(1) Clarify the limitations of the framework;   \n(2) If simulations of physics models are unavoidable, fundamental knowledge of other fields is necessary to present meaningful outcomes. For example, fluid dynamics is necessary for the authors to get familiar with.  \n(3) Illustrating what kinds of physics models could be solved by the Euler scheme and governed by the advection scheme with low accuracy loss and time steps cost.   ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "EClHfBXV6S", "reviewer": "Reviewer_seww", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes Physics Augmented Continuum Neural Radiance Fields (PAC-NeRF), to estimate both the unknown geometry and physical parameters of highly dynamic objects from multi-view videos. The core idea of this paper is a hybrid Eulerian-Lagrangian representation of the neural radiance field. The proposed method on a wide variety of complex systems outperforms state of the art methods by up to two orders of magnitude. ", "summary_of_the_review": "In summary, this paper studies an interesting problem and propose an effective solution. Experiments demonstrate it effectiveness. ", "strength_and_weaknesses": "Strengths:\n1. This motivation of this paper is very clear and the problem studied in this work is of great value to the field of machine vision.\n2. The proposed method has strong theoretical support and easy to understand. \n3. Extensive experiments are conducted to validate the performance of the proposed method. \n\nWeaknesses:\nSince there are no existing approaches that estimate both geometry and material properties from videos, the authors make a best-effort comparison by blending combinations of state-of-the-art approaches for each sub-task. Such comparison seems not sufficient. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-25%