NeuPhysics: Editable Neural Geometry and Physics from Monocular Videos
Authors: Yi-Ling Qiao
Alexander Gao
Ming Lin
Venue: NeurIPS 2022 Accept (2022)
Keywords: NeRF
Differentiable Physics
3D Reconstruction
Video Editing

Abstract: We present a method for learning 3D geometry and physics parameters of a dynamic scene from only a monocular RGB video input. To decouple the learning of underlying scene geometry from dynamic motion, we represent the scene as a time-invariant signed distance function (SDF) which serves as a reference frame, along with a time-conditioned deformation field. We further bridge this neural geometry representation with a differentiable physics simulator by designing a two-way conversion between the neural field and its corresponding hexahedral mesh, enabling us to estimate physics parameters from the source video by minimizing a cycle consistency loss. Our method also allows a user to interactively edit 3D objects from the source video by modifying the recovered hexahedral mesh, and propagating the operation back to the neural field representation. Experiments show that our method achieves superior mesh and video reconstruction of dynamic scenes compared to competing Neural Field approaches, and we provide extensive examples which demonstrate its ability to extract useful 3D representations from videos captured with consumer-grade cameras.

NeuPhysics: Editable Neural Geometry and Physics from Monocular Videos
Yi-Ling Qiao
Alexander Gaot Ming C. Lin
University of Maryland, College Park Equal Contribution
Abstract
We present a method for learning 3D geometry and physics parameters of dynamic scene from only a monocular RGB video input. To decouple the learning of underlying scene geometry from dynamic motion, we represent the scene as a time-invariant signed distance function (SDF) which serves reference frame, along with a time-conditioned deformation field. We further bridge this neural geometry representation with differentiaalee physics simulator by designing two- way conversion between the neural field and its corresponding hexahedral mesh, enabling to estimate physies parameters from the source video by minimizing a cycle consistency loss. Our method also allows user to interactively edit 3D objects from the source video by modifying the recovered hexahedral mesh, and propagating the operation back to the neural field representation. Experiments show that our method achieves superior mesh and video reconstruction of dynamic scenes compared to competing Neural Field approaches, and we provide extensive examples which demonstrate its ability to extract useful 3D representations from videos captured with consumer-grade cameras.
Introduction
Monocular RGB videos are ubiquitous today, and computers' ability to understand spatial and physical properties embedded in their contents is essential for many applications. Can you imagine one day being able to build a fully-interactive digital twin of your surrounding world using just
a smartphone? You could capture a moving, fully 3D portrait of yourself (Sec. 4.1 estimate the
physical material parameters of objects and interact with them virtually (Sec. 4.3) playback existing recordings from new angles (Sec. 4.2). or even edit your favorite cartoon videos (Sec. 4.4). In this paper, we design a system that makes progress toward realizing all of these things. Taking a single RGB video as input, our method estimates the geometry and dynamics parameters of the scene, parameterized by neural fields [37] and enables subsequent editing and physically-bon simulation.
While humans can easily infer 3D shapes and physical properties of their environment even from a single eye, these tasks remain challenging for computer algorithms. Geometry reconstruction from single view is an intrinsically under-constrained problem: there is usually more than one 3D structure that could result representation to model the complex geometry of the real world. Explicit representations like surface mesh 4 38] are widely used in computer graphics, but they suffer from limited resolution and fixed topology. Implicit representations such as signed distance functions (SDFs) describe complex geometric structure more elegantly, but are not directly suitable for reasoning about dynamics properties. Estimating dynamics parameters is yet more challenging, since it is contingent upon correctly estimating geometry and its variation over time.
given 2D projection of scene. Another key challenge lies in choosing a suitable
In this paper, we design an end-to-end differentiable rendering and simulation pipeline that can directly learn the geometry and dynamics from a monocular video. By assuming the existence of
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

one-to-one dense mapping between temporal frames in 3D space, our method decouples the geometry estimation task into two complementary parts: static reference frame, and mapping function that encodes motion. The static reference frame consists of three neural fields: Appearance, Rigidity, and an SDF for geometry. Given a 3D spatial coordinate as input, these fields respectively output its RGB color, rigidity mask, and signed distance to the nearest surface, in the reference frame. Time-variant motion is encoded in a separate Motion field, which takes as input a 3D spatial coordinate (sampled along ray), plus learned latent code denoting particular discrete time step, and outputs the 3D displacement from the sampled coordinate back to its corresponding reference frame coordinate. We utilize the density function from NeuS [61] for rendering. and the Rigidity mask from Non-Rigid NeRF (7 to regularize the motion and obtain foreground/background separation.
A convenient property that emerges from factorizing the scene into static geometry + dynamic motion fields is that by querying the motion field, we can establish a point-wise correspondence across frames, which we leverage to infer dynamics properties of the scene. Specifically, physics parameters are optimized by minimizing the difference between results from the differentiable simulator [12 and motion field. Upon learning reasonable parameters, the differentiable simulator can be used for physically-based animation and editing of the scene. Experiments show that our method outperforms competing approaches in surface reconstruction, video reconstruction, and novel view synthesis tasks, both qualitatively and quantitatively. Multiple examples also demonstrate our full pipeline's s physically-bait -aase scene understanding and interactive scene evenee-diting capabilities. Our code and data are available here: https::/ http://siiissss.ollll..... google. wwwoo//veew/eeeeee
In summary, the key contributions of this work are as follows:
We present a framework for estimating and editing the geometry, appearance, and physics parameters directly from a single monocular video (Figure 3)
By decoupling the learning of static SDF geometry and dynamic motion, our method effectively regularizes this under-constrained problem and enables reconstruction of high- quality dynamic surface mesh sequences from monocular videos (Figure
By connecting the neural- ccurralfields-bee -ieeld--based geometry reconstruction to a differentiable physics engine, this approach eliminates the need for manual construction of intermediate 3D models, and avoids poor initialization for the differentiable simulator, empowering users to estimate the modeling and physics parameters from scratch (Figure
We design a two-way conversion (Figure 2 3 between the explicit hexahedral mesh and implicit neural fields in our end-to-end modeling, simulation and rendering pipeline, enabling users to (1) extract high-quality explicit meshes from the implicit fields, and (2) interact with the implicit 3D field (e.g. add, delete, move, deform, and simulate) by directly manipulating its explicit mesh counterpart.
2 Related Work
Differentiable volume rendering with implicit neural representations has recently been used to synthesize novel views, estimate geometry and appearance from images and videos. Neural Radiance Fields (NeRF) 37 and following works 9 53 2 68 have achieved photorealistic rendering results
even on complex scenes. The coordinate-basee
multi-laye perceptron (MLP) representation equips
the scene with high resolution (74), can be effectively regularized 73, and is memory-efficient.
Compared with differentiable surface rendering [41 42]. volume rendering can backpropagate gradients through the entire space, which leads to better optimization of complex geometry that contains topological changes, and mitigates getting stuck in local minima. The original density field from NeRF cannot directly reconstruct high-quality surface meshes, so neural SDF fields [61,
44 43 67 have been proposed for multi-view geometry reconstruction. Compared to other discrete representations like meshes 660 25| 34] point clouds [433
33 and voxels 6 65], SDF fields have 26 for reconstruction tasks.
higher resolution, and do not require pixel-level segmentation 40
69
Current approaches that employ neural SDF fields do not converge to accurate geometry when faced with dynamic scenes, while our method does from single video, which we show in figure 4
Recent work has extended NeRF to videos of dynamic scenes [6 72 but they primarily focus on 2D image synthesis, not reconstructing 3D geometry. These works either directly condition the neural fields on time
or they learn canonical field separately from dieedependent We choose the latter strategy because it better constrains the learning problem
K3
16 6
motion 45 47 57.

---

Sanglini
Oprry points
Righlty NLP
Stantation
invarint Inheraaive
Ray fleding
Uer
Editig
Insevvatianl Inkernalion
Viluse Bendering
000
Motion MLL
Latret
Figure I: Overall workflow. NeuPhysics takes monocular video supervision. The inveriivar
fields are defined on reference frame, while time-dependent motion is applied to each point sampled in space. We then compute color loss from the volume rendered images. A differentialle simulator is embedded after optimizing the neural fields, to learn dynamics parameters and allow scene editing. detailed computational graph can be found in Figure
based on a reasonable inductive bias, and importantly, it provides us with point-wise correspondence between time steps that is useful for dynamics estimation. By representing geometry with neural SDF field, and applying strong geometric priors, we can reconstruct dynamic 3D geometry better than existing dynamic extensions of NeRF, while even improving image reconstruction shown in Section
Differentiable simulators enable physics priors to better fit into data-driven approaches, and have therefore increasingly been applied to robotics and computer graphics tasks [44
11
21
30
55
66
These physically simulation methods have been developed for dynamical systems with different
properties, including rigid bodies 17, 48
15 36), soft bodies 24
28
articulated bodies [9 62, 50], cloth 31 32 and fluids 58
59
22 56
3 Methods
Given a monocular video consisting of N RGB image frames {I our approach estimates the 3D geometry within the camera's field of view, point-wise correspondence of the dynamic 3D scene across time, and static/dynamic segmentation. By integrating the differentiable physics module, we can learn physics parameters from the dynamic motion sequence, which we leverage for video editing and forecasting. Figure shows overview of paper.
3.1 Rendering Pipeline
SDF and color networks. Similar to recent Neural Fields methods [37]. we parameterize the geometry and appearance of the scene using coordinated-based Multi Layer Perceptrons (MLPs), which can render 2D images using the volume rendering scheme proposed in NeuS [61]. Geometry represented by SDF network s0sps (p) R + R that maps spatial coordinate signed distance to the nearest surface. Appearance is represented by another MLP Co (p) RR3
R to its
Rd that describes the RGB value of each spatial point. During rendering, the points along a ray cast from a given pixel can be written as [p(() o + tv/e 2 0}, where o is the camera center, v the unit direction of the ray, and is distance from the camera center. The pixel's color then calculated by integrating along the ray:
I +oc
C(o, v)
ve(s(p(t))
where w((-) 61 is weight function describing the contribution of each point's color based its SDF value. More details can be found the Appendix.
Motion network. However, the above framework is only designed to handle multi-view images of statie scene, which motivates another component accommodate temporal information. Instead of directly conditioning the existing networks upon time, we choose decouple the dynamic and static
3

---

scene components. Therefore, the SDF and color networks are defined on a static reference frame, while a motion network Dumation (4, p): 1 R R estimates the offset field [57] for each time step, where 1 E Rd is learned latent feature encoding temporal information about the i-th frame I. The motion network bends each query ray {p(t) to {p'(t) ((t) + b(4, p(t)), which is then queried on the reference frame for SDF and color values.
Rigidity network. Since we want to segment and simulate dynamic objects within the scene downstream, apply rigidity network [57] Teriaidieu (p) IR + I to distinguish dynamic foreground from static background. The mask r(p) E [0, 1] defined on the reference frame indicates the likelihood that a point belongs to the swept volume of moving elements the scene. The offset value becomes b brigidity (I,, p(t)) Perigidity (p) b.g (4,, p(f)). This formulation pushes dynamic regions toward larger r(-) values during training, while driving static areas toward zero.
Demotion
3.2 Simulation Pipeline
Geometry reconstruction. Given the signed distance field s(-) and rigidity mask r(-), we can reconstruct the geometry of the dynamic and static parts of the scene. For example, the foreground in the i-th frame is contained within the swept volume of moving objects described by:
A {pls(p + b(1, S kedy, r(p + b(I, p)) 2 krigidity):
where kidf and Brigidity are constant thresholds for the SDF and rigidity values, respectively. This formulation is amenable to conversion into an explicit mesh representation. For rendering purposes, surface mesh could be obtained using the marching cubes algorithm [35]. Alternatively, we represent the dynamic elements as a volume mesh. We sample over discrete 3D grid to find vertices within volume A,, and a grid point p satisfies Equation [ we add that voxel to a hexahedral volume mesh m,, We choose hexahedral rather than tetrahedral volume mesh due to its simplicity and regular structure, and because hexahedral mesh can be simulated by DiffPD [1], a differentiable
FEM simulator. Figure g visualizes the conversion between the implicit and explicit representation. Triangle surface mesh and tetrahedral volume meshes could be explored in future work.
Physics Engine. From the input video of a dynamic scene, we can extract mesh sequence (m; ) with consecutive motion. An individual mesh model m; is described by its vertices and hexahedral elements {Q, E _=,x3 x3 E, E N where the rows of Q are the 3D coordinates of n, vertices, and each row in E; records the indices of cight vertices that form hexahedron. We focus on modeling dynamic objects as soft bodies in this work. Compared to rigid bodies, soft bodies offer more expressiveness to model a wide range of deformable objects that that are important in applications like physically-basee mixed reality and digital twins. As our physics engine, we use Projective Dynamics 3 implemented by DifpP 12. The governing equation can be written as
- -Q. 1:2 E( Ophyysics) + fent),
(3)
where M is the mass matrix, Q; is the vertex locations at frame i, h is the time step, Q is the velocity, E is the potential energy due to deformation, 9physics are the physics parameters, and fext are the external forces. Note that the gradient of the potential energy V E can be interpreted the internal forces within the soft materials. This system uses implicit time integration for increased numerical stability, thus F is defined on the frame 1. Objects are modeled by homogeneous co-rotated materials, where the material parameters Ophyeics include Young's 's modulus and Poisson's 's ratio. A detailed formulation of the elastic energy can be found in the supplementary materials of [12].
System identification can be valuable for constructing faithful digital twins, producing plausible animations, and control of dynamical systems, but estimating states Q and Q, and parameters 4physics and fext from only monocular video is challenging due to the high-dimensionality and nonlinearity of the system. Recent progress in differentiable physics tackles this challenge by making the entire simulation differentiable. Gradients aQ." aQ," 36 p0
ac d2 and aL where a user-defined scalar at... loss function, are all available, and allow us to estimate the parameters using a gradient-based optimizer like Adam 27

---

Parameter Estimation. Running one simula- tion step forward in time from the i-th frame gives us an updated model at frame i 4 1, depen- dent on parameters physies: m (Ophysics) = sim (6 physice, m i + i + 1). By simulating the reconstructed meshes in the differentiable physics engine sim(00 (ppyeiieess -) and comparing with the ground truth reconstruction results as supervision signal, we optimize the physics parameters Ppyyeic using gradient-based meth- ods.
Condract bemaantral moss
Spli ou daceeninn 3o trralstton
Netral Reide
B Wiy di wish +e Man
Cocppan bary cantaic coortinnies
ShaEE SEdOT oeforn beesheeni resh
The dedlsmed gaints
The adtual adaa Neand Fabls
One idea for a suitable loss function is to use metric like Chamfer Distance (istance(m;+1, ( (Ophgi(@ss) between the extracted and simulated mesh mi+1 and m4+1 (Ophyeic))) searching for the Ophysica that minimizes the distance. However, this strategy requires sampling and mesh reconstruction for every time step, which is computationally expensive. An ablation study and profiling numbers can be found in Figure IO Additionally, the reconstructed mesh might vary significantly across time steps, thus cannot guarantee point-wise correspondence, making the dynamics discontinuous and non -in-differentiable w.r.t. the networks.
Figure 2: The editing pipeline. Users can edit the neural fields by operating on a hexahedral mesh.
To save sampling time and make the gradient well-defined, we design a cycle-consisenncy loss to compute the distance between the simulation results and neural fields. Starting from an arbi- trary mesh m; sampled at the i-th frame, we run the differentiable simulation for steps and get (1/t,(0phssicn = sim (0rpoOpyyscc:: m i + i + j). When (
is deformed back to the reference frame using the learned motion field, the result should correspond to the original shape m,, Since m'+ (Ophysics) and m; share a common topology, an exact woint-wise correspondence exists, and we can directly subtract their point clouds to obtain their distance:
Ophysics = (m. + b Domtion
Prigidity (L,, m.)) - (m'++ (Ophyeice) b mottion
@ Origidity (l++j. (0ppy(0phssice)))
(4)
where b(I, m.) is the motion field at frame i that maps mesh m, back to the reference frame. Cphysice is differentiable w.r.t. the motion network, rigidity network, and physics parameters. Importantly, independent of the simulation duration, we only need to sample m; once, and can be used reference point to compare all other frames.
Scene Editing. The simulator in our pipeline imposes strong physics prior for forecasting motion and modifying dynamics. Figure shows how we edit the signed distance and color fields through the hexahedral mesh. Based on hexahedral mesh m; extracted from the i-th frame, we can simulate to get modified state m4411 In order to render the modified scene, we add an additional 'bending" layer basit (m(, -) on top of the primary motion field b(l;.)).
When querying ray point p' to render the modified scene, we first check if the point is displaced by our editing operation, i.c. if it is located within the hexahedral mesh m'. The inside-outside-tess would be trivial if all hexahedra in mf, remained in regular grid, however, they may have deformed during simulation. Thus, we split each hexahedron into five tetrahedra a query point p' is inside the hexahedron if and only if it is inside one of the tetrahedra. Following (71), we decide whether p' is inside a tetrahedron by its barycentric coordinates. For a tetrahedron consisting of four vertices can be computed
{p' (z/; v/,, 24) T Te-1- the barycentric coordinates
{c, 1- of p' = (x' v/ z²)
as C, = Det,/ Deto, where:
z
x'
y
z'
1
r
y
2
Deto
Vg
22
Det
y/2
z
Detz
x'
y
x
(5)
xz
V/
2g
1
T3
23
23
1
r
1/3
2
1
y4
24
1
3/
21
1
14
34
24
and Det, is obtained by substituting the i-th row of Detp with (a', y', 2,, 1). p' is inside or on the
surface of this tetrahedron if and only 0 S e S 1 holds for i = 1, 2, To determine tetrahedra
to check for a given point, we use k-nearest neighbors Op = NNNNNNN (m/, p') to locate mesh vertices Qr nearest to query point p. and select tetrahedra that contain those vertices.
5

---

Canonical Object
Cycle Consssency
Canonical Object
Cycle Comtitercy
Canonical Object
Reederinn
Volume
Fundboos
Rendering
Color Field
Rigidity Field
Motion Fi
Motion F)eld((s)
Motion Field(ta)
Warp
Warp Bark
Warp
Triinblle Paramoters
Ccttor
Prigidiry
mmotton
Back
mmotion
A morton
Non- Dillerntabeee Operators
SDF
Diff.
Diff.
Field
Raconsenicion
Mesh
Object(t, )
Sim.
(2)
Sim.
Object(
Difforentable Operators
0sor
Opkysica
9 ptoysicc
Figure 3: Computational Graph During Training. We show how data transfers among modules in basic 3-step dynamical systems. All operators are differentiable, except the hex mesh reconstruction for the first frame. In our designed cycle consistency mechanism, the rigidity and motion networks serve the bridge between the rendering and physics module.
If does belong to tetrahedron, then is displaced along with mesh. Assuming the modification can be linearly interpolated inside tetrahedron, the original coordinate in the i-th frame becomes the weighted p E CiP,, where p is the reference coordinate corresponding to p' (before editing). To account for the area that was once occupied by m, but now empty after editing: we simply assign those points large positive SDF value such that they effectively transparent during volume rendering and do not influence 3D reconstruction.
3.3 Training and Losses
With video the main supervision signal used to train our system, we wish to minimize the distance between rendered and the ground truth images Lestor = I, - -I. We include additional terms to regularize training. First, an important property of an SDF its unit norm. Eikonal term [8]. Fikonal = E..e( (Il...11l - 1)2 is therefore added to the loss. We also want to constrain the motion of our scene (i.e. to be locally rigid, incompressible, and contain mostly static ragions), so we include an offset loss foffset
Σ. [[b(I, ps, + r(Pr,)) and divergence loss [57
Ldivergence = Σ., [|div(b(1, Pr,x))l2-- Finally, we use the physics loss Cphysics when we train for physics parameters. The total loss funetion is:
Liotal Lootor CEikoCiakonal + Woff + Wdin Edivergence + Wphy UphyCphysics
By default, we set WEsk 0.1, Wofs 50.0, and Wdie 2.0, though these weights may be further tuned depending on the scene. Figure demonstrates the data and flow of gradients during training. In our proposed cycle-consistency physics loss, the rendering and simulation modules are connected by the rigidity and motion networks. Instead of jointly training all the parameters together, we adopt sequential training strategy where the rendering networks are first trained to optimize Lioral for 300,000 epochs, and the physics parameters are then trained to optimize Cphysics for 100 epochs. The physics module is trained independently after the geometry networks converge because the differentiable simulation is much more expensive than querying MIPs. An ablation study for the efficiency of our training strategy is conducted in Section 4.5) In future work it would be interesting to try a faster simulation engine which could be incorporated as a physics prior during training of the rendering networks. A full training cycle typically takes around 18 hours an NVIDIA A5000 GPU, For the training data (used with consent), we note that the background must contain sufficient detail, or the Structure from Motion implementation by COLMAP 51 52 could fail.
4 Experiments
4.1 Dynamic Geometry Reconstruction
We visualize mesh reconstruction quality of dynamic scenes in Figure Table shows quanti- tative results. Our method frequently outperforms competitive approaches. Additional geometry reconstruction comparisons can be found in Appendix

---

Sitting
Standing
Method
LPIPS.]
SSIM
PSNRT
LPIPS]
SSIM
PSNKT
LPIPS.
SSIM?
PSNRT
LPIPS]
SSIM
PSNRT
Neu8
0.239
0.911
34.563
0.301
0.829
808
0.330
0.862
34,400
0.305
0.90
34.485
NRNeRF
0.223
0.892
32,019
0.269
0.863
31.549
0.340
0.830
32.411
0.205
0.913
33,999
Table I: Quantitative evaluation of video reconstruction. We evaluate method on both real-
world and synthetic videos, outperforming alternative methods in most cases.
(e) Tnpur
(h) Ours
(c) NeuS
(d) D-NeRF
(c) Inpur
(f) Ours
(g) NeuS
(h) D.NeRF
Figure 4: Surface reconstruction from dynamic videos. Our method achieves higher-quality, more detailed reconstruction (e.g. on museles, faces, and books) than other approaches.
4.2 Novel View Synthesis
We simulate four cases of deformable objects moving in room. We render each scene with two different camera trajectories, so that one path may be used for training, while the other is used as ground truth to evaluate novel view synthesis. Quantitative results in Table show that our method achieves the best synthesis quality in all metrics. Datasets and results are shown in Figure (a) and (b) are frames from the training videos. (c) is taken at the same time step (b), but it novel view from the test path. Notice that our method successfully synthesizes the dynamic scene from a novel view, while other methods struggle reconstruct geometry and texture accurately.
4.3 Parameter Estimation
In these experiments, we first use our network to reconstruct the dynamic scene, then apply the embedded differentiable simulator to estimate physical parameters. Finally, edit the physical parameters to generate novel video frames. More visual results can be found in the supplementary video. Figure shows the initial frame of bouncing ball sequence, where the ball is released from height and collides with the ground. The ball is modeled by co-rotated material. Using DiffPD [12), estimate both Young's modulus and vertical acceleration in two separate optimization tasks. The physics loss is minimized for 100 epochs (as described in Section 3.2h using Adam [27] optimizer with learning rate of 0.01. We initialize Young's modulus to 2 10 Pa and acceleration to Om/s², and by the end of optimization, have estimated Young's 's modulus at 2.96 x 10 Pa and acceleration at may choose modify some of them
3. 3.78m/ Having identified accurate physics parameters, from their true values, and generate videos that contain modified physical properties from the original on which they are based. For example, the original ball has no horizontal speed, so we add 0.3m//s left and right initial velocities in Figure 6 (b) and (c), respectively. We also decrease the Young's s modulus in Figure 6 (d), (e), and (f) from 2 x 10 Pa x 10 Pa. where the deformation is getting more significant.
Tennis Ball
Basketball
Blue Ball
Blac Cube
Method
LPIPSS
SSIM
LPIPS
LPIPS
SSIM1
IPIPS
SSIM
NRNeRF
0.455
0.566
30.286
0.473
0,550
29.684
0.409
0.479
30.152
0.620
0.398
29 09.044
D-NoRF
937
0.775
Ours
0.336
0U632
31.103
0.368
0.625
30.568
0.334
0.513
30.407
0.589
0.449
29.997
Table Quantitative evaluation of novel view synthesis. In this experiment, we evaluate novel camera trajectory different from the training data. Even in this highly under-constrained task, our method benefits from the consistency of the geometry and significantly outperforms S.O.T.A. using all metrics.

---

(a) Input
(b) Input 2
(c) Novel view on 2
(d) Ours
(e) NRNeRF
(1 D-NeRF
Figure 5: Qualitative evaluation of geometry reconstruction and novel view synthesis. (a, b) Frames from the training videos. (c) Ground truth evaluation view. Consistent with numerical results in Table 2 our method generates images from novel views with higher quality than previous methods.
(a) Original Image
(b) Left velocity
(c) Right velocity
(d) Harder material
(e) Moderate material
(i) Softer material
Figure 6: Parameter estimation and editing by differentiable physics. NeuPhysics can estimate the
acceleration and Young's Modulus, generate videos with different initial velocities and material parameters.
4.4 Editing
In this section, we demonstrate how our hybrid representation provides flexible interface for users to directly modify the geometry on the 3D hexahedral mesh and volume render the results.
Region of Interest. First, select region of interest that wish to edit. Our system naturally provides the scene rigidity mask and SDF values, so foreground area A; at the i-th frame can be computed Equation 2 If users require more precisely defined region of interest, custom bounding box or even more complex hexahedra bounding volume A'' can be provided. A hexahedra mesh can be reconstructed from the final region of interest, A' (1 Ai..
Delete. Figure (a) and (b) are two distinct time steps from given input video. In one, a hand flexes and deforms, and in the second, an elastic ball bounces. First, we delete the foreground from View by simply setting the SDF value to positive number (e.g. 1.0) for the query points inside hexahedral mesh m. Since we assume bell-shaped density distribution centered at [61], area with large SDF value less likely to be sampled, and therefore appears transparent. After removing the foreground from View 1, even though some parts of the background the other views of the scene provide sufficient textural/strructural information to complete the missing background. Since editing takes place entirely in 3D space, the resulting 2D images feel sharp and physically grounded, as shown in (c).
occluded in the original,
ISCYCL
InSYYOLE
VBS
YES
VES
YES
(c) Duplicate Figure 7: Geometry and appearance editing videos. Our method can delete, add, move, or change the appearance. Editing in the 3D space can provide users with realistic and clean results.
(a) View
(b) View
(c) Delete
(d) Appearance

---

Appearance. We can also modify the output of the color network to change the appearance of specific region. For example, we might like to emphasize the moving foreground of a video by desaturating the background. This is easily done by changing the original color [r, g, bj ca. (p) to
be tggtb
rtg+b =tght]
for all the query points outside m. Results are shown in Figure 7 (d).
Duplicate. Translation and scaling are basic operations in graphic design software. With our pipeline, users are able to copy. move, and scale the hexahedral mesh m. creating new version m'. During volume rendering, the SDF and color values of the points p' inside m'' are set equal their counterparts p in m. In Figure (d), we duplicate an existing object and modify its size and position.
4.5 Ablation Study
In this section, we conduct several ablation studies using the bouncing ball example justify our network design choices and training strategies.
Rendering Module
Physics Medule
Epach
300.0001
20
Time per opoch
0.22 8
70
Sequential Vs. Joint Optimization. Our method learns the scene sequentially by first optimizing the rendering networks, followed by the physics parameters. A scem- ingly more concise strategy might be to jointly train all parameters from scratch. Unfortunately, the joint train- ing strategy is impractical for two reasons: (1) the table
Total time
0.75
SespentarOpemmisttr
0.70
Sendering Lags
0.65
0.60
in Figure 8 indicates that the physics module converges rapidly thanks to the effectiveness of the differentiable simulator, but running the forward and backward simula- tion is costly (~ 70 s) compared to querying the rendering networks (~ 0.22 s) - training the rendering module and physics module together would require many more iter- ations (300,000 epochs), with each taking much longer (~ 70sec per epoch), making the training infeasible, and (2) before the rendering module converges and provides satisfactory geometry estimation, the geometry makes little sense for simulation, so gradients flowing from the Ephysics do not help the optimization. The smoothed plot in F indicates that joint training slows down the rendering optimization early on. Sequential optimization is therefore more practical.
0.55
0.50
0.45
0.40
0
20
40
60
80
100
Epoch
Figure 8: Joint training can be time-
consuming and converges slowly.
We conducted two additional tests to study the relationship between the rendering and physics modules. First, during the sequential optimization, we also update the rendering networks. This harms the rendering quality (increases loss from 0.03 to 0.10) and does not help physics parameter estimation. Second, we attempt to optimize the rendered image from physics simulation to match the ground truth image. We found this process to be inefficient (2 minutes per image) and observed no significant benefits.
Physics Module. Within our sequential optimization strategy, there are still multiple options for physical parameter estimation. In pipeline, we reconstruct the volume mesh m, of the first frame, then predict the following mesh sequence m = sim m1, 1 + i) with the differentiable simulator. The physies loss for each frame can be computed in cycle-consistency manner in Equation 4 Our proposed cycle-consistency loss naturally guarantees a point-wise correspondence for each vertex, and does not require mesh reconstruction for every time step. For each epoch, only reconstruct the volume mesh for the first frame, which reduces computation and makes the loss differentiable w.r.t. the rendering networks.
A more direct physics loss could rely on reconstructing the geometry of each frame and computing the Chamfer distance between the simulated and reconstructed meshes. We compare the two strategies in Figure TO "Ours' is optimized with our proposed syclecconsistency loss Ephysies and 'Chamfer' is optimized with the Chamfer distance. (c) shows the value of acceleration, the optimization variable. (a) and (b) indicate that the losses have comparable performance in both metrics. However, 'Chamfer" takes significantly longer since every frame must be reconstructed. Our proposed cycle-consistency loss runs faster and maintains differentiability between the rendering and physics simulation module. Conceptually, decoupling the NeRF and differentiable physics has

---

Time Recanstraction
Our
Reconstract Chamfer
5.00
335
65
233
Total
~ 67
~v 115
Park
Paats Acxileraius
(60 Cydle Consisenccy Lots
Chammee Distome Lows
Figure 10: Compare with optimizing Chamfer Distance. An alternative our proposed cycle- consistency loss to use Chamfer Distance for the physics module. The two metrics have similar effects but our method can save much reconstruction time.
another drawback: a mesh reconstructed purely from SDF cannot capture the interior stress and deformation. Our proposed cycle-consistency loss addresses this concern, it takes advantage the learned motion field, so it is capable of finding the point-wiss wise correspondence and describing the deformation.
Rigidity Map. Finding the correct dynamic foreground object prerequisite for physical simulation. The raw output the SDF field only reconstruct global scene, the rigid- ity map is used to segment the dynamic part of the scene (assumed to be the foreground in this case). The rigidity map collects the mo- tion information from all frames and assigns a high rigidity mask value to dynamic regions. Figure 9 (c, d) visualizes the rough segmenta- tion results from using different rigidity thresh- olds. We empirically find 0.2 be a reasonable default threshold for separating the foreground from background. If the rigidity map does not provide clean separation, users may manually specify bounding box covering the moving object swept volume, and get the final foreground as shown in (e, f). An alternative segmentation approach is to use the magnitude of the motion field as criterion to separate dynamic foreground, i.e. large motion area corresponds to dynamic objects (and the converse). However, in some frames, dynamic parts might only contain small offsset from the canonical frame. For example, a cyclic bouncing ball could overlap with the canonical position and thus have offset values of zero, even though it should be classified as a dynamic area. Figure 2 (a) and (b) show results of filtering the scene using motion magnitude. It not a good strategy, since a large portion of the background remains even as the ball is partially clipped off in (b).
Motion Magaitule Rigidity 0.15te) Rigidity Bounding
Magnitude Rigidity
Fereground
Figure 9: Region of Interest. Motion magnitude (a, b) is an intuitive criterion for static/dynamic separation, but does not work well in practice. The rigidity map (c, d) provides a rough, but reason- able separation. (e, f) We can also specify more precise ROI with a bounding box.
5 Conclusion
In this paper, we design an end-to-end pipeline that can successfully estimate and enable physically- based editing of geometry, appearance, and physics of dynamic scene from single video. Some possible future directions include incorporating semantic priors to further constrain the optimization, and leveraging pre-trained networks [10] supervision to further regularize training. For example, optical flow scene flow networks could be used as motion detection signal, while a depth estimation network could help to initialize the signed distance field. Another useful future goal would be to estimate texture [64] and UV maps directly, allowing artists to construct 3D models in standard formats with less effort. Training and inference could also be accelerated dramatically by adopting a rusoluire hash encoding
(39]. Finally, we could couple this differentiable framework with other types of dynamics, to support wider range of objects and scenes, e.g. cloth, tetrahedra mesh 49 or even robots 20)
Acknowledgements. This research is supported in part by Dr. Barry Mersky and Capital One E-Nnovate Endowed Professorships, and ARL Cooperative Agreement W91 1N721
10

---

References
[1] Benjamin Attal, Eliot Laidlaw, Aaron Gokaslan, Changil Kim, Christian Richardt, James Tompkin, and Matthew O'Toole. Törf: Timc-of-fiiht radiance fields for dynamic scene view synthesis. Advances in neural information processing systems, 34, 2021.
[2] Mark Boss, Raphael Braun, Varun Jampani, Jonathan T Barron, Ce Liu, and Hendrik Lensch. Nerd: Neural reflectance decomposition from image collections. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 12684-12694, 2021.
[3] Sofien Bouaziz, Sebastian Martin, Tiantian Liu, Ladislav Kavan, and Mark Pauly. Projective dynamics: Fusing constraint projections for fast simulation. ACM Transactions on Graphics (TOG), 33(4). 2014.
[4] Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang. Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et al. Shapenet: An information-rich 3d model repository. arXiv preprint arXiv:1512.03012. 2015.
[5] Hsiao-yu Chen, Edgar Tretschk, Tuur Stuyck, Petr Kadlecek, Ladislav Kavan, Etienne Vouga, and Christoph Lassner. Virtual clastic objects. arXiv preprint arXiv:2201.0465 044233, 2022.
[6] Christopher B Choy, Danfei Xu. JunYoung Gwak. Kevin Chen, and Silvio Savarese. 3d-r2n2: A unified approach for single and multi-view 3d object reconstruction. In European conference computer vision. pages 628-644. Springer, 2016.
[7] Filipe de Avila Belbute-Peree Peres, Kevin A. Smith, Kelsey R. Allen, Josh Tenenbaum, and J. Zico Kolter. End-to-end differentiahle physies for learning and control. In Neural Information Processing Systems, 2018.
[8] Jonas Degrave, Michiel Hermans, Joni Dambre, and Francis wyffels. A differentiable physics engine for deep learning in robotics. Frontiery in Neurorobotics, 13, 2019.
[9] Frank Dellaert and Lin Yen-Chen. Neural volume rendering: Nerf and beyond. arXiv preprint 10Xiv:2101.052000 2020.
[10] Kangle Deng, Andrew Liu, Jun- Yan Zhu, and Deva Ramanan. Depth-supervised nerf: Fewer views and faster training for free. arXiv preprint vrXiv:2107.02791, 2021.
[11] Tao Du, Kui Wu, Andrew Spielberg, Wojeiech Matusik, Bo Zhu, and Eftychios Sifakis. Functional optimization of fluidic devices with differentiable stokes flow. ACM Transactions on Graphics (TOG), 2020.
[12] Tao Du, Kui Wu, Pingchuan Ma, Sebastien Wah, Andrew Spielberg, Daniela Rus, and Wojciech Matusik. DiffPD: Differentiable projective dynamics with contact. arXiv:2101.05917 2021.
[13] Yilun Du, Yinan Zhang, Hong-Xing Yu, Joshua B. Tenenbaum, and Jiajun Wu. Neural radiance flow for 4d view synthesis and video processing. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021.
[14] Haoqiang Fan, Hao Su, and Leonidas Guibas. point set generation network for abject reconstruction from a single image. In Procediigss of the IEEE conference on computer vision and pattern recognition, pages 605-61 (05-613,70 2017.
[15] C. Daniel Freeman, Erik Frey, Anton Raichuk, Sertan Girgin, Igor Mordatch, and Olivier Bachem. Brax a differentiable physics engine for large scale rigid body simulation, 2021. URL Utt/tttp://githui com./ google beogle/bbra e/brax
[16] Chen Gao, Ayush Saraf, Johannes Kopf, and Jia-Bin Huang. Dynamic view synthesis from dynamic monocular vidco. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5712 5721, 2021.
[17] Moritz Geilinger, David Hahn, Jonas Zehnder, Moritz Bächer, Bernhard Thomaszewski, and Stelian Coros. ADD: Analytically differentiable dynamics for multi-body systems with frictional contact. ACM Transactions on Graphics (TOG), 39(6). 2020.
[18] Amos Gropp, Lior Yariv, Niv Haim, Matan Atzmon, and Yaron Lipman. Implicit geometric regularization for learning shapes. arXiv preprint 2002: 10099, 2020.
[19] Sehoon Ha, Stelian Coros, Alexander Alspach. Joohyung Kim, and Katsu Yamane. Joint optimization of robot design and motion parameters using the implicit function theorem. In Robotics: Science and Systems (RSS), 2017.

---

[20] Eric Heiden, Miles Macklin, Yashraj S Narang. Dieter Fox. Animesh Garg. and Fabio Ramos. DiSECt: A Differentiable Simulation Engine for Autonomous Robotic Cutting. In Proceedings of Robotics: Science 20200//RSS..22.XIII..
and Systems, Virtual, July 2021. doi:
[21] Eric Heiden, David Millard. Erwin Coumans, Yizhou Sheng, and Gaurav S Sukhatme. Neuralsim: Augmenting differentiable simulators with neural networks. 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 9474-9481. IEHE, 2021.
[22] Philipp Holl, Vladlen Koltun, and Nils Thuerey. Learning control PDEs with differentiable physies. In ICLR, 2020.
[23] Yuanming Hu, Jiancheng Liu, Andrew Spielberg, Joshua Tenenbaum, William T. Freeman, Jiajun Wu, Daniela Rus, and Wojeiech Matusik. ChainQueen: A real-time differentiable physical simulator for soft roboties. In ICRA, 2019.
[24] Yuanming Hu, Luke Anderson, Tzu-Mao Li, Qi Sun, Nathan Carr, Jonathan Bagan-Kellep and Frédo Durand. DifhTiichi: Differentiable programming for physical simulation. In ICLR, 2020.
[25] Hiroharu Kato, Yoshitaka Ushiku, and Tatsuya Harada. Neural 3d mesh renderer. Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3907-3916, 2018.
[26] Petr Kellnhofer, Lars C Jebe, Andrew Jones, Ryan Spicer, Kari Pulli, and Gordon Wetzstein. Neural lumigraph rendering. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 4287-4297, 2021.
[27] Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv: 1412.6980, 2014.
[28] Jatavallabhula Krishna Murthy, Miles Macklin, Florian Golemo, Vikram Voleti, Linda Petrini, Martin Weiss, Breandan Considine, Jérôme Lévesuu, Kevin Xie, Kenny Erleben, et al. gradSim: Differentiable simulation for system identification and visuomotor control. In JCLR, 2021.
[29] Sizhe Li. Zhiao Huang, Tao Du Hao Su, Joshua Tenenbaum, and Chuang Gan. Contact Points Discovery for Soft-Body Manipulations with Differentiable Physics. In International Conference on Learning Representations (ICLR), 2022.
[30] Xuan Li. Jessica Wililliams, Minchen Li, Cynthia Sung, and Chenfanfu Jiang. Soft hybrid aerial vehicle via bistable mechanism. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 7107-7113, TEFE, 2021.
[31] Yifci Li, Tao Du, Kui Wu, Jie Xu, and Wojeiech Matusik. Diffcloth: Differentiable cloth simulation with dry frictional contact. ACM Trans. Graph. 2022.
[32] Junbang Liang, Ming C. Lin, and Vladlen Koltun. Differentiable cloth simulation for inverse problems. In Neural Information Processing Systems, 2019.
[33] Chen-Hsuan Lin, Chen Kong, and Simon Lucey. Leaminng efficient point cloud generation for dense 3d object reconstruction. In proceedings of the AAAI Conference Artificial Intelligence, volume 32, 2018.
[34] Shichen Liu, Tianye Li, Weikai Chen, and Hao Li. Soft rasterizer: A differentiable renderer for imagg-basedd reasoning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7708-7717, 2019.
[35] William E Lorensen and Harvey Cline. Marching cubes: A high resolution 3d surface construction algorithm. ACM siggruph computer graphics, 21(4):163-169, 1987.
[36] Pingchuan Ma, Tao Du, Joshua B. Tenenbaum, Wojeiech Matusik, and Chuang Gan. RISP: Rendering- invariant state predictor with differentiable simulation and rendering for cross-domain parameter estimation. In International Conference on Learning Representations, 2022.
[37] Ben Mildenhall, Pratul Srinivasan, Matthew Tancik, Jonathan Barron, Ravi Ramamoorthi, and Ren
Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In ECCV, 2020.
[38] Kaichun Mo. Shilin Zhu, Angll Chang, Li Yi. Subarna Tripathi. Leonidas Guibas, and Hao Su. PartNet: Large-scale Benchmark for Erniner and Hierarchical Part-level 3D Object Understanding, In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
[39] Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with a multiresolution hash encoding. ACM Trans. 4 July 2022. doi: 10. 10.1145/ 3528223.55 33330127. URL https: http:///w..orro..111////////////
353012
12

---

[40] Michael Niemeyer, Lars Mescheder, Michael Oechsle. and Andreas Geiger. Differentiable volumetric rendering: Learning implicit 3d representations without 3d supervision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3504-3515, 2020.
[41] Merlin Sumier-David Delio Vicini, Tizian Zeltner, and Wenzel Jakob. Mitsuba 2: A retargetable forward and inverse renderer. ACM Transactions on Graphics (TOG), 38(6), 2019.
[42] Merlin Nimier-David, Sébastien Speierer, Benoît Ruiz, and Wenzel Jakob. Radiative backpropagation: an adjoint method for lightning-fast differentiable rendering. ACM Transactions on Graphics (TOG), 39(4), 2020.
[43] Atsuhiro Noguchi, Umar Iqbal, Jonathan Tremblay, Tatsuya Harada, and Orazio Gallo. Watch it move: Unsupervised discovery of 3d joints for re-posing of articulated objects. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 3677-3687, 2022.
[44] Michael Oechsle, Songyou Peng, and Andreas Geiger. Unisurf: Unifying neural implicit surfaces and radiance fields multi-view reconstruction. International Conference on Computer Vision (ICCV), 2021.
[45] Keunhong Park, Utkarsh Sinha, Jonathan T. Barron, Sofien Bouaziz, Dan B Goldman, Steven M. Seitz, and Ricardo
Nerfies: Deformable neural radiance fields. ICCV, 2021.
[46] Keunhong Park, Utkarsh Sinha, Peter Hedman, Jonathan T. Barron, Sofien Bouaziz, Dan B Goldman, Ricardo Martin-Brualla, and Steven M. Seitz. Hypernerf: A higher-dimensionall representation for topologically varying neural radiance fields. ACM Trans. Graph., 40(6), dee 2021.
[47] Albert Pumarola, Enric Corona. Gerard Pons-Moll, and Francesc Moreno-Noguer. D-nerf: Neural radiance fields for dynamic scenes. arXiv preprint arXiv:201 I. 1.13961, 2020.
[48] Yi-Ling Qiao, Junbang Liang, Vladlen Koltun, and Ming C. Lin. Scalable differentiable physics for learning and control. In ICMI, 2020.
[49] Yi-Ling Qiao, Junbang Liang, Vladlen Koltun, and Ming Lin. Differentiable simulation of soft multi-body systems. In Conference on Neural Information Processing Syxtems (NewrIPS), 2021.
[50] Yi-Ling Qiao, Junbang Liang, Vladlen Koltun, and Ming C. Lin. Efficient differentiable simulation of articulated bodies. In ICMI., 2021.
[51] Johannes Lutz Schönberger and Jan-Michael Frahm. Structure-from-mmoiie revisited. In Conference on Computer Vision and Puttern Recognition (CVPR), 2016.
[52] Johannes Lutz Schönberger. Enliang Zheng, Mare Pollefeys, and Jan-Michael Frahm. Pixelwise view selection for unstructured nulti-view stereo. In European Conference Computer Vision (ECCV), 2016.
[53] Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein. Implicit neural representations with periodic activation functions. Advances Neural Information Processing Systems, 33:7462-7473, 2020.
[54] Changkyu Song and Abdeslam Boularias. Learning to slide unknown objects with differentiable physics simulations. In Robotics: Science and Systems (RSS), 2020.
[55] Andrew Spielberg, Allan Zhao, Yuanming Hu, Tao Du, Wojciech Matusik, and Daniela Rus. Learning- in-the-loop optimization: End-to end control and co-design of soft robots through learned deep latent representations. In Neural Information Processing Systems, 2019.
[56] Tetsuya Takahashi, Junbang Liang, Yi-Ling Qiao, and Ming C Lin. Differentiable fluids with solid coupling for learning and control. In AAAI, 2021.
[57] Edgar Tretschk, Ayush Tewari, Vladislav Golyanik, Michael Zollhöfer, Christoph Lassner, and Christian Theobalt. Non-rigid neural radiance fields: Reconstruction and novel view synthesis a dynamic scene from monocular video. In IEEE International Conference on Computer Vision (ICCV). TEEE, 2021.
[58] Kiwon Um. Robert Brand, Yun (Raymond) Fei, Philipp Holl, and Nils Thuerey. Solver-in-the-lop:: Learn- ing from differentiable physics to interact with iterative PDE-solvers. In Neural Information Processing Systems, 2020.
[59] Nils Wandel, Michael Weinmann, and Reinhard Klein. Learning incompressible fluid dynamics from scratch towards fast, differentiable fluid models that generalize. In ICLR, 2021.
13

---

[60] Nanyang Wang. Yinda Zhang, Zhuwen Li, Yanwei Fu, Wei Liu, and Yu-Gang Jiang. Pixel2mesh: Generating 3d mesh models from single rgb images. In Proceedings the European conference on computer vision (ECCV). pages 52-67, 2018.
[61] Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang. Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. NeurIPS, 2021.
[62] Keenon Werling, Dalton Omens, Jeongseok Lee, Ionnis Exarchos, and C Karen Liu. Fast and feature- complete differentiable physics for articulated rigid bodies with contact. In Robotics: Science and Systems (RSS), 2021.
[63] Wenqi Xian, Jia-Bin Huang, Johannes Kopf, Changil Kim. Space-time neural irradiance fields for free-viewpoint video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 9421-943 2021.
[64] Fanbo Xiang. Zexiang Xu, Miloš Hašan, Yannick Woldd-Geoffroy, Kalyan Sunkavalli, and Hao Su. NeuTex: Neural Texture Mapping for Volumetric Neural Rendering. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
[65] Haozhe Xie, Hongxun Yao, Xiaoshuai Sun, Shangchen Zhou, and Shengping Zhang. Pix2vox: Context- aware 3d reconstruction from single and multi-view images. In Proceedings of the IEEE/CVF International Conference on Computer Vision. pages 2690 2698, 2019.
[66] Jie Xu, Viktor Makoviychuk, Yashraj Narang, Fabio Ramos, Wojeiech Matusik, Animesh Garg, and Miles Macklin. Accelerated policy learning with parallel diffcrentiable simulation. In International Conference on Learning Representations, 2021.
[67] Gengshan Yang, Minh Vo, Natalia Neverova, Deva Ramanan, Andrea Vedaldi, and Hanbyul Joo. Banmo: Building animatable 3d neural models from many casual videos. Proceedings of the IFEF/CVF Conference on Computer Vision and Pattern Recognition, pages 2863-2873, 2022.
[68] Guandao Yang, Serge Belongie, Bharath Hariharan, and Vladlen Koltun. Geometry processing with neural fields. Advances in Neural Information Processing Systems, 34, 2021.
[69] Lior Yariv, Yoni Kasten, Dror Moran, Mcirav Galun, Matan Atzmon, Basri Ronen, and Yaron Lipman. Multiview neural surface reconstruction by disentangling geometry and appearance. Advances in Neural Information Processing Systems, 11:222225522 2020.
[70] Lior Yariv, Jiatao Gu, Yoni Kasten, and Yaron Lipman. Volume rendering of neural implicit surfaces. In Thirty-Fifth Conference Neural Information Processing Systems, 2021.
[71] Yu-Jic Yuan, Yang- Tian Sun, Yu- Kun Lai, Yuewen Ma, Rongfei Jia, and Lin Gao. Nerf-editing: Geometry editing of neural radiance fields. In Computer Vision and Pattern Recognition (CVPR), 2022.
[72] Jiakai Zhang, Xinhang Liu, Xinyi Ye, Fuqiang Zhao, Yanshun Zhang, Minye Wu, Yingliang Zhang, Lan Xu, and Jingyi Yu. Editable free-viewpoint vidco using layered neural representation. ACM Transactions on Graphics (TOG), 40(4):1-18, 2021.
[73] Kai Zhang, Gernot Riegler. Noah Snavely, and Vladlen Koltun. Nerf++: Analyzing and improving neural radiance fields. arXiv:2010.07492, 2020.
[74] Kai Zhang, Fujun Luan, Zhengqi Li, and Noah Snavely. Iron: Inverse rendering by optimizing neural sdfs and materials from photometric images. IEEE Conf. Comput. Vis. Pattern Recog., 2022.
14


Reviews:
[{"review_id": "uESZGMEi6kz", "reviewer": "Reviewer_1MJg", "summary": "The paper proposes a method to recover geometry, motion, and physics parameters from a single video. It recovers geometry and motion with differentiable volume rendering similar to Dynamic NeRF. To recover physics parameters, it uses Diff-PD. It shows results on synthetic and real data with relatively small motion. It also demonstrated applications of user editing.", "questions": "- l194: b(li,mi) needs to be explained. Otherwise, it's unclear what's the relation between mi and the reference frame.\n- l210-225 appears out of the blue. It will be helpful to start with the goal of the whole paragraph than diving into technical descriptions.\n- Scene editing is not clearly defined in the context of the this paper. Why is the physics simulator necessary for scene editing? Can't we edit the scene in an interactive manner? \n- Why it is necessary to have a rigidity network in addition to a motion network, given the magnitude of motion represents rigidity?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Originality and Significance**\n- (+) This work is the first to combine prior works of dynamic NeRF and differentiable physics in a joint optimization framework. \n- (-) However, it is not well-motivated why we need this joint optimization. Why not running differentiable rendering and simulation in a step-by-step manner? Where does joint optimization help in geometry and motion recovery, as well as physics parameter recovery? To give readers a better idea, experiments and a thorough analysis is needed. \n- (-) The challenges in combining differentiable rendering and differentiable physics is not clearly presented. As a result, the technical contribution appears weak. l183-l196 sparsely discussed an alternative solution but lacks in-depth analysis. How much does the proposed solution out-performs the alternative solution in compute and accuracy?\n\n**Clarity**\n- (-) The paper is difficult to follow in general. Some sentences are not logically connected. See questions. \n\n**Experiments**\n- (+)The quantitative results have nice details, benefiting from NeuS representation. The material parameter estimation experiments is interesting.  \n- (-) The baseline comparison only shows the proposed method is better but does not provide insight and analysis. What is the difference between the proposed method and the baselines? What does the results suggest? \n- (-) There is no ablation study. For example, I'd like understand how important to jointly solve physics parameters together with differentiable rendering. How important are the loss terms?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "bGGBmePHRK", "reviewer": "Reviewer_CXx9", "summary": "This paper presented a method for learning the geometry and physics parameters of a dynamic scene. The input source is a monocular RGB video. The paper decoupled the learning of dynamic scenes into a static reference neural field and a deformation field. The main structure of the learning part drew similarity to [57], including the offset field, the divergence field with rigidity masks. On top of these, the paper performed mesh reconstruction for the video sequence and send those meshes to a differentiable physics simulator to optimize between the extracted and reconstructed meshes. The method also allows scene editing.", "questions": "1. The abstract (Line 7) stated a design of a \"two-way conversion between the neural field and corresponding mesh representation...\". Where was the two-way explained in the paper?\n\n2. In Line 144, \"t\" was not defined.\n\n3. The connection between the simulator and the previous parts (SDF+motion+rigidity networks) was not clear. Does the simulator only also have static vs motion decomposition? \n\n4. The removal editing video showed remaining sparse object pieces. Is there an explanation?", "limitations": "This paper does not have limitations listed.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strength**\n1. The results presented are pretty impressive. Both quantitative and qualitative results were shown with comparison to previous approaches.\n\n**Weakness**\n1. The paper is hard to follow and not well organized, especially the motivation of the differentiable physics simulator. In fact, most of the demonstration should be used for the simulator, but the current version went over this part hastily. Therefore, it left much confusion to me. For example, \n1.1 The physical parameters to learn were unclear. The L_{physics} is only L-2 distance between two meshes with bending operation. \n1.2 The mesh extraction method (Line 177) is unclear. How were the meshes extracted and how were they used as supervision signal for the physics simulator?\n2. It seems that the simulator can be viewed as a post-processing step after D-NeRF optimization. (Line 242, 243) The integration of all the components was not well demonstrated.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "RO3fZkQ-MO", "reviewer": "Reviewer_zGis", "summary": "The paper presents a method that jointly optimizes shape, appearance, and foreground segmentation from a monocular video based on neural fields. The author then fits the physical parameters of the scene. The experiments demonstrate that the method works better than dynamic nerf and nonrigid nerf in nonrigid neural rendering. Further, the authors demonstrate interesting video editing experiments based on the learned physics.  \n\n", "questions": "**Two-stage procedure**: the method seems to be a two-stage optimization procedure according to Line 242-243. The first stage is optimizing the shape and appearance, and the second stage to optimize the physical parameters. This raises a few concerns. \n- Following the previous comment, it's not convincing to me if we should call the proposed work the first approach that simultaneously optimizes the shape, appearance, and physical parameters (Line 76). The entire pipeline seems to be a dynamic extension of Neus [61] + hexahedral mesh extraction + diffPD.   \n- It's hard to evaluate the contribution of the physical component in this case. It seems we can complete the second stage by directly calling DiffPD once reconstruction is done. I would appreciate if you could make your contribution more clear. What is the proposed method solves/enables for physics reasoning,   \n- Line 171 said the SDF to hexahedral mesh conversion is done through marching cube, which is not a differentiable operator.   Hence, it is not clear if the entire pipeline \"can\" be learned end-to-end.  \n- Following the previous comment, there needs to be a controlled experiment, i.e., dynamic nerf + surface extraction + DiffPD.\n\n**Physics parameter estimation**: the physics part is not clear in both the method description and experiments. Hence it's hard to evaluate the technical innovation and the performance. \n- The method section for the physics component is not clear. First, it's unclear what physical parameters the method optimizes and what are parameters the author needs to provide. Besides, it is unclear what the physical model is (e.g., rigid body physics or soft-body physics). Thirdly, the author should provide some detail on optimizing physical parameters to make it self-inclusive and reproducible (e.g., initialization and optimizer) \n- There needs to be a quantitative measurement for the physics component as a part of the major contributions. The current experiment does not seem to provide such. One could do this either in simulation or partially in the real world (e.g., measuring the acceleration or the mass of the basketball based on a video sequence) \n\n**Minor**: \n- Since the physics component is not very clear in the paper, I briefly checked the implementation provided by the author (thank you for doing this) and have two questions: 1) the physics parameters to optimize are only one-dimensional in both \"gravity\" and \"material\" case - - does your code support joint optimization of multiple physics parameters? 2) It seems you are initializing the gravity parameter to be 0.0378, which coincides with the final reported results of 3.78 m/s^2. ", "limitations": "See above", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Overall the quality of dynamic reconstruction and neural rendering from the author-provided data shows superior performance compared to prior SOTA. That said, I have a few concerns about the physical component as well as the full pipeline. I am currently on the borderline and open to going on either side based on whether the rebuttal addresses my major concerns on \"physics\" well\n\n**Pros**: \n- The paper tackles a very interesting yet challenging problem; \n- The solution is technically sound\n- Experiments show superior results to strong competing algorithms. \n- Code is provided and the video shows impressive editing results. \n\n**Cons**: \n- The method section (particularly the physical component) lacks many details, making it hard to evaluate the technical contribution there. \n- The experiment section lacks a thorough study of the quality of the physical component. \n- The two-stage framework is inconsistent with what is claimed in the intro. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept