NeuPhysics: Editable Neural Geometry and Physics from Monocular Videos

Abstract: We present a method for learning 3D geometry and physics parameters of a dynamic scene from only a monocular RGB video input. To decouple the learning of underlying scene geometry from dynamic motion, we represent the scene as a time-invariant signed distance function (SDF) which serves as a reference frame, along with a time-conditioned deformation field. We further bridge this neural geometry representation with a differentiable physics simulator by designing a two-way conversion between the neural field and its corresponding hexahedral mesh, enabling us to estimate physics parameters from the source video by minimizing a cycle consistency loss. Our method also allows a user to interactively edit 3D objects from the source video by modifying the recovered hexahedral mesh, and propagating the operation back to the neural field representation. Experiments show that our method achieves superior mesh and video reconstruction of dynamic scenes compared to competing Neural Field approaches, and we provide extensive examples which demonstrate its ability to extract useful 3D representations from videos captured with consumer-grade cameras.

Reviews and author responses:

 Thanks very much for your constructive feedback and for kindly increasing the score! We will switch the order of the figures. More discussions and failure cases about our experience of connecting the rendering and simulation modules will also be included.

We actually found that sequential training instead of joint training can mitigate inheriting limitations (error accumulation) from both sides. In our strategy, the trained NeRF would not be influenced by the suboptimal physics simulation. If we train them together, it is much more challenging to finetune the learning rate if we train physics parameters and rendering parameters *together*. Also, a bad physics module would break down the rendering module as well. We will show those finding qualitatively and quantitatively. Another potential limitation for both sides might be the running speed. It's an important future work in our calendar to improve the efficiency (e.g. faster GPU-based simulation, CUDA kernels for NeuS rendering)

 We would like to sincerely thank all reviewers for your review and discussion! Your suggestions help us better articulate the contributions and technical details. 

All reviewers are asking questions about the connection between the rendering and simulation, which is a key design in our pipeline. We appreciate those great questions and have added several experiments during the rebuttal period to more clearly highlight our contributions. Reviewer 1MJg proposed a new experiment during the discussion period. Until now, we have not observed significant improvement in that training. A much larger optimization space (with significantly higher dimensionality) would lead to much slower convergence if converging at all within a reasonable amount of time. We will add the results to the final version of the paper.

Based on that, we would further add one more experiment to further analyze and verify how the simulation part DiffPD can potentially benefit the NeRF: the rendered image from the physics simulation can actually be compared with the ground truth image. The closed-loop RGB loss in the image space is another way to update NeRF and this experimental result will also be included in the revision.


Connecting NeRF and differentiable physics modules is not as easy as it seems to be. There are some subtle details we are willing to validate and share. More experiments will be added in a later version.

Once again, we sincerely appreciate all reviewers' comments and discussions to help improving the exposition of this work. THANK YOU ALL.


 The reviewer thanks the authors for the explanation and clarification of several details. The reviewer has read the modified submission and is willing to increase the rating to borderline accept for the added experimentation. 

The reasons for not increasing to a clear accept or higher are due to concerns on presentation and novelty. 
1. Section 3 is divided into a "Rendering pipeline" and a "Simulation pipeline", with the first part talking about NeRF and the second about DiffPD. The integration of the two is perhaps demonstrated in Figure 2, which by itself is confusing and complicated. Figure 1 appears after Figure 2 and is also very hard to read. 
2. At the end of the day, this paper boils down to NeRF+DiffPD. The techniques developed in this paper are to integrate these two modules.  On one hand, the paper did a good in job presenting the benefits by combining the two for soft dynamic objects. But on the other hand, it is unclear and probably true that it inherits limitations from both ends, which is not discussed in the paper.

 Dear reviewer,

Thanks for your helpful and practical advice! 

We are starting to add the joint training comparisons as you suggested. Results will be added to the revised paper if we can catch the 8/9 deadline, or at least it will be in a future version.

Thank you for bringing the two papers to our attention.  Those two are using multiview videos with articulated assumptions. In contrast, our method uses a single video for general objects (e.g. balls, human body, face, hands). We will add the discussions to our paper. 

But for sure, we would tone down our claims, since NeRF is growing very fast and it's very likely there are some concurrent and similar works just like these two CVPR 2022 papers. CVPR 2022 is in this June after the Neurips deadline so it's hard for us to make the comparison.

Best, Authors



 I appreciate the effort the authors made in the rebuttal. Overall, I like the idea of combining differentiable simulation with NeRF, but the analysis and solution is still not satisfactory.

- Joint optimization. Given the additional information, it is still not clear whether the NeRF and physics parameters are jointly optimized or not. I would guess they are not after reading the response to other reviewers. "Sequential vs from scratch" is a different problem than "joint vs separate" optimization. It is fine to train sequentially due to efficiency considerations, but lack of joint optimization often gives suboptimal solutions. Coordinate descent-like strategies may allows joint optimization but in a sequential manner. It may be worth to study how to improve the speed of differentiable simulator and the joint optimization.

- Overclaim/Missing references. The point of using sdf representation in dynamic reconstruction is rather incremental. It is also not fair to claim "there is no paper known that can reconstruct high-quality dynamically changing surface mesh as this work does". For instance, [A,B] use sdf representation and show good results on dynamic objects from rgb videos.

[A] Noguchi, Atsuhiro, et al. "Watch It Move: Unsupervised Discovery of 3D Joints for Re-Posing of Articulated Objects." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.

[B] Yang, Gengshan, et al. "Banmo: Building animatable 3d neural models from many casual videos." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.

 Dear reviewers,

We wish you had a great weekend! Thanks to your constructive suggestions, our paper has been revised with additional ablation studies and technical details. 

For both rendering and simulation researchers, our method provides an easy-to-use pipeline reconstructing the geometry and dynamics. We also hope this technique can enable more general users to build and edit dynamic 3D models simply with their phones.

Could you please kindly let us know if there is anything we can do or clarify that might improve your rating? Looking forward to your post-rebuttal discussion.

Best, Authors

 
> The challenges in combining differentiable rendering and differentiable physics are not clearly presented. As a result, the technical contribution appears weak. l183-l196 sparsely discussed an alternative solution but lacks in-depth analysis. How much does the proposed solution out-perform the alternative solution in computing and accuracy?

Thanks for your question. The challenge is to make such combination both efficient and differentiable. We have added an experiment in NEW Section 4.5 to show that our way of combining differentiable rendering and differentiable physics is 75% more efficient than the alternative naive solution. NEW Figure 2 is added to demonstrate how to differentiate the cycle-consistency physics loss.

This comparison is added (in NEW section 4.5 and Figure 8) where we 1) extract the hex mesh sequence from our rendering module; 2) simulate another mesh sequence using the simulation module;  3) compute the pairwise Chamfer distance between the two sequences and update the physics parameters.

The results indicate that the disconnected workflow has comparable performance to ours in terms of loss convergence. However, it takes 75% more computational time than ours because it reconstructs the mesh for each frame, while our method avoids the expensive reconstruction by querying the motion network to compute the cycle-consistency physics loss.

This experiment also verifies the correctness of our designed loss function, since it is almost as effective as the Chamfer distance, yet much more computationally efficient.

There are two more potential advantages of using our loss instead of such a disconnected workflow:  (1) A mesh reconstructed purely from the SDF cannot reflect the inside stress and deformation gradients, as the reconstructed mesh elements are uniformly distributed. Our proposed cycle-consistency physics loss takes advantage of the learned motion fields, so it is capable of finding the point-wise correspondence and characterizing the deformation gradients. (2) Our cycle-consistency physics loss queries the motion networks, and can therefore pass gradients to the rendering module. The decoupled workflow, however, leads to a non-differentiable module for rendering.
 

> Joint optimization for geometry and motion recovery, as well as physics parameter recovery. 

Jointly training the entirely differentiable modeling-simulation-rendering pipeline looks elegant and was actually our initial goal. However, we can only do sequential optimization due to the performance issue. It could be an interesting and challenging future work to study the benefit of joint optimization for geometry and motion recovery, as well as physics parameter recovery. 

Our method is indeed trained in a sequential optimization manner for two reasons.  (1) The joint optimization is too expensive.  We have added an experiment and some profiling numbers to show this difference clearly in NEW Section 4.5 and Figure 7.  (2) The physics simulation is only well-defined after the geometry reconstruction converges reasonably well.

> The baseline comparison only shows the proposed method is better but does not provide insight and analysis. What is the difference between the proposed method and the baselines? What does the results suggest?

Thanks for asking! To the best of our knowledge, no existing NeRF methods can reconstruct high-quality dynamic geometry sequences from monocular RGB videos as ours does. Previous Dynamic NeRF papers (D-NeRF, NR-NeRF) use density for geometry representation. However, the mesh reconstruction using density values requires an ad-hoc cut-off threshold, making the surface mesh noisy. Our use of SDF + unbiased weighting scheme for volume rendering leads to much smoother geometry results. Our method also outperforms the original NeuS, as shown in the images and videos, since we have extended the static SDF representation to make it work well for dynamic scenes.  

In addition, the use of a physics engine also helps us edit the videos in an intuitive and convenient manner. Existing NeRF editing tools cannot perform physically-plausible editing (or otherwise require manual interaction), as they do not have an underlying physics module to create natural and realistic deformation due to interaction. 

Moreover, previous learning-based video prediction methods do not adopt a powerful differentiable physics engine as ours does. Their prediction often lacks interpretability, while our video prediction (like adding a horizontal velocity in NEW Figure 7) is totally understandable, explainable, and predictable, because the governing physics laws are explicitly encoded inside the physics engine.


 > There is no ablation study. For example, I'd like to understand how important to jointly solve physics parameters together with differentiable rendering. How important are the loss terms?

Thank you for your suggestions. We have now included ablation studies to analyze the design and effects of the physics component in NEW Section 4.5. We compare our designed cycle-consistency physics loss with the chamfer distance. Ours is as effective as the widely used chamfer distance but is 75% more efficient than the alternative naive solution. Other vanilla NeRF losses are already studied in some previous papers. We would recommend [57] as a reference.

We also add an ablation study regarding the training strategy in NEW Section 4.5 on “Sequential Vs. Joint Optimization”. It clearly illustrates that joint training is too time-consuming while the benefit is limited. 

> l194: b(li, mi) needs to be explained. Otherwise, it's unclear what's the relation between mi and the reference frame.

Thanks for your suggestions. We have modified the description in NEW Equation 4. $b(I_i, m_i)$ is the motion field at the i-th frame that can map the mesh $m_i$ to the canonical frame. 


> l210-225 appears out of the blue. It will be helpful to start with the goal of the whole paragraph than diving into technical descriptions.

Thank you for your advice. We rewrite this paragraph by starting with our intention as below:

“When rendering a point $p'$ for the edited image, we must first check if this point is being moved by our editing operation. An equivalent question is whether this point is located within the hexahedral mesh $m_i'$. The inside-outside-test would be trivial if all hexahedra in $m_i'$ remained in a regular grid”

> Scene editing is not clearly defined in the context of this paper. Why is the physics simulator necessary for scene editing? Can't we edit the scene in an interactive manner?

Thanks for asking this question. Scene editing in our paper includes adding, deleting, moving, deforming objects, or even simulating deformation using different physics parameters in an existing video. We show the examples in our supplementary video and NEW Figure 5 and 6.

Let’s take the editing of a bouncing-ball video as an example. Without our learned physics simulator, if an artist wants to edit the material with only interactive editing, they would first find the frames where the ball contacts the ground; then they would attach some handles to the ball and drag the handles to modify the shape.  Most importantly, the user needs to deform the ball **every frame** and ensure the physical correctness and consistency. This process would take a significant amount of time and manual efforts of a very skilled animator and/or artist – yet the results and visual quality may still appear unrealistic and/or inconsistent.

Conversely, in our framework, users can simply modify the physics parameters in the learned simulator and our simulator produces reasonably plausible, realistic animation results automatically. Our proposed editing pipeline in NEW Figure 1 could make video editing and generation much, much easier and faster in VR and/or 3D interactive metaverse applications.

> Why it is necessary to have a rigidity network in addition to a motion network, given the magnitude of motion represents rigidity?

This is an excellent question.  We have added the comparison in NEW Section 4.5 on “Rigidity Map” and in Figure 9. The rigidity network can disambiguate moving foreground vs. static background. An intuitive alternative is to use the magnitude of the motion field as a criterion to separate dynamic foreground, i.e. large motion area corresponds to dynamic objects (and the converse). However, in some frames, dynamic parts might only contain small offsets from the canonical frame. For example, a cyclic bouncing ball could overlap with the canonical position and thus have offset values of zero, even though it should be classified as a dynamic area. 

NEW Figure 9 (a, b) filters the scene using the motion magnitude. It is not a good criterion, since a large portion of the background is still left, while the ball is already incomplete in (b). The separation using the rigidity map in Figure 9 (d) is much better. The reason is that the rigidity map collects the motion information from all frames and it is not restricted to a single frame.

*We sincerely appreciate your comments. Please let us know if you have further feedback.*

Best, Authors

 > The motivation of the differentiable physics simulator. In fact, most of the demonstration should be used for the simulator, but the current version went over this part hastily. 

Thanks for your suggestions. We have added more details about the physics system in NEW Section 3.2 and Equation 3. More ablation studies relating to the physics component are conducted in NEW Section 4.5.

As for the motivation, our paper aims to contribute to both NeRF and differentiable physics.
 
On the NeRF side, there is no paper we know that can reconstruct high-quality dynamic surface mesh as ours does, which is shown in the video and in NEW Section 4.1. Moreover, the physics engine can greatly assist NeRF’s editing and interaction capabilities. A differentiable physics simulator can learn editable digital twins and animate dynamic objects in the scene automatically. Without such a differentiable physics engine, previous NeRF editing techniques would otherwise have to heavily rely on user interaction (manually designing motion trajectories and deformation handles), which is tedious and time-consuming, most probably physically incorrect and/or implausible. 

For differentiable physics, our geometry reconstruction module can provide accurate, dynamically changing 3D models for the simulator. In our limited experience dealing with differentiable physics, previous works are largely constrained by the modeling technique and mainly start with predefined, relatively simple, and fixed meshes. NeRF, as both a modeling and rendering module, can introduce new opportunities in applications of differentiable physics, since it is well-suited to handle unconstrained, out-of-domain, and/or real-world data.The input videos in our paper are either captured in the real world or synthesized in Blender. While the governing equations behind the source videos are different from those in DiffPD, we show that differentiable physics can still work well regardless of the model mismatch.  

> The physical parameters to learn were unclear. The $L_{physics}$ is only L-2 distance between two meshes with bending operation. 

Thanks for your question. The cycle-consistency physics loss, $L_{physics}$, is an important bridge between the rendering and simulation pipeline. We have added a diagram in NEW Figure 3 to show what parameters are learned by minimizing this loss. We also rewrite NEW Equation 4 to show how the rigidity network, motion network, and physics parameters are coupled and inter-connected in this loss function.

> The mesh extraction method (Line 177) is unclear. How were the meshes extracted and how were they used as supervision signals for the physics simulator? 

Thanks for letting us know the confusion. More details are added in New Section 3.2 Geometry Reconstruction: We sample over a 3D grid to find vertices within volume $\mathcal{A}_i$. If a grid point $p$ satisfies NEW Equation 2, we add this voxel to the hexahedral volume mesh $m_i$. 

We also add a visualization in NEW Figure 9 to show how we extract the mesh using the rigidity map and a region-of-interest bounding box. The extracted mesh is used to initialize the simulation. Most of the supervision signals come from the learned motion field. We show the computation of the cycle-consistency physics loss, $L_{pyhiscs}$, as defined by Equation 4
in NEW Figure 2.


 > It seems that the simulator can be viewed as a post-processing step after D-NeRF optimization. (Line 242, 243) The integration of all the components was not well demonstrated.

Each part of the rendering-simulation combination takes advantage of each other for editing, system identification, and digital twinning. NEW Figure 2 shows how they are coupled.

During training, we are indeed performing a sequential optimization for the performance issue. We have added an ablation study in NEW Section 4.5 Sequential Vs. Joint Optimization to show that joint training is too time-consuming while the benefit it provides is limited. 

A comparison of actually just using the simulator as post-processing is demonstrated in NEW Section 4.5 ​​Physics Module. In this experiment, we (1) extract the hex mesh sequence from our rendering module; (2) simulate another mesh sequence using the simulation module;  (3) compute the pairwise Chamfer distance between the two sequences and update the physics parameters. Results indicate that the disconnected workflow has comparable performance to ours in terms of loss convergence. However, it takes 75% more computational time than ours because it reconstructs mesh for each frame, while our method avoids the expensive reconstruction by querying the motion network to compute the cycle-consistency physics loss.


> The abstract (Line 7) stated a design of a "two-way conversion between the neural field and corresponding mesh representation...". Where was the two-way explained in the paper?

Thanks for pointing out the need for clarification on this.

Neural fields -> Mesh. The neural fields define a signed distance function that can be used to extract the mesh by the marching cube algorithm (OLD L171). We added more explanation in New Section 3.2 Geometry Reconstruction and New Figure 9 to visualize the process.

Mesh -> Neural fields. On the other hand, the mesh representation is used as a tool to edit the neural fields as described in OLD L197 (also NEW Figure 1). The deformed mesh can induce a piecewise linear bending field by interpolation.

The two-way conversion enables our method to take advantage of both the implicit and explicit representation, making it easier to edit and animate the neural field.


> In Line 144, $t$ was not defined.

Thanks! We have added the definition to the revised version. $t$ is the distance from the camera origin center ($o$), to a point along the ray in the direction $v$.

> The connection between the simulator and the previous parts (SDF+motion+rigidity networks) was not clear. Does the simulator also have static vs motion decomposition?

We have added a diagram in NEW Figure 2 to better show the connection. Yes, the simulator also has decomposition and only simulates the moving parts. The decomposition is from the rigidity networks. Its visualization is also added in NEW Figure 9.

> The removal editing video showed remaining sparse object pieces. Is there an explanation?

Good question! Yes, we have a conjecture about the artifacts. Since the MLP fields are continuous, there are some close-to-zero SDF values (i.e. high probability density) near the deleted object area. Therefore, the rays can terminate in those regions randomly due to the sampling strategy in volume rendering, even if the main object is already gone. It could be an interesting future work on how to improve the editing quality.

*We sincerely appreciate your comments. We hope that our response has addressed your concerns, and turns your assessment to the positive side. If you have any additional questions, please feel free to let us know during the rebuttal window.*

Best, Authors

 > I would appreciate it if you could make your contribution clear. What is the proposed method solves/enables for physics reasoning,

Our paper exploits the model and representation from NeRF as input to the differentiable physics.  This step enables us to couple NeRF and differentiable physics together and apply them to more complex real-world applications, by estimating geometry, appearance, and physical properties all from one monocular video.

Based on our prior experience working with differentiable physics, the domain and data are very important. Previous differentiable simulation works usually demonstrate results using simple scenes as input, and they assume that the optimization starts with an existing initial mesh, whereas we make no such assumptions.  This paper aims to provide an easy way to create high-quality 3D models directly from out-of-domain videos, such that the powerful differentiable physics engine can be applied and adopted to more diverse real-world scenarios. Although we only study soft body dynamics in this paper, other possible future avenues on fluids, cloth, and rigid bodies can also be very important for creating digital twins. 

Furthermore, this coupling of differentiable physics and NeRF to video prediction and editing can reproduce the physics environment directly from an existing video, and then generate physically-plausible variations automatically.

We design a differentiable rendering-modeling-simulation pipeline, as shown in the computational graph (NEW Figure 2).  In its current state, our training strategy does not fully utilize the gradients to improve the geometry reconstruction using the cycle-consistency physics loss (NEW Section 4.5 and Figure 7) and is left for future work (OLD Line 244) to explore how such a closed-loop approach can further improve the visual appearance. We hope that the power of physics reasoning will introduce new directions for NeRF-based rendering and modeling going forward.

> The method section (particularly the physical component) lacks many details, making it hard to evaluate the technical contribution there.

Thanks for pointing this out. The objects in our demos are soft bodies with co-rotated materials integrated by projective dynamics.  We have added more details about the physics module in NEW Section 3.2. 


> It's hard to evaluate the contribution of the physical component in this case. It seems we can complete the second stage by directly calling DiffPD once reconstruction is done. There needs to be a controlled experiment, i.e., dynamic nerf + surface extraction + DiffPD.

Thanks for your proposal. A controlled experiment is added (in NEW section 4.5 and Figure 8) where we (1) extract the hex mesh sequence from our rendering module; (2) simulate another mesh sequence using the simulation module;  (3) compute the pairwise Chamfer distance between the two sequences and update the physics parameters.

The results indicate that the uncoupled workflow has comparable performance to ours in terms of loss convergence. However, it takes 75% more computational time than ours because it reconstructs mesh for each frame, while our method avoids the expensive reconstruction by querying the motion network to compute the cycle-consistency physics loss.

This experiment also verifies the correctness of our designed loss function, since it is almost as effective as the Chamfer distance and significantly reduces the overall computation time. There are two more potential advantages of using our loss instead of such a decoupled workflow:  (1) A mesh reconstructed purely from the SDF cannot reflect the inside stress and deformation gradients because the reconstructed mesh elements are uniformly distributed. Our proposed cycle-consistency loss takes advantage of the learned motion fields, so it is capable of finding the point-wise correspondence and describing the deformation gradients. (2) Our cycle-consistency loss queries the motion networks, and can therefore pass gradients to the rendering module (which is added in NEW Section 4.5 and Figure 7). The decoupled workflow however leads to a totally non-differentiable module for rendering unlike ours.


 >  The method seems to be a two-stage optimization procedure according to OLD Line 242-243. Following the previous comment, it's not convincing to me if we should call the proposed work the first approach that simultaneously optimizes the shape, appearance, and physical parameters (OLD Line 76). The entire pipeline seems to be a dynamic extension of Neus [61] + hexahedral mesh extraction + diffPD.

We have added a computational graph in NEW Figure 2 to illustrate the forward and backward flow of our entire pipeline. Our method is indeed trained in a sequential optimization manner for two reasons.  (1) The joint optimization is too expensive.  We have added an experiment and some profiling numbers to show this difference clearly in NEW Section 4.5 and Figure 7.  (2) The physics simulation is only well-defined after the geometry reconstruction converges reasonably well. We believe that it would be an interesting but challenging future work to improve the physics training strategy and explore how physics priors can further improve or refine geometry reconstruction (OLD Line 244). 

As for the claim in OLD Line 76, The word 'simultaneously' is used because we treat our pipeline as a whole, where the geometry, appearance, and dynamics parameters are all the results of the sequential optimization. We now realize that this wording might be confusing, and have deleted it in the new version. To the best of our knowledge, our method is the first approach that estimates the shape, appearance, and physical parameters using merely a single monocular RGB video. However, we also tone down the claim to avoid any confusion. Now our claim for contribution is “We present a framework capable of estimating and editing the geometry, appearance, and physics parameters directly from a single monocular video.” 



> Line 171 said the SDF to hexahedral mesh conversion is done through marching cube, which is not a differentiable operator. 

Marching cube and sampling-based mesh reconstruction are indeed non-differentiable. 

Thanks for your questions. We will add a diagram to show the computational graph and how the gradients flow (in NEW Figure 2). We do not utilize the gradients of the mesh reconstruction. The differentiable path connecting the rendering module and the simulation module is the rigidity and motion network and cycle-consistency loss described in OLD L189-L196 (NEW Equation 4). In this paper, we mainly use the differentiable physics engine to perform physics parameter estimation and scene editing. We hope this work will inspire other follow-up work on the use of differentiable physics to improve both NeRF-based reconstruction and rendering. 


> The physics part is not clear in both the method description and experiments. 
A: Thanks for this suggestion. We have added more details about the physics engine in NEW Section 3.2. 

More training details are included in New Sections 3.3 and 4.3. We choose to use co-rotated material to model the bouncing ball and estimate Young's modulus and vertical acceleration in two separate optimization tasks. The cycle-consistency physics loss is minimized for 100 epochs using Adam optimizer with a learning rate of 0.01. The optimization starts with Young's modulus of $2\times 10^5 Pa$ and acceleration of $0m/s^2$. The estimated Young's modulus is $2.96\times 10^6 Pa$ and acceleration is $3.78m/s^2$.


> There needs to be a quantitative measurement for the physics component as a part of the major contributions. 

We have added the experiments as suggested and measure the Chamfer Distance as a quantitative metric in NEW Section 4.5. This experiment verifies the correctness of our designed loss function. It also shows that our physics component is significantly more efficient than a completely decoupled and disconnected workflow, where a chamfer loss is computed against each pair of sampled meshes and/or simulated meshes.


 > the physics parameters to optimize are only one-dimensional in both "gravity" and "material" case - - does your code support joint optimization of multiple physics parameters? 

A: As a proof of concept, the experiments shown in our paper only optimize and edit one parameter at a time sequentially, as in standard sequential optimization that achieves faster convergence in practice. We made the description more accurate in NEW Section 4.3. “We estimate Young's modulus and vertical acceleration respectively in two separate optimization tasks.” 

Thanks for your suggestions! We are happy to have better documented demo scripts and include multiple parameters to estimate when releasing our repository.

> It seems you are initializing the gravity parameter to be 0.0378, which coincides with the final reported results of 3.78 $m/s^2$.

A: The code snippets with 0.0378 are used to render the results of editing horizontal velocities (video 2:40). We first estimate the gravity and then use it as a known variable for editing. During editing, we set the physics_epoch=1. The 0.0378 -> 3.78 is due to the time scaling. We set $v_{i+1} = v_i + acc$ for each time step. The time step is 0.01 so the acceleration in one second is 0.0378 / 0.01 = 3.78. We also provide the learning curve in NEW Figure 8.

Thanks for pointing this out. The submitted code was packaged when we also used it for making the video. We apologize for the poor version control. In our released code, we will avoid reusing the same function name for both editing and training to improve the readability. Detailed tutorial scripts for training and editing will also be provided in the public release.

*We sincerely appreciate your comments. Please let us know if you have further feedback.*

Best, Authors


 We sincerely thank all reviewers for their insightful and helpful comments, and are glad that they found our work to be “tackling an interesting and challenging problem” and “having impressive results”. Their constructive suggestions help us identify confusing expressions or inadequate explanations due to our assumptions. We have added a few more qualitative and quantitative experiments to explain how our design choices benefit the physics part, as well as the entire pipeline. More visualization and technical details are also added to improve the presentation of our paper. We cordially invite the reviewers to read our newly uploaded revision to the paper. To avoid confusion, we use “NEW Section xxx” when referring to the rebuttal revision and “OLD Line xxx” for the prior reviewed version.

First, we would like to better motivate our work. The goal of this paper is to bridge two recent topics of significant and growing interest: NeRF and differentiable physics. Together they enable us to explore the potential applications of ML in creating digital twins and metaverse applications. This paper aims to demonstrate that the two techniques can take advantage of each other.

On the NeRF side, there is no paper known that can reconstruct high-quality *dynamically changing* surface mesh as this work does, which is shown in the video and in NEW Section 4.1. Moreover, the physics engine can greatly help NeRF’s editing and interaction capabilities. A differentiable physics simulator can learn editable digital twins and animate dynamic objects in the scene automatically. Without such a physics engine, previous NeRF editing techniques heavily rely on user interaction (manually designing motion trajectories and deformation handles).

For differentiable physics, our geometry reconstruction module can provide accurate, dynamic 3D models for the simulator. In our limited experience dealing with differentiable physics, previous works are largely constrained by the modeling technique and mainly start with predefined, relatively simple, and fixed meshes. NeRF, as both a modeling and rendering module, can introduce new opportunities in applications of differentiable physics, since it is well-suited to handle unconstrained, out-of-domain, and/or real-world data. The input videos in our paper are either captured in the real world or synthesized in Blender. While the governing equations behind the source videos may be different from those in DiffPD, we show that differentiable physics can still work well regardless of the model mismatch. 

This paper explores the concept of integrating a NeRF-based representation with differentiable physics.  We hope that this will inspire more follow-up works along this line of investigation – to model, render, and recreate dynamic scenes with highly complex physical behaviors, such as splashing water, flowing willows in the wind, and deformable and articulated robots from the real world.  We hope that this direction will open many new possibilities to create dynamic digital twins and Metaverse using ML.

There are also many great questions. For example, reviewer zGis asked what our physics system is. This work only deals with soft body dynamics.   But there could be other future work dealing with cloth, rigid bodies, fluids, granular materials, etc.  They are all very interesting and challenging. Reviewer CXx9 asked about the artifacts in the editing. We believe these are caused by the SDF-based volume rendering, which could possibly be another interesting NeRF problem to investigate. Reviewer 1MJg asked about the potential benefit of joint optimization. Jointly training the fully-differentiable modeling-simulation-rendering pipeline looks elegant and was actually our initial goal. However, the runtime performance is a major issue. A faster simulation or smarter training tricks are needed to fully exploit this system. Therefore, we did not claim the physics prior can help the geometry reconstruction, and leave the joint optimization as future work in our paper (OLD Line 243). Therefore, our current system opts for sequential optimization instead, in favor of higher performance.

In response to the reviews, we have added some extra explanations. Here is a summary of the key changes.

- Reword and clarify the claims about our contributions in NEW Section 1. 
- Add more details about the physics engine in NEW Section 3.2.
- Add a diagram to illustrate the computational graph in NEW Figure 2.
- Add more details on training in NEW Sections 3.3 and 4.3.
- Add three ablation studies to show the effectiveness of the physics module, our sequential training strategy, and the foreground selection process in NEW section 4.5.

*We hope that our response has addressed your concerns. If you have any additional questions, please feel free to let us know during the rebuttal window.*

*Thank you again for your time and consideration.*



 The paper presents a method that jointly optimizes shape, appearance, and foreground segmentation from a monocular video based on neural fields. The author then fits the physical parameters of the scene. The experiments demonstrate that the method works better than dynamic nerf and nonrigid nerf in nonrigid neural rendering. Further, the authors demonstrate interesting video editing experiments based on the learned physics.  

 Overall the quality of dynamic reconstruction and neural rendering from the author-provided data shows superior performance compared to prior SOTA. That said, I have a few concerns about the physical component as well as the full pipeline. I am currently on the borderline and open to going on either side based on whether the rebuttal addresses my major concerns on "physics" well

**Pros**: 
- The paper tackles a very interesting yet challenging problem; 
- The solution is technically sound
- Experiments show superior results to strong competing algorithms. 
- Code is provided and the video shows impressive editing results. 

**Cons**: 
- The method section (particularly the physical component) lacks many details, making it hard to evaluate the technical contribution there. 
- The experiment section lacks a thorough study of the quality of the physical component. 
- The two-stage framework is inconsistent with what is claimed in the intro. 
 **Two-stage procedure**: the method seems to be a two-stage optimization procedure according to Line 242-243. The first stage is optimizing the shape and appearance, and the second stage to optimize the physical parameters. This raises a few concerns. 
- Following the previous comment, it's not convincing to me if we should call the proposed work the first approach that simultaneously optimizes the shape, appearance, and physical parameters (Line 76). The entire pipeline seems to be a dynamic extension of Neus [61] + hexahedral mesh extraction + diffPD.   
- It's hard to evaluate the contribution of the physical component in this case. It seems we can complete the second stage by directly calling DiffPD once reconstruction is done. I would appreciate if you could make your contribution more clear. What is the proposed method solves/enables for physics reasoning,   
- Line 171 said the SDF to hexahedral mesh conversion is done through marching cube, which is not a differentiable operator.   Hence, it is not clear if the entire pipeline "can" be learned end-to-end.  
- Following the previous comment, there needs to be a controlled experiment, i.e., dynamic nerf + surface extraction + DiffPD.

**Physics parameter estimation**: the physics part is not clear in both the method description and experiments. Hence it's hard to evaluate the technical innovation and the performance. 
- The method section for the physics component is not clear. First, it's unclear what physical parameters the method optimizes and what are parameters the author needs to provide. Besides, it is unclear what the physical model is (e.g., rigid body physics or soft-body physics). Thirdly, the author should provide some detail on optimizing physical parameters to make it self-inclusive and reproducible (e.g., initialization and optimizer) 
- There needs to be a quantitative measurement for the physics component as a part of the major contributions. The current experiment does not seem to provide such. One could do this either in simulation or partially in the real world (e.g., measuring the acceleration or the mass of the basketball based on a video sequence) 

**Minor**: 
- Since the physics component is not very clear in the paper, I briefly checked the implementation provided by the author (thank you for doing this) and have two questions: 1) the physics parameters to optimize are only one-dimensional in both "gravity" and "material" case - - does your code support joint optimization of multiple physics parameters? 2) It seems you are initializing the gravity parameter to be 0.0378, which coincides with the final reported results of 3.78 m/s^2.  See above

 This paper presented a method for learning the geometry and physics parameters of a dynamic scene. The input source is a monocular RGB video. The paper decoupled the learning of dynamic scenes into a static reference neural field and a deformation field. The main structure of the learning part drew similarity to [57], including the offset field, the divergence field with rigidity masks. On top of these, the paper performed mesh reconstruction for the video sequence and send those meshes to a differentiable physics simulator to optimize between the extracted and reconstructed meshes. The method also allows scene editing. **Strength**
1. The results presented are pretty impressive. Both quantitative and qualitative results were shown with comparison to previous approaches.

**Weakness**
1. The paper is hard to follow and not well organized, especially the motivation of the differentiable physics simulator. In fact, most of the demonstration should be used for the simulator, but the current version went over this part hastily. Therefore, it left much confusion to me. For example, 
1.1 The physical parameters to learn were unclear. The L_{physics} is only L-2 distance between two meshes with bending operation. 
1.2 The mesh extraction method (Line 177) is unclear. How were the meshes extracted and how were they used as supervision signal for the physics simulator?
2. It seems that the simulator can be viewed as a post-processing step after D-NeRF optimization. (Line 242, 243) The integration of all the components was not well demonstrated. 1. The abstract (Line 7) stated a design of a "two-way conversion between the neural field and corresponding mesh representation...". Where was the two-way explained in the paper?

2. In Line 144, "t" was not defined.

3. The connection between the simulator and the previous parts (SDF+motion+rigidity networks) was not clear. Does the simulator only also have static vs motion decomposition? 

4. The removal editing video showed remaining sparse object pieces. Is there an explanation? This paper does not have limitations listed.

 The paper proposes a method to recover geometry, motion, and physics parameters from a single video. It recovers geometry and motion with differentiable volume rendering similar to Dynamic NeRF. To recover physics parameters, it uses Diff-PD. It shows results on synthetic and real data with relatively small motion. It also demonstrated applications of user editing. **Originality and Significance**
- (+) This work is the first to combine prior works of dynamic NeRF and differentiable physics in a joint optimization framework. 
- (-) However, it is not well-motivated why we need this joint optimization. Why not running differentiable rendering and simulation in a step-by-step manner? Where does joint optimization help in geometry and motion recovery, as well as physics parameter recovery? To give readers a better idea, experiments and a thorough analysis is needed. 
- (-) The challenges in combining differentiable rendering and differentiable physics is not clearly presented. As a result, the technical contribution appears weak. l183-l196 sparsely discussed an alternative solution but lacks in-depth analysis. How much does the proposed solution out-performs the alternative solution in compute and accuracy?

**Clarity**
- (-) The paper is difficult to follow in general. Some sentences are not logically connected. See questions. 

**Experiments**
- (+)The quantitative results have nice details, benefiting from NeuS representation. The material parameter estimation experiments is interesting.  
- (-) The baseline comparison only shows the proposed method is better but does not provide insight and analysis. What is the difference between the proposed method and the baselines? What does the results suggest? 
- (-) There is no ablation study. For example, I'd like understand how important to jointly solve physics parameters together with differentiable rendering. How important are the loss terms? - l194: b(li,mi) needs to be explained. Otherwise, it's unclear what's the relation between mi and the reference frame.
- l210-225 appears out of the blue. It will be helpful to start with the goal of the whole paragraph than diving into technical descriptions.
- Scene editing is not clearly defined in the context of the this paper. Why is the physics simulator necessary for scene editing? Can't we edit the scene in an interactive manner? 
- Why it is necessary to have a rigidity network in addition to a motion network, given the magnitude of motion represents rigidity? Yes

Meta-review: After rebuttal the new version of the paper reads much better and all reviewers were positive, despite some remaining criticisms. Hence the paper should be accepted.

Decision: Accept