Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper presents a method that jointly optimizes shape, appearance, and foreground segmentation from a monocular video based on neural fields. The author then fits the physical parameters of the scene. The experiments demonstrate that the method works better than dynamic nerf and nonrigid nerf in nonrigid neural rendering. Further, the authors demonstrate interesting video editing experiments based on the learned physics.  


strengths_and_weaknesses: Overall the quality of dynamic reconstruction and neural rendering from the author-provided data shows superior performance compared to prior SOTA. That said, I have a few concerns about the physical component as well as the full pipeline. I am currently on the borderline and open to going on either side based on whether the rebuttal addresses my major concerns on "physics" well

**Pros**: 
- The paper tackles a very interesting yet challenging problem; 
- The solution is technically sound
- Experiments show superior results to strong competing algorithms. 
- Code is provided and the video shows impressive editing results. 

**Cons**: 
- The method section (particularly the physical component) lacks many details, making it hard to evaluate the technical contribution there. 
- The experiment section lacks a thorough study of the quality of the physical component. 
- The two-stage framework is inconsistent with what is claimed in the intro. 

questions: **Two-stage procedure**: the method seems to be a two-stage optimization procedure according to Line 242-243. The first stage is optimizing the shape and appearance, and the second stage to optimize the physical parameters. This raises a few concerns. 
- Following the previous comment, it's not convincing to me if we should call the proposed work the first approach that simultaneously optimizes the shape, appearance, and physical parameters (Line 76). The entire pipeline seems to be a dynamic extension of Neus [61] + hexahedral mesh extraction + diffPD.   
- It's hard to evaluate the contribution of the physical component in this case. It seems we can complete the second stage by directly calling DiffPD once reconstruction is done. I would appreciate if you could make your contribution more clear. What is the proposed method solves/enables for physics reasoning,   
- Line 171 said the SDF to hexahedral mesh conversion is done through marching cube, which is not a differentiable operator.   Hence, it is not clear if the entire pipeline "can" be learned end-to-end.  
- Following the previous comment, there needs to be a controlled experiment, i.e., dynamic nerf + surface extraction + DiffPD.

**Physics parameter estimation**: the physics part is not clear in both the method description and experiments. Hence it's hard to evaluate the technical innovation and the performance. 
- The method section for the physics component is not clear. First, it's unclear what physical parameters the method optimizes and what are parameters the author needs to provide. Besides, it is unclear what the physical model is (e.g., rigid body physics or soft-body physics). Thirdly, the author should provide some detail on optimizing physical parameters to make it self-inclusive and reproducible (e.g., initialization and optimizer) 
- There needs to be a quantitative measurement for the physics component as a part of the major contributions. The current experiment does not seem to provide such. One could do this either in simulation or partially in the real world (e.g., measuring the acceleration or the mass of the basketball based on a video sequence) 

**Minor**: 
- Since the physics component is not very clear in the paper, I briefly checked the implementation provided by the author (thank you for doing this) and have two questions: 1) the physics parameters to optimize are only one-dimensional in both "gravity" and "material" case - - does your code support joint optimization of multiple physics parameters? 2) It seems you are initializing the gravity parameter to be 0.0378, which coincides with the final reported results of 3.78 m/s^2. 
limitations: See above
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper presented a method for learning the geometry and physics parameters of a dynamic scene. The input source is a monocular RGB video. The paper decoupled the learning of dynamic scenes into a static reference neural field and a deformation field. The main structure of the learning part drew similarity to [57], including the offset field, the divergence field with rigidity masks. On top of these, the paper performed mesh reconstruction for the video sequence and send those meshes to a differentiable physics simulator to optimize between the extracted and reconstructed meshes. The method also allows scene editing.
strengths_and_weaknesses: **Strength**
1. The results presented are pretty impressive. Both quantitative and qualitative results were shown with comparison to previous approaches.

**Weakness**
1. The paper is hard to follow and not well organized, especially the motivation of the differentiable physics simulator. In fact, most of the demonstration should be used for the simulator, but the current version went over this part hastily. Therefore, it left much confusion to me. For example, 
1.1 The physical parameters to learn were unclear. The L_{physics} is only L-2 distance between two meshes with bending operation. 
1.2 The mesh extraction method (Line 177) is unclear. How were the meshes extracted and how were they used as supervision signal for the physics simulator?
2. It seems that the simulator can be viewed as a post-processing step after D-NeRF optimization. (Line 242, 243) The integration of all the components was not well demonstrated.
questions: 1. The abstract (Line 7) stated a design of a "two-way conversion between the neural field and corresponding mesh representation...". Where was the two-way explained in the paper?

2. In Line 144, "t" was not defined.

3. The connection between the simulator and the previous parts (SDF+motion+rigidity networks) was not clear. Does the simulator only also have static vs motion decomposition? 

4. The removal editing video showed remaining sparse object pieces. Is there an explanation?
limitations: This paper does not have limitations listed.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper proposes a method to recover geometry, motion, and physics parameters from a single video. It recovers geometry and motion with differentiable volume rendering similar to Dynamic NeRF. To recover physics parameters, it uses Diff-PD. It shows results on synthetic and real data with relatively small motion. It also demonstrated applications of user editing.
strengths_and_weaknesses: **Originality and Significance**
- (+) This work is the first to combine prior works of dynamic NeRF and differentiable physics in a joint optimization framework. 
- (-) However, it is not well-motivated why we need this joint optimization. Why not running differentiable rendering and simulation in a step-by-step manner? Where does joint optimization help in geometry and motion recovery, as well as physics parameter recovery? To give readers a better idea, experiments and a thorough analysis is needed. 
- (-) The challenges in combining differentiable rendering and differentiable physics is not clearly presented. As a result, the technical contribution appears weak. l183-l196 sparsely discussed an alternative solution but lacks in-depth analysis. How much does the proposed solution out-performs the alternative solution in compute and accuracy?

**Clarity**
- (-) The paper is difficult to follow in general. Some sentences are not logically connected. See questions. 

**Experiments**
- (+)The quantitative results have nice details, benefiting from NeuS representation. The material parameter estimation experiments is interesting.  
- (-) The baseline comparison only shows the proposed method is better but does not provide insight and analysis. What is the difference between the proposed method and the baselines? What does the results suggest? 
- (-) There is no ablation study. For example, I'd like understand how important to jointly solve physics parameters together with differentiable rendering. How important are the loss terms?
questions: - l194: b(li,mi) needs to be explained. Otherwise, it's unclear what's the relation between mi and the reference frame.
- l210-225 appears out of the blue. It will be helpful to start with the goal of the whole paragraph than diving into technical descriptions.
- Scene editing is not clearly defined in the context of the this paper. Why is the physics simulator necessary for scene editing? Can't we edit the scene in an interactive manner? 
- Why it is necessary to have a rigidity network in addition to a motion network, given the magnitude of motion represents rigidity?
limitations: Yes
review_title: null

Meta-review: metareview: After rebuttal the new version of the paper reads much better and all reviewers were positive, despite some remaining criticisms. Hence the paper should be accepted.
confidence: Certain
Decision: Accept