Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper introduces a template-free method to reconstruct a high quality geometry and motion of a dynamic scene from a single RGB-D camera. It proposes a bijective deformation map to preserve the cycle consistency between two frames, thus it doesn’t require any scene or optical flow map. To handle the topology changes, the deformation network is combined with a topological-aware network. Experimental results show that the proposed method overcomes the state-of-the-art RGBD methods, such as DynamicFusion and OcclusionFusion, and RGB methods, such as BANMO.
strengths_and_weaknesses: # Strenghts

- A novel 3D reconstruction network of a dynamic scene from a single RGB-D camera. The combination between topology-aware network and deformation-based network makes the network to model the geometry and motion of a dynamic object.
- The proposed bijective map is able to preserve the cycle-consistency because it models the points in 3D observation space to the points in 3D canonical space.

# Weaknesses

- There is only a qualitative ablation study. The overall framework is like an extended version of HyperNERF for RGBD videos. Thus, it is essential to perform a quantitative ablation study, especially compared to HyperNERF.
- Lack of quantitative experimental results. The proposed method only performs a qualitative evaluation for various comparison methods. While qualitative evaluation can be subjective, it is essential to perform quantitative evaluation, especially with BANMo, HyperNERF, VolumeDeform, etc.
- It is unclear about the cycle-consistency performance of the proposed bijective map. The evaluation method only focuses on a single frame. There should be a way to evaluate the consistency between frames because it is also a part of the proposed contribution.
- It is recommended to follow the HyperNERF Fig. 8 to show the performance of the proposed method. It is unclear how the topology-aware and bijective map-based deformation networks affect the overall performance.

questions: - While the proposed method utilizes depth from Kinect as the ground truth, is there any effect if the captured depth information is noisy? Note that an active depth camera could not be used as the ground truth due to its noisy characteristic?
- Why don’t use a synthetic dataset for the ablation study to prove the proposed idea?

limitations: The authors have addressed the limitations in the conclusion.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes a template-free RGB-D based 3D scene reconstruction method that handles non-rigid changes with deformation in dynamic scenes.
The proposed method, Neural Dynamic Reconstruction (NDR) follows the similar steps with the classic dynamic scene reconstruction methods.
It uses a neural implicit function for surface representation, and thus using neural signed distance fields (SDF) and proposes a novel neural invertible deformation network that utilizes a bijective map between frames and a canonical space in order to constrain the non-rigid deformation of observed surfaces.
The paper also adds a topology aware network that tackles the well known challenges of dynamic scene reconstruction of the free-form deformation, where dramatic changes of topology (or assumptions of topology) could make handling deformation/motion constraints hard.
The experiments show that the proposed method outperforms (in the context of the accuracy of surface reconstruction over time) the other existing monocular dynamic scene reconstruction methods.

strengths_and_weaknesses: I appreciate the research effort from the authors. The results look very impressive and the contribution look very clear. Here are the +/- of the proposed methods and the submitted article.

+Convincing results compared to existing methods

+Provides solutions of each challenging limitation of classic methods (and other latest methods).

+Great idea on the use of bijective map together with topology aware network

-Need more detailed discussion and the evaluation of the bijective map and topology aware network

-Need more detailed discussion of computational expenses and how to handle them.  

questions: To make the paper more solid and to help the readers understand the article more clearly, I enumerated several questions below. Some questions may focus on how far the use of the proposed method is from the real world applications.

-How much offset the method can handle refining camera poses, and how much error or residual that wrongly initialized camera pose could be handled?

-How the segmentation results affect the overall results? How do the residuals in the boundary of target surfaces affect the quality of the reconstruction results?

-What is the sole inference time (including optimization steps) for each example? Particularly, providing some numbers, settings (settings, pose accuracy, # samples for each example demonstrated in the result sections would be very helpful to understand the correlation between the complexity of the scene/topology/motion.

-Is there any number or visualization that shows how accurately the bijective map is constructed?

-How wrongly handled topology affects the bijective map quality?

Regarding the importance of the major contribution of this paper, rather than the comparison to the 6D motion, extra discussion on the bijective map and its dependency on the actual topology aware network would make the paper more solid.

-The method is obviously very expensive. Is there any potential idea or open discussion to reduce the computational expenses? For example, not completely evaluate cycle consistency all over the frames, making some steps sequentially updatable. etc.

-In Figure 3. the results from DynamicFusion do not look right. What is the main source of the stripe artifacts in the most of results?
limitations: As addressed in the conclusion, the major limitation of the method is probably the computational expense.
At least adding a small section about the discussion of how to make the method scalable (even with the trade off between quality) or how to make it easier to use the method in the general use cases (end to end scenario) would make the paper more solid.
Finally, as the major contribution also lies in the use of bijective map and topology aware network, providing more discussion and detailed evaluation (in addition to ablation test) would make the paper even more solid.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposed a method for surface reconstruction from a sequential RGBD input. The main strategy for this method is the cycle consistency between canonical and observation space. This approach seems to represent non-rigid deformation. Moreover, to be topology-aware, this paper employed [45]. The results show some comparisons with DynamicFusion[42], OcclusionFusion[58], and so on. Both qualitative and quantitative results show that the method proposed in this paper is better than the methods for the comparison in some parts. The ablation study is also conducted but it is qualitative results only. 
strengths_and_weaknesses: Strength
- The idea of a cycle consistently between observation and canonical space is reasonable.
- Topology-awareness improves the results but it is basically the previously proposed method. 
- The organization and writing are good.

Weakness
- As written in the questions and limitations of this review, I cannot find how to define the canonical space.  Also, I have some concerns in the limitation of this method. 
questions: As for the canonical space, how is it decided? Moreover, does the choice, deciding, or learning of canonical space affect the performance?
In a case of a long sequence with big motion like a 360 rotation, does the cycle consistency correctly work?
limitations: If there are some motions that this method cannot apply, I think it should be mentioned as a limitation of this method.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper tackles the problem of dynamic object reconstruction from a single RGBD video. It solves the problem in a differentiable rendering pipeline and designs a novel 3D warping function that is guaranteed to be invertible. It achieves high-quality reconstruction results on KillingFusion, DeepDeform and iPhone videos. 
strengths_and_weaknesses: **Strengths**
- The method is conceptually simple and effective. The authors did well in integrating the best existing solutions (neural surface, canonical hyper-space, and invertible NNs) into a system that not only works well but also stays clean. 
- The bijective warping field is an interesting technical improvement over SE(3) fields. it ensures 3D warping functions to be invertible by design. 
- The results are high-quality and state-of-the-art.
- The paper is well-written with adequate amount of details. Design choices are also well-motivated. 

**Weakness**
- Some details can be clarified. As there is genuine ambiguity between camera motion and object motion, it is worth explaining the camera pose initialization in more detail and analyze the failure modes. When does Robust ICP fail? For instance, does it fail when the object exhibits rotational motion? How robust is the method to inaccurate camera pose initialization? 
- Some writing could be improved (also see questions). The technical part of sec. 3.1 is not easy to follow possible due to lack of concise equations in l148-165. It is also not obvious what design choice made it invertible. The key idea of coordinate splitting is only mentioned in l150 and the high-level intuition is not conveyed.

questions: - For the implementation of invertible wrapping fields in 155-165, if points in the canonical space happen to have the same $w'$ coordinate, their predicted $R_{uv}$ is constrained to be the same. Similarly if points in the observation space have the same $(u, v)$ coordinates, their predicted $\delta_w$ is constrained to be the same. Does this cause undesirable artifacts, for instance, when dealing with an object containing a flat surface with same $w'$ coordinates?
- Eq(13) is not clearly explained. What is v_p? What does it mean to force points on depth images to be visible from camera? 
- Fig. 6: the difference between 6D motion fields and full is not obvious. Consider adding more descriptive captions to highlight the difference or choose the example more carefully.
limitations: Yes
review_title: null

Meta-review: metareview: This paper had consistently positive reviews from all reviewers and weaknesses that were expressed were responded to coherently by the authors.  I recommend this paper be accepted. 
confidence: Certain
Decision: Accept