Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper proposed a novel view synthesis method for dynamic scenes using a deformable voxel formation. They show that using this representation is hard to train as there are large number of parameters. Sp device a static to dynamic paradigm to train faster resulting in 100x speed over previous dynamic Radiance fields.  
strengths_and_weaknesses: Strengths
- The paper is well written and easy to follow with key contributions well formulated and experimented.
- The dataset being used can be helpful in quickly starting the research in the dynamic reconstruction paradigm as it is low cost.
- Learning Deformable voxel radiance field helps understand dynamic scenes accurately and faster.
- Using the Static to dynamic learning paradigm is helpful in general for dynamic scene understanding and the coarse to fine optimization is useful in dynamic scenes.
- The ablation study shows the advantage of using different components in the optimization like optical flow to the overall accuracy of dynamic reconstruction

Weaknesses:
- The dataset captured in the real world is too small for accurate evaluation. Since the method is dynamic different motions of the same objects can be added to the dataset to make more evaluations and would make the method stronger.
- Fig 3 shows comparison to different methods but the synthetic examples seems to be very similar to other baselines. It is hard to understand the advantage of the current framework over such examples. Seems like the method performs well on the real videos. May be showing more such examples will be helpful.
- The method is proposed to be 100x faster, this contribution is mainly because of sparser input data with good flow. Doing more evaluations on what number of inputs is actually sufficient and are needed is will be an interesting insight.
questions: why was such a small dataset being proposed? It is easy to capture a larger data from the current configuration.


limitations: Limitations have been well addressed. It would be great if the authors provided negative societal impact for dynamic radiance fields.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: Modeling geometry and motions for a dynamic scene is an important task in computer vision. This paper proposes a volumetric representation to speed up learning dynamic radiance fields. It employs a 4D voxel grid to model deformations while continuing to use a 3D voxel grid to model the canonical space. To save optimization time and provide well initialization, this paper devises a 2-stage capturing procedure – the 1st stage is to capture a static state with a moving monocular camera, and the 2nd stage is to capture dynamic states with a few fixed cameras. Accordingly, the captured images in the 1st stage are utilized for training the canonical space, while the captured images in the 2nd stage are utilized for training the deformation field.
strengths_and_weaknesses: Strengths:
1.	The paper is well-written, including clarifying the motivation and introducing the experiment setup.
2.	The diagrammatic drawing (Fig. 1 and Fig. 2) is technical and clearly describes the claimed contributions.

Weakness
1.	The setup of capturing strategy is complicated and is not easy for applications in real life. To initialize the canonical space, the first stage is to capture the static state using a moving camera. Then to model motions, the second stage is to capture dynamic states using a few (4) fixed cameras. Such a 2-stage capturing is not straightforward.
2.	To utilize a volumetric representation in the deformation field is not a novel idea. In the real-time dynamic reconstruction task, VolumeDeform [1] has proposed volumetric grids to encode both the geometry and motion, respectively.
3.	The quantitative experiments (Tab. 2 and Tab. 3) show that the fidelity of rendered results highly depends on the 2-stage training strategy. In a general capturing case, other methods can obtain more accurate rendered images. Oppositely, Tab. 2 shows that it is not easy to fuse the designed 2-stage training strategy into current mainstream frameworks, such as D-NeRF, Nerfies and HyperNeRF. It verifies that the 2-stages training strategy is not a general design for dynamic NeRF.

[1] Innmann, Matthias, Michael Zollhöfer, Matthias Nießner, Christian Theobalt, and Marc Stamminger. "Volumedeform: Real-time volumetric non-rigid reconstruction." In European conference on computer vision (ECCV), pp. 362-379. Springer, Cham, 2016.
questions: 1, The design of the backward deformation is well-written, but the process of obtaining the forward motion is not clearly presented. Could you explain how to query the corresponding points of a canonical 3D point at different time steps?
2, Could you verify that the proposed DeVRF can still receive a well rendering performance without the 2-stages training strategy, like evaluating DeVRF with optical flow, cycle consistency, total variation, as well as a general training process? And compare it with the mainstream frameworks (D-NeRF, Nerfies and HyperNeRF).
3, If the users are able to employ a few (4) fixed cameras in capturing dynamic states, why do not use the binocular camera for recovering depth and then leverage a fusion strategy to model the deformation (a real-time procedure)?
4, Could you verify DeVRF in solving some challenging cases, like fast motions or highly variable poses?
limitations: The authors have discussed the limitations in the end.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: Authors propose an approach for novel view synthesis and reconstruction of dynamic scenes using a NeRF like model. The main idea is to split the capture/training phase between static and dynamic. Objects are first captured in a static setting using a single moving camera to build a canonical space. The dynamic performance is handled relying on multiple views rather than a single camera. A motion field is then learned to handle the dynamic of the scenes. Finally, temporal consistency enforces smooths results.
strengths_and_weaknesses: ++ Relevant problem, very active research topic in the community. Related work seems adequate.

++ Clarity. The paper is well described and reproducibility should not be a problem.

+/- Contributions: while the idea is definitely interesting, it is not clear the actual improvement over HyperNeRF (or even Nerfies). The results are marginally better and not clear why one should rely on this method instead of HyperNerf for instance. Nevertheless I find this idea of static and dynamic captures is interesting and I would suggest authors to add more results to support the claims. For instance, I would try the method on multi-view human datasets, where a single, multi-view frame could be used to learn the canonical space and then the rest of the sequence should learn a motion field. This could be particularly relevant for performance capture/avatars.

-- Experiments and Results. Authors should show more real-world examples. In particular results with less views for the dynamic part. I appreciated the quantitative results in the supplementary material, but would be good to show qualitatively the effect of using less views.
questions: I'd like authors to mostly comment on the weaknesses above. I think the paper has overall some interesting ideas that could be relevant for the community.
limitations: Some limitations are described, I would ask author to elaborate more on the actual capture requirements too. For instance, the dynamic part seems to need a multi-view setup, which limits the application in practice.
review_title: null

Meta-review: metareview: Reviewers are in agreement that the paper addresses an important task (modelling dynamic scenes) and is well presented and written.

Reviewer 7cq7 considers that the 2-stage capture is a limitation, and it is correct that the most general solution, towards which the field strives to move, is a small number of moving or static cameras capturing a complex dynamic scene.  However, as noted by other reviewers, this configuration is not impractical, and study of this case will likely contribute to the field overall.

confidence: Certain
Decision: Accept