Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: The paper proposes a method to reconstruct dynamic radiance fields from   a single monocular video for novel view synthesis. Different from previous 
methods that rely on RGB images only, the paper makes use of both RGB and 
time-of-flight cameras and use the phasor information of the ray to supervise 
the geometry reconstruction. They derive the image formation model for the 
ToF imagee. The experiment results show that the proposed method generates better depth estimation and dynamic scene view synthesis results. 

main_review: 
### Strengths
1. The usage of ToF camera for dynamic scene reconstruction is interesting 
and novel. The paper shows improvement over previous method that replies on 
RGB or RGB+depth.

2. The proposed method generates more accurate depth estimations. 

3. The paper shows results on diverse synthetic and real scenes to validate 
the proposed method and design choices. 

4. The method can generate better view synthesis results than baseline methods 
on sparse inputs. 

### Weakness

1. While the paper shows that the proposed method achieves better BRISQUE scores,
for most of the video results, the results of the proposed method are 
more blurry, and the results of VideoNeRF has more fine-grained details. What is the 
reason behind this? Also the proposed method has clear artifacts around depth 
discontinuities and obvious distortions (StudyBook, Dishwasher). While the paper 
acknowledges such limitations, they weaken the contributions of the paper. 

2. BRISQUE is not a common metric used for this task. I would suggest the paper should 
have comparison to ground truth on real scenes by capturing multi-view RGB images simultaneously and use some of the views as testing and report the standard metrics such as PSNR, SSIM 
and LPIPS. Currently the paper compares to VideoNeRF on synthetic data (Table 2 in supp) 
and use the GT depth for it, and from the results we can see that VideoNeRF is better than 
TORF consistently. However, we cannot draw the conclusion whether VideoNeRF is still better 
when using the ToF depth. Therefore, I would suggest adding quantitative comparisons on 
real data with GT.  

3. The infrared camera is usually sensitive to background light and sunlight. It's not 
clear to me whether the method works well for outdoor scenes and scenes with visible 
light sources. Is the method robust the potential corruptions in the ToF information?

4. The synthetic data such as Bedroom and bathroom has a lot of noise. It would be better 
to increase the number of samples during rendering and generate noise-free images. 

Overall I like the idea of using ToF cameras to provide additional supervision signals of 
neural radiance fields, and the paper is able to show improvement over previous methods. 
However, the quality of the results is not fully convincing to me, and considering that   
the paper uses a more complex acquisition setup, I would expect better results than the 
ones shown in the paper. Therefore I vote for "marginally above the acceptance threshold".


limitations_and_societal_impact: Please see the main review.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper is trying to render novel views of static/dynamic scenes by using RGB and ToF sensors. Rather than directly using depth from ToF sensors, this paper takes a step further to work on phase images, and is more likely to handle challenges of using ToF sensors. Experiment results show that the rendered images look more realistic, with less artifacts, especially for dynamic scenes.
main_review: 1. There are quite a lot endeavors trying to extend NeRF to dynamic scenes, which is higly illposed. To merge depth information from a ToF sensor is a nice choice, and this paper seems to be the first in utilizing TOF in the NeRF framework. Furthermore, this paper further works on the phase images, thus is more likely to avoid some commen issues of ToF, in the presence of complex reflectance and depth noise. 

2. The rendering model for ToF images has been carefully adapted in the NeRF framework, and those equations look correct to me. 

3. As for the results, clear improvement can be observed, when compared with existing methods in which depth information is not used. This is reasonable, since more information is used here. However, the contribution of using phase ToF images, rather than depth directly generated from a ToF sensor, is not clear. Comparison should be added to show the difference, so as to justify the necessity of working from phase images (which is the core claim of this paper). 

4. When comparing the results from this paper and those from NeRF, we can see that, the resolution and the sharpness of the proposed method are inferior to those from NeRF. One key reason might be the resolution difference between RGB and ToF. As we know, the rgb camera has high resolution, and the ToF sensor usually has low resolution. So, for a light ray in the ToF framework, there must be multiple corresponding light rays in the RGB framework.  This paper ignores this critical fact. It is highly recommended to examine this multi-resolution fact when using RGB and ToF bi-modality input. 

5. As the authors admitted, dark regions indeed have obvious artifacts, which is likely to be enhanced by using some supervised signal enhancement techniques. This can be an interesting task in the future. 
limitations_and_societal_impact: It's a nice paper that might interest many people in CV and graphics. The multi-resolution issue should be addressed for better visual quality. Also, the rendering speed should be accelerated. Both might be the main hurdles toward real applications.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The authors introduce a novel neural representation based on an image formation model for continuous-wave ToF cameras, thereby replacing the data-driven priors used by other approaches on the reconstruction of dynamic scene parts with measurements from a time-of-flight (ToF) camera. Using the modeling of raw ToF measurements instead of depth maps allows better handling challenging scenarios such as regions with low reflectance, multi-path interference, or the limited unambiguous depth range of sensors.


The proposed approach exhibits a potential benefit regarding robust static scene reconstruction from only a few views and regarding increased robustness of dynamic scene reconstruction. Still, the evaluation is lacking in some regards as discussed below.

main_review: Originality:
The proposed approach seems novel. There seems to be not much work yet on Time-of-Flight Radiance Fields. Related work seems to be mostly covered. I would recommend adding the recent work by Shen et al., Non-line-of-sight Imaging via Neural Transient Fields’ as it also considers time-resolved data.

Quality:
- Besides a discussion of the used hardware, the metrics used for evaluation and the used data the authors provide results for static and dynamic scenes.
- For static scenes the authors demonstrate in quantitative and qualitative comparisons that fewer views can be used in comparison to NeRF. Detailed experiments on different configurations/poses of the views have not been conducted.
- For dynamic scenes the authors provide quantitative and qualitative comparisons to NSFF and VideoNeRF, where the proposed method also seems to allow more plausible results in comparison to the other techniques. However, there seems to be a clear shift in the overall color distribution (Figure 5) of the depth which might indicate an offset or scale issue. This needs proper evaluation to allow getting insights on the real potential of the approach (I would also consider raising my voting by 1 point). Highlighting regions with zoom-ins could be another option to demonstrate the benefits for fine-grained structures.

Clarity:
- The paper is well-structured and easy to follow. Figures/tables and captions are informative.
- The introduction/motivation is good.
- The supplemental provides a comprehensive overview on different aspects and additional results. 

Significance:
- The paper seems reproducible by more experienced readers. The authors mention that they aim to release materials for reproduction later on, but this seems not guaranteed.


Post-Rebuttal Comments:

The authors addressed concerns regarding ...
- the their depth map results (Figure 5) where a the color shift bug and a resolution bug have been resolved,
- additional zoom-ins to improve the exposition of the results,
- the influence of the number of view configurations.
I also appreciate the additional results they provided on other reviewer comments, e.g. regarding the effect of the accuracy of the depth quality for VideoNeRF and the effect of the number of samples used for rendering.

The authors' promise that "We will include larger baseline view synthesis results in our revised submission." (also highlighted by Reviewer EBPE) is indeed interesting, but at the moment still kept vague, i.e., they did not yet provide details regarding the behavior in this regards.

As a result of the clarification of bugs and the extension of the evaluation, I also see the submission now above the threshold for acceptance.


limitations_and_societal_impact: The authors provide a discussion of limitations and the potential societal impact.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The goal is geometry and appearance reconstruction of a dynamic scene from monocular RGB and C-ToF (raw depth) measurements. The proposed method extends dynamic NeRF models by a principled image formation model of C-ToF measurements. It is assumed that only the ToF camera light enters the ToF camera sensor, that both are at the same position, and that there is only one bounce per light path. Experiments on few-view static reconstruction and monocular dynamic reconstruction are included.
main_review: The image formation model is a valuable extension of (dynamic) neural radiance fields to raw depth measurements.

The experiments show good static reconstruction quality from two views already. The dynamic reconstruction quality is similar to Video-NeRF (unpublished at the time of submission I believe) even though Video-NeRF uses additional regularization losses.

I almost missed the index.html file in the supplemental folder and suggest to mention it in the supplemental PDF or the main paper.

Extrinsics:

The novel view synthesis results seem to be along different camera paths (and resolutions) for different methods. The NSFF camera circles seem to be a bit larger than for ToRF, e.g. Cupboard has a much more aggressive path for NSFF than for ToRF. How are the extrinsics for novel view rendering made comparable between methods? Are quantitative results reported on these differing NVS paths?

Comparisons:

- Both NSFF and Video-NeRF originally estimate depth from RGB images during preprocessing and then use that estimated depth during optimization. If computationally feasible, evaluating Video-NeRF and NSFF on both recorded and their respectively own estimated depths would be nice. Measured depth might show other artifacts than estimated depth while measured depth might be more accurate than estimated depth, so it is not clear to me which one is fairer for comparisons.
- Video-NeRF shows similar quality to ToRF. Is that still the case when using estimated (instead of recorded) depth for Video-NeRF? I.e. are the regularization losses in Video-NeRF sufficient to make up for the lack of recorded depth?
- In general, it would be nice to show NVS from viewpoints that are more different than currently. The results show NVS that is very close to the input camera path. I understand that this is challenging for this type of method currently, but maybe geometry supervision helps? The reconstructions look like they are not really correct (disappearing hands, torn hair), so I presume that ToRF does not work well for such cases.
- Similarly, larger camera motion would be nice (in which case Colmap might also get enough baseline to estimate extrinsics). As long as NVS sticks relatively close to the input camera path similar to NSFF, the quality should be decent.


Summary:

Overall, the method is a non-trivial extension to dynamic NeRF models and is valuable. For the dynamic setting, I believe that clarifications regarding the extrinsics are necessary and that additional evaluations on NSFF and Video-NeRF would strengthen the paper and provide useful insight but are not strictly necessary for acceptance.

Minor points:

- Not necessary since it is a preprint, but there is another concurrent dynamic NeRF paper: Yilun Du, Yinan Zhang, Hong-Xing Yu, Joshua B. Tenenbaum, Jiajun Wu. "Neural Radiance Flow for 4D View Synthesis and Video Processing", arXiv:2011.13084.
- line 136: "component component"
- The DinoPear and Dishwasher depth videos of ToRF are in the wrong color coding.
- Is Eq. 6 or Eq. 7 used for the results? I.e. are multiple paths supported?

After rebuttal:

My points were addressed well enough. My main concern was the comparability of the extrinsics across methods, but I understand the difficulty in achieving that and I don't see a better option than what the authors are promising for the revision.

While not mentioned by the authors in their summary of changes, they also promised that "We will include larger baseline view synthesis results in our revised submission."

There are a few remaining downsides that keep this submission from being outstanding, especially important to me is the comment by reviewer SWuj: "However, the quality of the results is not fully convincing to me, and considering that the paper uses a more complex acquisition setup, I would expect better results than the ones shown in the paper."

Overall, however, I believe that the submission does rise from slightly below to slightly above the threshold for acceptance given its novelty and timeliness.
limitations_and_societal_impact: Limitations are addressed and potential negative societal impact is well described.
review_title: null

Meta-review: The paper proposed an approach to model geometry and appearance in a dynamic scene based on monocular RGB and ToF measurements. The key novelty is a principled extension of NeRF based on the image formation model of ToF measurements. The paper initially received a mixed rating, with two reviewers rated it below the bar and two above. Two reviewers were satisfied with the rebuttal and upgraded the ratings. While the meta-reviewer agrees that the quality of the results was not impressive, the ToF extension of NeRF for dynamic scenes is a solid technical contribution. The authors are encouraged to incorporate the review feedback in the final manuscript.
Decision: Accept (Poster)