Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper studies the effective metrics for evaluating recent works on novel view synthesis from monocular videos. The proposed effective multi-view factors (EMF) measures multi-view signals in the evaluation of view synthesis. It also comes with a new dataset which tries to mitigate the multi-view signals in captures. Two new metrics are proposed to measure the quality of view synthesis and motion. Several state-of-the-art methods are further improved, but still struggle in the cases of motion and few multi-view signal.
strengths_and_weaknesses: # Strengths
- EMF

The effective multi-view factors measure the amount of ''multi-viewness'' in the capture. Such ''multi-viewness'' makes the task of novel view synthesis on dynamic scenes easier as the problem degrades into a multi-view setup. EMF consists of scene motion and camera angular velocity. And higher value in either suggests there is a strong multi-viewness in the capture. EMF basically tells how easy/difficult the capture is to reconstruct. Such metrics is nice to have in the future dataset release.

- Masked PSNR and Correspondence

Two new metrics are proposed to further measure the quality of the synthesized image. Both are tailored for dynamic scenes, but I see them also useful in static scenes as well. Masked PSNR only calculates PSNR in the valid region, and PCK-T tells whether the predicted motion is correct. The design choices are smart.

# Weaknesses

- Correspondence

I see masked PSNR can be applied to any method as long as the ground truth pose and depth are available. However, the correspondence metrics lacks generalizability. As shown in the paper, only methods that explicitly models motion can calculate the PCK-T score. It does not apply to methods like T-NeRF.

- Camera angular velocity

Nerfies and HyperNeRF have a high camera angular velocity in Tbl. 1. As far as I see, this is because it is switching frames between two cameras. If the captures are reordered in a way that uses frames from camera 1 first and camera 2 second (or only frames from camera 1), will Nerfies still work? If so, it will have a much smaller velocity, thus less EMF. It would be interesting to see how the method performs vs. EMF.

- No view direction or appearance encoding

It is mentioned in L297 both are turned off during training. But the goal of PSNR_M is to evaluate the quality of NVS in seen region. In practice, view direction and appearance encoding are essential in synthesizing new views. Turning them off will hurt the PSNR_M score a lot. Is there a specific reason doing so aside from overfitting?

- Typo
L32 agnitude -> magnitude
questions: See above
limitations: Limitations are not discussed. The author can discuss the use case of EMF and two new metrics more.
review_title: null

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: This paper studied the problem of dynamic 3D scene synthesis from monocular video sequence and found flaws of over-representation of slow-moving objects with a fast-moving camera in existing datasets. The authors then proposed a new metric called effective multi-view actors (EMF) to quantify the amount of multi-view signal in the image sequence. The authors also introduced a new dataset with very low EMF and argued that the new dataset should be more suitable for evaluation of dynamic 3D scene synthesis methods. Finally the authors evaluated four representative algorithms on the new dataset and find performance gap not being noticed with previous existing datasets.
strengths_and_weaknesses: Strengths:
- The authors delve into the characteristic of existing dataset and managed to produce innovative metric to evaluate the difficulty of dataset in terms of monocular dynamics.

Weaknesses
- While the ability of build 3d representation from monocular dynamics is desirable, real word video sequences could also contain a well proportion of high EMF data. So the argument that low EMF datasets is better for evaluation may not hold unconditionally. The difficulty of dataset may also come from other aspects, such as shape complexity and surface property of objects.
questions: When evaluate on different sequences with different EMF value, do we always find lower loss for high EMF sequences on existing methods?
limitations: The authors answered YES for the question “Did you describe the limitations of your work” but not stating which section contains it.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper does a complete review for existing approaches recovering dynamic 3D scenes from monocular videos, especially Nerfies, HyperNeRF and NSFF. It studies the camera trajectory, proposes a metric Effective Multiview Factors (EMF) to quantify multi-view cues in a dynamic scene with moving cameras. Existing datasets typically have high multi-view cues. Therefore, a new dataset captured by iPhone is introduced with little multi-view cue. Additional metrics such as Masked PSNR and PCK-T are also introduced to ensure the fairness of evaluation. The new benchmark brings additional challenges for existing approaches of dynamic 3D capture.
strengths_and_weaknesses: Strengths:

- Quantifying multi-view cues using EMF is neat. Different datasets in dynamic 3D capture proposes different data including different camera trajectories. And EMF quatifies the difficulties of all the data.
- I also like PCK-T correspondence besides normal PSNR. Right now a lot of novel view synthesis approaches focus on PSNR but PSNR does not directly reflect how well the model understands the 3D world. PCK-T is more explicit and I think reporting both masked PSNR and PCK-T is a good idea.
- Supp webpage is fantastic. Thank you!

Weaknesses:

I don't see any weaknesses, although I think the paper is more like a new benchmark -- It might be more suitable for the benchmark track.

Additional comments (no need to address):

> Page 8 footnote: We find that this code base performs better than the original code release

missing a period.
questions: > (L213 - 215) Each training sequence is annotated with 10 to 20 keypoints in every 10 frames

How do you select these keypoints?

limitations: Limitations are not discussed in the paper.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes a dataset of monocular videos to evaluate non-rigid novel view synthesis methods. A factor for measuring the multi-view effect is proposed, and two evaluation metrics are proposed. 
strengths_and_weaknesses: Strengths:

The paper is well-organized, and it is easy to read.

The datasets with evaluation metrics are proposed.
 
Several methods of non-rigid novel view synthesis are evaluated on the proposed datasets. 



Weakness:

1. Using a multi-camera setup is more reliable than single-camera setup for performance analysis. After all, the goal is to evaluate methods instead of training models, I don’t think that it is the drawback of existing methods. We do not always need to capture new data for non-rigid novel view synthesis evaluation, so it is difficult for me to understand the strengths of the proposed solution.

2. The idea of Masked-PSNR is not surprising. First, if our goal is to evaluate the regions that are observed, it would be straightforward to mask other regions out. Second, we may also hope that the methods can predict and synthesize the regions that are not observed in training data. In this scenario, the mask should not be used. Overall, I don’t think that the masking is novel.

3. Although Nerfies trained models on images sampled from two cameras, the method is able to train models using a single-camera setup. 

4. The factor of the multi-view effect is just a simple trick, and there exist many other variants if we want. It is correct, but I don’t think that it is sufficiently significant in this problem. 

Post rebuttal,

After discussion, I agree with Reviewer vDbH, and I would strongly suggest the authors use the comments by vDbH in the final submission to position the work.

questions: If the authors propose a new method, it is good to evaluate it in the proposed datasets and evaluation metrics. However, from my perspective, the contribution is not significant enough for presentation in NeurIPS.
limitations: yes
review_title: null

Meta-review: metareview: Pre-rebuttal, this paper had mixed reviews. Post-rebuttal, the paper had two strong supporters, A6gt and vDbH, who argued that the paper provides valuable insights into an important field, as well as a supporter dLU6, who commented in the discussion below that they are in favor of the paper (although did not update their review). The only remaining criticism comes from 2BcV. The AC does not find 2BcV's review persuasive (A6gt's comments summarize the AC's perspective well) and 2BcV did not participate in discussion. The AC is inclined to accept the paper and encourages the authors to use their extra page to integrate their responses to the reviewers.
confidence: Certain
Decision: Accept