Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes a method to enable only sparse image set as an input to a NeRF network and enable rendering of artifact free novel views which are typically not possible using current existing methods. Their main contribution is to warp given input images to novel camera locations given the current estimate of the depth at novel view and minimize the error between the warped image and the NeRF rendered image at that novel viewpoint. The warped image is assumed to be a ground truth at the novel view-point. Since warping across translated views can lead to occlusion effects, a best estimate of occlusion mask is also calculated and used to mask out the cost error regions. The loss function is designed to be minimizing feature maps at multiple scales instead of direct pixel intensity difference because the pixel intensities can be different in reality if the scene contains non-Lambertian surfaces. The results are show to be better than standard and existing approaches.
strength_and_weaknesses: Strength:
* The paper addresses a very relevant area of NeRF application to real world where capturing a scene with many images maybe impossible due to scene motion or having a 100-camera multi-camera system to capture a scene is economically and practically prohibitive.
* The paper is well written and all aspects are well discussed.
* The results look good and show improvement over existing works.

Weakness:
* No major weakness except a few "?" in the paper which can be resolved.
clarity,_quality,_novelty_and_reproducibility: The paper is well written and addresses a relevant area of NeRF application.
summary_of_the_review: Please see Strengths above.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper introduces a geometric regularization technique for nerf training under a sparse view setting. The regularization is based on cross-view warping between seen/unseen views using rendered depth, where the warped image is utilized for supervision in feature space.


strength_and_weaknesses: Pros:

1. The overall idea is tidy and easy to follow

2. Experiment results seem promising

Cons:

1. Some important references which address the multi-view photo-consistency are missing, e.g., MVSDF, NeuralWarp, and GeuNeuS. Specifically, MVSDF also applies feature consistency to constrain the neural rendering optimization.

2. Following Cons 1, I would expect a comparison between the proposed method with the explicit multi-view consistency formulation in MVSDF/NeuralWarp/GeuNeuS. (maybe an ablation study plus some explanations). From my perspective, they are all some kind of similar photoconsistency but implemented in different ways.

3. There are several wrong citations in the [Datasets and metrics] section.
clarity,_quality,_novelty_and_reproducibility: See above
summary_of_the_review: See above. I would like to hear the author's feedback on the missing references and I would raise my rating if my concerns could be well-addressed.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a method that regularize nerf in a few-shot setting. The method utilize NeRF rendered depth as pseudo ground truth to warp rgb value to other near view. The error between source view and target view are taken as an additional loss to train NeRF in a few shot setting.
strength_and_weaknesses: Strengths:

1. A novel way to generate pseudo ground truth by warpping multiple input views to the novel view location. The local features are aligned using NeRF generated depth map by an occlusion-aware mask map.

2. Instead of vanilla NeRF only using pixel-level reconstruction loss, the paper proposes to add multi-level perceptual loss between wrapped pseudo ground truth image and generated novel view image. By adding such constrain at the feature level, the framework improves quality of generated novel views by a large margin in comparison to the baselines.

Weaknesses:

1. The novelty is not significant. The main contribution of this paper is the depth rewarping loss. However, using depth to for few-shot nerf is not new [1][2]. Also, warping input to unseen views is widely used in self-supervised MVS [3][4].

2. It is not convincing that depth estimation (Eq. 4) is accurate enough with few viewpoints. Also, the resolution could be limited given the fact that the depth is generated with grid sampling. 

3. It is weird to mention Monocular depth estimation in related work, which may be self-supervised photometric consistency.

4. There are a lot of missing citations. 

[1] Depth-supervised NeRF: Fewer views and faster training for free

[2] Dense depth priors for neural radiance fields from sparse input views. 

[a] M3vsnet: Unsupervised multi-metric multi-view stereo network 

[b] Learning Unsupervised Multi-View Stereopsis via Robust Photometric Consistency
clarity,_quality,_novelty_and_reproducibility: In terms of writing, the paper is organized, and the overall flow is clear. However, there are a few typos in the paper. For example, in Figure 1 illustration, it should be NeRF instead of NerF. Some citations with question marks (?) in the paper. 

Some idea needs more clarification, for example, in section 2, "... the density’s entropy in each ray and ensures consistency across rays in the neighborhood. However, the performance of these methods was still limited". It should be discussed in-depth that why their performance is limited, as these methods are direct competitors to the proposed approach. In section 4.4, quote “As explained above, we do not … generating Ml”. I think the authors try to explain that they don’t apply masks directly to the image pairs, but to the feature map level. However, the sentence needs to be reorganized. 

summary_of_the_review: Based on the weakness discussed above, I think the paper has limited novelty, and there is also some defacts in terms of technical and writing issues. Therefore, I vote for rejection at this stage.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper focus on the novel view synthesis with only sparse inputs. The authors propose a method called GeCo-NeRF that enforces geometric consistency to regularize the training of NeRF. Specifically, input images are warped to unseen viewpoints for supervision at feature level. They also filter out erroneous warped patches and engineer several training strategies to improve training. Experiments on both synthetic and real data demonstrated the effectiveness of GeCo-NeRF.
strength_and_weaknesses: Strength:
- The results are quite promising and achieve the state-of-the-art in the few-shot NeRF setting
- The authors put a lot of engineering efforts to make the warping technique work

Weaknesses:
- The method is not novel enough for me. Many methods propose to supervise at unseen viewpoints, e.g. DietNeRF.  Also, in [1], input image color was used to supervise unseen viewpoints at pixel level.
- The authors doesn't discuss the computational cost of the proposed method. In the paper, they report that 8 hours is needed to train 120k iterations with 2 RTX 3090Ti. As far as I know, training vanilla NeRF for 200k iterations only need 8 hours on a single 1080Ti.
- Unless I am missing something, ablation is not provided for the training strategy in sec 4.5, which I think is part of the core contribution of this paper
- Some references can be incorporated and discussed: concurrent work [2], [3] also uses estimated depth to warp input images to other camera poses to regularize NeRF (This point doesn't influence my rating, but including them would make this paper more complete)
- It seems that I cannot find the mentioned supplementary material

Minor: 
- some references are not compiled successfully, as can be seen in sec 5.1
- In "Plugging in NeRF'' in sec 5.3, table 3 should be table 1

[1] Ray Priors through Reprojection: Improving Neural Radiance Fields for Novel View Extrapolation

[2] Data augmentation for NeRF: a geometric consistent solution based on view morphing

[3] StructNeRF: Neural Radiance Fields for Indoor Scenes with Structural Hints
clarity,_quality,_novelty_and_reproducibility: The paper is well-written and clearly demonstrated its contribution. The results presented both qualitatively and quantitatively outperforms baselines. 

The paper should be reproduced with some efforts according to the details provided. Due to the several training strategies proposed, it might be hard to achieve the same results as in the paper. Therefore, it would be great if the code can be open-sourced after publication.
summary_of_the_review: Overall, I think this paper falls into the borderline. The techniques are not novel enough and some details are missing, but I appreciate the engineering effort and the results are quite promising. Therefore, I lean towards acceptance.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper proposes a method to enable a NeRF model to use a sparse set of images of a specific scene and render images from novel viewpoints of that scene. Their main contribution of the paper is to warp given input images to novel camera locations given estimated  depth and minimize the error between the warped image and the NeRF rendered image at that novel viewpoint. Since warping across views can lead to occlusions, a best estimate of occlusion mask is also calculated and used to mask out the loss at the estimated occluded regions. The loss function is designed to be minimizing feature maps at multiple scales instead of direct pixel intensity difference. The results show a boost in the quality of rendered views over previous models. 
The reviewers overall agree on the contribution of the paper and its improved quality of rendered images. One reviewer misunderstood the contribution thinking that the novel  viewpoints are given, but the authors clearly clarify this misconception in their rebuttal. They point out some missing or wrong citations that the rebuttal of the authors successfully addressed. The area chair has read the paper, reviews and rebuttal and believes that the contribution of the present paper may be too minor  for the ICLR audience. The proposed NeRF model does not learn representations that transfer across scenes in any way. We believe the contributions of this work more enthusiastically accepted by  the graphics community because the paper has not shown how to contribute to  visual recognition or representation learning that are  related themes of ICLR. The authors are encouraged to submit their work in upcoming relevant venues where papers that address single scene NeRF fitting are presented, such as SIGGRAPH or ICCV. 

justification_for_why_not_higher_score: The area chair has read the paper, reviews and rebuttal and believes that the contribution of the present paper may be too minor  for the ICLR audience. The proposed NeRF model does not learn representations that transfer across scenes in any way. We believe the contributions of this work more enthusiastically accepted by  the graphics community because the paper has not shown how to contribute to  visual recognition or representation learning that are  related themes of ICLR.
Decision: Reject