Well-NeRF: Ensuring Well-Posed Neural Radiance Fields via View Frustum and Shadow Zone Based Regularization

Authors: Geunho Kim, Jinwook Paeng, Jin Hong, Yoojin Han, Junseok Kwon
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Few-shot NeRF, Ill-posed problem, Artifacts removal, View frustum, Inside opaque, Boundary condition, Near-far threshold, Integrated model

Abstract:
Neural Radiation Field (NeRF) often produces many artifacts with sparse inputs. These artifacts are primarily caused by learning in regions where position inference is not feasible. We assume that the main cause of this problem is the incorrect setting of boundary conditions in the learning space. To address this issue, we propose a new regularization method based on two key assumptions: (1) the position of density and color cannot be inferred in regions where the view frustum does not intersect, and (2) information inside opaque surfaces cannot be observed and inferred, and thus cannot contribute to the rendering of the image. Our method aims to transform the NeRF model into a well-posed problem by regularizing learning in regions where position inference is not possible, allowing the network to converge meaningfully. Our approach does not require scene-specific optimization and focuses on regions where position inference is not possible, thereby avoiding degradation of model performance in main regions. Experimental results demonstrate the effectiveness of our method in addressing the sparse input problem, showing outstanding performance on the Blender synthetic datasets. Our method is designed to integrate seamlessly with existing techniques, providing an effective solution for sparse input scenarios and offering a foundational approach that serves as the first clue in addressing sparse input problems.

Decision: reject
Reviewer scores: 6, 6, 5, 3, 6

Reviews and discussion:
<thread>
creation date:
1730169732776

thread_type:
official_review

<thread_object>
creator: reviewer_stgh

<content>
summary:
This paper explores two key factors that lead to artifacts when training NeRF from sparse viewpoints. The first is that the regions cannot be inferred where the view frustum does not cover, and the second is that the opaque areas inside objects cannot be well constrained by the RGB loss, resulting in a NeRF that does not fully represent scene information. To address these issues, the paper proposes two regularization terms to constrain the training, focusing the network on areas with higher reliability for better results. Experiments demonstrate that the proposed method effectively resolves the two issues mentioned above.

strengths:
The paper proposes two different strategies to address the underfitting issues caused by sparse viewpoints, which are actually quite common. These strategies involve designing regularization terms to constrain the training of NeRF, and have been proven to be very effective in experiments. On the other hand, the proposed approach is very intuitive and relatively concise, and it can achieve better results without altering other network settings. Additionally, from what I've observed, the current design is essentially a plug-and-play module that can be used in any existing NeRF. If possible, the author could also emphasize this point and validate it with some experiments.

weaknesses:
For writing, firstly, regarding the paper's title, since the core problem addressed is how to regularize NeRF training under a "sparse perspective" to achieve better results, it's recommended that the title reflects the concept of "sparsity". Next, concerning the literature review section of the paper, I believe there should be relevant studies from the latest year (2024), and the author needs to thoroughly research the most recent advancements in this field. Additionally, there are issues with some of the formulas where the symbols are not clearly described, and there is inconsistency in their use. For example, the symbol (\sigma) in equation (3) should be consistent with equation (1). Also, it's unclear how (S_norm) in equation (6) is calculated—is it (S / num_views)?

For technical part, some design choices have not provided reasonable explanations for certain aspects.

In equation (3), the Frustum Score is a constant value for each sample point when the camera parameters are fixed. Therefore, the obtained \sigma_masked could be directly used in the integration calculation for RGB. Why then is there a need to further constrain its sparsity? Similarly, in equation (6), clipping is performed in the calculation of gradients. If, as previously mentioned, \sigma_masked is directly used in the integration, constraining both RGB and the gradients, wouldn't a similar effect be achieved with reduced computational effort? I hope the author can explain the design principles. 

Additionally, during the RGB blending process, introducing RGB values near the surface into internal sample points through blending might cause color bleeding in other views. Is it also possible that the color of sample points after weighting could bring noise from unconstrained foreground areas into the interior?

Furthermore, in terms of experimental design, the paper only compares with FreeNeRF, but the settings of FreeNeRF are completely different from this study (network structure, use of hash acceleration, etc.), making the comparison potentially unfair. Lastly, I hope the author could add a few examples of novel view synthesis, because having constrained NeRF, theoretically, the results of NVS should improve. Otherwise, it might be possible that the training data was overfitted through regularization strategies.

questions:
As discussed above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730626005458

thread_type:
official_review

<thread_object>
creator: reviewer_xenz

<content>
summary:
The work addresses NeRF reconstruction using sparse inputs. The authors assume that the primary cause of reconstruction artifacts is the incorrect setting of boundary conditions in the learning space. To tackle this issue, they propose a new regularization method based on two assumptions: (1) the position of density and color cannot be inferred in regions where the view frustum does not intersect, and (2) information inside opaque surfaces cannot be observed or inferred, and therefore cannot contribute to the rendering of the image. The proposed method can be seamlessly integrated with existing techniques.

strengths:
1. The proposed regularization methods, along with the propositions to develop them, are logical and well-founded.
2. The proposed method automatically adjusts the training space without requiring additional parameter tuning, making it easy to combine with other approaches.

weaknesses:
1. Regarding Shadow Zone Regularization, the authors claim that Equation 7 blends the opaque surface's color into the object's interior. However, there is no explicit determination of the opaque surface. How does this effect apply only to the interior of the object as claimed, without affecting other regions?
2. No video results are presented to demonstrate the reconstruction accuracy and view-consistency of the rendering. Additionally, the comparison baseline only involves nerfacto and FreeNeRF, which is insufficient.
3. Manually adjusting the bounding box is straightforward with popular NeRF frameworks and may achieve the same or even better results than the proposed Frustum Score Regularization.
4. Quotation marks are not used correctly. (Minor issue, not considered in my rating).

questions:
1. Could the authors provide additional frustum score visualizations, especially those associated with the presented qualitative results? This would aid in understanding the proposed regularization.
2. Could the authors include loss curves for the proposed regularizers?
3. Could the authors also provide visualizations of RGB values of samples along the ray to illustrate the behavior of Shadow Zone Regularization?
4. Could the proposed approach be combined with 3D Gaussian Splatting? This is significant, as 3DGS is becoming the mainstream approach for novel view synthesis, surpassing NeRF.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730677225033

thread_type:
official_review

<thread_object>
creator: reviewer_m1eu

<content>
summary:
This paper proposes, Well-NeRF, which leverages view frustum and shadow zone-based regularization to make NeRF a well-posed problem under sparse view setting. Authors show outstanding performance across various test datasets.

strengths:
+: The paper introduces the idea of addressing sparse input issues through frustum score and shadow zone regularization, which is highly simple and reasonable.
+: The experimental results are impressive, across different scenes.
+: The analysis is fundamental in 3D reconstruction community, which may inspire other 3D research task.

weaknesses:
-: The dataset are too small to well demonstrate the upper bound of the proposed method. And then the results are sensitive to the experimental setting, making the results less convincing.
-: Lack enough comparisons to other methods, like RegNeRF[1], ZeroRF[2], etc.
-: Lack experiments on various number of views, making me unclear about the sensitivity and scalability.
-: (not totally a weakness, but a suggestion) The whole analysis seems independent of how we represent the scene. So, why not enhance the paper with experiments on 3DGS? 

[1] Regnerf: Regularizing neural radiance fields for view synthesis from sparse inputs
[2] ZeroRF: Fast Sparse View 360◦ Reconstruction with Zero Pretraining

questions:
Please refer to the weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730679150030

thread_type:
official_review

<thread_object>
creator: reviewer_std1

<content>
summary:
This paper proposes a simple method to improve nerf results under sparse input views (as few as 6). This method captures two intuitions: (1) points that only appear in just one input view are not learnable because the mass to learn can lie anywhere along the rays (paired with a specific scale) to show up correctly in the camera, and (2) points inside the object are not learnable since they don’t contribute to the final observed colors. The authors argue that explicitly making the network not learn those points improves the sparse-view performance.

For (1), the authors compute a mask indicating the times each point is included in all input views’ frustums, and use it to mask the loss and scale the gradients. For (2), the authors encourage nearby points to have similar colors, by gradually blending each point’s RGB with the previous point along the ray and also computing loss between the current RGB and the blended RGB.

The method is compared against nerfacto as a plug-and-play module, and the results show nerf results get improved in general with this trick. It also gets compared with FreeNerf, where near and far planes need selecting carefully when input views are sparse; this limitation can be eliminated with this paper, and similar results can be acheived.

The authors also test their method on real-world datasets including the well-known “nerf datasets” and a real in-the-wild dataset captured by the authors themselves (in the supplemental PDF).

strengths:
This paper is strong in how it turns simple observations/intuitions into concrete implementations that improve nerf results in general. The intuitions make sense, and the end results indeed look improved.

The paper presents the ideas clearly with helpful visuals such as Figures 1 and 3.

The experiments from baseline comparisons to ablation studies are extensive and cover questions people may have very well.

weaknesses:
I like the simplicity and modularity of this approach but the real-world, in-the-wild results shown in the supplemental material PDF are of concerning quality. Admittedly, high-quality view synthesis from just 6 input views of an in-the-wild scene is hard, but the method is shown to work well for the famous real-world “nerf datasets”. Clearly a gap here, one that needs closing before this approach is useful for any real use case.

A bigger question follows: Under sparse input views, do such per-scene learning approaches still make sense? I think when input views are sparse like this, and the quality presented is bad like this, one may be better off with learning-based approaches that learn from many scenes and generalize reasonably to the test scene at hand.

questions:
Related to my point above about learning-based approaches that learn from multiple scenes, have the authors compared this approach against those approaches. There’s PixelNerf, and many nice works that followed. My intuition is the fewer input views you have, the better-suited a learning-based approach becomes, with priors learned from many scenes.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730876035781

thread_type:
official_review

<thread_object>
creator: reviewer_e43w

<content>
summary:
This paper introduces Well-NeRF, a method addressing sparse input problems in NeRF models. The proposed approach includes Frustum Score and Shadow Zone to constrain learning to well-posed regions in order to reduce artifacts. Experimental results on synthetic and real-world datasets demonstrate the method’s improvement over traditional models.

strengths:
1. The two key assumptions are insightful and crucial. 

2. Authors did well in the organization of the paper.

3. The idea of exploring view frustum seems interesting. It would be a great point for solving problems of sparse inputs in NeRFs.

weaknesses:
1. **Incremental Contribution**. The authors did not give enough theoretical proofs and arguments for effectiveness of their method. In the Experiment part, as the experiments largely focus on synthetic data, the contribution of this work seems incremental. Please see Question 2 for more information on this weakness.

2. **The proposed method**. Although the authors give a good and novel assumption, the proposed methods seems simple and incremental without insightful design. More validation can contribute to the soundness of your method.

3. **Insufficient Experiments**. The authors did not provide sufficient experiment about comparisons with prior works to show their performances. The experiments are mainly conducted on **NeRF Synthetic Dataset**. However, the huge amount of experiments on synthetic datasets would weaken the effectiveness of the proposed methods for real world applications. I would encourage authors to conduct more experiment to improve the soundness of the paper. 

4. **Writing**. Writing could be improved for clarity and soundness. Some typos and mistakes could be corrected i.e.  L45, 47, 50  incorrect quotation marks. L406 the sentence is not clear.

5. **Figures**. More detailed figures can improve the clarity of the paper and make your paper more understandable. Figures in the submission seem too simple for reader to fully understand your methods and arguments.

questions:
1. Can you give the **Frustum Score** of the input views in your training settings? Also can you give a statement with **Frustum Score** to explain what kind of inputs improves the most with your method?
2. Can you give more experiments on near/far plane setting? Is the proposed method still work well or does it still improve over baselines with near/far settings other than [0.05,1000] (such as [2,6] in Figure. 5, 6)?
3. Can you provide ablation studies on lambda of Equation. 9? You can set lambda as a pre-set hyperparameter which do not change in the training. It seems interesting to detach part of the loss as another parameter. How much does this design contribute to the convergence speed?
4. Can you give the ablation studies on the number of sampling points? In Equation. 7, it seems that the parameters depend on the number of sampling points. Does the number of sampling points influence the training results?
5. Can you give more results on large-scale datasets? The datasets with sparse inputs can further demonstrate the efficacy of your method.
6. Can you provide the experimental results on comparisons with similar works with Gaussian splatting? If the results still improve greatly compared with them, it would greatly improve the performances of the work.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732798910242

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Additional Materials for Rebuttal Uploaded

comment:
Dear Reviewers,

Once again, we would like to sincerely thank you for investing your time in reviewing our paper. Your in-depth feedback has been incredibly valuable in providing insights and improving our work.

As promised, we have compiled and uploaded the requested additional experiments and supplementary materials. The files added during the rebuttal process can be found in the Supplementary Material.zip file, which includes **ICLR_2025_Well_NeRF_Supplementary Materials for Rebuttal.pdf** and the **Videos folder.**

The contents of **Supplementary Materials for Rebuttal** can be summarized as follows:

**1. Near-Far Parameter Robustness**

This experiment demonstrates the robustness of our model to near-far parameters. While the near-far parameter may seem simple, it is, in fact, a highly sensitive parameter that significantly impacts the management of the training region in existing models.

**2. Effectiveness of Dynamic Lambda**

The **Dynamic Lambda** we designed through experiments duplicates the RGB loss value and uses it to regulate the normalization lambda. This approach prevents the risk of over-regularization and enhances training stability. Additionally, during the mid-to-late stages of training, it minimizes the impact of regularization, allowing the base model to dominate. This ensures that the potential of the base model is preserved and highlighted. We would like to emphasize this as another key contribution of our work.

**3. Visualization of Frustum Score**

To observe how the view frustum score influences the training region, we visualized it alongside NeRF rendering during the training process.

**4. New Dataset Proposal: Randomized Structures and Patterns**

We previously mentioned to some reviewers that "there is currently a lack of reasonable datasets for NeRF experiments." In response, we propose a newly developed dataset generator. This generator creates datasets that are infinitely randomized, preventing experimenters from designing biased methods due to insufficient datasets. We hope this contributes to a more accurate evaluation of NeRF model designs. The dataset generator will be made publicly available for everyone to use.

**5. Video Demonstration of NeRF Rendering and Loss Curve**

We included video materials alongside the loss curve to provide a more detailed examination of our model's performance.

If the feedback is positive, the rebuttal materials will be incorporated into the paper and the Supplementary Materials.

Thank you.
</content>
</thread_object>

</thread>


Metareview:
metareview:
In this paper, the authors have proposed Well-NeRF for reconstruction with sparse inputs. The motivation of this paper mainly comes from: 1) the points should not be able to inferred if they only appear in one view, and 2) the points inside the object are not observed so that should not contribute to the rendering process. By designing new regularization methods based on frustum score and shadow zone, Well-NeRF better restricts the boundary condition. The sparse view reconstruction is an important problem, and the proposed method should have potential to combine with more NeRF methods as a plug-and-play module. However, there are still some significant limitations of this paper. About the experiments, I suggest to apply the proposed method to more NeRFs and compare with more sota methods to better demonstrate the advantages of Well-NeRF. Reviewer xeNz raises concerns on the shadow zone regularization of the method. The writing of the paper should also be futher improved as mentioned by many reviewers. Based on the concerns above, I recommend a decision of rejection of this paper.