MutualNeRF: Improve the Performance of NeRF under Limited Samples with Mutual Information Theory

Authors: Zifan Wang, Jingwei Li, Yitang Li, Yunze Liu
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: nerf, mutual information, sparse view sampling, few-shot view synthesis

Abstract:
This paper introduces MutualNeRF, a framework enhancing Neural Radiance Field (NeRF) performance under limited samples using Mutual Information Theory. While NeRF excels in 3D scene synthesis, challenges arise with limited data and existing methods that aim to introduce prior knowledge lack theoretical support in a unified framework. We introduce a simple but theoretically robust concept, Mutual Information, as a metric to uniformly measure the correlation between images, considering both macro (semantic) and micro (pixel) levels.
  For sparse view sampling, we strategically select additional viewpoints containing more non-overlapping scene information by minimizing mutual information without knowing the ground truth images beforehand. Our framework employs a greedy algorithm, offering a near-optimal solution for this task.
  For few-shot view synthesis, we maximize the mutual information between inferred images and ground truth, expecting inferred images to gain more relevant information from known images. This is achieved by incorporating efficient, plug-and-play regularization terms.
  Experiments under limited samples show consistent improvement over state-of-the-art baselines in different settings, affirming the efficacy of our framework.

Decision: reject
Reviewer scores: 3, 5, 5, 5

Reviews and discussion:
<thread>
creation date:
1730451857196

thread_type:
official_review

<thread_object>
creator: reviewer_bhx7

<content>
summary:
This paper proposes utilizing Mutual Information Theory to enhance the performance of NeRF with limited samples. Mutual information serves as a metric to measure the correlation between images at both macro and micro levels. The macro perspective focuses on correlations in semantic features, while the micro perspective addresses correlations in the pixel space. By incorporating mutual information, the proposed framework effectively addresses challenges posed by sparse view sampling and few-shot view synthesis, resulting in improved performance. Extensive experiments demonstrate the effectiveness of assessing both semantic and pixel space distances.

strengths:
1. Introducing mutual information to enhance Neural Radiance Field (NeRF) performance is an innovative idea.
2. The experimental results in Table 3 show that the proposed method improves performance across various NeRF frameworks.
3. This paper is well-organized.

weaknesses:
1. Results in Figure 4 show limited improvement between FreeNeRF and the proposed MutualNeRF; only the zoomed-in sections reveal marginal enhancements, which are still not substantial.
2. The proposed method introduces more complex training requirements and utilizes a large model, CLIP, for assessing semantic space distance, which may impact training time and peak memory consumption.
3. Performance improvements become marginal with more powerful baseline frameworks. For instance, "RegNeRF + Ours" surpasses RegNeRF by 1.28 PSNR, while "FreeNeRF + Ours" outperforms FreeNeRF by only 0.50 PSNR.

questions:
1) Figure 4 shows that some regions, such as the stairs, are not improved. Why might this be the case? Could this indicate limitations in the Mutual Information design? Specifically, which types of regions can be enhanced by the macro and micro losses? Currently, it is unclear which specific issues the proposed framework addresses. Could the authors provide a more detailed explanation?

2) What are the comparisons in terms of training time and peak memory consumption?

3) Table 3 and Table 4 show that the performance gain for "FreeNeRF + Ours" is marginal. This raises concerns about whether MutualNeRF will improve performance for more advanced backbones, such as Sparsenerf [A] or Sparsenerf combined with FreeNeRF. Could the authors comment on this?

4) Does the proposed macro loss also improve the performance of 3DGS-based methods? If so, did the authors experiment with sparse-view 3DGS splatting methods, such as [B]?

5) What would happen if we applied CLIP as a perceptual loss to support FreeNeRF training?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730683345065

thread_type:
official_review

<thread_object>
creator: reviewer_hj8j

<content>
summary:
The paper introduces MutualNeRF, a framework that enhances Neural Radiance Field (NeRF) performance under limited sample conditions using Mutual Information Theory. It addresses sparse view sampling and few-shot view synthesis by minimizing and maximizing mutual information, respectively. The framework employs a greedy algorithm for viewpoint selection and plug-and-play regularization terms for training.

strengths:
Utilizes mutual information theory to improve NeRF under limited data.

Proposes a greedy algorithm for strategic viewpoint selection in sparse view sampling.

Introduces efficient regularization terms to enhance few-shot view synthesis.

weaknesses:
1.	How about using mutual information for 3DGS?
2.	What is the rendering speed of the proposed method, especially when compared with 3DGS?
3.	I suggest that the authors compare or discuss with more methods, like PixelNeRF[A], CR-NeRF[B], and MVSGaussian[C], to fully verify the effectiveness of the proposed mutual information.
4.	Why choose to pick images instead of training as a whole? Are there images that do not belong to the target scene? If we use a fast reconstruction method like 3DGS, training all images won't take much time.

**Reference**

[A] Yu, Alex, et al. "pixelnerf: Neural radiance fields from one or few images." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2021.

[B] Yang, Yifan, et al. "Cross-ray neural radiance fields for novel-view synthesis from unconstrained image collections." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.

[C] Liu, Tianqi, et al. "MVSGaussian: Fast Generalizable Gaussian Splatting Reconstruction from Multi-View Stereo." European Conference on Computer Vision. Springer, Cham, 2025.

questions:
Please refer to the weakness part.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730712994510

thread_type:
official_review

<thread_object>
creator: reviewer_ryb4

<content>
summary:
This paper proposes a framework to improve the performance of sparse view sampling and few-shot view synthesis by mutual information. For sparse view sampling, the authors first calculate the semantic space distance and pixel space distance, then use a greedy algorithm to select sparse views for training. For few-shot view synthesis, the authors add two regularization terms based on semantic space distance and pixel space distance, to improve the performance of view synthesis. The authors conduct detailed experiments to demonstrate the effectiveness.

strengths:
The authors demonstrate the effectiveness of both sparse view sampling and few-shot view synthesis. Using mutual information to select views for NeRF is reasonable.

weaknesses:
1. The motivation for pixel space distance is not clarified. According to Definition 3, the pixel space distance is the expectation of distance between any two points of rays. The authors should clarify the motivation behind Definition 3 since it is important for the following parts. The semantic space distance is fairly reasonable.

2. If my understanding is correct, for the few-shot view synthesis, the authors just added two regularization terms to the NeRF training. However, the relationship between mutual information and few-shot view synthesis is not clear. For sparse view sampling, minimizing mutual information is reasonable.

3. For the few-shot view synthesis, the proposed method needs to evaluate the semantic distance between randomly rendered images and ground truth images. Therefore, this will bring additional training costs. The analysis should be provided.

4. Compared with NeRF, 3D Gaussian splatting is much better in both training and rendering. The authors should conduct experiments based on 3DGS.

5.  There are many typos and grammatical errors. For example, formulas should come with indices.

questions:
please refer to the weaknesses part.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730717294321

thread_type:
official_review

<thread_object>
creator: reviewer_mjvb

<content>
summary:
The paper proposes MutualNeRF, a method to improve the performance of Neural Radiance Fields (NeRF) when training samples are limited. The authors integrate mutual information theory to develop a unified approach, enhancing NeRF's effectiveness in sparse view sampling and few-shot view synthesis, by modeling uncertainty in semantic space distance and pixel space distance.

strengths:
- This paper is well written and easy to follow.
- The framework’s design is comprehensive, considering both macro (semantic) and micro (pixel) perspective in the task of sparse view sampling and few-shot NVS.

weaknesses:
- **Complex methodology with marginal gains:** This methodology introduces significant complexity, especially in sparse view sampling, involving greedy algorithms and complex mutual information metrics. However, the observed improvements over simpler baselines are relatively minor, which may not justify the added complexity.
- **Lack of novelty.** The attempt to address the task of few-shot novel view synthesis through minimization of mutual information between viewpoints have already been explored, especially in CVRP 2022 paper **InfoNeRF: Ray Entropy Minimization for Few-Shot Neural Volume Rendering**. The methodology of this paper seems to be re-interpretation of ideas used in DietNeRF (semantic space) and InfoNeRF (pixel space) within the perspective of mutual information theory. I ask the authors to give a more thorough theoretical comparison between this work and the methods that I have mentioned.

questions:
See the weaknesses above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732364626851

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
We would like to express our sincere gratitude for the reviewer's constructive suggestions and comments. Since the deadline is approaching, we sincerely hope the reviewers can read our response. Please let us know if the reviewers have any comments about our response or any other additional concerns. We are eager to provide any further clarifications and discussions to help the evaluation.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper received negative feedback from the reviewers, who expressed concerns regarding the lack of novelty and the limited comparisons. Specifically, the use of mutual information for few-shot novel view synthesis has been explored in previous methods. The AC has carefully reviewed all the feedback and the authors' rebuttal and agrees that the current version is not yet ready for acceptance. The AC strongly encourages the authors to address all the reviewers' concerns and resubmit to a future venue.