Practical Epistemic Uncertainty Quantification for View Synthesis

Authors: Chuanhao Sun, Thanos Triantafyllou, Anthos Makris, Maja Drmač, Kai Xu, Luo Mai, Mahesh K. Marina
Primary area: probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords: Deep Learning, Epistemic Uncertainty Estimation, View Synthesis

Abstract:
View synthesis using Neural Radiance Fields (NeRF) and Gaussian Splatting (GS) has demonstrated impressive fidelity in rendering real-world scenarios. However, practical methods for accurate and efficient epistemic Uncertainty Quantification (UQ) in view synthesis are lacking. Existing approaches for NeRF either introduce significant computational overhead (e.g., "10x increase in training time" or "10x repeated training") or are limited to specific uncertainty conditions or models. Notably, GS models lack any systematic approach for comprehensive epistemic UQ. This capability is crucial for improving the robustness and scalability of neural view synthesis, enabling active model updates, error estimation, and scalable ensemble modeling based on uncertainty. In this paper, we revisit NeRF and GS-based methods from a function approximation perspective, identifying key differences and connections in 3D representation learning. Building on these insights, we introduce PH-Dropout, the first real-time and accurate method for epistemic uncertainty estimation that operates directly on pre-trained NeRF and GS models. Extensive evaluations validate our theoretical findings and demonstrate the effectiveness of PH-Dropout.

Decision: reject
Reviewer scores: 6, 6, 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1729694472130

thread_type:
official_review

<thread_object>
creator: reviewer_xcsr

<content>
summary:
The authors propose using post-hoc dropout as a tool for epistemic uncertainty quantification in novel view synthesis frameworks like NeRF and Gaussian Splatting. Starting from a model with trained parameters, they uniformly increase the dropout rate of every parameter as long as the training loss is unharmed. Sampling from the distribution defined by this dropout rate can be used to compute spatially-varying variances in each novel view, which is used as a proxy for uncertainty, which is validated experimentally.

strengths:
I congratulate the authors for their submission. The algorithm is simple, elegant and efficient. Previous attempts at using Dropout in novel view synthesis have failed: the authors have correctly identified the promise in the recent theoretically developed dropout injection methods and ported it to the Computer Vision community and the novel view synthesis problem in particular. Regardless of how exactly it compares against prior work, it surely advances the state of the art in uncertainty quantification for Vision and will inspire future work by the community (it has inspired *me* just by reading it!).

weaknesses:
The main weakness of the current manuscript is the quality of the technical exposition. Given that what is proposed is a fairly simple extension of the Ledda et al. 2023 work, this poor quality of technical exposition is not enough to worry me about the soundness of the methods, but nonetheless the authors should strive to improve it in a revision, and this (together with the evaluation concerns below) is the reason why my score is not higher (I will gladly increase it if these questions are addressed). For example:

- The pseudocode in Page 3 has several mistakes. Line 4 should be before Line 1, and Lines 1-5 should be indented. The line starting with “Ensure” is redundant.
- The wording around \sigma_max  (L164-168) is confusing. See questions below
- The theorems and proofs in section 3.2 and 4 are incredibly vague, to the extent that I would be more comfortable if they were described as “intuitions” rather than “proofs” or “sketch of proofs”. A proof sketch is an abbreviated version of a proof for which one can be confident a motivated student reader can fill out the details. This is very much not that, as evidenced by my questions below. I would propose the authors rewrite these sections, abandon the pretence of mathematical correctness, explain their intuition and refer the reader to experiments for validation.
- After reading Lemma 4.1 and its proof 10 times, I still do not understand what it means. See questions below.

I would have also appreciated some visual examples of the uncertainty predicted in novel views by this method and its competitors that go beyond the numerical evaluations. By looking at numbers on a table, it is hard to understand what makes this method more precise than others. It claims to be “modeling all epistemic sources of uncertainty”, but this could be said of, e.g., the Goli et al. work as well, which is philosophically very similar (“how much can one modify network weights without harming the loss?”). In which image regions is this method doing a better work? Is this method better in floater removal? The text claims “Bayes Rays fails to correlate depth uncertainty with high prediction error on the LF dataset”, which seems to contradict the results in Goli et al.’s work. I would appreciate a deeper dive into this. This method is theoretically interesting and fast enough that I do not think its acceptance hinges on the evaluation being flattering, but I would expect a scientific paper to be more transparent in this regard and include a more exhaustive evaluation of where its results stand in relation to previous work, as well as its flaws and the flaws of previous works.

The manuscript would also benefit from some English proofreading (e.g., “continuosity” should be “continuity”), even if this has no bearing on my recommendation.

questions:
I would love if the authors answered these questions in their revision:

- Why does the dropout mask in this method not need scaling the amplitude of the rest of the weights? I cannot think of a theoretical reason why one would do this, am I missing something? If not, is this due to an empirically observed advantage? I would be very curious to see this explored further, since it appears to go against the common wisdom of Dropout methods.
- About sigma_max (L164-168). Is the average being done over the training set only, or also the novel test view(s)? If the first one, isn’t the whole point of the algorithm that the training rendered RGB should not change after dropout injection, in which case sigma_max should be zero (or epsilon)? If the second one, how is something “quantifying the uncertainty of a model” if it depends on the specific view?
- What happens if the network is not perfectly overfit, as happens in almost every case? If the training is ended before the loss goes to zero, the network does not have sufficient resolution power, or it does but it is stuck in a local minima? The theoretical intuition seems to very deeply rely on this fact for the existence of redundancies: how much do the experiments suffer?
- The argument in Theorem 3.1 hinges on the conjecture that redundancy in the function space (e.g., one could modify the low power/high frequency Fourier components of this network without affecting the output much) translates into redundancy in the network parameters (i.e., one could turn network weights on and off without affecting the output much). I cannot immediately say that this conjecture is true (though it may very well be). Do the authors have any broader intuition or justification for this?
- On Lemma 4.1 and its proof: What is this D_KL in the space of weights? What does “p(a)” and “p(b)” mean? Are these conditioned on anything? I similarly do not understand the proof. Is all this lemma is doing saying that “if two weight sets are close, their Bayesian likelihoods are similar”? If so, this seems to me at least contestable. An argument about continuity may be made if the weights are changed only slightly, but if the observed redundancy is 20-30%, how can one state to be in a similar space in the Bayesian weight distribution?
- See “weakness” for questions on evaluation.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730517260873

thread_type:
official_review

<thread_object>
creator: reviewer_pmy8

<content>
summary:
This paper proposes a Post Hoc Epistemic UQ scheme, referred as PH-DROPOUT, to make uncertainty estimation directly on pre-trained Multi view Scene Reconstruction models (NeRF and Gaussian Splatting in particular). The algorithm estimates the variance of a well-trained model by introducing binary dropout mask into model parameters and greedily select drop out ratio to perturb the output within tolerance, and validate over test views. Authors explore the usage of the measurement from their PH-DROPOUT algorithm by conducting thorough analysis over different scenarios (active learning, correlation analysis, uncertainty driven model ensembles).

strengths:
Overall it is a good discovery and reflection on how to effectively and efficiently justify and analyze the performance of NeRF or GS model quantitatively. Authors justify their measurements is sound by extensive experiments and they indicate that most NeRF or GS models tend to have more redundant parameters as training views increase. This is a bold yet very inspiring claim. It implicitly conveys an intuitive thought: the information attained from more input views shall reduce the dimension of the model itself.  

Mostly the paper context is well written and well organized, and the paper itself answers most of my concerns while I was reading it.

weaknesses:
The math claim in this paper can be improved. There are too many "colloquial" proof rather than a mathematical modeling of phenomena. These claims seems to be redundant with respect to the integrity of the paper. For instance, to represent the significant redundancy in model parameters, authors state the following in line 190:
 
$$ \exists 0\ll r<1\rightarrow \forall x\in \mathcal{D}_{train}, \lvert F(x;\theta)- F(x;D(\theta,r))\rvert<\epsilon. $$

Firstly, greater than $\ll$ is not a rigorous notation. Secondly, $\mathcal{D}_{train}$ is not defined. Is it a point set or a multi-view image set? Lastly, the proof of line 190, i.e. line 192-210, is all text and there is no anchor point to refer to appendix for a comprehensive mathematical proof, and I failed to find the corresponding complete proof of theorem 3.1 in appendix. One cannot call statement in line 192-210 a proof of theorem. My suggestion is to make it as a conjecture or rewrite this theorem. Same question for lemma 4.1, thm 4.2, thm 4.3 and thm 4.4.

questions:
There are some clarification questions I wish to hear from authors:

*    In line 141, the model function $F(x;\theta)$ is introduced. Is $x$ a 3D point? This confuses me when I checked line 190.
*    In Algorithm 1 (line 143-159), does $M\cdot \theta$ refer the element-wise product? Where does subscript $i,j$ come from?
*    In line 173-175, it claims dropout only happens in one of the middle layes in NeRF-based models? Any ablation study over this choice of selective layer dropout?
*    To show performance in active learning, the paper evaluates the performance starting from sparse view training (as low as 2 views). Nevertheless, PH-Dropout has an assumption that the pre-trained model needs to be well trained or over-fitted. I understand that some NeRF-based alrogithm can achieve sparse-view reconstruction. Is there any quantitative assessment on judging whether or not PH-Dropout is applicable in these sparse-view training results?
*    What do AUSE RMSE, AUSE MSE and AUSE MAE, mean in Table 1 (line 443-450). Besides, layout of Table 1 and Figure 6 is bad as title and floatings are not properly aligned.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730668865964

thread_type:
official_review

<thread_object>
creator: reviewer_ptfb

<content>
summary:
The paper proposes a simple yet effective and extremely fast ad-hoc method for epistemic uncertainty estimation that operates directly on pre-trained NeRF and GS models in the task of novel view synthesis. At its core, the method proposes to use dropout at test time with the maximal drop rate that maintains an epsilon fit to the training views. The method is validated through proxy metrics like Spearman’s correlation of the uncertainty estimates with the RMSE from the GT, and the trend as a function of the number of training views. In addition, the authors demonstrate improved ensembling based on optimal ensemble member selection using their uncertainty estimates.

strengths:
* The authors considered multiple quantitative proxy metrics to benchmark their method, showcasing extensive evaluations.  
* The method is very simple to implement and adopt, yet it is extremely effective and efficient in terms of runtime.  
* Overall the paper is structured well and is easy to follow.  
* Related work is acknowledged, and the contribution of this paper is put in proper context.

weaknesses:
* Perhaps the major weakness of this paper is the relatively incremental contribution compared to Ledda et al 2023, which practically proposed the same technique up to the differences mentioned by the authors in L169-175. While I sincerely appreciate the honest citing of this related work by the authors, I find the contribution of PH-DROPOUT to be relatively small.    
* The English writing of the paper can be significantly improved in certain spots (see small list below), although for the most part it is not hard to understand the authors intention based on the context.  
* The proofs in the paper need to be more rigorous. For example, L192-207 are very hard to follow. Similarly, L235-243 seem to have mathematical inaccuracies such as calculating the KL divergence between two parameter instances instead of between two distributions. This overall proof sketch is not very clear and requires significant editing. The same applies to Theorem 4.2 and its proof sketch. More rigorous mathematical definitions and notations are needed. In the full proof from the appendix L777-782 seem to have a mistake in the variance formula. Where did $N\_F$ come from?

My rating is mainly due to the relatively limited contribution compared to Ledda et al. and the lack of mathematical rigor in the proofs. Nonetheless, I’m willing to increase my score if the authors' rebuttal can alleviate my concerns with respect to these two issues, as I still think this work does have the potential for practical value in quantifying the uncertainty of novel view synthesis. 

A few caught (minor) typos/english corrections:

* L103 \- hence hard \-\> hence **it is** hard  
* L104 \- find \-\> found  
* L128 \- in **ad** network \-\> in **a** network  
* L129 \- inject \-\> inject**ing**  
* L130 \- **non-**trivial \-\> **not** trivial  
* L149 \- step 3 in the algorithm is better split into 2 lines/formatted differently  
* L162 \- need \-\> need**ed**  
* L184 \- a common features \-\> common features/a common feature  
* L185 \- method \-\> method**s**  
* L198 \- signal \-\> **the** signal  
* L201 \- what does a “function with nearly discrete pattern” mean?  
* L208 of \-\> of **a**  
* L225 \- spa**c**ial \-\> spa**t**ial  
* L229 \- same structur**al** \-\> same **structure**  
* L229 \- number of parameter \-\> number of parameter**s**  
* L229 \- have similar distribution of parameter \-\> have **a** similar distribution of parameter**s**  
* L235 \- continuou**sity** \-\> continuity  
* ….

questions:
* What is the number of dropout samples N in your experiments?  
* Did you check the calibration of your uncertainty estimates (e.g. using metrics like ECE)?   
* Did you check the trend of the RMSE from the GT compared to thresholding out pixels with increasing uncertainty levels? Is this expected to behave similarly to a NeRF model predicting both the mean and the standard deviation of the RGB value along each ray?  
* In L262-264 you mention that your uncertainty estimate is biased. Is this bias not important? How does this affect your uncertainty calibration?  
* When you write down $\\rho\_p$ in the results tables you are referring to $\\rho\_{PE}$?  
* What does the acronym “AUSE” refer to in Table 1?
</content>
</thread_object>

</thread>
<thread>
creation date:
1731120613388

thread_type:
official_review

<thread_object>
creator: reviewer_6lns

<content>
summary:
The paper introduces PH-DROPOUT, an efficient method for epistemic uncertainty quantification in view synthesis, applicable to pre-trained NeRF and GS models. It leverages model redundancy to estimate uncertainty without retraining, showing strong performance across datasets. Despite its efficiency, it faces challenges with hash encoding and sparse scenarios.

strengths:
* The introduction of PH-DROPOUT as a post hoc epistemic uncertainty quantification (UQ) method is novel and addresses a critical gap in view synthesis research.
* The authors present a strong theoretical foundation, including proofs (e.g., Theorem 3.1) that justify the redundancy in NeRF and GS models, enabling effective dropout-based UQ.
* PH-DROPOUT achieves real-time performance, outperforming prior methods in computational efficiency.
* The method shows strong correlations between UQ metrics and prediction errors (e.g., RMSE), validating its reliability.

weaknesses:
* The method struggles with hash encoding-based NeRF models, which restricts its applicability to sparse or complex scenes where hash collisions are common.
* The results on 2DGS show limitations in few-view scenarios. 
* The method is specifically designed for view synthesis tasks and may not generalize to other domains.

questions:
* Can the authors provide a workaround or mitigation strategy for handling hash collisions in NeRF models? This would address a key limitation of the method.
* The inclusion of more real-world datasets, particularly in unbounded scenarios, could strengthen the empirical evaluation.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731138319774

thread_type:
official_review

<thread_object>
creator: reviewer_k2to

<content>
summary:
This paper proposes a fast post-hoc uncertainty quantification method for both NeRF and Gaussian Splatting Models. The method is based on test-time drop out of neurons/Gaussians in the model and estimating per-pixel uncertainty based on the error caused in the test views. The proposed method is orders of magnitude faster than other baselines and has competitive performance to them.

strengths:
- Explores how drop-out as a main UQ approach in ML, fits into the radiance field framework.
- It is applicable to both NeRF-based and 3DGS-based models.
- The proposed method is very fast, and is done post-hoc which makes it a useful method for downstream applications.

weaknesses:
Main:
- The rendering function in test time (the pre-trained radiance field) is not perfect. I do not see how comparing the rederings from the drop-out version of the model to the renderings of the model can specify uncertainty. Moreover, how are the test-views selected? Is there an assumption on the distribution of the cameras? If you select your camera far enough from the training ddistribution, the dropout rate would drop to zero , as the error from that view would be high anyway?
- The  volume rendering function itself (through its integration) hides the ambiguity and uncertainty in depth, so using the rendered image error as a source for identifying uncertainty can be less robust than a spatial method, unless queried densly from many test views.
- The authors claim this method is not architecture-dependant, a discussion about how drop-out in MLP-based NeRFs vs voxel-based or K-Plane ones results in the same metric would be useful. My confusion comes from the fact that some of these representations have more geometric meaning (like voxel or K-Planes or 3DGS) and in these cases dropping out a cell would directly affect a point in space while dropping out a node/layer from MLP might affect different places simultaneously.
- Qualitative results on uncertainty primarily reflect color uncertainty, as seen in flat regions like the train body in figure 4, where low-opacity, cloudy Gaussians exhibit low uncertainty. However, the paper should clarify whether the reported uncertainty correlates more with color error, depth error, or both, and include qualitative evidence to support this claim. This helps readers understand what use cases this uncertainty is useful for.
Minor:
- What are the PSNRs reported in Table 1? Are they PSNR after noise removal the same way explored in the baseline, if so the coverage % should also be reported alongside this metric.
- A few results on NGP-style NeRFs which are the main NeRF models used, and most results on MLP NeRFs.
- The proof sketches in the main paper can be more detailed.

questions:
See above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732893893544

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of Rebuttal

comment:
Dear Reviewers,

We sincerely thank all reviewers for their valuable feedback. We summarize modifications and the new experiments made in our revised paper.

Apologies for the delayed response to the reviews. We aimed to provide detailed clarifications and updated results wherever possible. We hope the extension allows you to share further comments or questions before the response deadline.

**Key Clarifications and New Results**

We acknowledge the reviewers' concerns regarding the structure of the reasoning in Sections 3 and 4\. We will refine the improper use of proofs, replacing them with intuition-driven analyses for improved clarity. Detailed responses to specific comments are provided below.

We will incorporate new results, including visualizations, to better address unbounded and real-world scenarios. These visual demonstrations will complement the quantitative metrics, enhancing overall understanding. Additionally, we will expand our discussion to include new baselines, such as k-plane.

**Summarized Response to Individual Reviewers**

To Reviewer *k2to*

1. PH-Dropout is not limited to specific test views; it works on arbitrary ones. Following prior works, we use randomly selected test views to illustrate the overall trend.  
2. Depth or spatial uncertainty is one of the contributors to epistemic uncertainty. This paper focuses on quantifying overall epistemic uncertainty that includes depth uncertainty. Empirically, our method matches the performance of state-of-the-art depth uncertainty estimation approaches like Bayes Rays while offering additional insights into other aspects of epistemic uncertainty.  
3. We would like to clarify that we did not claim our method is "not architecture-dependent," nor can we locate such a statement in our manuscript. We noted that some existing methods are limited to specific architectures like NeRF, while our method generalizes more effectively across NeRF and Gaussian splatting. Nonetheless, we will extend our discussion to include the new method mentioned by the reviewer, such as k-plane.

To Reviewer *6LNs*

1. Hash collision is orthogonal to the neural network after hash encoding, and therefore it is beyond the scope of PH-dropout.  
2. To avoid hash collisions, a straightforward approach is to use a larger hash table, though this comes at the cost of reduced efficiency in NGP-style methods.  
3. We evaluate PH-dropout on a more real world unbounded dataset, and will add them to the revised version.

To Reviewer *Ptfb*

1. PH-dropout differs from the interpretation in Ledda et al. (2023), as we do not view it as a straightforward approximation to MC-dropout. For example, it requires constraining fidelity on test views during dropout to align with the definition of epistemic uncertainty, which is ignored in Ledda et al. (2023)  
2. Our insights are embodied in the feasible dropout operations of PH-dropout, offering step-by-step guidance, including configuring the dropout rate—a critical aspect not addressed by Ledda et al.  
3. We agree that certain "proofs" in the paper should instead be framed as intuitions based on existing theorems and new observations. We have addressed each related comment and will thoroughly revise Section 4\.

To Reviewer *pmy8*

1. We have addressed all comments regarding the "theorems" in Sections 3 and 4 and will significantly revise these sections for improved clarity.  
2. The revised version will include the previously missing introductions of certain metrics and variables.

To Reviewer *xcSR*

1. PH-Dropout operates solely during the testing phase and does not require scaling, as a constant scale factor does not impact relative uncertainty. This allows it to support downstream applications effectively while slightly reducing computational costs.  
2. The evaluation follows the default setup from previous works, aiming to highlight overall trends rather than focusing solely on anchor views.  
3. The details of "how Fourier features are dropped" and "why single-layer dropout works" are explained and will be included in the revised version.  
4. The reasoning section will be thoroughly revised for improved clarity and soundness.

Thanks again for the valuable feedback.

Authors
</content>
</thread_object>

</thread>


Metareview:
metareview:
The evaluated paper deals with the uncertainty estimates added to NERFs and Gaussian splatting, based on test-time drop-out in the lines of Bayesian neural networks. Five different reviewers evaluated the contribution and appreciated the general idea, a strong theoretical foundation and the speed of the method. The main issues raised were,

- Novelty (the approach is very close to / incremental wrt an existing method),
- generality of the method, 
- the source of the estimated uncertainty,
- technical soundness and rigor of the proofs,
- Writing of the paper, lack of clarity.

During the reviewers/author discussion phase, the ratings of all reviewers converged to a stable "slightly above borderline". However, these increases anticipated the significant changes promised by the authors, not yet done in the revised version. In the reviewers/AC discussion period, the following assessment was reached: while the method is promising and was appreciated, and most of the questions have been answered by the authors, they failed to provide a revision of the paper itself. Anticipating edits promised by authors is tricky in peer review for any conference, but ICLR has a special open process, which allowed the authors to fully revise their paper. No doing so is a disappointment and provides sufficient cues (or a high risk) that these changes will not happen for the camera ready version. The discussion made it clear, that the reviewers judged the paper to not be ready for publication without these changes. This concerns not only the general presentation of the paper and lack of clarity, but in particular the theoretical aspects, including the quite vague "sketches of proofs". The authors had an opportunity, which they unfortunately missed. For these reasons, the AC judges that the paper is not yet ready for publication.