ProvNeRF: Modeling per Point Provenance in NeRFs as a Stochastic Field
Authors: Kiyohiro Nakayama
Mikaela Angelina Uy
Yang You
Ke Li
Leonidas Guibas
Venue: NeurIPS 2024 poster (2024)
Keywords: NeRF
Reconstruction
Stochastic Process
Sparse View
Novel View Synthesis
Uncertainty Estimation
Primary area: machine_vision
TL;DR: We model provenance -- the locations where each point is likely visible -- of a NeRF using a stochastic field.

Abstract: Neural radiance fields (NeRFs) have gained popularity with multiple works showing promising results across various applications. However, to the best of our knowledge, existing works do not explicitly model the distribution of training camera poses, or consequently the triangulation quality, a key factor affecting reconstruction quality dating back to classical vision literature. We close this gap with ProvNeRF, an approach that models the provenance for each point -- i.e., the locations where it is likely visible -- of NeRFs as a stochastic field. We achieve this by extending implicit maximum likelihood estimation (IMLE) to functional space with an optimizable objective.  We show that modeling per-point provenance during the NeRF optimization enriches the model with information on triangulation leading to improvements in novel view synthesis and uncertainty estimation under the challenging sparse, unconstrained view setting against competitive baselines. The code will be available at https://github.com/georgeNakayama/ProvNeRF.

ProvNeRF: Modeling per Point Provenance in NeRFs as a Stochastic Field
Kiyohiro Nakayama'
Mikaela Angelina Uy!
Yang You'
Ke Li
Leonida Guibas'
I Stanford University Nvidia
3 Simon Fraser University
w47566770stanford edu fmikacuy, yangyou, guibas)@cs www.:/ssssaaoood..oooooo edu
keli@sfu.co ca
Abstract
Neural radiance fields (NeRFs) have gained popularity with multiple works show- ing promising results across various applications. However, to the best of our knowledge, existing works do not explicitly model the distribution of training camera poses, or consequently the triangulation quality, a key factor affecting reconstruction quality dating back to classical vision literature. We close this gap with ProvNeRF, an approach that models the provenance for each point i.e., the locations where it likely visible of NeRFs as stochastic field. We achieve this by extending implicit maximum likelihood estimation (IMLE) to functional space with an optimizable objective. We show that modeling per-point provenance during the NeRF optimization enriches the model with information on triangulation leading improvements novel view synthesis and uncertainty estimation under the challenging sparse, unconstrained view setting against competitive baselines
Introduction
allowing for learning 3D scenes given only 2D images, have grown in popularity in recent years. It has shown promise in many different applications such as novel view synthesis [4] (). depth estimation [14], robotics [22 localization [36] (38, etc. Existing literature [1 15 45 show that the quality of NeRF reconstruction is correlated with the selection of training camera poses. Similar correlations are observed the classical literature too, triangulation highly dependent
Neural radiance fields (NeRFs)
camera poses [47 44 3), which greatly influences the reconstruction quality. One common and important setting in computer vision literature [7 40 16) is the sparse view [18] setting in unconstrained [53] environments, and triangulation is more critical, affecting the reconstruction quality as limited input views make the system more sensitive to noise.
Despite the corrllation between triangulation and reconstruction quality, to the best of our knowledge, existing works do not explicitly model the former when optimizing the latter. In this work, we address this gap in the literature by modeling for each point the locations where it is likely visible. We dub this as the provenances of a point. Modeling and learning per-point provenance can help NeRF understand how the training cameras are distributed in space, which inherently links it to triangulation and reconstruction quality.
However, determining the provenances of point without the underlying geometry is not straight- forward as many factors influence the visibility each point in the reconstructed geometry. For example, the literature stereo matching 44 47 has extensively studied the influences of camera locations on reconstruction. One such well-known challenge arises when selecting the baseline of pair of cameras in stereo system. As shown in Fig. 2
points' visibility can suffer from different sets of errors when the length of the camera pair's baseline changes. For NeRFs, the dependence
Code will be available at httpp//er//.ttt....oooo///egeeee
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

ProvNeRF enables many applications:
Symthesis
Uncertainty Estimation
ProvNeRF
Triangutation Uncertainty
Figure 1: (Left) ProvNeRF models provenance field that outputs provenances for each 3D point as likely samples (arrows). For 3D points (brown triangle and blue circle), the corresponding provenances (illustrated by the arrows), are locations that likely observe them. (Right) ProvNeRR enables better novel view synthesis and estimating the uncertainty of the capturing process because models the locations of likely observations that is critical for NeRF's optimization.
becomes more complex multiple cameras' visibility needs to be estimated. To overcome this challenge, we propose to model the provenance as the samples from a probability distribution, where location y is assigned with large likelihood if and only if is likely to be visible from y.
To handle the potential complexity this distribution, we represent the provenance of x as a set of location samples, generated from learned probability distribution. This is distinct from the existing "attribute" prediction extensions of NeRFs [69 30 S since provenance a distribution for every 3D point in space. Thus, this amounts to modeling infinite collection of distributions (per-point's provenance) over all 3D points, which is mathematically, stochastic field over IR. In our work, extend implicit maximum likelihood estimation (IMLE) [33]. sample-based generative model, to model stochastic fields by adapting the objective to functional space. Furthermore, we derive an equivalent pointwise objective that can be efficiently optimized with gradient descent and use it to model the provenance field.
We dub our method ProyNeRF which models per-point provenance during the training stage of NeRF (Fig. [.. This enriches the model with information on triangulation quality when the model parameters are optimized. Once the provenance stochastic field is trained, we show that we can use it to improve novel view synthesis (Sec. (TT.. and estimate triangulation uncertainty in the capturing
process (Sec. 5.2) under the challenging sparse, unconstrained view setting.
2 Related Works
NeRFss and their Extensions. Neural radiance fields (NeRFs) 42 have revolutionized the field of 3D reconstruction (20) and novel view synthesis (50 32 with its powerful representation of a scene using weights of an MLP that is rendered by volume rendering [41 58). Follow-ups on NeRF further tackle novel view synthesis under more difficult scenarios such unconstrained photo collections [39], unbounded (5), dynamic (35 and deformable (46) scenes, and reflective objects [59] 7) Going beyond novel view synthesis, the NeRF representation has also shown great promise in different applications such as autonomous driving 56 62), robotics [] 22 and editing 67 63). Recent works have also extended NeRFs model other fields in addition color and opacity such as semantics [69] 68), normals (66]. CLIP embeddings (28), image features [30] and scene flow [4.. Most of these works learn an additional function that predicts an auxiliary deterministic output at each point that is either scalar or a vector, trained with extra supervision using volume rendering. All of the above works use a deterministic field to output the additional information. However, because each point's provenance is probabilistic distribution, we need to model a stochastic field instead of a deterministic field for provenance.
Sparse View Novel View Synthesis. NeRFs with rendering supervision alone struggle with sparse view input due to insufficient constraints volume rendering. Several approaches have been proposed to train NeRFs under the sparse- view regime with regularization losses [43 64], semantic

---

Small Baseline:
Wide Baseline:
Large
aterco sange ence
P
Invisible
Figure 2: Complex influence of camera baseline distance on the 3D reconstruction, Right: With wide baseline, the reconstruction is more robust against 2D measurement noises. However, is more likely to omit hidden surfaces because the invisible region is larger than a small baseline camera pair. Left: With a small baseline, the 3D reconstruction is less likely suffer from occlusions the invisible region between cameras is small. However, the reconstruction can be noisy due to large stereo range errors (large deviation in depth with small amount of noise in the 2D measurement).
consistency [23]. and image [60] or cost volume 12 61
10
feature constraints. Other works also 57 54). Despite addressing
constrain the optimization using priors from data 24
65
or depth 49
the setting with limited number of input views, many works are not specifically designed to tackle our desired sparse, unconstrained views setting they either focus object-level (24 65
43
23
21
limited camera baselines [12 [1], or forward-facing scenes [60] scenes. Recent works
54
have looked into improving the NeRF quality on more difficult setting of sparse, unconstrained heeward-ffccing. input views by incorporating depth priors. However, none of these works consider the locations and orientations of training cameras in their optimization process despite it being one of the major factors influencing the NeRF's optimization in a sparse setup.
Uncertainty Modeling in Neural Radiance Fields. The current literature separates NeRF's uncertainty into aleatoric - e.g., transient objects or changes in lighting and epistemic data limitation due to weak texture or limited camera views uncertainties. Some works [39 26
model aleatoric uncertainty by directly predicting the uncertainty values through a neural network. However, their approach requires training on large-scale data and is not suited for estimating the uncertainty of a specific scene. On the other hand, several works explore epistemic uncertainty estimation in NeRFs through variational inference [51 52 48), ensemble learning [55] 31], and Bayesian inference [19 25 45). While these works estimate epistemic uncertainty, they still entangle different sources of uncertainty such as texture, camera poses, and model bias, resulting in unclear and inconsistent definitions of the uncertainty quantified. In our work, we specifically model the
uncertainty caused by the capturing process that is useful in various downstream tasks 45
26
48
3 Preliminaries
3.1 Neural Radiance Fields (NeRF)
neural radiance field (NcRF) a coordinate-base neural network that learns a field in 3D space, where each point 3 is of certain opacity and color. Mathematically, is parameterized by two functions representing the two fields Fo.u = (o,((x), co (xx d)), one for opacity a R33 - R. and one for color G9 R x 2 + |0, 1J3, where d E $ is the direction from where viewed from. One of the key underpinnings of NeRFs is volume rendering allowing for end-to-end differentiable learning with only training images. Concretely, given a set M images I1, I2. ..., Ias and their corresponding camera poses Pı, Ps ...,....., PM, the rendered color of pixel x is the expected color along a camera ray Ti. (t) = o, td.,., where o, is the camera origin and di,x is the ray direction coordinate x then given by the line integral:
for pixel that can be computed from the corresponding camera pose P.. The pixel value for 2D
=f"" oo (t) co (rr,x dt,
Co.e (Fi,z) (ra,)= =
(I)
3

---

where t, ty defines the near and far plane, and
T (rri (t)) - exp [-] [
(riz(()) ds
(2)
the transmittance of the point Ts (1) along the direction di,--
3.2 Implicit Maximum Likelihood Estimation
One choice of probabilistic model is implicit maximum likelihood estimation (IMLE) 33 that represents distribution set of samples and designed handle possibly multimodal distributions. As an implicit probabilistic model, IMLE learns a parameterized transformation Ho(.) of a latent random variable, e.g. Gaussian that output samples w = Hø(z,) with w Pe(w). Here, Pe is probability measure obtained by transforming the standard Gaussian distribution measure via Ha. Given set of data samples (w.,..., N }, the IMLE objective optimizes the model parameters with
(O, I), where Ha(-) often takes the form of a neural network
i = arg min E
Σ min H2()))) w6:12
(3)
It is shown that the above objective to be equivalent to maximizing the likelihood.
4 Method
In the following sections, we formally define the provenance all points as stochastic field (Sec. J. extend IMLE to model the provenance field (Sec. 4.2) and derive an equivalent pointwise loss for gradient descent (Sec. 4.3)
Notations. We denote a stochastic field with calligraphic font (Dş) and samples from the stochastic field using the same letter but bolded (Ds). Concretely D9 function sample, function defined over all points x E RR3 that is sampled from the stochastic field Dg, i.c. Dg ~ Dp. Da maps each point to one possible sample in its provenances. We also denote the distribution of the provenances point a as Dx(x). Moreover, a provenance sample (x from Do(x)) is equivalent to evaluating the function Dg Dø æ. Finally, we let a hat (?) denote the empirical samples/distributions..
4.1 Provenance as Stochastic Field
The provenance of a point is defined as the locations where it is likely visible from, and as a point can be visible from multiple locations, it can be represented as samples from distribution. Specifically, provenances of a point æ can be represented as samples from its provenance distribution (D That means that the likelihood of sampling location y E R3 from (x(x) determines how likely æe is visible from location y. Because such distributions are defined for each 3D point x E RR3 the R$.
collection of per-point provenances forms stochastic field Dd indexed by coordinates
Empirically, given sparse training camera views PI..... .......PM.... if æ is inside the camera frustum 11, for view P. and not occluded, empirical sample from the provenances of x can be parameterized as cistance-ddircttonn tuple (x) (fi..s d E R, m Considering all M training views, the empirical distribution of provenances at point x is defined by the following density function:
Pemp (1, d)
d((,d)) (Li, di..:))
M
where (fi, di,z)) V1, ||x
0ill
vell,00
Vi,
O;
(5)
x
o,
where ó is the Dirac delta function; a E [0. determines the length of d to handle occlusions. We modeled as the transmittance from the NeRF model. To recover the location that observes æ, we can write yi x t di
I denotes solid ball in 3

---

Z
Do (x)
(2(
(or)
C (udd
(t,
H θ
4
x8
12
2y(3.)
(x.)
min
Zi R
Zx(Ax
LProvNeRF
Figure 3: Training pipeline for ProvNeRF. For each point seen from provenance tuple (f, d). with direction d at distance t, we first sample K latent random functions (Zi} from distribution Z.
The learned transformation Hg transforms each 2 (x) to a provenance sample ( (x). Finally Hg is trained with ProoNeRR defined in Eq.
While the above empirical distribution of provenances is given by the training cameras, the actual distribution of provenances, i.e. for each point the locations that point likely visible from, can have more complex dependence on both the underlying geometry and the cameras. To capture this complexity, we model Da (z) as a learnable network, a probabilistic model that can model potentially complex distribution, and one choice of such model is implicit maximum likelihood estimate (IMLE) [3]. Similar to the empirical distribution, we also represent provenance samples from Do (x) distance-diiretioe tuple (f, d) defined in Eq 5 We optimized our network with the empirical distribution D training signals.
(x) defines a distribution for all point æ E R Treating R as the index set, D3 = (Dg (x) ( defines stochastic field on F$ as a collection of distributions Do (x) for all E R. Because stochastic field is composed of infinitely many random variables over R&, existing methods cannot be applied out of the box they only model finitedimmensional distributions. In the following sections, we extend IMLE 33 to model this stochastic field.
4.2 ProvNeRF
ProvNeRF models provenances of a NeRF as a stochastic field by extending IMLE [33 to functional space. IMLE learns a mapping that transforms latent distribution to the data distribution, where each data sample is either a scalar or a vector (Sec. 3.2p. However, in our context, since samples from the stochastic field 'D are functions mapping 3D locations provenances, need extend IMLE to learn a neural network mapping Ho that transforms a pre-defined latent stochastic field Z to the provenance distribution D9 (See Fig. 33.
Let Z be the stochastic field where each sample ~ a function R3 R². To transform Z to Dg, fIMLE learns deterministic mapping Ha that maps each latent function Z ~ Z to function Dg ~ Do via composition: D3 = Ho Z. Hg here is represented as a neural network to handle complex transformations from Z to Dg.
We define latent function sample Z Z be the concatenation of random linear transformation of x and æ itself. Mathematically, each latent function Z ~ 2 is a block matrix of size (b 4) x 3:
Z (x)
=
z
x, where z N~ N (0, . ERRS.
(6)
Although Z can be designed to have non-linear dependence on the input location æ, we experimentally show that this simple design choice works well across different downstream applications.
To train Ho. we maximize the likelihood of the training provenances (Eq. 4 under D, for each using the IMLE objective 33 extended to functional space. We term this extension as functional Implicit Maximum Likelihood Estimation (fIMLE). Because a direct extension to fIMLE leads to an intractable objective, we derive an efficient pointwise loss between the training provenances and model predictions equivalent to the fIMLE objective in the following section.
5

---

SCADE
Ours
Test Views
Figure Visual Efffct CPooNNVS in Eq. 10
Compared to pre-trained SCADE model, our method
can remove additional floaters in the scene (see the boxed region).
Scannet
Tanks and Temple
PSNR (t)
SSIM (t)
LPIPS (4)
PSNR (t)
SSIM (t)
LPIPS (4)
NeRF 42
19.03
0.670
0.398
17.19
0,559
0.457
DDP 49
19.29
0.695
0.368
19.18
0.651
0.361
SCADE [57
21.54
0.732
0.292
20.13
0,662
0.358
DaRF [44]
21.28
0.741
0.323
19.67
0.652
0.374
Ours
21.73
0.733
0.291
20.36
0.663
0.349
Table 1: Novel View Synthesis Results. Our method outperforms baselines in novel view synthesis on both Scannet and Tanks and Temple Datasets. This is because our novel NeRF regularizer in Eq. can remove additional floaters in the scene shown in Fig. See Sec. 5.1 for details.
4.3 Functional Implicit Maximum Likelihood Estimation
We construct IMLE objective for stochastic fields to maximize the likelihood of training prove- nances under Da. Similar to Eq. if have i.i.d. empirical samples D1,..., DM from the empirical stochastic field D (defined Sec, 4.1 and model samples D parameter stochastic field Dø, define the fIMLE objective
...... D ( from the
i = arg min E D ...... (
1 min D: -0
Unlike the original IMLE objective (Eq. that can be directly optimized, the fIMLE objective in Eq. requires the computation of I,2 integral norm functional analogy the I,2 vector norm which, in general, is not analytically in closed form. Furthermore, approximations of this integral are very expensive since each point query Dg needs forward pass through Hg.
To get around this, we use the calculus of variations to reformulate Eq, to minimize the pointwise difference between the empirical samples and model predictions This allows us to write the fIMLE objective as
LAMLE
E (DD
(K) .....D!
Σ min Ex Ex) D: (x:) D (x)
(8)
where ((() is uniform distribution the bound Ω. Eq. oonly requires computing the pointwise difference between samples from D(x) and (xx), making it efficiently optimizable with
See the supplementary for the full derivation

---

gradient descent. Ultimately, ProvNeRF jointly updates the underlying NeRF's parameters and Da by minimizing
1. a min (i.d) |(C, -0t (1y,a di
CrrovNeRY LNERF LIMLE LNERF ED!"
where (tj.a= d= = D (x), (t, d) are i..d. samples from D(x), and LNERF is the original objective of the NeRF model, e.g. photometric loss and depth loss. We provide implementation and architectural details in the supplementary material. See Figure 3 for the training pipeline illustration.
5
Experiments
Our ProvNeRF learns per-point provenance field D, by optimizing LPoONNeR on NeRF-based model. To validate ProvNeRF, we demonstrate that jointly optimizing the provenance distribution and NeRF representation can result in better scene reconstruction as shown the task of novel view synthesis (S1c. Moreover, also show that the learned provenance distribution enables other downstream tasks such as estimating the uncertainty of the capturing field (Sec. (.2). We provide an ablation study on fIMLE against other probabilistic methods in Sec. 5.3 Lastly, Sec. 5.4 we show a preliminary extension of ProvNeRF to 3DGS 22
Stochastic Provenance Field Visualization Fig. visualizes the provenance stochastic field by sampling 16 provenances on test view of the Scannet 758 scene. The directions of the samples are the negative of the predicted provenance directions for better illustration. Each sample is colored based on its predicted visibility. Notice that fIMLE allows ProvNeRF predict multimodal provenance distributions at different scene locations.
5.1 Novel View Synthesis
We show modeling per-point provenance improves sparse, unconstrained novel view synthesis. As point's 's provenances are sample locations from where the point is likely visible, the region between the provenance location samples and the query point should likely be empty. We design our provenance loss for NVS with this intuition.
Concretely, starting from given NeRF model, we first sample points æi,... NN for training camera ray parameter- ized as (t). Here we denote point 2;; = (44. We only take points ac; with transmittance greater than lected threshold x = 0.9. For each vis- ible point 2;, we sample provenances
PSNR (t)
SSIM (t)
LPIPS (1)
Determinisic Field
21.38
0.720
0.307
Frustum Check
21.56
0.728
0.297
Ours
21.73
0.733
0.291
Table 2: NVS Ablation Results on Scannet.
(t1 d ) .,.....(t.............. (tx dx) from Da(x;)
with | 0.7, Then each distance-
from which æ, is observed. This in turn means x should be equally visible when rendered from ray parameterized as (c)
direction tuple (t))) d, gives location y
= - -0²=
Te (t) y + td Vj. With this, we define our provenance loss for novel view synthesis as
N
K a + T(r T
ΣΣ
PPONNNS
(10)
=1
where [...] denotes the hinge loss and 0.05 is constant margin. LPoSNNS encourages the transmittance at æ; along training camera rays to be at least the visibility predicted by the sampled provenances from the provenance field with margin provenance directions and the training rays, LPowNvv can be used together with LProNNeR to optimize the NeRF representation and the provenance field, resulting in improved scene geometry. We apply ProvNeRF to SCADE [ for the task of novel view synthesis. See the supplement for details on the dataset, metrics, baselines, and implementation details.
By matching transmittances between the
Results. Table shows our approach outperforms the state-of-the-arr baselines in NVS on scenes from both the Scannet [3] and Tanks and Temples 29 dataset. Qualitative comparisons shown

---

View
Test
Ensemble
CF-NeRF Bayes' Ray
Ours
Ensemble
CF-NeRF
Bayes' Ray
Ours
Uncertaint
Centoi
Uncetain Carian
Unceraat
High
LDW
Figure 5: Qualitative Results for Uncertainty Modeling. We visualize uncertainty maps obtained using the method described in Sec. 5.2 The uncertainty and depth error maps are shown with color bars specified. Uncertainty values and depth errors are normalized per test image for the result to be comparable. As shown in the boxed regions, our method predicts uncertainty regions with more correlation with the predicted depth errors.
Scannet
Matterport
Avg.
#710
#758
#781
Avg.
Room 0
Room
Room
Ensemble
7.71
3.01
2.96
17.2
63.0
8.04
110
71.3
CF-NeRF 52
660
430
571
980
507
799
488
233
Bayes' Rays 19
5.47
5.11
5.23
6.07
5.49
5.67
5.77
5.91
Ours
-3.05
0.19
-1.93
-7.40
-11.0
-13.6
-10.2
-9.17
Table NLL Results for Triangulation Uncertainty.
in Fig. We see that compared the baseline SCADE, whose geometry is already relatively crisp, our ProwNys can further improve its NVS quality by removing additional cloud artifacts, as shown in the encircled regions. Note that this improvement does not require any additional priors and is only based on the provenance of the scene.
We also compare our performance with deterministic baselines: Deterministic Field regresses one provenance for each 3D location using a neural network and Frustum Check calculates the training provenance defined in Eq. by back-projg the sampled points one of the training camera and use that as the regularization information. Table shows that our provenance field outperforms these baselines on the novel view synthesis task because the deterministic field cannot model complex provenance distribution and the frustum check baseline lacks generalization ability as it cannot be optimized adapt the output provenance based the current NeRF's geometry.
5.2 Modeling Uncertainty in the Capturing Process
Provenances allow for estimating the uncertainty in triangulation, i.e., the capturing process. In classical multiview geometry [20). the angle between the rays is good rule of thumb that determines the accuracy of reconstruction. Fig. F illlusttrss this rule as the gion of uncertainty changes depending on the setup of the cameras. Formally, for 3D point æ, we sam- ple provenances {(t,, } from (x)). Treating d, as the principal axes and tú as the distances from to the camera origin, each provenance sample de- fines a pscudo camera P, that observes æ at pixel lo- cation xÿ Proj,
Segenct
Probiet
PrujOh
Figure Triangulation Uncertainty [20]. The figure shows that æ' is more uncertain com- pared to because the predicted provenances for æ' give narrower baseline than the baseline given by provenances æ.
(arj,(2=). (a:). Following chapter 12.6 of (20]. we define the trlangulation uncertainty of æ as the probability of given its noisy 2D pseudo observa- tions: P (,] TK) x P (II,..., xx ( PP (æ)
II (2,læ)

---

AP (1)
AUC (t)
Cemple
Deterministic Field
0.163
0.168
Gaussian-based w/ C
0.537
0,539
Gaussian- w/ C
0.629
0.631
VAE-based
0.323
0.325
ProvNeRF w/ Spatial Inv. Ours
0.742
0.744
0.745
0.747
Figure 7: Visualization of Provenance Field.
Table 4: Ablation Results Scannet.
The last two equalities are derived by assuming independence of the 2D observations and each P(zy) (x) follows Gaussian distribution A (Proj, This assumption is equivalent corrupting each 2D observation Proj,(z) by zero-mean Gaussian noise with o² variance, accounting for measurement noises in the capturing process. Assuming uniform prior of (a(x)) over the scene bound, the exact likelihood can be efficiently computed with importance sampling. This quantifies point's triangulation quality given the sampled provenances, which becomes measurement of the uncertainty of the capturing process. We apply our provenance field to ProvNeRF with different NeRF backbones (57 49 and compute the likelihood. See supplementary for details on the dataset, metrics, baselines, and implementations.
Results. Tab. shows the quantitative results on Scannet 13 and Matterport3D 99. We follow 51
551 to measure the negative log-likelihood (NLL) of the ground-truth surface under each model's uncertainty prediction. Since our ProvNeRF can be applied to any pre-trained NeRF module, we use pre-trained SCADE [57 for Scannet and DDP 49 for Matterport3D both of which are state- of-the-art approaches in each dataset. Our approach achieves the best NLL across all scenes in both datasets by margin because we compute a more fundamentally grounded uncertainty from classical multiview geometry (20) based on triangulation errors, while both CF-NeRF and Bayes' Rays require an approximation of the true posterior likelihood. Fig. shows qualitative comparisons between baselines' and method's uncertainty estimation. We expect a general correlation between uncertain regions with high-depth errors. An ideal uncertainty map should mark high-depth error regions with high uncertainty and vice versa. As shown in the boxed regions in the figure, method's 's uncertainty map shows better correlation with the depth error maps. We also quantitatively evaluate uncertainty maps using negative log- ligeliill following prior works Notice that in both examples, our method's certain (blue) regions mostly have low-depth errors (e.g., encircled parts in Fig- 5) because
formulation only assigns region to be certain if it is well triangulated (Fig. (6) On the other hand, baselines struggle these regions because they either use an empirical approximation from data or a Gaussian approximation of the ground truth posterior likelihood.
5.3 Ablation Study
We validate the choice of fIMLE as our probabilistic model by measuring the average precision (AP) [17] and area under the curve (AUC) [17] of predicted provenances (t, d) against ground truth provenances (f, d) for set of densely sampled points in the scene bound. See supplementary for metric and ablation implementation details.
Deterministic v.s. Stochastic Field. We validate the importance of modeling per-point provenance as stochastic field rather than a deterministic field. We model D3 with deterministic field parameterized by a neural network. Table the importance of modeling per-point provenance stochastic field. Since the provenances of a point are inherently multimodal, a deterministic field that only maps each æ to a single provenance cannot capture this multimodality.
Choice of Probabilistic Model. We validate our choice of fIMLE 33] as our probabilistic model. We first compare with explicit probabilistic models that model the provenance field as mixture of C' Gaussian processes and a VAE-based model. Table shows results for the Gaussian Mixture field with C = 2, 5 and the VAE-based process. Although the performances for the Gaussian-based models improve as we increase C., they still suffer from expressivity because of their explicit density

---

Rendering
Depth Error
FisherRf + 3DGS
ProvNeRF 3DGS
NLL(()
FisherRF
Ours
+710
5.19
-3.93
+758
2.73
Avg.
3.82
-3.00
Figure 8: Uncertainty Estimation Comparison with 3DGS. Compared with FishRF, our method is able to estimate uncertainties that correlate more with the depth error as shown by the encircled regions. The right shows a quantitative comparison of uncertainty in negative log-likelihood.
assumption. Similarly, the baseb model suffers from mode-collapse while our fIMLE enables capturing more complex distribution with learned transformation Ho-
Choice of Random Function Z. Lastly, we validate our latent stochastic field 2. We ablate our choice of Z with instead using a spatially invariant latent stochastic field 2+ with Z* (a+) = [e, æ) Vr. Here, is a Gaussian noise vector in Rd Table 4 shows the comparison between Dg obtained by transforming Z (Ours) and transforming Z" (Spatial Inv. 2). We see that using a spatially varying latent stochastic field further increases the expressivity of our model.
5.4 Preliminary Extension to 3D Gaussian Splatting
Because ProvNeRF is a post-hoc method that can model the provenance information for arbitrary novel view synthesis representations, conduct a preliminary experiment that extends our prove- nance field modeling to 3D) Gaussian Splatting (27). Specifically, given a pre-trained Gaussian representation G, model provenance distribution for each splat using IMLE with shared 6-layer MLP for Hg. The post-training takes around 30 minutes on a single A6000 Nvidia GPU, To show the usefulness of ProvNeRF applied to 3DGS, use the methodology in Sec. 5.2 to estimate uncertainty maps and compare them with the predicted depth errors. Fig. 8 shows a qualitative comparison of our uncertainty map w.r.t. FishRF 251 a recent 3DGS uncertainty estimation baseline. Compared to their uncertainty map, ours shows more correlation to the depth error as highlighted by the boxed regions. Quantitatively we evaluate NLL on the three Scannet scenes shown on the right side of the same figure and show substantial improvements over FishRF. This improvement over existing literature suggests applying ProvNeRF other representations such as 3DGS promising. We leave further exploration of the method and applications as future works.
6 Conclusion, Limitation, & Future Works
We present ProvNeRF, a model that enhances the traditional NeRF representation by modeling provenance through an extension of IMLE for stochastic processes. ProvNeRF can be easily applied to any NeRF model to enrich its representation. We showcase the advantages of modeling per per-point provenance in various downstream applications such improving novel view synthesis and modeling the uncertainty of the capturing process.
We note that our work is not without limitations. Our ProvNeRF requires post-hoc optimization, which takes around 8 hours on SCADE, limiting its current usability for real-time or on-demand applications. However, the idea presented in our work is not specific to the model design and can be adapted to other representations. See Sec. [.4 for preliminary adaption ProvNeRF to 3DGS.
We also note that the hyperparameters to incorporate ProvNeRF are chosen for better performance, e.g. for the uncertainty and novel view synthesis applications, and the future, will be beneficial to explore a more adaptive approach in integrating provenance to different downstream applications.
Acknowledgement This work is supported by Vannevar Bush Faculty Fellowship, ARL grant 19-1222-1-- an Apple Scholars in AI/MI, PhD Fellowship, and the Natural Sciences and Engineering Research Council of Canada (NSERC).
10

---

References
[1] Michal Adamkiewicz, Timothy Chen, Adam Caccavale, Rachel Gardner, Preston Culbertson, Jeannette Bohg, and Mac Schwager. Vision-only robot navigation a neural radiance world. JEEE Robotics and Automation Letters, 7(2):4606-4613, 2022.
[2] Sameer Agarwal, Noah Snavely, Ian Simon, Steven M. Seitz, and Richard Szeliski. Building rome a day. In ICCVCC
[3] Kalin Atanassov, Vikas Ramachandra, Sergio R. Goma, and Milivoje Aleksic. 3d image processing architecture for camera phones. In J. Angelo Beraldin, Geraldine S. Cheok, Michael B. McCarthy, Ulrich Neuschacfer-Rube,
McDowall, Margaret Dolinsky, and Atilla M. Baskurt, editors, Three- Dimensional Imaging, Interaction, and Measurement, volume 7864, page 786414, International Society for Optics and Photonics, SPIE, 2011.
[4] Jonathan T. Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin Brualla, and Pratul P. Srinivasan. Mip-nerf: A multiscale representation for anti anti-aliasing neural radiance fields. In ICCV, 2021.
[5] Jonathan T Barron, Ben Mildenhall, Dor Verbin, Pratul Srinivasan, and Peter Hedman. Mip-nerf 360: Unbounded anti-aliased neural radiance fields. In CVPR, 2022.
[6] Jonathan Barron, Ben Mildenhall, Dor Verbin, Pratul Srinivasan, and Peter Hedman. Zip-nerf: Anti-aliased grid-based neural radiance fields. ICCV, 2023.
[7] Sai Bi, Zexiang Xu, Pratul Srinivasan, Ben Mildenhall, Kalyan Sunkavalli, Miloš Hašan, Yannick Hold- Geoffroy, David Kriegman, and Ravi Ramamoorthi, Neural reflectance fields for appearance acquisition. arXiv preprint (rXiv:2008,03824 2020.
[8] Jiazhong Cen, Zanwei Zhou, Jiemin Fang. Chen Yang, Wei Shen, Lingxi Xie, Xiaopeng Zhang, and Qi Tian. Segment anything in 3d with nerfs. In NeurlPS, 2023.
[9] Angel Chang, Angela Dai, Thomas Funkhouser, Maciej Halber, Matthias Niessner, Manolis Savva, Shuran Song, Andy Zeng, and Yinda Zhang. Matterport3d: Learning from rgb-d data indoor environments. 3DV, 2017.
[10] Chang, Aljaž Božič, Tong Zhang, Qingsong Yan, Yingcong Chen, Sabine Süsstrunk, and Matthias NieBner. Re-mvsnet: Unsupervised multi-viww stereo with neural rendering. In ECCV, 2022.
[11] David Charatan, Sizhe Li, Andrea Tagliasacchi, and Vincent Sitzmann. pixelsplat: 3d gaussian splats from image pairs for scalable generalizable 3d reconstruction. arXiv, 2023.
[12] Anpei Chen. Zexiang Xu, Fugiang Zhao, Xiaoshuai Zhang. Fanbo Xiang, Jingyi Yu, and Hao Su. Mvsnerf: Fast generalizable radiance field reconstruction from multi-view stereo. In CVPR, 2021.
[13] Angela Dai, Angel X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. Scannet: Richly-annotated 3d reconstructions of indoor scenes. Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5828-5839 2017.
[14] Kangle Deng. Andrew Liu, Jun-Yan Yan Zhu, and Deva Ramanan. Depth-supervised nerf: Fewer views and faster training free. CVPR, 2022.
[15] Yilun Du Cameron Smith, Ayush Tewari, and Vincent Sitzmann. Learning to render novel views from wide-baseline stereo pairs. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, 2023.
[16] Jakob Engel. Vladlen Koltun, and Daniel Cremers. Direct sparse odometry. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(3):611- 1-625, 2018.
[17] Mark Everingham, Lue Van Gool, Christopher K. I. Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International Journal of Computer Vision, 88(2):303-338, 2010.
[18] Yasutaka Furukawa and Carlos Hernández. Multi- View Stereo: A Tutorial. Now Foundations and Trends, 2015.
[19] Lily Goli, Cody Reading, Silvia Sellán, Alec Jacobson, and Andrea Tagliasacchi. Bayes' rays: Uncertainty guantification in neural radiance fields. arXiv, 2023.
[20] Richard Hartley and Andrew Zisserman. Multiple View Geometry in Computer Vision. Cambridge University Press, 2 edition, 2004.

---

[21] Yi-Hua Huang, Yang-Tinn Tian Sun, Ziyi Yang, Xiaoyang Lyu, Yan-Pei Cao, and Xiaojuan Qi. Se-gs: Sparse- controlled gaussian splatting for editable dynamic scenes. arXiv preprini arXiv:2312. XXXx:2312.11937,, 2023.
[22] Jeffrey Ichnowski*, Yahav Avigal*, Justin Kerr, and Ken Goldberg. Dex-nerf: Using a neural radiance field to grasp transparent objects. In Conference on Robot Learning (CoRL), 2020.
[23] Ajay Jain, Matthew Tancik, and Pieter Abbecl. Putting nerf on a diet: Semantically consistent few-shot view synthesis. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), pages 5885-5894, October 2021.
[24] Wonbong Jang and Lourdes Agapito. Codenerf: Disentangled neural radiance fields for object categories. In ICCV, 2021.
[25] Wen Jiang, Boshu Lei, and Kostas Daniilidis. Fisherrf: Active view selection and uncertainty quantification for radiance fields using fisher information, 2023.
[26] Liren Jin, Xieyuanli Chen, Julius Rückin, and Marija Popović. uncertainty estimation in image-based neural rendering. In IROS, 2023.
Neu-nby: Next best view planning using
[27] Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Trunsactions on Gruphicx, 42(4). July 2023.
[28] Justin Kerr, Chung Min Kim, Ken Goldberg, Angjoo Kanazawa, and Matthew Tancik. Lerf: Language embedded radiance fields. In ICCV, 2023.
[29] Amo Knapitsch, Jacsik Park, Qian-Yt Yi Zhou, and Vladlen Koltun. Tanks and temples: Benchmarking large-scale scene reconstruction. ACM Transactions on Graphics, 36(4), 2017.
[30] Sosuke Kobayashi, Eiichi Matsumoto, and Vincent Sitzmann. Decomposing nerf for editing via feature field distillation. In NeurlPS, 2022.
[31] Georgios Kopanas and George Drettakis. Improving nerf quality by progressive camera placement for unrestricted navigation in complex environments. In Vision, Modeling, and Visualization, 2023.
[32] K.N. Kutulakos and S.M. Seitz. theory of shape by space carving. ICCV, 1999.
[33] Ke Li and Jitendra Malik. Implicit maximum likelihood estimation, 2018
[34] Zhengqi Li, Simon Niklaus, Noah Snavely, and Oliver Wang. Neural scene flow fields for space-time view synthesis of dynamic scenes. In CVPR, 2021.
[35] Zhengqi Li, Qianqian Wang, Forrester Cole, Richard Tucker, and Noah Snavely. Dynibar: Neural dynamic image-based rendering. In CVPR, 2023.
[36] Jianlin Liu, Qiang Nie, Yong Liu, and Chengjie Wang. Nerf-loc: Visual localization with conditional neural radiance field. In ICRA, 2023.
[37] Sheng Liu, Xiaohan Nie, and Raffay Hamid, Depth guided sparse structure- fencture-from-tee wo-o--froooottto coom-motion for movies and tv shows. In CVPR, 2022.
[38] Dominic Maggio, Marcus Abate, Jingnan Shi, Courtney Mario, and Luca Carlone. Loc-nerf: Monte carlo localization using neural radiance fields, 2022.
[39] Ricardo Martin-Brualla, Noha Radwan, Mehdi S. M. Sajjadi, Jonathan T. Barron, Alexey Dosovitskiy, and Daniel Duckworth. Nerf in the wild: Neural radiance fields for unconstrained photo collections. In CVPR, 2021.
[40] Hidenobu Matsuki, Raluca Scona, Jan Czarnowski, and Andrew J. Davison. Codemapping: Real-time dense mapping for sparse slam using compact scene representations. IEEE Robotics and Automation Letters, 2021.
[41] Nelson Max. Optical models for direct volume rendering. IEEE Transactions on Visualization and Computer Graphics, 1(2:999-108, 1995.
[42] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Rea Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In ECCV, 2020.
[43] Michael Niemeyer, Jonathan T. Barron, Ben Mildenhall, Mehdi S. M. Sajjadi, Andreas Geiger, and Noha Radwan. Regnerf: Regularizing neural radiance fields for view synthesis from sparse inputs. In CVPR, 2022.
12

---

[44] M. Okutomi and T. Kanade. A multiple-baseline stereo. IEEE Trunsactions on Puttern Analyis and Machine Intelligence, 15(4):353-363, 1993.
[45] Xuran Pan, Zihang Lai, Shiji Song, and Gao Huang. Activenerf: Learning where to see with uncertainty estimation. In ECCV, 2022.
[46] Keunhong Park, Utkarsh Sinha, Jonathan T. Barron, Sofien Bouaziz, Dan B Goldman, Steven M. Scitz, and Ricardo Martin- Brualla. Nerfies: Deformable neural radiance fields. In ICCV, 2021.
[47] Satyarth Praveen. Efficient depth estimation using sparse stereo-vision with other perception techniques. In Sudhakar Radhakrishnan and Muhammad Sarfraz, editors, Coding Theory, chapter 7. IntechOpen, Rijeka, 2019.
[48] Yunlong Ran, Jing Zeng. Shibo He, Jiming Chen, Lincheng Li, Yingke Chen, Gim Hee Lee, and Qi Ye. Neurar: Neural uncertainty for autonomous 3d reconstruction with implicit neural representations. IEEE Robotics and Automation Letters, 2022.
[49] Barbara Roessle, Jonathan T. Barron, Ben Mildenhall, Pratul P. Srinivasan, and Matthias Nießner. Dense depth priors for neural radiance fields from sparse input views. In CVPR, 2022.
[50] S.M. Seitz and C.R. Dyer. Photorealistic seene reconstruction by voxel coloring. In CVPR, 1997.
[51] J. Shen, A. Ruiz, A. Agudo, and F. Moreno- Noguer. Stochastic neural radiance fields: Quantifying uncertainty in implicit 3d representations. In 3DV, 2021.
[52] Jianxiong Shen, Antonio Agudo, Francesc Moreno-Noguer, and Adria Ruiz. Conditional nerf: Accurate 3d modelling with reliable uncertainty quantification. In ECCV, 2022.
[53] Noah Snavely, Steven M. Scitz, and Richard Szeliski. Photo tourism: Exploring photo collections in 3d. ACM Trans. Graph., 2006.
[54] Jiuhn Song, Seonghoon Park, Honggyu An, Seokju Cho, Min-Seop Kwak, Sungjin Cho, and Seungryong Kim. Därf: Boosting radiance fields from sparse inputs with monocular depth adaptation. In Neur!PS, 2023.
[55] Niko Sünderhauf, Jad Abou-Chakra, and Dimity Miller. Density- nerf ensembles: Quantifying predictive uncertainty in neural radiance fields. In ICRA, 2022.
[56] Matthew Tancik, Vincent Casser, Xinchen Yan, Sabeek Pradhan, Ben Mildenhall, Pratul P Srinivasan, Jonathan T Barron, and Henrik Kretzschmar. Block-nerf: Scalable large scene neural view synthesis. In CVPR, 2022.
[57] Mikacla Angelina Uy, Ricardo Martin- Brualla, Leonidas Guibas, and Ke L.i. Scade: Nerfs from space carving with ambiguity
depth estimates. CVPR, 2023.
[58] Mikaela Angelina Uy, George Kiyohiro Nakayama, Guandao Yang, Rahul Krishna Thomas, Leonidas Guibas, and Ke Li. Nerf revisited: Fixing quadrature instability in volume rendering. NeurlPS, 2023.
[59] Dor Verbin, Peter Hedman, Ben Mildenhall, Todd Zickler, Jonathan Barron, and Pratul P. Srinivasan. Ref-nerf: Structured view-dependent appearance for neural radiance fields. In CVPR, 2022.
[60] Qianqian Wang, Zhicheng Wang, Kyle Genova, Pratul Srinivasan, Howard Zhou, Jonathan Barron, Ricardo
Noah Snavely, and Thomas Funkhouser. Ibrnet Learning multi-view image-based rendering. In CVPR, 2021.
[61] Yi Wei, Shaohui Liu, Yongming Rao, Wang Zhao, Jiwen Lu, and Jie Zhou. Nerfingmvs: Guided optimization of neural radiance fields for indoor multi-view stereo. In ICCV, 2021.
[62] Yuanbo Xiangli, Linning Xu, Xingang Pan, Nanxuan Zhao, Anyi Rao, Christian Theobalt, Bo Dai, and Dahua Lin. Bungeenerf: Progressive neural radiance field for extreme multi-scale scene rendering. In ECCV, 2022.
[63] Tianhan Xu and Tatsuya Harada. Deforming radiance fields with cages. In ECCV, 2022.
[64] Jiawei Yang, Marco Pavone, and Yue Wang. Freenerf: Improving few-shot neural rendering with free frequency regularization. In Proc. IEEE Conf. Computer Vision and Pottern Recognition (CVPR), 2023.
[65] Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa, pixelnerf: Neural radiance fields from one few images. In CVPR, 2021.
13

---

[66] Zehao Yu, Songyou Peng. Michael Niemeyer, Torsten Sattler, and Andreas Geiger. Monosdf: Exploring monocular geometric cues for neural implicit surface reconstruction. In NeurlPS, 2022.
[67] Yu-Jie Yuan, Yang- Vang-Tian Sun, Yu-Kun Lai, Yuewen Ma, Rongfei Jia, and Lin Gao. Nerf-editing: Geometry editing of neural radiance fields. In CVPR, 2022.
[68] Xiaoshuai Zhang, Abhijit Kundu, Thomas Funkhouser, Leonidas Guibas, Hao Su, and Kyle Genova. Nerflets: Local radiance fields for efficient structure-aware 3d scene representation from 2d supervision. In CVPR, 2023.
[69] Shuaifeng Zhi, Tristan Laidlow, Stefan Leutenegger, and Andrew Davison. In-place scene labelling and understanding with implicit scene representation. In ICCV, 2021.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We discuss the limitations of our method in the supplementary
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: we provide the full derivation of our fIMLE objective in the supplementary.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide the training details, hyperparameters, network architectures the supplementary.
5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Question:
Answer: [Yes]
Justification: We will provide reproducible code upon the paper's acceptance.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
8. Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer: [Yes]
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Guidelines:
10. Broader Impacts
15

---

Question:
Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We address the potential societal impact climate change in the supplemen- tary materials
11. Safeguards
Question:
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
12. Licenses for existing assets
Question:
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
13. New Assets
Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Question:
Answer: [NA]
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
16


Reviews:
[{"review_id": "cM0kqFW4Cg", "reviewer": "Reviewer_GKL5", "summary": "This paper introduces a way to jointly learn/model provenance during NeRF training, where provenance is defined as locations where a 3D point is likely visible. This design is motivated by the classic idea of modeling triangulation quality, To implement this in NeRF, this paper extends implicit maximum likelihood estimation (IMLE) to functional space with an optimizable objective. Modeling provenance is beneficial to NeRF final results, and also enables a new way of uncertainty estimation.", "questions": "After reading, couple of questions remain:  \n- In Fig.2, how does camera baseline distance, occlusions/visibility, and stereo range errors relate to Provenance? The connection is less clear.\n- Is it possible to visualize the learned confident provenances together with the camera poses in a way that can better illustrate the correctness/soundness of the learned provenances?\n- For results in Tab.1, is `ours` built upon NeRF? If so, is the only difference between `ours` and `NeRF` is the extra loss?\n- In Eq.9 there are two expectations. So in practice, two samplings are needed. I'm wondering how are D, x, and (t,d) sampled in practice. \n- How important are L_provNVS and L_provNeRF individually? Which one is more important and why both are needed?", "limitations": "Limitations are discussed in the appendix where the authors are upfront about one important limitation of the running time. \nSocietal impact is also discussed in appendix.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The introduced idea of Provenance modeling is sound and motivated by classic idea in stereo matching. \n- The ability of estimating uncertainty is very important, and it's a free side-product of the introduced Provenance modeling.\n- The idea of using Implicit Maximum Likelihood Estimation in functional space is indeed a good solution for represents probabilistic distribution as a set of samples, and also for stochastic processes.\n- This paper also builds up a system of theory which is complete and reasonable. This theoretical framework will inspire many follow-up work in this field. \n- The appendix is very informative and discussed things like extra results, important alternative designs, key derivations and ablation studies. I found it to be a good complement of the main paper. \n- The findings of this method is helpful to sparse, unconstrained view setting is very encouraging.", "weaknesses": "I find a few things can be improved as listed below:\n- Some motivations are less clear or need clarification, specifically:\n  - L36 \"we propose to model the provenance as the samples from a probability distribution, where a location y is assigned with a large likelihood if and only if x is likely to be visible from y\": What's the alternative ways of modeling this probability distribution rather than using samples? Is it possible to learn a continuous representation rather than use discrete samples?\n  - For sample-based generative model, what are the alternatives to implicit maximum likelihood estimation (IMLE)? Why is this specific approach is taken? A discussion could be very useful here.\n\n- The visual results are hard to interpret. Denser and more informative caption can be considered.\n  - Why depth error is always shown with uncertainty. What's the connection and how can we interpret the depth? \n  - Compared to the baselines such as Bayes' Rays, what are considered as improvement from the uncertainty map? Is it more semantic? Or is it more correlated with depth, means near objects tends to have lower uncertainty. \n  - From results it appears some floaters/fuzziness are addressed but not totally. I'm wondering what's the reason and how can we address the remaining artifacts.\n  \n- Since there is no GT for uncertainty, what's the best way to evaluate it? In Tab.3 NLL is used, is this the best way? \n\n- L135, provenances is parameterized as a distance-direction tuple. What's the other options here? Can we directly model it as a 3D vector? Or using other parameterization of 3D vector?\n\n- Sec.2 first paragraph \"NeRFs and their Extensions\" missed dicussion of recent explicit representation such as 3DGS. Also, it's unclear whether the proposed solution can generalize to 3DGS? In theory it should work but we don't know in reality. This extra backbone experiment can make the results and theory more convincing. \n\n- This paper claims the proposed method helps  sparse, unconstrained view setting. Following this statement, the experiments should use a sparse setting such as few-shot NeRF. Also, I'm wondering why scannet and tanks and temple are picked as the testing benchmark but not other popular NeRF dataset (indoor, outdoor, synthetic, object-centric, etc).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for finding our method sound, theoretically complete, and has the potential to inspire follow-up works in the field. We answer the questions raised below.\n## Q1 Alternatives in modeling the probability distribution.\nInstead of as samples, we can either represent provenances as a discrete distribution in Eq.(4-5) main or as a closed-form continuous distribution (e.g. Gaussian Mixtures). However, discrete representations incur discretization errors and closed-form distributions have limited expressivity. In contrast, our provenances use an expressive deep net $H_\\theta$. See Table S1 for an NVS quality comparison.\n## Q2 Alternatives to functional IMLE.\nCompared to other sample-based implicit probabilistic models like GAN and VAE which are prone to mode collapses, IMLE can model multimodal distributions by implicitly maximizing likelihoods. Table 4 and Table S1 show quantitative comparisons of fIMLE against VAE highlighting its superior performance. We also tried GAN but it didn’t converge.\n## Q3 Visual results and caption improvement.\nWe include additional visualizations in the rebuttal. In Fig. 2 (a), we show provenances sampled on the ground truth surface. In Fig. 2 (b), we show improvements to depth images after applying $\\mathcal{L}\\_{provNVS}$ to SCADE. We will include them in the final version and update captions for Fig. 3, 4, and 6 of main for clarity.\n## Q4 Uncertainty’s connection with depth error.\nA poorly triangulated region likely has incorrect depth due to large depth ambiguity (c.f. Fig. 5). Consequently, regions with high depth error should have high triangulation uncertainty and vice versa. Thus, we validate our uncertainty map by examining its correlation with depth errors, which is a common approach in prior works [51-52]. \n## Q5 What are considered improvements in uncertainty maps?\nA good uncertainty map should correlate well with depth errors. E.g., the boxed regions in Fig. 4 (main) demonstrate the superiority of our uncertainty over the baselines. Our method accurately identifies both chairs as certain due to low depth errors while baselines incorrectly mark them as uncertain.\n## Q6 Floaters/Fuzziness not addressed totally.\nFig. 3 (main) shows visuals from the NVS experiment using our $\\mathcal{L}\\_{provNVS}$, improving scene reconstruction by clearing most floaters. Rendered depths included in Fig. 2 (b) of the rebuttal show further validation. While our method in theory may not eliminate all the artifacts when they are invisible from training cameras, in practice it removes most floaters as shown in the figures above. In Fig. 4 (main) that compares uncertainty estimation methods, floaters are present because we didn't apply the $\\mathcal{L}\\_{provNVS}$ regularizer; these floaters are from the pretrained SCADE.\n## Q7 Uncertainty evaluation.\nNLL, a common uncertainty metric [51-52, 55], measures the negative log-likelihood of ground truth surfaces under the model's uncertainty prediction. The metric is intuitive as effective uncertainty maps should assign high likelihood to the true scene surface. While Area Under Sparsification Error (AUSE) is also used [19. 73], Figure S5 shows its defects that can lead to unreliable scores. We therefore opted to evaluate NLL.\n## Q8 Provenance parameterization.\nYes, we can model provenance as a 3D vector that connects the observation center with the 3D location. Other parameterizations such as a camera pose in se(3) are also viable.\n## Q9 3DGS discussion.\nOur method is compatible with 3DGS. We integrate ProvNeRF to 3DGS and assess its triangulation uncertainty using the method in Sec. 5.2. See Fig. 1 of the attached PDF for both visual and quantitative comparisons with FishRF [73]. The results demonstrate the potential of applying ProvNeRF to explicit representations like 3DGS, and we plan to explore this further in the future. Details are in our global response.\n## Q10 Experiment setting: should be sparse and why use scannet, tanks and template dataset.\nOur experiments indeed use a sparse setting following SCADE [58]. Specifically, ScanNet and T&T are standard datasets [58] with 18-26 training images in unconstrained camera poses in each scene. We also test NVS on the In-the-Wild dataset from [58] achieving superior metrics over SCADE:\n||PSNR|SSIM|LPIPS|\n|-|-|-|-|\n|SCADE|22.82|**0.743**|0.347|\n|Ours|**22.85**|**0.743**|**0.343**|\n## Q11 Fig. 2 clarification.\nFig. 2 illustrates that modeling provenance is not straightforward, as different triangulation-related phenomena [20] such as camera baseline distance and occlusions need to be taken into account. We’ll improve this figure and caption in the final version.\n## Q12 Provenance visualization. We provide additional visualization of our learned provenances in Fig. 2(a) in the attached PDF. Note that we visualized provenances together with the training cameras in the supplementary video. \n## Q13 “Ours” built on NeRF?\nOurs is built on SCADE [58] (c.f. Ln 232-233). The only difference from SCADE is the extra $\\mathcal{L}\\_{fIMLE}$ loss in Eq. (8). \n## Q14 How sampling is done in practice.\nD is sampled every 1000 iterations following IMLE [32], and (x, t, d) are sampled every iteration. \n## Q15 $\\mathcal{L}\\_{provNeRF}$ v.s. $\\mathcal{L}\\_{provNVS}$. \n$\\mathcal{L}\\_{provNeRF}$ is crucial for learning the provenance stochastic process while $\\mathcal{L}\\_{provNVS}$ leverages this process to enhance scene reconstruction. Ablation studies show that training without $\\mathcal{L}\\_{provNeRF}$ leads to slightly worse results (`Ours*` in Table S1). This is because joint optimization synergistically adapts the provenance samples to the current geometry during joint optimization. Conversely, omitting $\\mathcal{L}\\_{provNVS}$ results in 21.44, 0.716 and 0.349 for PSNR, SSIM and LPIPS respectively. This is inferior to the pretrained SCADE because $\\mathcal{L}\\_{provNeRF}$ does not improve the reconstruction using provenances. \n\n[73] Jiang, W. et. al. (2023). FisherRF, CVPR 24’", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "J4fnDKNvBc", "reviewer": "Reviewer_u5T3", "summary": "This paper addresses gaps in existing Neural Radiance Fields research by modeling the provenance of each point as a stochastic process and enhancing triangulation quality through an extended Implicit Maximum Likelihood Estimation (IMLE) to functional space, resulting in improved novel view synthesis and uncertainty estimation under sparse, unconstrained view conditions. Experimental results demonstrate the effectiveness of the proposed method.", "questions": "1. Does the proposed method solve the uncertainty problem? Does the uncertainty problem really matter in the NeRF-based method? If so, why does the proposed method have inferior performance compared with other sOTA methods in evaluation, e.g. table 1?\n2. Can this method be applied to 3DGS or INGP-based methods? How much time it will cost to add the proposed framework in the post-optimiazation?", "limitations": "The authors demonstrate shortcomings concerning the optimization duration and mention a solution to address these deficiencies in the appendix section.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The idea is interesting, especially the defined problem in this paper makes a lot of sense to me. ProvNeRF enhances Neural Radiance Fields by integrating per-point provenance information during training, which enriches the model with critical insights on triangulation quality. This integration leads to a combination of traditional 3D vision-related fields, e.g. triangulation and uncertainty, to novel view synthesis, particularly under challenging sparse and unconstrained viewing conditions.\n2. The writing is very clear and easy to follow.", "weaknesses": "1. My main concern is about the evaluation of the proposed idea. According to the paper, modeling the provenance for each point could potentially improve NeRF's accuracy in predicting the positions of points in space. However, there are few direct comparative experiments on depth/position, with only Fig. 4 showing some reflection. The depth error does not seem to significantly differ from other methods. This makes the evaluation less convincing from my view.\n2. From Table 1,2, I wouldn't say that the result is significantly improved. And, the visualization in Fig.4, the difference between the proposed one and Bayes' is not notable. These make me a little bit concerned about the significance of the proposed method as it requires an 8-hour post-optimization.\n\nminor \n1. The description of the images in Fig. 2 is reversed, with left and right sides swapped.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for finding our method interesting, easy to follow, and enriches NeRFs with critical insights on triangulation quality. \n\nWe here clarify our experiment setup and avoid confusion. Our main experiments are split into two parts, each corresponding to a different application using ProvNeRF. 1) Novel view synthesis (Sec. 5.1): we use our learned provenance to formulate an additional regularizer $\\mathcal{L}\\_{provNVS}$ to improve scene reconstruction. We evaluate our approach in Fig. 3 and Tab. 1, demonstrating improvements compared to baselines. 2) Triangulation uncertainty estimation (Sec. 5.2): we use our learned provenance to estimate triangulation uncertainty. We show results on uncertainty estimation in Fig. 4 and Tab. 3 against baselines by measuring the uncertainty’s correlation to depth errors. This is a common evaluation protocol in prior works [51-52]. Note that in this context we do not compare the depth error across different models. We address your questions below. \n## Q1 Evaluation of the proposed idea.\nAs highlighted by other reviewers, we provide a “thorough” (GM1F) evaluation of our method and achieve improvements in both reconstruction and triangulation uncertainty metrics(o98v). In the attached PDF, we provide additional experimental results, including applying ProvNeRF on 3DGS for uncertainty estimation (Figure 1) and a depth map comparison between our method and SCADE (Fig. 2(b)).  \n## Q2 “Modeling the provenance improves NeRF's accuracy”.\nWe clarify that modeling the provenance alone does not improve the NeRF’s reconstruction. However, it can be used to formulate the $\\mathcal{L}\\_{provNVS}$ regularizer to improve NVS and scene reconstruction as shown by experiments in Sec. 5.1. We further provide improved depth rendering visuals in Figure 2(b) of the attached PDF that supplements Fig. 3. \n## Q3 “Few direct comparative experiments on depth/position, with only Fig. 4 showing some reflection.”\nWe clarify that the depth error maps in Figure 4 are NOT to show improvement in depth as it is part of our uncertainty experiments (Sec. 5.2). Note that our $\\mathcal{L}\\_{provNVS}$ regularizer is not used here. Instead, Fig. 4 visualizes uncertainty maps from different methods and compares their correlations to the depth errors, which serve as a qualitative comparison for uncertainty estimation. \n## Q4 Tables 1 and 2: “I wouldn’t say results significantly improved”.\nAs noted by GM1F, while the improvements in the quantitative metrics on the NVS experiments (Table 1 & 2) are not huge, “several examples where some of the common artifacts present in indoor NeRFs are mitigated if not completely removed.” making our method “quite compelling”. This suggests that our additional regularizer $\\mathcal{L}\\_{provNVS}$ can improve the NeRF reconstruction by modeling provenance that comes for free from the training images.\n## Q5 Difference between Bayes’ Rays in Figure 4.\nSec. 5.2, Fig. 4 compares our approach against baselines such as Bayes’ Rays on uncertainty estimation. Note that a good uncertainty map should correlate well with depth error maps, marking regions with high depth errors as highly uncertain and vice versa. The boxed regions in Fig. 4 demonstrate the superiority of our estimated uncertainty over Bayes’ Rays. For example, our method accurately identifies both chairs (left and right examples) as certain due to good triangulation, contrasting with baselines that incorrectly mark them as uncertain despite their low depth errors. Again, we clarify that the results in Fig. 4 (main) do not show improvement in scene reconstruction – we do not enforce our $\\mathcal{L}\\_{provNVS}$ regularizer here. In fact, the depth error for Bayes’ Rays and ours are the same in Fig. 4 as we use the same base model to compute for uncertainty. For improvements in scene reconstruction see our NVS experiments (Sec 5.1.) where we enforce the regularizer to clear out floaters. \n## Q6 Figure 2 caption.\nThanks for pointing this out. We will fix this in our final version. \n## Q7 Does the method solve the uncertainty problem and why is it important?\nYes, our proposed method can be used to estimate triangulation uncertainty as shown in Sec. 5.2. Uncertainty estimation is crucial for applications needing reliable reconstruction like continual learning and robotic navigation. Several studies [51-52, 55] have addressed this within NeRF settings.\n## Q8 Results are not SotA.\nWe achieve SotA results in both NVS and uncertainty estimation applications. In the NVS experiment, Figure 3 shows our results clearly improve the previous SotA method SCADE by removing floaters and fuzziness on surfaces. This is also reflected quantitatively in Table 1 as we achieve on average better than the baselines on both datasets and ours are better in almost all the metrics; In the uncertainty experiment, qualitative and quantitative comparisons suggest that our uncertainty map is more informative -- i.e., marking poorly triangulated regions with high uncertainty and vice versa -- compared with existing baselines. \n## Q9 Applying ProvNeRF to 3DGS.\nYes, we can apply our method to 3DGS. We apply ProvNeRF on top of 3DGS on Scannet dataset. The post-optimization takes around 30 minutes per scene on a single NVIDIA A6000 GPU. To study its applicability, we used our trained provenance stochastic process to estimate triangulation uncertainty as in Sec. 5.2. See our global response for implementation details. Figure 1 in the attached PDF shows the comparison of estimated uncertainty maps between ours and FisherRF [73]. Note that we obtain a more correlated uncertainty map w.r.t. the estimated depth error from 3DGS. The same figure also shows a NLL comparison against FisherRF where we outperform the baseline by a large margin. These results show that applying ProvNeRF to recent explicit representations such as 3DGS is promising and we leave further exploration as future works.\n\n[73] Jiang, W. et. al. (2023). FisherRF, CVPR 24’", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hobL7jiQYy", "reviewer": "Reviewer_GM1F", "summary": "The paper describes a method for explicitly modeling the visibility of a point in space in a NeRF model. This is done though modeling provenance which is the space of points where the given point is likely visible. The motivation is that modeling visibility enables the underlying NeRF model to better utilize triangulation information from the training images. The proposed method introduces a neural network to model the provenance function which allows for sampling points visible point estimates from a given 3D position. This secondary network can be applied to any base NeRF model. When trained with the provenance loss, the resulting model performs better in the task of novel view synthesis across a variety of scenes and metrics, and allows for explicit modeling of uncertainty in the reconstructed scene geometry.", "questions": "End of line 128,  \"we\" should be capitalized.\n\nThis is not necessarily a weakness, but it would be good to clarify. Are the 3D models being shown in the visualization based on evaluating the provenance on the ground truth mesh from depth fusion, or is that somehow extracted from the NeRF itself. They are shockingly clean to be extracted from a NeRF but it's not clear to me from the text if that is actually what's happening.\n\nLine 168 \" We define a latent function sample Z ~ Z to be the concatenation of a random linear transformation of x and x itself.\" I don't think concatenation is the right word here based on equation 6. I think this sentence is phrased poorly or I'm missing something important.", "limitations": "Yes", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "+ The motivation for the problem is interesting. If you have a way to approximate where a point is visible given the input data, that does provide a lot of insight about the quality of the input data for reconstruction of view synthesis. \n\n+ The results shown are good. The overall metric while better than other approaches are just a slight bump, but there are several examples shown where some of the common artifacts present in indoor NeRFs are mitigated if not completely removed using this method. That improved visual quality on top of the benefits of uncertainty modeling make this quite compelling.\n\n+ The ablation in Table 2 shows that the naive ideas that most people would have tried are worse than this, so I think the more obvious baselines are covered in comparison. Similarly Table 4 is a good ablation over several methods that I think would be the go to baseline for an idea like this. Overall the evaluation is very thorough. \n\n+ The main paper + supplemental is an immense amount of information and evaluation for a single paper. Clearly a lot of effort was put into describing and validating the method.", "weaknesses": "- Overall I found the presentation of the method to be very confusing. There is a lot of technical overview in 4.1-4.3 that I think over complicates what is actually happening. I had to reread these sections several times while making passes through the full paper to really get a grip on what is happening. The architecture diagram in Figure S1 makes it much clearer. I think the extremely technical discussion could be reduced and moved to the supplemental while a more direct description of what is actually being done like in the supplemental should be focused on more.\n\n- I don't follow why norm(d)>=0.7 has any berrying on the confidence of that provenance sample. The output provenance sample D(x) = (t,d) where the point y where x is estimated to be visible is y=x-td. So why is the length of the direction vector a signal for uncertainty? \n\n- The visualizations of the provenances for points don't align with what I expect. Why are the vectors all roughly the same length? Shouldn't it be more varied? Similarly, I don't think the definition of how to recover the point y where x is visible is correct based on the description and visualization. If y = x-td, then the output direction vector would be pointing away from points of visibility, not towards. \n\n- To reiterate my first point, the major weakness of this paper is the somewhat overwhelming amount of technical detail for what boils down to a pretty simple idea. Maybe if the details are reordered so that the simple explanation comes first followed by the more rigorous definition the paper would be easier to follow. After several reads I can follow the early sections, but they are still not extremely clear. I would recommend simplification if possible to make the core contribution as clear as possible and fill in the details where necessary instead of the current presentation. Though I admit that the technical groundwork is necessary so it is a difficult balance. But a clear example of what can be simplified is section 4.3. There is the initial equation 7 discussion, followed by the logic for simplifying to equation 8, which eventually reduces to equation 9 which is what I think most people would have assumed would be the natural conclusion. It's not to say that this presentation of the principles from equation 7 to equation 9 is not important if useful, but it is a bit heavy handed in my opinion.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for finding our work interesting and insightful and our evaluations thorough and compelling. We address the questions raised below. \n## Q1 Presentation of the method section. \nThank you for your valuable feedback. We will incorporate your suggestions and promise to improve the presentation of the method in the final version. We originally intended to describe the method in a mathematically precise way, but we understand this can cause confusion. We will lighten the mathematical notations, simplify the presentation of the idea, and move details to the supplementary. We will describe the method in a more intuitive way in the final version.\n## Q2 Clarification on length and norm of direction vector. \nThe length of the predicted direction determines how visible the provenance direction is to handle object/scene occlusion. That is, we model a visibility term $v \\in [0,1]$ (c.f., Eq. (5) of main) as the norm of the predicted direction that accounts for how “visible” this provenance sample is. For further clarity, we note that we overloaded the notation and used $\\bf{d}$ for both the unnormalized and normalized directions. In practice, once we extract a provenance’s visibility by computing the norm of the direction, we normalize the direction and use it for downstream tasks. In the final version, we will denote the normalized direction as $\\tilde{\\bf d}$ and fix the inconsistencies in the paper. Finally, in our NVS regularizer formulation in Eq. (10), we use the length of the direction vector to distinguish floaters from solid surfaces. I.e., if the predicted direction has $\\lVert\\bf d\\lVert<0.7$, we assume that there is a solid surface in that direction and it shouldn't be regularized. Otherwise if $\\lVert\\bf d\\lVert\\geq0.7$, we treat it as a floater where we apply our regularizer to remove it. We will add this clarification in the final version. \n## Q3 Variation in length of direction vectors. \nThey are varied, as shown by the length of vectors in the supplementary video. We further provide additional visualization in Figure 2 (a) in the rebuttal PDF; we will include this in the final version. We note that the visualization in Figure 6 only shows normalized directions. We apologize for this confusion and we will change it to reflect the length of the direction vector in the final version. \n## Q4 Misalignment of provenance visualization.\nYes, you are right. The visualizations in Figure 6 and the video show the negative of the predicted provenance directions. This is done for ease of illustration: otherwise the samples will be hidden by the scene. We will add this clarification in the final version. \n## Q5 Ln. 128. \nThanks for pointing this out. We will fix it in the final version. \n## Q6 Clarification of the provenance visualization. \nYes, the visualizations shown in the video and Figure 6 are provenance samples generated at the ground truth geometry surface provided by the dataset. We will include this clarification in the final version.\n## Q7 Ln 168. \nThanks for the suggestion. Yes, the expression in Eq.(6) is a more mathematically precise definition of our random function $\\bf{Z}$. We will modify the wording for a more precise definition in the final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mx8xjq78Xz", "reviewer": "Reviewer_o98v", "summary": "Paper presents a method for co-training an inference network that learns where points in a NeRF were “seen from” (ie. provenance) in the training data. This allows for quantification of uncertainty of the reconstructed geometry in terms of triangulation (location) and depth error. Mathematically frames these these provenances within a stochastic process, extending implicit maximum likelyhood estimation (IMLE) to the functional domain over infinite (euclidian) sets. Presensts a loss term for incorporation into NeRF training and demonstrate superior novel view synthesis with it, and lower triangulation uncertainty on two standard datasets.", "questions": "The exposition of section 4.1 is unclear and took me a while to understand. Primarily, I understand that for each point in space $x$ there is a distribution of provenances, But I could imagine two stochastic processes: 1) where each point is sampled from the distribution for a fixed location, or 2) where each point becomes the provenance distribution for the next point drawn. This could be made clearer. Perhaps a diagram would help. (Also note having figure S1 in the main text would have gone a long well to help me understand the method better).\n\n#25 \"modeling for each point, the locations where it is likely visible.\". Do you actually mean \"likely visible in the training dataset\", or \"likely visible in the scene\"? I believe you mean the former, yet this critical point was not clear to me until reading further into the paper. This seems like a critical distinction to make. ie. I could have a single view of a scene, yet properly infer that a point should be visible from many directions, even if not present in the training data.\n\n#126 \"forms a stochastic process, $\\mathcal{D}_\\theta$ indexed by coordinates $x \\in \\mathbb{R}^3$\" Shouldn't this be more correctly defined as a \"stochastic field\", given it's indexing by Euclidean space. In general this is not the \"stochastic process\" most readers would usually be accustomed to, as one indexed by the natural positive numbers. Also, you are defining samples as functions, not as $\\mathbb{R}^3$ positions, correct? Given this is the core of your contribution, extra clarity for readers would be appreciated for this.\n\n#136, since you call $\\mathbf{d}$ a \"direction\" it should be a normalized direction. Therefore the tuple should be in $R_+ \\times \\mathbb{S}^2$, right? ? If $\\mathbf{d}$ is not normalized, I'm not sure I understand why (and should be called a \"vector\", not a \"direction\"). This should be a direction pointing towards the camera origin?\n\n#138 I would define (5) before (4) for clarity. Also, presumably $\\mathbf{o}_i$ is the camera origin? Though I don't believe it's been defined yet.\n\n#211 Where is the actual architecture used for $\\mathbf{H}_\\theta$ described? This implementation detail seems critical to understanding the method, and how it might be helping NeRF training.", "limitations": "Authors leave discussion of limitations to a short paragraph in the supplemental. I’d prefer to see this in the main-text. Main current limitation is a long post-hoc training process. I’d also like to have seen a longer discussion on applicability of the method.\n\nAuthors sufficiently discuss the societal impact of their method.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "While I'm not convinced the \"Methods\" section is as crystal clear as it could be, and the problem domain widely applicable, the paper appears to present a novel contribution in the extension of IMLE to infinite sets and functional samples that should be of likely interest to the NeurIPS community.  The presentation is mathematically rigorous, though somewhat opaque, with extensive derivations in the supplement. \n\nDemonstrates improvements to novel view synthesis of existing NeRF scenes through co-training with their additional loss term and  synthesis term, achieving improvements in reconstruction metrics. Also demonstrates quantitative SotA results over competing methods on triangulation uncertainty.", "weaknesses": "Introduction still leaves me asking “Why is this an important problem?”. The method only addresses triangulation uncertainty, ignoring other forms of uncertainty like transients, perhaps limiting applicability. \n\nOverly mathematical presentation. Is this being presented in the absolute simplest way possible? Confusion over just exactly which the stochastic process is being discussed. Should clearly state the use-case (NeRF training add-on, or co-training objective?) up-front.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for finding our theoretical extension of IMLE to stochastic processes novel and mathematically rigorous and demonstrating improvements over existing methods. We address the questions raised below. \n## Q1 Introduction still leaves me asking “Why is this an important problem?” \nModeling provenances enables analysis of NeRF reconstruction from a classical stereo-matching point of view. As noted by the other reviewers, provenances are important concepts to model because they shed “insight about the quality of the input data for reconstruction” (GM1F) that “leads to a combination of traditional 3D vision-related fields, e.g. triangulation and uncertainty, to novel view synthesis” (u5T3). We will modify the introduction for better clarity and emphasis on the importance of provenance in the final version.\n## Q2 Method only addresses triangulation uncertainty. \nYes, one application of modeling provenances is modeling triangulation uncertainty in NeRFs. While uncertainty within NeRF’s reconstruction is multifaceted, isolating triangulation uncertainty can benefit downstream tasks such as next-view augmentation [19] that concern the capturing process (note other uncertainties such as transient are intrinsic to the scene). In the future, we hope to combine other types of uncertainty with triangulation uncertainty to achieve a more comprehensive understanding of NeRF reconstruction. \n## Q3 Mathematical presentation; should clearly state the use case up-front. \nThank you for your valuable feedback; we will modify our paper accordingly in our final version. Our ProvNeRF is a NeRF training add-on and we will state that up-front. We intend to describe the method in a mathematically precise way, but instead we will in the final version lighten the mathematical notations, simplify the idea's presentation and move details to the supplementary. \n## Q4 Stochastic process in the exposition of Sec 4.1.\nOur stochastic process is defined as a collection of distributions at each 3D location instead of distribution over different 3D locations. To provide an illustration, Figure 2(a) in our rebuttal shows the provenance direction samples at different 3D locations. We will include this in the final version and add further clarifications to avoid confusion. \n## Q5 Clarification on visibility. \nBy visibility, we refer to that given by the training dataset (so the former in your question), not from the scene. We will clarify this in our final version. But modeling visibility of the scene would be an interesting follow-up work. \n## Q6 Stochastic process v.s. Stochastic field. \nYes, stochastic field would be a more suitable name for our provenance model, as each of our samples is a function mapping each 3D location to a provenance of it. We note that there is an inconsistency in the stochastic process literature in terms of terminology. For example, [74] defines stochastic process as an n-dimensional and m-variate random function whereas [75] defines it as those with only a one-dimensional indexing set. However, stochastic field is a more suitable name given that our indexing set is a field. We will change the stochastic process to the stochastic field in our final version. Note that we still use the term provenance stochastic process in rebuttal for terminology continuity. \n## Q7 Why are directions not unit length? \nThe predicted direction samples are not unit length to handle object/scene occlusions. We model a visibility term $v \\in [0,1]$ (c.f., Eq. (5), main) as the norm of the predicted direction that accounts for how occluded this provenance sample is: this is why the distance-direction tuple lives in $\\mathbb{R}_+ \\times \\mathbb{D}^3$. We overloaded the notation and used $\\bf d$ for both the unnormalized and normalized directions. In practice, once we extract a provenance’s visibility by computing the norm of the direction, we normalize the direction and use it for downstream tasks. In the final version, we will denote the normalized direction as $\\tilde{\\bf d}$ and fix the inconsistencies in the paper. For further clarity, we note that our visualizations show the negative of the provenance direction (for clearer visuals) – that is the direction from a 3D location should be pointing away from the camera.\n## Q8 $o_i$; defining Eq. (5) before (4). \nYes, $o_i$ is the camera origin (see Ln. 99 main). We will reorder Eq. (4) and (5) in the final version; thank you for your suggestion. \n## Q9 Architecture of $H_\\theta$. \nIt is a 3-layer MLP with ReLU non-linearity and input feature, hidden feature, and output dimensions being 288, 256, and 4, respectively. We will include these details in Sec. S2 in the final version. \n## Q10 Limitation. \nThanks for the suggestion. We will move our limitation discussion to the main paper. \n## Q11 Longer discussion on method’s applicability. \nOur method can be applied to model stochastic fields other than provenance fields. For example, an interesting application other than provenance modeling is to model the material properties of a NeRF as a stochastic field. For instance, we could model the BRDF at each point in 3D as a stochastic field over the incoming and outgoing solid angles. Learning such a stochastic field would allow us to enable interesting applications such as relighting or material modification. Another possibility is to extend the deterministic framework in [76] to model the equivalent classes of signed distance fields given by a set of discrete signed distance samples as a stochastic field. This allows us to sample different but all plausible signed distance fields (thus implicit surfaces) given a set of discrete signed distance samples. We will include this discussion in the final version.\n\n[74] Shinozuka, M. (1987). Stochastic Fields and their Digital Simulation. \n\n[75] Knill, O. (2009) Probability Theory and Stochastic Processes with Applications. \n\n[76] Sellán, S., et. al. (2024). Reach For the Arcs, SIGGRAPH 24’", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank reviewers for their feedback and for finding our approach interesting (u5T3, GM1F), sound (GKL5), and novel (o98v). Our new formulation is classically motivated (GKL5), can be applied to any base NeRF models (GM1F), and leads to “a combination of traditional 3D vision-related fields to novel view synthesis” (u5T3). We also experimentally validate our provenance stochastic process in two applications that both lead to improvements over previous SotA methods (GKL5, o98v, GM1F). Below we include a summary of each response. Please see individual reviewer responses for more details.
## [GKL5, u5T3] Applying ProvNeRF to 3DGS.
We plug in ProvNeRF to a pretrained 3DGS on Scannet with additional depth supervision for convergence. We model a provenance distribution for each splat using IMLE with a shared 6-layer MLP for $H_\theta$, which takes around 30 minutes to train. After training we use it for uncertainty estimation as delineated in Sec. 5.2 (main). Figure 1 in the attached PDF shows a comparison to FisherRF [73], a recent uncertainty estimation work on 3DGS. Compared to their uncertainty map, ours shows more correlation to the depth error as highlighted by the boxed regions. Quantitatively we evaluate NLL on the three Scannet scenes shown on the right side of the same figure and show substantial improvements over FishRF. This improvement over existing literature suggests applying ProvNeRF to other representations such as 3DGS is promising. We leave further exploration of the method and applications as future works. 
## [GKL5] Alternative ways to model provenances.
We experimented with other modeling strategies such as deterministic fields, gaussian mixtures, and VAEs. However, all of these methods suffer from mode collapses. Table 4 and S1 show our method’s quantitative advantage in provenance sampling and NVS co-training. 
## [GKL5] How to interpret uncertainty visualization and uncertainty evaluation.
Uncertainty maps are typically measured by their correlation to depth error [19]. An ideal uncertainty map should mark regions with high-depth error with high uncertainty and vice versa. We quantitatively evaluate uncertainty maps using negative log-likelihood following prior works [51-52].
## [GKL5] Testing benchmarks.
ScanNet and T&T are standard evaluation datasets used in prior works [58] that are scene-level with varied training camera setups.
## [GKL5, GM1F, o98v] Provenance stochastic process visualization.
We visualize provenance samples on the ground truth scene surfaces in Figure 2 (a) of the attached PDF. Finally, we note that the visualized provenance directions are negative of the predicted directions for illustration. We will include this clarification and the figure in the final version. 
## [GKL5] $\mathcal{L}\_{provNVS}$ v.s. $\mathcal{L}\_{provNeRF}$.
$\mathcal{L}\_{provNeRF}$ is important for learning the provenance stochastic process and $\mathcal{L}\_{provNVS}$ is crucial for improving scene reconstruction using the provenance process. 
## [u5T3] Evaluation of the proposed idea.
The evaluation of ProvNeRF is split into two parts: Sec. 5.1 evaluates its NVS improvement using the additional regularizer $\mathcal{L}\_{provNVS}$. This enables the removal of common artifacts in NeRFs and improves the reconstruction metrics (GM1F, o98v). Sec. 5.2 evaluates our uncertainty estimation where we show SotA performance (o98v) compared with baselines.
## [u5T3] Results not significantly improved.
Table 1, 2, and Figure 3 in Sec. 5.1 shows $\mathcal{L}\_{provNVS}$ can remove common artifacts in indoor NeRFs (GM1F). This leads to improvements in reconstruction metrics (GM1F, o98v). We provide the rendered depth images in Figure 2 (b) of the attached PDF to demonstrate the improved geometry. 
## [GM1F, o98v] Overly mathematical presentation.
Thanks for the suggestions. We aimed to present the method section with precise definitions. Instead, we will lighten the mathematical notations, move derivation details in Sec 4 to supplementary, and present the method section in an intuitive manner.  
## [GM1F, o98v] Length of the provenance direction.
We use the length of the provenance direction to model its visibility (c.f. Eq.(5)). A lower direction norm usually means that the 3D location is occluded from this direction. Additionally, we made a notational mistake and represented both the normalized and unnormalized provenance directions as **d**. We will fix this in the final version. 
## [o98v] Importance of provenances.
Modeling provenances in NeRF allows for analyzing NeRF’s reconstruction from traditional 3D vision (u5T3), shedding critical insights on NeRF’s convergence (u5T3, GM1F). We will better motivate provenances in the intro of the final version. 
## [o98v] The method only addresses triangulation uncertainty.
While one of our applications is to model triangulation uncertainty, isolating triangulation uncertainty can benefit downstream tasks such as next-view augmentation [19]. We leave modeling other types of uncertainty (e.g., transients) as further work. 
## [o98v] Stochastic Process v.s. Stochastic Field.
Thanks for the suggestion. There seems to be a terminology inconsistency for random functions with a multivariate indexing set [74-75]. But we agree that stochastic field is a better terminology as we have an $\mathbb{R}^3$ indexing set. We will change stochastic processes to fields in the final version. Note that we still use the term provenance stochastic process in rebuttal for terminology continuity. 
## [GKL5, u5T3, GM1F, o98v] Typos and figures.
We thank all reviewers for their suggestions. We will fix the typos and improve the figures for better illustration and clarity in the final version. 

[73] Jiang, W. et. al. (2023). FisherRF, CVPR 24’

[74] Shinozuka, M. (1987). Stochastic Fields and their Digital Simulation. 

[75] Knill, O. (2009) Probability Theory and Stochastic Processes with Applications.

Meta-review: The reviewers generally agree that the proposed approach is well written/motivated, has technical novelty, and is evaluated sufficiently, outperforming baselines. However, there are also shared concerns about the clarity of the theoretical presentation, the significance of the results, and difficulty interpreting visualizations. A robust rebuttal was submitted that helped to address most of the reviewer concerns. After the discussion phase, there is a shared belief that this work will be of interest to the community. As such, the AC reached a decision to accept the paper. Please take the reviewer feedback into account when preparing the camera ready version.
Decision: Accept (poster)