Title: ProvNeRF: Modeling per Point Provenance in NeRFs as a Stochastic Field
Venue: NeurIPS 2024
Accepted: yes
Keywords: NeRF

Reconstruction

Stochastic Process

Sparse View

Novel View Synthesis

Uncertainty Estimation
Abstract: Neural radiance fields (NeRFs) have gained popularity with multiple works showing promising results across various applications. However, to the best of our knowledge, existing works do not explicitly model the distribution of training camera poses, or consequently the triangulation quality, a key factor affecting reconstruction quality dating back to classical vision literature. We close this gap with ProvNeRF, an approach that models the provenance for each point -- i.e., the locations where it is likely visible -- of NeRFs as a stochastic field. We achieve this by extending implicit maximum likelihood estimation (IMLE) to functional space with an optimizable objective.  We show that modeling per-point provenance during the NeRF optimization enriches the model with information on triangulation leading to improvements in novel view synthesis and uncertainty estimation under the challenging sparse, unconstrained view setting against competitive baselines. The code will be available at https://github.com/georgeNakayama/ProvNeRF.

ProvNeRF: Modeling per Point Provenance in NeRFs as a Stochastic Field

Abstract

Neural radiance fields (NeRFs) have gained popularity with multiple works showing promising results across various applications. However, to the best of our knowledge, existing works do not explicitly model the distribution of training camera poses, or consequently the triangulation quality, a key factor affecting reconstruction quality dating back to classical vision literature. We close this gap with ProvNeRF, an approach that models the provenance for each point - i.e., the locations where it is likely visible - of NeRFs as a stochastic field. We achieve this by extending implicit maximum likelihood estimation (IMLE) to functional space with an optimizable objective. We show that modeling per-point provenance during the NeRF optimization enriches the model with information on triangulation leading to improvements in novel view synthesis and uncertainty estimation under the challenging sparse, unconstrained view setting against competitive baselines 1 .

1 Introduction

Neural radiance fields (NeRFs) [42], allowing for learning 3D scenes given only 2D images, have grown in popularity in recent years. It has shown promise in many different applications such as novel view synthesis [4, 6], depth estimation [14], robotics [22, 1], localization [36, 38], etc. Existing literature [11, 15, 45] show that the quality of NeRF reconstruction is correlated with the selection of training camera poses. Similar correlations are observed in the classical literature too, triangulation is highly dependent on camera poses [47, 44, 3], which greatly influences the reconstruction quality. One common and important setting in computer vision literature [2, 37, 40, 16] is the sparse view [18] setting in unconstrained [53] environments, and triangulation is even more critical, affecting the reconstruction quality as limited input views make the system more sensitive to noise.

Despite the correlation between triangulation and reconstruction quality, to the best of our knowledge, existing works do not explicitly model the former when optimizing the latter. In this work, we address this gap in the literature by modeling for each point the locations where it is likely visible . We dub this as the provenances of a point. Modeling and learning per-point provenance can help NeRF understand how the training cameras are distributed in space, which inherently links it to triangulation and reconstruction quality.

However, determining the provenances of a point x without the underlying geometry is not straightforward as many factors influence the visibility of each point in the reconstructed geometry. For example, the literature on stereo matching [44, 47] has extensively studied the influences of camera locations on 3D reconstruction. One such well-known challenge arises when selecting the baseline of a pair of cameras in a stereo system. As shown in Fig. 2, points' visibility can suffer from different sets of errors when the length of the camera pair's baseline changes. For NeRFs, the dependence

ProvNeRFenablesmany applications:

NovelViewSynthesis

UncertaintyEstimation

TestView

DepthMSE

ProvNeRF

LProvNVS

TriangulationUncertainty

Training

Cameras

becomes more complex as multiple cameras' visibility needs to be estimated. To overcome this challenge, we propose to model the provenance as the samples from a probability distribution , where a location y is assigned with a large likelihood if and only if x is likely to be visible from y .

To handle the potential complexity of this distribution, we represent the provenance of x as a set of location samples , generated from a learned probability distribution. This is distinct from the existing 'attribute" prediction extensions of NeRFs [69, 30, 8] since provenance is a distribution for every 3D point in space. Thus, this amounts to modeling an infinite collection of distributions (per-point's provenance) over all 3D points, which is mathematically, a stochastic field over R 3 . In our work, we extend implicit maximum likelihood estimation (IMLE) [33], a sample-based generative model, to model stochastic fields by adapting the objective to functional space. Furthermore, we derive an equivalent pointwise objective that can be efficiently optimized with gradient descent and use it to model the provenance field.

We dub our method ProvNeRF which models per-point provenance during the training stage of NeRF (Fig. 1). This enriches the model with information on triangulation quality when the model parameters are optimized. Once the provenance stochastic field is trained, we show that we can use it to improve novel view synthesis (Sec. 5.1) and estimate triangulation uncertainty in the capturing process (Sec. 5.2) under the challenging sparse, unconstrained view setting.

2 Related Works

NeRFs and their Extensions. Neural radiance fields (NeRFs) [42] have revolutionized the field of 3D reconstruction [20] and novel view synthesis [50, 32] with its powerful representation of a scene using weights of an MLP that is rendered by volume rendering [41, 58]. Follow-ups on NeRF further tackle novel view synthesis under more difficult scenarios such as unconstrained photo collections [39], unbounded [5], dynamic [35] and deformable [46] scenes, and reflective objects [59, 7]. Going beyond novel view synthesis, the NeRF representation has also shown great promise in different applications such as autonomous driving [56, 62], robotics [1, 22] and editing [67, 63]. Recent works have also extended NeRFs to model other fields in addition to color and opacity such as semantics [69, 68], normals [66], CLIP embeddings [28], image features [30] and scene flow [34]. Most of these works learn an additional function that predicts an auxiliary deterministic output at each point that is either a scalar or a vector, trained with extra supervision using volume rendering. All of the above works use a deterministic field to output the additional information. However, because each point's provenance is a probabilistic distribution, we need to model a stochastic field instead of a deterministic field for provenance.

Sparse View Novel View Synthesis. NeRFs with rendering supervision alone struggle with sparse view input due to insufficient constraints in volume rendering. Several approaches have been proposed to train NeRFs under the sparse-view regime with regularization losses [43, 64], semantic

SmallBaseline:

WideBaseline:

Large stereo range error, but

Small stereo range error, but

unlikelytobeoccluded

likelytobeoccluded

Invisible

Invisible

consistency [23], and image [60] or cost volume [12, 61, 10] feature constraints. Other works also constrain the optimization using priors from data [24, 65] or depth [49, 57, 54]. Despite addressing the setting with limited number of input views, many works are not specifically designed to tackle our desired sparse, unconstrained views setting as they either focus on object-level [24, 65, 43, 23, 21], limited camera baselines [12, 61], or forward-facing scenes [60] scenes. Recent works [49, 57, 54] have looked into improving the NeRF quality on a more difficult setting of sparse, unconstrained (outward-facing) input views by incorporating depth priors. However, none of these works consider the locations and orientations of training cameras in their optimization process despite it being one of the major factors influencing the NeRF's optimization in a sparse setup.

Uncertainty Modeling in Neural Radiance Fields. The current literature separates NeRF's uncertainty into aleatoric - e.g., transient objects or changes in lighting - and epistemic - data limitation due to weak texture or limited camera views - uncertainties. Some works [39, 26] model aleatoric uncertainty by directly predicting the uncertainty values through a neural network. However, their approach requires training on large-scale data and is not suited for estimating the uncertainty of a specific scene. On the other hand, several works explore epistemic uncertainty estimation in NeRFs through variational inference [51, 52, 48], ensemble learning [55, 31], and Bayesian inference [19, 25, 45]. While these works estimate epistemic uncertainty, they still entangle different sources of uncertainty such as texture, camera poses, and model bias, resulting in unclear and inconsistent definitions of the uncertainty quantified. In our work, we specifically model the uncertainty caused by the capturing process that is useful in various downstream tasks [45, 26, 48, 31].

3 Preliminaries

3.1 Neural Radiance Fields (NeRF)

A neural radiance field (NeRF) is a coordinate-based neural network that learns a field in 3D space, where each point x ∈ R 3 is of certain opacity and color . Mathematically, a NeRF is parameterized by two functions representing the two fields F ϕ,ψ = ( σ ψ ( x ) , c ϕ ( x , d )) , one for opacity σ ψ : R 3 → R + and one for color c ϕ : R 3 × S 2 → [0 , 1] 3 , where d ∈ S 2 is the direction from where x is viewed from. One of the key underpinnings of NeRFs is volume rendering allowing for end-to-end differentiable learning with only training images. Concretely, given a set of M images I 1 , I 2 , ..., I M and their corresponding camera poses P 1 , P 2 , ..., P M , the rendered color of a pixel x is the expected color along a camera ray r i,x ( t ) = o i + t d i,x , where o i is the camera origin and d i,x is the ray direction for pixel x that can be computed from the corresponding camera pose P i . The pixel value for 2D coordinate x is then given by the line integral:

where t n , t f defines the near and far plane, and

is the transmittance of the point r i,x ( t ) along the direction d i,x .

3.2 Implicit Maximum Likelihood Estimation

One choice of probabilistic model is implicit maximum likelihood estimation (IMLE) [33] that represents a distribution as a set of samples and is designed to handle possibly multimodal distributions. As an implicit probabilistic model, IMLE learns a parameterized transformation H θ ( · ) of a latent random variable, e.g. a Gaussian z ∼ N (0 , I ) , where H θ ( · ) often takes the form of a neural network that output samples w j = H θ ( z j ) with w j ∼ P θ ( w ) . Here, P θ is a probability measure obtained by transforming the standard Gaussian distribution measure via H θ . Given a set of data samples { ˆ w 1 , ..., ˆ w N } , the IMLE objective optimizes the model parameters θ with

It is shown that the above objective to be equivalent to maximizing the likelihood.

4 Method

In the following sections, we formally define the provenance at all points as a stochastic field (Sec. 4.1), extend IMLE to model the provenance field (Sec. 4.2), and derive an equivalent pointwise loss for gradient descent (Sec. 4.3).

Notations. Wedenote a stochastic field with a calligraphic font ( D θ ) and samples from the stochastic field using the same letter but bolded ( D θ ). Concretely D θ is a function sample, a function defined over all points x ∈ R 3 , that is sampled from the stochastic field D θ , i.e. D θ ∼ D θ . D θ maps each point to one possible sample in its provenances. We also denote the distribution of the provenances at point x as D θ ( x ) . Moreover, a provenance sample D θ ( x ) from D θ ( x ) is equivalent to evaluating the function D θ ∼ D θ at x . Finally, we let a hat ( ˆ · ) denote the empirical samples/distributions.

4.1 Provenance as a Stochastic Field

The provenance of a point is defined as the locations where it is likely visible from , and as a point can be visible from multiple locations, it can be represented as samples from a distribution. Specifically, provenances of a point x can be represented as samples from its provenance distribution D θ ( x ) . That means that the likelihood of sampling a location y ∈ R 3 from D θ ( x ) determines how likely x is visible from location y . Because such distributions are defined for each 3D point x ∈ R 3 , the collection of per-point provenances forms a stochastic field D θ indexed by coordinates x ∈ R 3 .

Empirically, given sparse training camera views P 1 , . . . , P M , if x is inside the camera frustum Π i for view P i and is not occluded, an empirical sample from the provenances of x can be parameterized as a distance-direction tuple ˆ D i ( x ) = ( ˆ t i, x , ˆ d i, x ) ∈ R + × D 3 2 . Considering all M training views, the empirical distribution of provenances at point x is defined by the following density function:

where δ is the Dirac delta function; v i, x ∈ [0 , 1] determines the length of d to handle occlusions. We modeled it as the transmittance from the NeRF model. To recover the location that observes x , we can write y i, x = x -t i, x d i, x .

Z

D(x)

2

(x)

(tjxdjx)

(t, a)

(xt)

a

Zj;: R3 → Rb

D

0

min

j

Zk: R3 → Rb

Zk(xt)

θ

LprovNeRF

While the above empirical distribution of provenances is given by the training cameras, the actual distribution of provenances, i.e. for each point the locations that point is likely visible from, can have a more complex dependence on both the underlying geometry and the cameras. To capture this complexity, we model D θ ( x ) as a learnable network, a probabilistic model that can model potentially complex distribution, and one choice of such a model is implicit maximum likelihood estimate (IMLE) [33]. Similar to the empirical distribution, we also represent provenance samples from D θ ( x ) as a distance-direction tuple ( t, d ) as defined in Eq 5. We optimized our network with the empirical distribution ˆ D as training signals.

D θ ( x ) defines a distribution for all point x ∈ R 3 . Treating R 3 as the index set, D θ = {D θ ( x ) } x ∈ R 3 defines a stochastic field on R 3 as a collection of distributions D θ ( x ) for all x ∈ R 3 . Because a stochastic field is composed of infinitely many random variables over R 3 , existing methods cannot be applied out of the box as they only model finite-dimensional distributions. In the following sections, we extend IMLE [33] to model this stochastic field.

4.2 ProvNeRF

ProvNeRF models provenances of a NeRF as a stochastic field by extending IMLE [33] to functional space. IMLE learns a mapping that transforms a latent distribution to the data distribution, where each data sample is either a scalar or a vector (Sec. 3.2). However, in our context, since samples from the stochastic field D θ are functions mapping 3D locations to provenances, we need to extend IMLE to learn a neural network mapping H θ that transforms a pre-defined latent stochastic field Z to the provenance distribution D θ (See Fig. 3).

Let Z be the stochastic field where each sample Z ∼ Z is a function Z : R 3 → R b . To transform Z to D θ , fIMLE learns a deterministic mapping H θ that maps each latent function Z ∼ Z to a function D θ ∼ D θ via composition: D θ = H θ ◦ Z . H θ here is represented as a neural network to handle complex transformations from Z to D θ .

We define a latent function sample Z ∼ Z to be the concatenation of a random linear transformation of x and x itself. Mathematically, each latent function Z ∼ Z is a block matrix of size ( b +4) × 3 :

Although Z can be designed to have non-linear dependence on the input location x , we experimentally show that this simple design choice works well across different downstream applications.

To train H θ , we maximize the likelihood of the training provenances (Eq. 4) under D θ for each x using the IMLE objective [33] extended to functional space. We term this extension as functional Implicit Maximum Likelihood Estimation (fIMLE). Because a direct extension to fIMLE leads to an intractable objective, we derive an efficient pointwise loss between the training provenances and model predictions equivalent to the fIMLE objective in the following section.

SCADE

Ours

Views

Test

4.3 Functional Implicit Maximum Likelihood Estimation

We construct an IMLE objective for stochastic fields to maximize the likelihood of training provenances under D θ . Similar to Eq. 3, if we have i.i.d. empirical samples ˆ D 1 , . . . , ˆ D M from the empirical stochastic field ˆ D (defined in Sec. 4.1), and model samples D (1) θ , . . . , D ( K ) θ from the parameter stochastic field D θ , we define the fIMLE objective as

Unlike the original IMLE objective (Eq. 3) that can be directly optimized, the fIMLE objective in Eq. 7 requires the computation of a L 2 integral norm - a functional analogy to the L 2 vector norm which, in general, is not analytically in closed form. Furthermore, approximations of this integral are very expensive since each point query to D θ needs a forward pass through H θ .

To get around this, we use the calculus of variations to reformulate Eq. 7 to minimize the pointwise difference between the empirical samples and model predictions 3 . This allows us to write the fIMLE objective as

where U (Ω) is a uniform distribution over the scene bound Ω . Eq. 8 only requires computing the pointwise difference between samples from ˆ D ( x ) and D θ ( x ) , making it efficiently optimizable with

gradient descent. Ultimately, ProvNeRF jointly updates the underlying NeRF's parameters and D θ by minimizing

where ( t j, x , d j, x ) = D ( j ) θ ( x ) , ( ˆ t, ˆ d ) are i.i.d. samples from ˆ D ( x ) , and L NeRF is the original objective of the NeRF model, e.g. photometric loss and depth loss. We provide implementation and architectural details in the supplementary material. See Figure 3 for the training pipeline illustration.

5 Experiments

Our ProvNeRF learns per-point provenance field D θ by optimizing L ProvNeRF on a NeRF-based model. To validate ProvNeRF, we demonstrate that jointly optimizing the provenance distribution and NeRF representation can result in better scene reconstruction as shown in the task of novel view synthesis (Sec. 5.1). Moreover, we also show that the learned provenance distribution enables other downstream tasks such as estimating the uncertainty of the capturing field (Sec. 5.2). We provide an ablation study on fIMLE against other probabilistic methods in Sec. 5.3. Lastly, in Sec. 5.4, we show a preliminary extension of ProvNeRF to 3DGS [27].

Stochastic Provenance Field Visualization Fig. 7 visualizes the provenance stochastic field by sampling 16 provenances on a test view of the Scannet 758 scene. The directions of the samples are the negative of the predicted provenance directions for better illustration. Each sample is colored based on its predicted visibility. Notice that fIMLE allows ProvNeRF to predict multimodal provenance distributions at different scene locations.

5.1 Novel View Synthesis

We show modeling per-point provenance improves sparse, unconstrained novel view synthesis. As a point's provenances are sample locations from where the point is likely visible, the region between the provenance location samples and the query point should likely be empty. We design our provenance loss for NVS with this intuition.

Concretely, starting from a given NeRF model, we first sample points x 1 , ..., x N for a training camera ray parameterized as ˆ r x ( t ) . Here we denote point x i = ˆ r x ( ˆ t i ) . We only take points x i with transmittance greater than a selected threshold λ = 0 . 9 . For each visible point x i , we sample provenances ( t ( i ) 1 , d ( i ) 1 ) , . . . , ( t ( i ) K , d ( i ) K ) from D θ ( x i ) with ∥ d ( i ) 1 ∥ ≥ 0 . 7 . Then each distance-

2 direction tuple ( t ( i ) j , d ( i ) j ) gives a location y ( i ) j = x i -t ( i ) j d ( i ) j from which x i is observed. This in turn means x should be equally visible when rendered from ray parameterized as r ( i ) x ( t ) = y ( i ) j + t d ( i ) j , ∀ j . With this, we define our provenance loss for novel view synthesis as

where [ . . . ] + denotes the hinge loss and α = 0 . 05 is a constant margin. L ProvNVS encourages the transmittance at x i along training camera rays to be at least the visibility predicted by the sampled provenances from the provenance field with margin α . By matching transmittances between the provenance directions and the training rays, L ProvNVS can be used together with L ProvNeRF to optimize the NeRF representation and the provenance field, resulting in an improved scene geometry. We apply ProvNeRF to SCADE [57] for the task of novel view synthesis. See the supplement for details on the dataset, metrics, baselines, and implementation details.

Results. Table 1 shows our approach outperforms the state-of-the-art baselines in NVS on scenes from both the Scannet [13] and Tanks and Temples [29] dataset. Qualitative comparisons are shown

TestView

TestView

Ensemble

CF-NeRF

Bayes'

'Ray

Ours

Ensemble

CF-NeRF

Bayes' Ray

Ours

Uncertainty

Certain

UncertainiCertain

Uncertain

Depth Error

Low

High

Low

High

in Fig. 4. We see that compared to the baseline SCADE, whose geometry is already relatively crisp, our L ProvNVS can further improve its NVS quality by removing additional cloud artifacts, as shown in the encircled regions. Note that this improvement does not require any additional priors and is only based on the provenance of the scene.

We also compare our performance with deterministic baselines: Deterministic Field regresses one provenance for each 3D location using a neural network and Frustum Check calculates the training provenance defined in Eq. 4 by back-projecting the sampled points to one of the training camera and use that as the regularization information. Table 2 shows that our provenance field outperforms these baselines on the novel view synthesis task because the deterministic field cannot model complex provenance distribution and the frustum check baseline lacks generalization ability as it cannot be optimized to adapt the output provenance based on the current NeRF's geometry.

5.2 Modeling Uncertainty in the Capturing Process

Provenances allow for estimating the uncertainty in triangulation, i.e., the capturing process. In classical multiview geometry [20], the angle between the rays is a good rule of thumb that determines the accuracy of reconstruction. Fig. 6 illustrates this rule as the region of uncertainty changes depending on the setup of the cameras. Formally, for a 3D point x , we sample provenances { ( t j , d j ) } K j =1 from D θ ( x ) . Treating d j as the principal axes and t j as the distances from x to the camera origin, each provenance sample defines a pseudo camera P j that observes x at pixel location x j = Proj j ( x ) . Following chapter 12.6 of [20], we define the triangulation uncertainty of x as the probability of x given its noisy 2D pseudo observations: P ( x | x 1 , . . . , x K ) ∝ P ( x 1 , . . . , x K | x ) P ( x ) = ∏ K j =1 P ( x j | x ) P ( x ) .

Measurement

Noise

Proj(x)

Uncertainty

Region of

Proj2(x)

Proj2(x)

Proj1(x)

Visible

Sample

Visibitity

Invisible

The last two equalities are derived by assuming independence of the 2D observations and each P ( x j | x ) follows a Gaussian distribution N ( Proj j ( x ) , σ 2 ) . This assumption is equivalent to corrupting each 2D observation Proj j ( x ) by a zero-mean Gaussian noise with σ 2 variance, accounting for measurement noises in the capturing process. Assuming a uniform prior of P ( x ) over the scene bound, the exact likelihood can be efficiently computed with importance sampling. This quantifies a point's triangulation quality given the sampled provenances, which becomes a measurement of the uncertainty of the capturing process. We apply our provenance field to ProvNeRF with different NeRF backbones [57, 49] and compute the likelihood. See supplementary for details on the dataset, metrics, baselines, and implementations.

Results. Tab. 3 shows the quantitative results on Scannet [13] and Matterport3D [9]. We follow [51, 55] to measure the negative log-likelihood (NLL) of the ground-truth surface under each model's uncertainty prediction. Since our ProvNeRF can be applied to any pre-trained NeRF module, we use pre-trained SCADE [57] for Scannet and DDP [49] for Matterport3D, both of which are stateof-the-art approaches in each dataset. Our approach achieves the best NLL across all scenes in both datasets by a margin because we compute a more fundamentally grounded uncertainty from classical multiview geometry [20] based on triangulation errors, while both CF-NeRF and Bayes' Rays require an approximation of the true posterior likelihood. Fig. 5 shows qualitative comparisons between baselines' and our method's uncertainty estimation. We expect a general correlation between uncertain regions with high-depth errors. An ideal uncertainty map should mark high-depth error regions with high uncertainty and vice versa. As shown in the boxed regions in the figure, our method's uncertainty map shows better correlation with the depth error maps. We also quantitatively evaluate uncertainty maps using negative log-likelihood following prior works Notice that in both examples, our method's certain (blue) regions mostly have low-depth errors (e.g., encircled parts in Fig. 5) because our formulation only assigns a region to be certain if it is well triangulated (Fig. 6). On the other hand, baselines struggle in these regions because they either use an empirical approximation from data or a Gaussian approximation of the ground truth posterior likelihood.

5.3 Ablation Study

We validate the choice of fIMLE as our probabilistic model by measuring the average precision (AP) [17] and area under the curve (AUC) [17] of predicted provenances ( t, d ) against ground truth provenances ( ˆ t, ˆ d ) for a set of densely sampled points in the scene bound. See supplementary for metric and ablation implementation details.

Deterministic v.s. Stochastic Field. We validate the importance of modeling per-point provenance as a stochastic field rather than a deterministic field. We model D θ with a deterministic field parameterized by a neural network. Table 4 shows the importance of modeling per-point provenance as a stochastic field. Since the provenances of a point are inherently multimodal, a deterministic field that only maps each x to a single provenance cannot capture this multimodality.

Choice of Probabilistic Model. We validate our choice of fIMLE [33] as our probabilistic model. We first compare with explicit probabilistic models that model the provenance field as a mixture of C Gaussian processes and a VAE-based model. Table 4 shows results for the Gaussian Mixture field with C = 2 , 5 and the VAE-based process. Although the performances for the Gaussian-based models improve as we increase C , they still suffer from expressivity because of their explicit density

Uncertain

Rendering

DepthError

FisherRF+3DGS

ProvNeRF+3DGS

NLL(↓)丨FisherRF

Ours

#710

5.19

-3.93

#758

3.53

-6.04

#781

2.73

0.068

3.82

-3.30

LOV

High

Certain

assumption. Similarly, the VAE-based model suffers from mode-collapse while our fIMLE enables capturing a more complex distribution with a learned transformation H θ .

Choice of Random Function Z . Lastly, we validate our latent stochastic field Z . We ablate our choice of Z with instead using a spatially invariant latent stochastic field Z ⋆ with Z ⋆ ( x ) = [ ε , x ] ∀ x . Here, ε is a Gaussian noise vector in R d . Table 4 shows the comparison between D θ obtained by transforming Z ( Ours ) and transforming Z ⋆ ( Spatial Inv. Z ). We see that using a spatially varying latent stochastic field further increases the expressivity of our model.

5.4 Preliminary Extension to 3D Gaussian Splatting

Because ProvNeRF is a post-hoc method that can model the provenance information for arbitrary novel view synthesis representations, we conduct a preliminary experiment that extends our provenance field modeling to 3D Gaussian Splatting [27]. Specifically, given a pre-trained Gaussian representation G , we model a provenance distribution for each splat using IMLE with a shared 6-layer MLP for H θ . The post-training takes around 30 minutes on a single A6000 Nvidia GPU. To show the usefulness of ProvNeRF applied to 3DGS, we use the methodology in Sec. 5.2 to estimate uncertainty maps and compare them with the predicted depth errors. Fig. 8 shows a qualitative comparison of our uncertainty map w.r.t. FishRF [25], a recent 3DGS uncertainty estimation baseline. Compared to their uncertainty map, ours shows more correlation to the depth error as highlighted by the boxed regions. Quantitatively we evaluate NLL on the three Scannet scenes shown on the right side of the same figure and show substantial improvements over FishRF. This improvement over existing literature suggests applying ProvNeRF to other representations such as 3DGS is promising. We leave further exploration of the method and applications as future works.

6 Conclusion, Limitation, & Future Works

We present ProvNeRF, a model that enhances the traditional NeRF representation by modeling provenance through an extension of IMLE for stochastic processes. ProvNeRF can be easily applied to any NeRF model to enrich its representation. We showcase the advantages of modeling per-point provenance in various downstream applications such as improving novel view synthesis and modeling the uncertainty of the capturing process.

We note that our work is not without limitations. Our ProvNeRF requires post-hoc optimization, which takes around 8 hours on SCADE, limiting its current usability for real-time or on-demand applications. However, the idea presented in our work is not specific to the model design and can be adapted to other representations. See Sec. 5.4 for preliminary adaption of ProvNeRF to 3DGS.

We also note that the hyperparameters to incorporate ProvNeRF are chosen for better performance, e.g. for the uncertainty and novel view synthesis applications, and in the future, it will be beneficial to explore a more adaptive approach in integrating provenance to different downstream applications.

Acknowledgement This work is supported by a Vannevar Bush Faculty Fellowship, ARL grant W911NF-21-2-0104, an Apple Scholars in AI/ML PhD Fellowship, and the Natural Sciences and Engineering Research Council of Canada (NSERC).

References

Michal Adamkiewicz, Timothy Chen, Adam Caccavale, Rachel Gardner, Preston Culbertson, Jeannette Bohg, and Mac Schwager. Vision-only robot navigation in a neural radiance world. IEEE Robotics and Automation Letters , 7(2):4606-4613, 2022.

Sameer Agarwal, Noah Snavely, Ian Simon, Steven M. Seitz, and Richard Szeliski. Building rome in a day. In ICCV , 2009.

Kalin Atanassov, Vikas Ramachandra, Sergio R. Goma, and Milivoje Aleksic. 3d image processing architecture for camera phones. In J. Angelo Beraldin, Geraldine S. Cheok, Michael B. McCarthy, Ulrich Neuschaefer-Rube, Ian E. McDowall, Margaret Dolinsky, and Atilla M. Baskurt, editors, ThreeDimensional Imaging, Interaction, and Measurement , volume 7864, page 786414. International Society for Optics and Photonics, SPIE, 2011.

Jonathan T. Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin-Brualla, and Pratul P. Srinivasan. Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields. In ICCV , 2021.

Jonathan T Barron, Ben Mildenhall, Dor Verbin, Pratul P Srinivasan, and Peter Hedman. Mip-nerf 360: Unbounded anti-aliased neural radiance fields. In CVPR , 2022.

Jonathan T Barron, Ben Mildenhall, Dor Verbin, Pratul P Srinivasan, and Peter Hedman. Zip-nerf: Anti-aliased grid-based neural radiance fields. ICCV , 2023.

Sai Bi, Zexiang Xu, Pratul Srinivasan, Ben Mildenhall, Kalyan Sunkavalli, Miloš Hašan, Yannick HoldGeoffroy, David Kriegman, and Ravi Ramamoorthi. Neural reflectance fields for appearance acquisition. arXiv preprint arXiv:2008.03824 , 2020.

Jiazhong Cen, Zanwei Zhou, Jiemin Fang, Chen Yang, Wei Shen, Lingxi Xie, Xiaopeng Zhang, and Qi Tian. Segment anything in 3d with nerfs. In NeurIPS , 2023.

Angel Chang, Angela Dai, Thomas Funkhouser, Maciej Halber, Matthias Niessner, Manolis Savva, Shuran Song, Andy Zeng, and Yinda Zhang. Matterport3d: Learning from rgb-d data in indoor environments. 3DV , 2017.

Di Chang, Aljaž Božiˇ c, Tong Zhang, Qingsong Yan, Yingcong Chen, Sabine Süsstrunk, and Matthias Nießner. Rc-mvsnet: Unsupervised multi-view stereo with neural rendering. In ECCV , 2022.

David Charatan, Sizhe Li, Andrea Tagliasacchi, and Vincent Sitzmann. pixelsplat: 3d gaussian splats from image pairs for scalable generalizable 3d reconstruction. In arXiv , 2023.

Anpei Chen, Zexiang Xu, Fuqiang Zhao, Xiaoshuai Zhang, Fanbo Xiang, Jingyi Yu, and Hao Su. Mvsnerf: Fast generalizable radiance field reconstruction from multi-view stereo. In CVPR , 2021.

Angela Dai, Angel X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. Scannet: Richly-annotated 3d reconstructions of indoor scenes. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 5828-5839, 2017.

Kangle Deng, Andrew Liu, Jun-Yan Zhu, and Deva Ramanan. Depth-supervised nerf: Fewer views and faster training for free. In CVPR , 2022.

Yilun Du, Cameron Smith, Ayush Tewari, and Vincent Sitzmann. Learning to render novel views from wide-baseline stereo pairs. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2023.

Jakob Engel, Vladlen Koltun, and Daniel Cremers. Direct sparse odometry. IEEE Transactions on Pattern Analysis and Machine Intelligence , 40(3):611-625, 2018.

Mark Everingham, Luc Van Gool, Christopher K. I. Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International Journal of Computer Vision , 88(2):303-338, 2010.

Yasutaka Furukawa and Carlos Hernández. Multi-View Stereo: A Tutorial . Now Foundations and Trends, 2015.

Lily Goli, Cody Reading, Silvia Sellán, Alec Jacobson, and Andrea Tagliasacchi. Bayes' rays: Uncertainty quantification in neural radiance fields. arXiv , 2023.

Richard Hartley and Andrew Zisserman. Multiple View Geometry in Computer Vision . Cambridge University Press, 2 edition, 2004.

Yi-Hua Huang, Yang-Tian Sun, Ziyi Yang, Xiaoyang Lyu, Yan-Pei Cao, and Xiaojuan Qi. Sc-gs: Sparsecontrolled gaussian splatting for editable dynamic scenes. arXiv preprint arXiv:2312.14937 , 2023.

Jeffrey Ichnowski*, Yahav Avigal*, Justin Kerr, and Ken Goldberg. Dex-nerf: Using a neural radiance field to grasp transparent objects. In Conference on Robot Learning (CoRL) , 2020.

Ajay Jain, Matthew Tancik, and Pieter Abbeel. Putting nerf on a diet: Semantically consistent few-shot view synthesis. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , pages 5885-5894, October 2021.

Wonbong Jang and Lourdes Agapito. Codenerf: Disentangled neural radiance fields for object categories. In ICCV , 2021.

Wen Jiang, Boshu Lei, and Kostas Daniilidis. Fisherrf: Active view selection and uncertainty quantification for radiance fields using fisher information, 2023.

Liren Jin, Xieyuanli Chen, Julius Rückin, and Marija Popovi´ c. Neu-nbv: Next best view planning using uncertainty estimation in image-based neural rendering. In IROS , 2023.

Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics , 42(4), July 2023.

Justin Kerr, Chung Min Kim, Ken Goldberg, Angjoo Kanazawa, and Matthew Tancik. Lerf: Language embedded radiance fields. In ICCV , 2023.

Arno Knapitsch, Jaesik Park, Qian-Yi Zhou, and Vladlen Koltun. Tanks and temples: Benchmarking large-scale scene reconstruction. ACM Transactions on Graphics , 36(4), 2017.

Sosuke Kobayashi, Eiichi Matsumoto, and Vincent Sitzmann. Decomposing nerf for editing via feature field distillation. In NeurIPS , 2022.

Georgios Kopanas and George Drettakis. Improving nerf quality by progressive camera placement for unrestricted navigation in complex environments. In Vision, Modeling, and Visualization , 2023.

K.N. Kutulakos and S.M. Seitz. A theory of shape by space carving. In ICCV , 1999.

Ke Li and Jitendra Malik. Implicit maximum likelihood estimation, 2018.

Zhengqi Li, Simon Niklaus, Noah Snavely, and Oliver Wang. Neural scene flow fields for space-time view synthesis of dynamic scenes. In CVPR , 2021.

Zhengqi Li, Qianqian Wang, Forrester Cole, Richard Tucker, and Noah Snavely. Dynibar: Neural dynamic image-based rendering. In CVPR , 2023.

Jianlin Liu, Qiang Nie, Yong Liu, and Chengjie Wang. Nerf-loc: Visual localization with conditional neural radiance field. In ICRA , 2023.

Sheng Liu, Xiaohan Nie, and Raffay Hamid. Depth-guided sparse structure-from-motion for movies and tv shows. In CVPR , 2022.

Dominic Maggio, Marcus Abate, Jingnan Shi, Courtney Mario, and Luca Carlone. Loc-nerf: Monte carlo localization using neural radiance fields, 2022.

Ricardo Martin-Brualla, Noha Radwan, Mehdi S. M. Sajjadi, Jonathan T. Barron, Alexey Dosovitskiy, and Daniel Duckworth. Nerf in the wild: Neural radiance fields for unconstrained photo collections. In CVPR , 2021.

Hidenobu Matsuki, Raluca Scona, Jan Czarnowski, and Andrew J. Davison. Codemapping: Real-time dense mapping for sparse slam using compact scene representations. IEEE Robotics and Automation Letters , 2021.

Nelson Max. Optical models for direct volume rendering. IEEE Transactions on Visualization and Computer Graphics , 1(2):99-108, 1995.

Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In ECCV , 2020.

Michael Niemeyer, Jonathan T. Barron, Ben Mildenhall, Mehdi S. M. Sajjadi, Andreas Geiger, and Noha Radwan. Regnerf: Regularizing neural radiance fields for view synthesis from sparse inputs. In CVPR , 2022.

M. Okutomi and T. Kanade. A multiple-baseline stereo. IEEE Transactions on Pattern Analysis and Machine Intelligence , 15(4):353-363, 1993.

Xuran Pan, Zihang Lai, Shiji Song, and Gao Huang. Activenerf: Learning where to see with uncertainty estimation. In ECCV , 2022.

Keunhong Park, Utkarsh Sinha, Jonathan T. Barron, Sofien Bouaziz, Dan B Goldman, Steven M. Seitz, and Ricardo Martin-Brualla. Nerfies: Deformable neural radiance fields. In ICCV , 2021.

Satyarth Praveen. Efficient depth estimation using sparse stereo-vision with other perception techniques. In Sudhakar Radhakrishnan and Muhammad Sarfraz, editors, Coding Theory , chapter 7. IntechOpen, Rijeka, 2019.

Yunlong Ran, Jing Zeng, Shibo He, Jiming Chen, Lincheng Li, Yingke Chen, Gim Hee Lee, and Qi Ye. Neurar: Neural uncertainty for autonomous 3d reconstruction with implicit neural representations. IEEE Robotics and Automation Letters , 2022.

Barbara Roessle, Jonathan T. Barron, Ben Mildenhall, Pratul P. Srinivasan, and Matthias Nießner. Dense depth priors for neural radiance fields from sparse input views. In CVPR , 2022.

S.M. Seitz and C.R. Dyer. Photorealistic scene reconstruction by voxel coloring. In CVPR , 1997.

J. Shen, A. Ruiz, A. Agudo, and F. Moreno-Noguer. Stochastic neural radiance fields: Quantifying uncertainty in implicit 3d representations. In 3DV , 2021.

Jianxiong Shen, Antonio Agudo, Francesc Moreno-Noguer, and Adria Ruiz. Conditional-flow nerf: Accurate 3d modelling with reliable uncertainty quantification. In ECCV , 2022.

Noah Snavely, Steven M. Seitz, and Richard Szeliski. Photo tourism: Exploring photo collections in 3d. ACM Trans. Graph. , 2006.

Jiuhn Song, Seonghoon Park, Honggyu An, Seokju Cho, Min-Seop Kwak, Sungjin Cho, and Seungryong Kim. Därf: Boosting radiance fields from sparse inputs with monocular depth adaptation. In NeurIPS , 2023.

Niko Sünderhauf, Jad Abou-Chakra, and Dimity Miller. Density-aware nerf ensembles: Quantifying predictive uncertainty in neural radiance fields. In ICRA , 2022.

Matthew Tancik, Vincent Casser, Xinchen Yan, Sabeek Pradhan, Ben Mildenhall, Pratul P Srinivasan, Jonathan T Barron, and Henrik Kretzschmar. Block-nerf: Scalable large scene neural view synthesis. In CVPR , 2022.

Mikaela Angelina Uy, Ricardo Martin-Brualla, Leonidas Guibas, and Ke Li. Scade: Nerfs from space carving with ambiguity-aware depth estimates. In CVPR , 2023.

Mikaela Angelina Uy, George Kiyohiro Nakayama, Guandao Yang, Rahul Krishna Thomas, Leonidas Guibas, and Ke Li. Nerf revisited: Fixing quadrature instability in volume rendering. In NeurIPS , 2023.

Dor Verbin, Peter Hedman, Ben Mildenhall, Todd Zickler, Jonathan T. Barron, and Pratul P. Srinivasan. Ref-nerf: Structured view-dependent appearance for neural radiance fields. In CVPR , 2022.

Qianqian Wang, Zhicheng Wang, Kyle Genova, Pratul Srinivasan, Howard Zhou, Jonathan T. Barron, Ricardo Martin-Brualla, Noah Snavely, and Thomas Funkhouser. Ibrnet: Learning multi-view image-based rendering. In CVPR , 2021.

Yi Wei, Shaohui Liu, Yongming Rao, Wang Zhao, Jiwen Lu, and Jie Zhou. Nerfingmvs: Guided optimization of neural radiance fields for indoor multi-view stereo. In ICCV , 2021.

Yuanbo Xiangli, Linning Xu, Xingang Pan, Nanxuan Zhao, Anyi Rao, Christian Theobalt, Bo Dai, and Dahua Lin. Bungeenerf: Progressive neural radiance field for extreme multi-scale scene rendering. In ECCV , 2022.

Tianhan Xu and Tatsuya Harada. Deforming radiance fields with cages. In ECCV , 2022.

Jiawei Yang, Marco Pavone, and Yue Wang. Freenerf: Improving few-shot neural rendering with free frequency regularization. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR) , 2023.

Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa. pixelnerf: Neural radiance fields from one or few images. In CVPR , 2021.

Zehao Yu, Songyou Peng, Michael Niemeyer, Torsten Sattler, and Andreas Geiger. Monosdf: Exploring monocular geometric cues for neural implicit surface reconstruction. In NeurIPS , 2022.

Yu-Jie Yuan, Yang-Tian Sun, Yu-Kun Lai, Yuewen Ma, Rongfei Jia, and Lin Gao. Nerf-editing: Geometry editing of neural radiance fields. In CVPR , 2022.

Xiaoshuai Zhang, Abhijit Kundu, Thomas Funkhouser, Leonidas Guibas, Hao Su, and Kyle Genova. Nerflets: Local radiance fields for efficient structure-aware 3d scene representation from 2d supervision. In CVPR , 2023.

Shuaifeng Zhi, Tristan Laidlow, Stefan Leutenegger, and Andrew Davison. In-place scene labelling and understanding with implicit scene representation. In ICCV , 2021.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discuss the limitations of our method in the supplementary

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: we provide the full derivation of our fIMLE objective in the supplementary.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We provide the training details, hyperparameters, network architectures in the supplementary.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We will provide reproducible code upon the paper's acceptance.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Guidelines:

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We address the potential societal impact on climate change in the supplementary materials

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nPaper presents a method for co-training an inference network that learns where points in a NeRF were “seen from” (ie. provenance) in the training data. This allows for quantification of uncertainty of the reconstructed geometry in terms of triangulation (location) and depth error. Mathematically frames these these provenances within a stochastic process, extending implicit maximum likelyhood estimation (IMLE) to the functional domain over infinite (euclidian) sets. Presensts a loss term for incorporation into NeRF training and demonstrate superior novel view synthesis with it, and lower triangulation uncertainty on two standard datasets.\n\nSTRENGTHS:\nWhile I'm not convinced the \"Methods\" section is as crystal clear as it could be, and the problem domain widely applicable, the paper appears to present a novel contribution in the extension of IMLE to infinite sets and functional samples that should be of likely interest to the NeurIPS community.  The presentation is mathematically rigorous, though somewhat opaque, with extensive derivations in the supplement. \n\nDemonstrates improvements to novel view synthesis of existing NeRF scenes through co-training with their additional loss term and  synthesis term, achieving improvements in reconstruction metrics. Also demonstrates quantitative SotA results over competing methods on triangulation uncertainty.\n\nWEAKNESSES:\nIntroduction still leaves me asking “Why is this an important problem?”. The method only addresses triangulation uncertainty, ignoring other forms of uncertainty like transients, perhaps limiting applicability. \n\nOverly mathematical presentation. Is this being presented in the absolute simplest way possible? Confusion over just exactly which the stochastic process is being discussed. Should clearly state the use-case (NeRF training add-on, or co-training objective?) up-front.\n\nQUESTIONS:\nThe exposition of section 4.1 is unclear and took me a while to understand. Primarily, I understand that for each point in space $x$ there is a distribution of provenances, But I could imagine two stochastic processes: 1) where each point is sampled from the distribution for a fixed location, or 2) where each point becomes the provenance distribution for the next point drawn. This could be made clearer. Perhaps a diagram would help. (Also note having figure S1 in the main text would have gone a long well to help me understand the method better).\n\n#25 \"modeling for each point, the locations where it is likely visible.\". Do you actually mean \"likely visible in the training dataset\", or \"likely visible in the scene\"? I believe you mean the former, yet this critical point was not clear to me until reading further into the paper. This seems like a critical distinction to make. ie. I could have a single view of a scene, yet properly infer that a point should be visible from many directions, even if not present in the training data.\n\n#126 \"forms a stochastic process, $\\mathcal{D}_\\theta$ indexed by coordinates $x \\in \\mathbb{R}^3$\" Shouldn't this be more correctly defined as a \"stochastic field\", given it's indexing by Euclidean space. In general this is not the \"stochastic process\" most readers would usually be accustomed to, as one indexed by the natural positive numbers. Also, you are defining samples as functions, not as $\\mathbb{R}^3$ positions, correct? Given this is the core of your contribution, extra clarity for readers would be appreciated for this.\n\n#136, since you call $\\mathbf{d}$ a \"direction\" it should be a normalized direction. Therefore the tuple should be in $R_+ \\times \\mathbb{S}^2$, right? ? If $\\mathbf{d}$ is not normalized, I'm not sure I understand why (and should be called a \"vector\", not a \"direction\"). This should be a direction pointing towards the camera origin?\n\n#138 I would define (5) before (4) for clarity. Also, presumably $\\mathbf{o}_i$ is the camera origin? Though I don't believe it's been defined yet.\n\n#211 Where is the actual architecture used for $\\mathbf{H}_\\theta$ described? This implementation detail seems critical to understanding the method, and how it might be helping NeRF training.\n\nLIMITATIONS:\nAuthors leave discussion of limitations to a short paragraph in the supplemental. I’d prefer to see this in the main-text. Main current limitation is a long post-hoc training process. I’d also like to have seen a longer discussion on applicability of the method.\n\nAuthors sufficiently discuss the societal impact of their method.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper describes a method for explicitly modeling the visibility of a point in space in a NeRF model. This is done though modeling provenance which is the space of points where the given point is likely visible. The motivation is that modeling visibility enables the underlying NeRF model to better utilize triangulation information from the training images. The proposed method introduces a neural network to model the provenance function which allows for sampling points visible point estimates from a given 3D position. This secondary network can be applied to any base NeRF model. When trained with the provenance loss, the resulting model performs better in the task of novel view synthesis across a variety of scenes and metrics, and allows for explicit modeling of uncertainty in the reconstructed scene geometry.\n\nSTRENGTHS:\n+ The motivation for the problem is interesting. If you have a way to approximate where a point is visible given the input data, that does provide a lot of insight about the quality of the input data for reconstruction of view synthesis. \n\n+ The results shown are good. The overall metric while better than other approaches are just a slight bump, but there are several examples shown where some of the common artifacts present in indoor NeRFs are mitigated if not completely removed using this method. That improved visual quality on top of the benefits of uncertainty modeling make this quite compelling.\n\n+ The ablation in Table 2 shows that the naive ideas that most people would have tried are worse than this, so I think the more obvious baselines are covered in comparison. Similarly Table 4 is a good ablation over several methods that I think would be the go to baseline for an idea like this. Overall the evaluation is very thorough. \n\n+ The main paper + supplemental is an immense amount of information and evaluation for a single paper. Clearly a lot of effort was put into describing and validating the method.\n\nWEAKNESSES:\n- Overall I found the presentation of the method to be very confusing. There is a lot of technical overview in 4.1-4.3 that I think over complicates what is actually happening. I had to reread these sections several times while making passes through the full paper to really get a grip on what is happening. The architecture diagram in Figure S1 makes it much clearer. I think the extremely technical discussion could be reduced and moved to the supplemental while a more direct description of what is actually being done like in the supplemental should be focused on more.\n\n- I don't follow why norm(d)>=0.7 has any berrying on the confidence of that provenance sample. The output provenance sample D(x) = (t,d) where the point y where x is estimated to be visible is y=x-td. So why is the length of the direction vector a signal for uncertainty? \n\n- The visualizations of the provenances for points don't align with what I expect. Why are the vectors all roughly the same length? Shouldn't it be more varied? Similarly, I don't think the definition of how to recover the point y where x is visible is correct based on the description and visualization. If y = x-td, then the output direction vector would be pointing away from points of visibility, not towards. \n\n- To reiterate my first point, the major weakness of this paper is the somewhat overwhelming amount of technical detail for what boils down to a pretty simple idea. Maybe if the details are reordered so that the simple explanation comes first followed by the more rigorous definition the paper would be easier to follow. After several reads I can follow the early sections, but they are still not extremely clear. I would recommend simplification if possible to make the core contribution as clear as possible and fill in the details where necessary instead of the current presentation. Though I admit that the technical groundwork is necessary so it is a difficult balance. But a clear example of what can be simplified is section 4.3. There is the initial equation 7 discussion, followed by the logic for simplifying to equation 8, which eventually reduces to equation 9 which is what I think most people would have assumed would be the natural conclusion. It's not to say that this presentation of the principles from equation 7 to equation 9 is not important if useful, but it is a bit heavy handed in my opinion.\n\nQUESTIONS:\nEnd of line 128,  \"we\" should be capitalized.\n\nThis is not necessarily a weakness, but it would be good to clarify. Are the 3D models being shown in the visualization based on evaluating the provenance on the ground truth mesh from depth fusion, or is that somehow extracted from the NeRF itself. They are shockingly clean to be extracted from a NeRF but it's not clear to me from the text if that is actually what's happening.\n\nLine 168 \" We define a latent function sample Z ~ Z to be the concatenation of a random linear transformation of x and x itself.\" I don't think concatenation is the right word here based on equation 6. I think this sentence is phrased poorly or I'm missing something important.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses gaps in existing Neural Radiance Fields research by modeling the provenance of each point as a stochastic process and enhancing triangulation quality through an extended Implicit Maximum Likelihood Estimation (IMLE) to functional space, resulting in improved novel view synthesis and uncertainty estimation under sparse, unconstrained view conditions. Experimental results demonstrate the effectiveness of the proposed method.\n\nSTRENGTHS:\n1. The idea is interesting, especially the defined problem in this paper makes a lot of sense to me. ProvNeRF enhances Neural Radiance Fields by integrating per-point provenance information during training, which enriches the model with critical insights on triangulation quality. This integration leads to a combination of traditional 3D vision-related fields, e.g. triangulation and uncertainty, to novel view synthesis, particularly under challenging sparse and unconstrained viewing conditions.\n2. The writing is very clear and easy to follow.\n\nWEAKNESSES:\n1. My main concern is about the evaluation of the proposed idea. According to the paper, modeling the provenance for each point could potentially improve NeRF's accuracy in predicting the positions of points in space. However, there are few direct comparative experiments on depth/position, with only Fig. 4 showing some reflection. The depth error does not seem to significantly differ from other methods. This makes the evaluation less convincing from my view.\n2. From Table 1,2, I wouldn't say that the result is significantly improved. And, the visualization in Fig.4, the difference between the proposed one and Bayes' is not notable. These make me a little bit concerned about the significance of the proposed method as it requires an 8-hour post-optimization.\n\nminor \n1. The description of the images in Fig. 2 is reversed, with left and right sides swapped.\n\nQUESTIONS:\n1. Does the proposed method solve the uncertainty problem? Does the uncertainty problem really matter in the NeRF-based method? If so, why does the proposed method have inferior performance compared with other sOTA methods in evaluation, e.g. table 1?\n2. Can this method be applied to 3DGS or INGP-based methods? How much time it will cost to add the proposed framework in the post-optimiazation?\n\nLIMITATIONS:\nThe authors demonstrate shortcomings concerning the optimization duration and mention a solution to address these deficiencies in the appendix section.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a way to jointly learn/model provenance during NeRF training, where provenance is defined as locations where a 3D point is likely visible. This design is motivated by the classic idea of modeling triangulation quality, To implement this in NeRF, this paper extends implicit maximum likelihood estimation (IMLE) to functional space with an optimizable objective. Modeling provenance is beneficial to NeRF final results, and also enables a new way of uncertainty estimation.\n\nSTRENGTHS:\n- The introduced idea of Provenance modeling is sound and motivated by classic idea in stereo matching. \n- The ability of estimating uncertainty is very important, and it's a free side-product of the introduced Provenance modeling.\n- The idea of using Implicit Maximum Likelihood Estimation in functional space is indeed a good solution for represents probabilistic distribution as a set of samples, and also for stochastic processes.\n- This paper also builds up a system of theory which is complete and reasonable. This theoretical framework will inspire many follow-up work in this field. \n- The appendix is very informative and discussed things like extra results, important alternative designs, key derivations and ablation studies. I found it to be a good complement of the main paper. \n- The findings of this method is helpful to sparse, unconstrained view setting is very encouraging.\n\nWEAKNESSES:\nI find a few things can be improved as listed below:\n- Some motivations are less clear or need clarification, specifically:\n  - L36 \"we propose to model the provenance as the samples from a probability distribution, where a location y is assigned with a large likelihood if and only if x is likely to be visible from y\": What's the alternative ways of modeling this probability distribution rather than using samples? Is it possible to learn a continuous representation rather than use discrete samples?\n  - For sample-based generative model, what are the alternatives to implicit maximum likelihood estimation (IMLE)? Why is this specific approach is taken? A discussion could be very useful here.\n\n- The visual results are hard to interpret. Denser and more informative caption can be considered.\n  - Why depth error is always shown with uncertainty. What's the connection and how can we interpret the depth? \n  - Compared to the baselines such as Bayes' Rays, what are considered as improvement from the uncertainty map? Is it more semantic? Or is it more correlated with depth, means near objects tends to have lower uncertainty. \n  - From results it appears some floaters/fuzziness are addressed but not totally. I'm wondering what's the reason and how can we address the remaining artifacts.\n  \n- Since there is no GT for uncertainty, what's the best way to evaluate it? In Tab.3 NLL is used, is this the best way? \n\n- L135, provenances is parameterized as a distance-direction tuple. What's the other options here? Can we directly model it as a 3D vector? Or using other parameterization of 3D vector?\n\n- Sec.2 first paragraph \"NeRFs and their Extensions\" missed dicussion of recent explicit representation such as 3DGS. Also, it's unclear whether the proposed solution can generalize to 3DGS? In theory it should work but we don't know in reality. This extra backbone experiment can make the results and theory more convincing. \n\n- This paper claims the proposed method helps  sparse, unconstrained view setting. Following this statement, the experiments should use a sparse setting such as few-shot NeRF. Also, I'm wondering why scannet and tanks and temple are picked as the testing benchmark but not other popular NeRF dataset (indoor, outdoor, synthetic, object-centric, etc).\n\nQUESTIONS:\nAfter reading, couple of questions remain:  \n- In Fig.2, how does camera baseline distance, occlusions/visibility, and stereo range errors relate to Provenance? The connection is less clear.\n- Is it possible to visualize the learned confident provenances together with the camera poses in a way that can better illustrate the correctness/soundness of the learned provenances?\n- For results in Tab.1, is `ours` built upon NeRF? If so, is the only difference between `ours` and `NeRF` is the extra loss?\n- In Eq.9 there are two expectations. So in practice, two samplings are needed. I'm wondering how are D, x, and (t,d) sampled in practice. \n- How important are L_provNVS and L_provNeRF individually? Which one is more important and why both are needed?\n\nLIMITATIONS:\nLimitations are discussed in the appendix where the authors are upfront about one important limitation of the running time. \nSocietal impact is also discussed in appendix.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-18"}