Variational Multi-scale Representation for Estimating Uncertainty in 3D Gaussian Splatting
Authors: Ruiqi Li
Yiu-ming Cheung
Venue: NeurIPS 2024 poster (2024)
Keywords: Neural Rendering
Uncertainty Quantification
Primary area: machine_vision
TL;DR: We quantify the uncertainty in 3D Gaussian Splatting by deviating Gaussians to construction model space samples and learn with variational inference. .

Abstract: Recently, 3D Gaussian Splatting (3DGS) has become popular in reconstructing dense 3D representations of appearance and geometry. However, the learning pipeline in 3DGS inherently lacks the ability to quantify uncertainty, which is an important factor in applications like robotics mapping and navigation. In this paper, we propose an uncertainty estimation method built upon the Bayesian inference framework. Specifically, we propose a method to build variational multi-scale 3D Gaussians, where we leverage explicit scale information in 3DGS parameters to construct diversified parameter space samples. We develop an offset table technique to draw local multi-scale samples efficiently by offsetting selected attributes and sharing other base attributes. Then, the offset table is learned by variational inference with multi-scale prior. The learned offset posterior can quantify the uncertainty of each individual Gaussian component, and be used in the forward pass to infer the predictive uncertainty. Extensive experimental results on various benchmark datasets show that the proposed method provides well-aligned calibration performance on estimated uncertainty and better rendering quality compared with the previous methods that enable uncertainty quantification with view synthesis. Besides, by leveraging the model parameter uncertainty estimated by our method, we can remove noisy Gaussians automatically, thereby obtaining a high-fidelity part of the reconstructed scene, which is of great help in improving the visual quality.

Variational Multi-scale Representation for Estimating Uncertainty in 3D Gaussian Splatting
Ruiqi Li, Yiu-ming Cheung"
Department of Computer Science, Hong Kong Baptist University {csrqli, ymc))ccomp hkbu.edu.hi edu.hk
Abstract
Recently, 3D Gaussian Splatting (3DGS) has become popular in reconstructing dense 3D representations of appearance and geometry. However, the learning pipeline in 3DGS inherently lacks the ability quantify uncertainty, which is an important factor in applications like robotics mapping and navigation. In this paper, propose uncertainty estimation method built upon the Bayesian inference framework. Specifically, we propose a method to build variational multi-scale 3D Gaussians, where leverage explicit scale information in 3DGS parameters to construct diversified parameter space samples. We develop an offset table technique draw local multi-scale samples efficiently by offsetting selected attributes and sharing other base attributes. Then, the offset table is learned by variational inference with multi-scale prior. The learned offset posterior can quantify the uncertainty of each individual Gaussian component, and be used in the forward pass infer the predictive uncertainty. Extensive experimental results on various benchmark datasets show that the proposed method provides well-aligned calibration performance on estimated uncertainty and better rendering quality compared with the previous methods that enable uncertainty quantification with view synthesis. Besides, by leveraging the model parameter uncertainty estimated by our method, we can remove noisy Gaussians automatically, thereby obtaining a high- fidelity part of the reconstructed scene, which of great help in improving the visual quality.
Introduction
The radiance field methods [ for view synthesis have received increasing attention in the past few years due their capability of achieving photorealistic results. Among them, the recently proposed 3D Gaussian Splatting (3DGS) algorithm (2) has pushed the boundary of real-time view synthesis with prominent quality and efficiency. However, a major functionality deficiency of 3DGS is that it cannot provide uncertainty information regarding the reconstructed model and predictions. Such uncertainty information would be useful in removing the noisy components in the model and providing confidence maps to assess the quality view synthesis results, which is important in applications such as autonomous driving simulation and robotics navigation.
Previous works in view synthesis made attempts to quantify the uncertainty in Neural Radiance Field (NeRF) models via ensemble, variational inference or Laplace's 's approximation methods [ 4 S Although these works demonstrate the ability to quantify the uncertainty with NeRF models, they cannot be directly applied 3DGS models, due the intrinsic difference between the implicit NeRF representation and explicit 3DGS. Furthermore, some prior works deal with the uncertainty in learning models with methods such as Monte-Carke dropout 77, Deep Ensemble
g and Subnetwork
"orresponding author is Yiu-ming Cheung (wwe@comm...hbu..cuuuuuuoo Coode available at https:::/// thub, wttptt..c.m//srrr////aattttaaaa.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Full Scene
Centering Object
$11MB,100%H
255MB, 50%
102MB, 20%
51.1MB,
Figure The results of cleaning up an unbounded scene reconstructed with 3DGS using our uncertainty estimation. We remove the Gaussians with large parameter uncertainty, the majority of which are under-reconstruucte background. The desk at the center of the scene remains complete after removing 90% of the Gaussians.
9 A major characteristic of these methods is that they can be seen the approximation of Bayesian Inference, which estimates the distribution of posterior p(0)D) and prediction p(y]0, æ), instead of point estimation.
The methods mentioned above can be seen as inferring the posterior distribution using model space samples. As mentioned previous works [8] IO II]. the key objectives for good model space samples are diversity and efficiency. To improve the quality of uncertainty estimation by increasing the diversity, the generated model space samples should explore the potential cases of true posterior as much as possible, which was not achieved in previous work on NeRF uncertainty estimation. Furthermore, although naive methods like ensemble provide good approximations, they require large storage and the computational cost which increase linearly with the number of samples. Thus, on top of diversity, efficiency is another important aspect we focus on when estimating the uncertainty for 3DGS, which means that we should use as few samples of parameters possible.
In this paper, we design uncertainty estimation method for 3DGS by constructing diversified parameter space samples efficiently. Inspired by the Level-of-Detall (LoD) technique used in computer graphies [12 [3] and multi-scale representation widely used in computer vision [5), we propose to investigate the potential of scale information in fitting the scene representation with diversified samples. Specifically, we design multi-scale variational inference method for 3DGS, which increases the diversity of parameter samples by enforcing them to model local spatial areas with multiple scales. To reduce the number of extra parameters, we design a mechanism that spawns finer multi-scale Gaussians from the base Gaussian, which are heavily involved in the rendering process. Instead of creating actual new Gaussians, we maintain an offset table parameterizing only a subset of attributes and share the remainder with base Gaussian. This can further reduce the number of parameter samples by maintaining only the attributes that contribute our multi-scale representation.
Extensive experiments demonstrate the remarkable performance of our method on uncertainty estimation without affecting the rendering quality. We also show that our method provides both accurate posterior and predictive uncertainty estimation. The posterior uncertainty can be utilized directly to remove noisy Gaussians, thanks the explicit benefit of the 3DGS method. The predictive uncertainty can serve as a confidence map to interpret the synthesized novel views.
The main contributions of work summarized follows:
We propose a multi-scale variational inference framework for uncertainty-aware view syn- thesis with the 3DD Gaussian Splatting algorithm.
We develop spawning strategy to create multi-scale representations for 3DGS, and increase the sample diversity and inference efficiency by maintaining an offset table and sharing parameters.

---

Inference
Inference
Inference
t
Sampling Offset Posterior
((0)
Sampling Posterior
Sampling Models
or
Model
Model 2
Model 3
(c) Multi-scale Variational Inference
(a) Laplace's s Approximation
(b) Ensemble
The comparison between our multi-scale variational inference and other methods. (a) Laplace's s Approximation fits posterior with normal distribution where the mean equals maximum posteriori solution AP and precision equals fisher information {() (b) The ensemble method
Figure 2:
learns multiple models simultaneously to form the model space samples. (c) Our method builds
multi-scale representation of the scene, where inference is done by sampling the offset distribution and forming finer Gaussians.
We evaluate the accuracy of uncertainty estimation various benchmarks, and demonstrate the application of the parameter uncertainty in cleaning up noisy components in the scene.
2
Related Work
2.1 Uncertainty Quantification
Providing uncertainty together with predictions of learning models is of great such interpreting the model output [16] [7] or providing guidance active data collection [18] [9]. This goal can
be achieved by Bayesian learning, which characterizes the predictive distribution and posterior
distribution of model parameters with theoretical foundations (0) One can use Hamiltonian
Monte Carlo sampling for Bayesian learning in neural networks (1), which guarantees asymptotic performance. Previous works proposed other approximation methods such as Laplace's approximation [22] These methods consider the correlations between parameters, and are known for not requiring further assumption while reducing computations.
More recent works borrow from regularization techniques such as the dropout method approxi- mated Bayesian inference, such as Monte Carlo DropOut (MCDO) 77]. Concrete Dropout 23 and
Variational Dropout [24]. Another line of work builds model ensembles from various subsets of data
g 26 to infer the uncertainty. Some methods also introduce network modulation form model ensemble, among which Batchensemble [27] learns multiple low-rank weight matrices. Turkoglu et al. (28) introduced a set linear modulation parameters. Other works focus extra network components, such as Variational AutoEncoders (VAEs) methods
8 16 hyperparameters 25 or multiple subnetworks
22 or auxiliary network 30
The commonality of the above methods is that they introduce randomness into the model and draw model space samples. Furthermore, the importance the diversity of model space samples in approximated Bayesian learning is demonstrated in [T] B1]. Unlike the implicit characteristics of neural networks, the parameters of 3DGS have physical meanings that we can handle explicitly.
For algorithms in the multi-view geometry in computer vision, the uncertainty of camera pose can be estimated from the covariance matrix the transformation matrix using Monte Carlo methods [32] In the Simultaneous Localization and Mapping (SLAM) system, the uncertainty can be estimated from Kalman filter 33 Our method can be aggregated dense SL.AM system to provide uncertainty information for optimizing camera pose together with the map.
2.2 View Synthesis and Radiance Field
Recently, the radiance field has become popular 3D vision, which achieves great success in gener- ating photo-realistic images of novel view directions from set of calibrated images. Early research

---

Spawn
Inference
Base Gaussian
Finer Gaussians
Select Gaussians
Rendered View
Position Offset
Opacity Offset
Scale Offset
3D Gaussians
a b
Posteriar
Multi-scale
Offset Table
Uncertainty Map
Figure 3: The pipeline of our variational multi-scale representation. We spawn base Gaussians, which are the major components in the scene, into multi-scale finer Gaussians. We learn an offset table to perform the spawn operation by offsetting a subset of attributes. The offset table is learned with variational inference with multi-scale prior. The predictive and parameter uncertainty can be inferred from the variational parameters stored in the table.
developed light field [34 and multiplane images based 33 methods for this task. Building upon pre- vious research on implicit representations [36 37 38 39], NeRF and its variants achieve a successful framework combining volume rendering and positional encoding synthesize photorealistic novel views
[2] explicitly represents the 3D scene with points associated with the Gaussian function. Some works explore potential solutions the anti-alis aliasing rendering problem, for example, Turki et al.
40 41.
The recently proposed 3DGS
multi-resolution feature grid, and Yan al. 43 designed multi-scale Gaussian with level-of-detail. In work, propose variational multi-scale representation for 3DGS by spreading only local deviations to form multi-scale samples of model parameters.
22 developed
Some previous works focus on uncertainty estimation for NeRF models. Stochastic NeRF ad- dresses the uncertainty estimation in NeRF models via standard variational inference technique. NeRF Ensembles leverages neural network ensemble technique to quantify the predictive uncer- tainty. CF-NeRF models the predictive distribution via learning conditional normalizing flow. ActiveRMAP 44 focuses on the active reconstruction task, and models the predictive uncertainty via entropy of ray density distribution. NeurAR and ActiveNeRF [45 19] adopt the neural network to output uncertainty values for each pixel and learning with Negative Log-Likelihood loss. ProbNeRF
[46 designs learned variational autoencoder that generate 3D models from 2D images and uses the Monte Carlo method at inference to provide predictive uncertainty. Bayes' Ray 3 performs a post-hoc Laplace's approximation for NeRF quantify the uncertainty via applying perturbations spatial points. Compared to the NeRF models, the uncertainty estimation of 3DGS is less discussed. FisherRF [47] designs uncertainty quantification method for 3DGS and applies which adopts Laplace's ' approximation to compute the uncertainty from Fisher Information. However, the approximation of posterior uncertainty needs extensive computations of gradient aggregating over the whole training views. In our work, we leverage variational inference to approximate the posterior distribution parameters.
active learning,
3 Proposed Method
3.1 Preliminaries: Uncertainty Quantification for 3DGS
The goal of the view synthesis problem is to generate images from any viewpoint given set of calibrated input images. NeRF [ proposes to solve this problem by representing the geometry and appearance of the scene using learned radiance field f(x, d) - (c, σ),, where x and d represent spatial point and view direction, and a represent the corresponding color and opacity of that point viewing from d. Based on this radiance field representation, 3DGS algorithm [2] further proposes

---

that the values in the field can be explicitly stored in set of ellipsoids parameterized by the Gaussian function. The radiance value of each point in the scene x queried from its adjacent ellipsoid from the Gaussian function:
((p) exp (-2(xx (x--)) p) xx-
where and Σ are the center and covariance matrix of the ellipsoid. The covariance is further decomposed rotation R and scale S by Σ RSS R" We referred to each ellipsoid Gaussian G. In the rendering process, each Gaussian component is projected to the image space and transformed into its 2D projections. After that, they are accumulated via alpha blending to form pixel values in the rendered image:
N C (n
n-1
II (1 - ax)])((())))
Note that the rendering process is deterministic. For each Gaussian G. its learnable parameters composed of position, color, density and covariance: = {p. a. S, RJ.. The scale S and rotation R are learned respectively and form the covariance matrix Σ in rendering.
Standard 3DGS algorithm applies a non- aan-Bayesian approach to train the model with L1 loss function, which can be seen as Maximum Likelihood Estimation (MLE) with the error following Laplace distribution [20]. However, this only performs point estimation which lacks the ability to quantify predictive uncertainty and provide confidence information.
Previous methods in Bayesian learning provide tools like variational inference and ensemble methods for estimating the predictive uncertainty of models. For example, with ensemble methods, we can train multiple 3DGS models with different data subsets, random seeds or hyperparameters, and compute the variance of their output as the predictive uncertainty. With variational inference methods, the model posterior p(0 is approximated by a tractable variational distribution g0), where 'D is the training data. Then, the likelihood of pixel color p(c)x, 0) at pixel x together with the discrepancy between pl ( D)) and q( (()) is optimized.
3.2 Local Multi-scale 3D Gaussian
As discussed in previous works 8 11. the effectiveness of approximation methods in Bayesian inference highly depends on the diversity of model parameter space samples. With the explicit attributes of 3DGS parameters, we can manipulate the diversity of parameter samples and explore extensively the model parameter space. With more formations of the model representing the scene explored during learning. we can achieve better approximations inferring the parameter posterior distribution. In the following, we will introduce a practical approach to building representation with such diversification ability.
Local Multi-scale Representation. Different from NeRF-based scene representation where the model parameters are the neural network parameters with explicit meaning, 3D Gaussian models the local area of the spatial scene with attributes describing the geometry and appearance. Therefore, we can perform heterogeneous operations for Gaussian attributes by diversifying the scale of the local Gaussian to increase the performance in approximations. In our local multi-scale 3D Gaussian, we propose to learn scene representations with multiple Gaussian scales to represent a local spatial area. Specifically, we first select Gaussians that contribute more to the representation of the scene, and draw new multi-scale samples that are attached to these Gaussians. The parameters of the new Gaussians shared or learned with variational inference. Through this strategy, scale variance and achieve diversified model space samples our uncertainty estimation.
able to control the
Spawn from Base Gaussians. The pipeline for spawning local multi-scale 3D Gaussians is illustrated in Figure For every fixed step in training, we perform spawn operation for Gaussians. Specifically, we would like to select the Gaussians with larger scales and more contributions to the
5

---

scene representation. We refer to them as base Gaussians, which is found by selecting Gaussians whose scale and opacity are above certain thresholds the meantime. These base Gaussians are of greater contributions in the scene representation, which can be replaced by a set of Gaussians with various scales alternatively. Thresholding the gradient magnitude filters out trivial Gaussians such as those located in the distant background where they can hardly be seen.
Learn Finer Gaussians via Offset Table. After that, we spawn multi- finer Gaussians locally on top of base Gaussians. Instead of creating actual new Gaussians, to reduce the computational cost, we create a table & to store the parameters of offset distribution for K finer Gaussians. We find that some of the attributes of finer Gaussians can be shared as the same as base Gaussians with losing representation ability. We select position p. scale S and opacity a as the attributes maintained in the learned offset table, and the color and rotation R are shared by the base Gaussian associated. At inference, we apply the sampled offset to the base Gaussian to get the finer Gaussian. For example, the new position p* and scale S after offset are:
p" - p =
(3)
where Xp and Xs represent position and scale offset values sampled from the offset distribution parameterized by क_ There are total K entries in the offset table Φb while only M of them are selected randomly with equal probability and averaged to get the final offset distribution parameters Φ = 1 M Φ(im), where i represents the index for the selected entries. By doing so, we create a subset of finer Gaussians with multiple scales and select a random mixture of their attribute distribution parameters each time. Therefore, we build a random scale alternative to the original large and significant base Gaussians. For the base Gaussians, the densification, splitting, and cloning operations performed identically to the convention in 22, and these operations also performed for the offset table. We will introduce how to learn the offset table with variance in scale using variational inference in the following.
M Em
3.3 Infer the Posterior of the Offset Table
The offset table learn contains entries for K finer Gaussians, and the inference process introduces randomness in selecting the M entries that form the final offset distribution parameters. After obtaining these parameters, we perform variational inference for the offset distribution that enforces the finer Gaussains to be multi-scaled by assigning the prior distribution for the offset. We can infer the uncertainty from the distribution of 0", the parameter after applying the offset. Specifically, to infer the posterior and learn multi-scale representation, we let q(x) be the variational distribution for offset to approximate the true offset posterior distribution p(x(D) and minimize the Kullback-Leibee divergence between them:
dkl labx)ll(x | D)] = / g(x) log g(x)
(4)
p(x ())
-xx=o(( [logp(x. D) log {()) + log (DD),
(5)
where p(x) is the prior distribution for offset X. For position offset Xp and scale offset Xs, we assume normal distribution and uniform distribution respectively as the prior. Particularly, to learn multi-scale samples, we assume the offset prior distribution with the following parameters:
((xs) ~ U-(Snase + (1 - L/K)Sbaser 0): q(pp)~ ~ (0)
(6)
where Shase is the scale of base Gaussian that the finer Gaussian attached, 82 is the prior vari- ance for the position offset. The range of scale after offset for base Gaussian would be (1-1/K)S5 Sbasel, which means that the scale after offset would vary with the associated base Gaussian. The lower bound of the prior is (1 1/ KK) times the base Gaussian scale. This design choice ensures that the offset applied to the base Gaussian generates finer Gaussian with a larger scale range as the number of spawned Gaussians decreases, which means that larger diversity in scale applied when there are fewer spawned Gaussians. Additionally, we choose zero mean value for the prior of position offset to enforce that the position lies around the base Gaussian, and the prior of

---

Algorithm The pseudo-code of the training process of our uncertainty-aware 3DGS.
Input: Images and corresponding camera poses
Parameter: Maximum training step T; Spawn interval t; Threshold 7
Output: Trained scene representation with parameter 0; offset table Φ
while step I" do if' step % f === then
2:
3:
Select Gbase = [9.1 Σ [ > To. To.1|S.l1 > TS, a Ta
4:
Spawn Ibase. create offset table Φ { Øp. &a
5:
Assign prior p(x) for offsets
6:
end if
7:
Sample offset X, render image с and compute image loss L1, LSSTa
8:
Compute KL divergence LKL. dxrr,((x(D)ll(x
9: Optimize 0, á with total loss C L1 + LSSIM LKL
10: end while
opacity offset is given in the Appendix B We learn the offset table & and with the reparameterization trick (24). The total loss function Ltotal = (ē, Lsssar(CCC) dxr. [g(x)])((x) where is the ground truth of color and LSSTM is structural similarity index measure loss introduced in 2 At inference, predictive distribution is inferred by marginalizing over the offset distribution:
p x, x,
=
E
Ip(c
x, = p(c x, xxx(p( D)dx.
(7)
x~plx
where p(c D) is the color prediction for pixel The pseudo-code of proposer algorithm is shown in Algorithm]] !
Experiments
We evaluate uncertainty-aware view synthesis technique multiple real-world scenes. We compare the quantitative metrics for predictive uncertainty by estimating its correlation with the prediction error. Furthermore, we also validate the quality of our posterior uncertainty by removing Gaussians using different uncertainty threshold levels, which is quite practical for cleaning up and removing noisy regions of the scene, as known floaters 48). We will first introduce the datasets, metrics and baseline methods we used for evaluation, then the implementation details of our multi- scale variational inference algorithm. Finally, we will present quantitative results on uncertainty estimation, view synthesis and qualitative results on floater removal in the following.
4.1 Experimental Details
Datasets. We three datasets for evaluation: i) LF dataset 49 contains in total 8 indoor and outdoor scenes, each containing over 100 images from 360 view. Following the same setting with CF-NeRF (5), we use images from selected scenes torch, basket, africa, statue for evaluation. ii) LLFF dataset 3 contains 8 forward-facing and outdoor scenes, each containing 20 to 62 images where the camera positions are arranged in a grid pattern. iii) Mip-NeRF 360 dataset [50] contains 6 outdoor scenes, in each scene, more than 200 images are captured in 360 view. The scenes in this dataset are unbounded and contain detailed regions like grass fields, which challenging for reconstruction. We use this dataset to demonstrate the noisy Gaussian removal ability of our method.
Evaluation Metrics. For evaluating uncertainty estimation quantitatively, we first use the Area Under Sparsification Error (AUSE) with Mean Absolute Error (MAE) error. This metric evaluated the correlation between estimated uncertainty and true MAE error in each rendered view. It reorders the pixels according to the estimated uncertainty and calculates the difference between the reordered MAE array and the original ones. We normalize the error when calculating the difference. Secondly, we use the Negative Log-Likelihood (NLL) as a metric, which measures the likelihood of ground truth in the predictive distribution. This metric can evaluate both uncertainty and image quality at the same time. For image quality evaluation, we use Peak Signal-to-Noise Ratio (PSNR) to estimate
7

---

Novel View
Uncertainty
Error
Ensemble
S-NeRF
CF-NeRF
Ours
Figure 4: The visualization of predicted uncertainty map of novel view renderings. Our method demonstrates the best alignment of the uncertainty map with the error map.
the noise level, Structural Similarity Index Measure (SSIM) to measure the structural distortion and Learned Perceptual Image Patch Similarity (LPIPS) to measure the perceptual quality.
Baseline Methods.
CF-NeRF 5 enables the uncertainty estimation of NeRF models by
modeling latent variables and learning a conditional normalizing flows model. ii) S-NeRF 4 applies Bayesian learning to the NeRF model, and is able to quantify both depth and color uncertainty. iii) Bayes" Ray (3] constructs a spatial uncertainty field that can add perturbations to the position input of the radiance field and performs Laplace's approximation. iv) Easemble GS. In this method, we train 10 3DGS models with different subsets initialization points from Structure from motion (SfM) [5]] We also use different random seeds for each model. The variance of the predictions between all models regarded as the predictive uncertainty.
Implementation Details. We an AdamW optimizer update the learnable parameters of our variational multi-scale representation. The learning rate for 3DGS attributes is the same as the original algorithm (2]. We choose to spawn multi-scale variational inference. The learning rate of the offsett table is 0.1 times the learning rate of each attribute. The experiments are performed on single NVIDIA A100 GPU.
10 finer Gaussians the offset table to perform our
4.2 Uncertainty Estimation Quality Evaluation
We first evaluate the uncertainty of depth on the I.F dataset. The calibration between uncertainty and depth error is quantified by AUSE reported in Table On average, our method achieves the best performance. The performance of the ensemble method approaches better than other methods. However, our method requires much less computational cost compared to the model ensemble. In the basket scene, method largely improves the AUSE metric by 0.9 compared to the second-best ensemble method.
method while being
Furthermore, we evaluate the uncertainty quality of rendered images on the LF and LLFF datasets. In Table we report the AUSE NLL metrics for rendered images. The NLL is estimated by multivariate kernel density estimator used in [ On the LF dataset, our method shows the best AUSE, and outperforms S-NeRF in both metrics. terms of NL.., CF-NeRF shows comparable results in aligning the predictive distribution with the ground truth. A plausible reason is that CF-NeRF models

---

Table 1: The depth uncertainty estimation performance the LF dataset, quantified by the AUSE
with MAE error.
LF Dataset
africa
basket
statue
torch
Average
CF-NeRF
0.35
0.31
0.46
0.97
0.52
S-NeRF
0.66
0.38
0.67
0.74
0.61
Bayes" Ray
0.27
0.28
0.17
0.22
0.23
Ensemble GS (x 10)
0.16
0.22
0.17
0.26
0.20
Ours
0.19
0.13
0.21
0.23
0.19
Table The performance of novel view rendering and uncertainty estimation on rendered images within the LF and LLFF dataset.
Synthesized View Quality
Uncertainty Quality
Method
PSNR
SSIM
LPIPS 4
AUSE
NLL +
LF Dataset
CF-NeRF
24.32
0.835
0.202
0.49
-0.37
S-NeRF
20.21
0.761
0.248
0.62
1.32
Ensemble GS (x10)
27.64
0,902
0.088
0.29
-0.34
Ours
27.39
0.914
0.101
0.26
-0.30
LLFF Dataset
CF-NeRF
21.74
0.782
0.190
0.48
0.58
S-NeRF
20.10
0.744
0.221
0.59
0.91
Ensemble GS (x)
24.54
0.810
0.157
0.30
0.26
Ours
23.97
0.806
0.172
0.32
0.23
radiance distribution which helps improve performance. For the results on the LLFF dataset, our method demonstrates the best performance in terms of the NLL metric. We also surpass methods other than ensemble in terms of the AUSE metric. Bayes' Ray models spatial perturbation; therefore, only depth uncertainty quality is rendered and evaluated for this method. Our method does not offset color attributes directly to maintain efficiency, while still being able to model the uncertainty of predictive color by offsetting other attributes. We also provide visualizations of the predicted uncertainty map shown in Figure Our method aligns well with the prediction regions with complex scene content and overlapping object parts.
in detailed
4.3 Rendering Quality Evaluation
Apart from the quality of uncertainty, we also compare the quality of synthesized novel views on both LF and LLFF datasets. The results both datasets demonstrate the superior image quality of our method. Our method rivals the performance of the ensemble method in rendered image quality, without the need learn an extensive number of additional parameters for Spherical Harmonies (SH) coefficients that store radiance information. In the LF dataset, our method shows the best quality in terms of SSIM.
We provide rendered images of the fern scene in Figure Our method reconstructs the scene with clear details. The error pixels of the rendered image are located mainly in the region near the edge of the objects in the scene. A possible reason is that the slight error the camera pose estimated by SfM cause discrepancies between the rendered image and the ground truth. Some other error pixels are located on small distant object parts, such as small leaves in the background, which are more difficult reconstruct. The comparison of rendered images demonstrates that our variational multi-scale representation is capable of providing the extra functionality of uncertainty estimation while maintaining the performance of reconstructing various kinds of scenes with high fidelity.
4.4 Removing Noisy Floaters with Posterior Uncertainty
When reconstructing unbounded scenes such as those in Mip-NeRF 360, the training camera trajecto- ries are around the foreground object. Therefore, insufficient information regarding the background contained the collected images, and the geometry of the distant background object is hardly reconstructed. As result, the generalization ability over novel views those regions is poor, and

---

Counter
Bonsai
Kitchen
Bicycle
All Gaussians
50% Gaussians
30% Gaussians
Zoom in Details
Figure 5: The results of noisy Gaussian removal Mip-NeRF 360 scenes. By gradually deleting the Gaussians with large posterior uncertainty, our method removes the blurred floaters. The object of interest remains complete after the clean-up.
moving testing cameras away from the training trajectory leads significant drops in visual quality. In order to obtain a high-fidelity radiance field, one can remove these noisy Gaussians manually with editing tools. However, this process requires a large amount of human labor.
Our uncertainty estimation can be used to automatically remove floaters by deleting Gaussians where their parameters have relatively large posterior uncertainty estimated. The results of floater removal are shown in Figure 5 We retain 50% and 30% of the Gaussians with smaller posterior uncertainty in their parameters. From distant camera view, the novel view of the background scene fails to be synthesized due to the lack of the multi-view supervision signal. With the increasing number removed Gaussians, the noisy Gaussians with large scale and irregular covariance in the background are removed gradually. After removing 70% of the Gaussians in the scene, the floaters in the background scene are mostly eliminated while leaving the clear Gaussians that capture the complete object of interest. By using our method, we can obtain clear foreground objects for further editing and presenting in VR/AR scenes.
Conclusion
In this paper, we have proposed a probabilistic framework address the uncertainty estimation problem in the 3DGS algorithm. Different from previous work, we have identified the benefits of building multi-scale presentations enhance the diversity of parameter space samples when performing Bayesian inference. Our method improves the efficiency of variational inference by parameter sharing and sampling only a subset of attributes of the model. Experimental results have demonstrated the accuracy of our method in uncertainty estimation and its effectiveness in removing floaters. The potential usages of our method include quality assessment 3DGS scenes, robotics navigation and guided interactive active data acquisition.

---

[6] Niko Sünderhauf, Jad Abou-Chakra and Dimity Miller. Density-aware nerf ensembles: Quanti- fying predictive uncertainty in neural radiance fields. In 2023 IEEE International Conference 6 Acknowledgement
This work was supported in part by the NSFC Research Grants Council (RGC) Joint Research Scheme under the grant: N_HKBU214221, and the RGC Senior Research Fellow Scheme under the grant: SRFS2324-2502.
References
[1] Ben Mildenhall, Pratul Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In European Conference on Computer Vision, pages 405 421..42..20 Springer, 2020.
[2] Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphies, 42(4), 2023.
[3] Lily Goli, Cody Reading, Silvia Sellán, Alec Jacobson, and Andrea Tagliasacchi. Bayes' rays: Uncertainty quantification for neural radiance fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20061-20070, 2024.
[4] Jianxiong Shen, Adria Ruiz, Antonio Agudo, and Francesc Moreno-Noguer. Stochastic neural radiance fields: Quantifying uncertainty implicit 3d representations. In 2021 International Conference 3D Vision, pages 972-981. IEEE, 2021.
[5] Jianxiong Shen, Antonio Agudo, Francesc Moreno- -oguer, and Adria Ruiz. Conditional-fiow nerf: Accurate 3d modelling with reliable uncertainty quantification. In European Conference on Computer Vision, pages 540-557. Springer, 2022.
on Robotics and Automation, pages 9370-933 9376. IEEE, 2023.
[7] Yarin Gal and Zoubin Ghahramani. Dropout bayesian approximation: Representing model uncertainty in deep learning. In International Conference on Machine Learning, pages 1050-1059, PMLR, 2016.
[8] Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in neural information processing systems, 30, 2017.
[9] Erik Daxberger, Eric Nalisnick, James U Allingham, Javier Antorán, and José Miguel Hernández-Lobato Bayesian deep learning via subnetwork inference. In International Confer- ence on Machine Learning. pages 2510-2522 PMLR, 2021.
[10] Andrew G Wilson and Pavel Izmailov. Bayesian deep learning and probabilistic perspective of generalization. Advances in neural information processing systems, 33:4697-4708, 2020.
[11] Danny Wood, Tingting Mu, Andrew M Webb, Henry WJ Reeve, Mikel Lujan, and Gavin Brown. unified theory of diversity in ensemble learning. Journal of Machine Learning Research, 24(359):1-44 19,11-192222 2023.
[12] James H Clark. Hierarchical geometric models for visible surface algorithms. Communications of the ACM, 19(10):547555 1976.
[13] Michael Garland. Multiresolution Modeling: Survey and Future Opportunities. In Eurographics 1999 STARs. Eurographics Association, 1999.
[14] Shang- Hua Gao, Ming- Ming Cheng, Kai Zhao, Xin- Yin-Yulo Zhang, Ming Hsuan Yang, and Philip Torr. Res2net: A new multi-scale backbone architecture. IEEE Transactions on Pattern Analysis and Machine Intelligence, 43(2):652-66) 662, 2019.
[15] David G Lowe. Distinctive image features from scale-invariant keypoints. International journal of computer vision, 60:91-110, 2004.

---

[16] Jeremiah Liu, Zi Lin, Shreyas Padhy, Dustin Tran, Tania Bedrax Weiss, and Balaji Lakshmi- narayanan. Simple and principled uncertainty estimation with deterministic deep learning via distance awareness. Advances in neural information processing systems, 1147498-7512 2020.
[17] Joost Van Amersfoort, Lewis Smith, Yee Whye Teh, and Yarin Gal. Uncertainty estimation using a single deep deterministic neural network. In International Conference on Machine Learning, pages 9690-9700. PMLR. 2020.
[18] Yarin Gal, Riashat Islam, and Zoubin Ghahramani. Deep bayesian active learning with image data. In International Conference on Machine Learning, pages 1 1183-1192. PMLR, 2017.
[19] Xuran Pan, Zihang Lai, Shiji Song, and Gao Huang. Activenerf: Learning where to see with uncertainty estimation. In European Conference on Computer Vision, pages 230-246. Springer, 2022.
[20] Christopher M Bishop and Nasser M Nasrabadi. Pattern recognition and machine learning. Springer, 2006.
[21] Radford M Neal. Bayesian learning for neural networks, volume 118. Springer Science & Business Media, 2012.
[22] David JC MacKay. A practical bayesian framework for backpropagation networks. Neural computation, 4(3):448-472, 1992.
[23] Yarin Gal, Jiri Hron, and Alex Kendall. Concrete dropout. Advances in neural information processing systems, 30, 2017.
[24] Durk P Kingma, Tim Salimans, and Max Welling. Variational dropout and the local reparame- terization trick. Advances in neural information processing systems, 28, 2015.
[25] Florian Wenzel, Jasper Snoek, Dustin Tran, and Rodolphe Jenatton. Hyperparameter ensembles for robustness and uncertainty quantification. Advances in neural information processing systems,
2020.
[26] Marton Havasi, Rodolphe Jenatton, Stanislav Fort, Jeremiah Zhe Liu, Jasper Snoek, Balaji Lakshminarayanan, Andrew Mingbo Dai, and Dustin Tran. Training independent subnetworks for robust prediction. In 9th International Conference on Learning Representations, 2021.
[27] Yeming Wen, Dustin Tran, and Jimmy Ba. Batchensemble: Efficient ensemble of deep neural networks via rank-l perturbation. In Advances in neural information processing systems Workshop, 2019.
(28) Mehmet Ozgur Turkoglu, Alexander Becker, Hüseyin Anil Gündüzz Mina Rezaei, Bernd Bischl, Rodrigo Caye Daudt, Stefano D' Arroco,, Jan Wegner, and Konrad Schindler. Film-ensemble: probabilistic deep learning via feature-wise linear modulation. Advances in neural information processing systems, 35:22229-22244 2022.
[29] Gianni Franchi, Andrei Bursuc, Emanuel Aldea, Séverine Dubuisson, and Isabelle Bloch. Encoding the latent posterior of bayesian neural networks for uncertainty quantification. IEEE Transactions Pattern Analysis and Machine Intelligence, 2023.
[30] Ziyi Huang, Henry Lam, and Haofeng Zhang. Efficient uncertainty quantification and reduction for over-parameterized neural networks. Advances in neural information processing systems, 36, 2024.
[31] Taiga Abe, Estefany Kelly Buchanan, Geoff Pleiss, Richard Zemel, and John P Cunningham. Deep ensembles work, but are they necessary? Advances in neural information processing systems, 35:33646-33666 2022.
[32] Richard Hartley and Andrew Zisserman. Multiple view geometry in computer vision. Cambridge university press, 2003.
[33] Sebastian Thrun. Probabilistic robotics. Communications of the ACM, 45(3):52-557,200

---

[34] Ben Mildenhall, Pratul P Srinivasan, Rodrigo Ortiz- Catio, Nima Khademi Kalantari, Ravi Ramamoorthi, Ren Ng, and Abhishek Kar. Local light field fusion: Practical view synthesis with prescriptive sampling guidelines. ACM Transactions on Graphics, 38(4): 2019.
[35] Richard Tucker and Noah Snavely, Single-view view synthesis with multiplane images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 551-560, 2020.
[36] Lior Yariv, Jiatao Gu, Yoni Kasten, and Yaron Lipman. Volume rendering of neural implicit surfaces. Advances in neural information processing systems, 34:4805 4815, 2021.
[37] Ziyuan Luo, Boxin Shi, Haoliang Li, and Renjie Wan. Imaging interiors: An implicit solution to electromagnetic inverse scattering problems. In European Conference on Computer Vision, 2024.
[38] Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein. Im- plicit neural representations with periodic activation functions. Advances in neural information processing systems, 33:7462-7473, 2020.
[39] Vincent Sitzmann, Michael Zollhöfer, and Gordon Wetzstein. Scene representation networks: Continuous 3d-structure-awarr neural scene representations. Advances in neural information processing systems, 32, 2019.
[40] Qi Song, Ziyuan Luo, Ka Chun Cheung, Simon See, and Renjie Wan. Protecting nerfs' copyright via plug-and-play watermarking base model. In European Conference on Computer Vision, 2024.
[41] Xiufeng Huang, Ka Chun Cheung, Simon See, and Renjie Wan. Geometrysticker: Enabling ownership claim of recolorized neural radiance fields. In European Conference on Computer Vision, 2024.
[42] Haithem Turki, Michael Zollhöfer, Christian Richardt, and Deva Ramanan. Pynerf: Pyramidal neural radiance fields. Advances in neural information processing systems, 36, 2024.
[43] Zhiwen Yan, Weng Fei Low, Yu Chen, and Gim Hee Lee. Multi-scale 3d gaussian splatting for anti-aliased rendering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20923-2093 2024.
[44] Huangying Zhan, Jiyang Zheng. Yi Xu, Ian Reid, and Hamid Rezatofighi. Activermap: Radiance field for active mapping and planning. arXiv preprini 1xXix:2211.126:: 2022.
[45] Yunlong Ran, Jing Zeng, Shibo He, Jiming Chen, Lincheng Li, Yingfeng Chen, Gimhce Lee, and Qi Ye. Neurar: Neural uncertainty for autonomous 3d reconstruction with implicit neural representations. JEEE Robotics and Automation Letters, 8(2): 2023.
[46] Matthew D Hoffman, Tuan Anh Le, Pavel Sountsov, Christopher Suter, Ben Lee, Vikash K Mansinghka, and Rif Saurous. Probnerf: Uncertainty-aware inference of 3d shapes from 2d images. In International Conference on Artificial Intelligence and Statistics, pages 10425-10444. PMLR, 2023.
[47] Wen Jiang, Boshu Lei, and Kostas Daniilidis. Fisherrf: Active view selection and uncertainty quantification for radiance fields using fisher information. arXiv preprint arXiv:2311. 1.77744, 2023.
[48] Frederik Warburg, Ethan Weber, Matthew Tancik, Aleksander Holynski, and Angjoo Kanazawa. Nerfbusters: Removing ghostly artifacts from casually captured nerfs. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 18120-18130 October 2023.
[49] Kaan Yücer, Alexander Sorkine-Honne Hornung, Oliver Wang, and Olga Sorkine-Honne Hornung. Efficient
3d object segmentation from densely sampled light fields with applications to 3d reconstruction. ACM Transactions on Graphics, 35(3):1-15, 2016.
[50] Jonathan T Barron, Ben Mildenhall, Dor Verbin, Pratul P Srinivasan, and Peter Hedman. Mip- nerf 360: Unbounded anti-aliased neural radiance fields. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 5470-5479, 2022.

---

[51] Johannes L Schonberger and Jan-Michael Frahm. Structure- few---fooommee motion revisited. In Proceed- ings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4104 4113, 2016.

---

A Optimizing Variational Inference Objectives
In Section we have leveraged the Kl.-divergence the optimization goal of the multi-scale offset table with variational inference. However, the objective given by KL-divergence is intractable. Therefore, we have further transform the objective as follows:
J (x(x) log (a(x)
dkz (x1xx D)]
p(x D)
(xx(x) [log go (x( log p(x [D]
(xx(x) [log do (x)] (x) [og p(x: D) log)(D))
Ex)(x) [log (x( log D)] + p(D).
where @ the parameter of variational distribution, which we stored in the offset. We can transform this objective to the Evidence Lower BOund (ELBO) by considering only the expectation part, and taking the expectation of the term over the data distribution D. This yields tractable optimization objective to maximize as follows:
L(Ø) = Ex [log p(x, D) - log gx())
= (x) p())) + logp(x) - log (x))
E... [Bx~go(x) [logp(č
x, dxr(g(x)]lx(x))]
where logr(è p( Ir x) is the log-likelihood of ground truth color, p(x) is multi-scale prior distribution, 4(x) is the variational distribution. In our method, we assume Laplace distribution for the predictive variable c, then the loss function to optimize the prediction the & loss function.
B Prior Distribution of Opacity Offset
To get the opacity offset Xa. first sample n from the normal distribution, whose parameters learned and stored the offset table ø. Then, we apply a mapping using a sigmoid function with temperature get the opacity offset Xa:
Xa 1t--
The offset Xo is applied to opacity a by multiplication: a' a Xa- To derive the prior distribution for the offset one can use the change of variable technique. Firstly, the inverse transformation from Xa 17 is:
Xa Xa
11
In
The Probability Density Function (PDF) of is given by the normal distribution:
1
(m-p)2
p(n)
(10)
V2TT22
where y and a are the parameters of the normal distribution. Using the change of variables formula, we have:
dn dxa
p(xa)
p(n)
(11)
$ lnf
x)²
2
(12)
V
Xa Xa)

---

The Cumulative Distribution Function (CDF) of opacity offset prior Xa is shown in Figure 6 The intuition for this offset prior that we want apply a small perturbation the opacity in the variational inference. As shown in the CDF, with positive μs, the odds of offset Xo approaching 1 are high. Also, the sigmoid function is numerically stable in optimization. For computational simplicity, we minimize the KI. divergence between the variational distribution and prior of 7, instead of Xa
Cumulative Distribution Function (CDF) Opacity Offset
1.0
CDF
0.6
0
0.0
0.2
0.4
Opacity Offset
Figure 6: The Cumulative Distribution Function (CDF) of opacity offset prior.
C Rendering Time Analysis
Table 3: Inference time for variants of our method and the ensemble method.
Inference Time (s)
Ensemble GS (x10)
0.27 + 0.05
fuls fall
0.12 + 0.04
OurSp,S.c
0.08 + 0.02
Oursp.S.o
0.06 + 0.02
The time for inferring frame using our method compared with the ensemble method is shown in 3 We test on torch scene in the LF dataset with a single NVIDIA A100 GPU, and take the average time of 1, 0000 frames. Ours all means offset with all attributes, Oursp.S.c means offset with position, scaling and color. Oursp.5,a means offset with position, scale and opacity, which is the setting of our main experiments. The running time shows that our design choice can achieve the lowest inference time.
D Active Learning Experiments
In active data acquisition of 3DGS, image collection and the 3DGS model training are performed alternately. Our goal is to maximize the model quality with the same number of images used. At each image collection step, the most informative image is selected via acquisition function, in our case the uncertainty the rendered image. By using our uncertainty estimation method as the acquisition indicate where the model is uncertain about and acquire more data around there.
function,
We perform simple experiment on active data acquisition of 3DGS on the LLFF dataset. Specifically, the original training dataset serves as the candidate image pool, and 10% of the images are randomly chosen for training initially. Then, one image is chosen for every 500 steps until total 30% of images used. We render uncertainty map and aggregate the pixel values to choose the most

---

uncertain image from the pool as the next image added to the training set. After all images are chosen, the 3DGS model is further trained for 7K steps. The densification interval is 100 steps, and the spawning interval is 500 steps, and both operations are performed until training ends. As shown in T we found that the view synthesis quality of active 3DGS with our uncertainty estimation better than choosing images randomly. As the experiment setting for active learning is intricate, we prefer to fully investigate the application our uncertainty estimation on active learning, which is limitation of this paper.
Table 4: The experiment on active learning with our uncertainty estimation.
PSNR
SSIM
LPIPS
Random
20.97
0.65
0.234
Ours
21.35
0.69
0.212
E Ablation Study on the Number of Spawned Gaussian
We compare the view synthesis and uncertainty estimation performance using K 1,5, 10 number of finer-level Gaussians spawned in the offset table. We train on all scenes in the LLFF dataset and report the average results in Table We found that improving the number of finer-level Gaussians K shows notable increase in the quality of uncertainty estimation. More finer level Gaussians improve the sample space diversity, therefore providing precise estimation of model parameter uncertainty and novel views.
Table 5: Ablation study on the number of spawned Gaussians.
PSNR
SSIM
LPIPS
AUSE
NLL
Finer Gaussisans
23.42
0.792
0.183
0.37
0.24
Finer Gaussisans
23.94
0.795
0.178
0.34
0.27
10 Finer Gaussisans (Default Setting)
23.97
0.806
0.172
0.32
0.23
F Additional Visualization
Additional visualization results of view synthesis and uncertainty estimation of rendered images on scenes from the LLFF dataset using our method are provided in Figure 7 The images are down-scaled to half the original size.

---

Room
Fortress
Orchids
Ground Truth
Rendered Image
Error Map
Predicted Uncertainty
Figure 7: Addtional visualization results.

---

NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justification right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will published with the paper.
The reviewers of your paper will be asked use the checklist as one of the factors their evaluation. While "[Yes] is generally preferable to "[No]" ", it is perfectly acceptable to answer "[No] providedd proper justification is given (e.g.. "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering binary way, we
"[No] or "[NA]" is not grounds for rejection. While the questions are phrased acknowledge that the true answer is often more nuanced, please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS paper checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The contributions are reflected.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The limitations discussed.

---

Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, c.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification:
The proofs are complete.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We provided such information.
Guidelines:
The answer NA means that the paper does not include experiments.

---

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
We will release code once accepted.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (nttps://nips.co
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
7 /ttppp.cc/pbllc//uudeeee http://oeeeunmiiiioooooooooeeeee
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer:
[Yes]
Justification:
We provided such information.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification:
We provided such information.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes] Justification
Justification:
We provided running time of algorithms.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

---

The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethies
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer:
[Yes]
Justification:
We conform this.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g.. if there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification:
No significant negative impacts.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Answer: [NA]
Justification: No new data published.
Guidelines:
The answer NA means that the paper poses such risks.

---

Released models that have high risk for misuse or dual use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We conform that.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
No new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: No such experiments.

---

Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: No such risks.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.


Reviews:
[{"review_id": "uo4S1XVYQU", "reviewer": "Reviewer_UQFr", "summary": "This paper proposed a novel multi-scale variational representation for 3D gaussian splatting to estimates its uncertainty. Specifically, this paper introduced a spawn strategy to split a base gaussian into multiple sub gaussians and apply variational inference to estimate the uncertainty.", "questions": "(1), For the floater removal experiment, is removing the background actually the desired effect for real world use case? In my understanding , a good uncertainty-based gaussian pruning approach should be able to remove the blurry gaussians to improve the rendering clarity while preserving most of the objects regardless its foreground or background.\n\n(2), For the related works section, there are some additional related uncertainty quantification works could be discussed:\n[1] Naruto: Neural active reconstruction from uncertain target observations. CVPR 2024\n[2] Active neural mapping. ICCV 2023", "limitations": "The uncertainty qualification requires (additional) sub gaussian construction, might potentially increase the computational burden.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "(1), Estimating uncertainty in the 3D reconstruction / novel view synthesis systems is a critical task and has great application potential.\n\n(2), The paper proposed a novel representation and solid mathematical derivation to solve this problem.\n\n(3), The experiment shows strong relevance between the estimated uncertainty and the rendering error.", "weaknesses": "(1), Authors claimed one of the potential application is guided interactive active data acquisition. It would be great to have more detailed illustration or even experimental exploration.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful comment and positive feedback! We address your concerns as follows. \n\n## W1: Discuss more about active data acquisition. \n\nIn active data acquisition of 3DGS, image collection and the 3DGS model training are performed alternately. At each image collection step, the most informative image is selected via an acquisition function to maximize the model quality with the same number of images used. Our uncertainty estimation method can contribute to this acquisition function, indicating where the model is uncertain about and acquiring more data around there. \n\nWe perform a simple experiment on active data acquisition of 3DGS on the LLFF dataset. Specifically, the original training dataset serves as the candidate image pool, and 10% of images are randomly chosen for training initially. Then, one image is chosen for every 500 steps until 30% of images are used. We render our uncertainty map and aggregate the pixel values to choose the most uncertain image from the pool as the next image added to the training set. After all images are chosen, the 3DGS model is further trained for 3K steps. The densification interval is 100 steps, and the spawning interval is 500 steps, and both operations are performed until training ends. \n\nAs shown in the Table below, we found that the view synthesis quality of active 3DGS with our uncertainty estimation is better than choosing images randomly. Due to the various detailed settings in the active data acquisition task, we prefer to fully evaluate the performance of our uncertainty in active learning in the revision of this paper. \n\n\n|        | PSNR  | SSIM | LPIPS |\n|--------|-------|------|-------|\n| Random | 20.97 | 0.65 | 0.234 |\n| Ours   | 21.35 | 0.69 | 0.212 |\n\n\n## Q1: Floater removal should clear blurry Gaussians regardless of foreground or background. \n\nPlease refer to Q1 in global response, where we show that our method can clean foreground noisy Gaussians. \n\n## Q2: Discussion on related work. \n\nThank you for pointing these out! Different from our uncertainty estimation method, Naruto [8] learns the uncertainty of depth by the Negative Log Likelihood loss function. Active neural mapping [9] leverages the artificial neural variability [1] to indicate the predictive uncertainty. We will compare and cite them in the related work section. \n\n## L1: Constructing sub Gaussians might increase the computational burden \n\nWe analyze specifically the extra computational cost from the following perspective:  \n\n**Rendering RGB image**: The image rendering speed is basically the same as vanilla 3DGS rendering, since we do not require multiple sampling from variational distribution. \n\n**Calculating Per Gaussian Uncertainty for Pruning**: Calculating per Gaussian uncertainty for pruning simply aggregates the variance of learned posteriors for each Gaussian, which takes less than 1s for the garden scene in MipNeRF 360 in a V100 GPU. \n\n**Rendering Uncertainty Map**: Rendering the uncertainty map requires sampling from the variational distribution. If taking $N$ samples, the rendering cost would be $N$ times the vanilla rendering cost. In our experiments, we set $N$ equal to 10, the same as the number of finer gaussian. This cost growth follows the common practice in Variational Bayes approaches such as CF-NeRF [4] and others [2], [3]. \n\n**Memory**: The extra memory cost for the offset table of a base Gaussian is $12 \\cdot (K-1)$ floating numbers while $K$ is the number of finer Gaussians. We only build offset tables for base Gaussians which takes up around 50% of all Gaussians in the original scene. If rendering uncertainty maps is not required, we can retain only one entry in the offset table to render RGB images after pruning the noisy Gaussians. \n\nWe’ll be glad to discuss if you have any further concerns.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "34kkMLfyJY", "reviewer": "Reviewer_4RNR", "summary": "The paper aims to quantify uncertainty in the learning pipeline in 3DGS. To this end, the author(s) proposed to leverage explicit scale information to build variational multiscale 3D gaussians leading to the construction of diversified parameter space samples. This results in the proposition of a multiscale variational inference framework for uncertainty estimation in the 3D Gaussian model. Experimental results on popular benchmark datasets are shown to demonstrate the efficacy of the proposed method.", "questions": "1. Kindly clarify how removing uncertain 3D Gaussian still leads to complete scene images visualization on screen (rasterization).\n\n2. Kindly provide details on how distance of the points from the camera affects the uncertainty quantification. \n\n3. It is well-known that COLMAP 3D and camera pose are not perfect. How structure from motion uncertainty contributes to the current approach. Kindly comment.", "limitations": "Although a few limitations are obvious from the experiments section. It is not explicitly mentioned in the paper.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The paper solves an extremely useful problem for robot vision and control applications: quantifying uncertainty in 3D Gaussian splatting. Uncertainty modeling is essential for developing robot vision-based automation systems that are robust, safe, efficient, and capable of operating in complex and dynamic environments. By explicitly accounting for uncertainties, this approach can help design more effective and reliable robotic vision systems. Therefore, this paper clearly attempts to address a significant problem.\n\n2. Uncertainty quantification on explicit 3DGS by exploiting both the model space diversity and efficiency. Such a modeling strategy ensures efficient and compact model output.", "weaknesses": "1. The results are not as impressive as mentioned in the abstract and in the introduction of the paper, i.e., state-of-the-art.\n\n2. The approach to use only scale can cause problems with points that are far from the camera. This is critical particularly for the formulation that takes both scale and rotations to model the covariance —refer to 3DS original work. Kindly comment.\n\n3. Writing of the paper can be improved. \n\n        a. can not -> cannot\n\n        b. robotics navigation -> robot navigation\n\n        c. less samples of parameters as possible -> parameters samples as few (less) as  possible.\n\n        d. spawn strategy -> not clearly explained while introducing this term for the first time in the paper, i.e.,  what author(s) mean by this term—referring to contribution. Explaining at least briefly here will improve the draft.\n\n        e. Evaluation is generally not considered as contribution (contribution 3)\n\n        f. Section 2.2 line 97 “Rearly…this task”. Kindly rephrase this line, it is confusing in the current form.\n\n        g. Line 228: “the original is”...?\n\n        h. Line 234: Then-> then\n\n        There are more typos… kindly correct them.\n\n        i. Missing references in implicit uncertainty modeling Lee et al. RAL 2022 “uncertainty guided policy for active robotic 3d reconstruction using neural radiance fields”", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the kind and helpful comments! We will address your concerns below. \n\n## W1: The results are not as impressive as claimed\n\nFirstly, we’d like to clarify that the ensemble method is a naive baseline that trains 10 vanilla 3DGS with different random seeding, which incurs an extreme computational burden compared to other methods. Therefore, the ensemble method serves as a performance upper bound, with the same practice in [7]. \n\nApart from the ensemble method, our method is optimal in most cases. The only exception is the AUSE and NLL metric on the LF dataset, where our method is inferior to CF-NeRF, in Table 2. We discuss this result on line 250-252. On the other hand, in this setting, our view synthesis quality is better than other methods, which is another primary goal of building uncertainty-aware 3DGS models. \n\n## W2: The approach to use only scale can cause problems with points that are far from the camera; scale and rotation together form the covariance\n\nWe provide qualitative results regarding the points far from the training camera in Figure 2 in the attachment. In unbounded scenes from MipNeRF 360 dataset, using all training images, our method successfully renders both uncertainty maps and novel views of the distance background. Technically, although we refer to our method as multi-scale representations, as shown in Equation 6, we offset not only scale but also position to spawn the finer Gaussian. \n\nAdditionally, the scale of the finer Gaussian is restricted by our setting of prior distributions. The learned posterior of $K$ finer Gaussian scale would approach $U (S_{base}/K, S_{base})$, which means that the scale of finer Gaussians would be constrained softly to avoid unlimited variation in scale compared to the base Gaussian they attached. Therefore, for base Gaussians far from the camera, the scale of their spawned finer Gaussian is still adaptive, without harming the rendering quality. In practice, we found that the distributions of rotation over the scene are rather random and irregular compared to the scale, while the scale varies with object size. Therefore, we chose to decompose the covariance and only offset the scale to form our multi-scale representation. \n\n## W3: Writing and reference problems\n\nThanks for your careful inspection! We will fix the raised problems and improve the writing thoroughly in the revision. \n\nThe RAL paper proposes to use the entropy along the ray weight to evaluate pixel-wise uncertainty, which is used to guide active reconstruction. We will discuss and cite the RAL paper in the related work section. \n\n## Q1: How does removing uncertain 3D Gaussian still lead to complete scene renderings\n\nPlease refer to Q1 in the global response, where we visualize that when removing 10% of the Gaussians, the complete scene is preserved and small floaters are cleaned. We also illustrate why in the original paper Figure 1 and Figure 4 we remove at least 50% of the noisy Gaussians. \n\n## Q2: Details on how the distance of the points from the camera affects the uncertainty\n\nWhen estimating the per point (Gaussian) uncertainty in the MipNeRF 360 dataset, usually the model is more uncertain about the points (Gaussians) distant from the training camera trajectory. Generally, the uncertainty value of points depends on how well they are covered by multiple views (cameras) without occlusion. The distant, background points have less content provided from multiple views in a casually captured dataset, which makes the background modeling difficult. \n\nWhen rendering the uncertainty map, the points (Gaussians) far from testing cameras are projected to a smaller pixel region in image space, due to the inherent properties of perspective projection. However, the intensity of uncertainty pixels is consistent within the projected area in image space. \n\n## Q3: How does COLMAP uncertainty contribute to the current approach\n\nIn 3DGS training, the COLMAP generates sparse point clouds and camera poses as the input of 3DGS. Thus, from the 3DGS point of view, the uncertainty contained in camera poses generated by the structure from motion can be categorized as aleatoric uncertainty, which inherently exists in the input data to 3DGS [5]. The aleatoric uncertainty, together with epistemic uncertainty that exists in the model parameters of 3DGS, results in the predictive uncertainty estimated in our method. \n\n## L1: More discussion on limitations\n\nFor quantitative results, we discuss the limited results in line 250-252 of the original paper. We will talk more about the qualitative performance of rendering uncertainty maps in MipNeRF 360 in the revision. \n\nThank you again for your effort in reviewing our work! We’ll be glad to discuss if there are any further concerns.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CH55OeJGH8", "reviewer": "Reviewer_XD3Y", "summary": "This paper proposes an uncertainty estimation method for the 3D Gaussian Splatting (3DGS) radiance field reconstruction algorithm. The proposed methods leverage the multi-scale properties that lie inherently in a vast amount of Gaussian ellipsoids to improve the performance of variational inference. The paper validates quantitatively that the proposed method aligns with the novel view synthesis error better than previous methods on NeRF uncertainty estimation and naïve methods on 3DGS uncertainty estimation. The experiments also validate an interesting application of the proposed uncertainty estimation methods: to determine noisy Gaussians and remove them thus reduce the floater artifacts.", "questions": "Please see weakness.", "limitations": "More comparing results with simpler methods such as naïve variational inference can be provided.", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1. The motivation of this paper is clear. The problem of increasing diversity in the sampling of variational inference is challenging, and the idea of developing a multi-scale representation in 3DGS to solve it is intriguing. \n\n2. The method is sound and easy to follow. Representing the same scene with multiple scale levels is feasible in previous CG techniques. The proposed technique of learning the scene with different scales by assuming different variational distributions, and estimating the predictive uncertainty by sampling from these “multi-scale posteriors” is clear. Designing an offset table to increase inference efficiency is a plus. \n\n\n3. The experiments demonstrate the effectiveness of the method in uncertainty estimation by comparing AUSE and NLL. The synthesized image quality in Table 6 is better than NeRF based method, and the noisy Gaussians removal experiments are interesting.", "weaknesses": "1. Ambiguous details: i): In line 187, there are K alternative values in the offset table. If so, how many values that the offset table contain for one spawned Gaussian? ii): In Figure 3 (c), how are sampling and inference done from multiple spawned Gaussians? The Figure should contain more details to clarify the difference with other methods. \n\n2. More ablation studies. i): Replacing the multi-scale prior in equation 6 with the same prior distribution for all layers can be compared to study the performance gain of the multi-scale method. ii): The impact of the quantity of spawned Gaussians in the offset table (line 174) on the reconstruction performance can also be studied.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your acknowledgment of our work! We provide illustrations for your concerns below. \n\n## W1: Ambiguous values in the offset table for each spawned Gaussian\n\nSince we only offset the position and scale in the offset table, and each offset table contains $K$ spawned Gaussians, then for each spawned Gaussians the length of the offset vector is 12 (6 for position offset distributions and 6 for scale), so the number of alternative values in the offset table is $12 \\cdot K$. \n\n## W2: Ambiguous Figure\n\nGenerally, the inference pipeline of our method is to first sample from the spawned finer Gaussians and then sample from their learned posterior distribution. We will update the details in the figure.  \n\n## W3: How about setting the same prior for all finer levels\n\nThank you for the question. Actually, if setting the same prior for all finer levels, the learned posterior would lean to the same distribution after training. Therefore, this method could be regarded as equivalent to using a single finer level Gaussian. We perform additional experiments to explore the effectiveness of this method, and provide the results on the LLFF dataset in Q2 in the global response. We found that using only one finer level Gaussian would largely decrease the uncertainty estimation performance. \n\n## W4: The impact of the number of spawned Gaussians\n\nGenerally, the performance would increase when there is more number of fine level Gaussians. This is because more spawned Gaussians provide more capacity to fit the posterior distribution. We provide the comparison results and analysis of using 1, 5 and 10 spawned Gaussians in Q2 of the global response to validate this. \n\n## L1: Comparison with naive variational inference\nPlease also see the answer in W3. \n\nWe’ll be happy to address any further questions, please feel free to raise them.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NZudMKWaFZ", "reviewer": "Reviewer_ZTX9", "summary": "This work suggests a method to estimate uncertainty for the output of a 3D Gaussian Splatting (3DGS) model, by means of variational calculus.  \nSpecifically, the authors split the 3DGS model to two hierarchical levels (or- two scales). The coarser \"Base\" level is very similar to vanilla 3DGS, but now each Gaussian contains a list of parameters $\\{ \\mu_i, \\sigma_i\\}$ that are used to spawn the \"Finer\" level Gaussians.  \nThe latter \"Finer\" Gaussians are then fed into the standard 3DGS inference pipeline, only now these Finer Gaussians are fitted using the re-parametrization trick and the ELBO.  \nThe experimental section compares to other uncertainty estimation methods and shows favorable results.", "questions": "High level questions:\n* Can the proposed method be applied directly on the **Base** level Gaussians? i.o.w. - why does one needs more than a single Gaussian in the Finer level?\n* I would appreciate a deeper discussion on the similarities to [42]. \n---\nWhile the method is rather straightforward, some design choices are not clear to me:\n* why is $\\mu_n$ in eq(6) is limited to this range?\n* why is the spawning method of the Base level different from vanilla 3DGS?", "limitations": "The authors do not discuss limitation of their method.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The method builds on the simplicity of 3DGS, and seems rather straightforward to implement. Given additional prior knowledge on ELBO based methods, the proposed method is rather easy understand from the text (with the small exception mentioned below).  \nThe results show the power of this method compared to other methods (with the one exception mentioned below).", "weaknesses": "The method seems attractive, but needs to be better positioned w.r.t. previous works:\n* FisherRF, (citation [42] in the paper) is mentioned in line 116 but is not compared against. The justification for the lack of numerical comparison is that \"the posterior needs extensive computations\", but the authors of [42] state they run on a modest card (RTX3090) in 70 FPS. At the very least, results from [42] can be easily added to Table 1 based on Table 4 in [42]. Additionally, the [code](https://github.com/JiangWenPL/FisherRF) of [42] was made public. \n* latentSplat [arxiv 2403.16292](https://arxiv.org/abs/2403.16292) - while it operates under a **simpler setting than this work** , and while it is not yet accepted a peer-reviewed venue (AFAIK), I would recommend mentioning this method to prevent confusion.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the thoughtful review and the positive feedback! We address your comments and suggestions below. \n\n## W1: Comparison with FisherRF\n\nWe present the results of evaluating the quality of depth uncertainty maps using FisherRF. Note that our training view selection and implementation of AUSE in MAE metric follows the code of Bayes’ Ray [7]. The performance of other methods is provided in Table 1 in the main paper, and the average AUSE of our method is superior to FisherRF. \n\n|        | africa | basket | statue | torch | average |\n|--------|--------|--------|--------|-------|---------|\n| FisherRF | 0.29   | 0.27   | 0.31   | 0.42  | 0.32    |\n| Ours   | 0.18   | 0.17   | 0.13   | 0.29  | 0.19    |\n\nIn our original submission, the choice of not presenting FisherRF results is mainly because all other methods we listed can compute not only depth uncertainty maps but also RGB uncertainty maps, which forms our complete evaluation. However, FisherRF only reports the depth uncertainty results in the paper. The provided code only contains depth uncertainty map rendering, where they compute the per Gaussian uncertainty and perform alpha blending as the depth uncertainty map. \n\nRegarding computational efficiency. We would to further clarify that FisherRF actually requires a pre-computation step to compute the Hessian for all the Gaussians. For the basket scene in the LF dataset, we use 29.7s in a single V100 card to compute these Hessian. Then, the uncertainty map can be rendered for around 70 FPS. Instead, our method does not require any pre-computation step for rendering RGB uncertainty. \n\n## W2: Discussion about LatentSplat\n\nLatentSplat builds variational distribution for Gaussian features and achieves generalizable reconstruction of radiance field. Thank you for pointing this out, and we will discuss more and cite this work in the related work section. \n\n## Q1: Why do we need more than one single Gaussian in the Finer level?\n\nPlease refer to the results in Q2 in the global response. \n\nActually, we found that the number of finer level Gaussian is rather important for the quality of uncertainty estimation. The purpose of multiple finer Gaussian is to encourage a variety of scale distribution among different finer level Gaussians that are attached to the same base Gaussian. This leads to more diversified samples in inferring the uncertainty using the learned posterior, which helps improve the accuracy of uncertainty. \n\n## Q2: Deeper discussion on the similarity to FisherRF\n\nThe similarity between our method and FisherRF is that they both can estimate the uncertainty explicitly for each point (Gaussian) in 3DGS. However, the parameter uncertainty in FisherRF $\\mathbf{H}^{\\prime \\prime}\\left[\\mathbf{w} \\mid D^{\\text {train }}\\right]$ is approximated with the diagonal elements of the Hessian matrix of the parameter $\\mathbf{w}$ given the training data $D^{\\text {train}}$. \n\nThis approximation is made by employing assumptions like the predictive distribution should be a normal distribution with the mean equal to the maximum a posteriori solution and precision equal to the Fisher Information (Equation 18 in [6]). Instead, we follow a Variational Bayesian approach, in which we set priors for parameters, then minimize the KL divergence between prior and variational distribution to infer the variance of model parameters as the uncertainty. \n\n## Q3: Why $\\mu_{n}$ is limited to this range\n\nThe prior distribution of offset is set as this uniform distribution so that the scale of $K$ number of finer level Gaussian after offset would become $S_{offset} = S_{base}+\\mu_n \\sim U (S_{base}/K, S_{base})$, which encourage that the finer Gaussians scale not to exceed $S_{base}$, the scale of base Gaussian they attached or become too small and lose functionality. This setting enables the multi-scale representation by learning the distribution of scale of finer level Gaussian. \n\n## Q4: Why is the spawning method of base level different from vanilla 3DGS\n\nWe would like to clarify that densification and spawning are different operations. \nSpawning refers to the operations that select and offset base level Gaussians to construct finer level Gaussian. The densification of Gaussian is to clone all the attributes of Gaussians and translate the new ones. \n\nFor example, in our experiment, we train the LLFF scenes for 12K steps in total, during which the densification is performed for every 500 steps and spawning is performed for every 4K steps. \n\nNote that both base Gaussian and non-base Gaussian can perform densification, the difference is that the densification of base Gaussian also clones the attached finer Gaussians. Apart from that, the implementation of densification is identical to the original 3DGS. Thank you for pointing this out, and we will add the above details in the revision. \n\nIf you have further questions please feel free to raise them. We would be glad to provide materials and discuss!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We would like to thank all the reviewers for their time and efforts in reviewing our paper and providing insightful comments. Here we address two common questions from the reviewers in this global response, and the individual questions from each reviewer are provided separately.

## Q1: To Reviewer [UQFr, 4RNR]: Why does the floater removal experiment remove the background? Can we retain the background? 

Firstly, we observe that when training the 3DGS on MipNeRF 360 unbounded scenes, the majority of the floaters lie in the background region. This is because the training camera trajectory is placed around the centering object while providing insufficient multi-view information to reconstruct the background. Thus, we visualize the floater cleaning performance by removing at least 50% of the Gaussians to validate that our method can remove most of these noisy floaters. Moreover, our visualization results are consistent with Figure 1 in Bayes Ray [7], which also removes the noisy background of an unbounded scene represented by NeRF.  

We further show that our method can also perform the floater removal effect for noisy Gaussian in the foreground. As shown in Figure 1 in the attachment, when removing only 10% of the Gaussian and visualizing from a close view, our method can also remove smaller floaters in the foreground to improve the clarity of the synthesized view, while keeping the background complete. 

## Q2: To Reviewer [XD3Y, ZTX9]: Comparison with simpler methods, such as using less or even one finer level Gaussian. 

We compare the view synthesis and uncertainty estimation performance using $K \in {1, 5, 10}$ number of finer level Gaussians spawned in the offset table. Same as Section 4.2 in the original paper, we train on all 8 scenes in the LLFF dataset and report the average results. We found that improving the number finer level Gaussian $K$ shows a notable increase in the quality of uncertainty estimation. More finer level Gaussians improve the sample space diversity, therefore providing precise estimation of model parameter uncertainty. Nevertheless, the quality of novel views fluctuates when $K$ changes. We think this is because 5 spawned Gaussians are enough to represent the scene while the quality of uncertainty could be further promoted by more spawned Gaussians. 

|                                    |       |       |       |      |      |
|------------------------------------|-------|-------|-------|------|------|
|                                    | PSNR  | SSIM  | LPIPS | AUSE | NLL  |
| 1 Finer Gaussisans                      | 23.67 | 0.791 | 0.194 | 0.52 | 0.47 |
| 5 Finer Gaussisans                     | 23.92 | 0.797 | 0.179 | 0.43 | 0.34 |
| 10 Finer Gaussisans  (Default Setting)  | 23.84 | 0.805 | 0.186 | 0.38 | 0.32 |

We thank the reviewers again for their valuable feedback, and sincerely look forward to discussing any further questions!

## Reference

[1] Xie, Zeke, et al. "Artificial neural variability for deep learning: On overfitting, noise memorization, and catastrophic forgetting." Neural computation (2021). 

[2] Kingma, Durk P., Tim Salimans, and Max Welling. "Variational dropout and the local reparameterization trick." NeurIPS (2015). 

[3] Blundell, Charles, et al. "Weight uncertainty in neural network." International conference on machine learning. ICML (2015). 

[4] Shen, Jianxiong, et al. "Conditional-flow nerf: Accurate 3d modelling with reliable uncertainty quantification." ECCV (2022). 

[5] Kendall, Alex, and Yarin Gal. "What uncertainties do we need in bayesian deep learning for computer vision?." NeurIPS (2017). 

[6] Kirsch, Andreas, et al. “Unifying Approaches in Active Learning and Active Sampling via Fisher Information and Information-Theoretic Quantities.” TMLR (2022). 

[7] Goli, Lily, et al. "Bayes' Rays: Uncertainty Quantification for Neural Radiance Fields." CVPR (2024). 

[8] Feng, Ziyue, et al. "Naruto: Neural active reconstruction from uncertain target observations." CVPR (2024). 

[9] Yan, Zike, et al. "Active Neural Mapping." ICCV (2023).

Meta-review: This paper was reviewed by four experts in the field.  Based on the reviewers' feedback as well as the author feedback, the decision is to recommend the paper for acceptance.  The reviewers did raise some valuable comments that should be addressed in the final camera-ready version of the paper. The authors are encouraged to make the necessary changes to the best of their ability.    We congratulate the authors on the acceptance of their paper!
Decision: Accept (poster)