Variational Multi-scale Representation for Estimating Uncertainty in 3D Gaussian Splatting

Authors: ruiqi  li
yiu-ming  cheung
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 10:59:02

Abstract: Recently, 3D Gaussian Splatting (3DGS) has become popular in reconstructing dense 3D representations of appearance and geometry. However, the learning pipeline in 3DGS inherently lacks the ability to quantify uncertainty, which is an important factor in applications like robotics mapping and navigation. In this paper, we propose an uncertainty estimation method built upon the Bayesian inference framework. Specifically, we propose a method to build variational multi-scale 3D Gaussians, where we leverage explicit scale information in 3DGS parameters to construct diversified parameter space samples. We develop an offset table technique to draw local multi-scale samples efficiently by offsetting selected attributes and sharing other base attributes. Then, the offset table is learned by variational inference with multi-scale prior. The learned offset posterior can quantify the uncertainty of each individual Gaussian component, and be used in the forward pass to infer the predictive uncertainty. Extensive experimental results on various benchmark datasets show that the proposed method provides well-aligned calibration performance on estimated uncertainty and better rendering quality compared with the previous methods that enable uncertainty quantification with view synthesis. Besides, by leveraging the model parameter uncertainty estimated by our method, we can remove noisy Gaussians automatically, thereby obtaining a high-fidelity part of the reconstructed scene, which is of great help in improving the visual quality.

Main paper: 
 introduction: 
 The radiance field methods [1] for view synthesis have received increasing attention in the past few years due to their capability of achieving photorealistic results. Among them, the recently proposed 3D Gaussian Splatting (3DGS) algorithm [2] has pushed the boundary of real-time view synthesis with prominent quality and efficiency. However, a major functionality deficiency of 3DGS is that it cannot provide uncertainty information regarding the reconstructed model and predictions. Such uncertainty information would be useful in removing the noisy components in the model and providing confidence maps to assess the quality view synthesis results, which is important in applications such as autonomous driving simulation and robotics navigation.Previous works in view synthesis made attempts to quantify the uncertainty in Neural Radiance Field (NeRF) models via ensemble, variational inference or Laplace's approximation methods [3, 4, 5, 6] . Although these works demonstrate the ability to quantify the uncertainty with NeRF models, they cannot be directly applied to 3D Gaussian Splatting models, due to the intrinsic difference between the implicit NeRF representation and explicit 3DGS. Furthermore, some prior works deal with the uncertainty in learning models with methods such as Monte-Carlo dropout [7] , Deep Ensemble [8] and The results of cleaning up an unbounded scene reconstructed with 3DGS using our uncertainty estimation. We remove the Gaussians with large parameter uncertainty, the majority of which are under-reconstructed background. The desk at the center of the scene remains complete even after removing 90% of the Gaussians.Subnetwork [9] . A major characteristic of these methods is that they can be seen as the approximation of Bayesian Inference, which estimates the distribution of posterior p(θ|D) and prediction p(y|θ, x), instead of point estimation.The methods mentioned above can be seen as inferring the posterior distribution using model space samples. As mentioned in previous works [8, 10, 11] , the key objectives for good model space samples are diversity and efficiency. To improve the quality of uncertainty estimation by increasing the diversity, the generated model space samples should explore the potential cases of true posterior as much as possible, which was not achieved in previous work on NeRF uncertainty estimation. Furthermore, although naive methods like ensemble provide good approximations, they require large storage and the computational cost which increase linearly with the number of samples. Thus, on top of diversity, efficiency is another important aspect we focus on when estimating the uncertainty for 3DGS, which means that we should use as few samples of parameters as possible.In this paper, we design an uncertainty estimation method for 3DGS by constructing diversified parameter space samples efficiently. Inspired by the Level-of-Detail (LoD) technique used in computer graphics [12, 13] and multi-scale representation widely used in computer vision [14, 15] , we propose to investigate the potential of scale information in fitting the scene representation with diversified samples. Specifically, we design a multi-scale variational inference method for 3DGS, which increases the diversity of parameter samples by enforcing them to model local spatial areas with multiple scales. To reduce the number of extra parameters, we design a mechanism that spawns finer multi-scale Gaussians from the base Gaussian, which are heavily involved in the rendering process. Instead of creating actual new Gaussians, we maintain an offset table parameterizing only a subset of attributes and share the remainder with base Gaussian. This can further reduce the number of parameter samples by maintaining only the attributes that contribute to our multi-scale representation.Extensive experiments demonstrate the remarkable performance of our method on uncertainty estimation without affecting the rendering quality. We also show that our method provides both accurate posterior and predictive uncertainty estimation. The posterior uncertainty can be utilized directly to remove noisy Gaussians, thanks to the explicit benefit of the 3DGS method. The predictive uncertainty can serve as a confidence map to interpret the synthesized novel views.The main contributions of our work are summarized as follows:• We propose a multi-scale variational inference framework for uncertainty-aware view synthesis with the 3D Gaussian Splatting algorithm. • We develop a spawning strategy to create multi-scale representations for 3DGS, and increase the sample diversity and inference efficiency by maintaining an offset table and sharing parameters. • We evaluate the accuracy of uncertainty estimation on various benchmarks, and demonstrate the application of the parameter uncertainty in cleaning up noisy components in the scene.2 Related Work 
uncertainty quantification: 
 Providing uncertainty together with predictions of learning models is of great use such as interpreting the model output [16, 17] or providing guidance in active data collection [18, 19] . This goal can be achieved by Bayesian learning, which characterizes the predictive distribution and posterior distribution of model parameters with theoretical foundations [20] . One can use Hamiltonian Monte Carlo sampling for Bayesian learning in neural networks [21] , which guarantees asymptotic performance. Previous works proposed other approximation methods such as Laplace's approximation [22] . These methods consider the correlations between parameters, and are known for not requiring further assumption while reducing computations.More recent works borrow from regularization techniques such as the dropout method as an approximated Bayesian inference, such as Monte-Carlo DropOut (MCDO) [7] , Concrete Dropout [23] and Variational Dropout [24] . Another line of work builds model ensembles from various subsets of data [8, 16] , hyperparameters [25] or multiple subnetworks [9, 26] to infer the uncertainty. Some methods also introduce network modulation to form a model ensemble, among which BatchEnsemble [27] learns multiple low-rank weight matrices. Turkoglu et al. [28] introduce a set of linear modulation parameters. Other works focus on extra network components, such as Variational AutoEncoders (VAEs) methods [29] or auxiliary network [30] .The commonality of the above methods is that they introduce randomness into the model and draw model space samples. Furthermore, the importance of diversity of model space samples in approximated Bayesian learning is demonstrated in [11, 31] . Unlike the implicit characteristics of neural networks, the parameters of 3DGS have physical meanings that we can handle explicitly.For algorithms in the multi-view geometry in computer vision, the uncertainty of camera pose can be estimated from the covariance matrix of the transformation matrix using Monte Carlo methods [32] .In the Simultaneous Localization and Mapping (SLAM) system, the uncertainty can be estimated from the Kalman filter [33] . Our method can be aggregated to a dense SLAM system to provide uncertainty information for optimizing camera pose together with the map.Recently, the radiance field has become popular in 3D vision, which achieves great success in generating photo-realistic images of novel view directions from a set of calibrated images. Early research Figure 3 : The pipeline of our variational multi-scale representation. We spawn base Gaussians, which are the major components in the scene, into multi-scale finer Gaussians. We learn an offset table to perform the spawn operation by offsetting a subset of attributes. The offset table is learned with variational inference with multi-scale prior. The predictive and parameter uncertainty can be inferred from the variational parameters stored in the table. developed light field [34] and multiplane images based [35] methods for this task. Building upon previous research on implicit representations [36, 37, 38, 39] , NeRF and its variants achieve a successful framework combining volume rendering and positional encoding to synthesize photorealistic novel views [1, 40, 41] . The recently proposed 3DGS [2] explicitly represents the 3D scene with points associated with the Gaussian function. Some works explore potential solutions to the anti-aliasing rendering problem, for example, Turki et al. [42] develop a multi-resolution feature grid, and Yan et al. [43] design a multi-scale Gaussian with level-of-detail. In our work, we propose a variational multi-scale representation for 3DGS by spreading only local deviations to form multi-scale samples of model parameters. Some previous works focus on uncertainty estimation for NeRF models. Stochastic NeRF [4] addresses the uncertainty estimation in NeRF models via standard variational inference technique. NeRF Ensembles [6] leverages neural network ensemble technique to quantify the predictive uncertainty. CF-NeRF [5] models the predictive distribution via learning a conditional normalizing flow. ActiveRMAP [44] focuses on the active reconstruction task, and models the predictive uncertainty via entropy of ray density distribution. NeurAR and ActiveNeRF [45, 19] adopt the neural network to output uncertainty values for each pixel and learning with Negative Log-Likelihood loss. ProbNeRF [46] designs a learned variational autoencoder that can generate 3D models from 2D images and uses the Monte Carlo method at inference to provide predictive uncertainty. Bayes' Ray [3] performs a post-hoc Laplace's approximation for NeRF to quantify the uncertainty via applying perturbations to spatial points. Compared to the NeRF models, the uncertainty estimation of 3DGS is less discussed. FisherRF [47] design an uncertainty quantification method for 3DGS and apply it to active learning, which adopts Laplace's approximation to compute the uncertainty from Fisher Information. However, the approximation of posterior uncertainty needs extensive computations of gradient aggregating over the whole training views. In our work, we leverage variational inference to approximate the posterior distribution of parameters.3 Proposed Method 
preliminaries: uncertainty quantification for 3d gaussian splatting: 
 The goal of the view synthesis problem is to generate images from any viewpoint given a set of calibrated input images. NeRF [1] proposes to solve this problem by representing the geometry and appearance of the scene using a learned radiance field f (x, d) → (c, σ), where x and d represent a spatial point and view direction, c and σ represent the corresponding color and opacity of that point viewing from d. Based on this radiance field representation, 3D Gaussian Splatting algorithm [2] further proposes that the values in the field can be explicitly stored in a set of ellipsoids parameterized by the Gaussian function. The radiance value of each point in the scene x is queried from its adjacent ellipsoid from the Gaussian function:EQUATIONwhere p and Σ are the center and covariance matrix of the ellipsoid. The covariance is further decomposed to rotation R and scale S by Σ = RSS T R T . We referred to each ellipsoid as Gaussian G. In the rendering process, each Gaussian component is projected to the image space and transformed into its 2D projections. After that, they are accumulated via alpha blending to form pixel values c in the rendered image:EQUATIONNote that the rendering process is deterministic. For each Gaussian G, its learnable parameters θ are composed of position, color, density α, and covariance: θ = {p, c, α, S, R}. The scale S and rotation R are learned respectively and form the covariance matrix Σ in rendering.Standard 3D Gaussian Splatting algorithm applies a non-Bayesian approach to train the model with L 1 loss function, which can be seen as Maximum Likelihood Estimation (MLE) with the error following Laplace distribution [20] . However, this only performs point estimation which lacks the ability to quantify predictive uncertainty and provide confidence information.Previous methods in Bayesian learning provide tools like variational inference and ensemble methods for estimating the predictive uncertainty of models. For example, with ensemble methods, we can train multiple 3DGS models with different data subsets, random seeds or hyperparameters, and compute the variance of their output as the predictive uncertainty. With variational inference methods, the model posterior p(θ|D) is approximated by a tractable variational distribution q(θ), where D is the training data. Then, the likelihood of pixel color p(c|x, θ) at pixel x together with the discrepancy between p(θ|D) and q(θ) is optimized.As discussed in previous works [8, 10, 11] , the effectiveness of approximation methods in Bayesian inference highly depends on the diversity of model parameter space samples. With the explicit attributes of 3DGS parameters, we can manipulate the diversity of parameter samples and explore extensively the model parameter space. With more formations of the model representing the scene explored during learning, we can achieve better approximations in inferring the parameter posterior distribution. In the following, we will introduce a practical approach to building a representation with such diversification ability.Local Multi-scale Representation. Different from NeRF-based scene representation where the model parameters are the neural network parameters with no explicit meaning, 3D Gaussian models the local area of the spatial scene with attributes describing the geometry and appearance. Therefore, we can perform heterogeneous operations for Gaussian attributes by diversifying the scale of the local Gaussian to increase the performance in approximations. In our local multi-scale 3D Gaussian, we propose to learn scene representations with multiple Gaussian scales to represent a local spatial area. Specifically, we first select Gaussians that contribute more to the representation of the scene, and draw new multi-scale samples that are attached to these Gaussians. The parameters of the new Gaussians are shared or learned with variational inference. Through this strategy, we are able to control the scale variance and achieve diversified model space samples θ in our uncertainty estimation.Spawn from Base Gaussians.The pipeline for spawning local multi-scale 3D Gaussians is illustrated in Figure 3 . For every fixed step in training, we perform a spawn operation for Gaussians. Specifically, we would like to select the Gaussians with larger scales and more contributions to the scene representation. We refer to them as base Gaussians, which is found by selecting Gaussians whose scale is larger than a threshold, and the average gradient magnitude received together with the opacity is above certain thresholds in the meantime. These base Gaussians are of greater contributions in the scene representation, which can be replaced by a set of Gaussians with various scales alternatively. Thresholding the gradient magnitude filters out trivial Gaussians such as those located in the distant background where they can hardly be seen.Learn Finer Gaussians via Offset Table . After that, we spawn multi-scale finer Gaussians locally on top of base Gaussians. Instead of creating actual new Gaussians, to reduce the computational cost, we create a table ϕ to store the parameters of offset distribution for K finer Gaussians. We found that some of the attributes of finer Gaussians can be shared as the same as base Gaussians with losing representation ability. We select position p, scale S and opacity α as the attributes maintained in the learned offset table, and the color c and rotation R are shared by the base Gaussian associated. At inference, we apply the sampled offset to the base Gaussian to get the finer Gaussian. For example, the new position p * and scale S * after offset are:EQUATIONwhere χ p and χ S represent position and scale offset values sampled from the offset distribution parameterized by φ . There are total K entries in the offset table ϕ while only M of them are selected randomly with equal probability and averaged to get the final offset distribution parametersφ = 1 M M m ϕ(i m ), where i represents the index for the selected entries. By doing so, we create a subset of finer Gaussians with multiple scales and select a random mixture of their attribute distribution parameters each time. Therefore, we build a random scale alternative to the original large and significant base Gaussians. For the base Gaussians, the densification, splitting, and cloning operations are performed identically to the convention in [2] , and these operations are also performed for the offset table. We will introduce how to learn the offset table with variance in scale using variational inference in the following.The offset table to learn contains entries for K finer Gaussians, and the inference process introduces randomness in selecting the M entries that form the final offset distribution parameters. After obtaining these parameters, we perform variational inference for the offset distribution that enforces the finer Gaussains to be multi-scaled by assigning the prior distribution for the offset. We can infer the uncertainty from the distribution of θ * , the parameter after applying the offset. Specifically, to infer the posterior and learn a multi-scale representation, we let q(χ) be the variational distribution for offset to approximate the true offset posterior distribution p(χ|D) and minimize the Kullback-Leibler divergence between them:EQUATIONwhere p(χ) is the prior distribution for offset χ. For position offset χ p and scale offset χ S , we assume normal distribution and uniform distribution respectively as the prior. Particularly, to learn multi-scale samples, we assume the offset prior distribution with the following parameters:EQUATIONwhere S base is the scale of base Gaussian that the finer Gaussian attached, δ 2 is the prior variance for the position offset. The range of scale after offset for a base Gaussian would be [(1 -1/K)S base , S base ], which means that the scale after offset would vary with the associated base Gaussian. The lower bound of the prior is (1 -1/K) times the base Gaussian scale. This design choice ensures that the offset applied to the base Gaussian generates finer Gaussian with a larger scale range as the number of spawned Gaussians decreases, which means that larger diversity in scale is applied when there are fewer spawned Gaussians. Additionally, we choose zero mean value for the Optimize θ, ϕ with total loss L = L 1 + L SSIM + L KL 10: end while prior of position offset to enforce that the position lies around the base Gaussian, and the prior of opacity offset is given in the Appendix B. We learn the offset table ϕ and with the reparameterization trick [24] . The total loss function isL total = L 1 (ĉ, c) + L SSIM (ĉ, c) + d KL [q(χ)∥p(χ)],where ĉ is the ground truth of color and L SSIM is structural similarity index measure loss introduced in [2] . At inference, predictive distribution is inferred by marginalizing over the offset distribution:EQUATIONwhere p(c | x, D) is the color prediction for pixel x. The pseudo-code of our proposer algorithm in shown in Algorithm 1. 
experiments: 
 We evaluate our uncertainty-aware view synthesis technique on multiple real-world scenes. We compare the quantitative metrics for predictive uncertainty by estimating its correlation with the prediction error. Furthermore, we also validate the quality of our posterior uncertainty by removing Gaussians using different uncertainty threshold levels, which is quite practical for cleaning up and removing noisy regions of the scene, as known as floaters [48] . We will first introduce the datasets, metrics and baseline methods we used for evaluation, then the implementation details of our multiscale variational inference algorithm. Finally, we will present quantitative results on uncertainty estimation, view synthesis and qualitative results on floater removal in the following.Datasets. We use three datasets for evaluation: i) LF dataset [49] contains in total 8 indoor and outdoor scenes, each containing over 100 images from 360 • view. Following the same setting with CF-NeRF [5] , we use images from selected scenes torch, basket, africa, statue for evaluation.ii) LLFF dataset [34] contains 8 forward-facing and outdoor scenes, each containing 20 to 62 images where the camera positions are arranged in a grid pattern. iii) Mip-NeRF 360 dataset [50] contains 6 outdoor scenes, in each scene more than 200 images are captured in 360 • view. The scenes in this dataset are unbounded and contain details region like grass fields, which is challenging for reconstruction. We use this dataset to demonstrate the noisy Gaussian removal ability of our method.Evaluation Metrics. For evaluating uncertainty estimation quantitatively, we first use the Area Under Sparsification Error (AUSE) with Mean Absolute Error (MAE) error. This metric evaluated the correlation between estimated uncertainty and true MAE error in each rendered view. It reorders the pixels according to the estimated uncertainty and calculates the difference between the reordered MAE array and the original ones. We normalize the error when calculating the difference. Secondly, we use the Negative Log-Likelihood (NLL) as a metric, which measures the likelihood of ground truth in the predictive distribution. This metric can evaluate both uncertainty and image quality at Baseline Methods. i) CF-NeRF [5] enables the uncertainty estimation of NeRF models by modeling latent variables and learning a conditional normalizing flows model. ii) S-NeRF [4] applies Bayesian learning to the NeRF model, and is able to quantify both depth and color uncertainty. iii) Bayes' Ray [3] constructs a spatial uncertainty field that can add perturbations to the position input of the radiance field and performs Laplace's approximation. iv) Ensemble GS. In this method, we train 10 3DGS models with different subsets of initialization points from Structure from motion (SfM) [51] . We also use different random seeds for each model. The variance of the predictions between all models is regarded as the predictive uncertainty. Implementation Details. We use an AdamW optimizer to update the learnable parameters of our variational multi-scale representation. The learning rate for 3DGS attributes is the same as the original algorithm [2] . We choose to spawn K = 10 finer Gaussians in the offset table to perform our multi-scale variational inference. The learning rate of the offset table is 0.1 times the learning rate of each attribute. The experiments are performed on a single NVIDIA A100 GPU.We first evaluate the uncertainty of depth on the LF dataset. The calibration between uncertainty and depth error is quantified by AUSE reported in Table 1 . On average, our method achieves performance the best performance. The performance of the ensemble method approaches our method while being superior to other methods. Our method requires much less computational cost compared to the ensemble. In the basket scene, our method largely improves the AUSE metric by 0.9 compared to the second-best ensemble method. Furthermore, we evaluate the uncertainty quality of rendered images on the LF and LLFF datasets. In Table 2 , we report the AUSE and NLL metrics for rendered images. The NLL is estimated by the multivariate kernel density estimator used in [5] . On the LF dataset, our method shows the best AUSE, and outperforms S-NeRF in both metrics. In terms of NLL, CF-NeRF shows comparable results in aligning the predictive distribution with the ground truth. A plausible reason is that CF-NeRF models radiance distribution which helps improve performance. For the results on the LLFF dataset, our method demonstrates the best performance in terms of the NLL metric. We also largely surpass methods other than ensemble in terms of the AUSE metric. Bayes' Ray models spatial perturbation; therefore, only depth uncertainty quality is rendered and evaluated for this method. Our method does not offset color attributes directly to maintain efficiency, while still being able to model the uncertainty of predictive color by offsetting other attributes. We also provide visualizations of the predicted uncertainty map shown in Figure 4 . Our method aligns well with the prediction error, even in detailed regions with complex scene content and overlapping object parts.Apart from the quality of uncertainty, we also compare the quality of synthesized novel views on both LF and LLFF datasets. The results on both datasets demonstrate that our method shows superior image quality. Our method rivals the performance of the ensemble method in rendered image quality, without the need to learn an extensive number of additional parameters for Spherical Harmonics (SH) coefficients that store radiance information. In the LF dataset, our method shows the best quality in terms of SSIM.We provide rendered images of the flower scene in Figure 4 . Our method reconstructs the scene with clear details. The error pixels of the rendered image are located mainly in the region near the edge of the objects in the scene. A possible reason is that the slight error in the camera pose estimated by SfM can cause discrepancies between the rendered image and the ground truth. Some other error pixels are located on small distant object parts, such as small leaves in the background, which are more difficult to reconstruct. The comparison of rendered images demonstrates that our variational multi-scale representation is capable of providing the extra functionality of uncertainty estimation while maintaining the performance of reconstructing various kinds of scenes with high fidelity.When reconstructing unbounded scenes such as those in Mip-NeRF 360, the training camera trajectories are around the foreground object. Therefore, insufficient information regarding the background is contained in the collected images, and the geometry of the distant background object is hardly reconstructed. As a result, the generalization ability over novel views on those regions is poor, and moving testing cameras away from the training trajectory leads to significant drops in visual quality.In order to obtain a high-fidelity radiance field, one can remove these noisy Gaussians manually with editing tools. However, this requires a large amount of human labor.Our uncertainty estimation can be used to automatically remove floaters by deleting Gaussians where their parameters have relatively large posterior uncertainty estimated. The results of floater removal are shown in Figure 5 . We retain 50% and 30% of the Gaussians with smaller posterior uncertainty in their parameters. Viewing from a distant camera, the novel view of the background scene fails to be synthesized due to the lack of the multi-view supervision signal. With the increasing number of removed Gaussians, the noisy Gaussians with large scale and irregular covariance in the background are removed gradually. After removing 70% of the Gaussians in the scene, the floaters in the background scene are mostly elliminated while leaving the clear Gaussians that capture the complete object of interest. By using our method, we can obtain clear foreground objects for further editing and presenting in VR/AR scenes. 
conclusion: 
 In this paper, we have proposed a probabilistic framework to address the uncertainty estimation problem in the 3D Gaussian Splatting algorithm. Different from previous work, we have discovered the merits of building multi-scale presentations to enhance the diversity of parameter space samples when performing Bayesian inference. Our method improves the efficiency of variational inference by parameter sharing and sampling only a subset of attributes of the model. Experimental results have demonstrated the accuracy of our method in uncertainty estimation and its effectiveness in removing floaters. The potential usage of our method can be further explored, such as quality assessment of 3DGS scenes, robotics navigation and guided interactive active data acquisition. 
 
 Appendix: In Section 3.3, we have leveraged the KL-divergence as the optimization goal of the multi-scale offset table with variational inference. However, the objective given by KL-divergence is intractable. Therefore, we have to further transform the objective as follows:d KL [q ϕ (χ)∥p(χ | D)] := χ q ϕ (χ) log q ϕ (χ) p(χ | D) = E qχ(χ) [log q ϕ (χ) -log p(χ | D)] = E q ϕ (χ) [log q ϕ (χ)] -E q ϕ (χ) [log p(χ, D) -log p(D)] = E q ϕ (χ) [log q ϕ (χ) -log p(χ, D)] + log p(D),where ϕ is the parameter of variational distribution, which we stored in the offset. We can transform this objective to the Evidence Lower BOund (ELBO) by considering only the expectation part, and taking the expectation of the term over the data distribution D. This yields a tractable optimization objective to maximize as follows:L(ϕ) = E D,q ϕ (χ) [log p(χ, D) -log q ϕ (χ)] = E D,q ϕ (χ) [log p(D|χ) + log p(χ) -log q ϕ (χ)] = E x,ĉ E χ∼q ϕ (χ) [log p(ĉ = c | x, χ) -d KL (q(χ)||p(χ))] ,where log p(ĉ = c | x, χ) is the log-likelihood of ground truth color, p(χ) is our multi-scale prior distribution, q(χ) is the variational distribution. In our method, we assume Laplace distribution for the predictive variable c, then in practice the loss function to optimize the prediction is the L 1 loss function.To get the opacity offset χ α , we first sample η from the normal distribution, whose parameters are learned and stored in the offset table ϕ. Then, we apply a mapping using sigmoid function with temperature κ to get the opacity offset χ α :EQUATIONThe offset χ α is applied to opacity α by multiplication: α * = α • χ α . To derive the prior distribution for the offset χ α , one can use the change of variable technique. Firstly, the inverse transformation from χ α to η is:EQUATIONThe Probability Density Function (PDF) of η is given by the normal distribution:EQUATIONwhere µ and σ are the parameters for normal distribution. Using the change of variables formula, we have:EQUATION)The Cumulative Distribution Function (CDF) of opacity offset prior χ α is shown in Figure 6 . The intuition for this offset prior is that we want to apply a small perturbation to the opacity in the variational inference. As shown in the CDF, with a positive µ, the odds of offset χ α approaching 1 are high. Also, the sigmoid function is numerically stable in optimization. For computational simplicity, we minimize the KL divergence between the variational distribution and prior of η, instead of χ α . C Rendering Time Analysis The time for inferring a frame using our method compared with the ensemble method is shown in Table 3 . We test on torch scene in the LF dataset with a single NVIDIA A100 GPU, and takes the average time of 1, 000 frames. Ours f ull mean offset with all attributes, Ours p,S,c mean offset with position, scaling and color. Ours p,S,α mean offset with position, scale and opacity, which is the setting of our main experiments. The running time shows that our design choice can achieve the lowest inference time.In active data acquisition of 3DGS, image collection and the 3DGS model training are performed alternately. Our goal is to maximize the model quality with the same number of images used. At each image collection step, the most informative image is selected via an acquisition function, in our case the uncertainty of the rendered image. By using our uncertainty estimation method as the acquisition function, we can indicating where the model is uncertain about and acquiring more data around there.We perform a simple experiment on active data acquisition of 3DGS on the LLFF dataset. Specifically, the original training dataset serves as the candidate image pool, and 10% of images are randomly chosen for training initially. Then, one image is chosen for every 500 steps until in total 30% of images are used. We render our uncertainty map and aggregate the pixel values to choose the most uncertain image from the pool as the next image added to the training set. After all images are chosen, the 3DGS model is further trained for 7K steps. The densification interval is 100 steps, and the spawning interval is 500 steps, and both operations are performed until training ends. As shown in Table 4 , we found that the view synthesis quality of active 3DGS with our uncertainty estimation is better than choosing images randomly. As the experiment setting for active learning is intricate, we prefer to fully investigate the application of our uncertainty estimation on active learning, which is a limitation of this paper. We compare the view synthesis and uncertainty estimation performance using K ∈ 1, 5, 10 number of finer-level Gaussians spawned in the offset table. We train on all 8 scenes in the LLFF dataset and report the average results in Table 5 . We found that improving the number of finer-level Gaussians K shows a notable increase in the quality of uncertainty estimation. More finer level Gaussians improve the sample space diversity, therefore providing precise estimation of model parameter uncertainty and novel views. Additional visualization results of view synthesis and uncertainty estimation of rendered images on scenes from the LLFF dataset using our method are provided in Figure 7 . 


Decision: This paper was reviewed by four experts in the field.  Based on the reviewers' feedback as well as the author feedback, the decision is to recommend the paper for acceptance.  The reviewers did raise some valuable comments that should be addressed in the final camera-ready version of the paper. The authors are encouraged to make the necessary changes to the best of their ability.    We congratulate the authors on the acceptance of their paper!

Reviews: review: paper_summary: paper_summary: This paper proposed a novel multi-scale variational representation for 3D gaussian splatting to estimates its uncertainty. Specifically, this paper introduced a spawn strategy to split a base gaussian into multiple sub gaussians and apply variational inference to estimate the uncertainty.
strength_weakness: strength_weakness: (1), Authors claimed one of the potential application is guided interactive active data acquisition. It would be great to have more detailed illustration or even experimental exploration.
questions: questions: (1), For the floater removal experiment, is removing the background actually the desired effect for real world use case? In my understanding , a good uncertainty-based gaussian pruning approach should be able to remove the blurry gaussians to improve the rendering clarity while preserving most of the objects regardless its foreground or background.

(2), For the related works section, there are some additional related uncertainty quantification works could be discussed:
[1] Naruto: Neural active reconstruction from uncertain target observations. CVPR 2024
[2] Active neural mapping. ICCV 2023
limitations: limitations: The uncertainty qualification requires (additional) sub gaussian construction, might potentially increase the computational burden.
paper_summary: paper_summary: The paper aims to quantify uncertainty in the learning pipeline in 3DGS. To this end, the author(s) proposed to leverage explicit scale information to build variational multiscale 3D gaussians leading to the construction of diversified parameter space samples. This results in the proposition of a multiscale variational inference framework for uncertainty estimation in the 3D Gaussian model. Experimental results on popular benchmark datasets are shown to demonstrate the efficacy of the proposed method.
strength_weakness: strength_weakness: 1. The results are not as impressive as mentioned in the abstract and in the introduction of the paper, i.e., state-of-the-art.

2. The approach to use only scale can cause problems with points that are far from the camera. This is critical particularly for the formulation that takes both scale and rotations to model the covariance —refer to 3DS original work. Kindly comment.

3. Writing of the paper can be improved. 

        a. can not -> cannot

        b. robotics navigation -> robot navigation

        c. less samples of parameters as possible -> parameters samples as few (less) as  possible.

        d. spawn strategy -> not clearly explained while introducing this term for the first time in the paper, i.e.,  what author(s) mean by this term—referring to contribution. Explaining at least briefly here will improve the draft.

        e. Evaluation is generally not considered as contribution (contribution 3)

        f. Section 2.2 line 97 “Rearly…this task”. Kindly rephrase this line, it is confusing in the current form.

        g. Line 228: “the original is”...?

        h. Line 234: Then-> then

        There are more typos… kindly correct them.

        i. Missing references in implicit uncertainty modeling Lee et al. RAL 2022 “uncertainty guided policy for active robotic 3d reconstruction using neural radiance fields”
questions: questions: 1. Kindly clarify how removing uncertain 3D Gaussian still leads to complete scene images visualization on screen (rasterization).

2. Kindly provide details on how distance of the points from the camera affects the uncertainty quantification. 

3. It is well-known that COLMAP 3D and camera pose are not perfect. How structure from motion uncertainty contributes to the current approach. Kindly comment.
limitations: limitations: Although a few limitations are obvious from the experiments section. It is not explicitly mentioned in the paper.
paper_summary: paper_summary: This paper proposes an uncertainty estimation method for the 3D Gaussian Splatting (3DGS) radiance field reconstruction algorithm. The proposed methods leverage the multi-scale properties that lie inherently in a vast amount of Gaussian ellipsoids to improve the performance of variational inference. The paper validates quantitatively that the proposed method aligns with the novel view synthesis error better than previous methods on NeRF uncertainty estimation and naïve methods on 3DGS uncertainty estimation. The experiments also validate an interesting application of the proposed uncertainty estimation methods: to determine noisy Gaussians and remove them thus reduce the floater artifacts.
strength_weakness: strength_weakness: 1. Ambiguous details: i): In line 187, there are K alternative values in the offset table. If so, how many values that the offset table contain for one spawned Gaussian? ii): In Figure 3 (c), how are sampling and inference done from multiple spawned Gaussians? The Figure should contain more details to clarify the difference with other methods. 

2. More ablation studies. i): Replacing the multi-scale prior in equation 6 with the same prior distribution for all layers can be compared to study the performance gain of the multi-scale method. ii): The impact of the quantity of spawned Gaussians in the offset table (line 174) on the reconstruction performance can also be studied.
questions: questions: Please see weakness.
limitations: limitations: More comparing results with simpler methods such as naïve variational inference can be provided.
paper_summary: paper_summary: This work suggests a method to estimate uncertainty for the output of a 3D Gaussian Splatting (3DGS) model, by means of variational calculus.  
Specifically, the authors split the 3DGS model to two hierarchical levels (or- two scales). The coarser "Base" level is very similar to vanilla 3DGS, but now each Gaussian contains a list of parameters $\{ \mu_i, \sigma_i\}$ that are used to spawn the "Finer" level Gaussians.  
The latter "Finer" Gaussians are then fed into the standard 3DGS inference pipeline, only now these Finer Gaussians are fitted using the re-parametrization trick and the ELBO.  
The experimental section compares to other uncertainty estimation methods and shows favorable results.
strength_weakness: strength_weakness: The method seems attractive, but needs to be better positioned w.r.t. previous works:
* FisherRF, (citation [42] in the paper) is mentioned in line 116 but is not compared against. The justification for the lack of numerical comparison is that "the posterior needs extensive computations", but the authors of [42] state they run on a modest card (RTX3090) in 70 FPS. At the very least, results from [42] can be easily added to Table 1 based on Table 4 in [42]. Additionally, the [code](https://github.com/JiangWenPL/FisherRF) of [42] was made public. 
* latentSplat [arxiv 2403.16292](https://arxiv.org/abs/2403.16292) - while it operates under a **simpler setting than this work** , and while it is not yet accepted a peer-reviewed venue (AFAIK), I would recommend mentioning this method to prevent confusion.
questions: questions: High level questions:
* Can the proposed method be applied directly on the **Base** level Gaussians? i.o.w. - why does one needs more than a single Gaussian in the Finer level?
* I would appreciate a deeper discussion on the similarities to [42]. 
---
While the method is rather straightforward, some design choices are not clear to me:
* why is $\mu_n$ in eq(6) is limited to this range?
* why is the spawning method of the Base level different from vanilla 3DGS?
limitations: limitations: The authors do not discuss limitation of their method.

Comments: title: Thanks for the analysis
Thanks for the deatiled reply
Thank you for the response
comment: Thank you for the feedback! 

We will improve the presentation quality, and add the above ablations in the revision.
The rebuttal from the author has solved my concerns. I will keep my rating as accept. The author should address the problems raised in the comments in the revised version.
Dear Reviewer, 

We appreciate your precious time and efforts in reviewing our work! We look forward to addressing any concerns you may have during the remainder of the discussion period. 

Best Regards, 
Authors
Dear Reviewer, 

We appreciate your precious time and efforts in reviewing our work! We look forward to addressing any concerns you may have during the remainder of the discussion period. 

Best Regards, 
Authors
Thank you for the question! 

In our experiments, we found that spawning for more than 10 Gaussians shows a non-significant gain in uncertainty estimation metric. Actually, spawning between 5 and 10 already show a tradeoff between the quality of image and uncertainty, while 10 finer Gaussians lead to better uncertainty quality. Therefore, we chose 10 as the default setting and did not present the result of more Gaussians considering the extra computational cost. However, we only tested on LLFF scenes and the performance on more benchmarks might vary. We will add a more detailed ablation study in the revision on the effect of the spawned gaussian number, as raised by the reviewer [XD3Y, ZTX9].
We sincerely appreciate the feedback from the reviewer! The difference between densification and spawning operations is exactly what you describe above. 

Thank you for your positive score on our work!
What happes for 15 or even 20 finer Gaussians? (I looked for such a table in the paper but didn't find)
The authors have thoroughly addressed all of my questions and concerns, 
and I stand by my positive score.  

---
Indeed, I mistakenly asked about spawning rather than densification.  
If I understand correctly - you create/duplicate new base-Gaussians every 500 iterations, and re-spawn new finer-Gaussians every 4K iterations.
Thank you for affirming our response! We will add more detailed content on active data acquisition to the revision. We welcome discussion on any further questions.
I appreciate the author for the detailed response. This rebuttal resolves most of my concern, I would keep my rating as weak accept. Authors are encouraged to add the active data acquisition part to the main paper or at least the supplementary materials.