Title: Variational Multi-scale Representation for Estimating Uncertainty in 3D Gaussian Splatting
Venue: NeurIPS 2024
Accepted: yes
Keywords: Neural Rendering

Uncertainty Quantification
Abstract: Recently, 3D Gaussian Splatting (3DGS) has become popular in reconstructing dense 3D representations of appearance and geometry. However, the learning pipeline in 3DGS inherently lacks the ability to quantify uncertainty, which is an important factor in applications like robotics mapping and navigation. In this paper, we propose an uncertainty estimation method built upon the Bayesian inference framework. Specifically, we propose a method to build variational multi-scale 3D Gaussians, where we leverage explicit scale information in 3DGS parameters to construct diversified parameter space samples. We develop an offset table technique to draw local multi-scale samples efficiently by offsetting selected attributes and sharing other base attributes. Then, the offset table is learned by variational inference with multi-scale prior. The learned offset posterior can quantify the uncertainty of each individual Gaussian component, and be used in the forward pass to infer the predictive uncertainty. Extensive experimental results on various benchmark datasets show that the proposed method provides well-aligned calibration performance on estimated uncertainty and better rendering quality compared with the previous methods that enable uncertainty quantification with view synthesis. Besides, by leveraging the model parameter uncertainty estimated by our method, we can remove noisy Gaussians automatically, thereby obtaining a high-fidelity part of the reconstructed scene, which is of great help in improving the visual quality.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work suggests a method to estimate uncertainty for the output of a 3D Gaussian Splatting (3DGS) model, by means of variational calculus.  \nSpecifically, the authors split the 3DGS model to two hierarchical levels (or- two scales). The coarser \"Base\" level is very similar to vanilla 3DGS, but now each Gaussian contains a list of parameters $\\{ \\mu_i, \\sigma_i\\}$ that are used to spawn the \"Finer\" level Gaussians.  \nThe latter \"Finer\" Gaussians are then fed into the standard 3DGS inference pipeline, only now these Finer Gaussians are fitted using the re-parametrization trick and the ELBO.  \nThe experimental section compares to other uncertainty estimation methods and shows favorable results.\n\nSTRENGTHS:\nThe method builds on the simplicity of 3DGS, and seems rather straightforward to implement. Given additional prior knowledge on ELBO based methods, the proposed method is rather easy understand from the text (with the small exception mentioned below).  \nThe results show the power of this method compared to other methods (with the one exception mentioned below).\n\nWEAKNESSES:\nThe method seems attractive, but needs to be better positioned w.r.t. previous works:\n* FisherRF, (citation [42] in the paper) is mentioned in line 116 but is not compared against. The justification for the lack of numerical comparison is that \"the posterior needs extensive computations\", but the authors of [42] state they run on a modest card (RTX3090) in 70 FPS. At the very least, results from [42] can be easily added to Table 1 based on Table 4 in [42]. Additionally, the [code](https://github.com/JiangWenPL/FisherRF) of [42] was made public. \n* latentSplat [arxiv 2403.16292](https://arxiv.org/abs/2403.16292) - while it operates under a **simpler setting than this work** , and while it is not yet accepted a peer-reviewed venue (AFAIK), I would recommend mentioning this method to prevent confusion.\n\nQUESTIONS:\nHigh level questions:\n* Can the proposed method be applied directly on the **Base** level Gaussians? i.o.w. - why does one needs more than a single Gaussian in the Finer level?\n* I would appreciate a deeper discussion on the similarities to [42]. \n---\nWhile the method is rather straightforward, some design choices are not clear to me:\n* why is $\\mu_n$ in eq(6) is limited to this range?\n* why is the spawning method of the Base level different from vanilla 3DGS?\n\nLIMITATIONS:\nThe authors do not discuss limitation of their method.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes an uncertainty estimation method for the 3D Gaussian Splatting (3DGS) radiance field reconstruction algorithm. The proposed methods leverage the multi-scale properties that lie inherently in a vast amount of Gaussian ellipsoids to improve the performance of variational inference. The paper validates quantitatively that the proposed method aligns with the novel view synthesis error better than previous methods on NeRF uncertainty estimation and naïve methods on 3DGS uncertainty estimation. The experiments also validate an interesting application of the proposed uncertainty estimation methods: to determine noisy Gaussians and remove them thus reduce the floater artifacts.\n\nSTRENGTHS:\n1. The motivation of this paper is clear. The problem of increasing diversity in the sampling of variational inference is challenging, and the idea of developing a multi-scale representation in 3DGS to solve it is intriguing. \n\n2. The method is sound and easy to follow. Representing the same scene with multiple scale levels is feasible in previous CG techniques. The proposed technique of learning the scene with different scales by assuming different variational distributions, and estimating the predictive uncertainty by sampling from these “multi-scale posteriors” is clear. Designing an offset table to increase inference efficiency is a plus. \n\n\n3. The experiments demonstrate the effectiveness of the method in uncertainty estimation by comparing AUSE and NLL. The synthesized image quality in Table 6 is better than NeRF based method, and the noisy Gaussians removal experiments are interesting.\n\nWEAKNESSES:\n1. Ambiguous details: i): In line 187, there are K alternative values in the offset table. If so, how many values that the offset table contain for one spawned Gaussian? ii): In Figure 3 (c), how are sampling and inference done from multiple spawned Gaussians? The Figure should contain more details to clarify the difference with other methods. \n\n2. More ablation studies. i): Replacing the multi-scale prior in equation 6 with the same prior distribution for all layers can be compared to study the performance gain of the multi-scale method. ii): The impact of the quantity of spawned Gaussians in the offset table (line 174) on the reconstruction performance can also be studied.\n\nQUESTIONS:\nPlease see weakness.\n\nLIMITATIONS:\nMore comparing results with simpler methods such as naïve variational inference can be provided.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper aims to quantify uncertainty in the learning pipeline in 3DGS. To this end, the author(s) proposed to leverage explicit scale information to build variational multiscale 3D gaussians leading to the construction of diversified parameter space samples. This results in the proposition of a multiscale variational inference framework for uncertainty estimation in the 3D Gaussian model. Experimental results on popular benchmark datasets are shown to demonstrate the efficacy of the proposed method.\n\nSTRENGTHS:\n1. The paper solves an extremely useful problem for robot vision and control applications: quantifying uncertainty in 3D Gaussian splatting. Uncertainty modeling is essential for developing robot vision-based automation systems that are robust, safe, efficient, and capable of operating in complex and dynamic environments. By explicitly accounting for uncertainties, this approach can help design more effective and reliable robotic vision systems. Therefore, this paper clearly attempts to address a significant problem.\n\n2. Uncertainty quantification on explicit 3DGS by exploiting both the model space diversity and efficiency. Such a modeling strategy ensures efficient and compact model output.\n\nWEAKNESSES:\n1. The results are not as impressive as mentioned in the abstract and in the introduction of the paper, i.e., state-of-the-art.\n\n2. The approach to use only scale can cause problems with points that are far from the camera. This is critical particularly for the formulation that takes both scale and rotations to model the covariance —refer to 3DS original work. Kindly comment.\n\n3. Writing of the paper can be improved. \n\n        a. can not -> cannot\n\n        b. robotics navigation -> robot navigation\n\n        c. less samples of parameters as possible -> parameters samples as few (less) as  possible.\n\n        d. spawn strategy -> not clearly explained while introducing this term for the first time in the paper, i.e.,  what author(s) mean by this term—referring to contribution. Explaining at least briefly here will improve the draft.\n\n        e. Evaluation is generally not considered as contribution (contribution 3)\n\n        f. Section 2.2 line 97 “Rearly…this task”. Kindly rephrase this line, it is confusing in the current form.\n\n        g. Line 228: “the original is”...?\n\n        h. Line 234: Then-> then\n\n        There are more typos… kindly correct them.\n\n        i. Missing references in implicit uncertainty modeling Lee et al. RAL 2022 “uncertainty guided policy for active robotic 3d reconstruction using neural radiance fields”\n\nQUESTIONS:\n1. Kindly clarify how removing uncertain 3D Gaussian still leads to complete scene images visualization on screen (rasterization).\n\n2. Kindly provide details on how distance of the points from the camera affects the uncertainty quantification. \n\n3. It is well-known that COLMAP 3D and camera pose are not perfect. How structure from motion uncertainty contributes to the current approach. Kindly comment.\n\nLIMITATIONS:\nAlthough a few limitations are obvious from the experiments section. It is not explicitly mentioned in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-10"}

comments: SUMMARY:
This paper proposed a novel multi-scale variational representation for 3D gaussian splatting to estimates its uncertainty. Specifically, this paper introduced a spawn strategy to split a base gaussian into multiple sub gaussians and apply variational inference to estimate the uncertainty.

STRENGTHS:
(1), Estimating uncertainty in the 3D reconstruction / novel view synthesis systems is a critical task and has great application potential.

(2), The paper proposed a novel representation and solid mathematical derivation to solve this problem.

(3), The experiment shows strong relevance between the estimated uncertainty and the rendering error.

WEAKNESSES:
(1), Authors claimed one of the potential application is guided interactive active data acquisition. It would be great to have more detailed illustration or even experimental exploration.

QUESTIONS:
(1), For the floater removal experiment, is removing the background actually the desired effect for real world use case? In my understanding , a good uncertainty-based gaussian pruning approach should be able to remove the blurry gaussians to improve the rendering clarity while preserving most of the objects regardless its foreground or background.

(2), For the related works section, there are some additional related uncertainty quantification works could be discussed:
[1] Naruto: Neural active reconstruction from uncertain target observations. CVPR 2024
[2] Active neural mapping. ICCV 2023

LIMITATIONS:
The uncertainty qualification requires (additional) sub gaussian construction, might potentially increase the computational burden.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-10