3D Gaussian Splatting as Markov Chain Monte Carlo

Authors: shakiba  kheradmand
daniel  rebain
gopal  sharma
weiwei  sun
yang-che  tseng
hossam  isack
abhishek  kar
andrea  tagliasacchi
kwang  moo
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-11 20:27:09

Abstract: While 3D Gaussian Splatting has recently become popular for neural rendering, current methods rely on carefully engineered cloning and splitting strategies for placing Gaussians, which does not always generalize and may lead to poor-quality renderings. For many real-world scenes this leads to their heavy dependence on good initializations. In this work, we rethink the set of 3D Gaussians as a random sample drawn from an underlying probability distribution describing the physical representation of the scene—in other words, Markov Chain Monte Carlo (MCMC) samples. Under this view, we show that the 3D Gaussian updates can be converted as Stochastic Gradient Langevin Dynamics (SGLD) update by simply introducing noise. We then rewrite the densification and pruning strategies in 3D Gaussian Splatting as simply a deterministic state transition of MCMC samples, removing these heuristics from the framework. To do so, we revise the ‘cloning’ of Gaussians into a relocalization scheme that approximately preserves sample probability. To encourage efficient use of Gaussians, we introduce an L1-regularizer on the Gaussians. On various standard evaluation scenes, we show that our method provides improved rendering quality, easy control over the number of Gaussians, and robustness to initialization. The project website is available at https://3dgs-mcmc.github.io/.

Main paper: 
 introduction: 
 Neural rendering has seen a significant advancement with the introduction of Neural Radiance Fields (NeRF) [28] , and more recently, 3D Gaussian Splatting (3DGS) [19] . 3D Gaussian Splatting became highly popular thanks to its speed and efficiency-it can render high-quality images in a fraction of the time required by NeRF. Unsurprisingly, various extensions to 3D Gaussian Splatting have been proposed, such as extending 3D Gaussians to dynamic scenes [43, 41] , making them robust to aliasing effects [48, 42] , and generative 3D content creation [52, 49, 36] .However, despite the various extensions, a common shortcoming of these methods is that they mostly rely on the same initialization and densification strategy for placing the Gaussians: either the one originally suggested by [19] , or other recently proposed variations [4] . Specifically, they rely on carefully engineered cloning and splitting heuristics for placing Gaussians [19, see "adaptive density control"]. Depending on the state of each Gaussian, they are cloned, split, or pruned, which is the primary way to control the number of Gaussians within the 3DGS representation. Moreover, Gaussians are regularly 'reset' by setting their opacities to small values to remove floaters in the representations. This heuristic-based approach requires multiple hyperparameters to be carefully tuned and, as we will show later on in the paper, can fail in some scenes.These heuristics further lead to various problems. It causes the method to heavily rely on good initial point clouds for it to work well, especially when applied to real-world scenes. It is also nontrivial to estimate how many 3D Gaussians will be used for a given scene from just the hyperparameters, making it difficult to control the computation and memory budget in advance without affecting reconstruction quality during inference time. Concurrent works [10, 9] thus focus on having better initialization to solve the former, or study the latter problem [4] . However, even these recent concurrent solutions still rely on heuristics, and they do not always generalize well. In some cases, this leads to sub-optimal placement of Gaussians resulting in poor quality renderings and wasted compute.To solve this problem, we take a step back and rethink the set of 3D Gaussians as random samplesmore specifically, as Markov Chain Monte Carlo (MCMC) samples-drawn from an underlying probability distribution that is proportional to how faithfully these Gaussians reconstruct the scene. With this considered, we show in Section 3.2 that the conventional 3D Gaussian Splatting update rule is very similar to a Stochastic Gradient Langevin Dynamics update [3, 20] , where the only term missing is a noise term that promotes the exploration of samples. We thus reformulate 3D Gaussian Splatting into an SGLD framework, an MCMC algorithm, which naturally explores the scene landscape, and samples Gaussians that are effective in reproducing the scene faithfully. It is important to note that while we assume an underlying distribution, this distribution does not need to be explicitly modelled, as the Gaussians themselves are already representing it.Given this view, the heuristics involved in the densification and pruning of Gaussians, as well as resetting their opacities, are no longer necessary. 3D Gaussians are simply the samples used for MCMC, thus exploring their sample locations is naturally dealt through SGLD updates. Any modification to the set of Gaussians, including increasing or decreasing their cardinality, can be reformulated as a deterministic state transition-relocation of Gaussians-where we move our sample (the set of Gaussians) to another sample (another set of Gaussians with different configuration). Importantly, to minimally disturb the MCMC sampling chain, we make sure that the two states, before and after the move, are of similar probability. This implies that the training loss value does not change as we alter combinatorial properties, such as the number of Gaussians within the representation, preventing the training process from becoming unstable. This simple strategy, under the MCMC framework, is enough to provide renderings of high quality, beyond what is provided by the conventional heuristics.In more detail, we propose to relocate Gaussians using a 'cloning' strategy, where we move 'dead' Gaussians (with low opacity) to where other 'live' Gaussians exist, but in a way that has minimal impact on the rendering, and thus on the probability distribution of the Gaussians. In other words, we set the composition of the cloned Gaussians to render the same images as before cloning. While a modification of cloning [4] was recently suggested as a way to ensure the rendering is equal at the Gaussian centers, we show that this is not enough-the whole Gaussian must be considered. Without our careful strategy MCMC sampling provides sub-optimal training. Finally, to encourage efficient use of the Gaussians, we apply L1-regularization. As the extent of a Gaussian is defined both the opacity and the scale of Gaussians, we apply our regularization to both of them. This effectively encourages them to 'disappear' if unnecessary.We evaluate our method on standard scenes evaluated in [19] (NeRF Synthetic [28] , MipNeRF 360 [2] , Tank & Temples [22] , Deep Blending [16] ), as well as the OMMO [27] dataset that exhibit large scene context. With our method, one does not need to initialize the Gaussians carefully. Our method provides high-quality renderings, regardless of whether Gaussians are initialized randomly or from Structure-from-Motion points.To summarize, our contributions are: • we reveal the link between 3DGS and MCMC sampling, leading to a simpler optimization; • we replace the heuristics in 3D Gaussian Splatting with a principled relocation strategy; • we introduce regularizer to encourage parsimonious use of Gaussians; • we improve robustness to initialization; • we provide higher rendering quality.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes] Justification: Please see Abstract and Introduction.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
related work: 
 Novel-view synthesis via Neural Radiance Fields. Since the introduction of Neural Radiance Fields (NeRF) [28] , it has become extremely popular for building and representing 3D scenes. The core idea behind the method is to learn a neural field that encodes radiance values in the modeling volume, which is then used to render via volume rendering with light rays. Since its first introduction, it has been extended to deal with few views [47] , to generalize to new scenes without training [34, 47] , to dynamic [12] and unbounded scenes [2] , to roughly posed images [39] , to speed up training [29, 46] , and even to biomedical applications [7, 51] to name a few. These extensions are by no means an exhaustive list and demonstrate the impact that NeRF had. For a more in-depth survey we refer the readers to [13] .Amongst works on NeRF, most relevant to ours is [20] , which also employs Stochastic Gradient Langevin Dynamics (SGLD) [3] to identify the most promising samples to train with, and allow faster training convergence. While we ground ourselves in the same Markov Chain Monte Carlo (MCMC) paradigm based on SGLD, the application context is entirely different. In their work, SGLD is used to perform a form of 'soft mining', so to accelerate NeRF training. In our case, we are instead rethinking 3DGS as samples from an underlying distribution that represents the 3D scene.Gaussian Splatting. 3D Gaussian Splatting (3DGS) [19] is a recent alternative to NeRF that rely on differentiable rasterization instead of volume rendering. In a nutshell, instead of querying points along the ray, it stores Gaussians, which can then be rasterized into each view to form images. This rasterization operation is highly efficient, as instead of querying hundreds of points along a light ray to render a pixel, one can simply rasterize the (few) Gaussians associated with a given pixel. Therefore, Gaussian Splatting allows 1080p images to be rendered at 130 frames per second on modern GPUs, which catalyzed the research community.Unsurprisingly, various extensions immediately followed. These include ones that focus on removing aliasing effects [48, 42] , allowing reflection [44] or capture of dynamic scenes [41, 43] , 3D content generation [36, 52, 49] , controllable 3D avatars [23] , and prediction of 3D representations from few-shot images [5] . Methods that focus on more compact representation, thus suitable for rendering on mobile devices, have also been proposed. These methods prune/cluster Gaussians, adaptively selecting the number of spherical harmonics to encode color [32] , and quantize the parameters of the representation [31] . All of these extensions are extremely recent, demonstrating the large interest sparked within the community. With the exception of [5] , one core limitation that these methods share is that they all rely on the original adaptive density control heuristics that 3DGS [19] proposed. As we demonstrate in this work, this does not necessarily always work, and it require either careful initialization or appropriate tuning. Even then, the rendering outcomes may be suboptimal. For additional research, we direct interested readers to a recent survey [6] .Concurrent work. Bulo et al. [4] recently proposed to modify the densification strategy to address issues with cloning Gaussians, as well as densifying Gaussians at locations with high training error. Their method partially addresses the issue with cloning, but it is not enough as we will show in Section 3.4. Their error-based densification is orthogonal to our research direction and could easily be incorporated into our method as well, which we leave to future works. Other works explore how to better initialize through an auxiliary NeRF network [10] or via trained dense geometry estimators [9] , such as [37] . In our case, we tackle the source of the problem and reduce the dependence on initialization itself.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes]Justification: See Appendix C.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
method: 
 We first reformulate Gaussian Splatting as Markov Chain Monte Carlo (MCMC) sampling. We then introduce the new update equations under the MCMC framework with Stochastic Gradient Langevin Dynamics [3, 20] . We then discuss how the heuristics in 3D Gaussian Splatting [19] can be folded into a novel relocalization scheme. Finally, we discuss the L1 regularization that we use to encourage efficient use of the Gaussians, and the implementation details.Before reformulating, we first briefly review 3D Gaussian Splatting [19] for completeness. 3D Gaussian Splatting represents the scene as a set of 3D Gaussians, which are then rasterized into a desired view via α-blending. This can be viewed as an efficient way to perform volume rendering as in NeRFs [28] . Specifically, for a camera pose θ to render a pixel x we order the N Gaussians by sorting them in the order of increasing distance from the camera and writeEQUATIONwhere c i is the color of each Gaussian stored as Spherical Harmonics that are converted to color according to the pose θ, and if we denote their opacity as o, centers as µ, and covariance as Σ,EQUATIONwhere R is the camera projection operation.Then, with the color values for each pixel, Gaussians are trained to minimize the loss:EQUATIONwhere L 1 is the average L1 error between C(x) and the ground-truth colour C gt (x), and L D-SSIM is the Structural Similarity Index Metric (SSIM) [38] between the rendered and ground-truth image.Where λ D-SSIM =0.2 as proposed by [19] .Unlike existing approaches to 3D Gaussian Splatting, we propose to interpret the training process of placing and optimizing Gaussians as a sampling process. Rather than defining a loss function and simply taking steps towards a local minimum, we define a distribution G which assigns high probability to collections of Gaussians which faithfully reconstruct the training images. This choice allows us to leverage the power of MCMC frameworks to draw samples from this distribution in a way that is mathematically well-behaved, even when making discrete changes in the parameter space. As such, we can design discrete operations analogous to the original splitting and pruning heuristics of Gaussian Splatting without breaking the assumptions of continuity that underlie typical gradient-based optimization.To achieve this, we start from the Stochastic Gradient Langevin Dynamics (SGLD) [40, 3] method, which is an MCMC framework that has also recently been applied to novel view synthesis applications [20] . This particular choice is convenient, as SGLD already resembles the commonly used Stochastic Gradient Descent (SGD) update rule, but with additional stochastic noise. Specifically, if we consider the updates of a single Gaussian g in 3DGS, and momentarily ignore their split/merge heuristics:EQUATIONwhere λ lr is the learning rate, and I is an image sampled from the set of training images I. Let us now compare the former to a typical SGLD update:EQUATION)where P is the data-dependent probability density function for the distribution one wishes to sample from, and ϵ is the noise distribution for exploration. The hyperparameters a and b together control the trade-off between convergence speed and exploration 1 We note the striking similarity between (4) and (5) . In other words, by having the loss as the negative log likelihood of the underlying distribution,EQUATIONthe equations become identical if λ lr = -a and b=0. Hence, the standard Gaussian Splatting optimization could be understood as having Gaussians that are sampled from a likelihood distribution that is tied to the rendering quality. We further note that this addition of noise to optimization is highly related to traditional optimization methods that inject noise [35, 30, 8] or that perform perturbed gradient descent [17, 18] . Here, we formulate it as MCMC in favour of the probabilistic relationship that it provides in (6) , and the removal of heuristics that it enables, which we discuss in Sec. 3.4.(a) Cloning in 3DGS [19] (b) Cloning in [4] (c) Our method Figure 1 : Illustration of different respawn strategies -We show a 1D example of rasterizing a Gaussian with opacity 0.95, before and after cloning them into four identical Gaussians and rasterizing them together, with different strategies. Existing methods cannot be used for MCMC as they broaden the extent of the selected Gaussian, significantly violating distribution invariance.Having revealed the link between SGLD and conventional 3DGS optimization, we rewrite (4) as:EQUATIONwhere λ lr and λ noise are the hyperparameters controlling the learning rate and the amount of exploration enforced by SGLD. In practice, instead of the raw gradients ∇ g E I∼I [L total (g; I)], we use the Adam [21] optimizer with default parameters for β 1 and β 2 [25] .In Equation (7) , it is important that the noise term ϵ is designed carefully. The noise term ϵ needs to be added in a way that it can be 'balanced' by the gradient term ∇ gi L total , or otherwise (7) reduces to random updates. For example, it is quite common for Gaussians to be narrow when reconstructing scenes to represent lines and edges. Should the added noise force Gaussians to move to locations outside of the previous support regions, this random walk would be irrecoverable, breaking the MCMC sampling chain.We further notice that exploration is not critical for opacity, scale, and color, and we do not add noise to these parameters. In fact, we empirically found that adding noise to them to slightly harm performance; see Section 4.1. Conversely, it has been shown by [11] that 3DGS reconstruction can be harmed significantly when areas of space are left unexplored, e.g. due to missing points in the initialization. This suggests potential for noise-based exploration to improve results for both random initialization as well as SFM-derived initializations which suffer from missing geometry.Finally, as we are interested in the 'converged' quality not just exploration, we reduce the amount of noise when Gaussians are well-behaved, that is, when their opacities are high enough to be guided well by the gradients. Thus, we design the noise term only on the locations of the Gaussians such that it is dependent on their covariances and also its opacities, as well as the learning rate:EQUATIONwhere η ∼ N (0, I), σ is the sigmoid function, and k and t are hyperparameters controlling the sharpness of the sigmoid, which we set as k=100 and t=(1 -0.005) to make a sharp transition function that goes from zero to one, centered around the default pruning threshold of 3D Gaussian Splatting [19] for the opacity values of Gaussians. In simple terms, (8) perturbs a Gaussian with anisotropic noise with the same anisotropy profile Σ of the Gaussian, while the sigmoid term reduces the effect of noise on opaque Gaussians.Inspired by jump and resampling moves in MCMC [26, 15] , we now discuss how heuristics in 3D Gaussian Splatting can be rewritten as simple state transitions. In 3D Gaussian Splatting, heuristics are used to 'move', 'split', 'clone', 'prune', and 'add' Gaussians to encourage more 'live' Gaussians (o i ≥0.005). 2 We explain all of these modifications moving from one sample state g old to another sample state g new . This applies also to cases where the number of Gaussians changes, as one could think of the state with a smaller number of Gaussians simply as the equivalent state with more Gaussians, but with those that have zero opacity, that is, dead Gaussians. Importantly, for these kinds of deterministic moves to be integrated into MCMC frameworks, it is important that they do not cause MCMC sampling to collapse. Specifically, we aim to preserve the probability of the sample state before and after the move that is, P(g new ) = P(g old ) such that the move can be seen as simply hopping to another sample with equal probability. We now detail how we achieve this.There can be various ways, but we opt for a simple strategy where we move 'dead' Gaussians (o i < 0.005) to the location of 'live' Gaussians. In doing so, we set the parameters of the Gaussians to minimize the difference between the impact on renderings that g new and g old provide. We provide the exact derivation in Appendix A and here we provide the update equation. Without loss of generality, consider moving N -1 Gaussians, g 1,...,N -1 , to g N . Then, denoting the old Gaussian parameters with superscript old and the new ones with new, we writeEQUATIONWhile, at first glance, the strategy we opt for may seem similar to 'cloning' in 3D Gaussian Splatting [19] , the difference (9) brings is critical. In Figure 1 , we illustrate this for a simplified 1D case, when o old N =0.95. As shown, classical cloning and the recently proposed 'centre corrected' version [4] both lead to a significant difference in the rasterized Gaussian as cloning is performed. This is because in (1), the composed opacity is a product of multiple Gaussian shapes and its negation. Both existing strategies lead to the extent of the selected Gaussian growing as shown in Fig. 1 , thus significantly differ in terms of likeness of these states, that is P(g new ) ̸ = P(g old ). This, in fact, leads to sub-optimal training. Implementation. While our method results in P(g new ) ≈ P(g old ), it is not exact. Hence we apply this move every 100 iterations to avoid disruptions in the training process. To choose where to move, for each dead Gaussian, we first chose a target Gaussian to move/teleport to via multinomial sampling of the live Gaussians with the probabilities proportional to their opacity values. Please note that only after all movement decisions have been made we apply (9) . Finally, as we rely on the Adam optimizer, moment statistics should also be adjusted. We reset the moment statistics for the target Gaussian (the original one that is cloned) so that it is biased to stay stationary, while for the new ones (source) we retain the moment statistics to encourage exploration. This is because 'dead' (source) Gaussians are dominated by the noise term in (7) , and the moment statistics are hence appropriate to foster exploration.To make effective use of the memory and compute while improving the performance, we encourage Gaussians to disappear in non-useful locations and 'respawn' elsewhere. As the existence of a Gaussian is effectively determined by its opacity o and covariance Σ, we apply regularization to both of these. Our full training loss is:EQUATIONwhere eig j (.) denotes the j-th eigenvalues of the covariance matrix (the variance along the principle axes of the covariance matrix), and λ o and λ Σ are hyperparameters.We implement our method on top of the 3DGS [19] framework using PyTorch [33] .Gradual increase in the number of Gaussians. As in other works [19, 4] , we allow the number of Gaussians to gradually grow, so that Gaussians are placed at useful locations. We do this simply by initially starting with a selected number of Gaussians, then allowing more 'dead' Gaussians to become 'alive' through our relocation strategy we previously detailed in Section 3.4. Specifically, Table 1 : Quantitative results with same number of Gaussians -Our method outperforms all baselines even when starting from random initialization, with a large gap in performance when compared with 3DGS [19] -Random. We highlight the best and second-best for each column.NeRF Synthetic [28] MipNeRF 360 [2] Tank & Temples [22] Deep Blending [16] we gradually increase the number of live Gaussians by 5% until the maximum desired number of Gaussians is met.Initialization and training. We initialize our samples either randomly or from point clouds, typically from Structure-from-Motion (SfM) as in 3DGS [19] . For random initialization, we follow 3DGS [19] and uniformly random sample 100k Gaussians within three times the extent of the camera bounding box. We also use the same learning rate and learning-rate schedulers to enable comparisons. For the location of Gaussians, we start at a learning rate of 1.6e -4 and decay it exponentially to 1.6e -6 . For all experiments, unless specified otherwise, we use λ noise =5 × 10 5 , λ Σ =0.01, and λ o =0.01. For Deep Blending [16] , we use λ o =0.001. Following 3DGS [19] , we start with 500 warmup iterations, during which we do not perform our relocalization in Sec. 3.4 nor increase the number of Gaussians.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [Yes]Answer: [Yes] Justification: Code is public at the project page.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
experiments: 
 We use various datasets, both synthetic and real. Specifically, as in 3DGS [19] , we use all scenes from NeRF Synthetic [28] dataset, the same two scenes used in [19] of Tank & Temples [22] , and Deep Blending [16] and all publicly available scenes from MipNeRF 360 [2] . We do not use 'Flower' and 'Treehill' scenes from MipNeRF 360 [2] as they are not publicly available. We further use all scenes from the OMMO [27] dataset as in [10] , for the large scenes with distant objects it provides. For MipNeRF 360 [2] , to make our results compatible with [19] , we downsample the indoor scenes by a factor of two, and the outdoor scenes four. For OMMO [27] scene #01, we downsample the images four times to keep the image size reasonable (1000 × 750). For all other scenes, we use the original image resolutions. In the main paper, we report our results by summarizing the average statistics for each dataset. Results for individual scenes, including the standard deviation of multiple runs, can be found in Appendix B. License information for each dataset can be found in Appendix E.Metrics. We evaluate each method using three standard metrics: Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index Metric (SSIM) [38] , and Learned Perceptual Image Patch Similarity (LPIPS) [50] . To account for randomness, we run all experiments three times and average the results.Baselines. We compare against conventional 3DGS [19] , with both random and SfM point cloudbased initialization strategies. We use their official code for our experiments. We also report the original numbers in 3DGS [19] , but where we correct their LPIPS scores, as reported by [4] . We also include state-of-the-art baselines for each dataset.Performance with the same number of Gaussians. As the number of Gaussians is directly related to the quality of novel-view rendering, we first compare our method with existing baselines using the same number of Gaussians as 3DGS [19] . In more details, we simply set the number of Gaussians used in the original 3DGS [19] and set it as our maximum number of Gaussians to be used during training and inference. We show the quantitative results in Table 1 , and qualitative highlights Train from Tank & Temples [22] Playroom from Deep Blending [16] Room from MipNeRF 360 [2] 10 from OMMO [27] Stump from MipNeRF 360 [2] Figure 2 : Qualitative highlights with the same number of Gaussians -We provide examples of novel-view rendering of 3DGS [19] and our approach on multiple scenes from different datasets (with either random or SFM initialization). We highlight the differences in inset figures. Our method faithfully represents details of the various regions thanks to our hybrid MCMC re-formulation that allows exploration without heuristics. Our results provide higher quality reconstructions. Please zoom-in to see details.in Figure 2 . As shown, our method provides better performance than 3DGS [19] . Interestingly, our method, whether initialized randomly or with SfM point clouds, only displays minor variations in performance, thanks to the exploration that our MCMC formulation provides. This allows our method to outperform 3DGS [19] , and regardless of the initialization strategy. Limited budget. We further verify the effectiveness of our formulation by limiting the budget for the number of Gaussians on all the datasets (we omit NeRF Synthetic [28] , as performance on the synthetic dataset is highly saturated, as also highlighted by 3DGS [19] in their initialization experiments). To limit the number of Gaussians for the conventional 3DGS [19] , we simply stop their densification strategy from spawning more points if the limit on the number of Gaussians has been reached. Note that the pruning strategy can cause densification to resume, should some Gaussians get pruned after this threshold is met. Note also that our experiments for this setup is using 3DGS [19] with its default parameters, and do not perform further hyper-tuning. We use the exact same hyperparameters as in Sec. 3.6 for this experiment as well. We report the summary of the results in Fig. 3 . With a limited budget, the gap between our method and 3DGS [19] increases.Sensitivity to initialization. An important benefit of our method is that it allows exploration through the MCMC sampling scheme, removing the heavy reliance of 3DGS [19] on initialization. To verify this, we deviate from the default initialization strategy of 3DGS [19] , which is to randomly place Gaussians within 3× the camera extent as defined by [19] . Instead, we use 1× the camera extent.Table 3 : Ablation study -We report the quantitative metrics without various components of our method, and with the L1 regularizer in 3D Gaussian Splatting [19] . We use the MipNeRF 360 [2] dataset, and random initialization. All components contribute to the final rendering quality.3DGS [19] 3DGS [ Figure 4 : Effect of the noise term (ϵ) -We visualize our reconstruction with (left half) and without (right half) the noise term in (7) . The noise terms are essential to explore the full scene extent. We summarize our results for the average of all scenes in MipNeRF 360 [2] in Table 2 . As shown, our method provides robustness to the initialization strategy, whereas 3DGS [19] requires careful initialization. This is also evident in Section 4.1, where our results with random initialization remain competitive with those initialized by SfM point clouds.Ablations. We further perform ablation studies in Tab. 3. We evaluate whether our regularizers defined on opacity and scale of Gaussians can help conventional 3DGS [19] as well and how essential they are for our method, using the MipNeRF 360 [2] dataset and with random initialization, as it makes their effect easier to observe. In the case of our method, where exploration is encouraged, they are essential to prevent stray Gaussians that shoot off into spaces that are not well updated by the reconstruction loss, e.g., regions outside of the view frustum. The existence of noise is critical to achieving best performance, as without it Gaussians cannot explore the full extent of the scene, as we also illustrate in Fig. 4 . Further, our regularizers are rather harmful when used with classical 3DGS [19] , as they are not compatible with the heuristics proposed therein.We also explore in Tab. 3 adding noise to other parameters. In addition to our location noise in (8), we apply N (0, I) to scale and rotation and N (0, 0.1 × I) for opacity, and decrease them exponentially with a decay rate of 0.9995 for each iteration. This slightly worse performance suggests that the other parameters do not require as much exploration as the Gaussian locations do.On Tank & Temples [22] , we additionally investigate the design choice of our noise scheduler by removing the covariance term and the opacity term. Without the covariance term, we achieve a PSNR of 23.16, whereas with the covariance term 24.21. Without the opacity term, we achieve 22.47, which is also performing worse than with the two terms. We note that removing the opacity term requires λ noise to be set to 0.05 or the model fails to train (PSNR < 7).Finally, we also compare the impact that noise scheduling has. Using the same exponential scheduling as in 3DGS [19] -which is what we use-achieves best performance of 24.21 PSNR, while a linear scheduler gives 17.64. We further compare with the scheduler suggested in [30] , which gives 22.46.As our method results in the same 3D Gaussian Splat representation as prior work, inference time is the same with 3DGS [19] , which is highly efficient. To provide an exact comparison, we took our Gaussians trained at 1M Gaussian, and re-measured the single optimization iteration time for our method and the original 3DGS. In this case, ours takes 80 milliseconds while 3DGS takes 76 milliseconds. That is, the added time for sampling and noise addition is not substantial, even with our implementation that implements resampling naively with PyTorch [33] 's torch.multinomial-this could be further accelerated with a CUDA implementation.Furthermore, the configuration of Gaussians (i.e. where they are, their sizes and opacity) matters greatly to the runtime because they affect the speed of rasterization (among them the opacity regularizer affects the speed the greatest). Hence, we evaluate the runtime for the 'Room' scene in MipNeRF 360 dataset, all with SfM initializations and maximum number of 1.5M Gaussians per the original 3DGS implementation. We report the timings in Tab. 4. As shown, our method, while it may take longer to achieve the highest PSNR, trains faster when a similar PSNR is desired. An independent implementation of our method [45] also confirms a 20% reduction in training time and a 65% reduction in required memory when using our method. 
conclusion: 
 In this paper, we reformulated 3D Gaussian Splatting [19] training as Markov Chain Monte Carlo (MCMC) and implement it via Stochastic Gradient Langevin Dynamics (SGLD). By doing so, we show that we can eliminate the need for point-cloud initialization, and avoid heuristic-based densification, pruning and reset. Not only do we show that this strategy generalizes well across various scenes, outperforming the original 3D Gaussian Splatting [19] , but for the first time we show that this leads to a 3DGS implementation that beats NeRF backbones on the challenging MipNeRF3360 [2] dataset. 
experimental setting/details: 
 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes]Justification: We provide them in Sec. 3 and Sec. 4.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes]Justification: We report all results after running them thrice and report the average. We report the standard deviation in Tab. 5.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean. Justification: We adhere to the NeruIPS Code of Ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: While not strongly applicable as our work focuses on foundational methods for 3D reconstruction, we briefly discuss potential impact in Appendix D.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: Our method is a 3D reconstruction method, hence we are not releasing any model. Hence, the question is not applicable.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: See Appendix E Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix: To have minimal impact on the rendering outcome we propose a strategy that minimizes the difference between the rasterization outcomes of a Gaussian, before and after cloning. For simplicity, let us drop the subscript for the Gaussian index, and consider only the case when a selected Gaussian is to be cloned to multiple copies. We set c new =c old for the cloned Gaussians, and also, as the Gaussians we use in 3D Gaussian Splatting are unnormalized, we set their central C(x) values to be identical before and after the split, that is, C new (µ)=C old (µ). Plugging x = µ in (1), this meansEQUATIONwhich is the same as [4] when N =2.However, (11) is not enough to preserve rasterization as already noted in [4] , for these Gaussians are not point sources-we thus need to alter the covariance Σ of these Gaussians. One way to guarantee minimal impact would be to minimize the mean squared error, that is minimizeEQUATIONBut solving this results in a complex equation without a simple analytical form.We thus opt for an alternative solution, where we consider random 1D slices of Gaussians that pass through their centers, inspired by sliced Wasserstein methods [24] . Given that our Gaussians share their centers, the integral of the rasterization before and after the cloning operation must be equal for rasterization to remain the same for any arbitrary slice. Thus, denoting points along this arbitrary slice S(•) as x = S(x), we instead write minimize∞ -∞ C new (x)dx - ∞ -∞ C old (x)dx 2 2, ∀S.This equation has a simple analytical solution and we can achieve∞ -∞ C new (x)dx = ∞ -∞ C old (x)dx in many ways, including our update equation in (9) . Given this, we now derive how we modify the scales.Without loss of generality, let us consider the mean, µ, of the cloned Gaussian to be zero to simplify the equations further. In addition to the points along the slice x, let us further denote the 'sliced' covariance as Σ = S(Σ). Then, by plugging ( 2) into (1), for the new Gaussians, we can now writeEQUATION)Interestingly, the N -power here can be made even simpler by the fact that for p < 1EQUATIONwhich allows us to rewrite (14) asEQUATION)Table 5 : All results with same number of Gaussians -We report the performance of our method and 3DGS [19] with the same number of Gaussians. Tab. 1 reports the average entries from this table.Our method outperforms 3DGS [19] across the board. We do not include results for non-random initialization on NeRF Synthetic as this dataset does not include COLMAP point clouds.3DGS [19] EQUATIONSolving for Σ new we getEQUATIONSince we want (18) to hold for any arbitrary slice S(•), we can simply replace Σ with Σ, which is then the update equation in (9) .We report all numbers for all scenes in Tab. 5. The standard deviation (std) is computed based on the averages obtained from three different runs, where each average is calculated across all scenes in a dataset for a given seed.While our method allows more robustness to initialization and higher rendering quality thanks to the exploration introduced by MCMC, it is still subject to the same limitations as 3DGS [19] in terms of its modelling capacity. For example, the aliasing issue solved in [48] can also be a problem with our method, as well as modelling reflections [44] . Our method, however, should be compatible with these advancements in Gaussian Splatting, as our method can be viewed as a better training framework for Gaussian Splats. In other words, it should enhance all other Gaussian Splatting methods that are available.While our method is focusing on the core problem of 3D reconstruction, thus not having immediate societal implications, it may, however, have a rippling impact on downstream applications. Because our method reduces the reliance that Gaussian Splatting has on initialization, it may enhance these downstream applications. For example, 3D content generation [36, 52, 49] often suffers from the heuristics in Gaussian Splatting, which we remove. For controllable human modeling [23] , our method also has the potential to enhance its quality. Thus, our method may indirectly enhance what is capable with generative methods, as well as human avatars. Both applications have the potential to be misused. This is a concern with any technology, and we urge users to think about the implications before applying our method.We use the following datasets:• NeRF Synthetic [28] : made available under Creative Commons Attribution 3.0 License. Available at https://www.matthewtancik.com/nerf. • Mip-NeRF 360 [2] : no license terms provided. Available at https://jonbarron.info/ mipnerf360/. • OMMO [27] : no license terms provided. Available at https://ommo.luchongshan. com/. • Deep Blending [16] : no license terms provided. Available at http://visual.cs.ucl.ac. uk/pubs/deepblending/. • Tank & Temples [22] : made available under Creative Commons Attribution-NonCommercial-ShareAlike 3.0 License https://www.tanksandtemples.org/ license/.NeurIPS Paper Checklist38th Conference on Neural Information Processing Systems (NeurIPS 2024).In typical SGLD formalism, b is typically expressed as a function of a and an additional hyper-parameter, but we rewrite it in this form without any loss of generality following[20].This is the default threshold used in3D Gaussian Splatting[19]. 


Decision: The submission received consistently high scores. The reviewers and the authors engaged in positive discussion during the rebuttal phase, and a few of the initial queries were satisfactorily resolved (the biggest one was a misunderstanding of ms wrt time). The proposed solution was found to be simple and effective, removing the heuristic step in the original GS initialization/spawning step. The work is appreciated both due to its theoretical grounding and its practical relevance. 

Please include the additional clarifications while preparing the final version of the paper. Congratulations on the nice piece of work.

Reviews: review: paper_summary: paper_summary: The paper presents a simple and effective method to enhance the training of 3D Gaussian Splatting (3DGS). It offers two main contributions. First, it demonstrates that adding carefully designed noise to the Gaussian centers after each gradient step can boost the performance of 3DGS. This encourages more exploration, which is especially helpful when Gaussian centers are randomly initialized. Second, the method performs densification by replacing low opacity Gaussians with clones of Gaussians sampled through multinomial sampling of the "live" ones based on their opacity values. The parameters of the Gaussians are adjusted to minimize the impact on the rendering outcome. The method is tested on both synthetic and real datasets and shows better performance compared to the 3DGS baseline, regardless of whether the Gaussian centers are randomly or "SFM" initialized.
strength_weakness: strength_weakness: - **Relevance of MCMC framework**: Adding noise to the parameters [1] or to the gradients [2][3] is a common practice in optimization  to escape from saddle points and find local minima. Since the training relies on momentum-based optimizers, I believe the proposed method is more closely related to Perturbed Gradient Descent methods [4][5] or noise injection methods  [1] [2] [3] than to MCMC methods. For a proper momentum based update, noise would typically be added to the momentum estimate instead of the parameters. This doesn't make the method any less relevant or novel. **However, it does make me question the relevance of the MCMC framework that the paper is built around. In which part of the analysis or the method is this  framework necessary ?**
- **The design of the noise term**:  One important contribution of the paper is the design of the noise term. It would be great to have more explanations and ablations about these design choices. Why is this particular choice important for the method? Additionally, how does it compare to simpler noise schedules such as the one in [2]?
- Concerning the update in equation 9 of the paper, please clarify what assumptions and approximations are needed for this derivation and in which case they are no longer valid.


[1] Mark Steijkvers.A recurrent network that performs a context-sensitive prediction task. 1996.

[2] Neelakantan et al.,Adding gradient noise improves learning for very deep networks. 2015

[3]  Deng et al., How shrinking gradient noise helps the performance of neural networks. 2021. 

[4] Jin et al., How to Escape Saddle Points Efficiently. 2017.

[5] Jin et al., On Nonconvex Optimization for Machine Learning: Gradients,
Stochasticity, and Saddle Points.  2021.
questions: questions: - Relevance of the MCMC framework (See Weaknesses).
- How is the term designed and how does it affect the performance?
- What assumptions and approximations are needed for the derivation of equation 9 ?
limitations: limitations: The authors adequately addressed the limitations in the Appendix.
paper_summary: paper_summary: Current 3DGS-based methods require carefully designed strategies such as cloning and splitting to assign a 3D Gaussian at a location. Further, they also require initializing points from SFM to generate high-quality novel views. The proposed work assumes that a set of 3D Gaussians are drawn from an underlying probability distribution, which is representative of the scene. Further, 3DGS updates are converted to SGLD updates by introducing noise.

The main contributions of this work are as follows:
- A fresh perspective that 3D Gaussians are sampled from a distribution and relocation strategy is compatible with MCMC samples.
- Robustness to initialization. 3DGS-MCMC is not dependent on the initialization step in 3DGS.
- The proposed method outperforms other NeRF-based methods and 3DGS on standard datasets.
strength_weakness: strength_weakness: - The proposed method exhibits robustness to the initialization step. However, is there a reduction in training time between the version using SFM points and the version using random initialization?
	
- **Missing evaluation on Scannet++[A1] dataset:** It is a large-scale dataset for indoor scenes. It will be interesting to see how this method performs on this challenging dataset. It is difficult to perform this experiment in a short duration of time. I leave it to authors to decide if they want to include in their manuscript. 

- **Training Time:** The proposed method generates very high-quality novel-views. However, this comes with an added training cost. 3DGS-MCMC takes 90 minutes for 1M Gaussians, whereas 3DGS can be optimized within 30 minutes. The authors mention in L467-468 that a CUDA implementation can accelerate the proposed method. 

[A1] Yeshwanth, C., Liu, Y.C., Nießner, M. and Dai, A., 2023. Scannet++: A high-fidelity dataset of 3d indoor scenes. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 12-22).
questions: questions: - L40-41: "lead to .... waste compute."Can "poor-quality renderings" be substantiated by some examples? Also, can the authors elaborate why it is a wasted compute? InstantSplat[7] reconstructs an unbounded scene with sparse views in under 40 seconds on a commercial GPU. 

- Does this method accurately represent the surface better than 3DGS? Can we extract a high-quality mesh from 3DGS-MCMC? Will it be better than recent works such as 2D Gaussian Splatting[A2]? 

[A2] Huang, B., Yu, Z., Chen, A., Geiger, A. and Gao, S., 2024. 2d Gaussian splatting for geometrically accurate radiance fields. arXiv preprint arXiv:2403.17888.
limitations: limitations: The authors have discussed the limitations of their method in Appendix D in the supplementary material. 
As such, this work has no societal impact. However, some downstream applications can have a societal impact. The authors have discussed this in Appendix E.
paper_summary: paper_summary: The paper discusses improvements to 3D Gaussian Splatting in neural rendering. Current methods rely on complex cloning and splitting strategies for placing Gaussians, which often do not generalize well and depend heavily on good initializations. The authors propose rethinking 3D Gaussians as random samples from an underlying probability distribution of the scene, using Markov Chain Monte Carlo (MCMC) sampling. They show that 3D Gaussian updates can be converted into Stochastic Gradient Langevin Dynamics (SGLD) updates by adding noise. This allows for the removal of heuristic densification and pruning strategies, replacing them with a deterministic state transition of MCMC samples. Additionally, the authors introduce an L1-regularizer on Gaussians to encourage efficient usage. Their method improves rendering quality, provides easy control over the number of Gaussians, and is robust to initialization across various standard evaluation scenes.
strength_weakness: strength_weakness: /
questions: questions: /
limitations: limitations: /
paper_summary: paper_summary: This paper proposes a novel densification strategy of 3D Gaussian Splatting (3DGS) based on the Markov Chain Monte Carlo (MCMC) sampling scheme.
The authors address the ‘heuristic’ densification of standard 3DGS and adopt a distribution-aware resampling pipeline. 
Consequently, they achieve higher rendering quality compared to standard 3DGS with a similar number of Gaussian primitives.
strength_weakness: strength_weakness: - The additional computational costs are required for MCMC resulting in the increase of training time compared to standard 3DGS.

- They only suggest the resampling strategy of 3DGS. Therefore, the technical contribution is inefficient despite the detailed analysis.
questions: questions: - Although they have mentioned the computational time in Sec. B of the appendix, I cannot understand how much it takes compared to standard 3DGS. Can you provide the comparison in training time of this method and standard 3DGS for each scene?

- Although it has been described in L184-186, I am confused about the reason why adding noise to other parameters leads to harmful results. Can you describe more details about it?
limitations: limitations: It only tackles the cloning strategy without any other problems for achieving higher rendering quality. Thus, the technical improvement seems to be inefficient.

Comments: comment: Thank you for the responses. The authors have addressed my concerns in the rebuttal. I think the noise ablation should be included in the  paper. 

Concerning the relevance of the hybrid MCMC framework, I understand how the relocation of Gaussians can be seen as  “jump” moves [B]. However, I don't see how this makes sure that "Gaussians are moved toward these surfaces quickly" since the  “jump” moves happen between "states" which depends on how the reloacation is performed.

Overall, I think this is a good contribution and the authors did a good job addressing my concerns.  I have decided to maintain my rating in support of acceptance.
Thanks for the authors' sincere effort and detailed experiments. I keep my original score.
Thank you for the responses. The authors have addressed all of my concerns in the rebuttal. Thus, I have decided to maintain my rating in support of acceptance.
Thank you for the comments! and thanks again for acknowledging the efficiency of our method.

Regarding your second point, we will for sure include them in the camera ready or the supplementary (if we run out of space)

Thanks,
Authors
I have reviewed the rebuttal and my additional comments are as follows:
- First, I initially misunderstood "ms" as minutes. My mistake. According to the provided table, the proposed method takes 12 minutes longer than 3DGS. However, if fewer Gaussians are used, for example, 300k, the method converges in 21 minutes with a similar PSNR to 3DGS. This demonstrates the high efficiency of the proposed method.

- Secondly, I hope these results, along with any additional experiments, will be included in the supplementary materials of the final camera-ready version.