3D Gaussian Splatting as Markov Chain Monte Carlo
Authors: Shakiba Kheradmand
Daniel Rebain
Gopal Sharma
Weiwei Sun
Yang-Che Tseng
Hossam Isack
Abhishek Kar
Andrea Tagliasacchi
Kwang Moo Yi
Venue: NeurIPS 2024 spotlight (2024)
Keywords: Novel View Synthesis
3D Gaussian Splatting
Primary area: machine_vision

Abstract: While 3D Gaussian Splatting has recently become popular for neural rendering, current methods rely on carefully engineered cloning and splitting strategies for placing Gaussians, which does not always generalize and may lead to poor-quality renderings. For many real-world scenes this leads to their heavy dependence on good initializations. In this work, we rethink the set of 3D Gaussians as a random sample drawn from an underlying probability distribution describing the physical representation of the scene—in other words, Markov Chain Monte Carlo (MCMC) samples. Under this view, we show that the 3D Gaussian updates can be converted as Stochastic Gradient Langevin Dynamics (SGLD) update by simply introducing noise. We then rewrite the densification and pruning strategies in 3D Gaussian Splatting as simply a deterministic state transition of MCMC samples, removing these heuristics from the framework. To do so, we revise the ‘cloning’ of Gaussians into a relocalization scheme that approximately preserves sample probability. To encourage efficient use of Gaussians, we introduce an L1-regularizer on the Gaussians. On various standard evaluation scenes, we show that our method provides improved rendering quality, easy control over the number of Gaussians, and robustness to initialization. The project website is available at https://3dgs-mcmc.github.io/.

3D Gaussian Splatting as Markov Chain Monte Carlo
Shakiba Kheradmand', Daniel Rebain', Gopal Sharma', Weiwei Sun', Yang-Che Tseng', Hossam Isack?, Abhishek Kar?, Andrea Tagliasacchi*> 4,5, Kwang Moo Yi!
Univeessity of British Columbia, Google Research, Google DeepMind, *Simon Fraser University, University of Toronto
https://ubc-vision vision github 10/ /dgs mcmc
Abstract
While 3D Gaussian Splatting has recently become popular for neural rendering. current methods rely on carefully engineered cloning and splitting strategies for placing Gaussians, which can lead poor-quality renderings, and reliance on a random
good initialization. In this work, we rethink the set of 3D Gaussians sample drawn from an underlying probability distribution describing the physical representation of the scene in other words, Markov Chain Monte Carlo (MCMC) samples. Under this view, we show that the 3D Gaussian updates can be converted as Stochastic Gradient Langevin Dynamics (SGLD) update by simply introducing noise. We then rewrite the densification and pruning strategies 3D Gaussian Splatting as simply deterministic state transition of MCMC samples, removing these heuristics from the framework. To so, we revise the 'cloning' of Gaussians into relocalization scheme that approximately preserves sample probability. To encourage efficient use of Gaussians, we introduce regularizer that promotes the removal of unused Gaussians. On various standard evaluation scenes, show that our method provides improved rendering quality, easy control over the number Gaussians, and robustness to initialization.
Introduction
Neural Radiance Fields (NeRF) have been the forefront of these advancements, providing impres- sive results through implicit modeling via neural networks, the landscape is further evolving with the introduction of 31D Gaussian splatting.
Neural rendering has seen significant advancement with the introduction of Neural Radiance Fields (NeRF) (28), and more recently, 3D Gaussian Splatting (3DGS) [19] 3D Gaussian Splatting
became highly popular thanks to its speed and efficieecy--it can render high-quality images in
fraction of the time required by NeRF. Unsurprisingly, various extensions 3D Gaussian Splatting
have been proposed, such as extending 3D Gaussians to dynamic scenes
43 41 making them robust
aliasing effects 48 42, and generative 3D content creation [2
49
36
However, despite the various extensions, a common shortcoming of these methods is that they mostly rely on the same initialization and densification strategy for placing the Gaussians: either the one originally suggested by [19], or other recently proposed variations (4). Specifically, they rely on carefully engineered cloning and splitting heuristies for placing Gaussians [19 see "adaptive density control"]. Depending on the state of each Gaussian, they are cloned, split, or pruned, which is the primary way control the number of Gaussians within the 3DGS representation. Moreover,
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Gaussians are regularly 'reset' by setting their opacities to small values to remove floaters in the representations. This ceuristic-based approach requires multiple hyperparameters be carefully tuned and, as we will show later on in the paper, can fail in some scenes.
These heuristics further lead various problems. It causes the method to heavily rely good initial point clouds for it to work well, especially when applied to real-world scenes. It is also nontrivial to estimate how many 3D Gaussians will be used for given scene from just the hyperparameters, making it difficult to control the computation and memory budget in advance without affecting reconstruction quality during inference time. Concurrent works TO thus focus on having better initialization to solve the former, or study the latter problem 4 However, even these recent concurrent solutions still rely on heuristics, and they do not always generalize well. In some cases, this leads to sub-optimal placement of Gaussians resulting in poor quality renderings and wasted compute.
To solve this problem, take a step back and rethink the set of 3D Gaussians as random samples- more specifically, as Markov Chain Monte Carlo (MCMC) samples-drame from an underlying probability distribution that is proportional to how faithfully these Gaussians reconstruct the scene. With this considered, we show in Section 3.2 that the conventional 3D Gaussian Splatting update rule is very similar to Stochastic Gradient Langevin Dynamics update B 20), where the only term missing is a noise term that promotes the exploration of samples. We thus reformulate 3D Gaussian Splatting into an SGLD framework, an MCMC algorithm, which naturally explores the scene landscape, and samples Gaussians that are effective in reproducing the scene faithfully. It is important to note that while we assume an underlying distribution, this distribution does not need to be explicitly modelled, as the Gaussians themselves ves are already representing it.
Given this view, the heuristics involved in the densification and pruning of Gaussians, as well as resetting their opacities, are no longer necessary. 3D Gaussians are simply the samples used for MCMC, thus exploring their sample locations is naturally dealt through SGLD updates. Any modification to the set of Gaussians, including increasing or decreasing their cardinality, can be of Gaussians- where we move our sample
reformulated as deterministic state (the set of Gaussians) to another sample (another set of Gaussians with different configuration). Importantly, to minimally disturb the MCMC sampling chain, we make sure that the two states, before and after the move, are of similar probability. This implies that the training loss value does not change as we alter combinatorial properties, such as the number of Gaussians within the representation, preventing the training process from becoming unstable. This simple strategy, under the MCMC framework, is enough to provide renderings of high quality, beyond what is provided by the conventional heuristics.
In more detail, we propose relocate Gaussians using a 'cloning' strategy, where we move 'dead' Gaussians (with low opacity) to where other 'live' Gaussians exist, but in a way that has minimal impact on the rendering, and thus on the probability distribution of the Gaussians. In other words, we set the composition of the cloned Gaussians to render the same images as before cloning. While modification of cloning 4 was recently suggested as a way to ensure the rendering equal at the Gaussian centers, we show that this is not enough-the whole Gaussian must be considered. Without our careful strategy MCMC sampling provides sub-optimal training. Finally, to encourage efficient use of the Gaussians, we apply L Ll-regularization. As the extent of Gaussian is defined both the opacity and the scale of Gaussians, we apply our regularization to both of them. This effectively encourages them to 'disappear' if unnecessary.
We evaluate our method on standard scenes evaluated in 19] (NeRF Synthetic 28
MipNeRF 360 22)
Tank & Temples (22), Deep Blending [66), as well as the OMMO (27) dataset that exhibit large scene context. With our method, one does not need to initialize the Gaussians carefully. Our method provides high-quality renderings, regardless of whether Gaussians are initialized randomly or from Structure- Motion points.
To summarize, our contributions are:
we reveal the link between 3DGS and MCMC sampling, leading a simpler optimization;
we replace the heuristics in 3D Gaussian Splatting with a principled relocation strategy;
we introduce regularizer to encourage parsimonious use of Gaussians:
we improve robustness to initialization;
we provide higher rendering quality.
2

---

2 Related Work
Novel-view synthesis via Neural Radiance Fields. Since the introduction of Neural Radiance Fields (NeRF) [28], it has become extremely popular for building and representing 3D scenes. The core idea behind the method is to learn a neural field that encodes radiance values in the modeling volume, which is then used to render via volume rendering with light rays. Since its first introduction, has been extended to deal with few views [47. to generalize to new scenes without training [44 47 to
dynamic [12] and unbounded scenes 2 to roughly posed images [39,, to speed up training 29 46].
and even to biomedical applications 7 S to name few. These extensions are by means an exhaustive list and demonstrate the impact that NeRF had. For a more in-depth survey we refer the
readers to 13
Amongst works on NeRF, most relevant to ours is 20), which also employs Stochastic Gradient Langevin Dynamics (SGLD) [3] to identify the most promising samples to train with, and allow faster training convergence. While we ground ourselves in the same Markov Chain Monte Carlo (MCMC) paradigm based on SGLD, the application context is entirely different. In their work, SGLD used to perform a form of 'soft mining'. so to accelerate NeRF training. In our case, we are instead rethinking 3DGS as samples from an underlying distribution that represents the 3D scene.
Gaussian Splatting. 3D Gaussian Splatting (3DGS) [9] is recent alternative to NeRF that rely on differentiable rasterization instead of volume rendering. In a nutshell, instead of querying points along the ray. it stores Gaussians, which can then be rasterized into each view form images. This rasterization operation is highly efficient, as instead of querying hundreds of points along light ray to render a pixel, one can simply rasterize the (few) Gaussians associated with a given pixel. Therefore, Gaussian Splatting allows 1080p images to be rendered at 130 frames per second on modern GPUs, which catalyzed the research community.
Unsurprisingly, various extensions immediately followed. These include ones that focus on removing 42
aliasing effects 48
allowing reflection [ capture of dynamic scenes [4] 43 3D content
generation 336
52
49 controllable 3D avatars (23). and prediction of 3D representations from
few-shot images 5 Methods that focus on more compact representation, thus suitable for rendering on mobile devices, have also been proposed. These methods prune/cluster Gaussians, adaptively selecting the number of spherical harmonics encode color [32], and quantize the parameters of the representation [31]. All of these extensions are extremely recent, demonstrating the large interest sparked within the community. With the exception of (5). one core limitation that these methods share is that they all rely on the original adaptive density control heuristics that 3DGS [9] proposed. As we demonstrate in this work, this does not necessarily always work, and it require either careful initialization or appropriate tuning. Even then, the rendering outcomes may be suboptimal. For additional research, we direct interested readers to a recent survey [6]
Concurrent work. Bulo et al. recently proposed to modify the densification strategy to address issues with cloning Gaussians, well densifying Gaussians locations with high training error. Their method partially addresses the issue with cloning, but it is not enough as we will show in Section 3.4 Their brror-based densification is orthogonal to our research direction and could easily be incorporated into our method as well, which we leave to future works. Other works explore how to better initialize through an auxiliary NeRF network [10] or via trained dense geometry estimators (9) such as 37 In our case, we tackle the source of the problem and reduce the dependence on initialization itself.
3 Method
We first reformulate Gaussian Splatting as Markov Chain Monte Carlo (MCMC) sampling. We then introduce the new update equations under the MCMC framework with Stochastic Gradient Langevin Dynamics 3 20 We then discuss how the heuristics in 3D Gaussian Splatting [19 can be folded into a novel relocalization scheme. Finally, we discuss the I. regularization that we use to encourage efficient use of the Gaussians, and the implementation details.
3

---

3.1 Brief review of 3D Gaussian Splatting
Before reformulating, we first briefly review 3D Gaussian Splatting 19 for completeness. 3D Gaussian Splatting represents the scene as set of 3D Gaussians, which are then rasterized into desired view via a-blending. This can be viewed as an efficient way to perform volume rendering as in NeRFs (28). Specifically, for camera pose & to render pixel x we order the N Gaussians by sorting them the order of increasing distance from the camera and write
c(x)= = E (x)((x)
II oz ox(x))
where c; is the color of each Gaussian stored as Spherical Harmonics that are converted to color according to the pose 0, and if we denote their opacity o, centers μr, and covariance Σ,
oe (x) = exp( (= [x
R(,, () Ro(E.)) R(p 01).
where R is the camera projection operation.
Then, with the color values for each pixel, Gaussians are trained to minimize the loss:
Corig (1 AD-SSIM) Li + AD-SSIM CD-SSIM:
(3)
where & is the average LI error between C(x) and the ground-truth colour Cgi(x), and Lp sSIM is the Structural Similarity Index Metric (SSIM) 38 between the rendered and ground-truth image.
Where Äp =02 as proposed by [19].
3.2 3D Gaussian Splatting as Markov Chain Monte Carlo (MCMC)
Unlike existing approaches to 3D Gaussian Splatting, we propose interpret the training process of placing and optimizing Gaussians as sampling process. Rather than defining a loss function and simply taking steps towards local minimum, we define distribution G which assigns high probability to collections of Gaussians which faithfully reconstruct the training images. This choice allows us to leverage the power of MCMC frameworks to draw samples from this distribution in way that is mathematically well-behaved, even when making discrete changes in the parameter space. As such, we can design discrete operations analogous to the original splitting and pruning heuristics of Gaussian Splatting without breaking the assumptions of continuity that underlic typical gradient-based optimization.
To achieve this, we start from the Stochastic Gradient Langevin Dynamics (SGLD) 440 3 method, which is an MCMC framework that has also recently been applied to novel view synthesis appli- cations [20). This particular choice is convenient, as SGLD already resembles the commonly used Stochastic Gradient Descent (SGD) update rule, but with additional stochastic noise. Specifically, we consider the updates of single Gaussian g in 3DGS, and momentarily ignore their split/merge heuristics:
g g+ X1e V Ei-I [Fronal (g: I)],
(4)
where A is the learning rate, and I is an image sampled from the set of training images I. Let us now compare the former to a typical SGLD update:
g g+g+o Vg log P(g)+
(5)
where P is the data-dependent probability density function for the distribution one wishes to sample from, and is the noise distribution for exploration. The hyperparameters and together control the trade-off between convergence speed and exploration[] We note the striking similarity between (4) and (5). In other words, by having the loss as the negative log likelihood of the underlying distribution,
g = P exp(-fuoul).
the equations become identical if A - a and b=0. Hence, the standard Gaussian Splatting optimization could be understood as having Gaussians that are sampled from likelihood distribution that is tied to the rendering quality. We further note that this addition noise to optimization highly related to traditional optimization methods that inject noise [35] 30 g or that perform perturbed gradient descent [17 [8]. Here, we formulate as MCMC favour of the probabilistic relationship that it provides in 66. and the removal of heuristics that it enables, which we discuss in Sec. 3.4
In typical SGLD formalism, b is typically expressed as a function of and an additional
but rewrite it in this form without any loss of generality following 20

---

( Cloning SDGS
Cloning
(c) method
Figure Illustration of different respawn strategies We show ID example of rasterizing a Gaussian with opacity 0.95, before and after cloning them into four identical Gaussians and rasterizing them together, with different strategies. Existing methods cannot be used for MCMC as they broaden the extent of the selected Gaussian, significantly violating distribution invariance.
3.3 Updating with Stochastic Gradient Langevin Dynamics
Having revealed the link between SGLD and conventional 3DGS optimization, we rewrite (4) as:
g g Avr VglEr~I [Crocal (g: 1)] + Anoise 6,
where Åir and Amoise the hyperparameters controlling the learning rate and the amount of exploration enforced by SGL.D. In practice, instead of the raw gradients Vg EgwZ (Froaa (g: I)]. we use the
Adam [21] optimizer with default parameters for and 32 (25
In Equation (7. is important that the noise term e designed carefully. The noise term e needs to be added in way that it can be 'balanced" by the gradient term V gg. Cumal or otherwise 7 reduces to random updates. For example, it quite common for Gaussians to be narrow when reconstructing scenes to represent lines and edges. Should the added noise force Gaussians to move locations outside of the previous support regions, this random walk would be irrecoverable, breaking the MCMC sampling chain.
We further notice that exploration not critical for opacity, scale, and color, and we do not add noise these parameters. In fact, we empirically found that adding noise them to slightly harm performance; see Section 4.1 Conversely, it has been shown by [II] that 3DGS reconstruction can be harmed significantly when areas of space are left unexplored, e.g. due to missing points in the initialization. This suggests potential for noise-based exploration to improve results for both random initialization well as SFM-derived initializations which suffer from missing geometry.
Finally, as we are interested in the 'converged' quality not just exploration, we reduce the amount of noise when Gaussians are well-behaved, that is, when their opacities are high enough to be guided well by the gradients. Thus, we design the noise term only on the locations of the Gaussians such that it is dependent on their covariances and also its opacities, as well as the learning rate:
E = o a-k((o-t))) En where
E c le 0).
(8)
where 77 NN(O I), is the sigmoid function, and It and are hyperparameters controlling the sharpness of the sigmoid, which we set as k=100 and t=(1 - 0.005) to make a sharp transition function that goes from zero one, centered around the default pruning threshold of 3D Gaussian Splatting (19] for the opacity values of Gaussians. In simple terms, (8) perturbs a Gaussian with
anisotropic noise with the same anisotropy profile of the Gaussian, while the sigmoid term reduces the effect of noise on opaque Gaussians.
3.4 Heuristics as state transitions via relocation
we now discuss how heuristics in 3D Gaussian Splatting can be rewritten as simple state transitions. In 3D Gaussian Splatting, heuristics are used to 'move". 'split', 'clone", 'prune", and 'add' Gaussians encourage more 'live' Gaussians (o,20.005) We explain all of these modifications moving from sample state gold to another sample state g" new This applies also to cases where the number Gaussians changes, as one could
Inspired by jump and resampling moves in MCMC 26 15
This is the default threshold used in 3D Gaussian Splatting 19

---

think of the state with a smaller number of Gaussians simply as the equivalent state with more Gaussians, but with those that have zero opacity, that is, dead Gaussians. Importantly, for these kinds of deterministic moves to be integrated into MCMC frameworks, it is important that they do not cause MCMC sampling to collapse. Specifically, we aim to preserve the probability of the sample state before and after the move that is, (eg P(g such that the move can be seen as simply hopping to another sample with equal probability. We now detail how we achieve this.
There can be various ways, but we opt for simple strategy where we move 'dead' Gaussians (o: < 0.005) to the location of 'live' Gaussians. In doing so, we set the parameters of the Gaussians to minimize the difference between the impact on renderings that g"ew and gold provide. We provide the exact derivation in Appendix and here we provide the update equation. Without loss of generality, consider moving N 1 Gaussians, 81.....N- to gN. Then, denoting the old Gaussian parameters with superscript old and the new ones with new, we write
MI,..,N N MNN
old
TEE I..... N
= 1
ood
N i-1
(o
-2 NN
Mnew L = (ox)))
ΣΣ k=
k
Vk+1
While, at first glance, the strategy we opt for may seem similar to 'cloning' in 3D Gaussian Splat- ting [19], the difference (9) brings is critical. In Figure [I, we illustrate this for simplified ID case, when -0.955 As shown, classical cloning and the recently proposed 'centre corrected' version 44] both lead to significant difference in the rasterized Gaussian as cloning is performed. This is because [ the composed opacity is product of multiple Gaussian shapes and its negation. Both existing strategies lead to the extent of the selected Gaussian growing as shown in Fig, 1 thus significantly differ in terms of likeness of these states, that is P(g"eew * P(g This, in fact, leads to sub-optimal training.
Implementation. While our method results in P(gnew) P(gald) it not exact. Hence we apply this move every 100 iterations to avoid disruptions in the training process. To choose where to move, for each dead Gaussian, we first chose a target Gaussian to move/teleport to via multinomial sampling of the live Gaussians with the probabilities proportional to their opacity values. Please note that only after all movement decisions have been made we apply ( Finally, we rely on the Adam optimizer, moment statistics should also be adjusted. We reset the moment statistics for the target Gaussian (the original one that is cloned) so that is biased to stay stationary, while for the new ones (source) we retain the moment statistics to encourage exploration. This is because 'dead' (source) Gaussians are dominated by the noise term in 77. and the moment statisties are hence appropriate to foster exploration.
3.5 Encouraging fewer Gaussians
To make effective use of the memory and compute while improving the performance, we encourage Gaussians to disappear in non-useful locations and 'respawn' elsewhere. As the existence of Gaussian is effectively determined by its opacity o and covariance E, we apply regularization to both of these. Our full training loss is:
Listal (1 >D.MSSMM Li + AD.SSIM LD.SSIM + Ao Elod) loch Ax Elveig, (Z.), (10)
where (.) denotes the j-th eigenvalues of the covariance matrix (the variance along the principle axes of the covariance matrix), and A and A are hyperparameters.
3.6 More implementation details
We implement our method on top of the 3DGS [19 framework using PyTorel
33
Gradual increase in the number of Gaussians. As in other works 19 we allow the number of Gaussians to gradually grow, so that Gaussians are placed at useful locations. We do this simply by initially starting with a selected number of Gaussians, then allowing more 'dead" Gaussians to become 'alive' through our relocation strategy we previously detailed in Section 3.4 Specifically,

---

Table 1: Quantitative results with same number of Gaussians Our method outperforms all baselines even when starting from random initialization, with a large gap in performance when
compared with 3DGS [9] Random. We highlight the best and second-best for each column.
NeRF Synthetic 28
MipNeRF 360 2
Tank & Temples [22] Deep Blending [16
OMMO 27
PSSRT ISINI ILPISS
PSNK1 ssse LLPIS.
PSNRT LIIIS)
PENRY /558M1 (LMPS)
PANRT /55IMT LLMPS)
NcRF
29
31.01/
24.85 0.66 0043
21.18 0.34
Plenoxels
46
31.76/
23.63 0.67 /0.4
21.08 /0.72 70.38
INGP-Big
29
33,18/
26.75 / 0.75 0.30
21.92 /0.31
MipNeRIF
1
33,09/
27.60 0.81 0.25
21.4/0078/00
MipNeRF360
29.23 / 0.84 0.21
22.22 /0.76 0,22
3DGS 19
33.32/
28.69 / 0.87 / 0.22
/..44 /0.21
3DGS 19
(Random)
33.42 / 0.97 0.04
27.89 / 0.84 0.26
21.93 /0.79 /0,27
29.55 0.90
0.33
28.24 / 0.88 0.24
Ours
(Random)
33.80 0.97 0.04
29.72 0.89 /..19 0.19
24.21 0.86 0.19
29.71 0.90
0.32
29.3/ / 0.90 0.20
3DGS
19
(SIM)
29.30 0.88 / 0.21
23.67 / /0.84 0.22
29.64 / 0.90
0.32
28.83 / 0.89 0.22
Ours
(SIM)
29.89
0.90 / 0.19
24.29 0.86 0.19
29.67 / 0.89
0.32
29.52 / 0.91 0.20
we gradually increase the number of live Gaussians by 5% until the maximum desired number of Gaussians is met.
Initialization and training. We initialize our samples either randomly from point clouds, typically from Structure- senecture-from-aae from-rooooeoooo Motion (SfM) as in 3DGS [19]. For random initialization, we follow 3DGS [99] and uniformly random sample 100k Gaussians within three times the extent of the camera bounding box. We also use the same learning rate and learning-rate schedulers to enable comparisons. For the location of Gaussians, For all experiments, unless specified otherwise, we use Anoise 55 x 10 =x=0.01, and 0.00..1. For Deep Blending [16), we use A =0.001. Following 3DGS [19], we start with 500 warmup iterations,
start learning rate of 1.6e and decay it exponentially to 1.6e-6.
during which we do not perform our relocalization in Sec. 3.4 nor increase the number of Gaussians.
4 Experiments
We use various datasets, both synthetic and real. Specifically, as in 3DGS [9]. we use all scenes from NeRF Synthetic (28) dataset, the same two scenes used in [19 of Tank & Temples [22]. and Deep Blending [16) and all publicly available scenes from MI 360 (2). We do not use 'Flower' and 'Trechill' scenes from MipNeRF 360 2 as they are not publicly available. We further use all scenes from the OMMO 27 dataset as in 10), for the large scenes with distant objects it provides. For MipNeRF 360 (2) to make our results compatible with [19., downsample the indoor scenes by a factor of two, and the outdoor scenes four. For OMMO 27 scene #01, we downsample the images four times to keep the image size reasonable (1000 750). For all other scenes, we use the original image resolutions. In the main paper, we report our results by summarizing the average statistics for each dataset. Results for individual scenes, including the standard deviation of multiple runs, can be found in Appendix B License information for each dataset can be found in Appendix E
Metrics. We evaluate each method using three standard metrics: Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index Metric (SSIM) (38,, and Learned Perceptual Image Patch Similar- ity (LPIPS) [50]. To account for randomness, run all experiments three times and average the results.
Baselines. We compare against conventional 3DGS 19 with both random and SfM point cloud- based initialization strategies. We use their official code for our experiments. We also report the original numbers in 3DGS [19), but where we correct their LPIPS scores, reported by 4.. We also include thee-of- baselines for each dataset.
4.1 Results
Performance with the same number of Gaussians. As the number of Gaussians is directly related to the quality of novel-view rendering, we first compare our method with existing baselines using the same number of Gaussians as 3DGS [19. In more details, we simply set the number of Gaussians used the original 3DGS [19] and set as our maximum number of Gaussians to be used during training and inference.
We show the quantitative results in Table I and qualitative highlights
7

---

Random
3DGS
3DGS
SFM
Oure Random
Ours SFM
Train from Tank Temples
Playroom from Deep Blending [66]
Room from MipNeRF 360
10 from OMMO 27
Stump from MipNeRF 360
Figure 2: Qualitative highlights with the same number of Gaussians We provide examples of novel-view rendering of 3DGS [19] and our approach on multiple scenes from different datasets (with either random or SFM initialization). We highlight the differences faithfully represents details of the various regions thanks to our hybrid MCMC re- fe-fommulation that allows exploration without heuristics. Our results provide higher quality reconstructions. Please zoom-in to see details.
inset figures. Our method
in Figure 2 As shown, our method provides better performance than 3DGS [19 Interestingly, our method, whether initialized randomly or with SIM point clouds, only displays minor variations in performance, thanks to the exploration that MCMC formulation provides. This allows method to outperform 3DGS 19) and regardless of the initialization strategy.
Limited budget. We further verify the effectiveness of our formulation by limiting the budget for the number of Gaus- sians all the datasets (we omit NeRF Synthetic (28), performance on the synthetic dataset highly saturated, also highlighted by 3DGS [9 in their initialization experiments). To limit the number of Gaussians for the conventional 3DGS [19), simply stop their densifica- tion strategy from spawning more points the limit on the number of Gaussians has been reached. Note that the pruning strategy can cause densification to resume, should some Gaussians get pruned after this threshold is met. Note also that our experiments for this setup is using 3DGS [19 with its default parameters, and do not perform further hyper-tuning. We use the exact same hyperparam- eters in Sec. 3.6 for this experiment as well. We report the summary of the results in Fig. 3 With a limited budget, the gap between our method and 3DGS [19 increases.
28
27
3DGS (Random)
(Random)
Figure 3: Varying the #Gaussians We report the PSNR of 3DGS [19] and method averaged over all datasets (ex- cept NeRF Synthetic).
Sensitivity to initialization. An important benefit of our method is that it allows exploration through the MCMC sampling scheme, removing the heavy reliance of 3DGS [19] initialization. To verify this, we deviate from the default initialization strategy of 3DGS [19) which to randomly place Gaussians within 3x the camera extent as defined by [9 Instead, use 1 1x the camera extent.

---

Table 3: Ablation study We report the quantitative metries without various components of method, and with the L1 regularizer in 3D Gaussian Splatting [19]. We use the MipNeRF 360 2 dataset, and random initialization. All components contribute the final rendering quality.
3DGS 19
Ours foxit
Ours
Ours
Ours Full Method
3DGS 19
Lintal
A =
Noise all param.
PSNRT PSSI LLAMS)
PSNIR SUIMY LLUUS)
PSMOC SSINET LIUIS
PSNKE / ssuse Liters
PSNEC ISSMI ruPUSS
PSNN SSUpp LiNiPs)
27.89/ / 0.84 / 0.26
23.84/ / 0.77 /0.3
23.90/ 0.67 /0,42 27.41 / 0.83 0.26
29.11 0. / 0.24
29.72 0.89 / 0.19
(A) 'Rieycle' from MigNeRF 360
(b) 'Stmmpp from MiRNeRF 360
2
(c) '0S' from OMMO
23
Figure 4: Effect of the noise term (6) We visualize our reconstruction with (left half) and without
(right half) the noise term in The noise terms essential explore the full scene extent.
We summarize our results for the average of all scenes in MipNeRF 360 2 Table 2 As
Table 2: Initialization ablation Our method
provides similar performance regardless of the initialization strategy, whereas the performance of the original 3DGS [19] differs significantly.
shown, our method provides robustness to the
initialization strategy, whereas 3DGS [19] re- quires careful initialization. This is also evident in Section [4.]. where our results with random initialization remain competitive with those ini- tialized by SfM point clouds.
cameea exient
camera extent
3DGS
19
(Random)
27.88 37.89/ 0.84 / 0.26
22.72 0.75//0...0.
Ours
(Random)
0.19
Ablations. We further perform ablation studies in Tab. 3
We evaluate whether our regularizers defined on opacity and scale of Gaussians can help conventional 3DGS [9] as well and how essential they are for our method, using the MipNeRF 360 (2) dataset and with random initialization, as it makes their effect easier to observe. In the case of our method, where exploration is encouraged, they are essential to prevent stray Gaussians that shoot off into spaces that are not well updated by the reconstruction loss, e.g., regions outside of the view frustum. The existence noise is critical to achieving best performance, as without it Gaussians cannot explore the full extent of the scene, as we also illustrate in Fig. Further, our regularizers are rather harmful when used with classical 3DGS (19, as they are not compatible with the heuristics proposed therein.
We also explore in Tab. adding noise other parameters. In addition to our location noise 88. we apply (0, I) to scale and rotation and N (O, 0.1 x I) for opacity, and decrease them exponentially with decay rate of 0. 0.9995 for each iteration. This slightly worse performance suggests that the other parameters do not require much exploration as the Gaussian locations do.
On Tank& & Temples 222 we additionally investigate the design choice of our noise scheduler by removing the covariance term and the opacity term. Without the covariance term, we achieve a PSNR of 16, whereas with the covariance term 24.21. Without the opacity term, we achieve 22.47, which is also performing worse than with the two terms. We note that removing the opacity term requires Anoise to be set to 0.05 or the model fails to train (PSNR < 7).
Finally, we also compare the impact that noise scheduling has. Using the same exponential scheduling as in 3DGS [9] which is what we use achieves best performance of 24.21 PSNR, while linear scheduler gives 17.64. We further compare with the scheduler suggested in (30, which gives 22.46.
4.2 Computational time
As our method results in the same 3D Gaussian Splat representation as prior work, inference time the same with 3DGS [19), which highly efficient. To provide an exact comparison, we took our Gaussians trained at IM Gaussian, and re-measured the single optimization iteration time for

---

Table 4: Timings- We report the timings of our method with various opacity regularizer (A,) settings and the maximum number of Gaussians to 300k. Our method can deliver faster training while still maintaining reconstruction quality.
3DGS [99
Ours =0.01)
Ours (A,=0.001)
Ours (A, =0,=0.01, 300k)
PSNR Time
31.7 25 minutes
32.5 42 minutes
32.4 30 minutes
31.8 21 minutes
method and the original 3DGS. In this case, ours takes 80 milliseconds while 3DGS takes 76 milliseconds. That is, the added time for sampling and noise addition not substantial, even with our implementation that implements resampling naively with PyTorch 33 torch mult multinooiall. this could be further accelerated with a CUDA implementation.
Furthermore, the configuration of Gaussians (i.e. where they are, their sizes and opacity) matters greatly to the runtime because they affect the speed of rasterization (among them the opacity regularizer affects the speed the greatest). Hence, we evaluate the runtime for the 'Room' scene in MipNeRF 360 dataset, all with SfM initializations and maximum number of 1.5M Gaussians per the original 3DGS implementation. We report the timings in Tab. 4 As shown, our method, while it may take longer to achieve the highest PSNR, trains faster when a similar PSNR is desired. An independent implementation of our method [45 also confirms 20% reduction in training time and 65% reduction in required memory when using our method.
5 Conclusion
In this paper, we reformulated 3D Gaussian Splatting 19] training as Markov Chain Monte Carlo (MCMC) and implement it via Stochastic Gradient Langevin Dynamies (SGLD). By doing so, we show that we can eliminate the need for point-cloud initialization, and avoid heuristic- based densification, pruning and reset. Not only do we show that this strategy generalizes well across various scenes, outperforming the original 3D Gaussian Splatting [9]. but for the first time we show that this leads 3DGS implementation that beats NeRF backbones on the challenging MipNeRF3360 [2 dataset.
Acknowledgements We would like thank Charatan al. [5], the inspiration deal with 3DGS optimization as a distribution was inspired by their work. Similarly, we would like to thank Goli al. [14], the notion of adding noise to Gaussians stemmed from wanting explore their positional null-space. We would also like to thank Sergio Orts Escolano and Federico Tombari for their feedback. This work was supported in part by the Natural Sciences and Engineering Research Council of Canada (NSERC) Discovery Grant, NSERC Collaborative Research and Development Grant, Google, Digital Research Alliance of Canada, and Advanced Research Computing at the University of British Columbia, and by the SFU Visual Computing Research Chair program.
References
[1] Barron, J.T., Mildenhall, B., Tancik, M., Hedman, P., Martin-Brualla, R., Srinivasan, P.P.: Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields. In: Int. Conf. Comput. Vis. (2021)
[2] Barron, J.T., Mildenhall, B., Verbin, D., Srinivasan, P.P., Hedman, P.: Mip-nerf 360: Unbounded anti-aliased neural radiance fields. In: IEEE Conf. Comput. Vis. Pattern Recog. (2022)
[3] Brosse, N., Moulines, E., Durmus, A.: The Promises and Pitfalls of Stochastic Gradient Langevin Dynamics. In: Adv. Neural Inform. Process. Syst. (2018)
[4] Bulò, S.R., Porzi, L.. Kontschieder, P.: Revising densification in gaussian splatting. arXiv preprint (2024)
[5] Charatan, D., Li, S., Tagliasacchi, A., Sitzmann, V.: pixelsplat: 3d gaussian splats from image pairs for scalable generalizable 3d reconstruction. arXiv preprint (2023)
10

---

[6] Chen, G., Wang, W.: A survey on 3d gaussian splatting. arXiv preprint (2024)
[7] Corona-Figueroa, A., Frawley, J., Bond-Taylor, S., Bethapudi, S., Shum, H.P.H., Willcocks, C.G.: Mednerf: Medical neural radiance fields for reconstructing 3d-aware ct-projections from a single X-ray (2022)
[8] Deng, Z., Huang, J., Kawaguchi, K.: How shrinking gradient noise helps the performance of neural networks. In: IEEE International Conference on Big Data (Big Data) (2021)
[9] Fan, Z., Cong, W., Wen, K., Wang, K., Zhang, J., Ding, X., Xu, D., Ivanovic, B., Pavone, M., Pavlakos, G., Wang, Z., Wang, Y.: Instantsplat: Unbounded sparse-view pose-free gaussian splatting in 40 seconds (2024)
[10] Foroutan, Y., Rebain, D., Yi, K.M., Tagliasacchi, A.: Does gaussian splatting need sfm initial- ization? arxiv preprint (2024)
[11] Foroutan, Y., Rebain, D., Yi, K.M., Tagliasacchi, A.: Evaluating alternatives sfm point cloud initialization for gaussian splatting
wwtp:////rrrabb////4/////// 2404. 12547
[12] Gafni, G., Thies, J. Zollhofer, M., Niessner, M.: Dynamic neural radiance fields for monocular 4D facial avatar reconstruction. https:/arriv.orgabb//21122.22..0
(2020)
[13] Gao, K., Gao, Y., He. H., Lu, D., Xu, L., Li, J.: Nerf: Neural radiance field in vision, a comprehensive review. arXiv preprint (2022)
[14] Goli, L.., Reading, C., Sellan, S., Jacobson, A., Tagliasacchi, A.: Bayes' rays: Uncertainty quantification for neural radiance fields. In: IEEE Conf. Comput. Vis. Pattern Recog. (2024)
[15] Green, P.J.: Reversible jump markov chain monte carlo computation and bayesian model determination. Biometrika 82(4), 711-732 (1995)
[16] Hedman, P., Philip, J., Price, T., Frahm, J.M., Drettakis, G., Brostow, G.: Deep blending for free- viepoint image- based rendering. ACM Trans. Graph. (2018)
[17] Jin, C., Ge. R., Netrapalli, P., Kakade, S.M., Jordan, M.I.: How to escape saddle points efficiently. In: Int. Conf. Mach. Learn. (2017)
[18] Jin, C., Netrapalli, P., Ge, R., Kakade, S.M., Jordan, M.I.; On nonconvex optimization for machine learning: Gradients, stochasticity, and saddle points. Journal of the ACM (JACM) 68(2), 1-29 (2021)
[19] Kerbl, B., Kopanas, G., Leimkuhler, T., Drettakis, G.: 3d gaussian splatting for real-time radiance field rendering. ACM Transactions Graphics (2023)
[20] Kheradmand, S., Rebain, D., Sharma, G., Isack, H., Kar, A., Tagliasacchi, A., Yi, K.M.: Accelerating Neural Field Training via Soft Mining, In: IEEE Conf. Comput. Vis. Pattern Recog. (2024)
[21] Kingma, D.P., Ba, J.: Adam: A method for stochastic optimisation. In: Int. Conf. Learn. Represent. (2015)
[22] Knapitsch, A., Park, J., Zhou, Q.Y., Koltun, V.: Tanks and temples: Benchmarking large-scale scene reconstruction. ACM Trans. Graph. (2017)
[23] Kocabas, M., Chang, J.H.R., Gabriel, J., Tuzel, O., Ranjan, A.: Hugs: Human gaussian splats. arXiv preprint (2023)
[24] Kolouri, S., Nadjahi, K., Simsekli, U., Badeau, R., Rohde, G.: Generalized sliced wasserstein distances. In: Adv. Neural Inform. Process. Syst. (2019)
[25] Li, J., Wang, W. Ji, H.: Self-supervised deep learning for image reconstruction: A langevin monte carlo approach. SIAM Journal on Imaging Sciences (2023)
[26] Lindsey, M., Weare, J., Zhang, A.: Ensemble markov chain monte carlo with teleporting walkers. SIAM/ASA Joumna Uncertainty Quantification (2022)
11

---

[27] Lu, C.. Yin, F., Chen, X., Liu, W., Chen, T., Yu, G., Fan, J.: A large-scale outdoor multi modal dataset and benchmark for novel view synthesis and implicit scene reconstruction. In: Int. Conf. Comput. Vis. (2023)
[28] Mildenhall, B., Srinivasan, P.P., Tancik, M., Barron, J.T., Ramamoorthi, R., Ng. R.: Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM (2021)
[29] Muller, T., Evans, A., Schied, C., Keller, A.:
Instant neural graphics primitives with multireso- lution hash encoding, ACM Transactions on Graphics (2022)
[30] Neelakantan, A., Vilnis, L., Le, Q.V., Sutskever, I., Kaiser, L., Kurach, K., Martens, J.: Adding gradient noise improves learning for very deep networks. arXiv preprint (2015)
[31] Niedermayr, S., Stumpfegger, J., Westermann, R.: Compressed 3d gaussian splatting for accelerated novel view synthesis (2023)
[32] Papantonakis, P., Kopanas, G., Kerbl, B., Lanvin, A., Drettakis, G.: Reducing the memory foot- print of 3d gaussian splatting, Proceedings of the ACM on Computer Graphics and Interactive Techniques (2024)
[33] Paszke, A., Gross, S., Massa, F. Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al.; Pytorch: An imperative style, high-performance deep learning library. In: Adv. Neural Inform. Process. Syst. (2019)
[34] Rematas, K., Mruall- R., Ferrari, V:: Sharf: Shape-conditioned radiance fields from a single view. In: Int. Conf. Mach. Learn. (2021)
[35] Steijvers, M., Grünwald, P.: recurrent network that performs a context- context-sensitive prediction task. In: Proceedings of the Eighteenth Annual Conference of the Cognitive Science Society (1996)
[36] Tang, J., Ren, J., Zhou, H., Liu, Z., Zeng, G.: Dreamgaussian: Generative gaussian splatting for efficient 3d content creation. arXiv preprint (2023)
[37] Wang, S., Leroy, V., Cabon, Y., Chidlovskii, B., Revaud, J.: Dust3r: Geometric 3d vision made
easy. In: CVPR (2024)
[38] Wang, Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P.: Image quality assessment: From error visibility to structural similarity. IEEE Trans. Image Process. (2004)
[39] Wang, Z., Wu, S., Xie, W., Chen, M., Prisacariu, V.A.: NeRF--: Neural radiance fields without known camera parameters. arXiv preprint (2021)
[40] Welling, M., Teh, Y.W.; Bayesian learning via stochastic gradient langevin dynamics, In: Int. Conf. Mach. Learn. (2011)
[41] Wu, G., Yi, T., Fang. J., Xie, L., Zhang, X., Wei, W., Liu, W., Tian, Q., Wang, X.: 4d gaussian splatting for real-time dynamic scene rendering. arXiv preprint (2023)
[42] Yan, Z., Low, W.F. Chen, Y., Lee, G.H.: Multi-scale 3d gaussian splatting for anti-aliased rendering. arXiv preprint (2023)
[43] Yang, Z., Gao, X., Zhou, W. Jiao, S., Zhang, Y., Jin, X.: Deformable 3d gaussians for high- fidelity monocular dynamic scene reconstruction. arXiv preprint (2023)
[44] Ye, K., Hou, Q., Zhou, K.: 3d gaussian splatting with deferred reflection. arXiv preprint (2024)
[45] Ye, Li, R., Kerr. J., Turkulainen, M., Yi, B., Pan, Z., Sciskari, O., Yc, J., Hu, J., Tancik, M.,
Kanazawa, A.: gsplat: An Open-Source Library for Gaussian Splatting, arXiv preprint (2024)
[46] Yu, A., Li, R., Tancik, M., Li, H., Ng. R., Kanazawa, A.: PlenOctrees for real-time rendering of neural radiance fields. In: ICCV (2021)
12

---

[47] Yu, A., Ye, V., Tancik, M., Kanazawa, A.: pixelnerf: Neural radiance fields from one or few images. In: IEEE Conf. Comput. Vis. Pattern Recog. (2021)
[48] Yu, Z., Chen, A., Huang, B., Sattler, T., Geiger, A.: Mip-splatting: Alias-free gaussian splatting. arXiv preprint (2023)
[49] Yuan, Y., Li, X., Huang, Y., De Mello, S., Nagano, K., Kautz, J., Iqbal, U.: Gavatar: Animatable 3d gaussian avatars with implicit mesh learning. arXiv preprint (2023)
[50] Zhang, R., Isola, P., Efros, A.A., Shechtman, E.,, Wang, O.:
The Unreasonable Effectiveness of
Deep Features as a Perceptual Metric. In: IEEE Conf. Comput. Vis. Pattern Recog. (2018)
[51] Zhong, E.D., Lerer, A., Davis, J.H., Berger, B.: Cryodrgn2: Ab initio neural reconstruction of 3d protein structures from real cryo-em images. In: Int. Conf. Comput. Vis. (2021)
[52] Zou, Z.X., Yu, Z., Guo, YCC. Li, Y., Liang, D., Cao, Y.P., Zhang, S.H.: Triplane meets gaussian splatting: Fast and generalizable single-view 3d reconstruction with transformers. arXiv preprint (2023)
13

---

Appendix
A Derivation for the cloning strategy
To have minimal impact on the rendering outcome we propose a strategy that minimizes the difference between the rasterization outcomes of a Gaussian, before and after cloning. For simplicity, let us drop the subscript for the Gaussian index, and consider only the case when selected Gaussian is to be cloned to multiple copies. We set c"ew=ee" for the cloned Gaussians, and also, as the Gaussians we use in 3D Gaussian Splatting are unnormalized, we set their central C(x) values to be identical before and after the split, that is, Che (μ)=Cald (pt). Plugging x μ in (I. this means
(1 - o N = 1 - oold
(11)
which is the same as when N=2.
However, [I not enough to preserve rasterization as already noted 44, for these Gaussians are point sources- we thus need to alter the covariance Σ of these Gaussians. One way to guarantee minimal impact would be to minimize the mean squared error, that is
minimize
[| - CCold( dx,
(12)
But solving this results in a complex equation without simple analytical form.
We thus opt for an alternative solution, where we consider random ID slices of Gaussians that pass through their centers, inspired by sliced Wasserstein methods (24.. Given that our Gaussians share their centers, the integral of the rasterization before and after the cloning operation must be equal for rasterization to remain the same for any arbitrary slice. Thus, denoting points along this arbitrary slice S-) as x = (xx), we instead write
| CREw (z) )x L (rld(r)
minimize
SS.
(13)
This equation has simple analytical solution and we can achieve _ Cnew (x)dx
1 (r) in many ways, including our update equation in 1 Given this, we now derive how we modify the scales.
Without loss of generality, let consider the mean, μa, of the cloned Gaussian to be zero to simplify the equations further. In addition to the points along the slice x, let us further denote the 'sliced' covariance Σ = S(E). Then, by plugging (2) into [I) for the new Gaussians, can now write
x²
Cnew (z)
new exp
new exp
2 new
25
j=1
(14)
new exp
new
x²
1
new
exp
2
x²
Interestingly, the N-power here can be made even simpler by the fact that for < 1
N
(1-p)
Σ
N
(b)
(15)
k
k=0
which allows to rewrite [4]
x²
(k+1)
(-
mo
k+1
exp
k
2y
(16)
(_(k: (k++ + 1)x²
(1)0 exp
k
nnww
14

---

Table 5: All results with same number of Gaussians We report the performance of our method and 3DGS [19] with the same number of Gaussians. Tab. rrports the average entries from this table. Our method outperforms 3DGS [99] across the board. We do not include results for non-random initialization on NeRF Synthetic as this dataset does not include COLMAP point clouds.
3DGS 19 (Random) rSNRy /SMM [1PS]]
3DGS [9] (SIM)
Ours (Random)
Ours (SfM)
PSNRY /5SM/
PSKR sSM PPSS
PSNR (SIM) (1.151]
Mic
35.35 / 0.99 0.01
37.29 / 0.99 0.01
NeRF Synthetic 28
Ship
30.93 / 0. 90 0.13
30.82/ / 0.91 00.1
Lego
35.84 0.98 0.02
36.01 0.98 0.02
Chair
36.17 0.99 0.02
36.51 0.99 / 0.02
Materials
30.00 0.96 0.04
30.59 0.96 / 0.04
Hotdog
37.83 / 0,99 / 0.03
37.82/ / 0.99 0.02
Drums
26.22 / 0,95 / 0.04
26.29 / 0.95 / 0.04
Ficus
35.01 0,99 / 0.01
35.07 0.99 / 0.01
Average
33.42 0.97 / 0.044
33 80 / 0.97 / 0.04
Std
0.0076 0,0000 / 0.0000
0.0105 0.0000 0.0000
Counter
28.09 88 0.30
29.12 0.91
0.91 0.24
29.16/ 0.92 0.23
29.51 /0.22
MipNeRF 360 ?
Stump
23.91 / 0.68 / 00.33
26.99 0.78 0.24
27.67 0.82 / 0.20
27.80 0.82
Kitchen
30.54 / 0.92 0.
31,58 0.93 14.14
32.23 9.94 0.14
32,27 / 10.14
Bicycle
24.57 7. / 0.33
25.64 0,.64/0.78// 0.22
26.06 / 0.81 / 0.19
26.15 / 0.81 70.18
Bonsai
30,94/ 0.93 0.26
32.32/0.3 0.95 / 0.24
32.67 0,95 / 0.23
32.88 0.95/ /0.22
Room
30.04 0.90 0.32
31,70/ 0,93 0.27
32.30 0.94 / 0.25
32.48 0.94
Garden
27.16 0.86 0.14
77.73/0.81 0.12
27.99 0.88 0011
28.16 1/166/0.8 0.89 00.10
Average
27.89 0 0.26
29.30/ 0.88 0.21
29.72 0.89 0.19
29.89/ 0.90 0..19
Std
0 0,0021 0,0016
0.0276 / 0.0003 0.0003
0.0246 0.0001 /0,0003
0.0154/ 0.0001 0.0001
Trmple H
Train
21.24 / 0,77 / 00.00
21.94 0.81 / 0.2
22.40/ 0.83 0.224 0.24
22.47 / 0.83 / 0.24
Truck
22.63 0.82 0.22
25.40 / 0.88 0.18
26.02 / 0.89 00.14
26.11 /0889 0.14
Average
21.93 / 0,80 10.80/00.2 0.27
23.67 0,84 0.22
24.21 0.86 0.19
24.29/ / /.86 / 0,19
Std
0.0521 / 0.0008 0.0007
0.0639
0,0004 0.0004
0.0729 0.0007 0.0007
0.0688 0.0004 0.0008
Dr Johnson
28.94 0. /.334
29.14 0.90/ 0.33
29.05 0.89 0..3
29.00 / 0.89 0.33
Playroom
30.16 / 0,90/ 0.32
30.15 / 0.90 0.32
30.37 / 0.90 0.31
30.33 / 0.90 0.31
Average
29.55 / 0.90 0.33
29.64/ 0.90/ 0.32
29.71/ 0.90 0.32
29.67 / 0.89 0.32
Std
0.0688 / 0.0007 0,0010
0.0487/
0.0003 0,0002
0.0556 0,0015 / 0.0010
0.0458/ 0.0022 0,0022
01
25.13 0.77 /0227
25.64/ 0,79 0.25
25.90/ 0.80 0.23
25.89/ / 0.80 0.22
25.61 / 0,86 0.27
25.80 0,87 0.26
27.38 0,89 0.22
27.62 0,90 0,22
27.66 0. 86 0.29
28.37 0.87 0.28
28.81 0.88 0.27
28.87 0.88 0.27
06
27.12/ 0.91 0.25
26.79 0.91 / 0.25
27.52 0,94 / 0.20
27.65 / 0.94 / 0.
OMMO
10
29.64 0.87 0.26
29.99 / 0.89 / 0..3
31.20 / 0.90 0.22
31.51 / 0.91 / 0.20
13
31.60 0.92 / 0.22
12.75/0.94 0.18
32.65 0,94 0.18
33.14 / 0.95 0.16
14
30.33 0.93 0.19
30.87 /.994 / 0..17
31.03 0.94 00.7
31.26 0.94 0.16
15
28.79 0.900 0.19
30.46 0.93 0.16
29.96 0.93 0.16
30.25 0.93 00.55
Average
28.24 0.88 0.24
28.83 0.89 0.22
29.31 / 0.90 0.20
29,52 0.91 /,200
Std
0.0265 / 0.0006 0.0007
0.0299
0.0004 0.0003
0.0233 0.0000 0.0001
0.0226 0.0001 0.0003
Finally, as S exp(-ax") = v w/a for some value a, plugging (6) into _ 1 (old(x) we write
Cnew (n)w())
N Σ
2n Nnew
(o
k+1 old old, oldd/2272 2
(17)
k
1=1 O=
Solving for Deew we get
NN
i 1 (-
Inew ((ool))
Σ
yold
(18)
k
+1
Since we want 18) to hold for any arbitrary slice S(-), we can simply replace with E, which is then the update equation in (9)
15

---

B Detailed results
We report all numbers for all scenes in Tab. 5 The standard deviation (std) is computed based on the averages obtained from three different runs, where each average is calculated across all scenes in dataset for given seed.
C Limitations and future work
While our method allows more robustness to initialization and higher rendering quality thanks to the exploration introduced by MCMC, is still subject to the same limitations as 3DGS [19] in terms of its modelling capacity. For example, the aliasing issue solved in [48] can also be problem with method, as well as modelling reflections [44]. Our method, however, should compatible with these advancements in Gaussian Splatting, as method be viewed better training framework for Gaussian Splats. other words, should enhance all other Gaussian Splatting methods that are available.
D Broader impact
While our method is focusing on the core problem of 3D reconstruction, thus not having immediate societal implications. may, however, have a rippling impact downstream applications. Because our method reduces the reliance that Gaussian Splatting has initialization, it may enhance these downstream applications. For example, 3D content generation 36
often suffers from the heuristics in Gaussian Splatting, which we remove. For controllable human modeling [23], our method also has the potential to enhance its quality. Thus, our method may indirectly enhance what is capable with generative methods, as well human avatars. Both applications have the potential to be misused. This is concern with any technology, and we urge users to think about the implications before applying our method.
52 40
E
Dataset licenses
We use the following datasets:
NeRF Synthetic (28) made available under Creative Commons Attribution 3.0 License.
Available at httpp::/ww..attthewettaaaaa
com/nerf
Mip-NeRF 360
2 no license terms provided. Available at https://jonbarroo.
mipnerf360/
OMMO 27 no license terms provided. Available at
com/
Deep Blending [16]: license terms provided. Available at htttpp//visuull.. a.s.uul.er aak/pubs/ddepblenddee
Tank & Temples [2]: made available under Creative Commons Attribution
NonCommercial -iiaeerrill-SaarAAlkke 3.0 License
https://wwwn tanksandtemples..com org/
license/
16

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Please see Abstract and Introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: See Appendix
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
17

---

Justification:
We provide a mathematical derivation on part of our algorithm in Appendix
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification:
We provide all hyperparemeters and details required to reproduce the paper in
Sec.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
18

---

Answer: [Yes]
Justification:
Code is public at the project page.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https guwdes/dde/eeee Cooeeeoodeesssseeeeee 7/nips.com / casssiinnccccyyc for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification:
We provide them Sec. and Sec.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We report all results after running them thrice and report the average. We report the standard deviation in Tab.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
19

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We report the computational time in Sec. 4.2
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics http:p///..aarr....ooooooo///aaa
Answer: [Yes]
Justification:
We adhere to the NeruIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: While not strongly applicable our work focuses on foundational methods for 3D reconstruction, we briefly discuss potential impact in Appendix D
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
20

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Answer: [NA]
Justification: Our method is a 3D reconstruction method, hence we are not releasing any model. Hence, the question is not applicable.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: See Appendix
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the Datasets.paaersstt paperswi thcode. cow/ datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
21

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do not introduce new assets.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
We do not do any user studies nor involve human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
We do not do any user studies nor involve human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
22


Reviews:
[{"review_id": "V4Um8yaP2p", "reviewer": "Reviewer_Cpa7", "summary": "The paper presents a simple and effective method to enhance the training of 3D Gaussian Splatting (3DGS). It offers two main contributions. First, it demonstrates that adding carefully designed noise to the Gaussian centers after each gradient step can boost the performance of 3DGS. This encourages more exploration, which is especially helpful when Gaussian centers are randomly initialized. Second, the method performs densification by replacing low opacity Gaussians with clones of Gaussians sampled through multinomial sampling of the \"live\" ones based on their opacity values. The parameters of the Gaussians are adjusted to minimize the impact on the rendering outcome. The method is tested on both synthetic and real datasets and shows better performance compared to the 3DGS baseline, regardless of whether the Gaussian centers are randomly or \"SFM\" initialized.", "questions": "- Relevance of the MCMC framework (See Weaknesses).\n- How is the term designed and how does it affect the performance?\n- What assumptions and approximations are needed for the derivation of equation 9 ?", "limitations": "The authors adequately addressed the limitations in the Appendix.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper tackles an important problem (optimization, cloning and splitting strategies for placing Gaussians) and introduces relevant concepts and ideas to analyze it.\n- The paper proposes a simple and easy-to-implement way to improve the optimization of Gaussian positions by encouraging more exploration.\n- The proposed relocation strategy works effectively  and pairs nicely with the L1-regularizer on the Gaussians.\n- **Evaluation**: The evaluation is performed on various types of scenes (Nerf Synthetic, MipNeRF 360, Tank & Temples, Deep Blending, OMMO) and performance is reported with and without \"SFM\" initialization. Reporting the average over 3 runs and the corresponding standard deviation  is a good practice for the reliability of the results.\n- **Performance**: The provided results show that the proposed method improves over the 3DGS baseline. More importantly, the method obtains competitive results without initializing the Gaussians with SFM points.", "weaknesses": "- **Relevance of MCMC framework**: Adding noise to the parameters [1] or to the gradients [2][3] is a common practice in optimization  to escape from saddle points and find local minima. Since the training relies on momentum-based optimizers, I believe the proposed method is more closely related to Perturbed Gradient Descent methods [4][5] or noise injection methods  [1] [2] [3] than to MCMC methods. For a proper momentum based update, noise would typically be added to the momentum estimate instead of the parameters. This doesn't make the method any less relevant or novel. **However, it does make me question the relevance of the MCMC framework that the paper is built around. In which part of the analysis or the method is this  framework necessary ?**\n- **The design of the noise term**:  One important contribution of the paper is the design of the noise term. It would be great to have more explanations and ablations about these design choices. Why is this particular choice important for the method? Additionally, how does it compare to simpler noise schedules such as the one in [2]?\n- Concerning the update in equation 9 of the paper, please clarify what assumptions and approximations are needed for this derivation and in which case they are no longer valid.\n\n\n[1] Mark Steijkvers.A recurrent network that performs a context-sensitive prediction task. 1996.\n\n[2] Neelakantan et al.,Adding gradient noise improves learning for very deep networks. 2015\n\n[3]  Deng et al., How shrinking gradient noise helps the performance of neural networks. 2021. \n\n[4] Jin et al., How to Escape Saddle Points Efficiently. 2017.\n\n[5] Jin et al., On Nonconvex Optimization for Machine Learning: Gradients,\nStochasticity, and Saddle Points.  2021.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their constructive comments and suggestions.\n\n## Relevance of the hybrid MCMC framework\n\nOur work is indeed related to gradient descent methods with noise and perturbations as suggested. In our case, our motivation for opting to interpret our framework as a hybrid MCMC method is due to the necessity of “jump” moves, and because we simply are much more familiar with the MCMC literature than what you pointed us to. As you suggest, we will extend our related works with the suggested literature in a revision. However, note that because much of the modeling space (the scene) is empty and Gaussians should only be located near surfaces, it is critical that Gaussians are moved toward these surfaces quickly. Under the MCMC formulation, they can be naturally integrated as “jump” moves or “resampling” moves [A, B], which we leverage.\n\nRegarding the noise in the momentum component, this is clearly an interesting direction to explore. We are excited that our reformulation seems to be opening doors to these various design choices which were originally not possible due to the heuristics that were involved in the optimization, such as abrupt reset of Gaussian opacities or manual splitting of large Gaussians. These heuristics clearly break any hope to formally study the convergence characteristics of the method, and in our work we remove all of them, which will hopefully lead to more work on this topic.\n\n[A] Lindsey et. al,  “Ensemble Markov chain Monte Carlo with teleporting walkers”, SIAM/ASA Journal on Uncertainty Quantification. 2022\n\n[B] Green, “Reversible jump Markov chain Monte Carlo computation and Bayesian model determination”, Biometrika, 1995\n\n## Design of the noise term\n\nIn Table 3 we already had a brief comparison of an alternative noise strategy where we add noise also to other Gaussian parameters. We have further tested without using the covariance term and the opacity term for the *Tank and Temples* dataset (we chose this dataset due to the short amount of time available for the rebuttal). We report the results in the table below:\n\n|              | No Covariance | No opacity* | With both |\n| ------------ | ------------- | ---------- | --------- |\n| PSNR - Train | 21.29         | 21.68      | 22.40     |\n| PSNR - Truck | 25.04         | 23.27      | 26.02     |\n| PSNR - Avg.  | 23.16         | 22.47      | 24.21     |\n\nHaving both covariance and opacity is important to achieve the best performance:\n- Without the covariance term, the noise is difficult to “undo” with gradients for narrow Gaussians, which then cause the Gaussians to randomly walk about and no longer represent the desired distribution.\n- Without the opacity term, opaque Gaussians that are significantly contributing to the reconstruction loss would not be able to converge/stabilize, leading to a loss of high-frequency details.\n\nFurther, note that in the table above we had to use $\\lambda_{noise}=5\\times10^2$ for “No opacity*”, as if we use the same hyper-parameters as in the paper, the model does not train at all (PSNR<7).\n\n## Noise term scheduler\n\nAs suggested, we also tried the scheduling from [2], and we additionally tested a simple linear scheduler. Again, we ran our experiment on the *Tank and Temples* dataset. As shown in the table below, our exponential decay scheduler (which uses the same scheduler as in the original 3DGS paper) works the best:\n\n|              | Noise Scheduler in our paper | Best Noise scheduler in [2] | Linear scheduler |\n| ------------ | ---------------------------- | --------------------------- | ---------------- |\n| PSNR - train | 22.40                        | 21.86                       | 15.37            |\n| PSNR - Truck | 26.02                        | 23.06                       | 19.90            |\n| PSNR - Avg.  | 24.21                        | 22.46                       | 17.64            |\n\n## Derivation of Eq. 9\n\nThe detailed derivation of Eq. 9 is provided in Appendix. A. The main approximation is the approximation of the integral that integrates the squared difference between the two distributions. We instead minimize the difference between the individual integrals, projected into each view, which is inspired by sliced Wasserstein methods [17]. As the color values that we integrate over are always non-negative, there are no underlying assumptions that could break.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BfCOAHUtWk", "reviewer": "Reviewer_jFyG", "summary": "Current 3DGS-based methods require carefully designed strategies such as cloning and splitting to assign a 3D Gaussian at a location. Further, they also require initializing points from SFM to generate high-quality novel views. The proposed work assumes that a set of 3D Gaussians are drawn from an underlying probability distribution, which is representative of the scene. Further, 3DGS updates are converted to SGLD updates by introducing noise.\n\nThe main contributions of this work are as follows:\n- A fresh perspective that 3D Gaussians are sampled from a distribution and relocation strategy is compatible with MCMC samples.\n- Robustness to initialization. 3DGS-MCMC is not dependent on the initialization step in 3DGS.\n- The proposed method outperforms other NeRF-based methods and 3DGS on standard datasets.", "questions": "- L40-41: \"lead to .... waste compute.\"Can \"poor-quality renderings\" be substantiated by some examples? Also, can the authors elaborate why it is a wasted compute? InstantSplat[7] reconstructs an unbounded scene with sparse views in under 40 seconds on a commercial GPU. \n\n- Does this method accurately represent the surface better than 3DGS? Can we extract a high-quality mesh from 3DGS-MCMC? Will it be better than recent works such as 2D Gaussian Splatting[A2]? \n\n[A2] Huang, B., Yu, Z., Chen, A., Geiger, A. and Gao, S., 2024. 2d Gaussian splatting for geometrically accurate radiance fields. arXiv preprint arXiv:2403.17888.", "limitations": "The authors have discussed the limitations of their method in Appendix D in the supplementary material. \nAs such, this work has no societal impact. However, some downstream applications can have a societal impact. The authors have discussed this in Appendix E.", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- **Qualitative and Quantitative Results:** The proposed method is evaluated on NeRF synthetic, Tank&temples, Deep Blending, MipNeRF360 and OMMO dataset. Quantitatively, 3DGS-MCMC outperforms NeRF-based methods and 3DGS. Qualitatively, the novel-views from 3DGS-MCMC are sharp compared to the 3DGS method (Fig. 2). This is due to the MCMC formulation proposed in this work, which allows exploration.\n\n- **High-Performance compared to 3DGS with a limited budget(L278-297):** The authors show an interesting experiment where they limit the budget for number of Gaussians during optimization. As expected, 3DGS has a significant drop in performance, whereas the performance drop in the proposed method is limited. Notably, there is a difference of 4 dB when the maximum number of Gaussians is set to 100k.\n\n- Unlike 3DGS, the proposed method is not sensitive to the initialization. This robustness to initialization is illustrated in Tab. 2. When a camera extent of $1\\times$ is used, 3DGS achieves a PSNR of 22.72 dB, whereas 3DGS-MCMC achieves a PSNR of 29.64. This result show that the proposed method is robust to the initialization. Also, this substantiates the exploration claim proposed in the paper. \n\n- **Exhaustive ablation for all the key design choices:** In Tab. 3, the authors present ablation on the regularizers for Gaussians. Interestingly, in this framework, regularization of the Gaussian parameters improves the performance, whereas it is harmful in the 3DGS framework. Further, noise in the update step update allows for more exploration. This claim is further substantiated in Fig. 4. Finally, when noise is used for all the parameters, overall performance is slightly dropped.", "weaknesses": "- The proposed method exhibits robustness to the initialization step. However, is there a reduction in training time between the version using SFM points and the version using random initialization?\n\t\n- **Missing evaluation on Scannet++[A1] dataset:** It is a large-scale dataset for indoor scenes. It will be interesting to see how this method performs on this challenging dataset. It is difficult to perform this experiment in a short duration of time. I leave it to authors to decide if they want to include in their manuscript. \n\n- **Training Time:** The proposed method generates very high-quality novel-views. However, this comes with an added training cost. 3DGS-MCMC takes 90 minutes for 1M Gaussians, whereas 3DGS can be optimized within 30 minutes. The authors mention in L467-468 that a CUDA implementation can accelerate the proposed method. \n\n[A1] Yeshwanth, C., Liu, Y.C., Nießner, M. and Dai, A., 2023. Scannet++: A high-fidelity dataset of 3d indoor scenes. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 12-22).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for sharing the enthusiasm that we have for our method.\nHere is our response to your comments:\n\n## Training time between SfM points version vs Random points version\n\nSfM has faster convergence, but to obtain the best PSNR performances both were run for about the same amount of wall-clock time. However, note our primary objective was not to improve convergence, but to reduce reliance on good initialization and also remove the heuristics. We believe the latter two to be critical in situations where initialization could be difficult, such as 3D generative modeling or dynamic scene reconstruction.\n\n## ScanNet++\n\nIndeed this would be very interesting. We were not able to run these experiments due to the resource crunch during the rebuttal period, but we hope to be able to add these results for the camera ready.\n\n## Training time\n\nPlease see the global response for the training time. In short, 90ms was *milliseconds, not minutes* and considering only the compute time for each Gaussian the added time is negligible. Ours does run slower if trained with a high opacity regularizer due to more transparent Gaussians, but it ultimately gives higher quality results, and it also can be tuned to **train faster while still outperforming 3DGS**.\n\n## L40--L41 Suboptimal placements leading to wasted compute and poor quality renderings\n\nOur existing experiments show that existing relocation heuristics place Gaussians at worse locations than ours, leading to lower rendering quality (this is especially true when random initialization is used). In regards to *wasted compute*, we wanted to highlight the fact that Gaussian splitting heuristics becomes unnecessary, but we realize that this can be misinterpreted and will remove this comment.\n\n## Surfaces\n\nThis is a great suggestion. While we were not able to finish this during the short rebuttal period as it involves setting up a new evaluation pipeline, we will investigate this in the future.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "I63Vgauy3Q", "reviewer": "Reviewer_m7Kb", "summary": "The paper discusses improvements to 3D Gaussian Splatting in neural rendering. Current methods rely on complex cloning and splitting strategies for placing Gaussians, which often do not generalize well and depend heavily on good initializations. The authors propose rethinking 3D Gaussians as random samples from an underlying probability distribution of the scene, using Markov Chain Monte Carlo (MCMC) sampling. They show that 3D Gaussian updates can be converted into Stochastic Gradient Langevin Dynamics (SGLD) updates by adding noise. This allows for the removal of heuristic densification and pruning strategies, replacing them with a deterministic state transition of MCMC samples. Additionally, the authors introduce an L1-regularizer on Gaussians to encourage efficient usage. Their method improves rendering quality, provides easy control over the number of Gaussians, and is robust to initialization across various standard evaluation scenes.", "questions": "/", "limitations": "/", "rating": 7, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1) This work is very insightful\n2) Experiments are extensive and detailed", "weaknesses": "/", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the positive assessment of our paper. Please let us know if you have any concerns or questions and we will try our best to respond.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Y3lKwLdmgZ", "reviewer": "Reviewer_cJQR", "summary": "This paper proposes a novel densification strategy of 3D Gaussian Splatting (3DGS) based on the Markov Chain Monte Carlo (MCMC) sampling scheme.\nThe authors address the ‘heuristic’ densification of standard 3DGS and adopt a distribution-aware resampling pipeline. \nConsequently, they achieve higher rendering quality compared to standard 3DGS with a similar number of Gaussian primitives.", "questions": "- Although they have mentioned the computational time in Sec. B of the appendix, I cannot understand how much it takes compared to standard 3DGS. Can you provide the comparison in training time of this method and standard 3DGS for each scene?\n\n- Although it has been described in L184-186, I am confused about the reason why adding noise to other parameters leads to harmful results. Can you describe more details about it?", "limitations": "It only tackles the cloning strategy without any other problems for achieving higher rendering quality. Thus, the technical improvement seems to be inefficient.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper is well-written to understand.\n\n- The authors tackle the limitation of heuristic densification of standard 3DGS and suggest an MCMC sampling strategy for 3DGS. Also, the detailed analysis effectively supports their theoretical statements.\n\n- It achieves standard 3DGS in rendering quality while preserving the fast inference time with the same format of 3DGS representation. Therefore, it can be used in various applications of 3DGS without any modification.", "weaknesses": "- The additional computational costs are required for MCMC resulting in the increase of training time compared to standard 3DGS.\n\n- They only suggest the resampling strategy of 3DGS. Therefore, the technical contribution is inefficient despite the detailed analysis.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their constructive comments and suggestions.\nHere are our responses to your comments:\n\n## Technical contribution\n\nWe respectfully disagree, as our method cannot be summarized as a mere resampling strategy. It is a completely new take on the 3D Gaussian Splatting optimization that removes ALL the non-differentiable heuristics (and inherent training instability) that were introduced by the original authors: “opacity reset”, “densification”, \"splitting\", and “pruning”.\nWe have further shown that this is non-trivial to achieve, as one must carefully design components in a way that obeys the basic principles of hybrid MCMC. Thanks to our re-formulation, future works could integrate various advancements in perturbed Gradient Descent, as reviewer Cpa7 suggests. \n\n## Additional compute cost\n\nPlease see the global response for the training time. In short, considering only the compute time for each Gaussian the added time is negligible. Ours does run slower if trained with a high opacity regularizer due to more transparent Gaussians, but it ultimately gives higher quality results, and it also can be tuned to **train faster while still outperforming 3DGS**.\n\n\n## Noise to other parameters being harmful\n\nThere have been a number of recent papers demonstrating that the locations of the Gaussians suffer heavily from local optima, which leads to their reliance on good initializations [7, 8]. Hence, adding noise to these parameters helps escape these local optima, and Gaussians to “explore” the scene. Other degrees of freedom (rotation, scale, opacity) do not suffer as heavily, hence the noise simply slows down convergence unnecessarily. Please see additional details showing more experiments on the noise design in our answer to reviewer Cpa7 if the reviewer is interested.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We are glad to see that all reviewers are positive towards our paper.
Reviewers commend the effectiveness of our method (**Cpa7**, **jFyG**, **cJQR**), especially when random initialization is used, the thoroughness of our ablations (**jFyG**, **cJQR**).
They also acknowledge the ease of use of our method as a drop-in to existing Gaussian Splatting methods (**jFyG**, **cJQR**).
Please see the individual rebuttals for reviewer-specific responses.

## Training time (**jFyG** and **cJQR**)

First of all, we would first like to clarify that what we provided in the Appendix for training time is 90ms (milliseconds) and *not 90 minutes* (**jFyG**).

To provide an exact comparison, we took our Gaussians trained at 1M Gaussian, and re-measured the single optimization iteration time for our method and the original 3DGS. In this case, ours takes 80 milliseconds while 3DGS takes 76 milliseconds. That is, the added time for sampling and noise addition is not substantial, even with our implementation.

Still, to achieve the PSNR reported in the paper, our method does take longer. This is because the configuration of Gaussians (i.e. where they are, their sizes and opacity) matters greatly to the runtime as this affects the speed of rasterization (among them the opacity regularizer affects the speed the greatest). For the “room” scene in the MipNeRF 360 dataset, we find the following timings, all with SfM initializations and max 1.5M Gaussians per the original 3DGS implementation:

|              | opacity regularizer ($\lambda_o$) | PSNR | Total Training Time |
| ------------ | ------------- | ---------- | ---------- | 
| 3DGS | --  |31.7 | 25 minutes |
| Ours (paper) | 0.01 |  32.5 | 42 minutes |
| Ours  | 0.001 |  32.4 | 30 minutes |
| Ours (300k Gaussians)  | 0.01 | 31.8 | 21 minutes|

Note how our method still **outperforms 3DGS regardless of our choice**. We found $\lambda_o=0.01$ to work well for various scenarios, including when using random initialization (with lower opacity regularizer random initialization performs slightly worse 31.7 vs 32.3 with $\lambda_o=0.001$), thus we report performance with 0.01. Finally, when using fewer Gaussians, our method **still outperforms 3DGS and trains faster**.
Due to the scarcity of compute caused by the rebuttal period, we were unable to provide the full per-scene compute times, but we will include them all in our camera ready.

Meta-review: The submission received consistently high scores. The reviewers and the authors engaged in positive discussion during the rebuttal phase, and a few of the initial queries were satisfactorily resolved (the biggest one was a misunderstanding of ms wrt time). The proposed solution was found to be simple and effective, removing the heuristic step in the original GS initialization/spawning step. The work is appreciated both due to its theoretical grounding and its practical relevance. 

Please include the additional clarifications while preparing the final version of the paper. Congratulations on the nice piece of work.
Decision: Accept (spotlight)