Relaxing Accurate Initialization Constraint for 3D Gaussian Splatting

## Abstract

Abstract 3D Gaussian splatting (3DGS) has recently demonstrated impressive capabilities in real-time novel view synthesis and 3D reconstruction. However, 3DGS heavily depends on the accurate initialization derived from Structure-from-Motion (SfM) methods. When the quality of the initial point cloud deteriorates, such as in the presence of noise or when using randomly initialized point cloud, 3DGS often undergoes large performance drops. To address this limitation, we propose a novel optimization strategy dubbed RAIN-GS ( R elaxing A ccurate IN itialization Constraint for 3D G aussian S platting). Our approach is based on an in-depth analysis of the original 3DGS optimization scheme and the analysis of the SfM initialization in the frequency domain. Leveraging simple modifications based on our analyses, RAIN-GS successfully trains 3D Gaussians from sub-optimal point cloud (e.g., randomly initialized point cloud), effectively relaxing the need for accurate initialization. We demonstrate the efficacy of our strategy through quantitative and qualitative comparisons on multiple datasets, where RAIN-GS trained with random point cloud achieves performance on-par with or even better than 3DGS trained with accurate SfM point cloud.

## 1 Introduction

Novel view synthesis is one of the essential tasks in computer vision and computer graphics, aiming to render novel views of a 3D scene given a set of images. It has a wide range of applications in various fields, including augmented reality and virtual reality [], robotics [], and data generation []. Neural radiance fields (NeRFs) [] have demonstrated remarkable success in this task, learning implicit representations that capture intricate 3D geometry and specular effects solely from images. However, NeRFs’ reliance on multi-layer perceptrons (MLPs) [, , , , ] results in slow and computationally intensive volume rendering, hindering real-time applications.

Recently, 3D Gaussian splatting (3DGS) [] has emerged as a compelling alternative for both high-quality results and real-time rendering. Unlike NeRFs training an MLP to model the scene with an implicit representation, 3DGS models the scene using explicit 3D Gaussians. The learned 3D Gaussians are then projected to the image space using an efficient CUDA-based differentiable tile rasterization [, ] technique which enables the rendering of the Gaussians in real-time.

Despite its remarkable results, 3DGS requires an additional input of initial point cloud compared to NeRFs. The quality of the initial point cloud is one of the essential requirements of 3DGS, showing large performance drops when trained with randomly initialized point cloud []. A similar type of performance degradation can occur in real-world scenarios, specifically in scenes where SfM techniques struggle to converge, such as scenes with symmetry, and textureless regions [, ]. Also, accurate initial point cloud become a strict requirement even for situations where camera poses can be obtained through external sensors [, ] or pre-defined such as in text-to-3D generation [, ]. In these cases, where the initial point cloud from SfM may contain noise or even unavailable, 3DGS requires an additional module to obtain the point cloud, adding extra computation cost [, ].

In this work, we start with a natural question: “Why is accurate initial point cloud so important in 3D Gaussian splatting?” and conduct an in-depth analysis of the original 3DGS optimization scheme and SfM initialization. Our analyses reveal that the original 3DGS optimization scheme struggles to transport Gaussians further from their initialized locations, leading to performance degradation when starting from random or noisy SfM initialization. In addition, the analysis of the SfM initialization in the frequency domain reveals that SfM initialization provides the low-frequency components of the true distribution. From the provided low-frequency components, 3DGS clones and splits existing Gaussians, which is then used to learn the remaining high-frequency details of the scene []. In this process, the low-frequency components guide the Gaussians to successfully learn the target distribution in a coarse-to-fine manner utilizing the clone/split process, without the need to transport the Gaussians to correct locations. Based on this observation, we propose that to train 3DGS with noisy or random point cloud, we need an effective strategy that can transport the Gaussians further from their initial positions and encourage the Gaussians to prioritize the learning of the coarse structure information, which will then guide the overall optimization process to learn in a similar coarse-to-fine manner.

Based on our analysis of accurate initial point cloud, we propose a novel optimization strategy called RAIN-GS (Relaxing Accurate INitialization Constraint for 3D Gaussian Splatting), which is composed of three key components: 1) a novel initialization method starting with sparse Gaussians with large variance, 2) progressive Gaussian low-pass filtering utilized in the rendering process, and 3) a novel adaptive bound-expanding split (ABE-Split) algorithm utilized in the adaptive density control. We demonstrate that RAIN-GS effectively relaxes the requirement of accurate initialization for 3DGS, as applying RAIN-GS to random point cloud successfully priorities the learning of low-frequency components and guides the Gaussians to learn in a coarse-to-fine manner, and enables the Gaussians to traverse further from their initialized locations. After optimization, RAIN-GS applied to random point cloud achieves on-par or even better results compared to 3DGS trained with SfM. By effectively relaxing the requirement of initial point cloud, RAIN-GS opens up new possibilities for applying 3DGS to settings where obtaining accurate initial point cloud is challenging.

## 2 Related work

### 2.1 Novel view synthesis

Neural radiance fields (NeRF) [] have succeeded in significantly boosting the performance of novel view synthesis by optimizing an MLP that can estimate the density and radiance of any continuous 3D coordinate. With the camera poses of the given images, NeRF learns the MLP by querying dense points along randomly selected rays, that outputs the density and color of each of the queried coordinates. Various follow-ups [, , , , , , , ] adopted NeRF as their baseline model and further extend the ability of NeRF to model unbounded or dynamic scenes [, , ], lower the required number of images for successful training [, ], learn a generalizable prior to resolve the need of per-scene optimization [, ], or utilize an external hash-grid to fasten the overall optimization process []. Although all of these works show compelling results, the volume rendering from dense points along multiple rays makes NeRF hard to apply in real-time settings achieving lower rendering rates of < 1 fps.

Recently, 3D Gaussian splatting (3DGS) [] has been recognized as a new solution for the task of novel view synthesis, succeeding in high-quality real-time novel-view synthesis at 1080p resolution with frame rates exceeding 90 fps. 3DGS achieves this by modeling the 3D scene with explicit 3D Gaussians, followed by an efficient and differentiable tile rasterization implemented in CUDA to accelerate the rendering process. Due to the efficient and fast architecture of 3DGS, it gained massive attention from various fields [, , , ], extended to model dynamic scenes [, ], or used as an alternative for text-to-3D tasks [, ] for fast generation. Nevertheless, the essential requirement of both accurate pose and initial point cloud limits the application of 3DGS to only scenes where SfM can provide accurate point cloud. In this work, we analyze the optimization strategy of 3DGS and present a simple yet effective change in the optimization strategy to expand the versatility of 3DGS to successfully learn the scene without accurate initialization.

### 2.2 Structure-from-Motion (SfM)

SfM techniques [, , ] have been one of the most widely used algorithms to reconstruct a 3D scene. Typical SfM methods output the pose of each input image and a sparse point cloud that includes rough color and position information for each point. These methods employ feature extraction, matching steps, and bundle adjustment. For the task of novel view synthesis, NeRF [] utilizes the estimated pose from the SfM and trains an implicit representation to model the scene. The recently proposed 3DGS [] utilizes both the accurate pose and initial point cloud as an initialization for the position and color of 3D Gaussians, showing large performance drops when the initial point cloud becomes noisy or not accessible.

Despite the effectiveness of SfM in generating accurately aligned point cloud and precise camera poses, its incremental nature and the computational intensity of the bundle adjustment process significantly increase its time complexity, often to $O(n^{4})$ with respect to $n$ cameras involved []. Due to the high computational demand, SfM being a hard prerequisite limits the feasibility of NeRF or 3DGS for scenarios that require real-time processing. In addition, SfM often struggles to converge, such as in scenes with symmetry, specular properties, and textureless regions, and when the available views are limited. In these cases, an additional module is required to provide accurate initial point cloud to 3DGS, which can take longer times than SfM [].

## 3 Preliminary: 3D Gaussian splatting (3DGS)

Recently, 3DGS [] has emerged as a promising alternative for the novel view synthesis task, modeling the scene with multiple 3D Gaussians [].
Each $i$-th Gaussian $G_{i}$ represents the scene with the following attributes: a position vector $\mu_{i}\in\mathbb{R}^{3}$, an anisotropic covariance matrix $\Sigma_{i}\in\mathbb{R}^{3\times 3}$, spherical harmonic (SH) coefficients [, ], and an opacity logit value $\alpha_{i}\in[0,1)$. With these attributes, each Gaussian $G_{i}$ is defined in the world space [] as follows:

$$ $G_{i}(x)=e^{-\frac{1}{2}(x-\mu_{i})^{T}\Sigma_{i}^{-1}(x-\mu_{i})}.$ (1) $$

To render an image from a pose represented by the viewing transformation $W$, the projected covariance $\Sigma_{i}^{\prime}$ is defined as follows:

$$ $\Sigma_{i}^{\prime}=JW\Sigma_{i}W^{T}J^{T},$ (2) $$

where $J$ is the Jacobian of the local affine approximation of the projective transformation. The 2D covariance matrix is simply obtained by skipping the third row and column of $\Sigma_{i}^{\prime}$ []. Finally, to render the color $C(p)$ of the pixel $p$, 3DGS utilizes alpha blending according to the Gaussians depth. For example, when $N$ Gaussians are sorted by depth, the color $C(p)$ is calculated as follows:

$$ $C(p)=\sum_{i=1}^{N}c_{i}\alpha_{i}G^{\prime}_{i}(p)\prod_{j=1}^{i-1}(1-\alpha_ {j}G^{\prime}_{j}(p)),$ (3) $$

where $c_{i}$ is the view-dependent color value of each Gaussian calculated with the SH coefficients, and $G^{\prime}_{i}$ is the 3D Gaussian projected to the 2D screen space.

Since 3DGS starts learning from sparse point cloud, they utilize an adaptive density control to obtain a more accurate and denser representation. By considering if the scene is under-/over-constructed, each of the Gaussians undergoes a clone/split operation. Gaussians that have small covariance are cloned and Gaussians considered to be too large are splitted into two new smaller Gaussians, where the means of new Gaussians are sampled from the probability density function (PDF) of the original Gaussian. When the Gaussian is considered to be too small, the Gaussian is cloned and simply located in the mean of the original Gaussian.

## 4 Motivation

Although point cloud can be noisy or unavailable in real-world scenarios [, ], 3DGS shows large performance drops depending on the accuracy of the initial point cloud []. To understand the large performance gap of 3DGS, we conduct an in-depth analysis of the original 3DGS optimization scheme and the analysis of SfM initialization in the frequency domain. Our analyses reveal two important characteristics of the optimization scheme of 3DGS: 1) the optimization scheme of 3DGS struggles to transport Gaussians from their initialized locations and 2) the coarse structure information (low-frequency components) provided by the accurate initialization enables the adaptive density control method of 3DGS to robustly model the remaining fine details of the scene in a coarse-to-fine manner.

##### 3DGS lacks the ability to transport Gaussians.

To represent and learn the scene with explicit 3D Gaussians, 3DGS first initializes the Gaussians $G_{i}$ in the world space, whose means $\mu_{i}$ are defined by the initial point cloud. The point cloud can be either achieved from SfM or initialized randomly.

As mentioned in [], the process of fitting a 3DGS model is similar to fitting a Gaussian Mixture Model (GMM), which is well-known for being non-convex and generally solved with the Expectation-Maximization (EM) algorithm []. They further note that, similar to the EM algorithm, training 3DGS from randomly initialized point cloud becomes prone to falling into local minima due to two main reasons. 1) The Gaussians can only receive gradients close to their means, mostly from the range not exceeding the distance of a few standard deviations, and 2) there is no existing path for the Gaussians that will decrease the loss monotonically.

Although [] only analyzes the case of starting from random initialization, we verify that Gaussians can also easily fall into local minima when SfM-initialized point cloud become noisy. As shown in Table 2, we find that adding a small constant noise or adding a small noise $\epsilon$ sampled from a normal distribution ($\epsilon\sim\mathcal{N}(0,1))$ to the SfM-initialized point cloud, leads to large performance drops. Based on these observations, we hypothesize that the optimization scheme of 3DGS lacks the ability to correct or move the positions of the Gaussians. We empirically verify our hypothesis by calculating the average distance each Gaussian traversed after optimization, as shown in Table 2. It can be seen that the average distance each Gaussian moved is close to zero, indicating that the optimization scheme of 3DGS lacks the ability to move Gaussians, which can lead to the failure of capturing objects located far from the initial positions of the Gaussians. This emphasizes the need for a strategy that can enable the Gaussians to transport further from their initialized locations, in order to successfully train 3DGS from sub-optimal initializations.

**Table 1: Effect of noise on the initial SfM point cloud.**
| Init. | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ |
| --- | --- | --- | --- |
| SfM | 27.205 | 0.815 | 0.214 |
| SfM + ${\epsilon}$ | 24.944 | 0.771 | 0.270 |
| SfM + constant | 26.204 | 0.761 | 0.277 |

##### Accurate initialization guides 3DGS to learn in a coarse-to-fine manner.

Figure: Figure 2: Analysis of SfM initialization in 3DGS. (a) The top shows the GT image, and the bottom is the rendered image by 3DGS after only 10 steps with SfM initialization. We can observe that the rendered image is already coarsely-close to GT image. We randomly sample a horizontal line from the image marked in red. (b) The pixel intensity along this line are shown, with the GT indicated in blue and the rendered image in orange. (c) This graph visualizes the magnitude of the frequency components of (b). Since frequencies further from the middle of the x-axis represent high-frequency components, we observe that SfM provides coarse approximation of the true distribution.
Refer to caption: x2.png

To further investigate the benefits of SfM initialization, we analyze the rendered images in the frequency domain using Fourier transform []. As shown in Figure 2, the analysis in the frequency domain demonstrates that SfM initialization provides a coarse approximation of the target distribution.

As the goal of novel view synthesis is to understand the 3D distribution of the scene, it is necessary to model both low- and high-frequency components of the true distribution. However, prior NeRF frameworks [, , ] argue that NeRF is prone to overfitting and naïve optimization leads to over-fast convergence of high-frequency components, expressed with high-frequency artifacts in the rendered image. To circumvent this problem, they adopt a coarse-to-fine learning strategy, which regularizes NeRF to learn the low-frequency components first. Similarly, prior works [, ] utilizing GMMs for the task of point cloud registration or generation also mention that naïve fitting of GMMs can result in converging to local minima. In order to robustly train GMMs, they also adopt a coarse-to-fine strategy, implemented by starting with a small number of Gaussians and recursively increasing the number of total Gaussians. In both NeRFs and GMMs, coarse-to-fine strategy guides the network to learn more robustly, leading to better performance.

In this perspective, starting the optimization of 3DGS from SfM-initialized point cloud can be understood as benefitting from a similar coarse-to-fine process, where SfM provides the low-frequency components (Figure 2), and the adaptive density control method of 3DGS adds the Gaussians to learn the remaining high-frequency details. Based on our observations, the success of 3DGS from accurate initialization can be attributed to the low-frequency components guiding the overall training process, preventing the Gaussians from falling into local minima.
This highlights the need for a strategy that can prioritize the learning of the low-frequency components even from sub-optimal initializations, which will then be used to guide the remaining optimization process of 3DGS.

## 5 Methodology

Following our motivation (Section 4), we propose a novel optimization strategy, dubbed RAIN-GS (Relaxing Accurate INitialization Constraint for 3D Gaussian Splatting), which enables the Gaussians to prioritize the learning of low-frequency components, and enables the Gaussians to transport further from their initialized locations. Our strategy consists of three key components: 1) sparse-large-variance (SLV) initialization (Section 5.1), 2) progressive Gaussian low-pass filtering (Section 5.2), and 3) a novel adaptive bound-expanding split (ABE-Split) algorithm (Section 5.3).

### 5.1 Sparse-large-variance (SLV) initialization

Figure: Figure 3: Toy experiment to analyze different initialization methods. This figure visualizes the result of our toy experiment predicting the target distribution using a collection of 1D Gaussians, starting from different initialization methods.
Refer to caption: x3.png

Drawing inspiration from GMMs [, ], which gradually increase the number of Gaussians to accurately model target point cloud, we observe that the adaptive density control of 3DGS can be viewed as a similar process. Through cloning and splitting operations, 3DGS generally increases the number of Gaussians to find the adequate number of Gaussians required to represent the scene. Based on our findings, we hypothesize that initializing 3DGS with a sparse set of Gaussians will prioritize the learning of low-frequency components, akin to the progressive refinement approach employed by GMMs. This sparse initialization strategy is expected to capture the overall structure of the target point cloud in the early stages of the optimization process, with finer details being added as the number of Gaussians increases.

To verify our hypothesis, we conduct a toy experiment in a simplified 1D regression task. Following the original 3DGS which can be interpreted as the learning process of a 3D target distribution with multiple Gaussians, we use $N$ Gaussians each with learnable means, variances, and weights, which are then blended to model a 1D target signal. Specifically, we follow the initialization methods of 3DGS [], where the means are initialized randomly and the variances are initialized based on the distances of the three nearest neighbors. As a result, sparse initialization of Gaussians leads to a larger initial covariance (SLV) and dense initialization leads to a smaller covariance (DSV). To verify our hypothesis that learning with sparse Gaussians will prioritize the learning of low-frequency components, we conduct our toy experiment using $N=15$ and $N=1000$ for the SLV and DSV initialization respectively. Note that our 1D toy experiment without the adaptive density control method of 3DGS provides a controlled environment isolating the effects of initialization. A detailed explanation of our toy experiment can be found in Section C of the supplementary materials.

As shown in Figure 3, SLV initialization prioritizes the learning of low-frequency components compared to DSV initialization verifying our hypothesis. After 1,000 steps, SLV also shows a better prediction of the target distribution. Similar results can be observed when SLV is applied to 3DGS, as lowering the number of initial Gaussians $N$ in randomly initialized settings significantly improves performance. Following the random initialization method of [], which randomly samples point cloud from a scene extent defined as three times the bounding box of the camera poses, SLV prioritizes the learning of low-frequency components, producing fewer high-frequency artifacts. Surprisingly, SLV becomes more effective even until extremely sparse settings (e.g., as low as $N=10$), verifying the effectiveness of our novel SLV initialization method.

### 5.2 Progressive Gaussian low-pass filter control

Although our SLV initialization method is effective, we find that after multiple densification steps, the number of 3D Gaussians increases exponentially due to the adaptive density control, which can collapse into similar problems with the DSV initialization. In order to guide the 3D Gaussians to learn in a coarse-to-fine manner and let the 3D Gaussians sufficiently explore the low-frequency components during the early stage of training, we propose a novel progressive control of the Gaussian low-pass filter which is utilized in the rendering stage.

##### Gaussian low-pass filter for 3DGS.

In the rendering stage of 3DGS, the 2D Gaussian $G_{i}^{\prime}$ projected from a 3D Gaussian $G_{i}$ is defined as follows:

$$ $G_{i}^{\prime}(x)=e^{-\frac{1}{2}(x-\mu_{i}^{\prime})^{T}{\Sigma_{i}^{\prime}} ^{-1}(x-\mu_{i}^{\prime})}.$ (4) $$

However, directly using projected 2D Gaussians can lead to visual artifacts when they become smaller than the size of a single pixel [, ]. To ensure coverage of at least one pixel, [] enlarge the 2D Gaussian’s scale by adding a small value to the covariance’s diagonal elements as follows:

$$ $G_{i}^{\prime}(x)=e^{-\frac{1}{2}(x-\mu_{i}^{\prime})^{T}(\Sigma_{i}^{\prime}+ sI)^{-1}(x-\mu_{i}^{\prime})},$ (5) $$

where $s$ is a pre-defined value of $s=0.3$ and $I$ is an identity matrix. This process can also be interpreted as the convolution between the projected 2D Gaussian $G_{i}^{\prime}$ and a Gaussian low-pass filter $h$ (mean $\mu=0$ and variance $\sigma^{2}=0.3$) of $G_{i}^{\prime}\otimes h$, which is shown to be an essential step to prevent aliasing []. After applying convolution with the low-pass filter, the area of the projected Gaussian $G_{i}^{\prime}$ is approximated by a circle. The radius of this circle is defined by three times the larger eigenvalue from the 2D covariance matrix $(\Sigma_{i}^{\prime}+sI)$(^1^11We provide a detailed proof in Section B.1 in the supplementary materials.).

##### Progressive low-pass filter control.

Instead of using a fixed value of $s$ through the entire optimization process, we notice that this value $s$ can ensure the minimum area each Gaussians have to cover in the screen space. As Gaussians only receive gradients inside the range of a few standard deviations [], learning from wider areas is essential for the Gaussians to learn the coarse structure information of the scene. Therefore, we control $s$ to regularize the Gaussians to cover wider areas during the early stage of training and progressively learn from a more local region. Specifically, as the value $s$ ensures the projected Gaussians area to be larger than $9{\pi}s$(^2^22We provide a detailed proof in Section B.2 in the supplementary materials.), we define the value $s$ such that $s=HW/9\pi N$, where $N$ indicates the number of Gaussians and $H,W$ indicates the height and width of the image respectively.

### 5.3 Adaptive bound-expanding split (ABE-Split) algorithm

As discussed in Section 4, in 3DGS, Gaussians struggle to model scenes far from their initialized locations. Through visualizations of the positions of the Gaussians after training, we empirically find that Gaussians particularly struggle to model scenes outside the initial bound defined by the starting point cloud positions (see Section D.5 of the supplementary materials for visualizations).

This type of behavior is largely attributed to the characteristics of the adaptive density control method of 3DGS [], which applies a clone/split algorithm to the Gaussians. As the new Gaussian is sampled from the PDF of the original Gaussian, it is likely to be located near the original Gaussians, which leads to an increased density of Gaussians near initial locations. Consequently, 3DGS more easily models scenes inside the initial bound compared to those outside of it, highlighting the need for a strategy that can encourage the Gaussians to move outside the initial bound.

Based on these observations, we propose the adaptive bound-expanding split algorithm, a novel splitting algorithm that can encourage the Gaussians to move outside the initial bound during early steps. Specifically, ABE-Split algorithm splits each of the Gaussians into three, where two of the Gaussians undergo the same split process as the original split algorithm. However, for the additional third Gaussian, rather than sampling from the PDF of the original Gaussian, we initialize the new location by multiplying a constant value (proportional to the size of the scene extent). Also, we retain the same scale as the original Gaussian without dividing it. This is to encourage the transported Gaussians to receive gradients from larger regions in their new positions. We provide a more detailed explanation of ABE-Split in Section A in the supplementary materials.

### 5.4 Analysis of RAIN-GS

All three key components of our strategy can be interpreted as a component that acts as the “warm-up” step for the Gaussians, which guides the Gaussians to first learn the coarse structure of the scene. SLV initialization and progressive Gaussian low-pass filtering facilitate coarse-to-fine optimization whereas ABE-Split enables the Gaussians to transport to further locations. We demonstrate that our strategy successfully guides 3DGS to learn the coarse structure without any given information (e.g., accurate initial point cloud) which can widen the application of 3DGS to real-world scenarios where the initial point cloud can be noisy or absent. Specifically, as RAIN-GS does not have any restrictions or conditions for the initial point cloud other than being sparse (SLV initialization), we can apply our strategy to both point cloud achieved from SfM [] or random. The effectiveness and versatility of our strategy are further evaluated in Section 6.

## 6 Experiments

### 6.1 Experimental Settings

##### Datasets.

To assess the effectiveness of our strategy, we conduct qualitative and quantitative comparisons on the Mip-NeRF360 dataset [], Tanks&Temples dataset [], and Deep Blending dataset [] previously utilized in the evaluation of the 3DGS method []. We evaluate the PSNR, LPIPS, and SSIM metrics by constructing a train/test split using every 8th image for testing, as suggested in Mip-NeRF360 []. We use the same image resolution as 3DGS.

##### Tasks.

As mentioned in Section 4, SfM can fail in real-world scenarios where the point cloud become noisy or sometimes inaccessible. To assess the robustness of 3DGS [] in these scenarios, we first compare our strategy with 3DGS each trained with SfM-initialized point cloud, SfM-initialized point cloud with added noise $\epsilon$ ($\epsilon\sim N(0,1)$), and random point cloud, where our strategy only utilizes random point cloud (Table 3). In addition, to assess the performance of our strategy, we compare our strategy with Plenoxels [], InstantNGP-Base, InstantNGP-Big [], and 3DGS [] (Table 4). In the latter experiment, we evaluate 3DGS trained with SfM-initialized point cloud, and random point cloud. For our strategy, we train RAIN-GS with SfM-initialized point cloud and random point cloud.

### 6.2 Implementation details

We implement our model based on 3DGS []. We follow the same training process of the existing implementation in all datasets. We apply our strategy to both randomly initialized point cloud and SfM-initialized point cloud. When applying RAIN-GS to randomly initialized point cloud (Table 3, Table 4), we set the initial number of Gaussians to $N=10$. When applying RAIN-GS to SfM-initialized point cloud (Table 4), we select a sparse set from the entire point cloud. We either use the top 10% with the least reprojection error (denoted as <10%) or use cluster centers (denoted as cluster) obtained through HDBSCAN []. As RAIN-GS guides the Gaussians to learn the coarse structure of the scene in the early stages of training, we set the first 10,000 steps as the warm-up phase and extend the densification steps by 10,000 steps. During the warm-up phase, the learning rate of the mean $\mu$ is unchanged, and ABE-Split replaces the original split algorithm. Additional details are included in supplementary materials A.

**Table 3: Quantitative comparison on Mip-NeRF360, Tanks&Temples and Deep Blending datasets in different initial points conditions. We compare our method with 3DGS under three different initial point conditions: SfM points, noisy SfM points, and the absence of initial points.**
| Methods | Init.<br>points | Mip-NeRF360 Outdoor Scene |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| bicycle | flowers | garden | stump | treehill |  |  |  |  |  |  |  |  |  |  |  |  |
| PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ | PSNR$\uparrow$ | SSIM $\uparrow$ | LPIPS$\downarrow$ | PSNR$\uparrow$ | SSIM $\uparrow$ | LPIPS$\downarrow$ |  |  |
| 3DGS [] | SfM | 25.246<br>(    –    ) | 0.771<br>(    –    ) | 0.205<br>(    –    ) | 21.520<br>(    –    ) | 0.605<br>(    –    ) | 0.336<br>(    –    ) | 27.410<br>(    –    ) | 0.868<br>(    –    ) | 0.103<br>(    –    ) | 26.550<br>(    –    ) | 0.775<br>(    –    ) | 0.210<br>(    –    ) | 22.490<br>(    –    ) | 0.638<br>(    –    ) | 0.317<br>(    –    ) |
| Noisy SfM | 24.289<br>(-0.957) | 0.692<br>(-0.079) | 0.307<br>(+0.102) | 21.028<br>(-0.492) | 0.565<br>(-0.040) | 0.374<br>(+0.038) | 26.515<br>(-0.895) | 0.846<br>(-0.022) | 0.133<br>(+0.030) | 26.028<br>(-0.522) | 0.743<br>(-0.032) | 0.252<br>(+0.042) | 21.728<br>(-0.762) | 0.607<br>(-0.031) | 0.380<br>(+0.063) |  |
| Random | 21.034<br>(-4.212) | 0.575<br>(-25.42) | 0.378<br>(+0.173) | 17.815<br>(-3.705) | 0.469<br>(-22.48) | 0.403<br>(+0.067) | 23.217<br>(-4.193) | 0.783<br>(-0.085) | 0.175<br>(+0.072) | 20.745<br>(-5.805) | 0.618<br>(-0.157) | 0.345<br>(+0.135) | 18.986<br>(-3.504) | 0.550<br>(-0.088) | 0.413<br>(+0.096) |  |
| Ours | Random | 25.042 | 0.747 | 0.238 | 21.762 | 0.616 | 0.324 | 26.884 | 0.854 | 0.114 | 26.680 | 0.768 | 0.215 | 22.528 | 0.621 | 0.342 |
| Methods | Init.<br>points | Mip-NeRF360 Indoor Scene | Mip-NeRF360<br>Average |  |  |  |  |  |  |  |  |  |  |  |  |  |
| room | counter | kitchen | bonsai |  |  |  |  |  |  |  |  |  |  |  |  |  |
| PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ | PSNR$\uparrow$ | SSIM $\uparrow$ | LPIPS$\downarrow$ | PSNR$\uparrow$ | SSIM $\uparrow$ | LPIPS$\downarrow$ |  |  |
| 3DGS [] | SfM | 30.632<br>(    –    ) | 0.914<br>(    –    ) | 0.220<br>(    –    ) | 28.700<br>(    –    ) | 0.905<br>(    –    ) | 0.204<br>(    –    ) | 30.317<br>(    –    ) | 0.922<br>(    –    ) | 0.129<br>(    –    ) | 31.980<br>(    –    ) | 0.938<br>(    –    ) | 0.205<br>(    –    ) | 27.205<br>(    –    ) | 0.815<br>(    –    ) | 0.214<br>(    –    ) |
| Noisy SfM | 30.740<br>(+0.108) | 0.901<br>(-0.013) | 0.260<br>(+0.040) | 23.872<br>(-4.828) | 0.838<br>(-0.067) | 0.285<br>(+0.081) | 24.477<br>(-5.840) | 0.877<br>(-0.045) | 0.175<br>(+0.046) | 25.823<br>(-6.157) | 0.875<br>(-0.063) | 0.265<br>(+0.060) | 24.944<br>(-2.261) | 0.771<br>(-0.044) | 0.270<br>(+0.056) |  |
| Random | 29.685<br>(-0.947) | 0.894<br>(-0.020) | 0.265<br>(+0.045) | 23.608<br>(-5.092) | 0.833<br>(-0.072) | 0.276<br>(+0.072) | 26.078<br>(-4.239) | 0.893<br>(-0.029) | 0.161<br>(+0.032) | 18.538<br>(-13.442) | 0.719<br>(-0.219) | 0.401<br>(+0.196) | 22.190<br>(-5.015) | 0.704<br>(-0.111) | 0.313<br>(+0.099) |  |
| Ours | Random | 30.809 | 0.906 | 0.247 | 28.529 | 0.895 | 0.223 | 31.270 | 0.920 | 0.137 | 31.547 | 0.934 | 0.218 | 27.228 | 0.807 | 0.229 |

| Methods | Init.<br>points | Tanks&Temples | Deep Blending |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Truck | Train | DrJohnson | Playroom |  |  |  |  |  |  |  |  |  |  |
| PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ | PSNR$\uparrow$ | SSIM $\uparrow$ | LPIPS$\downarrow$ |  |  |
| 3DGS [] | SfM | 25.187<br>(    –    ) | 0.879<br>(    –    ) | 0.148<br>(    –    ) | 21.097<br>(    –    ) | 0.802<br>(    –    ) | 0.218<br>(    –    ) | 28.766<br>(    –    ) | 0.899<br>(    –    ) | 0.244<br>(    –    ) | 30.044<br>(    –    ) | 0.906<br>(    –    ) | 0.241<br>(    –    ) |
| Noisy SfM | 23.367<br>(-1.820) | 0.838<br>(-0.041) | 0.195<br>(+0.047) | 20.212<br>(-0.885) | 0.753<br>(-0.049) | 0.284<br>(+0.156) | 27.084<br>(-1.682) | 0.882<br>(-0.017) | 0.283<br>(+0.039) | 30.194<br>(+0.150) | 0.901<br>(-0.005) | 0.252<br>(+0.011) |  |
| Random | 20.149<br>(-5.038) | 0.758<br>(-0.121) | 0.248<br>(+0.100) | 20.824<br>(-0.273) | 0.772<br>(-0.030) | 0.255<br>(+0.127) | 28.668<br>(-0.098) | 0.894<br>(-0.005) | 0.258<br>(+0.014) | 28.358<br>(-1.686) | 0.896<br>(-0.010) | 0.258<br>(+0.017) |  |
| Ours | Random | 24.816 | 0.865 | 0.169 | 21.436 | 0.786 | 0.244 | 28.675 | 0.896 | 0.260 | 30.165 | 0.903 | 0.250 |

**Table 4: Quantitative comparison on Mip-NeRF360, Tanks&Temples and Deep Blending datasets. We compare our method with previous approaches, including the random initialization method described in the original 3DGS []. ‘Cluster’ and ‘<10%’ means our model utilizing clustered SfM points and top 10% of points with the least reprojection error, respectively.**
| Method | SfM<br>points | Mip-NeRF360 Outdoor Scene |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| bicycle | flowers | garden | stump | treehill |  |  |  |  |  |  |  |  |  |  |  |  |
| PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ | PSNR$\uparrow$ | SSIM $\uparrow$ | LPIPS$\downarrow$ | PSNR$\uparrow$ | SSIM $\uparrow$ | LPIPS$\downarrow$ |  |  |
| Plenoxels [] | ✗ | 21.912 | 0.496 | 0.506 | 20.097 | 0.431 | 0.521 | 23.495 | 0.606 | 0.386 | 20.661 | 0.523 | 0.503 | 22.248 | 0.509 | 0.540 |
| INGP-Base [] | ✗ | 22.193 | 0.491 | 0.487 | 20.348 | 0.450 | 0.481 | 24.599 | 0.649 | 0.312 | 23.626 | 0.574 | 0.450 | 22.364 | 0.518 | 0.489 |
| INGP-Big [] | ✗ | 22.171 | 0.512 | 0.446 | 20.652 | 0.486 | 0.441 | 25.069 | 0.701 | 0.257 | 23.466 | 0.594 | 0.421 | 22.373 | 0.542 | 0.450 |
| 3DGS [] | ✗ | 21.034 | 0.575 | 0.378 | 17.815 | 0.469 | 0.403 | 23.217 | 0.783 | 0.175 | 20.745 | 0.618 | 0.345 | 18.986 | 0.550 | 0.413 |
| 3DGS [] | ✓ | 25.246 | 0.771 | 0.205 | 21.520 | 0.605 | 0.336 | 27.410 | 0.868 | 0.103 | 26.550 | 0.775 | 0.210 | 22.490 | 0.638 | 0.317 |
| Ours | ✗ | 25.042 | 0.747 | 0.238 | 21.762 | 0.616 | 0.324 | 26.884 | 0.854 | 0.114 | 26.680 | 0.768 | 0.215 | 22.528 | 0.621 | 0.342 |
| Ours | cluster | 25.211 | 0.767 | 0.211 | 21.704 | 0.617 | 0.321 | 27.166 | 0.862 | 0.103 | 26.816 | 0.775 | 0.204 | 22.589 | 0.630 | 0.328 |
| Ours | <10% | 25.214 | 0.769 | 0.208 | 21.858 | 0.619 | 0.320 | 27.130 | 0.861 | 0.107 | 26.899 | 0.777 | 0.202 | 22.618 | 0.632 | 0.327 |
| Method | SfM<br>points | Mip-NeRF360 Indoor Scene | Mip-NeRF360<br>Average |  |  |  |  |  |  |  |  |  |  |  |  |  |
| room | counter | kitchen | bonsai |  |  |  |  |  |  |  |  |  |  |  |  |  |
| PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ | PSNR$\uparrow$ | SSIM $\uparrow$ | LPIPS$\downarrow$ | PSNR$\uparrow$ | SSIM $\uparrow$ | LPIPS$\downarrow$ |  |  |
| Plenoxels [] | ✗ | 27.594 | 0.842 | 0.419 | 23.624 | 0.759 | 0.441 | 23.420 | 0.648 | 0.447 | 24.669 | 0.814 | 0.398 | 23.080 | 0.625 | 0.462 |
| INGP-Base [] | ✗ | 29.269 | 0.855 | 0.301 | 26.439 | 0.798 | 0.342 | 28.548 | 0.818 | 0.254 | 30.337 | 0.890 | 0.227 | 25.302 | 0.671 | 0.371 |
| INGP-Big[] | ✗ | 29.690 | 0.871 | 0.261 | 26.691 | 0.817 | 0.306 | 29.479 | 0.858 | 0.195 | 30.685 | 0.906 | 0.205 | 25.586 | 0.699 | 0.331 |
| 3DGS [] | ✗ | 29.685 | 0.894 | 0.265 | 23.608 | 0.833 | 0.276 | 26.078 | 0.893 | 0.161 | 18.538 | 0.719 | 0.401 | 22.190 | 0.704 | 0.313 |
| 3DGS [] | ✓ | 30.632 | 0.914 | 0.220 | 28.700 | 0.905 | 0.204 | 30.317 | 0.922 | 0.129 | 31.980 | 0.938 | 0.205 | 27.205 | 0.815 | 0.214 |
| Ours | ✗ | 30.809 | 0.906 | 0.247 | 28.529 | 0.895 | 0.223 | 31.270 | 0.920 | 0.137 | 31.547 | 0.934 | 0.218 | 27.228 | 0.807 | 0.229 |
| Ours | cluster | 30.363 | 0.909 | 0.233 | 28.594 | 0.901 | 0.211 | 31.136 | 0.923 | 0.133 | 31.695 | 0.935 | 0.217 | 27.253 | 0.813 | 0.218 |
| Ours | <10% | 30.843 | 0.912 | 0.230 | 28.703 | 0.902 | 0.209 | 31.364 | 0.922 | 0.133 | 31.829 | 0.938 | 0.214 | 27.384 | 0.815 | 0.217 |

| Methods | SfM<br>points | Tanks&Temples | Deep Blending |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Truck | Train | DrJohnson | Playroom |  |  |  |  |  |  |  |  |  |  |
| PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ | PSNR$\uparrow$ | SSIM $\uparrow$ | LPIPS$\downarrow$ |  |  |
| Plenoxels [] | ✗ | 23.221 | 0.774 | 0.335 | 18.927 | 0.663 | 0.422 | 23.142 | 0.787 | 0.521 | 22.980 | 0.802 | 0.465 |
| INGP-Base [] | ✗ | 23.260 | 0.779 | 0.274 | 20.170 | 0.666 | 0.386 | 27.750 | 0.839 | 0.381 | 19.483 | 0.754 | 0.465 |
| INGP-Big [] | ✗ | 23.383 | 0.800 | 0.249 | 20.456 | 0.689 | 0.360 | 28.257 | 0.854 | 0.352 | 21.665 | 0.779 | 0.428 |
| 3DGS [] | ✗ | 21,149 | 0.758 | 0.248 | 20.824 | 0.772 | 0.255 | 28.668 | 0.894 | 0.258 | 28.358 | 0.896 | 0.258 |
| 3DGS [] | ✓ | 25.187 | 0.879 | 0.148 | 21.097 | 0.802 | 0.218 | 28.766 | 0.899 | 0.244 | 30.044 | 0.906 | 0.241 |
| Ours | ✗ | 24.816 | 0.865 | 0.169 | 21.436 | 0.786 | 0.244 | 28.675 | 0.896 | 0.260 | 30.165 | 0.903 | 0.250 |
| Ours | cluster | 24.893 | 0.869 | 0.160 | 21.180 | 0.786 | 0.243 | 28.529 | 0.895 | 0.260 | 29.994 | 0.902 | 0.250 |
| Ours | <10% | 25.190 | 0.873 | 0.159 | 21.741 | 0.796 | 0.235 | 28.567 | 0.896 | 0.261 | 30.065 | 0.902 | 0.249 |

Figure: Figure 4: Qualitative results on Mip-NeRF360, Tanks&Temples, and Deep Blending dataset.
Refer to caption: x4.png

### 6.3 Quantitative comparison with 3DGS in real-word scenarios

As mentioned above, SfM might fail or produce noisy point cloud in real-world scenarios. To assess the robustness of 3DGS [] in these cases, we evaluate 3DGS trained with SfM points, noisy SfM points, and random points. The results are compared with ours trained with random points.

As shown in Table 3, 3DGS is highly dependent on the accuracy of initial point cloud, showing large performance drops when trained with noisy SfM point cloud (denoted as Noisy SfM) and random point cloud (denoted as Random). In contrast, our strategy trained from random point cloud shows on-par or even better results compared to 3DGS trained with accurate SfM point cloud.

### 6.4 Qualitative and quantitative comparison

In Table 4, we compare our model with Plenoxels [], InstantNGP-Base, InstantNGP-Big [], and 3DGS [] on the Mip-NeRF360 dataset [], Tanks&Temples dataset [], and Deep Blending dataset []. All NeRF methods [, , ] are trained without SfM points. We evaluate 3DGS [] and ours trained with SfM-initialized and randomly initialized point cloud. For ours trained with SfM-initialized point cloud, we evaluate utilizing different clustering methods of selecting the top 10% with the least reprojection error (denoted as <10%) or using cluster centers (denoted as cluster) obtained through HDBSCAN [].

In Figure 4, we provide qualitative results on novel test views. We compare with InstantNGP-Base ((a)), 3DGS trained with random point cloud ((b)) and SfM initialized point cloud ((c)), and ours trained with random point cloud ((d)). By comparing (b) and (d), we demonstrate the effectiveness of our strategy, which removes any unwanted high-frequency artifacts or floaters and also successfully captures scene objects located far away from the viewpoint. By comparing (c) and (d), ours trained with random point cloud show better results than 3DGS trained with SfM initialized point cloud, capturing fine details more effectively and rendering the background more accurately. More qualitative images are included in Section E in the supplementary materials.

### 6.5 Ablation studies

**Table 5: Ablation on core components**
| Low-pass filter | Init. | ABE-Split | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ |
| --- | --- | --- | --- | --- | --- |
| Constant | DSV | ✗ | 22.190 | 0.704 | 0.313 |
| Constant | SLV | ✗ | 25.675 | 0.749 | 0.288 |
| Ours | SLV | ✗ | 26.180 | 0.761 | 0.280 |
| Ours | SLV | ✓ | 27.228 | 0.807 | 0.229 |

In Table 5, we validate the effectiveness of each component in our method trained in the Mip-NeRF360 dataset [] with randomly initialized point cloud. The comparison of Sparse-large-variance (SLV) and dense-small-variance (DSV) initialization is done by using $N=10$ and $N=1,000,000$ respectively. We compare our progressive Gaussian low-pass filter control method with using a constant low-pass filter value ($s=0.3$) as done in the original 3DGS []. We also compare utilizing our novel ABE-Split method and the original split algorithm utilized in 3DGS. All three ablations show that using our components achieves significantly better results and using all three key components achieves the best results. More detailed and various ablation studies can be found in Section D.2 in the supplementary materials.

## 7 Conclusion

In this work, we introduce RAIN-GS (Relaxing Accurate INitialization Constraint for 3D Gaussian Splatting), a novel optimization strategy that effectively removes the constraint for accurate initialization. By combining sparse-large-variance (SLV) initialization, progressive Gaussian low-pass filter control, and the adaptive bound-expanding split (ABE-Split) algorithm, our strategy successfully guides 3D Gaussians to prioritize the learning of low-frequency components and encourages the Gaussians to transport to further locations from the initialized locations. RAIN-GS effectively removes the strict reliance on accurate point cloud achieved from Structure-from-Motion (SfM), opening up new possibilities for 3DGS in scenarios where achieving accurate point cloud is challenging.

## Supplementary materials

In the supplementary materials, we provide a more detailed analysis of our experiments and implementation details, together with additional results of rendered images using our strategy. Specifically, in Section A, we provide the implementation details of how our RAIN-GS (Relaxing Accurate INitialization Constraint for 3D Gaussian Splatting) can be applied to original 3D Gaussian splatting (3DGS) []. In Section B, we give a more detailed explanation of the convolution process with the Gaussian low-pass filter used in the rendering stage and how we choose the low-pass filter value $s$ (diagonal values of the covariance matrix $sI$ of the Gaussian low-pass filter) for our progressive Gaussian low-pass filter control. In Section C, we explain the details of our toy experiment shown in Section 5.1 of our main paper: learning to regress 1D signal with multiple 1D Gaussians. In Section D, we provide additional analysis including comparisons of computational resources with different methods and limitations of our strategy. In Section E, we provide additional qualitative results. In Section F, we show how our strategy can be applied to training 3DGS with a limited number of images.

## Appendix A Implementation Details

### A.1 RAIN-GS

We implement our strategy based on the official code of 3DGS []. As mentioned in Section  6.2 of our main paper, except for the hyperparameters for increasing spherical harmonic (SH) degrees and 3D Gaussian scale division ratio, we follow the original implementation details and hyperparameters of 3DGS []. We start increasing SH degree every 1,000 steps after 5,000 steps and we lower 3D Gaussian scale division ratio 1.6 to 1.4.
Our novel strategy consists of three components: 1) sparse-large-variance (SLV) random initialization, 2) progressive Gaussian low-pass filter control and 3) ABE-Split.
While 3DGS begins with the SFM point cloud which provides a coarse approximation of the scene, RAIN-GS starts with significantly fewer Gaussians and without any coarse information. Consequently to achieve an initial coarse approximation, it requires more time and steps to densify the Gaussians. Thus we add warm-up phase in training and our three main components are only applied during the warm-up phase. Specifically, the traditional 3DGS training can be divided into two main phases: (1) densification and (2) optimization. The densification phase typically involves about 15,000 steps, followed by 15,000 steps of optimization. The learning rate of the mean $\mu$ start with $1.6e^{-4}$ and exponentially decay to $1.6e^{-6}$ during densification phase. For RAIN-GS, however, the optimization phase is reduced to 5,000 steps, and 10,000 steps of warm-up phase is introduced prior to densification phase. While warm-up phase, the learning rate of the mean $\mu$ is kept $1.6e^{-4}$ and our componets are applied to help the Gaussians learn a coarse approximation. Then, the remaining process follows the same learning process as the original 3DGS.

#### A.1.1 Sparse-large-variance (SLV) random initialization

Following the original implementation of 3DGS [], we initialize random point cloud within the boundary defined as three times the size of the camera’s bounding box. Instead of the original dense-small-variance random initialization where the initial number of Gaussians $N$ is set as $N>100K$, we only initialize 10 Gaussians as $N=10$. As the initial covariance of 3D Gaussian is defined based on the distances of the three nearest neighbors, sparse initialization leads to a larger initial covariance, resolving the requirement for additional modification. This simple change of code (expressed with a gray box in Algorithm 1) largely improves the performance of 3DGS based on our analysis of sparse-large-variance initialization. An illustrations of SfM initialization, dense-small-variance (DSV) initialization and sparse-large-variance (SLV) initialization are shown in Figure 5.

Figure: Figure 5: Visualization of different initialization methods. This figure illustrates the effect of different initialization methods. (a) Ground truth image. (b) Initialized point cloud from Structure-from-Motion (SfM). (c) Dense-small-variance (DSV) random initialization with small initial covariances due to the short distance between Gaussians. (d) Sparse-large-variance (SLV) random initialization with large initial covariances due to wider distance between Gaussians.
Refer to caption: x5.png

Figure: Algorithm 1 Sparse-large-variance (SLV) random initialization

#### A.1.2 Progressive Gaussian low-pass filter control

In the original implementation of 3DGS, the Gaussian low-pass filter is used in the rendering stage to ensure the projected 2D Gaussians to cover at least one pixel in the screen space. To enlarge the 2D Gaussians, 3DGS uses a fixed Gaussian low-pass filter (mean $\mu=0$ and variance $\sigma^{2}=s=0.3$). Instead of using a fixed sigma value $\sigma$ for this low-pass filter, we propose a progressive Gaussian low-pass filter control, where the sigma value starts with a large value and progressively reduces to $\sigma^{2}=0.3$. This can be efficiently implemented. Instead of passing a fixed value of $s=0.3$ as the diagonal values of the covariance matrix $sI$ of the Gaussian low-pass filter, we pass a progressive value adaptively defined with the image height, width, and number of Gaussians taken into account as $\text{min(}\text{max(}HW/9\pi N,0.3\text{), 300.0)}$ every 1,000 steps. Our implementation is expressed in line 8 of Algorithm 2.

Figure: Algorithm 2 Progressive Gaussian low-pass filter control & ABE-Split

#### A.1.3 ABE-Split

In addition to the split of 3DGS, the ABE-Split generates an additional Gaussian and transport it to a distant location. Specifically, new position is determined by calculating the vector between the center of bounding box and the original Gaussian position, then multiplying it by 0.3 times the scene extent. The pseudo code for this process is provided in line 21 and 22 of Algorithm 2.
Due to our SLV initialization and progressive Gaussian low-pass filtering, the Gaussians are sparse and have large covariance during the early steps of training, resulting in a majority of the Gaussians only undergoing the split process. Thus, the ABE-Split can transport a sufficient number of Gaussians in early stage.

## Appendix B Proof

Figure: Figure 6: Visualization of low-pass filter. This figure shows the visualization of the effect of the low-pass filter. As shown in (b), the convolution of the splatted 2D Gaussian with the low-pass filter expands the area the Gaussian is splatted onto, resulting in the Gaussians affecting larger areas than naïve splatting as shown in (a).
Refer to caption: x6.png

### B.1 Proof on radius of a Gaussian convolved with a low-pass filter

As mentioned in Section 5.2 of our main paper, the 3D Gaussians $G_{i}$ is projected to 2D Gaussians $G_{i}^{\prime}$ in the screen space as follows:

$$ $G_{i}^{\prime}(x)=e^{-\frac{1}{2}(x-\mu_{i}^{\prime})^{T}{\Sigma_{i}^{\prime}} ^{-1}(x-\mu_{i}^{\prime})}.$ (6) $$

To ensure the 2D Gaussian $G_{i}^{\prime}$ to cover at least one pixel, 3DGS adds a small value $s$ to the diagonal elements of the 2D covariance $\Sigma_{i}^{\prime}$ as follows:

$$ $G_{i}^{\prime}(x)=e^{-\frac{1}{2}(x-\mu_{i}^{\prime})^{T}(\Sigma_{i}^{\prime}+ sI)^{-1}(x-\mu_{i}^{\prime})},$ (7) $$

where $I$ is the $2\times 2$ identity matrix. This process can be understood as the convolution between the 2D Gaussian $G_{i}^{\prime}$ and the Gaussian low-pass filter $h$ (mean $\mu=0$ and variance $\sigma^{2}=s=0.3$) of $G_{i}^{\prime}\otimes h$. This is due to the nature of Gaussians where the convolution of Gaussians with the variance matrices $V$ and $Z$ results in a Gaussian with the variance matrix $V+Z$ as follows:

$$ $G_{1}(x)=e^{-\frac{1}{2}(x-\mu_{i})^{T}{V}^{-1}(x-\mu_{i})}\quad G_{2}(x)=e^{- \frac{1}{2}(x-\mu_{i})^{T}{Z}^{-1}(x-\mu_{i})},$ (8) $$

$$ $(G_{1}\otimes G_{2})(x)=e^{-\frac{1}{2}(x-\mu_{i})^{T}({V+Z})^{-1}(x-\mu_{i})}.$ (9) $$

Following the convolution process, 3DGS estimates the projected 2D Gaussian’s area to identify its corresponding screen tiles. This is done by calculating $k$ times the square root of the larger eigenvalue of $(\Sigma_{i}^{\prime}+sI)$, which represents the radius of the approximated circle, and $k$ is the hyperparameter that determines the confidence interval of the 2D Gaussian. Figure 6 illustrates the low-pass filter’s effect, where the projected Gaussian is splatted to wider areas in (b) compared to (a).

### B.2 Proof on progressive low-pass filter size

In Section 5.2 of our main paper, we define the value $s$ for our progressive Gaussian low-pass filter control based on the fact that the area of the projected 2D Gaussians is at least $9{\pi}s$. As the area of the projected 2D Gaussian is defined as the circle whose radius is $k$ times the square root of the larger eigenvalue of $(\Sigma_{i}^{\prime}+sI)$, we have to first calculate the eigenvalues of $(\Sigma_{i}^{\prime}+sI)$. If we define the eigenvalues of $\Sigma_{i}$ as $\lambda_{i1},\lambda_{i2}$, since the eigenvalue of $sI$ is $s$, the eigenvalues of $(\Sigma_{i}^{\prime}+sI)$ can be defined as $\lambda_{i1}+s,\lambda_{i2}+s$. This leads to the following proof:

$$ $\begin{split}&r=k\cdot\sqrt{\text{max(}\lambda_{i1},\lambda_{i2}\text{)}+s},\\ &r\geq k\cdot\sqrt{s},\\ &\pi r^{2}\geq k^{2}\pi s,\end{split}$ (10) $$

where $k$ is the hyperparameter that defines the confidence interval of the Gaussian. We follow the original implementation of 3DGS as $k=3$ which gives the 99.73$\%$ confidence interval. Using the value $k=3$ leads to the proof of the area of each Gaussian being at least $9\pi s$.

## Appendix C Toy experiments

### C.1 Implementation details

In this section, we provide a detailed explanation of the 1D regression toy experiment shown in Section 5.1 of our main paper. We design the experiment to learn a target distribution $Y(x)$, where $x$ is in the range $[0,10,000)$, using $N$ 1D Gaussians. Each of the $N$ 1D Gaussians includes its mean $\mu_{i}$ and variance $\sigma_{i}^{2}$ as learnable parameters to model a random 1D signal $Y(x)$ as the ground truth distribution. The target distribution $Y(x)$ is generated from random 10 1D Gaussian distributions. By blending $N$ 1D Gaussians with a learnable weight $w_{i}$, we train the parameters $\{\mu_{i},\sigma_{i},w_{i}\}$ through the L1 loss between the blended distribution from $N$ 1D Gaussians and $Y(x)$ as follows:

$$ $\mathcal{L}=\sum_{x}\ \left\lVert Y(x)-\sum_{i=0}^{N}w_{i}\cdot\exp\left(- \frac{(x-\mu_{i})^{2}}{2\sigma_{i}^{2}}\right)\right\rVert,$ (11) $$

For the dense-small-variance (DSV) scenario, we initialize the number of Gaussians as $N=1,000$ and randomly select $\mu_{i}$ and $\sigma_{i}$ values within the range $[0,1)$. For the dense-large-variance (DLV) scenario, we use the same number of Gaussians as $N=1,000$ but choose $\mu_{i}$ and $\sigma_{i}$ values randomly from the range $[300,301)$. For the sparse-large-variance (SLV) scenario, we set the number of initial Gaussians as $N=15$ and randomly select $\mu_{i}$ and $\sigma_{i}$ values from the range $[300,301)$. In all scenarios, we train the parameters for a total of 1,000 steps using the Adam optimizer with a learning rate of 0.01.

Figure: Figure 7: Toy experiment to analyze different initialization methods. This figure visualizes the result of our toy experiment predicting the target distribution using a collection of 1D Gaussians, starting from different initialization methods. Dense-small-variance (DSV) and dense-large-variance (DLV) initialize 1,000 1D Gaussians, where DLV is initialized with large variances by adding $s$ to the initial variance. Small-large-variance (SLV) initializes 15 1D Gaussians with the same $s$ added to the initial variance. We can observe the over-fast convergence of high-frequency components on DSV initialization, which is resolved in the DLV initialization but fails to converge due to fluctuation. SLV initialization resolves both problems, learning the low-frequency components first and also converging to successfully model the target distribution.
Refer to caption: x7.png

### C.2 Analysis on toy experiments

As mentioned in Section 5.1, we conduct a toy experiment in a simplified 1D regression task to examine how DSV and SLV initialization influences the optimization process. As shown in the top row of Figure 7, training with the DSV random initialization exhibits a tendency towards over-fast convergence on high frequencies. This is evident in the model’s ability to capture high-frequency components after only 10 steps. However, this rapid focus on high-frequencies leads to undesired high-frequency artifacts in the final reconstruction (zoomed-in box at 1,000 steps). We hypothesize that this behavior arises from the small initial variances of the dense Gaussians, which restrict each Gaussian to model a very localized region, increasing the susceptibility to overfitting.

To verify our hypothesis, we repeat the experiment with a value $s$ added to the initial variance changing our initialization to dense-large-variance (DLV). Note that this modification effectively encourages the Gaussians to learn from wider areas. The results (middle row of Figure 7) support our hypothesis: the learned signal at 10 steps demonstrates a prioritization of low-frequency components when compared to DSV initialization. However, even with the ability to prune Gaussians by learning weights as $w_{i}=0$, the dense initialization causes fluctuations in the learned signal, preventing convergence after 1,000 steps. These observations highlight a crucial point: while learning from wider areas (enabled by a large variance) is necessary to prioritize low-frequency components, a dense initialization can still lead to instability and convergence issues.

To resolve both problems, we propose a sparse-large-variance (SLV) initialization. The sparsity reduces fluctuations throughout the optimization, while the large variance ensures initial focus on the low-frequency distribution. This is demonstrated in the bottom row of Figure 7, where the experiment is repeated with $N=15$ Gaussians with $s$ added to the initial variance. SLV initialization succeeds in both the prioritization of low-frequency components at 10 steps and the modeling of the true distribution with minimal errors after 1,000 steps.

Therefore, although initializing random points from the same volume which is defined as three times the size of the camera’s bounding box, we initialize a significantly sparser set of 3D Gaussians. As the initial covariance of 3D Gaussian is defined based on the distances of the three nearest neighbors, sparse initialization leads to a larger initial covariance, encouraging each Gaussian to model a wider region of the scene.

### C.3 Analysis on dense-large-variance (DLV) initialization in 3DGS

We compare the influence of different dense initialization methods through the toy experiment of learning to regress a random 1D signal with multiple 1D Gaussians. Through the toy experiment, we analyzed that while Gaussian learning from wider areas (enabled by the large initial variance) leads to prioritizing low-frequency components, the dense initialization results in instability and convergence issues. As shown in Figure 7, starting from Gaussians with large variance (dense-large-variance (DLV)) leads to prioritizing the learning of low-frequency components, compared to dense-small-variance (DSV) initialization, observed in the coarse signal in the bottom row of step 10.

Although DLV initialization succeeds in learning the low-frequency components of the true distribution first, due to the dense number of Gaussians, DLV initialization fails to converge with large fluctuations. This can be observed by comparing the learned signals from steps 10, 100, and 1000. The learned signal is already quite close to the true distribution in 10 steps but fails to converge even after a sufficient number of steps.

To verify that our findings also apply to the initialization of 3D Gaussians for 3DGS, we implement the DLV initialization and evaluate the performance of 3DGS (DLV) on the Mip-NeRF360 dataset. For this experiment, we set the initial number of Gaussians as $N=1,000,000$, equal to our setting for DSV initialization. As the initial covariance is defined by the mean distance of the three nearest neighbors, we manually scale up the initial covariance to guide the Gaussians to learn from wider areas. The results are shown in Table 6, where the performance of DLV random initialization is between DSV and SLV random initialization, aligned with the results of our toy experiment. Note that 3DGS (SLV) in Table 6 indicates 3DGS trained using only the sparse-large-variance (SLV) random initialization of our strategy.

**Table 6: Quantitative comparison on Mip-NeRF360 dataset with various initialization methods. We compare our method with the DSV random initialization method described in the original 3DGS [], DLV random initialization, and SLV random initialization method. We report PSNR, SSIM, LPIPS.**
| Method | Outdoor Scene |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| bicycle | flowers | garden | stump | treehill |  |  |  |  |  |  |  |  |  |  |  |
| PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ | PSNR$\uparrow$ | SSIM $\uparrow$ | LPIPS$\downarrow$ | PSNR$\uparrow$ | SSIM $\uparrow$ | LPIPS$\downarrow$ |  |
| 3DGS (DSV) | 21.034 | 0.575 | 0.378 | 17.815 | 0.469 | 0.403 | 23.217 | 0.783 | 0.175 | 20.745 | 0.618 | 0.345 | 18.986 | 0.550 | 0.413 |
| 3DGS (DLV) | 21.595 | 0.552 | 0.426 | 18.959 | 0.469 | 0.445 | 23.338 | 0.757 | 0.218 | 19.845 | 0.555 | 0.427 | 19.605 | 0.550 | 0.440 |
| 3DGS (SLV) | 23.227 | 0.610 | 0.373 | 20.498 | 0.528 | 0.392 | 25.670 | 0.816 | 0.157 | 23.748 | 0.647 | 0.339 | 21.423 | 0.578 | 0.402 |
| Method | Indoor Scene | Average |  |  |  |  |  |  |  |  |  |  |  |  |  |
| room | counter | kitchen | bonsai |  |  |  |  |  |  |  |  |  |  |  |  |
| PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ | PSNR$\uparrow$ | SSIM $\uparrow$ | LPIPS$\downarrow$ | PSNR$\uparrow$ | SSIM $\uparrow$ | LPIPS$\downarrow$ |  |
| 3DGS (DSV) | 29.685 | 0.894 | 0.265 | 23.608 | 0.833 | 0.276 | 26.078 | 0.893 | 0.161 | 18.538 | 0.719 | 0.401 | 22.190 | 0.705 | 0.313 |
| 3DGS (DLV) | 29.284 | 0.886 | 0.280 | 26.253 | 0.857 | 0.272 | 27.975 | 0.900 | 0.161 | 22.363 | 0.793 | 0.360 | 23.246 | 0.702 | 0.337 |
| 3DGS (SLV) | 29.966 | 0.892 | 0.268 | 27.473 | 0.868 | 0.260 | 29.934 | 0.915 | 0.159 | 29.132 | 0.900 | 0.251 | 25.675 | 0.750 | 0.287 |

## Appendix D Additional analysis

### D.1 Analysis on prioritized learning of low-frequency components

In Section 4 of our main paper, we find that SfM initialization guides 3D Gaussians with a coarse approximation of the true distribution to robustly learn the remaining high-frequency components. In Section 5.1 of our main paper, we then show that sparse-large-variance (SLV) random initialization successfully guides the Gaussians to prioritize the learning of low-frequency components. To verify that our strategy successfully guides 3DGS to prioritize the learning of low-frequency components, we provide additional analysis of our strategy through the rendered images achieved from different steps of training.

Figure: Figure 8: Visualization of our strategy prioritizing the learning of coarse approximation. This figure visualizes the rendering result of the ’bicycle’ scene from the Mip-NeRF360 dataset trained with 3DGS using different initialization methods. We show the images rendered from different steps of 100, 3,000, 5,000, and 30,000. The results of 3DGS trained from SfM initialization, DSV random initialization, and Ours are shown from top to bottom.
Refer to caption: x8.png

As shown in Figure 8, we compare the rendered results of 3DGS trained with different initialization. As SfM provides a coarse approximation of the true distribution, 3DGS trained with SfM initialization (first row) directly learns the remaining high-frequency details. This tendency to directly learn the high-frequency components is highlighted in the second row where 3DGS is trained with DSV random initialization. Without any strategy to prioritize the learning of low-frequency components, the high-frequency components such as the edges of the bench and bicycle are already learned as shown in the rendered image of step 3,000. In contrast, our strategy (last row) successfully guides the Gaussians to first model the scene’s coarse approximation. At step 3,000, high-frequency details are absent, resulting in smoothed appearances for the grass, road, and background trees.

Figure: Figure 9: Images from Figure 8 analyzed in the frequency domain.
Refer to caption: x9.png

In Figure 9, we further analyze the rendered images of Figure 8 in the frequency domain. We sample a horizontal line from the images as in (a) and perform FFT []. From (b) to (d), the normalized DFT magnitude of the transformed signal from the GT image and the rendered image is shown in sequential timesteps of 100, 3,000, and 5,000.

As mentioned in Section 4 of our main paper, 3DGS (top row) starts from a coarse approximation of the signal, which can be observed by the low-frequency components already being captured in step 100 ((b), top row). This is more evident when compared to the transformed signal of 3DGS (DSV) and Ours at step 100 ((b), middle and bottom row), where the low-frequency components are largely different from the GT signal. Aligned with our analysis of the toy experiment in Section 5.1 of our main paper, DSV random initialization exhibits a tendency towards over-fast convergence on high-frequencies. This can be observed by comparing (b) and (c), where the high-frequency components are quickly learned in (c). This leads to high-frequency artifacts after optimization, as shown in the rendered image (step 30,000, middle row) in Figure 8.

On the other hand, our strategy successfully guides the Gaussians to prioritize the learning of low-frequency components. When comparing (b), (c), and (d) of the last row, the low-frequency components are learned in (c) and then the high-frequency details are captured in (d). Note that this prioritization of learning low-frequency components makes the overall training process similar to starting with SfM initialization. The similarity is further highlighted by comparing the transformed signals of 3DGS in (b) and Ours in (c).

### D.2 Ablation study on warm up phase

##### Ablation settings for low-pass filter control.

To demonstrate the effectiveness of our progressive Gaussian low-pass filter control strategy, we employ three different decreasing functions of convex, linear, and concave to control the Gaussian low-pass filter value $s$.
Different from our strategy, where the value $s$ is defined adaptively by image height, width, and the number of Gaussians $N$ at each time step, the remaining functions are manually defined to achieve $s=300$ at step 0 and $s=0.3$ at about 3,000 steps across all scenes. The intuition behind this design is based on our analysis that our adaptive Gaussian low-pass filter value reaches $0.3$ between 2,000-3,000 steps. Also, we empirically find that the initial Gaussian low-pass filter value $s>300$ offers no significant improvement, only making the overall computation inefficient. Based on these findings, we define the max value of the Gaussian low-pass filter as $s=300$.

For the convex function, we use the following formula for $s$ scheduling:

$$ $s=\max(7^{-\frac{x}{1000}}*300,0.3).$ (12) $$

For the linear function, we use the following formula for $s$ scheduling:

$$ $s=\max(300-0.0997084x,0.3).$ (13) $$

For the concave function, we use the following formula for $s$ scheduling:

$$ $s=\max(300*(1+7^{-3}-7^{\frac{x-3000}{1000}}),0.3).$ (14) $$

The illustration of different Gaussian low-pass value formulas is shown in Figure 11 and Figure 11 where our formula is adaptively defined, showing different functions for each scene.

##### Comparison.

In Table 11, we show a more detailed ablation study on the components belonging to the warm up phase. Additional experiments regarding (1) fixing learning rate during the warm up phase (2) the other progressive low-pass filter control functions mentioned above are shown here. Results show that our choice of dynamically adjusting the low-pass filter value based on the number of Gaussians achieves the best performance compared to the others. Also as can be seen in the Table, each of fixing learning rate, SLV initialization and ABE-Split solely contributes to the performance improvement, verifying the effect of our proposed methods. Compared to the original DSV random initialization, we achieve an significant average PSNR improvement over 5dB.

Figure: Figure 10: Illustration of different Gaussian low-pass filter value formulas.
Refer to caption: x10.png

**Table 7: Ablation study on warm up phase.**
| Low-pass filter | Init. | Fixed learning rate | ABE-Split | Mip-NeRF360 |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ |  |  |  |  |
| Constant | DSV | ✗ | ✗ | 22.190 | 0.704 | 0.313 |
| Constant | SLV | ✗ | ✗ | 25.675 | 0.749 | 0.288 |
| Progressive (Convex) | SLV | ✗ | ✗ | 25.799 | 0.756 | 0.286 |
| Progressive (Linear) | SLV | ✗ | ✗ | 25.898 | 0.754 | 0.290 |
| Progressive (Concave) | SLV | ✗ | ✗ | 25.974 | 0.754 | 0.288 |
| Ours | SLV | ✗ | ✗ | 26.180 | 0.761 | 0.280 |
| Ours | SLV | ✓ | ✗ | 26.362 | 0.791 | 0.241 |
| Ours | SLV | ✓ | ✓ | 27.228 | 0.807 | 0.229 |

**Table 8: Quantitative comparison on Mip-NeRF360 dataset in sparse SfM points setting. We compare our method with 3DGS in sparse SfM points setting, We report PSNR, SSIM and LPIPS.**
| Method | SfM<br>points | Outdoor Scene |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| bicycle | flowers | garden | stump | treehill |  |  |  |  |  |  |  |  |  |  |  |  |
| PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ | PSNR$\uparrow$ | SSIM $\uparrow$ | LPIPS$\downarrow$ | PSNR$\uparrow$ | SSIM $\uparrow$ | LPIPS$\downarrow$ |  |  |
| 3DGS | cluster | 24.542 | 0.703 | 0.289 | 21.044 | 0.569 | 0.365 | 26.883 | 0.852 | 0.118 | 25.774 | 0.735 | 0.250 | 22.102 | 0.601 | 0.376 |
| 3DGS | <10% | 24.456 | 0.700 | 0.294 | 21.260 | 0.580 | 0.360 | 26.832 | 0.850 | 0.127 | 26.165 | 0.744 | 0.247 | 22.260 | 0.612 | 0.365 |
| 3DGS | full | 25.246 | 0.771 | 0.205 | 21.520 | 0.605 | 0.336 | 27.410 | 0.868 | 0.103 | 26.550 | 0.775 | 0.210 | 22.490 | 0.638 | 0.317 |
| Ours | cluster | 25.211 | 0.767 | 0.211 | 21.704 | 0.617 | 0.321 | 27.166 | 0.862 | 0.103 | 26.816 | 0.775 | 0.204 | 22.589 | 0.630 | 0.328 |
| Ours | <10% | 25.214 | 0.769 | 0.208 | 21.858 | 0.619 | 0.320 | 27.130 | 0.861 | 0.107 | 26.899 | 0.777 | 0.202 | 22.618 | 0.632 | 0.327 |
| Method | SfM<br>points | Indoor Scene | Average |  |  |  |  |  |  |  |  |  |  |  |  |  |
| room | counter | kitchen | bonsai |  |  |  |  |  |  |  |  |  |  |  |  |  |
| PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ | PSNR$\uparrow$ | SSIM $\uparrow$ | LPIPS$\downarrow$ | PSNR$\uparrow$ | SSIM $\uparrow$ | LPIPS$\downarrow$ |  |  |
| 3DGS | cluster | 29.919 | 0.899 | 0.255 | 28.256 | 0.893 | 0.227 | 30.127 | 0.917 | 0.140 | 31.247 | 0.932 | 0.220 | 26.655 | 0.789 | 0.249 |
| 3DGS | <10% | 30.795 | 0.908 | 0.243 | 28.523 | 0.896 | 0.223 | 30.120 | 0.915 | 0.140 | 31.625 | 0.934 | 0.221 | 26.893 | 0.793 | 0.247 |
| 3DGS | full | 30.632 | 0.914 | 0.220 | 28.700 | 0.905 | 0.204 | 30.317 | 0.922 | 0.129 | 31.980 | 0.938 | 0.205 | 27.205 | 0.815 | 0.214 |
| Ours | cluster | 30.363 | 0.909 | 0.233 | 28.594 | 0.901 | 0.211 | 31.136 | 0.923 | 0.133 | 31.695 | 0.935 | 0.217 | 27.253 | 0.813 | 0.218 |
| Ours | <10% | 30.843 | 0.912 | 0.230 | 28.703 | 0.902 | 0.209 | 31.364 | 0.922 | 0.133 | 31.829 | 0.938 | 0.214 | 27.384 | 0.815 | 0.217 |

### D.3 Ablation on sparse SfM initialization

In Table 8, we compare two methods for creating sparse SfM points to reduce the uncertainty of the SfM point cloud and use our coarse-to-fine method with the SfM point cloud: cluster and <10%. For the cluster, we use HDBSCAN [] to cluster the points based on the location of the SfM point cloud, and use the centroids of the clusters for the sparse SfM initialization. We conduct experiments using the default hyperparameters in scikit-learn [] for HDBSCAN []. For <10%, we create sparse SfM points by using only the top 10% of points utilizing the reprojection error value that comes out with the SfM output. We confirm that our method achieves additional performance improvements by using the SfM points together.

**Table 9: Quantitative comparison on Mip-NeRF360 dataset in noisy initial SfM point cloud settings. We compare 3DGS method with different noisy inital SfM point cloud. We report PSNR, SSIM, LPIPS.**
| Initialization | Outdoor Scene |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| bicycle | flowers | garden | stump | treehill |  |  |  |  |  |  |  |  |  |  |  |
| PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ | PSNR$\uparrow$ | SSIM $\uparrow$ | LPIPS$\downarrow$ | PSNR$\uparrow$ | SSIM $\uparrow$ | LPIPS$\downarrow$ |  |
| SfM | 25.246 | 0.771 | 0.205 | 21.520 | 0.605 | 0.336 | 27.410 | 0.868 | 0.103 | 26.550 | 0.775 | 0.210 | 22.490 | 0.638 | 0.317 |
| SfM + ${\epsilon}$ | 24.289 | 0.692 | 0.307 | 21.028 | 0.565 | 0.374 | 26.515 | 0.846 | 0.133 | 26.028 | 0.743 | 0.252 | 21.728 | 0.607 | 0.380 |
| SfM+constant | 23.619 | 0.625 | 0.358 | 21.139 | 0.569 | 0.364 | 25.663 | 0.809 | 0.163 | 23.382 | 0.641 | 0.335 | 21.989 | 0.593 | 0.380 |
| Initialization | Indoor Scene | Average |  |  |  |  |  |  |  |  |  |  |  |  |  |
| room | counter | kitchen | bonsai |  |  |  |  |  |  |  |  |  |  |  |  |
| PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ | PSNR$\uparrow$ | SSIM $\uparrow$ | LPIPS$\downarrow$ | PSNR$\uparrow$ | SSIM $\uparrow$ | LPIPS$\downarrow$ |  |
| SfM | 30.632 | 0.914 | 0.220 | 28.700 | 0.905 | 0.204 | 30.317 | 0.922 | 0.129 | 31.980 | 0.938 | 0.205 | 27.205 | 0.815 | 0.214 |
| SfM + ${\epsilon}$ | 30.740 | 0.901 | 0.260 | 23.872 | 0.838 | 0.285 | 24.477 | 0.877 | 0.175 | 25.823 | 0.875 | 0.265 | 24.944 | 0.771 | 0.270 |
| SfM+constant | 30.790 | 0.899 | 0.264 | 28.041 | 0.875 | 0.250 | 30.203 | 0.914 | 0.145 | 31.006 | 0.924 | 0.230 | 26.204 | 0.761 | 0.277 |

### D.4 Analysis on the impact of noise on the initial SfM point cloud

In Table 2, we present detailed results to further investigate the ability of the 3DGS optimization scheme to transport Gaussians to the correct 3DGS locations on Mip-NeRF360 datasets. Here, we conduct this experiments by adding random noise $\epsilon\sim\mathcal{N}(0,1)$ and constant systematic noise, whose value equals 2, to the initial SfM points. The results shown in Table 9 prove that 3DGS strongly depends on the initial point. Figure 12 shows initial SfM points and points with noise $\epsilon$. Even with the small amount of noise, 3DGS fails to move to the correct position.

Figure: Figure 12: Visualization of SfM points and points with noise $\epsilon$.
Refer to caption: x12.png

Figure: Figure 13: Displacements of Gaussians from initial positions
Refer to caption: x13.png

### D.5 Analysis on movement of each Gaussian

To observe the movement of the each Gaussian, we measure how far the Gaussian moves from its initial position during the training. An additional parameter is incorporated to record the initial position, ensuring that even when Gaussians are split or cloned, the initial position parameters are retained. Then, the movement is calculated as difference between the final position of each Gaussian after training and its respective initial position.

We conduct analysis on “Truck” scene of Tanks&Temples dataset, comparing the settings of 3DGS with SfM point initialization, 3DGS with random initialization, and our method. The mean, standard deviation, and the top 1% values of the movements is shown in Table 2. Additionally, Figure 13 shows the overall scene from the same camera viewpoint for each experiment to observe the differences in distribution of overall Gaussians between the beginning and 30,000 step. In case such as 3DGS with SfM initialization and random initialization, the positions of the Gaussians does not change significantly. However, our method shows substantial changes in comparison.

### D.6 Expanding Scene bound

As discussed in Section 4 and can be seen in supplementary materials Section D.5, Gaussians struggle to move further from the initialized locations. Therefore, in Table 10, we compare the results according to the initial scene bound and our method using the ABE-Split algorithm on the Mip-NeRF360 dataset to see if it is sufficient to initialize with a sufficiently wide scene bound. Extent $\times$ 3 and Extent $\times$ 10 are the initial scene bounds created by multiplying the camera extent given in the dataset by 3 and 10, respectively, and 3 is used in the original 3DGS. As can be seen in Extent x 10, it is confirmed that simply starting with a wide initial scene bound does not yield sufficient performance.

### D.7 Detailed results of Oracle scene bound experiments

To find out if the performance degradation of using noisy or random point cloud solely comes from losing the information of where the Gaussians should be located, we conduct a simple experiment of training 3DGS with random point cloud sampled from the oracle scene bounds. Specifically, here we assume the oracle scene bounds as the final scene bound defined by the final positions of the Gaussians, which are trained from point cloud achieved from SfM. Table 11 presents the results for each scene. The experiment reveals that SfM initialization provides additional benefits to 3DGS beyond merely placing the Gaussians in proper locations, as starting from random point cloud even in oracle scene bounds leads to sub-optimal results compared to the one starting from SfM-initialized point cloud.

**Table 10: Quantitative comparison on Mip-NeRF360 dataset within different scene bound settings. We compare our method (SLV) with 3DGS using random initialization within different scale bounding box. We report PSNR, SSIM, LPIPS.**
| Scene bound | Outdoor Scene |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| bicycle | flowers | garden | stump | treehill |  |  |  |  |  |  |  |  |  |  |  |
| PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ | PSNR$\uparrow$ | SSIM $\uparrow$ | LPIPS$\downarrow$ | PSNR$\uparrow$ | SSIM $\uparrow$ | LPIPS$\downarrow$ |  |
| Extent$\times$3 | 21.034 | 0.575 | 0.378 | 17.815 | 0.469 | 0.403 | 23.217 | 0.783 | 0.175 | 20.745 | 0.618 | 0.345 | 18.986 | 0.550 | 0.413 |
| Extent$\times$10 | 23.864 | 0.643 | 0.354 | 20.974 | 0.553 | 0.381 | 25.862 | 0.823 | 0.151 | 24.876 | 0.696 | 0.300 | 22.067 | 0.590 | 0.394 |
| Ours | 25.042 | 0.747 | 0.238 | 21.762 | 0.616 | 0.324 | 26.884 | 0.854 | 0.114 | 26.680 | 0.768 | 0.215 | 22.528 | 0.621 | 0.342 |
| Scene bound | Indoor Scene | Average |  |  |  |  |  |  |  |  |  |  |  |  |  |
| room | counter | kitchen | bonsai |  |  |  |  |  |  |  |  |  |  |  |  |
| PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ | PSNR$\uparrow$ | SSIM $\uparrow$ | LPIPS$\downarrow$ | PSNR$\uparrow$ | SSIM $\uparrow$ | LPIPS$\downarrow$ |  |
| Extent$\times$3 | 29.685 | 0.894 | 0.265 | 23.608 | 0.833 | 0.276 | 26.078 | 0.893 | 0.161 | 18.538 | 0.719 | 0.401 | 22.190 | 0.704 | 0.313 |
| Extent$\times$10 | 29.834 | 0.889 | 0.276 | 27.441 | 0.863 | 0.267 | 30.424 | 0.913 | 0.147 | 30.482 | 0.917 | 0.242 | 26.203 | 0.765 | 0.279 |
| Ours | 30.809 | 0.906 | 0.247 | 28.529 | 0.895 | 0.223 | 31.270 | 0.920 | 0.137 | 31.547 | 0.934 | 0.218 | 27.228 | 0.807 | 0.229 |

**Table 11: Quantitative comparison on Mip-NeRF360 dataset in oracle scene bound settings. We compare 3DGS method with random initialization in paper setting, SfM initialization and random initialization in oracle scene bound. We report PSNR, SSIM, LPIPS.**
| Initialization | Outdoor Scene |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| bicycle | flowers | garden | stump | treehill |  |  |  |  |  |  |  |  |  |  |  |
| PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ | PSNR$\uparrow$ | SSIM $\uparrow$ | LPIPS$\downarrow$ | PSNR$\uparrow$ | SSIM $\uparrow$ | LPIPS$\downarrow$ |  |
| 3DGS (Random) | 21.034 | 0.575 | 0.378 | 17.815 | 0.469 | 0.403 | 23.217 | 0.783 | 0.175 | 20.745 | 0.618 | 0.345 | 18.986 | 0.550 | 0.413 |
| 3DGS (SfM) | 25.246 | 0.771 | 0.205 | 21.520 | 0.605 | 0.336 | 27.410 | 0.868 | 0.103 | 26.550 | 0.775 | 0.210 | 22.490 | 0.638 | 0.317 |
| 3DGS (Oracle) | 24.227 | 0.664 | 0.328 | 21.353 | 0.582 | 0.356 | 26.791 | 0.843 | 0.131 | 24.634 | 0.678 | 0.318 | 22.219 | 0.600 | 0.378 |
| Initialization | Indoor Scene | Average |  |  |  |  |  |  |  |  |  |  |  |  |  |
| room | counter | kitchen | bonsai |  |  |  |  |  |  |  |  |  |  |  |  |
| PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ | PSNR$\uparrow$ | SSIM $\uparrow$ | LPIPS$\downarrow$ | PSNR$\uparrow$ | SSIM $\uparrow$ | LPIPS$\downarrow$ |  |
| 3DGS (Random) | 29.685 | 0.894 | 0.265 | 23.608 | 0.833 | 0.276 | 26.078 | 0.893 | 0.161 | 18.538 | 0.719 | 0.401 | 22.190 | 0.704 | 0.313 |
| 3DGS (SfM) | 30.632 | 0.914 | 0.220 | 28.700 | 0.905 | 0.204 | 30.317 | 0.922 | 0.129 | 31.980 | 0.938 | 0.205 | 27.205 | 0.815 | 0.214 |
| 3DGS (Oracle) | 30.192 | 0.895 | 0.265 | 27.812 | 0.873 | 0.251 | 30.797 | 0.917 | 0.140 | 30.303 | 0.916 | 0.238 | 26.481 | 0.774 | 0.267 |

### D.8 Comparison of computational resources

In this section, we now compare the computational resources of Plenoxels [], InstantNGP [], 3DGS [], 3DGS with random initialization, and 3DGS with our strategy. We evaluate different methods on the Mip-NeRF360 [], Tanks&Temples [], and Deep Blending [] dataset.

##### Training time.

Note that when pre-defined camera poses are available, methods other than 3DGS [] can bypass the time-consuming Structure-from-Motion (SfM) process. To highlight this overhead, in Table 12 we compare the training time of 3DGS with the SfM process taken into account. For 3DGS, we find that the number of Gaussians in each training step has a direct correlation to the overall training time - more Gaussians lead to longer training time.

**Table 12: Training time.(^4^44Except for the SfM processing time and training times of 3DGS(Random) and Ours, we copied the values from 3DGS []. Note that 3DGS utilized a single A6000 GPU and we utilized a single RTX 3090 GPU for evaluation.)**
|  | Mip-NeRF360 | Tanks&Temples | Deep Blending |
| --- | --- | --- | --- |
| Plenoxels | 25m49s | 25m5s | 27m49s |
| INGP-Base | 5m37s | 5m26s | 6m31s |
| INGP-Big | 7m30s | 6m59s | 8m |
| 3DGS(^5^55The processing time of SfM is denoted with a $\dagger$.) | 41m33s $+$ (19m9s)^† | 26m54s $+$ (19m43s)^† | 36m2s $+$ (6m23s)^† |
| 3DGS(Random) | 28m22s | 18m31s | 29m33s |
| Ours | 31m34s | 15m13s | 27m51s |

### D.9 Limitations

Although RAIN-GS robustly guides 3D Gaussians to model the scene even from randomly initialized point cloud, our novel strategy is not without limitations. As our strategy prioritizes learning the coarse approximation, it sometimes fails to detect the need for further densification to capture high-frequency details. This happens especially for regions where the L1 rendering loss cannot distinguish between a coarse approximation and a high-frequency true distribution. An example of this limitation is shown in Figure 14, where the grass region in the front of the rendered image lacks high-frequency details compared to the ground truth image. However, we find that this limitation also appears when training 3DGS with SfM initialized point cloud, which is also shown to start optimization from a coarse approximation of the true distribution in our main paper. We posit that this limitation mainly comes from the lack of supervision, using the L1 rendering loss as the main signal. Although adding additional supervision from depth or error maps could open up new possibilities to resolve these limitations, we leave this direction as future work.

Figure: Figure 14: Visualization of failure cases. This figure shows the rendering results of the “flowers” scene of the Mip-NeRF360 dataset from Ours and 3DGS. When compared to the GT image shown in (c), the front of the images of (a) and (b) lack high-frequency details.
Refer to caption: x14.png

## Appendix E Additional qualitative results

We show additional qualitative results in Figure 15.

Figure: Figure 15: Additional qualitative results.
Refer to caption: x15.png

Figure: Figure 16: Additional qualitative results with depth.
Refer to caption: x16.png

## Appendix F Additional results of 3DGS in sparse view settings

Due to the impressive performance shown by NeRF, various follow-ups have been proposed, specifically to succeed in learning NeRF with a limited number of images [, , , , ]. However, due to the requirement of accurate pose and initial point cloud of 3DGS, the task of successfully guiding 3D Gaussians with a limited number of images becomes a much more complicated task. In this section, we evaluate our strategy in training 3DGS with few-shot images, where SfM struggles to converge. In Table 13 and Figure 17, we show quantitative and qualitative results comparing 3DGS(Random) and Ours in the sparse view setting in the Mip-NeRF360 dataset []. To show the effectiveness of our strategy when SfM struggles to converge, after excluding every 8th image for evaluation, we train the model on randomly sampled 10$\%$ of the remaining images. Ours outperforms 3DGS trained with random initialization both qualitatively and quantitatively, demonstrating the potential of our work to be extended to the task of training 3DGS with few-shot images. In Figure 17, it is shown that the rendered results of Ours much more robustly models the real geometry of the scene.

**Table 13: Quantitative comparison on Mip-NeRF360 dataset in sparse-view setting. We compare our method with the random initialization method described in the original 3DGS []. We report PSNR, SSIM, LPIPS.**
| Method | Outdoor Scene |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| bicycle | flowers | garden | stump | treehill |  |  |  |  |  |  |  |  |  |  |  |
| PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ | PSNR$\uparrow$ | SSIM $\uparrow$ | LPIPS$\downarrow$ | PSNR$\uparrow$ | SSIM $\uparrow$ | LPIPS$\downarrow$ |  |
| 3DGS(Random) | 11.787 | 0.170 | 0.637 | 10.948 | 0.118 | 0.648 | 16.752 | 0.377 | 0.444 | 15.239 | 0.194 | 0.602 | 12.675 | 0.206 | 0.599 |
| Ours | 12.436 | 0.157 | 0.636 | 11.546 | 0.120 | 0.649 | 17.955 | 0.463 | 0.425 | 14.675 | 0.142 | 0.614 | 12.820 | 0.191 | 0.597 |
| Method | Indoor Scene | Average |  |  |  |  |  |  |  |  |  |  |  |  |  |
| room | counter | kitchen | bonsai |  |  |  |  |  |  |  |  |  |  |  |  |
| PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ | PSNR$\uparrow$ | SSIM $\uparrow$ | LPIPS$\downarrow$ | PSNR$\uparrow$ | SSIM $\uparrow$ | LPIPS$\downarrow$ |  |
| 3DGS(Random) | 18.864 | 0.702 | 0.418 | 15.311 | 0.575 | 0.475 | 18.181 | 0.629 | 0.384 | 13.246 | 0.508 | 0.538 | 14.778 | 0.386 | 0.527 |
| Ours | 19.804 | 0.722 | 0.424 | 18.201 | 0.637 | 0.451 | 21.474 | 0.718 | 0.359 | 20.664 | 0.721 | 0.408 | 16.620 | 0.430 | 0.507 |

Figure: Figure 17: Qualitative results in sparse view settings.
Refer to caption: x17.png

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 5
presentation: 2
questions: I have a few comments .. 

Typo: 
page 2, were propose -> we propose

* Pan et al. 2024 accelerates the SfM pipeline but does not use learnable modules.
* In the abstract and in Section 1, the authors argue that solely using photometric loss is the main cause for poor optimization performance with noisy initialization. However, throughout the paper, the standard photometric loss continues to be used in the optimization objective without any modifications. Thus, it would help to put less emphasis on the particular point when motivating the problem.
soundness: 2
strengths: The authors propose certain modifications to the initialization steps prior to optimizing 3D Gaussian splatting representations. It leads to small but consistent improvements in the reconstruction quality on several datasets especially when the optimizer is started with noisy or random initialization of the 3D point coordinates. In general, the arguments and insights they present sounds reasonable in theory and seems to be helping in practice as shown in the empirical results.
summary: This submission investigates the initialization strategy commonly used in existing 3D Gaussian Splatting (3DGS) optimization schemes. The authors present some technical insights towards the question of why the 3DGS optimizers tend to perform poorly when the centers of the 3D Gaussians are initialized from random values. They coin a new term in the paper and call the phenomena “low Gaussian transportability”. The authors also argue that the sole use of photometric error leads to the aforementioned limitation.

To address the observed limitations in optimizing 3DGS representations from noisier initialization, the authors have proposed some remedies, such as a) starting with fewer Gaussians with significantly larger covariances, applying low pass filtering progressively, introducing heuristics for reducing the gaussian size as the optimization progresses and the number of Gaussians in the scene increase, new splitting criteria.

Empirically these ideas lead to incremental improvements on the popular datasets (Mip-NeRF 360, RealEstate 10K, Tanks & Temples, Deep Blending) in terms of standard metrics (pSNR, SSIM, LPIPS, etc.). An ablation study was also reported to measure the benefit of each of the proposed modifications in isolation.
weaknesses: The authors try to improve 3DGS performance when the Gaussians are initialized from random 3D points. While their results show some promise towards that end, they however assume that the camera poses are accurate. Typically, a full SfM pipeline must be executed to obtain accurate camera poses – must run bundle adjustment till completion. When running a full BA, accurate points are typically obtained as a by-product and no extra computation is needed. So, in other words, I don’t see a compelling situation where the 3DGS optimizer must be initialized from a random 3D point cloud when accurate camera poses are already available. In case of Re10K, the 3D points could be initialized using triangulated feature matches where both matching and triangulation could leverage the accurate poses.

However, it becomes much more interesting and relevant when camera poses are obtained from a different source and not necessarily extremely accurate.

While the paper is readable, the main paper lacks sufficient technical detail. The authors introduce the idea of limited Gaussian portability but in Section 4.2 there is an empirical study on the Mip-NeRF360 dataset without any deeper insights. Notably other authors have already investigated the issue of local minima in the optimization landscape. So I am not convinced that this paper extends the insight beyond what is already known in the literature. So I would argue against the first claimed contribution on page 2 “For the first time, we conduct an in-depth analysis and identify … “.

The ideas of improving the initialization seem reasonable but none of these modifications guarantees escaping local minima. For example, the authors state in Section 5.1 “To prevent the Gaussians from falling into local minima, we propose a simple yet effective modification to the original random initialization”. Even though some benefits of the proposed scheme are shown in the empirical evaluation, in the absence of any formal guarantees, the claims should be toned down a bit. Similarly, the specific split heuristic describes here (called ABE-Split) is a variation of existing splitting strategies and there is not much detail why this specific criteria should be used.

---

confidence: 5
contribution: 3
initial_rating: 5
presentation: 3
questions: The authors proposed the progressive low-pass filter to ensure 3D Gaussians receive more gradients during training. I wonder if is this equivalent to training 3D Gaussians from low-resolution images to high-resolution images in a coarse-to-fine manner. I would appreciate the authors to provide a fair comparison between the progressive low-pass filter and coarse-to-fine training strategy/
soundness: 3
strengths: (1) The proposed method is simple to add to the existing 3DGS method

(2) The authors provide exhaustive experiments to verify the effectiveness of the proposed method

(3) The proposed method improves the original 3DGS clearly on the tanks-and-temples and the MipNeRF360 dataset.
summary: This paper aims to alleviate the initialization/densification problem in the original 3DGS, where regions are under-reconstructed in SfM. To solve the issue, the paper proposes initializing 3D Gaussians with larger variances so that Gaussians can cover more regions of the image and learn more about the global structure of the scene. Moreover, the paper proposes progressive low-pass filtering techniques to encourage Gaussians to receive more gradients during training. The paper also uses an adaptive bound-expanding split algorithm to overcome the limitation that the variance of Gaussians becomes small after multiple splitting steps. The experiments on the MipNeRF360 dataset and tanks-and-temples dataset show the effectiveness of the proposed method.
weaknesses: (1) The ideas are intuitive and the contributions are not significant.

(2) While I appreciate their improvements to the original 3DGS methods, there are more variants of 3DGS, and I wonder would the same strategies proposed in this paper can improve the other 3DGS methods(such as scaffold-gs, etc).

(3) The method can improve 3DGS on the tanks-and-temples and MipNeRF360 dataset, I would like to see the effectiveness of this method on texture-less areas. And I would like to raise my score if the authors do so.

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 3
questions: 1. In Line 431, the authors mention that the reconstruction results are based on poses from sensors. Does the the pose errors impact the final reconstruction performance, and can the proposed method mitigate this issue?

2. As shown in Figures 1 and 3, why does the improvement mainly occur in edge regions or distant areas?

3. Some further discussions: With the recent advancements in monocular reconstruction algorithms, such as Dust3R and DepthPro, which allow for effective 3D reconstruction without relying on Colmap or even poses, would the proposed method conflict with or complement these approaches? Could case studies be conducted to explore this?

I might raise my rating if my concerns are addressed.
soundness: 3
strengths: 1. This paper proposes to reconstruct Gaussian scenes using randomly initialized point clouds. In some cases, the proposed method surpasses Colmap+3DGS, which is impressive.
2. The proposed method encourages Gaussian scenes to first learn global structural information before focusing on detailed information.
3. The analysis in this paper is well-organized and clearly written.
summary: This paper aims to reconstruct Gaussian scenes using randomly initialized point clouds. The designs guide the Gaussian scenes to first establish a global structure, thereby avoiding local optima. Although relatively simple, this method even achieves superior reconstruction results in certain scenes compared to Colmap+3DGS. However, more experiments and analysis are needed to support the motivation.
weaknesses: 1. The design in this paper mainly comes from experimental results. While the approach is simple and effective, it lacks theoretical novalities, and many hyperparameters require tuning. It would be beneficial for the authors to provide more detailed ablation experiments on these hyperparameters.

2. The paper assumes known multi-view image poses without point cloud reconstruction, but further analysis and explanation are needed regarding in which case this setting makes scene:
- If the image poses are obtained from sensors, how does the method handle pose errors? How much error can it handle? Should the poses being optimized to improve robustness? 
- If the multi-view images are generated from generative models, such as text-to-3D models, please validate the effectiveness of the proposed method in these cases.

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 2
questions: 1. The author claims that the limited Gaussian transportability is mainly due to suboptimal performance caused by exclusive reliance on image photometric loss. However, is there any possibility that the suboptimal results are from batch size 1 optimization? Can this problem be well resolved by simply using a parallel training paradigm like GrendelGS?
2. In Table 5, the author claims that when combined with the low-pass filter and ABE-Split algorithm, SLV initialization shows the best performance. However, there seems to be a lack of an experiment combining the proposed low-pass filter, N=100K, and ABE-Split to provide a comparison.
soundness: 2
strengths: 1. Significance. Relaxing the initialization constraint of 3DGS is a critical and meaningful problem. The RAIN-GS realizes an impressive performance without the need of SfM points, proving it is an efficient solution.
summary: This paper aims to alleviate the critical requirements for accurate initialization for 3DGS. The proposed RAIN-GS attributes the problem to limited Gaussian transportability. Based on this observation, it introduces a novel initialization strategy and adaptive density control strategy, as well as a progressive Gaussian low-pass filtering to resolve the problem.
weaknesses: 1. As a submission to ICLR, it's not reasonable to only include 3DGS and Mip-Splatting for comparison. 2DGS sacrifices representation ability for more accurate geometry. Thus it's meaningless and unfair to compare with it purely on rendering quality. A more reasonable choice should be ScaffoldGS [CVPR24] or other improvements on ECCV24 or NIPS24, especially when they are randomly initialized.
2. The authors claim limited Gaussian transportation as a reason for the problem, and provide the analysis on movement of Gaussians as evidence. However, it remains unclear where do RAINGS's improvement comes from. The author should compare their method in both Table 2 and Figure 2 to validate the motivation and effectiveness of their method.

Meta-review: This paper describes an initialization strategy for improving the quality of 3D Gaussian splatting-based novel-view synthesis. The strength of this paper is the improvement over the existing 3DGS initialization strategy. On the other hand, the downside is the technical contribution of the work. The proposed method is rather heuristic based on the empirical observation. More importantly, as one of the reviewer commented, if the accurate camera poses are obtained by structure-from-motion, we already have sparse tie points which can be used for initialization of Gaussians. While the authors responded on this respect saying that the feature matching and triangulation is the bottleneck, but for knowing camera poses, these steps are actually already included. The overall reviewer ratings were mixed around the borderline, and importantly none of the reviewers were excited about the paper. The AC weighed on the lack of the motivation for the designed initialization scheme and reached this decision.

Decision: Reject