Relaxing Accurate Initialization Constraint for 3D Gaussian Splatting

Authors: Jaewoo Jung, Jisang Han, Honggyu An, Jiwon Kang, Seonghoon Park, Seungryong Kim
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Novel View Synthesis, 3D Gaussian Splatting

Abstract:
In this work, we investigate the limitations of the 3D Gaussian Splatting (3DGS) optimization scheme, revealing why it undergoes significant performance drops when initialized with noisy or random point clouds. Through in-depth analysis, we identify a key limitation of the 3DGS optimization: limited Gaussian transportability. Since Gaussians are optimized solely based on image photometric loss, the optimization tends to overfit the parameters of the projected Gaussians to improve reconstruction at their current positions, rather than relocating them to more optimal locations. This leads to producing under-reconstructed regions when starting with noisy or random initialization, failing to transport Gaussians to correct locations. Based on our findings, we propose RAIN-GS (Relaxing Accurate Initialization Constraint for 3D Gaussian Splatting), a set of simple yet effective modifications, including initializing sparse Gaussians with large variances, progressive Gaussian low-pass filtering, and an Adaptive Bound-Expanding split algorithm. These modifications enable Gaussians to effectively redistribute across the scene, capturing both coarse structure and fine details. By addressing the inherent limitations of 3DGS, RAIN-GS allows effective training even with random point clouds, significantly enhancing reconstruction quality.

Decision: reject
Reviewer scores: 5, 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1729511892250

thread_type:
official_review

<thread_object>
creator: reviewer_gqex

<content>
summary:
This paper aims to alleviate the critical requirements for accurate initialization for 3DGS. The proposed RAIN-GS attributes the problem to limited Gaussian transportability. Based on this observation, it introduces a novel initialization strategy and adaptive density control strategy, as well as a progressive Gaussian low-pass filtering to resolve the problem.

strengths:
1. Significance. Relaxing the initialization constraint of 3DGS is a critical and meaningful problem. The RAIN-GS realizes an impressive performance without the need of SfM points, proving it is an efficient solution.

weaknesses:
1. As a submission to ICLR, it's not reasonable to only include 3DGS and Mip-Splatting for comparison. 2DGS sacrifices representation ability for more accurate geometry. Thus it's meaningless and unfair to compare with it purely on rendering quality. A more reasonable choice should be ScaffoldGS [CVPR24] or other improvements on ECCV24 or NIPS24, especially when they are randomly initialized.
2. The authors claim limited Gaussian transportation as a reason for the problem, and provide the analysis on movement of Gaussians as evidence. However, it remains unclear where do RAINGS's improvement comes from. The author should compare their method in both Table 2 and Figure 2 to validate the motivation and effectiveness of their method.

questions:
1. The author claims that the limited Gaussian transportability is mainly due to suboptimal performance caused by exclusive reliance on image photometric loss. However, is there any possibility that the suboptimal results are from batch size 1 optimization? Can this problem be well resolved by simply using a parallel training paradigm like GrendelGS?
2. In Table 5, the author claims that when combined with the low-pass filter and ABE-Split algorithm, SLV initialization shows the best performance. However, there seems to be a lack of an experiment combining the proposed low-pass filter, N=100K, and ABE-Split to provide a comparison.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730449376876

thread_type:
official_review

<thread_object>
creator: reviewer_caqa

<content>
summary:
This paper aims to reconstruct Gaussian scenes using randomly initialized point clouds. The designs guide the Gaussian scenes to first establish a global structure, thereby avoiding local optima. Although relatively simple, this method even achieves superior reconstruction results in certain scenes compared to Colmap+3DGS. However, more experiments and analysis are needed to support the motivation.

strengths:
1. This paper proposes to reconstruct Gaussian scenes using randomly initialized point clouds. In some cases, the proposed method surpasses Colmap+3DGS, which is impressive.
2. The proposed method encourages Gaussian scenes to first learn global structural information before focusing on detailed information.
3. The analysis in this paper is well-organized and clearly written.

weaknesses:
1. The design in this paper mainly comes from experimental results. While the approach is simple and effective, it lacks theoretical novalities, and many hyperparameters require tuning. It would be beneficial for the authors to provide more detailed ablation experiments on these hyperparameters.

2. The paper assumes known multi-view image poses without point cloud reconstruction, but further analysis and explanation are needed regarding in which case this setting makes scene:
- If the image poses are obtained from sensors, how does the method handle pose errors? How much error can it handle? Should the poses being optimized to improve robustness? 
- If the multi-view images are generated from generative models, such as text-to-3D models, please validate the effectiveness of the proposed method in these cases.

questions:
1. In Line 431, the authors mention that the reconstruction results are based on poses from sensors. Does the the pose errors impact the final reconstruction performance, and can the proposed method mitigate this issue?

2. As shown in Figures 1 and 3, why does the improvement mainly occur in edge regions or distant areas?

3. Some further discussions: With the recent advancements in monocular reconstruction algorithms, such as Dust3R and DepthPro, which allow for effective 3D reconstruction without relying on Colmap or even poses, would the proposed method conflict with or complement these approaches? Could case studies be conducted to explore this?

I might raise my rating if my concerns are addressed.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730534189853

thread_type:
official_review

<thread_object>
creator: reviewer_ploa

<content>
summary:
This paper aims to alleviate the initialization/densification problem in the original 3DGS, where regions are under-reconstructed in SfM. To solve the issue, the paper proposes initializing 3D Gaussians with larger variances so that Gaussians can cover more regions of the image and learn more about the global structure of the scene. Moreover, the paper proposes progressive low-pass filtering techniques to encourage Gaussians to receive more gradients during training. The paper also uses an adaptive bound-expanding split algorithm to overcome the limitation that the variance of Gaussians becomes small after multiple splitting steps. The experiments on the MipNeRF360 dataset and tanks-and-temples dataset show the effectiveness of the proposed method.

strengths:
(1) The proposed method is simple to add to the existing 3DGS method

(2) The authors provide exhaustive experiments to verify the effectiveness of the proposed method

(3) The proposed method improves the original 3DGS clearly on the tanks-and-temples and the MipNeRF360 dataset.

weaknesses:
(1) The ideas are intuitive and the contributions are not significant.

(2) While I appreciate their improvements to the original 3DGS methods, there are more variants of 3DGS, and I wonder would the same strategies proposed in this paper can improve the other 3DGS methods(such as scaffold-gs, etc).

(3) The method can improve 3DGS on the tanks-and-temples and MipNeRF360 dataset, I would like to see the effectiveness of this method on texture-less areas. And I would like to raise my score if the authors do so.

questions:
The authors proposed the progressive low-pass filter to ensure 3D Gaussians receive more gradients during training. I wonder if is this equivalent to training 3D Gaussians from low-resolution images to high-resolution images in a coarse-to-fine manner. I would appreciate the authors to provide a fair comparison between the progressive low-pass filter and coarse-to-fine training strategy/
</content>
</thread_object>

</thread>
<thread>
creation date:
1730686722740

thread_type:
official_review

<thread_object>
creator: reviewer_y9vo

<content>
summary:
This submission investigates the initialization strategy commonly used in existing 3D Gaussian Splatting (3DGS) optimization schemes. The authors present some technical insights towards the question of why the 3DGS optimizers tend to perform poorly when the centers of the 3D Gaussians are initialized from random values. They coin a new term in the paper and call the phenomena “low Gaussian transportability”. The authors also argue that the sole use of photometric error leads to the aforementioned limitation.

To address the observed limitations in optimizing 3DGS representations from noisier initialization, the authors have proposed some remedies, such as a) starting with fewer Gaussians with significantly larger covariances, applying low pass filtering progressively, introducing heuristics for reducing the gaussian size as the optimization progresses and the number of Gaussians in the scene increase, new splitting criteria.

Empirically these ideas lead to incremental improvements on the popular datasets (Mip-NeRF 360, RealEstate 10K, Tanks & Temples, Deep Blending) in terms of standard metrics (pSNR, SSIM, LPIPS, etc.). An ablation study was also reported to measure the benefit of each of the proposed modifications in isolation.

strengths:
The authors propose certain modifications to the initialization steps prior to optimizing 3D Gaussian splatting representations. It leads to small but consistent improvements in the reconstruction quality on several datasets especially when the optimizer is started with noisy or random initialization of the 3D point coordinates. In general, the arguments and insights they present sounds reasonable in theory and seems to be helping in practice as shown in the empirical results.

weaknesses:
The authors try to improve 3DGS performance when the Gaussians are initialized from random 3D points. While their results show some promise towards that end, they however assume that the camera poses are accurate. Typically, a full SfM pipeline must be executed to obtain accurate camera poses – must run bundle adjustment till completion. When running a full BA, accurate points are typically obtained as a by-product and no extra computation is needed. So, in other words, I don’t see a compelling situation where the 3DGS optimizer must be initialized from a random 3D point cloud when accurate camera poses are already available. In case of Re10K, the 3D points could be initialized using triangulated feature matches where both matching and triangulation could leverage the accurate poses.

However, it becomes much more interesting and relevant when camera poses are obtained from a different source and not necessarily extremely accurate.

While the paper is readable, the main paper lacks sufficient technical detail. The authors introduce the idea of limited Gaussian portability but in Section 4.2 there is an empirical study on the Mip-NeRF360 dataset without any deeper insights. Notably other authors have already investigated the issue of local minima in the optimization landscape. So I am not convinced that this paper extends the insight beyond what is already known in the literature. So I would argue against the first claimed contribution on page 2 “For the first time, we conduct an in-depth analysis and identify … “.

The ideas of improving the initialization seem reasonable but none of these modifications guarantees escaping local minima. For example, the authors state in Section 5.1 “To prevent the Gaussians from falling into local minima, we propose a simple yet effective modification to the original random initialization”. Even though some benefits of the proposed scheme are shown in the empirical evaluation, in the absence of any formal guarantees, the claims should be toned down a bit. Similarly, the specific split heuristic describes here (called ABE-Split) is a variation of existing splitting strategies and there is not much detail why this specific criteria should be used.

questions:
I have a few comments .. 

Typo: 
page 2, were propose -> we propose

* Pan et al. 2024 accelerates the SfM pipeline but does not use learnable modules.
* In the abstract and in Section 1, the authors argue that solely using photometric loss is the main cause for poor optimization performance with noisy initialization. However, throughout the paper, the standard photometric loss continues to be used in the optimization objective without any modifications. Thus, it would help to put less emphasis on the particular point when motivating the problem.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732423625656

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response

comment:
We are significantly grateful for all the reviewers thoroughly reviewing our manuscript and providing valuable feedbacks which can improve our work when addressed. We are also grateful that the reviewers recognize the strengths of our work, including the effectiveness of our strategy of largely improving the results of 3DGS with random point clouds (y9Vo, pLoa, Caqa, gqEX), the importance of the task (gqEX), analysis being intuitive (y9Vo, Caqa), impressive results with extensive experiments (pLoa,Caqa,gqEx) and the overall writing being well-written (Caqa).

Encouraged by the valuable feedbacks from the reviewers, we have conducted extra experiments which can verify the effectiveness and application of our approach. The revised pdf includes:

- Additional qualitative and quantitative results from generated images with text-to-3D models (Figure 6, Table 5)
- More detailed ablations of our components (Appendix F.1)
- RAIN-GS with other 3DGS extensions (Appendix F.2)
- RAIN-GS with noisy camera poses (Appendix F.3)
- Replacing progressive low-pass filtering with low-to-high resolution image training (Appendix F.4)
- Analysis of Movement of Each Gaussian in RAIN-GS (Appendix F.5)
- Additional analysis on Grendel-GS (Appendix F.6)
- Fixed typos

We have expressed the modified or newly added parts in red.

Furthermore, in this rebuttal, we have addressed key points raised by the reviewers, including:

- Where can RAIN-GS be effectively used? (y9Vo, Caqa)
- Can RAIN-GS improve other extensions of 3DGS? (pLoa)
- Additional comparisons and analysis (gqEx)
</content>
</thread_object>

</thread>
<thread>
creation date:
1733231604011

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of our work

comment:
Dear Reviewers,

We sincerely appreciate your valuable time and effort in reviewing our manuscript. As the author-reviewer discussion period has concluded, we would like to provide a final summary of our work.

Our work, **RAIN-GS**, addresses the critical question: "Why is 3D Gaussian Splatting (3DGS) largely dependent on the accuracy of the initial point cloud, even when provided with accurate camera poses?" Through our analysis, we reveal that this dependency stems from the limited transportability of Gaussians within the 3DGS framework. We further verify our analysis by proposing a simple yet effective modification that enables the Gaussians to transport further, largely improving the performance of 3DGS even with random point clouds.

We have demonstrated that RAIN-GS can be effectively applied in the following scenarios:
1. **Text-to-3D Generated Images** (Figure 5, Table 6)
2. **Images with Camera Poses Obtained from Sensors** (Figure 4, Table 4)
3. **Images with Noisy Camera Poses** (Appendix F.3)
4. **Joint Application with Other 3DGS Extensions** (Appendix F.2)

In scenarios 1 to 3, traditional Structure-from-Motion (SfM) algorithms often fail to converge, failing to provide accurate initial point clouds to the user. These scenarios verify the effectiveness and necessity of our work which can widen the application of 3DGS to real-world scenarios. Furthermore, the joint application of RAIN-GS with Scaffold-GS shown that applying our work can be beneficial even when point clouds from SfM is accessible, improving the performance of Scaffold-GS.

Thanks to all the valuable feedbacks from the reviewers, we have included several experiments in the Appendix that can strengthen and verify our contributions. These experiments and discussions will be incorporated into the main paper in the final version. We also thank the reviewers from acknowledging our responses and a positive evaluation of our work after discussion. We believe that our extensive experiments validate the effectiveness of RAIN-GS and its potential to significantly reduce the burden on users to obtain accurate initial point clouds, thereby broadening the practical applications of 3DGS.

Thank you once again for your thoughtful reviews and constructive feedback,

Authors of submission 6403.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper describes an initialization strategy for improving the quality of 3D Gaussian splatting-based novel-view synthesis. The strength of this paper is the improvement over the existing 3DGS initialization strategy. On the other hand, the downside is the technical contribution of the work. The proposed method is rather heuristic based on the empirical observation. More importantly, as one of the reviewer commented, if the accurate camera poses are obtained by structure-from-motion, we already have sparse tie points which can be used for initialization of Gaussians. While the authors responded on this respect saying that the feature matching and triangulation is the bottleneck, but for knowing camera poses, these steps are actually already included. The overall reviewer ratings were mixed around the borderline, and importantly none of the reviewers were excited about the paper. The AC weighed on the lack of the motivation for the designed initialization scheme and reached this decision.