Lightweight Predictive 3D Gaussian Splats

Authors: Junli Cao, Vidit Goel, Chaoyang Wang, Anil Kag, Ju Hu, Sergei Korolev, Chenfanfu Jiang, Sergey Tulyakov, Jian Ren
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Gaussian splatting

Abstract:
Recent approaches representing 3D objects and scenes using Gaussian splats show increased rendering speed across a variety of platforms and devices. While rendering such representations is indeed extremely efficient, storing and transmitting them is often prohibitively expensive. To represent large-scale scenes, one often needs to store millions of 3D Gaussian, which can occupy up to gigabytes of storage. This creates a significant practical barrier, preventing widespread adoption on resource-constrained devices.
In this work, we propose a new representation that dramatically reduces the hard drive footprint while featuring similar or improved quality when compared to the standard 3D Gaussian splats. This representation leverages the inherent feature sharing among splats in the close proximity using a hierarchical tree structure, with which only the parent splats need to be stored. We present a method for constructing tree structures from naturally unstructured point clouds. Additionally, we propose the adaptive tree manipulation to prune the redundant trees in the space, while spawn new ones from the significant  children splats during the optimization process. On the benchmark datasets, we achieve 20x storage reduction in hard-drive footprint with improved fidelity compared to the vanilla 3DGS and 2-5x reduction compared to the exiting compact solutions.  More importantly, we demonstrate the practical application of our method in real-world rendering on mobile devices and AR glasses.

Decision: accept (poster)
Reviewer scores: 5, 8, 8

Reviews and discussion:
<thread>
creation date:
1730513982654

thread_type:
official_review

<thread_object>
creator: reviewer_9hav

<content>
summary:
This paper proposes a storage-effective 3D gaussian splatting representation. It uses hierarchical tree structure use similarity between nearby 3D gaussians.  The tree structures are manipulated adaptively in the optimization processes. The method not only outperform SOTA methods, but it is also shown to run effectively in mobile devices.

strengths:
Storage-effective representation for 3D gaussians.  

The proposed method shown to outperform existing SOTA methods. 

The proposed method shown to run on mobile devices. 

Ablation study to show necessities of the proposed components.

weaknesses:
There is only one figure (i.e., fig.2) for explaining the proposed method. Thus, it may not be easy to follow the process of the method.

questions:
The format of the sub-section titles is not consistent. Some of them are ended with ":", some with "." and some with " ".
</content>
</thread_object>

</thread>
<thread>
creation date:
1730550369744

thread_type:
official_review

<thread_object>
creator: reviewer_zk78

<content>
summary:
This paper presents a new representation for 3D Gaussian splats that drastically reduces disk space requirements while providing similar or improved quality compared to previous methods. The method detects inherent common features between splats in close proximity using an attention mechanism (ATM) and exploits these using a hierarchical tree structure in which only the parent splats need to be stored. As a very extensive evaluation on current benchmark data sets shows, the new representation achieves an average reduction in disk footprint of 20x compared to the original 3DGS, with improved PSNR and comparable SSIM and LPIPS, and a 2-5x reduction in storage compared to new works on more efficient representation. In addition, the method is presented to real rendering applications on mobile devices and AR glasses.
In contrast to the competing method CompactGS, it uses a combination of neural fields and self-attention layers to predict not only view-dependent colours but also geometric properties. Secondly, in contrast to competing approaches that store the position of each splat in the point cloud explicitly like ScaffoldGS, the method stores only a small subset of splats, referred to as parents, while the remaining points are predicted on-the-fly during rendering, significantly reducing memory requirements. In contrast to recently presented anchor-based methods, the hierarchical tree representation in combination with the proposed Adaptive Tree Manipulation takes into account the importance of both parent and child splats, which enables a subtree expansion strategy.

strengths:
The idea of deriving the positions of child splats and associated attributes – position, color, scale, etc. – from the parent using a small neural network and only storing parent splats together with the weights of the neural network is new and should definitely be published.
The paper gives a very nice introduction to the problem and a very good overview of the existing approaches.
The method itself is explained in detail and is easy to follow. Comprehensive ablation studies show the influence of the different components of the algorithm on its performance. The extensive evaluation impressively demonstrates the strength of the method compared to the state of the art. It is interesting to note that, in contrast to the original 3DGS, the quality in some of the benchmark comparisons is better despite the considerable compression of the representation.

weaknesses:
Unfortunately, there is no code to go with the paper, so that the procedure can be tested independently.

questions:
In the  text to Figure 2 "patent" should be replaced by "parent".

Maybe you comment already in section 4.1. that in your code you have chosen $K$ to be 2?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730622229572

thread_type:
official_review

<thread_object>
creator: reviewer_zq2p

<content>
summary:
This paper proposes a lightweight 3D Gaussian framework that models inherent spatial relationships within a hierarchical tree structure. To optimize the tree structure, the authors suggest adaptive tree manipulation (ATM), an adaptive growth and pruning strategy for parent and children nodes. Consequently, it demonstrates efficient storage size and superior rendering quality compared to compact 3D Gaussian representations, e.g., LightGS, CompactGS, and EAGLES.

strengths:
- The proposed tree structure enables parent nodes to represent children nodes using an on-the-fly decoding pipeline. This results in the small disk usage of this representation with high rendering quality.

- The optimization schemes, ATM for growth and warm-up for initial training, lead to achieving stable and effective optimization of the proposed representation.

weaknesses:
- The main weakness of this paper is the limited technical novelty. The proposed tree structure mainly comes from the anchor-based representation, Scaffold-GS [1]. Also, the adaptive manipulation of children nodes is proposed in HAC [2], with more efficient learnable masks. Moreover, the usage of a hash grid for efficient 3DGS representation is also proposed in HAC and CompactGS [3]. 

- Also, it demonstrates a slower rendering speed compared to 3DGS, as we can see in L458. It indicates that this approach requires more 3D Gaussians splats (both parent splats and children splats) for the rendering phase and only consumes less disk usage in the storage. Therefore, the claim for **"lightweight"** representation seems overclaimed.

- Furthermore, there is limited comparison to existing compact representations. Recently, many papers have been addressing the inefficiency issue of 3DGS. However, there are too less methods in the comparison. Please refer to the question section for the related approaches.

---
**Reference**
1. Lu et al., Scaffold-GS: Structured 3D Gaussians for View-Adaptive Rendering, CVPR 2024
2. Chen et al., HAC: Hash-grid Assisted Context for 3D Gaussian Splatting Compression, ECCV 2024
3. Lee et al., Compact 3D Gaussian Representation for Radiance Field, CVPR 2024

questions:
- The results of CompactGS in Table 1 seem to be not applying post-processing, such as quantization of the hash grid. As this paper reports the storage size for the quantized hash grid as described in L443, it is fair to report the post-processed results of CompactGS.

- Moreover, it seems that the tree structure and hash grid design require more training time compared to 3DGS. Please provide the training time to optimize this representation.

- Also, the detailed number of 3D Gaussians for 3DGS should be added to prove that this method requires less number of Gaussians as described in the quantitative results section.

- Furthermore, it is recommended to add more comparisons for the compact 3DGS representations as below. According to the reviewer guideline, the authors do not have to compare papers which are recently published in ECCV'24. But, as the idea this paper is significantly related to HAC [1], it could be helpful for comparing with HAC in terms of both performance and methodology.

  - Compressed3D [2] (CVPR 2024)
  - Reduced3DGS [3] (I3D 2024)
  - HAC [1] (ECCV 2024)
  - SOG [4] (ECCV 2024)
  - Compact3D [5] (ECCV 2024)

---
**Reference**
1. Chen et al., HAC: Hash-grid Assisted Context for 3D Gaussian Splatting Compression, ECCV 2024
2. Niedermayr et al., Compressed 3D Gaussian Splatting for Accelerated Novel View Synthesis, CVPR 2024
3. Papantonakis et al., Reducing the Memory Footprint of 3D Gaussian Splatting, I3D 2024
4. Morgenstern et al., Compact 3D Scene Representation via Self-Organizing Gaussian Grids, ECCV 2024
5. Navaneet et al., Compact3D: Smaller and Faster Gaussian Splatting with Vector Quantization, ECCV 2024
</content>
</thread_object>

</thread>
<thread>
creation date:
1732340506478

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
We sincerely appreciate your time and effort in reviewing our submission and providing valuable and positive feedback to enhance our work. We are delighted that the reviewers have acknowledged the strength of our paper, including:

**novel hierarchical tree structure and representation** leading to small-disk usage with high rendering quality (Reviewer ZQ2P), storage-effective for 3D gaussians (Reviewer 9HaV), and is new and should definitely be published (Reviewer zk78);

**extensive experiments** outperforming existing SOTA methods (Reviewer zk78, ZQ2P) with efficient storage size and superior rendering quality (Reviewer 9HaV), and better than 3DGS on some benchmarks despite the considerable compression of the representation (Reviewer zk78);

**comprehensive ablation** showing the influence and necessities of different components from the proposed algorithm (Reviewer zk78,  9HaV);

**deployment of real rendering** demonstrating our model can run effectively on mobile devices and AR glasses (Reviewer zk78,  9HaV); 

**nice introduction, good overview, and detailed explained method** of the paper (Reviewer zk78). 

We also **revise the manuscript** with the following updates suggested from reviewers, including:
1. adding a visual explanation to illustrate our method in *Appendix A.1*;
2. adding a discussion of related work (*e.g.*, HAC [1]) in  *Appendix A.2*;
3. adding the analysis of training convergence in *Appendix A.3.5*;
4. fixing the typos in caption of Fig. 2;
5. fixing the typos in subsection titles.

The added content is highlighted in light blue color.

In the following, we provide detailed responses to each reviewer.

---

**Reference**
1. Chen et al., HAC: Hash-grid Assisted Context for 3D Gaussian Splatting Compression, ECCV 2024
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper introduces a novel hierarchical 3DGS representation that effectively reduces storage requirements while maintaining similar or improved rendering quality compared to the standard 3DGS. By leveraging on inherent feature sharing among proximate splats, the method represents nearby splats using a reduced number of parent splats, leading to a more efficient and compact representation. Experimental results demonstrate that the proposed approach significantly decreases the storage footprint relative to existing or baseline 3DGS methods, without compromising rendering quality. Therefore, I recommend its acceptance.