CAT-3DGS: A Context-Adaptive Triplane Approach to Rate-Distortion-Optimized 3DGS Compression

Authors: Yu-Ting Zhan, Cheng-Yuan Ho, Hebi Yang, Yi-Hsin Chen, Jui Chiu Chiang, Yu-Lun Liu, Wen-Hsiao Peng
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: 3D Gaussian Splatting, Rate-Distortion Optimization, Context Models, Grid Representations

Abstract:
3D Gaussian Splatting (3DGS) has recently emerged as a promising 3D representation. Much research has been focused on reducing its storage requirements and memory footprint. However, the needs to compress and transmit the 3DGS representation to the remote side are overlooked. This new application calls for rate-distortion-optimized 3DGS compression. How to quantize and entropy encode sparse Gaussian primitives in the 3D space remains largely unexplored. Few early attempts resort to the hyperprior framework from learned image compression. But, they fail to utilize fully the inter and intra correlation inherent in Gaussian primitives. Built on ScaffoldGS, this work, termed CAT-3DGS, introduces a context-adaptive triplane approach to their rate-distortion-optimized coding. It features multi-scale triplanes, oriented according to the principal axes of Gaussian primitives in the 3D space, to capture their inter correlation (i.e. spatial correlation) for spatial autoregressive coding in the projected 2D planes. With these triplanes serving as the hyperprior, we further perform channel-wise autoregressive coding to leverage the intra correlation within each individual Gaussian primitive. Our CAT-3DGS incorporates a view frequency-aware masking mechanism. It actively skips from coding those Gaussian primitives that potentially have little impact on the rendering quality. When trained end-to-end to strike a good rate-distortion trade-off, our CAT-3DGS achieves the state-of-the-art compression performance on the commonly used real-world datasets.

Decision: accept (poster)
Reviewer scores: 6, 6, 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1730607605144

thread_type:
official_review

<thread_object>
creator: reviewer_vhe1

<content>
summary:
In this paper, the authors present a context-adaptive and rate-distortion-optimized 3DGS coding approach named CAT-3DGS. The approach introduces several modules for efficient attribute 3DGS compression, including a triplane-based hyperprior, spatial autoregressive models (SARM), channel-wise autoregressive models (CARM) and view frequency-aware masking. These modules are jointly trained in a rate-distortion-optimized framework. Experimental results demonstrate that CAT-3DGS achieves rate reductions of 78× and 26× compared to 3DGS and Scaffold-GS, respectively, while surpassing existing 3DGS compression schemes in rate-distortion performance.

strengths:
1.The coding architecture is well-structured and theoretically solid, with four distinct modules specifically designed to address various redundancies.

2.The proposed method achieves exceptional rate-distortion performance, surpassing other 3DGS compression techniques.

weaknesses:
1.The effectiveness of view frequency-aware masking is somewhat ambiguous. In Sec. 5.3, the authors report a 16% BD-rate reduction due to view frequency-aware masking. However, it’s unclear whether the anchor configuration (labeled as “w/o VFM” in Fig. 10) excludes masking altogether, or retains the masking operation but removes the weights p_{n,k} only.

2.Spatial redundancies within attributes are not fully exploited. The proposed SARM only addresses inter-redundancies within triplanes (hyperpriors) rather than anchors. Given the fact that spatial redundancies exist between anchors [1], this aspect is not leveraged in the current framework.

[1] Yufei Wang, Zhihao Li, Lanqing Guo, Wenhan Yang, Alex C Kot, and Bihan Wen. Contextgs: Compact 3d gaussian splatting with anchor level context model. arXiv preprint arXiv:2405.20721, 2024b.

questions:
Please see weakness
</content>
</thread_object>

</thread>
<thread>
creation date:
1730618311548

thread_type:
official_review

<thread_object>
creator: reviewer_geta

<content>
summary:
The paper introduces CAT-3DGS, a context-adaptive compression method for 3D Gaussian splatting (3DGS) that optimizes rate-distortion performance. CAT-3DGS uses multi-scale triplanes to embed spatial correlations of Gaussian primitives and employs autoregressive models for encoding. Additionally, it uses a channel-wise autoregressive model to enhance compression efficiency within each primitive and introduces a view frequency-aware masking mechanism to skip encoding primitives that minimally impact image quality. Experimental results demonstrate the effectiveness of the proposed method.

strengths:
1. PCA-guided triplanes as the hyperprior
2. Channel-wise Autoregressive Model: Enhances compression by leveraging internal correlations within each voxel.
3. View Frequency-aware Masking Mechanism: Skips encoding for primitives with minimal impact on rendering quality, reducing redundancy and enhancing overall performance.

weaknesses:
1. The ablation study of PCA guidance need to be included since there are already serval works that utilizes triplanes to compress the 3DGS/Nerf.
2. The efficiency of the model. The model seems a bit complicated. The coding time greatly increased due to the proposed context model. Besides, how about the training time?
3. Missing quantitative comparison with ContextGS, which is an important baseline method which also uses context model. For many benchmarks, the improvements appear somewhat marginal.
4. Figure 2 may not be entirely accurate; to my knowledge, CompGS does not utilize a context model.

questions:
Please see the weakness part.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730622065420

thread_type:
official_review

<thread_object>
creator: reviewer_tujm

<content>
summary:
The paper presents CAT-3DGS, a context-adaptive triplane approach for compressing 3D Gaussian Splatting (3DGS) data with a focus on optimizing the rate-distortion trade-off. CAT-3DGS introduces multi-scale triplanes aligned with the principal axes of Gaussian primitives in 3D space, leveraging both spatial and channel-wise autoregressive models to capture inter and intra correlations for efficient compression. This approach also includes a view frequency-aware masking mechanism to skip encoding less impactful Gaussian primitives. Experiments demonstrate CAT-3DGS achieves superior rate-distortion performance compared to prior 3DGS compression models like HAC, particularly on commonly used real-world datasets.

strengths:
1. Innovative Triplane-Based Hyperprior: 

By projecting 3D Gaussian primitives onto triplanes, the paper introduces an efficient hyperprior structure that captures spatial correlation, which improves entropy coding performance.

2. Enhanced Coding Efficiency: 

The combination of spatial and channel-wise autoregressive models effectively utilizes both inter and intra correlations, which is particularly valuable in reducing the bit rate while maintaining high rendering quality.

3. Practical Masking Mechanism: 

The view frequency-aware masking mechanism selectively encodes only those primitives crucial for rendering, further optimizing storage and bandwidth usage.

4. Comprehensive Experimental Evaluation: 

The method is tested across several datasets and compared against notable baselines, showcasing significant rate savings and quality retention.

weaknesses:
1. (Minor) Autoregressive Model Use: 

While the channel-wise and pixel-wise autoregressive models are novel in this context, they are a commonly applied strategy in image compression tasks. However, this paper does make a unique contribution by applying it to 3DGS.

2. Lack of Comparative Analysis with ContextGS: 

Although ContextGS is cited as a related work, there is no direct performance comparison with it. Given that ContextGS provides detailed results and its code is available, a comparative evaluation would enhance clarity on CAT-3DGS's improvements over previous models, especially regarding contextual coding efficiency.​

questions:
I do not have other questions.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730726485930

thread_type:
official_review

<thread_object>
creator: reviewer_stvt

<content>
summary:
This paper proposed a context-adaptive triplane based hyperprior entropy model to capture the inter correlation among Gaussian primitives in the 3D space  (i.e. spatial correlation) for spatial autoregressive coding in the projected 2D planes. The channel-wise autoregressive coding is performed to leverage the intra correlation within each individual Gaussian primitive. Besides, the view frequency-aware masking mechanism is proposed to actively skip from coding those Gaussian primitives with little impact on the rendering quality. Experimental results show that the proposed CAT-3DGS achieves the state-of-the-art compression performance on the commonly used real-world datasets.

strengths:
- ***Novelty of Framework:*** The key idea of decomposing the dense 3D hash-graid in HAC with triplane hyperprior entropy model is novel and reasonable.
- ***Technological Innovation:*** This paper proposed the SARM and FARM modules to exploit the inter-correlation and intra-correlation, respectively. Speciflly, the view frequency-aware masking mechanism is designed to to skip less critical Gaussian primitives from coding.
- This paper is well organized and easy to follow.

weaknesses:
- Ablation experiment for CARM is not sufficient, the proposed method is only evaluated on one scenario, *e.g.*, bicycle scene.
- The paper is based on the work of HAC. However, the rate parameter setting used by HAC in the comparison experiment is not the same as that used by the proposed method. I doubt the fairness of this setting.

questions:
- The FARM proposed by the author is interesting. According to the manuscript, the anchor feature is a latent representation of the anchor point, which does not have a clear mathematical correlation like the spherical correlation coefficient of 3DGS. What is the motivation of the proposed channel-wise autoregressive processing of anchor features?
- It is noted that the rate parameters setting used by the authors on different datasets are not consistent. Why?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730744346125

thread_type:
official_review

<thread_object>
creator: reviewer_egdp

<content>
summary:
The paper introduces a new method called CAT-3DGS for compressing 3D Gaussian Splatting (3DGS) representations. This method aims to optimize the rate-distortion trade-off by using a context-adaptive triplane approach. It captures spatial correlations through multi-scale triplanes and leverages intra correlations within Gaussian primitives for efficient coding. Additionally, it incorporates a view frequency-aware masking mechanism to skip less impactful primitives, achieving state-of-the-art compression performance on real-world datasets.

strengths:
+ The paper introduces a new compression method for 3DGS that utilizes a triplane-based hyperprior. This method leverages multi-scale triplanes oriented along the principal axes of Gaussian primitives.
+ The paper presents a novel pruning approach in 3DGS using a view frequency-aware masking mechanism. This mechanism assesses the significance of Gaussian primitives based on their impact on rendering quality, allowing less critical ones to be skipped during coding.
+ The paper presents comprehensive ablation studies of various modules in the proposed compression method.

weaknesses:
- Entropy coding is a variable-length coding technique, typically decoded sequentially. However, 3DGC, used in rendering, is a highly parallelized method. The paper does not clearly explain how the proposed compression method can manage this parallelization without impacting rendering speed.
- The paper does not address decoding complexity. While CHARM enhances compression performance, it could significantly impact decoding speed, which in turn may affect rendering speed.
- The paper does not clarify whether real entropy coding was used during rendering or if the bit-rate was calculated based on theoretical entropy coding.

questions:
* Are there scenarios where CAT-3DGS underperforms? Which modules in CAT-3DGS contribute to the drop in performance?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732116604180

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Response to All Reviewers

comment:
We thank the reviewers for their insightful and constructive comments. In particular, we appreciate all reviewers for recognizing the effectiveness of our work. We are also pleased that Reviewers (egdp, stVt, and tuJM) found our work **novel**, and that Reviewer (VHE1) considered our **coding architecture to be well-structured and theoretically solid**. Additionally, we are grateful to Reviewers (egdp, tuJM) for appreciating the **comprehensive experimental results**.

We have carefully addressed all the concerns and comments and hope the reviewers find our responses satisfactory. We kindly request that the reviewers consider increasing the ratings for this early attempt at efficient coding of 3DGS representations, which we believe would invite more contributions from the community to advance this largely unexplored yet important area.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This is a paper discussing scene compression using a Gaussian splatting model. The paper introduces several novelties (e.g., triplane-based hyperprior, view frequency-aware masking mechanism), and evaluates against the most important baselines. No major weaknesses are noted. The work represents a significant contribution, albeit somewhat incremental.