An Exploration with Entropy Constrained 3D Gaussians for 2D Video Compression

Authors: Xiang Liu, Bin Chen, Zimo Liu, Yaowei Wang, Shu-Tao Xia
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: video compression, 3d gaussian splatting, entropy coding

Abstract:
3D Gaussian Splatting (3DGS) has witnessed its rapid development in novel view synthesis, which attains high quality reconstruction and real-time rendering. At the same time, there is still a gap before implicit neural representation (INR)  can become a practical compressor due to the lack of stream decoding and real-time frame reconstruction on consumer-grade hardware. It remains a question whether the fast rendering and partial parameter decoding characteristics of 3DGS are applicable to video compression. To address these challenges, we propose a Toast-like Sliding Window (TSW) orthographic projection for converting any 3D Gaussian model into a video representation model. This method efficiently represents video by leveraging temporal redundancy through a sliding window approach. Additionally, the converted model is inherently stream-decodable and offers a higher rendering frame rate compared to INR methods. Building on TSW, we introduce an end-to-end trainable video compression method, GSVC, which employs deformable Gaussian representation and optical flow guidance to capture dynamic content in videos. Experimental results demonstrate that our method effectively transforms a 3D Gaussian model into a practical video compressor.  GSVC further achieves better rate-distortion performance than NeRV on the UVG dataset, while achieving higher frame reconstruction speed (+30%~40% fps) and stream decoding. Code is available at [Github](https://github.com/actcwlf/GSVC)

Decision: accept (poster)
Reviewer scores: 5, 8, 6, 6

Reviews and discussion:
<thread>
creation date:
1730093548897

thread_type:
official_review

<thread_object>
creator: reviewer_nojs

<content>
summary:
This paper proposes a new framework for video compression from the perspective of 3D Gaussian Splatting (3DGS). The key idea is to construct an interesting spatial-temporal modeling way called toast-like sliding window to cope with the 3D scene modeling in a short time period. This design enables the introduction of several 3DGS-based compression methods into video compression areas. With further careful designs like optical-flow supervision and time-aware Gaussians, the method achieves better compression quality then previous implicit-representation-based approaches like NeRV.

strengths:
1. The idea of toast-like sliding windows is interesting. It provides an insightful design by treating the time step as an extra dimension to construct a 3D structure.
2. The overall writing is clear and well-motivated. The logical flow is very clear and easy to understand.
3. The overall idea of the exploration of video compression from the 3DGS perspective is novel.

weaknesses:
1.  Why do toast-like sliding windows need to render a short future period of content? It would be better to provide more illustrations about the detailed rendering process.
2. Ablation study of FiLM design. The analysis of the change from MLP to FiLM-based network structure is missing.
3. It would be better to include a more comprehensive comparison of video compression approaches like H.264/265 and HNeRV for in-depth analysis.
4.  There are several typos in the writing like line 243, us-> use.

questions:
1. It would be better to provide the rasterization process when applying the Orthographic Projection in the context of Gaussian Splatting. Including these changes in the appendix will help the readers understand the paper more clearly.
2. How the anchor points are selected in the toast-like sliding windows?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730485270370

thread_type:
official_review

<thread_object>
creator: reviewer_jdph

<content>
summary:
This paper explores 3DGS for video compression. It Introduces TSW projection to adapt 3D Gaussian Splatting for 2D video representation. This paper also develops an end-to-end trainable video compression method based on deformable Gaussian and optical guidance to better model motion content in videos.

strengths:
1. Orthographic projection and sliding windows for videos are reasonable ideas for adapting the original perspective projection in 3DGS. 

2. Experiments on the UVG dataset show that the developed GSVC codec achieves RD performance compared to NeRV and offers efficient rendering with inherent stream decoding capability. Besides, this paper presents a relatively complete work for the codec workflow. 

3. The paper mentions that the code will be open-source. I appreciate the promise of open source, especially in video compression.

weaknesses:
1. the proposed method still needs to train per video. Which i think is a limitation for real-world usage. 

2. As video codecs are a well-developed method, this paper does not include comparisons to traditional codecs. I know neural representation for compression is a novel and hot feild. But it’s more valuable and practice to highlight the specific strengths of each methods, especially in terms of the computation costs for encoding.

questions:
1. Figure 1's ground truth image seems to be with low quality. Based on my knowdege, 3DGS's reconstruct is prune to have high numerical results but lack of details. Have the author consider this issue for compresssion task ?
2. It is hard to see if there is a temporal inconsistency without video examples. Can you provide video comparisons ?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730581494254

thread_type:
official_review

<thread_object>
creator: reviewer_jglw

<content>
summary:
The authors propose a novel approach to 2D video compression using 3D Gaussian Splatting (3DGS). They represent 2D video sequences as a volumetric distribution in 3D space, using X-Y-T coordinates. To reduce temporal redundancy, they introduce a Toast-like Sliding Window (TSW) projection method that considers Gaussian points within a temporal window centered on the rendering plane. The framework incorporates optical flow information and a FiLM-based deformation field to better capture temporal dynamics. An entropy coding scheme is implemented to optimize model compression and enhance streaming efficiency.

The paper presents three main contributions:
* Introduction of a novel paradigm that represents 2D video using 3D Gaussian models, supported by the TSW projection technique
* Development of GSVC, a comprehensive compression pipeline that combines HAC, deformable fields, and opacity flow regularization
* Experimental validation showing GSVC achieves comparable quality to NeRV while providing 30% faster rendering and flexible bitrate-availability tradeoffs

strengths:
* Pioneering application of 3DGS to video compression
* Minimal assumptions about video content, relying primarily on temporal continuity, which enables broad applicability
* High adaptability with various 3DGS models, opening promising research directions
* Demonstrated practical utility through flexible bitrate-availability tradeoffs, particularly valuable for streaming applications

weaknesses:
I do really like the idea of the paper yet the evaluation section could be further polished.
* Limited comparison baseline, with NeRV as the only reference. The evaluation would benefit from comparisons with other neural video compression methods
* Evaluation only focus on low bitrate/bpp scenarios, while contemporary neural compression methods (e.g., VCT, DCVC) typically demonstrate performance at higher bitrates with PSNR values exceeding 34 on the UVG dataset.

questions:
- In Figure 5, there remains a gap in frame availability even after complete bit decoding. What factors contribute to this limitation?
- Another concern is about the depth sorting when it comes to rendering. there could be a scenario when two gaussians (lets say Ga and Gb), in the frame t-1, fall into Vf, while in the next frame t, they fall into Vb, due to the time plane moving forward. Yet in this case the relative depths of Ga and Gb to the image plane are reversed 
  - for example, in t-1, Ga, Gb in Vf, and Ga is close to the plane, and Gb is further to the plane, then in t, Ga, Gb in Vb, it must be that Ga is further to the plane and Gb looks closer to the plane
  - Such kind of reversing might lead to totally different coloring of the same pixel after alpha blending. has such inconsistency in consecutive frames ever been observed in your experiments?
  - Also how the two images projected from Vb and Vf get mixed into a single image? simply averaging them?
  - It would be interesting if you could visualize the xyt space as a normal 3D space and in the meantime, visualize the rendering of one of the frames from nearby Gaussians. That will be cool.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730620400797

thread_type:
official_review

<thread_object>
creator: reviewer_czlv

<content>
summary:
This paper introduces a 2D video compression method using 3D Gaussian Splatting (3DGS), aiming for faster frame reconstruction and stream decoding. However, concerns remain about its limited novelty, as many techniques (e.g., time embeddings, Gaussian attribute adjustments) are adapted from NeRF and 4DGS. The framework largely assembles prior methods (e.g., HAC, GPCC, and optical flow loss from 4DGS) without substantial innovation, raising questions about the applicability and originality of the approach in RGB video compression.

strengths:
1. Improved Inference Speed:

The compression framework outperforms INR-based inference in rendering speeds, a notable improvement in practical applications where real-time decoding is essential.

2. Broad Performance Evaluation: 

The authors provide comprehensive experiments and comparisons with alternative compression methods, demonstrating that GSVC achieves favorable rate-distortion performance across different quality metrics.

weaknesses:
1. Lack of Novelty:

The technical novelty of the work is limited. Key techniques, such as time-position embeddings and Gaussian attribute variation over time, have been previously introduced in NeRF and 4DGS [1]. The approach of offsetting Gaussian positions based on features at different timestamps is conceptually similar to the offset extraction in dynamic 4DGS methods. The overall compression framework aligns closely with HAC [2], while the compression of x,y,z coordinates with GPCC was introduced in other prior work. Similarly, optical flow loss is a component already implemented in 4DGS [3]. And the GPU-based ANS ae/ad is already implemented in CompressAI[4].


2. No Comparison with HEVC/VVC: 

The paper lacks a direct performance comparison with widely-used standards like HEVC or VVC, making it difficult to gauge its practical effectiveness against traditional video compression methods.


3. Questionable Value for INR/3DGS in RGB Video Compression: 

The application of INR/3DGS techniques to RGB video compression seems inherently misaligned with the goals of video compression. Compression frameworks for RGB video typically rely on extensive data to learn a generalized data distribution, leveraging priors that do not exist within a single video or image.

[1] 4D Gaussian Splatting for Real-Time Dynamic Scene Rendering

[2] HAC: Hash-grid assisted context for 3D Gaussian splatting compression

[3] MotionGS: Exploring Explicit Motion Guidance for Deformable 3D Gaussian Splatting

[4] https://github.com/InterDigitalInc/CompressAI

questions:
Considering the typical aim of RGB image/video compression to leverage extensive priors learned from large datasets, how does the proposed framework add value when applied to compress a single video? Is there any particular benefit in this context?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732125526372

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Common Concern

comment:
We sincerely appreciate all the reviewers for dedicating their valuable time and effort to review our paper and provide insightful comments and suggestions. Encouragingly, reviewers praise the GSVC's novelty and reasonableness ( R#JGLW, R#jDPh, R#nojS ), completeness (R#JGLW, R#jDPh), high adaptability (R#JGLW ) and appreciate the writing ( R#nojS) and broad evaluation (R#czLV) of our work. 

We would like to express our sincerest gratitude once more for your invaluable time and efforts in evaluating our work! A revision of our paper that incorporates your precious suggestions will soon be uploaded to further improve the paper's quality.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper leverages 3D Gaussian Splatting (3DGS) for video compression, introducing the Toast-like Sliding Window (TSW) for efficient 2D video representation. The proposed GSVC framework uses deformable Gaussians and optical flow to improve temporal modeling and reduce redundancy. GSVC outperforms NeRV in rate-distortion and achieves 30–40% faster frame reconstruction, offering a practical and efficient solution for video compression.

Strengths:
Novelty: First to apply 3D Gaussian Splatting (3DGS) to video compression with innovative temporal modeling via Toast-like Sliding Window (TSW).
Efficiency: Achieves faster rendering and stream decoding, outperforming INR-based methods.
Robust Performance: Shows strong rate-distortion performance and adaptability through comprehensive evaluations.
Clear Writing and Open Source: Well-presented work with a commitment to releasing open-source code.

Weaknesses:
Limited Novelty: Adapts existing techniques from NeRF, 4DGS, and HAC, offering minimal originality.
Lacking Comparisons: Missing benchmarks against HEVC/VVC and advanced neural methods like VCT and DCVC.
Per-Video Training: Requires per-video training, reducing practicality for real-world use.
Evaluation Gaps: Limited focus on high-bitrate scenarios and missing ablations on FiLM design and rendering details.

The paper received divergent reviews.
We weakly accept the paper because it is the first attempt of using 3DGS on video compression task. But we would like the authors to include limitation section in the paper to detail the weaknesses of the method and also release the code as promised.