An Exploration with Entropy Constrained 3D Gaussians for 2D Video Compression
Authors: Xiang Liu
Bin Chen
Zimo Liu
Yaowei Wang
Shu-Tao Xia
Venue: ICLR 2025 Poster (2025)
Keywords: Video Compression
3D Gaussian Splatting
Entropy Coding
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: 3D Gaussian Splatting (3DGS) has witnessed its rapid development in novel view synthesis, which attains high quality reconstruction and real-time rendering. At the same time, there is still a gap before implicit neural representation (INR)  can become a practical compressor due to the lack of stream decoding and real-time frame reconstruction on consumer-grade hardware. It remains a question whether the fast rendering and partial parameter decoding characteristics of 3DGS are applicable to video compression. To address these challenges, we propose a Toast-like Sliding Window (TSW) orthographic projection for converting any 3D Gaussian model into a video representation model. This method efficiently represents video by leveraging temporal redundancy through a sliding window approach. Additionally, the converted model is inherently stream-decodable and offers a higher rendering frame rate compared to INR methods. Building on TSW, we introduce an end-to-end trainable video compression method, GSVC, which employs deformable Gaussian representation and optical flow guidance to capture dynamic content in videos. Experimental results demonstrate that our method effectively transforms a 3D Gaussian model into a practical video compressor.  GSVC further achieves better rate-distortion performance than NeRV on the UVG dataset, while achieving higher frame reconstruction speed (+30%~40% fps) and stream decoding. Code is available at [Github](https://github.com/actcwlf/GSVC)

Published as a conference paper at ICLR 2025
AN EXPLORATION WITH ENTROPY CONSTRAINED 3D GAUSSIANS FOR 2D VIDEO COMPRESSION
Xiang Liu
Bin Chen?.3+ Zimo Liu Yaowei Wang Shu-Tao Xia
'Tsinghua Shenzhen International Graduate School, Tsinghua University
Harbin Institute of Technology, Shenzhen
Peng Cheng Laboratory
1 liuxiang22 20aanngg2mmmmmmmmmmm tsinghua. edu. en, chenbin2021@hit.eee acc. liuzmêpel ac. cn, wangywêpel, ac. en, xiastêsz. ts tsinghua, edu. cn
ABSTRACT
3D Gaussian Splatting (3DGS) has witnessed its rapid development in novel view synthesis, which attains high quality reconstruction and real-time rendering. At the same time, there is still gap before implicit neural representation (INR) become a practical compressor due to the lack of stream decoding and real-time frame reconstruction on consumer-gras grade hardware. It remains question whether the fast rendering and partial parameter decoding characteristics of 3DGS are ap- plicable to video compression. To address these challenges, we propose a Toast- like Sliding Window (TSW) orthographic projection for converting any 3D Gaus- sian model into a video representation model. This method efficiently represents video by leveraging temporal redundancy through sliding window approach. Additionally, the converted model is inherently stream-decodable and offers a higher rendering frame rate compared to INR methods. Building TSW, we introduce an end-to-end trainable video compression method, GSVC, which em- ploys deformable Gaussian representations and optical flow guidance to capture dynamic content in videos. Experimental results demonstrate that our method effectively transforms 3D Gaussian model into a practical video compressor. GSVC further achieves better rate-distortion performance than NeRV on the UVG dataset, while achieving higher frame reconstruction speed (+30% fps) and stream decoding. Code is available at Github
INTRODUCTION
Video compression is classical issue in the domain of signal processing. With the growing amount of video data on the Internet, numerous video compression algorithms have emerged. Classic video compressors employ hand-crafted rules to reduce redundant information in the data (Le Gall 1991 Wiegand et al. 2003,, (Sulliaan et al. 2012).
saarning-basedd approaches, on the other hand, learn the characteristics of video from large quantity of video data through data-driven manner and achieve highly efficient compression (Lu et al., 2019; Shi et al.] 2022 Li et al., 2024).. Besides, methods based on implicit neural representation (INR) have also started be applied in video compression, which represent signals by directly mapping coordinates or timestamps to the corresponding RGB values, offering new insights for the field (Chen et al.) 2021 Lee al. 2023).
Similar to the success of deep learning in other fields (Huang et al., 2023 Qin et al.) 2024 Xia
al., 2024; Gao et al., (2024), current learning-based neural video compression approaches have achieved considerable progress in rate-distortion (RD) performance. However, these methods do have certain issues. Firstly, many methods have a high decoding computational complexity, limiting their practicality in real-world scenarios. INRebased method have to some extent alleviated this problem (Chen et al,, (02T), but for those with superior RD performance, their rendering frame rates cannot reach the speed for practical usage (Kwan et al. 2023). Secondly, fully decoding the representation network is necessary before rendering any frame for INR-based method, making it challenging to achieve stream decoding. Given that neural representation is more suitable for
"Corresponding author

---

Published as a conference paper at ICLR 2025
Original
Original
GSVC 0. 0.028bpp
Original
0.0288PP
0.028opp
NeRv D. 0300pp
NeRv 0,030bPP
Figure 1:
Comparison between proposed GSVC and NeRV (Chen et al. 2021). Our method achieves effective video compression while surpassing NeRV in some aspects such as background texture.
encoding longer videos (Chen et al. 2021 this characteristic significantly limits its applications, especially in popular streaming media.
Recently, 3D Gaussian Splatting (3DGS) (Kerbl et al. 2023 has gained increasing attention for its impressive capability of reconstructing 3D scene from multiple photos, which outperforms NeRF (Mildenhall al. 2020) in training and rendering speed. Although 3D Gaussian representation is method specifically ly designed for 3D scenes, shares similarities with neural representation in that they are both a new type of signal representation and have the potential be a general-purpose representation method applied to other domains (Dupont et al. 2021 2022 Zhang et al. (024). These fields will also benefit from the exclusive features 3DGS, such explicit representation, fast rendering, and partial parameter rendering.
In this paper, focusing on the weakness of INR video compressors and advantages of 3DGS men- tioned before, we explore the feasibility of 3DGS in video compression. We propose novel regional orthographic projection called Toast-like sliding window (TSW) orthographic projection. TSW or- thographic projection utilizes the redundancy between adjacent frames without requiring any explicit or implicit 3D structure in the signal. Using the method, any model sharing same rendering process with 3DGS can be casily extended to 2D video representation model. Based on TSW, propose an end-to-end trainable Gaussian Splatting Video Compressor (GSVC) by extending the framework of HAC (Chen et al. 2024). Furthermore, introduce time-aware Gaussians generation, which offers increased parameter efficiency. This design, cooperated with proposed optical flow (Horn & Schunck 1981 I) guided deformable Guassians (Yang al.) 2024c), significantly improve the capa- bility of GSVC to efficiently model dynamic objects in video. Fig. [ demonstrates a qualitative results of GSVC.
The proposed TSW has been tested with vanilla 3DGS(Kerbl et al. 2023), Scaffold-GS (Lu et al. 2024) and HAC (Chen al. 2024) validate its universality, demonstrating that our method ef- fectively transforms 3D Gaussian model into practical video compressor. Thorough experiments over UVG dataset (Mercat al., 2020) were performed, in which proposed GSVC demonstrates better RD performance than NeRV (Chen et al. 2021) and superior rendering efficiency. Besides, our method achieves stream decoding by natural.
The primary contributions of this work are summarized below:
We propose novel TSW orthographic projection, which bridging the gap between 3D Gasussian Splatting and 2D video representation.
We build an end-to-end trainable video codec GSVC based on TSW by extending an exist- ing 3DGS compressor.
We refine GSVC by time-aware Gaussian generation and optical flow guided deformable
3D Gaussian improve parameter efficiency and capture dynamic contents in videos.
2

---

Published as a conference paper at ICLR 2025
2
RELATED WORK
2.1 LEARNED VIDEO COMPRESSION
Traditional video codecs utilize variety of hand-crafted rules to eliminate spatial and temporal dundancies such as different intra-prediction and inter- interprrdiccton patterns in order to achieve high- efficiency encoding (Le Gall, 1991; Wiegand et al. 2003 (uullivan al., 2012). In video com- pression methods using deep learning, one approach is to improve existing traditional methods by replacing certain components, such as entropy coding (Song al. (017) and post-processing (Lu et al. 2018, Song et al. 2018) with neural network. (Tian et al. (2024) propose a machine-friendly video compressor with traditional- neurall mixed coding framework.
Another approach is to build an end-to-end deep learning vidco codec which jointly optimizes all the components in video compression pipeline (Lu et al. 2019 Hu al., a (02I). Such methods typically take advantage of representation ability of neural networks to achieve functions such as feature transformation and motion compensation (Lin et al. 2020). Liu et al.] (2020) utilize the conditional entropy between frames. Li et al. (2021) propose using feature domain context as condition.
Besides typical end-to-end models, INR-based compression methods, which general require lower computational resources, provide a different view of compression task. These representation meth- ods encode the signal itself implicitly in the weights of a neural network, The neural representation has attracted attention for its performance in synthesizing new views in 3D scenes (Mildenhall et al., 2020), and it has gradually been applied to the representation and compression of images and videos (Dupont et al. 2021 2022 Ladune et al. 2023). Chen et al. (2021) first propose use timestamps as the input of the representation network, outputting corresponding video frames. Combining with network pruning and quantization, this representation method can more effectively capture intra- frame and inter-frame relationships than NeRF (Mildenhall et al. 2020) in video compression. Sub- sequent work built on this foundation, with more detailed design of the network structure, achieving
better rate-distortion performance (Chen et al. 2021 2023 Lee et al. 2023 Kwan et al. 2023).
In comparison to end-to-end video compression models, INR-based methods can achieve lower computational overhead while not requiring pre-trained models to be deployed at the encoding and decoding ends. However, neural representation-baat based video compression methods still have diffi culty meeting the real-time characteristics in rendering rate, especially for some models with better RD performance. In addition, these methods need to fully decompress the entire model for frame reconstruction, making it impractical to apply to scenarios where streaming transmission is required.
2.2 3D GAUSSIAN SPLATTING
3D Gaussian Splatting (3DGS) (Kerbl et al., (023) is a method for 3D scene reconstruction that has emerged in recent years, which has attracted the attention of many researchers with its fast re- construction speed and real-time rendering characteristics. Different from implicit representation of NeRF, 3DGS uses series of Gaussian points in space to represent the 3D information of the scene. This representation method also provides a new perspective for many tasks. For example, 4D dynamic videos can be represented through moving and deforming Gaussian points in a defor- mation field (Lin et al., 20244] Yang et al.] (002c)). (Shi et al. (2024) embed semantics in 3DGS for open-vocabulary scene understanding YYnng al.) 2024a). There have also been many works on improving 3D Gaussian itself, such as Yu et al. (2024) improving the reconstruction quality through Mip filter, and Liu et al. (2024) enabling 3D Gaussian represent super-large scenes through level- of-detail strategy. In terms of practical applications. some work has explored reducing the storage overhead of 3D Gaussian through pruning (Yang al., 2024b) and entropy coding (Chen et al.) 2024).
Although 3D Gaussian specifically designed for 3D scenes, it has the potential be a general sig- nal representation method that can be applied to more fields. Zhang et al. (2024) transform the 3D splatting rendering process into a 2D splatting rendering process, building an image representation and compression pipeline based on Gaussian representation. Shin al. (2024) represent video by foreground Gaussian and background Gaussian, and further achieve video editing. Despite its strong expressive capabilities and fast rendering speed, the exploration of Gaussian in non-3D representa- tion fields is insufficient. How to design better representations to apply 3D Gaussian to different signal like video remains question.
3

---

Published as a conference paper at ICLR 2025
3 METHOD
The implicit assumption of 3DGS for effective scene representation and novel view synthesis is that the dataset describes a 3D scene that can be observed from multiple viewpoints. The assumption may be violated when extend 3DGS to general-purpose representation model (Zhang et al. 2024). To address the issue, we transform the original camera model of 3DGS to TSW orthographic pro- jection (abbreviated TSW in following sections). TSW is based the assumption of similarity between neighboring frames, which is inherent characteristics of videos. With TSW, we can easily transform any 31D Gaussian model video representation model. In section 3.1 we briefly review the background of 3DGS and entropy constrained 3DGS, HAC(Chen et al. 2024). Section 33.2 in- troduces TSW in details. To enhance the clarity of our method, we illustrate the overall framework of GSVC in Fig. 3 Section 3.3 and section 3.3 explain then time-aware Gaussian generation and optical flow guidance respectively. Section summarizes the pipeline of GSVC and introduce the cost-compression process.
3.1 PRELIMINARIES
3D Gaussian Splatting is designed to synthesis high-quality novel view of 3D scene efficiently from photos, which is known as inverse rendering. 3DGS models a 3D scene as a large amount of 3ID Gaussian points represented by
1 1(2 11) - (ar 11)
G(æ) exp(-
(1)
where μl and E are the mean position and covariance matrix of 3D Gaussian particle. In rendering process, Gaussian particle are projected to pixel space to obtain 2D Gaussian parameterized by E
'' = JWEW
where W is view transformation and J the Jacobian of the affine approximation of the perspec- tive transformation. Each pixel color C' is calculated following a-blending
C
Σ Ca. II (1-
- az).
where a is derived from E'. To maintain the positive semi-definiteness covariance matrix is further decomposed as rotation R and scaling S, i.e. Σ = RSS R
In Scaffold-GS (Lu et al. 2024), all previous parameters are generated from anchor with location æ and attributes A = i E RDD E RR o E RR33), where each component represents anchor feature, scaling and offsets, respectively. HAC (Chen et al.) 2024) further exploits mutual relation between A and f Interp(æ",, H), to achieve efficient compression, where H. is binary hash grid.
3.2 BRIDGING 3D GAUSSIAN SPLATTING AND 2D VIDEO
Few of previous work investigates the method of representing video by 3DGS, Shin et al. (2024) decompose video foreground Gaussians and background Gaussians. This method requires explicit foreground and background content in the video, but this assumption is not always valid in many videos, such as screen recordings and hand-drawn animations, which are non-natural scene videos. Our core idea is to view 2D videos as 3D manifold and represent the video using 3D Gaussians directly in the manifold space. This means that every Gaussian point not only affects different pixels at the same time but also affects the same pixel at different times.
Specifically, we view 2D axis xy and time axis t in video as 3D axis xyz in 3D space, and represent the 3D space by Gaussian particles. In render stage, we introduce Toast-like sliding window (TSW) orthographic projection, as shown in Fig. 2c When rendering frame with time stamp f1, we select all Gaussians with coordinate 2 falling into it h, t + hl, which is similar near clip and far clip in normal orthographic projection. We note space [t] h, tj] and [t tj + h] as V, and Vú respectively. After culling stage, plane i as camera plane and perform 3D Gaussian
4

---

Published as a conference paper at ICLR 2025
culled at
Vo
Vs
h
camera
cemeraa
f1
culled at 22
(a) Perspective Projection
(b) Orthographic Projection
(c) TSW Orthographic Projection
Figure 2: Comparison of different camera models.
Splatting rendering process in Vj and Vg, Since affine approximation is unnecessary for orthographic projection, the Jacobian matrix J is identity matrix. Eq. 2 can be simplified
WEWT.
(4)
To utilizes the bi-directional similarity along time axis, we obtain final reconstructed frame from the average of rendering results of V, and V,, We show more details and visualization in Appendix A.3
When using a 3I) manifold to represent a 2D) video, the redundancy in the t/z axis is often greater than that of the xy axis, which is one of the most important differences from the usual 3D ordinates. The sliding window mechanism improves the representation efficiency by reusing and effectively reduces the redundant information tion method only assumes the similarity in the time axis of the video, and does not require explicit 3D structures, which more suitable for general video representation scenarios. Moreover, when reconstructing frames, only Gaussian points within the sliding window participate in the rendering process, naturally achieving the goal of stream decoding and random decoding.
the time axis. At the same time, this representa-
3.3 TIME-AWARE GAUSSIAN GENERATION
In Scaffold-GS, the generation process of Gaussian points is performed by taking the viewpoint coordinates and anchor point features as inputs, achieving perspective-aware Gaussian Splatting rendering. However, previous studies have shown that directly using coordinates as inputs prevents the network from learning high-frequency features, deteriorates reconstruction quality. Therefore, we use time stamp of frame and anchor location relative to the camera plane dæa as input after positional encoding (Mildenhall et al. 2020 Tancik et al. 2020
pe(p)
cos (oskkrr)))
(5)
We set L 16 for both & and 82".
In terms of network architecture, we add feature-wise linear modulation (FiLM) layers (Perez et al., 2018 to MLP. The FiLM layers apply affine transformations features based on condition inputs. Compared with the original MLP, incorporating FiLM layers increase the network capacity and improve parameter efficiency, which important to compression task. In our method, the transfor- mation parameters of the FiLM layers are computed from input positional encoding
a. 9e (pe(t), pe(5=")), )), Br, = (p((())) pe(o="))
where i denotes the index of anchor, denotes the different FiLM layer of different attributes such as color The final attribute value are computed from anchor features and FiLM transformation
MLP
f. FiLMM Br,c) Ye.ff Bi, c= c=MM.P'.(..)
(8)
(9)
Here color example, and the calculation for other attributes similar.
5

---

Published as a conference paper at ICLR 2025
In addition to the attribute network, also added deformation network YYnng al., 2024c) to enhance the capability of modeling dynamic content. The deformation network takes the positional encoding and anchor feature input and calculates the dynamic offset delta do of the spawned Gaussians at time t.
d0 MLP pe()))).
(10)
Given the anchor x,, final position of spawned Gaussians are
(oo: P.....; = x"h + (60 + fou, OK-1])
(11)
where {oo, 01...., with that anchor.
OK- .......,.-11 IR k the learnable offiets and la the scaling factor associated
3.4 OPTICAL FLOW GUIDED DEFORMATION
High dynamic objects often appear in real-world videos. Currently, research on using 3D Gaussian representation for dynamic content mainly focuses on 4D video (Yang al. (024c). Studies on high dynamic content are still insufficient Some work uses optical flow information as auxiliary supervisory signals to guide the training process (Tang et 2023) However, such methods usually require an additional network to estimate the optical flow information between reconstructed frames. The performance is affected by the auxiliary network. The explicit representation of 3D Gaussian provides another way to utilize optical flow information.
During the training, we render adjacent two frames simultaneously. For Gaussian points that appear in both frames, we directly supervise their displacement in deformation field using the optical flow information corresponding to the pixel coordinate where the Gaussian point is projected according to
N
Coptical,ti
P Piit
Uiig l1:
(12)
where Pait fiitt are the coordinates of the i-th Gaussian shared between (V,, VoJa, and (V,, Vo l and l2 in pixel space respectively. Uir,, .1 is the estimated optical flow of corresponding image pixel. The explicit supervision is more effective than implicit information from original image. Another benefit is that the optical flow is estimated independent from training process and can be reused for same video with different model settings.
3.5 TRAINING PIPELINE, POST TRAINING COMPRESSION AND ENTROPY CODER
Fig. iillssrrtes the full pipeline of GSVC. We integrated all of the above modules to achieve joint training of reconstruction quality, bit rate, and optical flow constraints.
L - N(Da + 3K) (Centropy + Lhash) do Co X₂ Cscaffold
(13)
where Lree is fusion loss of L1 loss and SSIM loss between ground truth and average render results of Vy and Vo, Coptical is the optical loss. Lscaffold, Lentropy and Lhash are similar to previous work. X, do and Å are trade-off hyper-parameters among each component.
Despite HAC-based architecture can estimate the end-to-end entropy most parameters, the anchor coordinates are only quantized during training and not included in the entropy estimation pipeline. The network parameters are also saved in full precision bit In 3D scene compression, the bit rate of anchor coordinates and network parameters is not high, saving them in this way will not significantly affect performance. However, in video tasks, we found that the proportion of these two parts has significant increase. Therefore, further encoded these parameters improve the final RD performance.
For the anchor points, we geometry-based point cloud compression tool, G-PC Schwarz jet al. 2018), to ensure the accuracy of the reconstruction. We compress the quantized anchor point coordinates in a lossless manner. Since G-PCC compression reorders the anchor point sequence, we also adjust the order of related attributes before entropy coding them. For all network parameters,

---

Published as a conference paper at ICLR 2025
B
Chwsh
Centropy
Lacaffold
Entropy Nets
TSW Rendering
spawned
Altribute
3D Gaussians
Interp
Coptical
Gausslans
AE/AD
$29
Projection
Re M441
TSW Culling
Trainable networks
Operetion flow Gradient flow Desode
G-PCC
Trainable parameters
Masked offsets
Bitstream
The framework of GSVC. t is the timestamp of frame to be rendered. q. μl and or are estimated quantization step, mean and scale used in entropy coding. N denotes encode anchor attributes A = S E RD"", , E Re, E R3K } by Normal distribution. Similarly, B represent Bernoulli, used encode binary hash grid H. AE/AD represent ANS (Duda et al. 2015) encoder and decoder. Note the offsets mask is also decoded from bitstream.
Figure 3:
following previous work (Chen et al. 2021), we directly quantize them to bits. After this post- processing, the compression ratio can be further improved.
Furthermore, we implemented GPU-based entropy codec according to ANS theory (Duda et al, 2013 Bamler (022). Most of entropy coders utilize the powerful serial computation ability of CPU cores but misalign with the requirements of emerging compressor building on neural network, which conduct most of the computation on GPU. Frequent communication between CPU and GPU will deteriorate the coding speed, let alone the bottleneck of pure serial processing. In proposed entropy codec, take advantage of parallelization by probing decoding. Appendix A.J explains the detail designs.
4 EXPERIMENTS
4.1 EXPERIMENTS SETUP
Dataset All experiments are performed on popular UVG dataset (Mercat et al.
2020). We select 6 + 300 3900
first 7 videos as dataset, consisting videos with resolution of 1920 1080 and 600
frames in total, following previous works (Chen et al./ [021 Lee et al. 2023). In GSVC, we use
optical flow estimated by pre-trained model of VideoFlow (Shi et al. 2023).
Metries We use the peak signal-to-noise ratio (PSNR) in RGB 4:4:4 and MS-SSIM (Wagg et al.
2003) distortion quality metric, which are the most widely used metric in compression. We also report LPIPS (Zhang et al. 2018) in main experiment, which is popular perceptual metric to measure realism. Bit-per-pixel (BPP) is used as coding efficiency metric.
Implementation details We implement differentiable rasterizer with orthographic projection based the repository of HAC. The sliding window mechanism and GSVC are implemented with PyTorch. In experiments of 3DGS family methods, we build Structure-from conee-frro-ooioon (SfM) cloud directly from UVG dataset by COLMAP (Schönberger Frahm 2016 Schönberger et al., 2016). For video fails to converge generates multiple SIM clouds, we skip the video in experiments. For TSW variant of 3DGS family methods and GSVC, there is no trivial method to obtain a meaningful initialization like SfM. We simply start training from random initialized point cloud. We assign camera position of each frame as (0, O, t, where t, is normalized timestamp of corresponding frame. We set h = 0.4 for HoneyBee, h 0.2 for Shake and h 0.05 for other videos. We reproduce the results of
7

---

Published as a conference paper at ICLR 2025
Table 1: Numerical results of all methods. For vanilla 3DGS Kerbl al. 2023). Scaffold-GS (abbreviated as S-GS) (Lu et al. (024) and HAC (Chen et al. 2024). COLMAP fails to reconstruct valid scene on HoneyBee, Jockey and Sra so skip these videos. Subseript TSW denotes the TSW variants of these methods. The best and second best results are highlighted red and yellow cells. The size is measured in MB.
Data
Metrics
3DGS
S-GS
HAC
3DGSrsw
S-GS
HACrsw
GSVC
PSNR
25.87
28.52
26.27
28.54
29.21
29.77
30.25
SSIM f
0.8102
0.8367
0.8116
0.8414
0.8426
0.8549
0.8532
Beauty
LPIPS
0.6033
0.6034
0.6191
0.6022
0.6126
0.5925
0.5745
SIZE +
28.10
4.930
1.493
25.18
5.995
2.316
1.107
FPS
126.3
125.3
154.6
93.05
84.32
89.0
83.37
PSNR
23.47
26.59
25.42
29.03
29.04
32.00
33.19
SSIM
0.7461
0.8815
0,8542
0.8726
0.8753
0.9260
0,9517
Bospho.
LPIPS
0.4427
0.2601
0,2985
0,3503
0.3551
0.2660
0.1780
SIZF
121.0
7.990
2.105
24.80
5.903
5.091
2.050
FPS
53.24
92.61
137.0
77.63
60.81
50.82
67.03
PSNR
35.23
31.00
36.50
37.90
SSIM
0.9755
0.9525
0.9801
0.9830
Honey.
LPIPS
0.1844
0.2520
0.1728
0.1571
SIZE
23.87
5.903
3,169
2.313
FPS
81.33
98.48
76.73
39.63
PSNR
21.71
22.93
22.76
29.38
SSIM
0.7399
0.7673
0.7760
0.8956
Jockey
LPIPS
0,5342
0.5298
0.4975
0,3777
SIZE
22.48
6.741
3.062
2,429
FPS
83.96
49.17
41,42
61.70
PSNR
21.30
23.51
22.43
18.27
18.88
19.75
26.96
SSIM
0.8044
0.8655
0,8354
0.6143
0,6349
0.6943
0,9263
Ready.
LPIPS
0.3120
0.2718
0.2890
0.6213
0.6432
0.5381
0.2326
SIZE
631.4
85,3900
21.63
26.45
7.047
6.433
6,350
FPS
76.21
98.82
119.3
71.38
76.62
51.45
$1.90
PSNR
29.75
30.41
31.75
34.22
SSIM
0.9068
0.9113
0.9272
0,9470
Shake.
LPIPS
0.3279
0.3218
0.2719
0,2239
SIZE
24.16
9,824
3,729
3.153
FPS
73.39
83.22
77.81
42.93
PSNR
20.31
23.37
20.59
23.84
23.94
25.69
28.61
SSIM
0.6511
0.7705
0.6904
0.7779
0.7851
0.8491
0.9201
Yacht.
LPIPS
0.5379
0.4239
0.4855
0.4610
0.4703
0.3684
0.2469
SIZE L
50.77
38.60
5.485
24.34
5.902
5.502
3.7015
FPS
47.35
83.72
123.2
68.13
66.4746
50.3142
64.00
NeRV (Chen et al. 2021) according to original hyper- parameters setting but in per video way. We conduct all experiments single RTX 3090. More details can be found in Appendix A.2
4.2 QUANTITY AND QUALITY RESULTS
Table T demonstrates the result among 3DGS family methods and proposed method. Not surpris- ingly, vanilla 3DGS, Scaffold-GS and HAC cannot be considered valid video representation model, failing to reconstruct all videos in dataset. It obvious that not all videos contain a 3D scene, which key modeling assumption of 3DGS. For TSW variants of these methods, only require ad- jacent similarity among frames, which effectively transform these methods to video representation models. HACTSw achieves second best results in many metrics.
But we also notice that for ReadySetGo, TSW variants fail to surpass vanilla methods. The result indicates the insufficiency of representing video by TSW only for videos including high dynamic
8

---

Published as a conference paper at ICLR 2025
Setter
0.34
H224
Heatt
11265
B.MC
26
NeRV
0.30
0.28
02
11.04
VC1
N254
H1205
DEVE
DEVC
0.24
0.22
3.02
0.03
0.04
0.08
0.93
0.05 0.37
0.23
0.03
0.00
Figure 4: Comparison with other methods on UVG dataset. GSVC significantly outperforms NeRV (Chnn et al. 2021) and HNeRV (Chen al. 2023) MS-SSIM and LPIPS and achieves compara- ble MS MSSM and LPIPS performance with H.265 (veryslow) at low bit rate region.
36
NeRVINNeRV
35
Better
0,0005
0.8
GSVC A = 0.001
Optimall stream decoding
0.004
+300%
PSNR (d8)
33
0.6
GSVC 0.01
32
Frame available
40x FPS
0.0005
20 000
30
H.264
HNeRV
VCT
0.2
29
NeRV/HNeRV
0.0
FPS
Decoded bits (16)
(a) Results of rendering speed
(b) Results of stream decoding
Figure 5: Comparison with other methods on practical metrics. GSVC outperforms all learning- based methods on rendering speed. We omit DCVC (Li et al.] 2021 here due to its slow auto- regressive decoding (< 1 FPS). In stream decoding. when bit stream is completely decoded, all frames are available for rendering in all methods.
contents. Cooperating with proposed time-aware generation and optical guided deformation, GSVC achieves best result in majority of metrics except for FPS, Since GSVC requires rendering twice for each frame, this result is reasonable.
Fig. 4 demonstrates the comparison of GSVC and other methods including H.264, H.265, DCVC (L et al., 2021). VCT (Menterr et al. 2022). NeRV (Chen et al.) 202I) and HNcRV (Chen et al,, 2023}. We use 10.1, 0.04, 0.001, 0.005 and averages the results over same X. GSCV attains comparable performance in PSNR and significantly better performance in MS-SSIM and LPIPS than NeRV and HNeRV. This is a positive proof-of-coneey of applying 3DGS in video compression. We present more results and visualizations of all methods in Appendix A.5
4.3 RENDERING SPEED AND STREAM DECODING
Rendering speed is important for real-world application. Note here we refer decoding as the process of entropy decoding for GSVC and Huffman decoding for NeRV. Rendering speed only account for reconstruction since the decoding process can be easily overlapped by multiprocessing. Fig. [5a shows GSVC achieves % 40% higher FPS than NeRV and over 40x FPS than VCT, demon- strating the efficiency of our method.
Stream decoding is another important feature for practical usage. Fig. 5b shows the stream decoding result. NeRV must decoding whole network before rendering frames. But for GSVC, after receiv- ing common information including meta, anchor position and weights of network, the remaining bitstream be decoded demand. Besides, we also notice that better stream decoding perfor- mance is achieved when using high bit rate setting (smaller A). This is because the size of common information similar different bitrates.

---

Published as a conference paper at ICLR 2025
Table 2: Ablation results. We report BD-rate (%) Bjøntegaard 2001 of all settings relative to GSVC. Negative value denote the setting has better RD performance than GSVC. Failing to calculate BD-rate means the highest PSNR among all À of the setting smaller than the lowest PSNR of GSVC. w/o deformation means removing deformation network. w/o FiLM represents using original
MLP instead of network with FiLM layer.
Settings
Beauty
Bospho.
Honey.
Jockey
Ready.
Shake.
Yacht.
w/o optical guidance
-5.374
2.618
-5.060
81.21
78.37
-10.56
1.753
w/o post compression
72.81
42.75
68.66
40.68
32.24
58.49
39.31
w/o deformation
0.347
38.15
41.39
w/o FiLM
64.82
134.1
3.039
64.67
34.5
31.5
34.0
31.0
33.5
PSNR (dB)
S9N0 (2B)
30.0
29.5
0.05
h
0.2
28.5
h 0.2
31.5
0.02
0.03
0.04
0.05
0.02
0.03
0.04
0.05
BPP
BPP
(a) Results on ShakeNDry
(b) Results on Jockey
Figure 6: Ablation results of h. ShakeNDry contains an almost static background while Jocky is highly dynamic video.
4.4 ABLATION STUDY
Table. demonstrates the effectiveness of different components of GSVC. In general, all the pro- posed modules effectively improve the final performance except optical flow guidance. For videos with high dynamic contest, optical flow guidance is critical. For the data with negative effects, fur- ther inspection reveals that they are mainly caused by noise in the optical flow estimation results. This also indirectly indicates the importance of optical flow information. How overcome these inevitable noises is direction worth exploring in the future.
Fig. shows the impact of the h in TSW. For video that is close static, larger h performs better, while for dynamic video, a smaller h performs better. Fig. 6b also indicates that large h may lead unstable training for dynamic video. Due to the non-uniform dynamic characteristics of videos, manually specifying or determine by warm-up up training for all frames is far from optimal. How adaptively assign h to each frame according dynamics of adjacent frames is another valuable question.
DISCUSSION
Conclusion In this paper, we propose novel TSW orthographic projection effectively transform any 3D Gaussian model to video representation model. Based on TSW we further introduce prac- tical video compressor GSVC. As proof-of-concepl. video codec by extensive experiments. Our approach has demonstrated promising RD performance with practical fast rendering and stream decoding. With the advancements of 3DGS compression techniques (Wang et al. 2024), 3DGS-based video codec is worth exploring further in the future.
explore the characteristics of 3DGS-base
Limitation Similar to other INR-based methods, GSVC still relies videv training encoding process, which limits its application in real-time encoding tasks. Moreover, the proposed model still has room for improvement. These limitations also suggest more directions for further exploration.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work is supported in part by the National Natural Science Foundation of China under grant 62171248, 62301 189, the project of Peng Cheng Laboratory (PCL2023A08), and Shenzhen Sci- ence and Technology Program under Grant KJZD00040001107700000 BCBS20221008092 240611, CXWW20022081117296600 htt.1/11100000444000000000111110
REFERENCES
Robert Bamler. Understanding entropy coding with asymmetric numeral systems (ans): statisti- cian's perspective. arXiv preprint arXiv:2201.01744 2022.
Gisle Bjontegaard. Calculation of average psmr differences between rd-curves. 2001. URL https:
7/api. wtw.aattiiicccillaaa..roooooraaa
Hao Chen, Bo He, Hanyu Wang, Yixuan Ren, Ser Nam Lim, and Abhinav Shrivastava. Nerv: Neu- ral representations for videos. Advances in Neural Information Processing Systems, 34:21557-
21568, 2021.
Hao Chen, Matthew Gwilliam, Ser-Nam Lim, and Abhinav Shrivastava. Hnerv: hybrid neural representation for videos. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pp. 10270-10279, 2023.
Yihang Chen, Qianyi Wu, Jianfei Cai. Mehrtash Harandi, and Weiyao Lin. Hac: Hash-grid assisted context for 3d gaussian splatting compression. arXiv preprint arXiv:2403. 14530, 2024.
Jarek Duda, Khalid Tahboub, Neeraj J Gadgil, and Edward Delp. The use of asymmetric numeral systems an accurate replacement for huffmman coding. In 2015 Picture Coding Symposium (PCS), pp. 65-69. IEEE, 2015.
Emilien Dupont, Adam Goliñski, Milad Alizadeh, Yee Whye Teh, and Arnaud Doucet. Coin: Com- pression with implicit neural representations. arXiv preprint 1rXiv:2103.03/11 2021.
Emilien Dupont, Hrushikesh Loya, Milad Alizadeh, Adam Golinski, Yee Whye Teh, and Arnau Doucet. Coin++: Data agnostic neural compression. arXiv preprint arXiv:2201.12904, 1(2)4, 2022.
Kuofeng Gao, Huanqia Cai, Qingyao Shuai, Dihong Gong, and Zhifeng Li. Embedding self- correction inherent ability in large language models for enhanced mathematical reasoning. arXiv preprint arXiv:2410.1077 2024.
Berthold KP Horn and Brian G Schunck. Determining optical flow. Artificial intelligence, 17(1-3): 185-203, 1981.
Zhihao Hu, Guo Lu, and Dong Xu. Fve: A new framework towards deep video compression in feature space. Proceedings of the IEEE/CVF Conference on Computer Vision and Patlern Recognition, pp. 1502-1511, 2021.
Yujun Huang, Bin Chen, Shiyu Qin, Jiawei Li, Yaowei Wang, Tao Dai, and Shu-Tao Xia. Learned distributed image compression with multi-scale patch matching in feature domain. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 4322-4329 2023.
Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splat- ting for real-time radiance field rendering. ACM Trans. Graph., 1244):139-1,202 2023.
Ho Man Kwan, Ge Gao, Fan Zhang. Andrew Gower, and David Bull. Hinerv: Video compres- sion with hierarchical encoding-based neural representation. Advances in Neural Information Processing Systems, 36, 2023.
Théo Ladune, Pierrick Philippe, Félix Henry, Gordon Clare, and Thomas Leguay. Cool-chie: Coordinate-basse low complexity hierarchical image codec. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 13515-13522, 2023.

---

Published as a conference paper at ICLR 2025
Didier Le Gall. Mpeg: A video compression standard for multimedia applications. Communications of the ACM, 34(4):46-58, 1991.
Joo Chan Lee, Daniel Rho, Jong Hwan Ko, and Eunbyung Park. Ffnerv: Flow-guided Trame-wise neural representations for videos. In Proceedings of the 31st ACM International Conference Multimedia, pp. 7859-7870, 2023.
Jiahao Li, Bin Li, and Yan Lu. Deep contextual video compression. Advances Neural Information Processing Systems, 34:181 14-18125, 2021.
Jiahao Li, Bin Li, and Yan Lu. Neural video compression with feature modulation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 26099-26108, 2024.
Jianping Lin, Dong Liu, Houqiang Li, and Feng Wu. M-lve: Multiple frames prediction for learned video compression. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 3546,3554, 2020.
Youtian Lin, Zuozhuo Dai, Siyu Zhu, and Yao Yao. Gaussian-flow: 4d reconstruction with dy- namic 3d gaussian particle. In Proceedings of the IEEE/CVF Conference on Compuler Vision and Pattern Recognition, pp.
21145, 2024.
Jerry Liu, Shenlong Wang. Wei-Chiu Ma, Meet Shah, Rui Hu, Pranaab Dhawan, and Raquel Ur- tasun. Conditional entropy coding for efficient video compression. In European Conference Computer Vision, pp. 453-468. Springer. 2020.
Yang Liu, He Guan, Chuanchen Luo, Lue Fan, Junran Peng, and Zhaoxiang Zhang. City- gaussian: Real-time high-quality large-scale scene rendering with gaussians. arXiv preprint arXiv:2404.01133 2024.
Guo Lu, Wanli Ouyang, Dong Xu, Xiaoyun Zhang, Zhiyong Gao, and Ming- Ting Sun. Deep kalman filtering network for video compression artifact reduction. Proceedings of the European Con- ference on Computer Vision (ECCV), pp. 568-584, 2018.
Guo Lu, Wanli Ouyang, Dong Xu, Xiaoyun Zhang, Chunlei Cai, and Zhiyong Gao. Dve: An end-to-end deep video compression framework. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp. 11006-11015, 2019.
Tao La, Mulin Yu, Linning Xu, Yuanbo Xiangli, Limin Wang. Dahua Lin, and Bo Dai. Scaffold-gs: Structured 3d gaussians for adaptvde rendering. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 10654-206644 2024.
Fabian Mentzer, Eirikur Agustsson, Michael Tschannen, Radu Timofte, and Lue Van Gool. Practical full resolution learned lossless image compression. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
Fabian Mentzer, George Toderici, David Minnen, Sung-Jin Hwang, Sergi Caelles, Mario Lucic, and Eirikur Agustsson. Vct: A video compression transformer. arXiv preprint arXiv:2206007007, 2022.
Alexandre Mercat, Marko Viitanen, and Jarno Vanne. Uvg dataset: 50/120fps 4k sequences for video codec analysis and development. In Proceedings of the 11th ACM Multimedia Systems Conference, pp. 297-302, 2020.
Ben Mildenhall, Pratul P. Srinivasan, Matthew Taneik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes neural radiance fields for view synthesis. 2020.
Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville. Film: Visual reasoning with a general conditioning layer. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018
Shiyu Qin, Yi-Min Zhou, Jin in-Peng Wang, Bin Chen, Bao- Yi An, Tao Dai, and Shu- Xia. Pro- gressive learning with visual prompt tuning for variable-rate image compression. In 2024 IEEE International Conference Image Processing (ICIP), pp. 1767-1773. IEEE, 2024.
12

---

Published as a conference paper at ICLR 2025
Johannes Lutz Schönberger and Jan-Michael Frahm. Structure-froot motion revisited. In Confer- ence on Computer Vision and Pattern Recognition (CVPR), 2016.
Johannes Lutz Schönberger, Enliang Zheng, Mare Pollefeys, and Jan-Michael Frahm. Pixelwise view selection for unstructured multi-view stereo. In European Conference on Computer Vision (ECCV), 2016.
Sebastian Schwarz, Marius Preda, Vittorio Baroncini, Madhukar Budagavi, Pablo Cesar, Philip A Chou, Robert A Cohen, Maja Krivokuća, Sébastien Lasserre, Zhu Li, et al. Emerging mpeg standards for point cloud compression. and Systems, 9()): 133-113,, 2018.
on Emerging and Selected Topics in Circuits
Jin-Chuan Shi, Miao Wang, Hao-Bin Duan, and Shao-Hua Guan. Language embedded 3d gaus- sians for open- vocabulary scene understanding. Proceedings of the IEEE/CVF Conference on Computer Vision and Pultern Recognition, pp. 5333-5343, 2024.
Xiaoyu Shi, Zhaoyang Huang, Weikang Bian, Dasong Li, Manyuan Zhang, Ka Chun Cheung, Simon See, Hongwei Qin, Jifeng Dai, and Hongsheng Li. Videoflow: Exploiting temporal cues for multi-frame optical flow estimation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 12469-12480, 2023.
Yibo Shi, Yunying Ge, Jing Wang, and Jue Mao. Alphave: High-performance and efficient learned video compression. In European Conference on Computer Vision, pp. 616-631. Springer, 2022.
Inkyu Shin, Qihang Yu, Xiaohui Shen, So Kweon, Kuk-Jin Yoon, and hancing temporal consistency in video editing by reconstructing videos with 3d gaussian splatting. arXiv preprint ar arXiv:2406.0254) 2024.
Chen. En-
Rui Song, Dong Liu, Houqiang Li, and Feng Wu. Neural network-based arithmetic coding of intra prediction modes in heve. In 2017 IEEE Visual Communications and Image Processing (VCIP), pp. 1-4. IEEE, 2017.
Xiaodan Song, Jiabao Yao, Lulu Zhou, Li Wang, Xiaoyang Wu, Di Xie, and Shiliang Pu. A practical convolutional neural network as loop filter for intra frame. In 2018 25th IEEE International Conference on Image Processing (ICIP), pp. 1133-1137. IEEE, 2018.
Gary Sullivan, Jens-Rainer Ohm, Woo-Jin Han, and Thomas Wiegand. Overview of the high efficiency video coding (heve) standard. IEEE Transactions circuits and systems for video technology, 22(12): 2012.
Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Fridovich-Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. Advances in neural information processing systems, 33:7537-7547, 2020.
Ly Tang, Xinfeng Zhang, Gai Zhang, and Xiaoqi Ma. Scene matters: Model-based deep video compression. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 12481-12491, 2023.
Lucas Theis, Wenzhe Shi, Andrew Cunningham, and Ferenc Huszár. Lossy image compression with compressive autoencoders. International conference on learning representations, 2022.
Yuan Tian, Guo Lu, Yichao Yan, Guangtao Zhai, Li Chen, and Zhiyong Gao. A coding framework and benchmark towards low-bitrate video understanding. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.
Yufei Wang, Zhihao Li, Lanqing Guo, Wenhan Yang, Alex C Kot, and Bihan Wen. Contextgs: Compact 3d gaussian splatting with anchor level context model. arXiv preprint a722211405.20721, 2024.
Zhou Wang, Eero P Simoncelli, and Alan C Bovik. Multiscale structural similarity for image quality assessment. In The Thrity-Seventh Asilomar Conference on Signals, Systems Computers, 2003, volume 2, pp. 1 1398-1402 leee, 2003.
13

---

Published as a conference paper at ICLR 2025
Thomas Wiegand, Gary Sullivan, Gisle Bjontegaard, and Ajay Luthra. Overview of the h. 264/ave video coding standard. IEEE Transactions on circuits and systems for video technology, 13(7): 560-576, 2003.
Yichong Xia, Bin Chen, Yan Feng, Tianshuo Ge, Yujun Huang, Haoqian Wang, and Yaowei Wang. Multi-scale architectures matter: Examining the adversarial robustness of flow-based lossless compression. Pattern Recognition, 149:110242, 2024.
Jianing Yang, Xuweiyi Chen, Shengyi Qian, Nikhil Madaan, Madhavan Iyengar, David Fouhey, and Joyce Chai. Lim-grounder: Open-vocabulary 3d visual grounding with large language model as an agent. 2024 IEEE International Conference on Robotics and Automation (ICRA), pp- IEEE, 2024a.
Runyi Yang, Zhenxin Zhu, Zhou Jiang, Baijun Ye, Xiaoxue Chen, Yifei Zhang, Yuantao Chen, Jian Zhao, and Hao Zhao. Spectrally pruned gaussian fields with neural compensation. arXiv preprint arXiv:2405.cm 240.02465000666 2024b.
Ziyi Yang, Xinyu Gao, Wen Zhou, Shaohui Jiao, Yuqing Zhang, and Xiaogang Jin. Deformable 3d gaussians for high-fidelity monocular dynamic scene reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 20331-20344 2024c.
Zehao Yu, Anpei Chen, Binbin Huang, Torsten Sattler, and Andreas Geiger. Mip-splatting: Alias- free 3d gaussian splatting. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 19447-19456, 2024.
Richard Zhang, Phillip Isola, Alexci A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 386-595, 2018.
Xinjie Zhang, Xingtong Ge, Tongda Xu, Dailan He, Yan Wang, Hongwei Qin, Guo Lu, Jing Geng, and Jun Zhang. Gaussianimage: 1000 fps image representation and compression by 2d gaussian splatting, arXiv preprint ar arXiv:2403.08533 2024.
14

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1 GPU-BASED ANS CODEC
The ANS codec often considered to have higher encoding and decoding speeds while keeping similar coding efficiency to arithmetic coding. We implement our codec by extending the algorithm introduced by [Bamler (2022) to massively parallel environment. Algo. and Algo. demonstrates the encoding and decoding process.
Algorithm Encoding process
1: procedure ENCODE S4M,,LS(8, 1t,
2: Ps + PMF(*, ,t, o))
3: Ca CDF(s, pr, o))
4: head < 0
>> head is an unsigned 32 bit integer Empty array store bit stream
5: bitstream
6:
mask t 216 - -1
7:
cursor 0
8:
for i + 0, N - do
9:
p + Pa
10:
c Col
11:
if (head >> 16) > then
12:
cursor cursor +1 cestream/cursien <- head&rmask head head >> 16
13:
14:
15:
end
head mod p+0 head + head/p) 17:
head head 16)
19:
end for
while head do
Transfer the reaming bits head bitstream
21:
cursor cursor + bitstream [cursor] +- head&mask head +- head >> 16
22:
23:
24:
end while
25:
return bitstream
26:
end procedure
In Algo. is symbols vector to be encoded with N elements. o are distribution parameters associated with each symbol. On GPU, we can quickly compute probability mass function (PMF) value and cumulative density function (CDF) value of each symbol, shown in line in the algo- rithm. The remain part of algorithm follows the common ANS encoding process in serial fashion.
Algo. 2 demosttattes the decoding process of our codec. Different from common ANS decoder, we suppose all symbols in support set are valid symbol to be decoded. Obviously only the trial holds the correct symbol will succeed. For massively parallel GPU, performing all attempts will not take much longer time than completing just one. In serial decoding, it is difficult to decode the correct symbol with just a few trials. Therefore, even if the GPU thread slower than CPU thread, our method can still achieve faster decoding speed, especially with larger support sets.
In Algo. PMF() and CDF() compute the full PMF and CDF for whole support set of all symbols to be decoded. Fig. 7 demonstrates the encoding and decoding speed of proposed codec and torchac (Mentzer et al, 2019) on random generated message with 20000 symbols. Since our method does not compute full PMF values and CDF values, the acceleration is more significant in encoding stage.
15

---

Published as a conference paper at ICLR 2025
Algorithm 2 Parallel decoding process
1: procedure RECODE.SYMBBOLSStiittrrrmmm
M. o, .)
2: Pa f PMF 0T
D Ps: C. are computed outside the function,
3: C. CDF
put them here for demonstration.
4: while cursor 2 0 and head >> 16 == 0 do
5: token t bitstream[cursor/
6:
cursor t cursor
7:
head t- head << 16 token
8:
end while
9:
_shared__ symbols D
Empty array to store decoded symbols >> Current decoded symbol Residue decoding
10: shared
shared
12:
mask t- 2 -
for <- 0, N
do
14:
<- Pa :
15:
- C. i
16:
head&mask
17:
head t- head >>
18:
tid t thread ID of CUDA kernel
19:
1b = c(tid!
20:
ub cllid +
21:
if z > lb and = < ub then
Only thread holds correct symbol pass the check
tid
23:
r + 2 lb symbols[i] end if
25:
26:
block.sync()
D Syne threads CUDA kernel
27:
head <- head x pla] +
28:
(head >> 16) = 0 and cursor 2 then
29:
token +- bitstream[cursor]
30:
cursor + cursor 1
31:
head head << token
32:
end
33:
end for
34:
return symbols
end procedure
torchac Ours
10a
Time
10:1
10-2
Encoding
Decoding
Figure 7: Comparison results. Our method significantly faster than torchac
16

---

Published as a conference paper at ICLR 2025
A.2 MORE IMPLEMENTATION DETAILS
A.2.1 HYPER-PARAMETEKS SETTINGS
For original 3DGS, Scaffold-GS and HAC, we use the default setting. Note Xe is a tunable hyper- parameter to control the rate-distortion trade-off. order to obtain more intuitive results, we set Ae 10 to Beauty, Åe- 10 to Bosphorus and A 10-3 to the others.
In experiments of TSW variants, we sample coordinates used to initialize Gaussian from uniform distribution. We set initial anchor number 100000 for GSVC and 20000 to the others. Note GSVC achieves lower bit rate with more initialization anchors. For HAC1sW. we set de 0.001 for all videos.
Besides, we set densification threshold to 0.0005, which limited the densification speed in some high increase the capacity of hash grid. For the
dynamic videos. We also set grid dimension maining haper-parameters of GSVC shared with HAC, including learning rates and their scheduler, follow the original setting.
A.2.2 TRAINING SCHEDULE
In all experiments, train GSVC 40000 iterations in total. In first 10000 iterations, precision training and adding uniform noise to simulate quantization error from iteration 10000 to 35000. After 35000 iterations, we employ a straight-through estimator (Theis al. 2022) align with real quantization. The entropy constraints is added from iteration 15000. We apply anchor densification and pruning between iteration 1500 and 25000. We also pause the operations at the beginning of quantization training i.e. iterations between 10000 and 11000 to improve training stability.
full
A.2.3 BUILDING SFM CLOUDS FROM VIDEO
Because SfM clouds initialization is necessary for 3DGS family methods, we generate the point clouds using COLMAI Schönberger & Frahm 2016 Schönberger et al. 2016).
import pycolmap import pathlib
fiipuutpattbe
(ttihiib..aahh//pppaatttt/o///o/ _drr')
image_dir htanbb..Path(//aaahtt///rraa
database_path output_path / 'database.db'
image_dir)
maps = www.:o/mmp..icceeeetaaaaappppppe
database_pathe image_diz: dir, output_path
12 maps [0] write (output_path)
Listing I: Snippet used to build SfM clouds from video frames
We successfully build SIM for Bosphorus and YachtRide, COLMAP extract multiple SIMs from Beauty, Jockey and ReaySetGo. Since the biggest SIM of Beauty and ReadySetGo includes more than 500 frames of original video, we consider them as valid SfM. For HoneyBee and ShakeNDry, COLMAP fails converge.
A.2.4 DETAILS OF BASELINE METHODS
For H.265 and H.264 we follow the settings in DCVC al. 202T) and average the results over same qp value. For DCVC we use the pre-trained models to evaluate the performances. Instead of
17

---

Published as a conference paper at ICLR 2025
Rendering
V
Camera plane
Figure 8: Example of rendering one frame in real anchors cloud.
testing only 120 frames in original paper, we test all frames in UVG dataset. The results of VCT (Mentzer al., 2022) is copied from original paper. Different from other methods using models training with same specified loss function in all experiments, the results of VCT on PSNR and MS-SSIM are from corresponding models i.e. test PSNR using models trained for PSNR and test MS-SSIM using models trained for MS-SSIM.
A.3 MORE DETAILS ABOUT RENDERING
Fig. 8 illustrates real example of rendering one frame. To render frame at time l,, camera plane z 40- Based on the camera plane, we cull anchors that do not participate in the rendering (gray points in Fig. 8. For anchors fall into Vy and Vo. we render them separately. The rendering follows standard 3D Gaussian Splatting rendering procedure, except skipping view frustum culling and replacing perspective projection matrix with orthographic projection matrix (Eq. 2
Fig. 9 demonstrates progressive rendering results. In each setting, we use subset of Gaussians in Vy and V, render a frame, according Gaussian particles' distance to camera plane. For example, when d = 0.03, we only select Gaussians with distance smaller than 0.03. It is obvious that Gaussians near camera plane tend to focus on dynamic objects i.e. dog the scene, while distant Gaussians pay more attention to static background, which is shared by more frames.
A.4 RESULTS OF ENCODING TIME
Table demonstrates the comparison encoding time. GSVC outperforms other INR-based meth- ods. It should be noted that smaller total rendered frames during training does not necessarily mean faster converging. Training time depends on model size, model structure, and whether entropy con- straint is enforced during training. For GSVC, the time of one iteration with entropy constraint is longer than without entropy constraint due to the consumption the entropy network. Since there no accepted metric like FLOPS to measure the computational complexity of 3D Gaussian Splat- ting rendering, we use GPU utilization from nvidia-smi as reference. A utilization below 100% indicates that GSVC's computational consumption is lower than other methods. However, it also suggests that the GPU's capability is not fully tapped. This limitation calls for further exploration and improvement in the future.
18

---

Published as a conference paper at ICLR 2025
(a) d 0.03
(b) d 0.06
(c) 0.09
(d) d 0.12
(e) d 0.15
( = 0.20
Figure 9: Progressive rendering results. d is the max distance between camera plane and Gaussian particles participating rendering.
Table 3:
Training/Encoding time comparison. Frames represents number of total frames rendered during training. We show the results of typical 600 frames video UVG dataset. Training time of HNeRV is sensitive to model size i.e. bpp, so we demonstrate a range here. GPU Util. is from nvidia-smi.
Method
Frames
Time
GPU Util.
NeRV
240k
22ks (6h6m)
100%
HNeRV
1 180k
13.8ks 28.2ks (3h50m~7h50m)
100%
GSVC
80k
12.4ks (3h26m)
85%
A.5 VISUALIZATION
We demonstrate the visualization results in following pages.
19

---

Published as a conference paper at ICLR 2025
0.00
213
8.0
314
ONC
0.33
092
6.50
(a) PSNR
(b) MS-SSIM
(c) LPIPS
oohuk
ohuuk
igiturk
igiturk
ioiturk
o
(d) Ground truth
(e) GSVC
(f) NeRV
eelukk
ouuk
ioirok k
ioiturk
igiturk
g 3DGSTsw
(h) Scaffold-GSnsw
HACTsw
ituak
oiruk
oituck
i) 3DGS
(k) Scaffold-GS
HAC
Figure 10: Beauty
20

---

Published as a conference paper at ICLR 2025
0.2
107
4.1
12.
5:4
412
0.02
2.55
0.02
0.87
(a) PSNR
(b) MS-SSIM
(c) LPIPS
(d) Ground truth
(e) GSVC
(1) NeRV
g 3DGSTsw
(h) Scaffold-GSssw
HACrsw
i 3DGS
(k) Scaffold-GS
HAC
Figure 11: Bosphorus
21

---

Published as a conference paper at ICLR 2025
3.44
02080
3.42
0:90
0.108
3.49
031
4.22
(a) PSNR
(b) MS-SSIM
(c) LPIPS
(d) Ground truth
(e) GSVC
(I) NeRV
gg) 3DGSIsw
(h) Seaffold-GSrsw
HACısw
Figure 12: HoneyBee
22

---

Published as a conference paper at ICLR 2025
4.2
(a) PSNR
(b) MS-SSIM
(c) LPIPS
Zghuts
100
100
100
(d) Ground truth
(e) GSVC
(I NeRV
g 3DGS
(h) Scaffold-GSrsw
ii HACIsw
Figure 13: Jockey
23

---

Published as a conference paper at ICLR 2025
770
355
34.
ONL
5.2
(a) PSNR
(b) MS-SSIM
(c) LPIPS
2
2
2
(d) Ground truth
(c) GSVC
(I) NeRV
2
g 3DGSTSW
(h) Scaffold-GSnsw
HACTsw
2
2
2
i) 3DGS
(k) Scaffold-GS
HAC
Figure 14: ReadySetGo
24

---

Published as a conference paper at ICLR 2025
3.90
3.50
3.44
37.
0.23
8.00
11.
Hard
0.64
6.6
6.6
(a) PSNR
(b) MS-SSIM
(c) LPIPS
(d) Ground truth
(e) GSVC
II NeRV
g 3DGS
(h) Scaffold-GSngw
HACrsw
Figure 15: ShakeNDry
25

---

Published as a conference paper at ICLR 2025
185
212
20.
0.44
212
ONT
0:2
6.2
(a) PSNR
(b) MS-SSIM
(c) LPIPS
(d) Ground truth
(e) GSVC
( NeRV
g 3DGSTsW
(h) Scaffold-GSrsw
HACTsw
i 3DGS
(k) Scaffold-GS
HAC
Figure 16: YachtRide
26


Reviews:
[{"review_id": "fdbxejn1TJ", "reviewer": "Reviewer_czLV", "summary": "This paper introduces a 2D video compression method using 3D Gaussian Splatting (3DGS), aiming for faster frame reconstruction and stream decoding. However, concerns remain about its limited novelty, as many techniques (e.g., time embeddings, Gaussian attribute adjustments) are adapted from NeRF and 4DGS. The framework largely assembles prior methods (e.g., HAC, GPCC, and optical flow loss from 4DGS) without substantial innovation, raising questions about the applicability and originality of the approach in RGB video compression.", "questions": "Considering the typical aim of RGB image/video compression to leverage extensive priors learned from large datasets, how does the proposed framework add value when applied to compress a single video? Is there any particular benefit in this context?", "limitations": "", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. Improved Inference Speed:\n\nThe compression framework outperforms INR-based inference in rendering speeds, a notable improvement in practical applications where real-time decoding is essential.\n\n2. Broad Performance Evaluation: \n\nThe authors provide comprehensive experiments and comparisons with alternative compression methods, demonstrating that GSVC achieves favorable rate-distortion performance across different quality metrics.", "weaknesses": "1. Lack of Novelty:\n\nThe technical novelty of the work is limited. Key techniques, such as time-position embeddings and Gaussian attribute variation over time, have been previously introduced in NeRF and 4DGS [1]. The approach of offsetting Gaussian positions based on features at different timestamps is conceptually similar to the offset extraction in dynamic 4DGS methods. The overall compression framework aligns closely with HAC [2], while the compression of x,y,z coordinates with GPCC was introduced in other prior work. Similarly, optical flow loss is a component already implemented in 4DGS [3]. And the GPU-based ANS ae/ad is already implemented in CompressAI[4].\n\n\n2. No Comparison with HEVC/VVC: \n\nThe paper lacks a direct performance comparison with widely-used standards like HEVC or VVC, making it difficult to gauge its practical effectiveness against traditional video compression methods.\n\n\n3. Questionable Value for INR/3DGS in RGB Video Compression: \n\nThe application of INR/3DGS techniques to RGB video compression seems inherently misaligned with the goals of video compression. Compression frameworks for RGB video typically rely on extensive data to learn a generalized data distribution, leveraging priors that do not exist within a single video or image.\n\n[1] 4D Gaussian Splatting for Real-Time Dynamic Scene Rendering\n\n[2] HAC: Hash-grid assisted context for 3D Gaussian splatting compression\n\n[3] MotionGS: Exploring Explicit Motion Guidance for Deformable 3D Gaussian Splatting\n\n[4] https://github.com/InterDigitalInc/CompressAI", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1NgibnLnCb", "reviewer": "Reviewer_JGLW", "summary": "The authors propose a novel approach to 2D video compression using 3D Gaussian Splatting (3DGS). They represent 2D video sequences as a volumetric distribution in 3D space, using X-Y-T coordinates. To reduce temporal redundancy, they introduce a Toast-like Sliding Window (TSW) projection method that considers Gaussian points within a temporal window centered on the rendering plane. The framework incorporates optical flow information and a FiLM-based deformation field to better capture temporal dynamics. An entropy coding scheme is implemented to optimize model compression and enhance streaming efficiency.\n\nThe paper presents three main contributions:\n* Introduction of a novel paradigm that represents 2D video using 3D Gaussian models, supported by the TSW projection technique\n* Development of GSVC, a comprehensive compression pipeline that combines HAC, deformable fields, and opacity flow regularization\n* Experimental validation showing GSVC achieves comparable quality to NeRV while providing 30% faster rendering and flexible bitrate-availability tradeoffs", "questions": "- In Figure 5, there remains a gap in frame availability even after complete bit decoding. What factors contribute to this limitation?\n- Another concern is about the depth sorting when it comes to rendering. there could be a scenario when two gaussians (lets say Ga and Gb), in the frame t-1, fall into Vf, while in the next frame t, they fall into Vb, due to the time plane moving forward. Yet in this case the relative depths of Ga and Gb to the image plane are reversed \n  - for example, in t-1, Ga, Gb in Vf, and Ga is close to the plane, and Gb is further to the plane, then in t, Ga, Gb in Vb, it must be that Ga is further to the plane and Gb looks closer to the plane\n  - Such kind of reversing might lead to totally different coloring of the same pixel after alpha blending. has such inconsistency in consecutive frames ever been observed in your experiments?\n  - Also how the two images projected from Vb and Vf get mixed into a single image? simply averaging them?\n  - It would be interesting if you could visualize the xyt space as a normal 3D space and in the meantime, visualize the rendering of one of the frames from nearby Gaussians. That will be cool.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* Pioneering application of 3DGS to video compression\n* Minimal assumptions about video content, relying primarily on temporal continuity, which enables broad applicability\n* High adaptability with various 3DGS models, opening promising research directions\n* Demonstrated practical utility through flexible bitrate-availability tradeoffs, particularly valuable for streaming applications", "weaknesses": "I do really like the idea of the paper yet the evaluation section could be further polished.\n* Limited comparison baseline, with NeRV as the only reference. The evaluation would benefit from comparisons with other neural video compression methods\n* Evaluation only focus on low bitrate/bpp scenarios, while contemporary neural compression methods (e.g., VCT, DCVC) typically demonstrate performance at higher bitrates with PSNR values exceeding 34 on the UVG dataset.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "j47tPkJED3", "reviewer": "Reviewer_jDPh", "summary": "This paper explores 3DGS for video compression. It Introduces TSW projection to adapt 3D Gaussian Splatting for 2D video representation. This paper also develops an end-to-end trainable video compression method based on deformable Gaussian and optical guidance to better model motion content in videos.", "questions": "1. Figure 1's ground truth image seems to be with low quality. Based on my knowdege, 3DGS's reconstruct is prune to have high numerical results but lack of details. Have the author consider this issue for compresssion task ?\n2. It is hard to see if there is a temporal inconsistency without video examples. Can you provide video comparisons ?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Orthographic projection and sliding windows for videos are reasonable ideas for adapting the original perspective projection in 3DGS. \n\n2. Experiments on the UVG dataset show that the developed GSVC codec achieves RD performance compared to NeRV and offers efficient rendering with inherent stream decoding capability. Besides, this paper presents a relatively complete work for the codec workflow. \n\n3. The paper mentions that the code will be open-source. I appreciate the promise of open source, especially in video compression.", "weaknesses": "1. the proposed method still needs to train per video. Which i think is a limitation for real-world usage. \n\n2. As video codecs are a well-developed method, this paper does not include comparisons to traditional codecs. I know neural representation for compression is a novel and hot feild. But it’s more valuable and practice to highlight the specific strengths of each methods, especially in terms of the computation costs for encoding.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cA5bdn0Pmq", "reviewer": "Reviewer_nojS", "summary": "This paper proposes a new framework for video compression from the perspective of 3D Gaussian Splatting (3DGS). The key idea is to construct an interesting spatial-temporal modeling way called toast-like sliding window to cope with the 3D scene modeling in a short time period. This design enables the introduction of several 3DGS-based compression methods into video compression areas. With further careful designs like optical-flow supervision and time-aware Gaussians, the method achieves better compression quality then previous implicit-representation-based approaches like NeRV.", "questions": "1. It would be better to provide the rasterization process when applying the Orthographic Projection in the context of Gaussian Splatting. Including these changes in the appendix will help the readers understand the paper more clearly.\n2. How the anchor points are selected in the toast-like sliding windows?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The idea of toast-like sliding windows is interesting. It provides an insightful design by treating the time step as an extra dimension to construct a 3D structure.\n2. The overall writing is clear and well-motivated. The logical flow is very clear and easy to understand.\n3. The overall idea of the exploration of video compression from the 3DGS perspective is novel.", "weaknesses": "1.  Why do toast-like sliding windows need to render a short future period of content? It would be better to provide more illustrations about the detailed rendering process.\n2. Ablation study of FiLM design. The analysis of the change from MLP to FiLM-based network structure is missing.\n3. It would be better to include a more comprehensive comparison of video compression approaches like H.264/265 and HNeRV for in-depth analysis.\n4.  There are several typos in the writing like line 243, us-> use.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)