Title: LightGaussian: Unbounded 3D Gaussian Compression with 15x Reduction and 200+ FPS
Venue: NeurIPS 2024
Accepted: yes
Keywords: Gaussian Splatting

Efficient Rendering
Abstract: Recent advances in real-time neural rendering using point-based techniques have enabled broader adoption of 3D representations. However, foundational approaches like 3D Gaussian Splatting impose substantial storage overhead, as Structure-from-Motion (SfM) points can grow to millions, often requiring gigabyte-level disk space for a single unbounded scene. This growth presents scalability challenges and hinders splatting efficiency. To address this, we introduce LightGaussian, a method for transforming 3D Gaussians into a more compact format. Inspired by Network Pruning, LightGaussian identifies Gaussians with minimal global significance on scene reconstruction, and applies a pruning and recovery process to reduce redundancy while preserving visual quality. Knowledge distillation and pseudo-view augmentation then transfer spherical harmonic coefficients to a lower degree, yielding compact representations. Gaussian Vector Quantization, based on each Gaussian’s global significance, further lowers bitwidth with minimal accuracy loss. LightGaussian achieves an average 15 times compression rate while boosting FPS from 144 to 237 within the 3D-GS framework, enabling efficient complex scene representation on the Mip-NeRF 360 and Tank & Temple datasets. The proposed Gaussian pruning approach is also adaptable to other 3D representations (e.g., Scaffold-GS), demonstrating strong generalization capabilities.

LightGaussian: Unbounded 3D Gaussian Compression with 15x Reduction and 200+ FPS

Abstract

Recent advances in real-time neural rendering using point-based techniques have enabled broader adoption of 3D representations. However, foundational approaches like 3D Gaussian Splatting impose substantial storage overhead, as Structure-fromMotion (SfM) points can grow to millions, often requiring gigabyte-level disk space for a single unbounded scene. This growth presents scalability challenges and hinders splatting efficiency. To address this, we introduce LightGaussian , a method for transforming 3D Gaussians into a more compact format. Inspired by Network Pruning, LightGaussian identifies Gaussians with minimal global significance on scene reconstruction, and applies a pruning and recovery process to reduce redundancy while preserving visual quality. Knowledge distillation and pseudo-view augmentation then transfer spherical harmonic coefficients to a lower degree, yielding compact representations. Gaussian Vector Quantization, based on each Gaussian's global significance, further lowers bitwidth with minimal accuracy loss. LightGaussian achieves an average 15 × compression rate while boosting FPS from 144 to 237 within the 3D-GS framework, enabling efficient complex scene representation on the Mip-NeRF 360 and Tank & Temple datasets. The proposed Gaussian pruning approach is also adaptable to other 3D representations (e.g., Scaffold-GS), demonstrating strong generalization capabilities.

1 Introduction

Novel view synthesis (NVS) aims to generate photo-realistic images of a 3D scene from unobserved viewpoints, given a set of calibrated multi-view images. This capability has widespread applications

in virtual reality [1], augmented reality [2], digital twins [3], and autonomous driving [4]. Neural Radiance Fields (NeRFs) [5-7] have shown promise in 3D modeling from multi-view images by mapping 3D locations and view directions to view-dependent color and volumetric density, with pixel intensity rendered through volume rendering [8]. However, NeRF and its variants face limitations in rendering speed, limiting their deployment in real-world scenarios. To address this, voxel-based representations [9-13], hash grids [14], and neural light fields [15] have been developed. Despite improvements, these methods often compromise between quality and speed.

Recent progress in point-based 3D Gaussian Splatting (3D-GS) [16] has enabled real-time rendering with photo-realistic quality for complex scenes. By representing the scene with explicit 3D Gaussians and using a splatting technique [17], 3D-GS balances speed and quality, making it suitable for large-scale scenarios like digital twins and autonomous driving. However, point-based methods incur high storage costs, as each point and its attributes require independent storage. Additionally, heuristic densification of sparse SfM points into dense Gaussians often results in overparameterization, leading to excessive storage and slower rendering speeds. For instance, a typical unbounded 360-degree scene in 3D-GS [7] may require over 1GB of storage (e.g., 1.4GB for the Bicycle scene).

In this paper, we address storage and rendering speed issues by developing a compact representation that retains the original rendering quality. The heuristic densification process in 3D-GS results in significant redundancy. Our method, LightGaussian , reduces redundancy by targeting both Gaussian count (N) and feature dimension (F) through a comprehensive pipeline:

To reduce Gaussian count (N), we propose a Gaussian Pruning and Recovery step, identifying and removing Gaussians with minimal impact on visual quality, followed by recovery to ensure smooth adaptation.

For compressing features (F), we introduce an SH Distillation process to compact higherdegree spherical harmonic (SH) coefficients, supported by pseudo-view augmentation. Additionally, we employ Vector Quantization (VQ) to adaptively select a codebook of Gaussian attributes (e.g, positions, scales, and rotations), reducing precision for less significant features and applying quantization-aware fine-tuning to maintain quality.

In summary, LightGaussian achieves substantial compression (e.g., from 782MB to 45MB) while maintaining visual fidelity (SSIM decrease of only 0.007 on Mip-NeRF 360). Rendering speed also improves, reaching over 200 FPS on complex scenes. Our Gaussian Pruning and Recovery method generalizes well, enhancing performance across different 3D Gaussian formats, such as Scaffold-GS.

2 Related Works

Efficient 3D Scene Representations. Neural radiance fields (NeRF) [5] use multi-layer perceptrons (MLPs) to represent scenes, setting new standards for view synthesis quality. However, NeRF models face challenges with slow inference speeds, limiting practical use. Efforts to address this have explored ray re-parameterizations [6, 7], explicit spatial data structures [18-21, 9, 14, 13], caching and distillation techniques [15, 22-24], and ray-based representations [25, 26]. Nevertheless, achieving real-time rendering remains challenging for NeRF methods, especially for large-scale scenes where multiple queries per pixel hinder performance.

Point-based representations, like Gaussians, have been explored in various applications, such as shape reconstruction [27], molecular modeling [28], and point-cloud replacement [29], as well as in shadow [30] and cloud rendering [31]. Pulsar [32] demonstrated efficient sphere rasterization, while 3D Gaussian Splatting (3D-GS) [16] applies anisotropic Gaussians [33] with tile-based sorting to achieve real-time speed and quality comparable to MLP-based methods like Mip-NeRF 360 [7].

Despite its strengths, 3D-GS has high storage demands due to the extensive attributes stored with each Gaussian, often requiring gigabytes per scene and hindering rendering efficiency. Recent concurrent works address these issues by using region-based vector quantization [34], K-means codebooks [35], view-direction exclusion [36], or learned binary masks and grid-based neural networks instead of spherical harmonics (SHs) [37] to reduce model size.

Pruning and Quantization. Model pruning reduces neural network complexity by removing nonsignificant parameters, balancing performance and resource use. Unstructured [38] and structured

Gaussian Prune & Recovery

SH Distillation

High-degree

Low-degree

Rasterizer

Rasterizer

Distillation

Attributes

XYZ

Spherical Harmonics

Opacity/Rot/Scale

FP16

FP16

Vector Quantization

FP16

Gaussian Vector

Quantization

Score

Calculation

Multi-view

Images

Pseudo cameras

Training cameras

pruning [39, 40] eliminate parameters at the weight level and neuron/channel levels, resulting in a smaller, more efficient architecture. Iterative magnitude pruning (IMP), where low-magnitude weights are progressively removed, has proven effective in methods like lottery ticket rewinding [41, 42].

Vector quantization (VQ) [43] compresses data by representing it with discrete codebook entries (tokens), using mean square error (MSE) to select the closest match in the codebook for each data vector. Prior studies [44-46] have shown that learning discrete, compact representations enhances visual understanding and model robustness. VQ has thus been widely applied in image synthesis [47], text-to-image generation [48], and novel view synthesis [18, 49, 50].

Knowledge Distillation. Knowledge Distillation (KD) [51-54, 15] trains a smaller student model by transferring knowledge from a larger teacher model [55]. In 3D vision, KD has been applied to neural scene representations, leveraging view renderings to incorporate 2D priors. For example, DreamFusion [56] and NeuralLift-360 [57] use pre-trained diffusion models for 3D generation, while models like DFF [58], NeRF-SOS [59], INS [60], and SA3D [61] distill 2D image feature extractors to 3D tasks. KD has also been central to compressing scene representation models [15, 24].

3 Methods

Overview. The LightGaussian framework is illustrated in Fig. 2. The 3D-GS model is trained on multi-view images, initialized from SfM point clouds, and expanded to millions of Gaussians to represent the scene comprehensively. Our pipeline then processes the 3D-GS model into a compact format using Gaussian Prune and Recovery to reduce the number of Gaussians, SH Distillation to eliminate redundant SHs while retaining key lighting information, and Vector Quantization to store Gaussians with lower bit-width.

3.1 Background: 3D Gaussian Splatting

3D Gaussian Splatting (3D-GS) [16] is an explicit point-based 3D scene representation, utilizing Gaussians with various attributes to model the scene. When representing a complex real-world scene, 3D-GS is initialized from a sparse point cloud generated by SfM, and Gaussian Densification is applied to increase the Gaussian counts that are used for handling small-scale geometry insufficiently covered and over reconstruction. Formally, each Gaussian is characterized by a covariance matrix Σ and a center point X , which is referred to as the mean value of the Gaussian:

where Σ can be decomposed into a scaling matrix S and a rotation matrix R .

The complex directional appearance is modeled by an additional property, Spherical Harmonics (SH), with n coefficients, { c i ∈ R 3 | i = 1 , 2 , . . . , n } where n = D 2 represents the number of coefficients of SH with degree D . A higher degree D equips 3D-GS with a better capacity to model the viewdependent effect but causes a significantly heavier attribute load.

When rendering 2D images from the 3D Gaussians, the technique of splatting [62, 17] is employed for the Gaussians within the camera planes. With a viewing transform denoted as W and the Jacobian

PSNR: 27.2 (3D-GS)

PSNR: 25.3(Opacity prune)

CDF

二

PDF

CDF

of the affine approximation of the projective transformation represented by J , the covariance matrix Σ ′ in camera coordinates can be computed as Σ ′ = JW Σ W T J T . Specifically, for each pixel, the color and opacity of all the Gaussians are computed using the Gaussian's representation Eq. 1. The blending of N ordered points that overlap the pixel is given by the formula:

Here, c i , α i represents the view-dependent color and opacity, calculated from a 2D Gaussian with covariance Σ multiplied by an optimizable per-3D Gaussian's opacity. In summary, each Gaussian point is characterized by attributes including: position X ∈ R 3 , color defined by spherical harmonics coefficients SH ∈ R ( k +1) 2 × 3 (where k represents the degrees of freedom), opacity α ∈ R , rotation factor R ∈ R 4 , and scaling factor S ∈ R 3 .

3.2 Gaussian Pruning & Recovery

Gaussian densification [16], which clones and refines the initial SfM point cloud, enhances small-scale geometry and detailed scene appearance by improving coverage. While this approach significantly boosts reconstruction quality, it increases the number of Gaussians from thousands to millions after optimization, resulting in substantial storage demands.Inspired by neural network pruning techniques [63] that remove less impactful neurons while preserving overall performance, we propose a tailored pruning strategy for 3D Gaussian Splatting to reduce over-parameterized points while maintaining accuracy. Identifying redundant yet recoverable Gaussians is essential to our approach. However, pruning Gaussians based on simple criteria (e.g., point opacity) risks degrading modeling performance, as it may eliminate essential scene details, as shown in Fig. 3.

Global Significance Calculation. Inspired by Eq. 2, we go beyond using Gaussian opacity alone to assess significance by evaluating 3D Gaussians within the view frustum, projecting them onto the camera viewpoint for rendering. The initial significance score of each Gaussian (denoted as G ( X j ) ) can then be quantified based on its contribution to each pixel (ray, r i ) across all training views using the criteria ✶ ( G ( X j ) , r i ) whether they intersect or not. Consequently, we iterate over all training pixels to calculate the hit count of each Gaussian. The score is further refined by the adjusted 3D Gaussian's volume γ ( Σ j ) and 3D Gaussian's opacity σ j , as they all contribute for the rendering formula. The volume calculation equation of the j -th 3D Guassian is V ( Σ j ) = 4 3 πabc , where abc are the 3 dimensions of Scale ( S ). We also consider the transmittance T is calculated with one subtract all the 3D Gaussian's opacity the ray hit before the j -th 3D Gaussian, T = ∏ j -1 i =1 (1σ i ). Finally, the global significance score can be summarized as:

where j is the Gaussian index, i denotes a pixel, and M , H , and W represent the number of training views, image height, and width, respectively. ✶ is an indicator function that determines whether a Gaussian intersects a given ray. However, using Gaussian volume alone tends to overemphasize background Gaussians, leading to excessive pruning of Gaussians modeling fine geometry. Thus, we propose a more adaptive approach to measuring volume dimensions:

Here, the calculated Gaussian volume is firstly normalized by the 90% largest of all sorted Gaussians, clipping the range between 0 and 1, to avoid excessive floating Gaussians derived from vanilla 3D-GS. The β is introduced to provide additional flexibility.

Gaussian Co-adaptation. Werank all Gaussians by their global significance scores to quantitatively guide pruning of the lower-ranked Gaussians. The remaining Gaussians are then jointly adapted by fine-tuning their attributes-without additional densification-to offset the minor loss from pruning. This adaptation is performed using photometric loss on the original training views, aligned with 3D-GS training for 5,000 iterations.

3.3 Distilling into Compact SHs

In the uncompressed Gaussian Splat representation, a substantial 81.3 percent of the feature dimension is occupied by Spherical Harmonics (SH) coefficients, requiring (45+3) floating-point values per splat. Directly reducing the SH degree can save disk space but also diminishes surface 'shininess' and affects specular reflections.

To balance model size with scene quality, we introduce a knowledge distillation approach. Knowledge is distilled from a teacher model with full-degree SHs to a student model with truncated, lower-degree SHs. Supervision is based on the difference in predicted pixel intensities between the two models, with images synthesized at camera positions by sampling around each training view according to a Gaussian distribution:

where R and t denote rendering camera rotation and position, t pseudo and t train represent the newly synthesized and training camera positions, respectively. N denotes a Gaussian distribution with mean 0 and variance σ 2 , which is added to the original position to generate the new position.

3.4 Gaussian Attributes Vector Quantization

Vector quantization (VQ) clusters voxels into compact codebooks, enabling high compression rates. However, quantizing Gaussian attributes in a point-based, inherently non-Euclidean representation poses notable challenges. Our empirical findings indicate that applying quantization across all elements-especially for attributes like position, rotation, and scale-results in significant accuracy losses and a marked reduction in precision when represented discretely.

We propose applying VQ to the Spherical Harmonics (SH) coefficients, based on the assumption that a subgroup of 3d Gaussians typically exhibits a similar appearance. Fundamentally, VQ segments the Gaussians G = { g 1 , g 2 , . . . , g N } (here we apply it on SH) to the K codes in the codebook C = { c 1 , c 2 , . . . , c K } , where each g j , c k ∈ R d and K ≪ N. d means SH dimension. We aim to strike a balance between rendering quality loss and compression rate by leveraging the pre-computed significance score from Eq. 3. Based on this, we apply VQ selectively on the least significant elements in the Spherical Harmonics (SHs). Specifically, we initialize C via K-means, iteratively sample a batch of G , associates them to the closest codes by euclidean distance, and update each c k via moving average rule: c k = λ d · c k +(1 -λ d ) · 1 / T k · ∑ g j ∈R ( c k ) GS j · g j , where T k = ∑ g j ∈R ( c k ) GS j is the significance score (Eq. 3) which is assigned to the code vector c k , R ( c k ) is the set of Gaussians associated to the k -th code. λ d = 0 . 8 represents the decay value, which is utilized to update the

Ground-truth

RGB (3D-GS)

RGB (Ours)

code vector using a moving average. We fine-tune the codebook for 5,000 iterations, while fixing the gaussian-to-codebook mapping. We disable additional clone/split operations and leverage photometric loss on the training views. To preserve essential attributes, including Spherical Harmonics with higher global significance scores, along with Gaussian position, shape, rotation, and opacity, we skip VQ for these elements and store them directly in float16 format.

4 Experiments

4.1 Experimental Settings

Datasets and Metrics. We conduct comparisons using the scene-scale view synthesis dataset provided by Mip-NeRF360 [64], which comprises nine real-world large-scale scenes, including five unbounded outdoor and four indoor settings with complex backgrounds. In addition, we utilize the Tanks and Temples dataset [65], a comprehensive unbounded dataset, and select the same scenes as used in [16]. Performance metrics on synthetic object-level datasets will be detailed in the supplementary materials. We report metrics including the peak signal-to-noise ratio (PSNR), structural similarity (SSIM), and perceptual similarity as measured by LPIPS [66].

Compared Baselines. We compare our approach with methods suited for large-scale scene modeling, including Plenoxel [12], Mip-NeRF360 [7], and 3D-GS [16]. Additionally, we evaluate

against efficient techniques like Instant-NGP [14], which uses a hash grid for storage efficiency, and VQ-DVGO [49], which extends DVGO [9] with voxel pruning and VQ for optimized representation.

Implementation Details. Our framework is implemented in PyTorch and integrates the differentiable Gaussian rasterization technique from 3D-GS [16]. All performance evaluations are conducted on an A6000 GPU. In the Global Significance Calculation phase, we assign a power value of 0.1 in Eq. 4 and proceed to fine-tune the model for 5,000 steps during the Gaussian Co-adaptation process. For SH distillation, we downscale the 3-degree SHs to 2-degree, thereby reducing 21 elements for each Gaussian. This is further optimized by setting σ to 0.1 in the pseudo view synthesis stage. In the Gaussian VQ step, the codebook size is configured to 8192, selecting SHs with the least 60% significance score for the vector quantization (VQ ratio), to balance the trade-off between compression efficiency and fidelity.

4.2 Experimental Results

Quantitative Results. We assess the performance of various methods for novel view synthesis, with quantitative metrics summarized in Tab. 1. This includes efficient voxel-based NeRFs like Plenoxel [12] and Instant-NGP [14], the compact MLP-based Mip-NeRF360 [7], vector-quantized NeRF [49], and 3D Gaussian Splatting [16].

On the Mip-NeRF360 dataset, MLP-based NeRF methods achieve competitive accuracy with compact representations but suffer from slow inference speeds (0.06 FPS), limiting their practicality. Voxelbased NeRFs improve rendering efficiency but still fall short of real-time performance; for example, Plenoxel requires 2.1GB for a single large-scale scene. In contrast, 3D-GS offers a good balance between quality and speed but demands substantial storage per scene.

Our method, LightGaussian, exceeds existing techniques with rendering speeds over 200 FPS, enabled by efficient rasterization that prunes insignificant Gaussians. This approach reduces 3D Gaussian model redundancy, cutting storage from 782MB to 45MB on Mip-NeRF360-a 15 × reduction. LightGaussian also outperforms 3D-GS on the Tank & Temple datasets, nearly doubling rendering speed and reducing storage from 380MB to 22MB.

Qualitative Results. We conducted a comparative analysis of rendering results between 3D-GS and LightGaussian, focusing on intricate details and background regions, as shown in Fig. 4. Both 3D-GS and LightGaussian exhibit comparable visual quality, even in challenging scenes with thin structures, showing that LightGaussian effectively removes redundancy while preserving reconstruction fidelity.

Generalization to Other Point-based Representations We further apply our Gaussian Pruning approach to Scaffold-GS [67], an advanced neural Gaussian-based 3D representation. In this method, we prune Gaussians that contribute minimally to scene reconstruction, reducing redundancy while preserving visual fidelity. Experiments on the MipNeRF360 dataset demonstrate that pruning 80% of neural Gaussians increases the rendering speed from 152 to 178 FPS, as shown in Tab. 5. These results underscore LightGaussian's potential as an effective optimization tool for other Gaussian-based representations.

4.3 Ablation Studies

We performed ablation studies on each component of our method to evaluate their individual impacts. Results show that Gaussian Pruning and Recovery effectively removes redundant Gaussians, retaining only those crucial for scene representation. SH Distillation reduces the Spherical Harmonics degree, simplifying the lighting model with minimal quality loss. Furthermore, Vector Quantization efficiently compresses the feature space, creating a more compact representation. Combined, these modules substantially improve the overall efficiency and performance of our framework.

Significance Criteria To identify the optimal criteria for measuring each Gaussian's global significance, we evaluated key characteristics: Gaussian-ray interaction count, Gaussian opacity, and functional volume. Tab. 3 illustrates that integrating these three factors - by weighting the hit count with opacity and Gaussian volume - yields the highest rendering quality post zero-shot pruning. A quick Gaussian Co-adaptation (the last row) which optimizes the remaining Gaussians, can effectively recover the rendering accuracy to its pre-pruning level. The visual comparison of rasterized images, before and after pruning, alongside the depiction of pruned Gaussians, is presented in Fig. 5.

SH Distillation & Vector Quantization. Directly removing high-degree components in Spherical Harmonics (SHs) causes significant performance degradation compared to the full model (Exp #3 →

Before Pruning

Rasterized Residual

After Pruning

1.8M(pts)

, 0.847

(SSIM)

5.6M(pts)

, 0.856

(SSIM)

0.7M(pts)

, 0.866

(SSIM)

2.3M(pts)

, 0.863

(SSIM)

4), particularly with the loss of specular reflections across varying viewpoints. However, introducing knowledge distillation from the full model (Exp #4 → 5) allows for size reduction while preserving essential viewing effects. Additionally, incorporating pseudo-views during training (Exp #5 → 6) demonstrates the effectiveness of simulated views in enhancing the learning of specular reflections. While applying VQ to all attributes yields poorer results (see Tab. 4), this impact is mitigated by leveraging Gaussian Global Significance.

Degradation & Speed vs. Compression Ratio We investigate the interplay between rendering quality and speed across various model compression ratios. Specifically, we adjust parameters such

as Gaussian Pruning & Recovery , SH Distilling , and VQ . As depicted in Fig 6, we note a marked decline in rendering quality when the pruning ratio reaches 70%, with a more rapid deterioration as the VQ ratio approaches 65%. In Tab 6, we observe that 2-degree SH compactness reduces the SSIM from 0.927 to 0.926, while 1-degree SH compactness further reduces it to 0.923.

5 Conclusion, Limitations, and Broad Impact

We present LightGaussian, a novel framework that transforms heavy point-based representations into a compact format for efficient novel view synthesis. Designed for practical use, LightGaussian leverages 3D Gaussians to model large-scale scenes, effectively identifying and pruning the least significant Gaussians generated through densification. It achieves over 15 × data reduction, boosts FPS to over 200, and minimally impacts rendering quality. Exploring zero-shot compression across various 3D-GS-based frameworks remains a promising direction for future research.

Broadly, LightGaussian's compact representation has the potential to democratize high-quality 3D content for applications in VR, AR, and autonomous driving by reducing resource demands, enabling more accessible and scalable deployment across industries. While we do not foresee any significant risk from this specific technique, 3D reconstruction may infringe on personal privacy when applied in public spaces or with drone footage, contradicting our ethical intentions.

References

Jianmei Dai, Zhilong Zhang, Shiwen Mao, and Danpu Liu. A view synthesis-based 360° vr caching system over mec-enabled c-ran. IEEE Transactions on Circuits and Systems for Video Technology , 30(10):3843-3855, 2019.

Tinghui Zhou, Richard Tucker, John Flynn, Graham Fyffe, and Noah Snavely. Stereo magnification: Learning view synthesis using multiplane images. arXiv preprint arXiv:1805.09817 , 2018.

Luca De Luigi, Damiano Bolognini, Federico Domeniconi, Daniele De Gregorio, Matteo Poggi, and Luigi Di Stefano. Scannerf: a scalable benchmark for neural radiance fields. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , pages 816-825, 2023.

Zirui Wu, Tianyu Liu, Liyi Luo, Zhide Zhong, Jianteng Chen, Hongmin Xiao, Chao Hou, Haozhe Lou, Yuantao Chen, Runyi Yang, et al. Mars: An instance-aware, modular and realistic simulator for autonomous driving. arXiv preprint arXiv:2307.15058 , 2023.

Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing Scenes As Neural Radiance Fields for View Synthesis. Communications of the ACM , 65(1):99-106, 2021.

Jonathan T Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin-Brualla, and Pratul P Srinivasan. Mip-Nerf: A Multiscale Representation for Anti-Aliasing Neural Radiance Fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 5855-5864, 2021.

Jonathan T. Barron, Ben Mildenhall, Dor Verbin, Pratul P. Srinivasan, and Peter Hedman. Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fields. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 5460-5469, 2022.

Robert A Drebin, Loren Carpenter, and Pat Hanrahan. Volume rendering. ACM Siggraph Computer Graphics , 22(4):65-74, 1988.

Cheng Sun, Min Sun, and Hwann-Tzong Chen. Direct Voxel Grid Optimization: Super-Fast Convergence for Radiance Fields Reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 5459-5469, 2022.

Liao Wang, Jiakai Zhang, Xinhang Liu, Fuqiang Zhao, Yanshun Zhang, Yingliang Zhang, Minye Wu, Jingyi Yu, and Lan Xu. Fourier PlenOctrees for Dynamic Radiance Field Rendering in Real-Time. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 13524-13534, 2022.

Katja Schwarz, Axel Sauer, Michael Niemeyer, Yiyi Liao, and Andreas Geiger. Voxgraf: Fast 3d-Aware Image Synthesis With Sparse Voxel Grids. ArXiv Preprint ArXiv:2206.07695 , 2022.

Alex Yu, Sara Fridovich-Keil, Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angjoo Kanazawa. Plenoxels: Radiance Fields Without Neural Networks. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 5491-5500, 2022.

Anpei Chen, Zexiang Xu, Andreas Geiger, Jingyi Yu, and Hao Su. Tensorf: Tensorial radiance fields. In European Conference on Computer Vision , pages 333-350. Springer, 2022.

Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant Neural Graphics Primitives With a Multiresolution Hash Encoding. ACM Transactions on Graphics (TOG) , 41:1 - 15, 2022.

Huan Wang, Jian Ren, Zeng Huang, Kyle Olszewski, Menglei Chai, Yun Fu, and Sergey Tulyakov. R2l: Distilling neural radiance field to neural light field for efficient novel view synthesis. In European Conference on Computer Vision , pages 612-629. Springer, 2022.

Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics (ToG) , 42(4):114, 2023.

Georgios Kopanas, Julien Philip, Thomas Leimkühler, and George Drettakis. Point-based neural rendering with per-view optimization. In Computer Graphics Forum , volume 40, pages 29-43. Wiley Online Library, 2021.

Alex Yu, Ruilong Li, Matthew Tancik, Hao Li, Ren Ng, and Angjoo Kanazawa. Plenoctrees for real-time rendering of neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 5752-5761, 2021.

Sara Fridovich-Keil, Alex Yu, Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angjoo Kanazawa. Plenoxels: Radiance Fields Without Neural Networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 5501-5510, 2022.

Lingjie Liu, Jiatao Gu, Kyaw Zaw Lin, Tat-Seng Chua, and Christian Theobalt. Neural Sparse Voxel Fields. Advances in Neural Information Processing Systems , 33:15651-15663, 2020.

Tao Hu, Shu Liu, Yilun Chen, Tiancheng Shen, and Jiaya Jia. EfficientNeRF Efficient Neural Radiance Fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 12902-12911, 2022.

Stephan J Garbin, Marek Kowalski, Matthew Johnson, Jamie Shotton, and Julien Valentin. Fastnerf: High-fidelity neural rendering at 200fps. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 14346-14355, 2021.

Peter Hedman, Pratul P Srinivasan, Ben Mildenhall, Jonathan T Barron, and Paul Debevec. Baking neural radiance fields for real-time view synthesis. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 5875-5884, 2021.

Christian Reiser, Songyou Peng, Yiyi Liao, and Andreas Geiger. Kilonerf: Speeding up neural radiance fields with thousands of tiny mlps. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 14335-14345, 2021.

Benjamin Attal, Jia-Bin Huang, Michael Zollhoefer, Johannes Kopf, and Changil Kim. Learning Neural Light Fields With Ray-Space Embedding Networks. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 19787-19797, 2022.

Vincent Sitzmann, Semon Rezchikov, William T. Freeman, Joshua B. Tenenbaum, and Frédo Durand. Light Field Networks: Neural Scene Representations With Single-Evaluation Rendering. Advances in Neural Information Processing Systems , 34, 2021.

Leonid Keselman and Martial Hebert. Flexible techniques for differentiable rendering with 3d gaussians. arXiv preprint arXiv:2308.14737 , 2023.

James F Blinn. A generalization of algebraic surface drawing. ACM transactions on graphics (TOG) , 1(3):235-256, 1982.

Benjamin Eckart, Kihwan Kim, Alejandro Troccoli, Alonzo Kelly, and Jan Kautz. Accelerated generative models for 3d point cloud data. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 5497-5505, 2016.

Manjushree Nulkar and Klaus Mueller. Splatting with shadows. In Volume Graphics 2001: Proceedings of the Joint IEEE TCVG and Eurographics Workshop in Stony Brook, New York, USA, June 21-22, 2001 , pages 35-49. Springer, 2001.

Petr Man. Generating and real-time rendering of clouds. In Central European seminar on computer graphics , volume 1. Citeseer Castá-Papiernicka, Slovakia, 2006.

Christoph Lassner and Michael Zollhofer. Pulsar: Efficient sphere-based neural rendering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1440-1449, 2021.

Matthias Zwicker, Hanspeter Pfister, Jeroen Van Baar, and Markus Gross. Ewa volume splatting. In Proceedings Visualization, 2001. VIS'01. , pages 29-538. IEEE, 2001.

Simon Niedermayr, Josef Stumpfegger, and Rüdiger Westermann. Compressed 3d gaussian splatting for accelerated novel view synthesis. arXiv preprint arXiv:2401.02436 , 2023.

KL Navaneet, Kossar Pourahmadi Meibodi, Soroush Abbasi Koohpayegani, and Hamed Pirsiavash. Compact3d: Compressing gaussian splat radiance field models with vector quantization. arXiv preprint arXiv:2311.18159 , 2023.

Yuanhao Cai, Yixun Liang, Jiahao Wang, Angtian Wang, Yulun Zhang, Xiaokang Yang, Zongwei Zhou, and Alan Yuille. Radiative gaussian splatting for efficient x-ray novel view synthesis. In European Conference on Computer Vision , pages 283-299. Springer, 2025.

Joo Chan Lee, Daniel Rho, Xiangyu Sun, Jong Hwan Ko, and Eunbyung Park. Compact 3d gaussian representation for radiance field. arXiv preprint arXiv:2311.13681 , 2023.

Yann LeCun, John Denker, and Sara Solla. Optimal brain damage. Advances in neural information processing systems , 2, 1989.

Yang He, Guoliang Kang, Xuanyi Dong, Yanwei Fu, and Yi Yang. Soft filter pruning for accelerating deep convolutional neural networks. arXiv preprint arXiv:1808.06866 , 2018.

Sajid Anwar, Kyuyeon Hwang, and Wonyong Sung. Structured pruning of deep convolutional neural networks. ACM Journal on Emerging Technologies in Computing Systems (JETC) , 13(3):1-18, 2017.

Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. arXiv preprint arXiv:1803.03635 , 2018.

Jonathan Frankle, Gintare Karolina Dziugaite, Daniel Roy, and Michael Carbin. Linear mode connectivity and the lottery ticket hypothesis. In International Conference on Machine Learning , pages 3259-3269. PMLR, 2020.

Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems , 30, 2017.

Gabriella Csurka, Christopher Dance, Lixin Fan, Jutta Willamowski, and Cédric Bray. Visual categorization with bags of keypoints. In Workshop on statistical learning in computer vision, ECCV , volume 1, pages 1-2. Prague, 2004.

Robert Gray. Vector quantization. IEEE Assp Magazine , 1(2):4-29, 1984.

Chengzhi Mao, Lu Jiang, Mostafa Dehghani, Carl Vondrick, Rahul Sukthankar, and Irfan Essa. Discrete representations strengthen vision transformer robustness. arXiv preprint arXiv:2111.10493 , 2021.

Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 12873-12883, 2021.

Shuyang Gu, Dong Chen, Jianmin Bao, Fang Wen, Bo Zhang, Dongdong Chen, Lu Yuan, and Baining Guo. Vector quantized diffusion model for text-to-image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10696-10706, 2022.

Lingzhi Li, Zhen Shen, Zhongshu Wang, Li Shen, and Liefeng Bo. Compressing volumetric radiance fields to 1 mb. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 4222-4231, 2023.

Haoyu Guo, Sida Peng, Yunzhi Yan, Linzhan Mou, Yujun Shen, Hujun Bao, and Xiaowei Zhou. Compact neural volumetric video representations with dynamic codebooks. In Thirty-seventh Conference on Neural Information Processing Systems , 2023.

Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531 , 2015.

Zhiqiang Shen, Zechun Liu, Dejia Xu, Zitian Chen, Kwang-Ting Cheng, and Marios Savvides. Is label smoothing truly incompatible with knowledge distillation: An empirical study. arXiv preprint arXiv:2104.00676 , 2021.

Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108 , 2019.

Lin Wang and Kuk-Jin Yoon. Knowledge distillation and student-teacher learning for visual intelligence: A review and new outlooks. IEEE transactions on pattern analysis and machine intelligence , 44(6):3048-3068, 2021.

Cristian Buciluˇ a, Rich Caruana, and Alexandru Niculescu-Mizil. Model compression. In Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining , pages 535-541, 2006.

Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. arXiv preprint arXiv:2209.14988 , 2022.

Dejia Xu, Yifan Jiang, Peihao Wang, Zhiwen Fan, Yi Wang, and Zhangyang Wang. Neurallift360: Lifting an in-the-wild 2d photo to a 3d object with 360 {\ deg } views. arXiv preprint arXiv:2211.16431 , 2022.

Sosuke Kobayashi, Eiichi Matsumoto, and Vincent Sitzmann. Decomposing nerf for editing via feature field distillation. Advances in Neural Information Processing Systems , 35:23311-23330, 2022.

Zhiwen Fan, Peihao Wang, Yifan Jiang, Xinyu Gong, Dejia Xu, and Zhangyang Wang. Nerf-sos: Any-view self-supervised object segmentation on complex scenes. arXiv preprint arXiv:2209.08776 , 2022.

Zhiwen Fan, Yifan Jiang, Peihao Wang, Xinyu Gong, Dejia Xu, and Zhangyang Wang. Unified implicit neural stylization. In European Conference on Computer Vision , pages 636-654. Springer, 2022.

Jiazhong Cen, Zanwei Zhou, Jiemin Fang, Wei Shen, Lingxi Xie, Xiaopeng Zhang, and Qi Tian. Segment anything in 3d with nerfs. arXiv preprint arXiv:2304.12308 , 2023.

Wang Yifan, Felice Serena, Shihao Wu, Cengiz Öztireli, and Olga Sorkine-Hornung. Differentiable surface splatting for point-based geometry processing. ACM Transactions on Graphics (TOG) , 38(6):1-14, 2019.

Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. Advances in neural information processing systems , 28, 2015.

Jonathan T Barron, Ben Mildenhall, Dor Verbin, Pratul P Srinivasan, and Peter Hedman. Mipnerf 360: Unbounded anti-aliased neural radiance fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 5470-5479, 2022.

Arno Knapitsch, Jaesik Park, Qian-Yi Zhou, and Vladlen Koltun. Tanks and temples: Benchmarking large-scale scene reconstruction. ACM Transactions on Graphics (ToG) , 36(4):1-13, 2017.

Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 586-595, 2018.

Tao Lu, Mulin Yu, Linning Xu, Yuanbo Xiangli, Limin Wang, Dahua Lin, and Bo Dai. Scaffoldgs: Structured 3d gaussians for view-adaptive rendering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 20654-20664, 2024.

Appendix

Algorithm Explanation. We detail the procedures of LightGaussian in Algorithm 1. The trained 3D-GS [16] features a Gaussian location with a dimension of 3, Spherical Harmonics coefficients with a dimension of 48, and opacity, rotation, and scale, whose dimensions are 1, 4, and 3, respectively.

Algorithm 1 The overall pipeline of LightGaussian

Specifically, we detail how we calculate each Gaussian's Global Significance Score in Algorithm 2.

Algorithm 2 Global significance calculation for gaussians.

Post-Finetuning of Vector Quantization. Applying vector quantization (VQ) based on the significance score facilitates a reduction in the required codebook size compared to the full model. Nonetheless, this method results in a decrease in accuracy as it groups the attributes into several clusters (codes). We observe that the SSIM decreases from 0.923 (#6) to 0.915 (#7), as illustrated in Table 2 of the main draft. This observation motivates us to joint optimize all other attributes and the codes post the VQ. Specifically, by assigning a unique code to each Gaussian, we proceed to implement differentiable finetuning using photometric loss, which is similar to the methodology

Exp #1

Exp #2

Exp #6

Exp #7

Exp #8

Exp #9

Exp #3

RGB

Residual(SSIM)

RGB

Exp #4

Residual(SSIM)

employed in 3D-GS, and optimize all the attributes in conjunction with VQ. This approach permits precise adjustments to the codebook and the rest attributes, improving its alignment with the training images (#8, SSIM improves from 0.915 to 0.923).

6 More Experiment Results

In addition to realistic indoor and outdoor scenes in the Mip-NeRF360 [7] and Tank and Temple datasets [65], we further evaluate our method on the synthetic Blender dataset [5], and provide a scene-wise evaluation on all datasets, accompanied by detailed visualizations.

Results on NeRF-Synthetic 360°(Blender) Dataset. The synthetic Blender dataset [5] includes eight photo-realistic synthetic objects with ground-truth controlled camera poses and rendered viewpoints (100 for training and 200 for testing). Similar to 3D-GS [16], we start training the model using random initialization. Consequently, we calculate the Global Significance of each Gaussian, work to reduce the SH redundancy, and apply the Vector Quantization (codebook size set at 8192) to the learned representation. Overall comparisons with previous methods are listed in Table 7, where we observe our LightGaussian markedly reduces the average storage size from 52.38MB to 7.89MB, while improving the FPS from 310 to 411 with only a slight rendering quality decrease.

Additional Qualitative Results on Mip-NeRF360. Weprovide extra visualizations for 3D-GS [16], LightGaussian (ours), and VQ-DVGO [49], accompanied by the corresponding residual maps from the

ground truth. As evidenced in Figure 8 and Figure 9, LightGaussian outperforms VQ-DVGO, which utilizes NeRF as a basic representation. Furthermore, LightGaussian achieves a comparable rendering quality to 3D-GS [16], demonstrating the effectiveness of our proposed compact representation.

Additional Quantitative Results on Mip-NeRF360. Tables 8, 9, and 10 present the comprehensive error metrics compiled for our evaluation across all real-world scenes (Mip-NeRF360 and Tank and Temple datasets). Our method not only compresses the average model size from 782MB to 45MB, but also consistently demonstrates comparable metrics with 3D-GS on all scenes. LightGaussian additionally shows better rendering quality than Plenoxel, INGP, mip-NeRF360, and VQ-DVGO.

Implementation Details of VQ-DVGO. In the implementation of VQ-DVGO [49], we initially obtain a non-compressed grid model following the default training configuration of DVGO [9]. The pruning quantile β p is set to 0.001, the keeping quantile β k is set to 0.9999, and the codebook size is configured to 4096. We save the volume density and the non-VQ voxels in the fp16 format without additional quantization. For the joint finetuning process, we have increased the iteration count to 25,000, surpassing the default setting of 10,000 iterations, to maximize the model's capabilities. All other parameters are aligned with those specified in the original VQ-DVGO paper [49], ensuring a faithful replication of established methodologies.

Ground-truth

RGB (3D-GS)

RGB (Ours)

RGB (VQ-DVGO)

Residual Map (VQ-DVGO)

Ground-truth

RGB (3D-GS)

RGB (Ours)

RGB (VQ-DVGO)

Residual Map (VQ-DVGO)

Ground-truth

RGB (3D-GS)

RGB (Ours)

RGB (VQ-DVGO)

Residual Map (VQ-DVGO)

Overall Analysis. We report the performance of the investigation on the proposed modules in Tab. 2 and Fig. 7. We verify that our design of Gaussian Pruning & Recovery is effective in removing redundant Gaussians (Exp #1 → 3) with negligible quality degradation while preserving rendering accuracy. This proves that the proposed global significance, based on principle of splatting, accurately represents the critical aspects of each Gaussian. By removing the high-degree SHs and transferring the knowledge to a compact representation (Exp #3 → 5), our method successfully demonstrates the benefits from using soft targets and extra data from view augmentation, results in negligible changes

Ground-truth

RGB (3D-GS)

RGB (Ours)

RGB (VQ-DVGO)

Residual Map (VQ-DVGO)

Ground-truth

RGB (3D-GS)

RGB (Ours)

RGB (VQ-DVGO)

Residual Map (VQ-DVGO)

Ground-truth

RGB (3D-GS)

RGB (Ours)

RGB (VQ-DVGO)

Residual Map (VQ-DVGO)

Ground-truth

RGB (3D-GS)

RGB (Ours)

RGB (VQ-DVGO)

Residual Map (VQ-DVGO)

in specular reflection. In practical post-processing, the vector quantization on the least important Gaussians (Exp #7), showcases the advantage of adopting VQ to further reduce model size.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We have clearly stated the contributions and scope in the Abstract and Introduction.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We describe the limitations in the Section 5.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: The paper does not include theoretical results.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We have presented the information in Section 4.1.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification: We will release our code after our paper gets accepted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: The training and test details are specified in Section 4.1 Implementation Details.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: The paper does not involved statistical test.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We report the type of resources used in the experiments in Section 4.1.

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research conducted in the paper conform the NeurIPS Code of Ethics.

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: We have discusses the broader impact in Section 5.

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The released model does not have a high risk for misuse.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We cite corresponding papers for the asserts we use in Section 2.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification:

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This work does not involve crowdsourcing nor research with human subjects.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This work does not involve crowdsourcing nor research with human subjects.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nLightGaussian introduces a three-stage technique to efficiently reduce the number of Gaussian primitives. In the first stage, redundant Gaussian primitives are pruned based on global significance, rather than opacity. The second stage involves SH distillation, which utilizes data augmentation from synthesized pseudo views. Finally, the third stage employs Vector Quantization of the SH coefficients. By incorporating these three approaches, LightGaussian achieves a remarkable 15x compression while maintaining competitive rendering quality.\n\nSTRENGTHS:\n1. LightGaussian introduces a novel pruning strategy based on global significance, along with SH Distillation and Vector Quantization techniques, to effectively reduce both primitive redundancy and feature redundancy.\n2. LightGaussian functions as a plugin and can be utilized in any GS representation model.\n3. The authors conducted a comprehensive ablation study to demonstrate the effectiveness of the proposed method.\n4. This method efficiently reduces the number of Gaussian primitives, leading to a significant improvement in rendering speed.\n5. The paper is well-written and easy to follow.\n\nWEAKNESSES:\n1. **Lack of necessary experimental analysis.** Although the authors compare LightGaussian with Compressed 3D-GS and Compact 3D-GS in Table 1, they fail to provide any analysis in the experimental results section, which is quite peculiar. Moreover, most of the experimental analyses focus on NeRF, which seems unnecessary. Furthermore, in Table 1, it is difficult to distinguish the visual quality performance and compressed capacity between LightGaussian and Compressed 3D-GS, and, strangely, there is no qualitative number provided for the FPS of Compressed 3D-GS.\n2. **Lack of novelty.** This method appears to be a post-processing approach of 3D-GS and lacks essential characteristics, unlike Scaffold-GS [1], which proposes a hierarchical 3D Gaussian representation and only stores the information of anchors. It would be beneficial for the authors to compare LightGaussian with Scaffold-GS and its subsequent work, HAC [2], in terms of both experimental results and analysis.\n3. **Poor layout formatting.** In Figure 7, the author incorrectly typeset the paper, leading to a compromised reading experience. By the way, the comparison in Figure 6 is not clearly distinguishable, as the eight images appear to be nearly identical.\n\n[1] Scaffold-GS: Structured 3D Gaussians for View-Adaptive Rendering\n\n[2] HAC: Hash-grid Assisted Context for 3D Gaussian Splatting Compression\n\nQUESTIONS:\n1. Please provide a more detailed analysis of compression GS methods, such as Compressed 3D-GS and Compact 3D-GS, in the experimental section. In addition to comparing FPS, it would be valuable to include a comparison of training times as well.\n2. It would greatly enhance the paper if the authors compare LightGaussian with Scaffold-GS and HAC in the experimental section for a more comprehensive evaluation.\n\nLIMITATIONS:\nThe authors discuss the limitation in Section 5.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-14"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper focuses on compressing 3D Gaussian Splatting (3D-GS) models by mainly focusing on reducing the number of Gaussians and compressing the feature size of Gaussians. With three key steps: gaussian pruning and recovery, spherical harmonics distillation, and vector quantization, the paper achieves ~15x reduction in disk usage and ~2x faster rendering speeds with minimal loss in rendering quality. The effectiveness is verified on Mip-NeRF 360, Tank and Temple, and NeRF synthetic datasets.\n\nSTRENGTHS:\n* Compacting the storage cost and rendering speed of 3DGS with minimal loss of rendering quality is an interesting task and has a lot of applications such as rendering on VR devices.\n* The paper proposed a heuristic metric to evaluate the importance of each Gaussian, to help identify the less important Gaussian which are then pruned out to reduce the storage cost and accelerate the rendering speed.\n* The paper used knowledge distillation after reducing the dimension of the spherical harmonic features to recover the rendering quality.\n* The paper did detailed ablation studies on the three key steps and the sub-steps, plus different design choices and hyperparameters, which clearly shows the effectiveness of each components.\n\nWEAKNESSES:\n* The proposed vector quantization step lacks novelty, for example both [34] and [36] used vector quantization to compress Gaussian attributes, with minor differences in the method. \n* In comparison to the prior 3DGS compression works, the improvement is limited. As shown in Table 1, the improvement on the Mip-NeRF 360 dataset is not clear compared to [34], and the results are worse than [34] on the Tank and Temple dataset in terms of both rendering quality and model size. Besides, why the rendering speeds of [34] are not available?\n* The method requires at least two finetuning steps (Gaussian co-adaptation and VQ finetuning) for the compression, beyond the pre-trained 3DGS model, and each finetuning step requires the same iterations as pre-training. Prior work for example [36] can compress 3DGS in an end-to-end manner, and achieve similar performance. How long does the method take to compress one 3DGS model, and in comparison with the 3DGS compression baselines?\n\nIn conclusion, the paper is not significantly better than the 3DGS compression baselines. The proposed two novel compression methods such as the Gaussian pruning by a heuristic metric and the knowledge distillation do not show considerable improvement over the baselines in terms of rendering quality, rendering speed, and storage cost.\n\nQUESTIONS:\nPlease address the issues I pointed out in the 'weakness' section. Furthermore, I would appreciate clarification on the following questions in the authors' rebuttal.\n* Can authors provide the rendering speeds of [34] which are missing in Table 1?\n* In Table 2 are the reported FPSs for training (finetuning) or testing? If for testing why the FPS changes after some steps that should only affect the training procedure, for example from step [4] to [5]. Are they just noises?\n* In the experiments do you use a universal pruning ratio in the Gaussian pruning step, or is it scene-specific? \n* (Minor) In Algorithm 1 line 12 why use G' rather than G as the teacher? As the rendering quality of G' is slightly worse than G. Is it because G' is faster to render?\n\nLIMITATIONS:\nThe authors provide a discussion of limitations in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors delivered a compact 3D Gaussian representation, i.e., LightGaussian for novel view synthesis. There are three technical contributions. Firstly, the authors present Gaussian Pruning and Recovery that measure the significance of each Gaussian to the view quality and then prune 3D Gaussian that has minimal impact on the visual quality. Secondly, the authors propose Spherical Harmonics (SH) distillation that condenses the information from higher-order coefficients to a more compact form. Thirdly, a Vector Quantization step is presented to reduce the redundancy in spatial and lighting representations.\n\nSTRENGTHS:\ni) The technology in this paper is very comprehensive, including the three most mainstream methods of compressing deep learning models, i.e., pruning, distillation, and quantization. Thus, this paper has great engineering significance and can inspire subsequent works of compressing 3DGS from three points of view.\n\nii) The idea of Gaussian Pruning & Recovery and SH distillation is novel. Computing the score for each 3D Gaussian to represent its significance is fancy and reasonable. The insight of designing Eq. (3) as the score function is interesting. As for the distillation, few people pay attention to the learnable numbers, especially the SH coefficients. Distilling the high-order coefficients with pseudo views also provides a new idea of how to reduce the attributes / learnable parameters of 3D Gaussian.\n\niii) The proposed LightGaussian achieves a better balance between the reconstruction quality, storage usage, and inference speed. To be specific, in table 1, LightGaussian yields improvements of 0.2 dB in PSNR, 81 fps in inference speed, and 6 MB in storage memory than the state-of-the-art method compact 3DGS on the Mip-NeRF 360 datasets. Meanwhile, LightGaussian achieves faster speed and small model size on the Tank and Template datasets with only a 0.2 dB drop.\n\niv) The experiments are sufficient. The results in Table 2 clearly demonstrates the effect of each component. The experiments in Table 3, 4, and 5 study the variants of the three compressing techniques. The authors consider very comprehensive situations in these ablation study experiments.\n\nv) The static webpage and video demo in the supplementary are very attractive and Exquisite, which save a lot of time for readers to understand what the authors did in this work.\n\nWEAKNESSES:\ni) The story line is not coherent and the motivation may need more explanation. In particular, the three technical contributions, i.e., Gaussian Pruning and Recovery, SH Distillation, and Vector Quantization seem like three independent works. Combining them together without clear motivation make this paper like a technical report. Although I appreciate the work the authors have done, it would be better to improve the writing by establishing a coherent story line and adding clear motivation for the proposed methods.\n\nii) It is better to add discussion and fair comparison with previous 3DGS compressing works such as compressed 3D-GS [34] and compact 3D-GS [36]. For example, what is the difference between the compress techniques? And what about the performance with the same baseline model?\n\niii) Some technical details may require more explanation, for example, why designing Eq. (3) like this as the contribution score of each 3DGS? It is interesting to know the process and key insight of the designing process. How you think about this question?\n\niv) For the main results in table 1, why the Compressed 3D-GS does not have the inference speed? And why LightGaussian is better than SOTA on the Mip-NeRF 360 dataset and performs worse on the Tank and Template datasets? It is better to add analysis and discussion here to explain this performance gap.\n\nQUESTIONS:\nI have a question for the presented SH distillation step\n\nSome existing works just use an view-independent component (also low-order SH coefficients) to represent the color of 3D Gaussian [1]. Did you ever try this method? It is interesting to have a discussion between the SH distillation with this option.\n\n[1] Radiative Gaussian Splatting for Efficient X-ray Novel View Synthesis. In ECCV 2024.\n\nLIMITATIONS:\nYes, the authors have analyzed the limitations and broader impact of the method in section 4.4 of the main paper.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis manuscript presents a pipeline to drastically reduce the size of pretrained 3D Gaussian splatting models in a way that preserves novel view image fidelity and increases rendering speed. This pipeline consists of three parts: 1) pruning based on an introduced global significance score followed by fine-tuning, 2) reducing the spherical harmonic dimension of the Gaussians and using knowledge distillation to recover the lost specular information, 3) vector quantization of the spherical harmonics of gaussians below a certain global significance score threshold. The global significance score per Gaussian is accumulated over all pixel rays that that intersect that Gaussian and is the sum of the opacity of the Gaussian multiplied by a normalized volume of the Gaussian. Experimental results show a marked decrease in the model size and increase in rendering speeds while image quality is only slightly degraded.\n\nSTRENGTHS:\nS1: This manuscript is well written and easy to follow. Details are provided that should allow for the ability to replicate experiments.\n\nS2: The current size of 3D-GS models makes them unusable in low resource settings — such as VR/AR — and compressing these models will open up many new advancements in downstream applications. The proposed pipeline demonstrates great performance on real world scenes and will be a benefit to researchers working in resource constrained settings.\n\nWEAKNESSES:\n**Main weaknesses:**\n\nW1: I’m surprised that contribution per pixel ray is sufficient for an effective the global significance score. It seems like a per Gaussian score would need to account for both the Gaussian’s contribution to the input image pixels AND the location of those pixels in 3D space. A Gaussian observed by multiple cameras should hold more importance as it will be more likely to faithfully reconstruct the 3D scene than a Gaussian only observed by a single camera.\n\nW2: I’m also surprised that the global significance score doesn’t need to account for the exponential drop in contribution to the pixel color due to alpha blending from Gaussians in front of the Gaussian whose score is being computed.\n\nW3: Does the global significance score get updated for all Gaussians along a pixel’s ray or is it updated for only those Gaussians that contribute to the pixel color? I suspect it is the latter, but this should be clearly stated in the manuscript.\n\nW4: It’s surprising to me that recovering the higher degree spherical harmonic representation in the lower spherical harmonic degrees via knowledge distillation and sampled views is needed to maintain high image quality after spherical harmonic reduction.  I’d expect that recovering the color via fine-tuning in the just the training views should work quite well and knowledge distillation and sampled views should be ablated against this.\n\n\n**Cosmetic issues:**\n\nEq. 4: Volume should always be positive, so there shouldn’t be a need to take the max of normalized volume and 0.\n\nLine 200: “network” should be “model”, 3D-GS models aren’t networks.\n\nQUESTIONS:\nI’m rating this manuscript as “Accept”. I encourage the authors to engage with the weaknesses I’ve listed and will consider raising my score if they are addressed in the rebuttal.\n\nLIMITATIONS:\nThis manuscript has adequately covered the limitations of the proposed pipeline.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-14"}