Lightweight Predictive 3D Gaussian Splats

## Abstract

Abstract. Recent approaches representing 3D objects and scenes using Gaussian splats show increased rendering speed across a variety of platforms and devices.
While rendering such representations is indeed extremely efficient, storing and transmitting them is often prohibitively expensive.
To represent large-scale scenes, one often needs to store millions of 3D Gaussians, occupying gigabytes of disk space.
This poses a very practical limitation, prohibiting widespread adoption.
Several solutions have been proposed to strike a balance between disk size and rendering quality, noticeably reducing the visual quality.
In this work, we propose a new representation that dramatically reduces the hard drive footprint while featuring similar or improved quality when compared to the standard 3D Gaussian splats. When compared to other compact solutions, ours offers higher quality renderings with significantly reduced storage, being able to efficiently run on a mobile device in real-time 1 1 1 More demo examples in our webpage: https://github.com/plumpuddings/LPGS . Our key observation is that nearby points in the scene can share similar representations.
Hence, only a small ratio of 3D points needs to be stored.
We introduce an approach to identify such points—called parent points. The discarded points— children points—along with attributes can be efficiently predicted by tiny MLPs.

## 1. Introduction

Gaussian Splatting (3DGS)-based methods are taking the graphics and vision communities by a storm [Luiten et al., ; Wu et al., ; Yang et al., ]. They promise to strike the right balance between high-fidelity rendering, fast convergence, and efficient inference [Kerbl et al., ]. The latter two benefits make 3DGS-based methods superior to Neural Radiance Fields (NeRFs)-based techniques [Mildenhall et al., ; Martin-Brualla et al., ; Barron et al., ]. Indeed, while NeRFs [Barron et al., ] show high-fidelity renderings too, apart from several exceptions [Cao et al., ; Wang et al., ; Chen et al., ; Müller et al., ], their training and inference time is often prohibiting real-time and edge-based applications.

3DGS-based approaches represent a 3D scene using an explicit, point-based representation [Aliev et al., ]. Geometry and color are stored as millions of 3D Gaussians. The 3D Gaussians are efficiently rasterized to 2D images, with much faster rendering than neural volumetric rendering approaches [Kerbl et al., ]. However, to represent sophisticated geometry and texture, especially for large-scale scenes, a significant amount of points along with their attributes need to be stored, amounting to gigabytes of storage.

In a world of connected devices, real-time experiences and applications, this storage requirement imposes a heavy toll on the hard-drive and the transmission bandwidth. Hence, several initial solutions have been proposed to reduce the storage for 3DGS, such as incorporating a sparse voxel grid [Lu et al., ] or applying more aggressive pruning of the 3D points [Fan et al., ; Lee et al., ].
Yet, existing studies still suffer either from large storage requirements [Lu et al., ] or inferior rendering quality compared to 3DGS [Fan et al., ; Lee et al., ].

In this work, we introduce a lightweight Gaussian Splat representation, with storage significantly reduced, and featuring superior rendering quality. Empowered with our approach, a practitioner does not have to compromise the quality. The key to our approach is a realization that not all points are equally important for rendering (also discussed in [Lu et al., ]). In fact, only a fraction of points need to be stored.
We refer to such points as *parent* points. The remaining points, named as *children* points, do not have to be stored, but can be *predicted* instead.
Besides, we show that even the Gaussian attributes can be estimated during rendering.

Fig. 1 shows the Garden scene [Barron et al., ] reconstructed by the standard Gaussian Splats [Kerbl et al., ], Compact GS [Lee et al., ], Light GS [Fan et al., ] and the proposed approach. First, we observe a significantly reduced density of points in the point cloud reconstructed by our approach. This, and the predicting of the attributes instead of storing them, significantly reduces the storage requirement for our method. Second, we show improved PSNR scores and visual quality, when we zoom-in into the details of the rendered images.

To identify the *parent* points, we first allow every point to be considered as a *parent* point, and be used to predict its *children* points.
We allow children nodes to be promoted to new parents when required.
If found unnecessary during optimization, we can delete *parent* points together with all their *children* points. Intuitively, this allows certain regions with sophisticated geometry to contain more points for accurate modelling. Therefore, starting with points obtained using SfM [Schonberger and Frahm, ], our method learns a compact set of *parent* points. *Children* points do not have to be stored and can be predicted instead. We use a 3D hash grid [Müller et al., ] to encode the offsets used to estimate the 3D locations of such points. Besides that, in the same hash grid, we store the features necessary for predicting Gaussian attributes for each point. Intuitively, nearby points represent similar geometry and texture and, thus, could be used when predicting the attributes. To do so, we introduce an attention-based mechanism attending to both parent and child points. Prediction is performed using shallow 2-layer fully-connected networks, adding a negligible overhead to the process.

Equipped with the above contributions, to represent large-scale real-world scenes, our method requires dramatically less storage, while having comparable or improved quality metrics. Compared with the original 3DGS [Kerbl et al., ], we show up to $20\times$ reduction on average in hard-drive footprint, with improved PSNR and comparable SSIM and LPIPS. Compared with current voxel grid-based approach [Lu et al., ], our method provides $5\times$ storage reduction, while showing improved metrics on the dataset introduced in [Barron et al., ], and $2\times$ reduction and comparable metrics on Tanks&Temples [Knapitsch et al., ] (see Tab. 1). Besides, our method offers a reader a practical means to satisfy the storage requirements of their task at a hand. In Fig. 3 we show several configurations depending on the total hard-drive footprint. Our smallest configuration, which is 2$\times$ smaller than the smallest prior work [Fan et al., ], shows similar scores. Our largest model, which is still smaller than all other works, reaches higher quality. Due to smaller storage requirements, efficient rendering, and superior quality our method can unlock wide adoption of GS-based applications on resource-constrained devices, such as mobiles.

## 2. Related Work

Novel View Synthesis.
Research on rendering scenes from unseen viewpoints with photorealism has evolved over several decades [Greene, ; Chen and Williams, ; Levoy and Hanrahan, ; Buehler et al., ; Srinivasan et al., ]. Traditional approaches typically rely on explicit depth estimation to warp pixels for generating novel views [Kalantari et al., ; Penner and Zhang, ; Choi et al., ; Riegler and Koltun, ]. However, the accuracy of depth estimation algorithms is critical, and handling disocclusions during rendering adds complexity. An alternative approach involves Multiplane Images (MPI) [Zhou et al., ; Srinivasan et al., ; Flynn et al., ], which learn a representation associating objects within the scene with fronto-parallel layers. This structured representation facilitates efficient rendering from different viewpoints while preserving depth relationships and occlusions. More recently, Neural Radiance Fields (NeRF) [Mildenhall et al., ] have gained popularity for their ability to achieve highly realistic rendering, even in scenarios involving complex view-dependent lighting effects such as transparency and reflectance. However, the weakness of NeRF lies in its volumetric rendering formulation, which necessitates sampling a large number of points per ray to render a single pixel. This high computational cost limits the usage of NeRF for real-time or on-device applications. While efforts to reduce computational requirements for volumetric rendering have been a focus of recent research [Liu et al., ; Neff et al., ; Garbin et al., ; Reiser et al., ; Lindell et al., ; Yu et al., ; Müller et al., ; Fridovich-Keil et al., ; Lombardi et al., ; Cao et al., ; Gupta et al., ], point-based rendering, particularly 3D Gaussian Splatting (3DGS) [Kerbl et al., ],
presents another promising direction for real-time view synthesis.

Point-based Rendering.
The basic form of point-based rendering rasterizes an unstructured set of points and utilizes graphics APIs for rendering [bot, ; Ren et al., ], providing speed and editing flexibility, yet often resulting in artifacts such as holes and outliers. To address this, researchers have explored learnable point-based representations. One approach is to associate neural features with each point, rendering images using CNNs with rasterized feature maps as input [Rakhimov et al., ; Aliev et al., ; Kopanas et al., ; Feng et al., ]. Another relevant approach is to learn parameters of point primitives such as ellipsoids, or surfels [Wiles et al., ; Yifan et al., ; Gross and Pfister, ; Insafutdinov and Dosovitskiy, ; Lin et al., ; Kerbl et al., ]. Notably, 3D Gaussian Splatting [Kerbl et al., ] learns anisotropic 3D Gaussians, treating them as volumes of radiance and splatting them to 2D images using alpha-blending. This technique achieves higher-quality results compared to NeRF-based methods and can render high-resolution images in real-time. One drawback of 3D Gaussian splatting and many point-based methods is their inefficiency in terms of storage complexity. This is due to their unstructured representation, which ignores spatial correlations between points. Consequently, they require storing a large number of points to represent complex scenes, often numbering in the millions. Therefore, reducing the consumption of storage in 3D Gaussian splatting has become an emerging research topic.

Efficient Representation for 3D Gaussian Splatting.
There are two main directions to improve the efficiency of 3D Gaussian splatting representations: reducing the number of points and compressing storage for point-wise attributes (*e.g*., spherical harmonics and geometric properties). We find three concurrent works which are most related to ours. LightGS [Fan et al., ] introduces a point pruning and recovery process to minimize redundancy in Gaussian splats, utilizes distillation and pseudo-view augmentation to distill spherical harmonics to a lower degree, and employs quantization to further reduce storage. While LightGS achieves considerable storage reduction, it results in noticeable fidelity degradation compared to the original Gaussian splatting due to quantization. CompactGS [Lee et al., ] proposes using a grid-based neural field to implicitly represent view-dependent colors rather than explicitly storing spherical harmonics per point, offering promising storage efficiency without significant fidelity loss. ScaffoldGS [Lu et al., ] suggests distributing local splats using anchor points, re-parameterizing point positions relative to these anchors to enable anchor-based point growing and pruning strategies for redundancy reduction in 3DGS.
Our method shares similarities with CompactGS [Lee et al., ] and ScaffoldGS [Lu et al., ] while exhibits crucial *differences*. *First*, unlike CompactGS, we utilize a combination of neural fields and self-attention layers to predict not only view-dependent colors but also geometric properties. *Second*, in contrast to both approaches which explicitly store the position of every point in the point cloud, our method only stores a small subset of points, referred to as *parent* points, while predicting the remaining points on-the-fly during rendering. This substantially reduces memory footprint. *Third*, we introduce a new point growing and pruning strategy that allows our approach to achieve high-fidelity rendering with fewer points.

Figure: Figure 2. *Top*: We use a *parent* node to estimate its *children* nodes and the Gaussian attributes. The *parent* node retrieves a pair of features, used for attributes $f_{\mathrm{a}}$ and displacement ($f_{\mathrm{\Delta}}$) prediction, from the feature grid. The displacement features $f_{\mathrm{\Delta}}$ are used to estimate the positions of *children* nodes. To estimate the Gaussian attributes, such as scale, rotation, color, and opacity, attribute features $f_{\mathrm{a}}$ are aggregated with self-attention. *Bottom Left*: The process of querying the features. *Bottom Right*: A visualization of *parent* nodes and their predicted *children* nodes.
Refer to caption: x2.png

## 3. Preliminaries

3D Gaussian Splatting (3DGS) [Kerbl et al., ] represents a scene with 3D points x. The points are initialized with a coarse point cloud obtained using Structure-from-Motion (SfM) [Schonberger and Frahm, ].
These Gaussians, $G(\textbf{x})$, serve as the anisotropic volumetric splats defined by their position (mean $\mu$) and 3D covariance ($\Sigma$) as
$G\left(\textbf{x}\right)=e^{-\frac{1}{2}\left(\textbf{x}-\mu\right)^{T}\Sigma^
{-1}\left(\textbf{x}-\mu\right)}$.

To ensure $\Sigma$ remaining positive semi-definite during optimization, it is represented with an equivalent yet effective formulation with the scaling matrix $S$ and the rotation matrix $R$, such that $\Sigma=RSS^{T}R^{T}$.
The attributes of the 3D splats (*e.g*., location, covariance, and opacity) together with the directional appearance of the radiance filed, represented via the spherical harmonics (SH) [Sara Fridovich-Keil and Alex Yu et al., ], are end-to-end learned using optimization. During the optimization the number of Gaussians is changed via cloning, splitting, and pruning operations.

To render an image, 3D $G(\textbf{x})$ are first transformed into 2D Gaussians (denoted as $G^{\prime}(\textbf{x})$) [Zwicker et al., ].
3DGS uses an efficient tiled-based rasterizer that presorts primitives for the entire image, allowing fast $\alpha$-blending of anisotropic splats. The color $C$ of a pixel is computed by blending $N$ 2D Gaussians that overlap at the pixel as: $C=\sum_{i\in N}c_{i}\alpha_{i}G^{\prime}_{i}(\textbf{x})\prod_{j=1}^{i-1}(1-
\alpha_{j}G^{\prime}_{j}(\textbf{x})),$
where $c_{i}$ represents view-dependent colors for each splat, $\alpha_{i}$ is the opacity.
Thanks to the highly optimized rasterizer for modern GPUs, 3DGS render high-fidelity scenes in real-time across a number of platforms.

These benefits come with a cost. 3DGS require a significant number of 3D Gaussians, sometimes needing gigabytes for complex large-scale scenes. This requirement limits their application on edge devices, as downloading gigabytes over the network and storing them is hardly feasible or practical for edge devices at scale.

## 4. Method

We show a high-level overview of our approach in Fig. 2. Our key motivation is that particular splats carry greater importance than others, due to their position, opacity, scale, *etc*. [Lu et al., ]. We show that these important splats can be used to derive the attributes—position, color, scale, *etc*.—of other splats using a small neural network. This allows us to store only the important splats along with the weights of the neural network. To do so, we represent a scene using a forest of a depth-$1$ tree structures, where we use *parent* nodes to represent the important splats using which we predict $K$ *children* nodes.
Formally, we represent a scene using $\mathcal{S}=\{\mathcal{X}_{1},\mathcal{X}_{2},\dots\mathcal{X}_{n}\}$, where $\mathcal{X}_{i}$ is tree and each node contains the attributes, such as position ($x$), color ($c$), opacity ($\alpha$), scale ($s$), and rotation ($r$). This representation can be stored very efficiently, as for each tree we need to save only the positions and scales of *parent* nodes and small neural network shared across the trees, to predict all the other attributes of the tree.

### 4.1. Neural Representation for Lightweight Predictive Splats

We design the architecture to model the close relationship between a parent and children nodes. In particular, we assume that the children nodes are in the vicinity of parent node and have similar physical attributes such as shape, color and opacity. We satisfy these requirements by using a hash-grid based approach [Müller et al., ; Chen et al., ] as our representation. Hash-grids have an inherent property to return similar features when queried with the points located nearby. Below we describe in detail how a tree ($\mathcal{X}_{i}$) can be represented in storage efficient manner. In what follows we drop the index $i$.

Given a hash-grid $\mathcal{H}(\cdot)$ shared across the trees and parent node positions $x_{p}$, we query the features as $f=\mathcal{H}({x}_{p})$ and use them to predict all the attributes of the tree. Specifically, we divide $f$ into two halves $f\equiv\{f_{\Delta}\in\mathbb{R}^{D/2},f_{a}\in\mathbb{R}^{D/2}\}$, where the first half ($f_{\Delta}$) represents displacement and is used to predict the position of children. The second half ($f_{a}$) is used to predict rest of the attributes.

Predicting Position.
We want children and parent nodes to represent similar geometry and appearance. Hence, children should be located in the vicinity of the parents nodes. We model the position of children as their displacement from their parent nodes. For the parent we predict the position of $k^{\mathrm{th}}$ child using $x_{k}=x_{p}+g_{\mathrm{pos}}(f_{\mathrm{\Delta}})[k]$ where $g_{\mathrm{pos}}$ is an MLP with output shape $K\times 3$.

Having the positions of all nodes in the tree, we can predict the rest of the attributes, such as scale, rotation, color, and opacity. We reuse the hash-grid to get the attribute feature (${f_{\mathrm{a}}}_{k}$) for $k^{\mathrm{th}}$ child node using $\mathcal{H}(x_{k})$. A naive approach to extract the remaining attributes using $f_{\mathrm{a}}$ and ${f_{\mathrm{a}}}_{k}$ is to pass the latter to an MLP get scale, rotation, color and opacity. We found such approach to be sub optimal. A hash-grid representation implicitly makes the representation of spatially points similar. There is no mechanism to share information between the features after they are computed. Since there is relation between physical attributes of the parent and children nodes, having such information sharing mechanism is beneficial. To this end, we propose a modified self-attention mechanism to better capture the inter-dependencies between children and parent nodes. To do so, we first obtain the aggregated feature $\mathcal{F}_{\mathrm{a}}$ $\in\mathbb{R}^{K+1\times D/2}$ by concatenating features of all the nodes in the tree, such as $\mathcal{F}_{\mathrm{a}}=\texttt{Concat}(\{f_{\mathrm{a}},({f_{\mathrm{a}}}_{1
},\dots,{f_{\mathrm{a}}}_{K}\})$, where Concat is a concatenation operation. We then apply a modified self-attention operation on $\mathcal{F}_{\mathrm{a}}$ to get the final feature $\mathcal{F}^{\prime}_{\mathrm{a}}$:

$$ (1) $\displaystyle\mathcal{F}^{\prime}_{\mathrm{a}}=\mathcal{F}_{\mathrm{a}}+ \lambda\sigma(\frac{\mathcal{P}_{1}(\mathcal{F}_{\mathrm{a}})*\mathcal{P}_{2}( \mathcal{F}_{\mathrm{a}})^{T}}{\sqrt{d}})*\mathcal{F}_{\mathrm{a}},$ $$

where $\sigma(\cdot)$ is a Softmax function, $\mathcal{P}_{i}(\cdot)$ is a projection matrix, $d$ is a scaling factor set as $D/2$, $\lambda$ is a hyper-parameter for balancing the information trade-off from the attention mechanism and $*$ denotes the matrix multiplication. In contrast to vanilla attention mechanism [Vaswani et al., ], we do not apply positional embedding, so that Eq. 1 is permutation invariant which is an important property to maintain while working with point clouds [Qi et al., ].
Further, we use the unprojected $\mathcal{F}_{\mathrm{a}}$ when multiplying with $\sigma(\cdot)$, since we empirically found no performance gain by projecting $\mathcal{F}_{\mathrm{a}}$.
Next, we split $\mathcal{F}^{\prime}_{\mathrm{a}}$ in $K+1$ attribute feature vector to predict the remaining attributes for each node in the tree.

Predicting Scale and Rotation: It is vital to properly initialize the scale of Gaussians for stable training. For instance, Gaussians with small scales make minimal contributions to the rendering quality, mainly because of their limited volume.
In contrast, large Gaussians can potentially contribute to every pixel during rasterization, leading to a significant amount of GPU memory. Hence, to make training stable and minimize storage needs at the same time, we adopt a middle-ground strategy. More specifically, we represent the scales of children as a scaled version of their parents ($s_{\mathrm{p}}$): $s_{k}=\hat{s}_{k}\ s_{p}$ where $\hat{s}_{k}$ is predicted by an MLP. In case of rotation, we directly regress it for both parents and children nodes using the corresponding attribute feature vector.
We share the weights of the MLP to regress both scale and rotation.
We experimentally found, that including position of node ($x_{k}$), the distance of the point to the center of the axis aligned bounding box ($b_{k}$) along with attribute feature ((${f^{\prime}_{\mathrm{a}}}_{k}$) improves performance: $\hat{s}_{k},r_{k}=g_{rs}({f^{\prime}_{\mathrm{a}}}_{k},x_{k},b_{k})$.

Predicting Color and Opacity: 3DGS uses degree-$3$ spherical harmonics (SH) for view-dependent color representation [Kerbl et al., ]. However, we find such design is unnecessary and the color can be directly predicted using from feature vectors and a viewing direction. We use an MLP that takes them as an input and directly predicts the color as output, $c_{k}=g_{c}({f^{\prime}_{\mathrm{a}}}_{k},d_{k})$ where $d_{k}$ is the viewing direction of the node in the tree. This helps in reducing the storage by a significant amount as previously each splat had to store the spherical harmonics individually.
To predict opacity, we use another MLP with inputs as ${f^{\prime}_{\mathrm{a}}}_{k}$ and the position of the node to get corresponding opacity, $o_{k}=g_{o}({f^{\prime}_{\mathrm{a}}}_{k},x_{k})$.

We described all the operations above for a single tree. The same operation is extended for all the trees. Further, the neural networks for all the operations share their weights across all the trees.
To summarize, the proposed representation can efficiently represent the tree structure using hash-grid based neural representations $\mathcal{H}(\cdot)$ and a few MLPs. We only need to store position and scale of parent nodes and the weights of our neural networks, while the rest of the properties of the tree is regressed as described above.

### 4.2. Adaptive Tree Manipulation

3DGS starts by using an initial point cloud from SfM [Schonberger and Frahm, ]. To allow for some flexibility in the point cloud structure they propose several techniques to add and delete points during optimization. The techniques are based on position gradient, size, and opacity of the gaussians.
Extending these techniques to our depth-$1$ tree structure of is not straightforward. Treating parent nodes as individual points and following 3DGS can lead to sub-optimal or incorrect results (see Tab. 2).
In the proposed depth-$1$ tree representation both parents and children impact the rendering quality. Hence, to allow for flexibility one needs to incorporate children nodes, when adding or deleting trees in the scene.

Along with maintaining the gradient of position of parent nodes we also track them for the children nodes. When the gradient of a child node is above a certain threshold $\tau^{c}_{\mathrm{pos}}$ we split the tree and promote the child node to become a new parent. This is crucial to represent complicated regions in the scene where there might not be many parent nodes. Once the child has been promoted to parent node we apply clone and split operations to all the parents following similar practices in 3DGS. This gives flexibility to children nodes to add trees and help in the densification of region if necessary.

To delete an entire tree we only rely on the statistics of parent node. This is because if a child node was important then it would have been already promoted to become a new parent. Hence, we can safely delete the current parent that will in turn delete all the corresponding children nodes. Specifically, for deleting the trees we track the scale and opacity of the parent nodes and delete them if they are below a certain threshold similar to 3DGS.

### 4.3. Training

Our model, including the hash-grid and MLPs, is *end-to-end* learnable. These components are guided by the $\mathcal{L}_{1}$ loss between the rendered images and ground-truth images along with a D-SSIM loss, such that:

$$ (2) $\displaystyle\mathcal{L}$ $\displaystyle=(1-\beta)\mathcal{L}_{1}+\beta\mathcal{L}_{\mathrm{D-SSIM}},$ $$

where $\beta$ is set as $0.2$ following the setting in  [Kerbl et al., ].

We use a warm-up training scheme that helps in convergence of the model [Kerbl et al., ]. The warm-up consists of training the model in a low resolution setting, eventually moving to higher resolution after a certain number of steps have been completed. We found that the warm-up strategy is crucial to correctly position the splats and densify the regions.
Without the warm-up, the model struggles to populate enough splats in the background area, despite the importance of the area resulting in substandard performance. Please refer to *Supplementary* material for details.

## 5. Experiments

Our aim is to build an efficient GS representation, with low storage requirements, high-fidelity rendering and real-time execution. These features are particularly important for resource constrained devices such as mobile phone, as transferring large volumes of data across the cell network and storing it locally is not practical.

We provide extensive experiments to evaluate and validate the contributions introduced in the previous section.

Implementation Details.
We use Instant-NGP [Müller et al., ] as our hash-grid owing to its compact and efficient design and 2 layers MLP for all the MLPs. Following the practices in [Barron et al., ; Müller et al., ] we use scene contraction to map the position into $[0,1]$ before feeding it to the Instant-NGP. This helps bring the splats that are occasionally outside the Axis-Aligned-Bounding-Box (AABB) due to the densification of splatsalong with the position updates. We estimate the AABB with the initial COLMAP [Schönberger and Frahm, ] point cloud. This algorithms of AABB estimation and contraction are included in *supplementary*.
We set $\lambda=0.5$ for all the experiments and train the model for $30K$ steps where initial $7.5K$ steps is warm-up stage. The number of *children* splats
($K$) used in our experiment varies across scenes, with at most 2 children. Please refer to *Supplementary* material for details.

**Table 1. Quantitative comparisons of our approach and other works evaluated on three widely used benchmark datasets, including Mip-NeRF 360^∘ dataset [Barron et al., ], Tanks&Temples [Knapitsch et al., ], and Deep Blending [Hedman et al., ]. We report the image quality metrics, such as PSNR, SSIM, and LPIPS, and the required storage. We also report the on-device capability of each Gaussian Splatting based work (On-Device in the table), where OOM denotes Out-of-Memory error and ✓denotes the real-time capability (¿$30$ fps) on our tested device, *i.e*., iPhone14. The evaluation results on other works are obtained from their papers. Compared with the methods that are capable to run on mobile devices, our models (Ours-C1, C2, C3) can obtain smaller model size with higher rendering quality (*i.e*., PSNR).**
| Method | On-Device | Mip-NeRF $360^{\circ}$ Dataset | Tank&Temples | Deep Blending |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  |  | PSNR $\uparrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$ | Storage $\downarrow$ | PSNR $\uparrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$ | Storage $\downarrow$ | PSNR $\uparrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$ | Storage $\downarrow$ |
| Mip-NeRF 360^∘ | - | 29.23 | 0.844 | 0.207 | 8.6MB | 22.22 | 0.759 | 0.257 | 8.6MB | 29.40 | 0.901 | 0.245 | 8.6MB |
| iNGP | - | 26.43 | 0.725 | 0.339 | 48MB | 21.72 | 0.723 | 0.330 | 48MB | 23.62 | 0.797 | 0.423 | 48MB |
| Plenoxels | - | 23.62 | 0.670 | 0.443 | 2100MB | 21.08 | 0.719 | 0.379 | 2300MB | 23.06 | 0.795 | 0.510 | 2700MB |
| ScaffoldGS | OOM | 29.02 | 0.848 | 0.220 | 156MB | 23.96 | 0.853 | 0.177 | 87MB | 30.21 | 0.906 | 0.254 | 66MB |
| 3DGS | OOM | 28.69 | 0.870 | 0.182 | 693MB | 23.14 | 0.841 | 0.183 | 411MB | 29.41 | 0.903 | 0.243 | 676MB |
| LightGS | ✓ | 28.45 | 0.857 | 0.210 | 42.48MB | 22.83 | 0.807 | 0.242 | 22.43MB | - | - | - | - |
| CompactGS | ✓ | 28.60 | 0.855 | 0.211 | 46.98MB | 23.32 | 0.831 | 0.201 | 39.43MB | 29.79 | 0.901 | 0.258 | 43.21MB |
| Ours-C1 | ✓ | 28.45 | 0.837 | 0.235 | 23.40 MB | 23.19 | 0.810 | 0.239 | 22.00 MB | 29.32 | 0.895 | 0.282 | 22.90MB |
| Ours-C2 | ✓ | 28.86 | 0.851 | 0.217 | 29.50 MB | 23.47 | 0.820 | 0.228 | 29.05MB | 29.61 | 0.896 | 0.277 | 29.15MB |
| Ours-C3 | ✓ | 29.11 | 0.857 | 0.210 | 35.60MB | 23.82 | 0.829 | 0.210 | 35.32MB | 29.89 | 0.902 | 0.267 | 35.40MB |

Dataset and Metrics We evaluate our method using *seven* scenes from the Mip-NeRF $360^{\circ}$ dataset [Barron et al., ], *two* scenes from Tank&Temples [Knapitsch et al., ], and *two* scenes from Deep Blending [Hedman et al., ]. We use the widely adopted metrics like PSNR, SSIM [Wang et al., ], and LPIPS [Zhang et al., ] to assess the quality for image reconstruction. We also report the storage size (in MB) for various methods along with their on-device capabilities. We benchmark the Gaussian Splatting based methods on iPhone 14 with our implementation of the mobile application.
We report three configurations of our method named C1, C2, and C3 by varying feature dimension $D$ of the hash-grid $\mathcal{H}$. C1 is our smallest model with $D=32$ followed by C2 with $D=48$ and C3 is largest with $D=64$.Since our framework adds and removes points during optimization, the final storage for each model can vary. For each dataset we report the average size of all the scenes within one configuration. The metrics, too, are averaged over all scenes of each dataset. Per-scene quantitative results are in the *Supplementary* material.

Figure: Figure 3. We plot the PSNR score of several configurations of our method and prior works computed over the dataset introduced by [Barron et al., ].
Refer to caption: x3.png

### 5.1. Comparison Results

Figure: Figure 4. Visual comparisons with methods offering efficient GS representations ([Lee et al., ; Fan et al., ]). We magnified regions to show qualitative differences. Our approach (C3) can render images with high-quality while greatly saving the storage. Zoom-in for greater detail.
Refer to caption: x4.png

Quality vs Storage. First, we show that our approach provides a practical means of satisfying diverse technical requirements. We can reduce or increase the feature dimension of the hash-grid and the number of points, while still maintaining similar or superior rendering quality. In Fig. 3, we plot PSNR, evaluated on the dataset introduced by [Barron et al., ], for contemporary models as well as for three configurations of our approach. Our smallest configuration is almost 50% smaller than the smallest prior work (LightGS [Fan et al., ]), and shows the same rendering quality. Our largest configuration, which is still 32% smaller than the smallest existing work, shows significantly increased PSNR. To give the reader a better perspective, we also plot conventional works with large hard-drive footprint [Lu et al., ; Kerbl et al., ]. Our largest configuration, which uses only 20% of ScaffoldGS [Lu et al., ] and only 4.5% of 3DGS [Kerbl et al., ] storage, shows higher quality than both of these much larger works.
These advantages of our method are crucial for mobile deployment.
Less disk storage also helps in speeding up transmission that significantly impacts user experience when sharing content.

Figure: Figure 5. Visual comparison of model trained with and without ATM. We can see that model trained without ATM fails to model intricate details in the scene.
Refer to caption: x5.png

Quantitative Results.
Tab. 1 shows the quantitative results performed on real-world scenes, spanning from large-scale urban landscapes to intricate indoor and outdoor environments.
We compare our approach against NeRF-based methods, 3DGS [Kerbl et al., ], and concurrent works (*i.e*., LightGS [Fan et al., ], CompactGS [Lee et al., ], and ScaffoldGS [Lu et al., ]).
On the Mip-NeRF 360^∘ dataset, our approach achieves the *best* PSNR among all the approaches. Compared with 3DGS [Kerbl et al., ], we obtain a significant storage *reduction*, *i.e*., $19.5\times$, and require $3.5\times$ *fewer* 3D points.
On the Tank&Temples [Knapitsch et al., ] dataset, although ScaffoldGS [Lu et al., ] has better PSNR than our approach, our storage is almost $2.4\times$ *smaller* than ScaffoldGS.

Compared with 3DGS [Kerbl et al., ] on this dataset, we require
$1.9\times$ *fewer* 3D points and $11.3\times$ *less* storage.
Lastly, on the Deep Blending [Hedman et al., ] dataset, our method has *higher* PSNR and $19\times$ storage *reduction* than 3DGS [Kerbl et al., ].

It is worth noting that there always exist a configuration of our method where we achieve smallest size and best PSNR when compared with recent works [Lee et al., ; Fan et al., ] on all the datasets.

Qualitative Results.
Fig. 4 demonstrates the high-quality rendering of our method produced using C3 configuration across $5$ example scenes covering all the datasets. We see various examples where our method outperforms previous compression works. We can see our models can better capture background details (row 3, 5), better capture reflections (row 2) while being the smallest or of comparable size. It can also capture intricate details where other methods fail such as ceilings (row 1, 4). We encourage the reader to check out the webpage for more results.

On-Device Capability.
We explore the feasibility of running splatting based methods on mobile devices. We use iPhone14 as the platform to implement a testing application. For fair comparison, we unpack splats from all methods to a standard 3DGS format [Kerbl et al., ] for rendering. We observe Out-of-Memory error when running all scenes from the three benchmark datasets for 3DGS and ScaffoldGS, owing to their large number of splats. Our method can successfully run on device, and achieves smaller and better rendering quality compared to LightGS and CompactGS.

**Table 2. Analysis of the key components in our framework. *Full* denotes the full pipeline. We report the PSNR results for different settings.**
|  | FE | w/o Attn. | w/o ATM. | w/o Contract. | Full |
| --- | --- | --- | --- | --- | --- |
| Bicycle | 19.72 | 22.80 | 22.66 | 22.88 | 23.68 |
| Playroom | 23.36 | 28.74 | 28.65 | - | 29.27 |

### 5.2. Ablation Analysis

We perform comprehensive analysis on various components of our methods using our C3 configuration. Here we choose two representative scenes to perform experiments: one unbounded outdoor scene Bicycle from Mip-NeRF 360^∘ dataset [Barron et al., ] and one indoor scene Playroom from the Deep Blending dataset [Hedman et al., ]. We report the best PSNR that is achieved within $10$K steps for all experiments.

**Table 3. Analysis of self-attention. We show the PSNR for the attention with different number of heads (H) and attention head dimension (F), and different $\lambda$ in Eqn. 1.**
|  | H1-F32 | H2-F16 | H4-F8 | $\lambda$=0.1 | $\lambda$=0.5 | $\lambda$=1.0 |
| --- | --- | --- | --- | --- | --- | --- |
| Bicycle | 23.68 | 23.31 | 23.20 | 23.55 | 23.68 | 23.53 |
| Playroom | 29.27 | 29.15 | 29.10 | 29.14 | 29.27 | 29.22 |

**Table 4. Analysis of the inputs used for attributes prediction.**
|  | w/o Distance | w/o Position | SH D1 | SH D2 | SH D3 |
| --- | --- | --- | --- | --- | --- |
| Bicycle | 23.32 | 9.72 | 23.05 | 23.60 | 23.68 |
| Playroom | 29.18 | 6.19 | 29.23 | 29.15 | 29.27 |

Figure: Figure 6. The effect of Adaptive Tree Manipulation (ATM). Yellow points indicate the splats who have not changed the parent status during entire optimization. Green points represent former children that have been promoted to parents. Around 80% of parents are from our ATM operation.
Refer to caption: x6.png

Importance of Hash Grid. We replace the hash grid with the frequency encoding of the 3D position followed by a $2$-layer MLP to output a $D=64$-dimensional feature vector, which has the same dimension as the one from hash grid. We denote the setting as *FE*.
Without hash-grid we see a significant drop in the performance, highlighting the importance of the feature alignment encoded within the spatial hash grid.

Importance of Attention Mechanism. It can be seen that when we remove self attention mechanism between the nodes of the tree it is detrimental to the performance (Tab. 2 w/o Attn). This validates our motivation that there is relation between various physical attributes of the nodes of tree hence there needs to be a mechanism to facilitate the sharing of information. We also noticed that adding attention mechanism reduces the number of parent points making our method storage efficient: 884K v.s 1.06M averaged across all scenes in dataset [Barron et al., ]. We hypothesise that a configuration with attention can pull information from nearby points, allowing the method to reduce the number of points to store and represent the scene efficiently. Additionally, we ablated various configurations by varying $\lambda$ in Eq. 1 and the number of heads in attention to find the best configuration (Tab. 3). We can see a right balance between the input features and attention features is important for best performance.

Adaptive Tree Manipulation (ATM). We remove Adaptive Tree Manipulation (ATM) and add or delete the trees based only on parent nodes statistics and observe a drop in PSNR (Tab. 2) also visible in rendered images Fig. 5.
This is because there is no mechanism to promote important children to parents that might hinder in populating trees correctly and failing to represent complex scenes effectively. On the other hand, this might also lead to deletion of important children nodes when deleting a parent. Our proposed ATM method can effectively alleviate these issues.
Additionally, in Fig. 6 we show the point clouds of three scenes. Green points represent parents promoted from children during the optimization. Yellow points show parents that stayed parents during entire optimization.
It is clearly seen that the majority of the parent nodes are formed by promoting child nodes. Further note that parts with relatively flat geometry exhibit more yellow, while sophisticated geometry with high frequency details contain more green. Hence, ATM brings a further benefit of being able to fit sophisticated geometry better.

Inputs of MLP for attribute prediction Tab. 4 shows the analysis for the inputs used to predict the attributes.
We conduct the experiments of without using the distance from points to the center of AABB (denoted as *w/o Distance*) and without using the 3D position information (denoted as *w/o Position*) to predict attributes. It can be seen that position is very crucial for training while distance further improves the performance.
Lastly, we analyze the degrees of the SH encoding on the view directions by performing degree from 1 to 3 (denoted as *SH D1* to *SH D3*). Degree of 3 gives the best performance as it has more capacity to model complicated light effects.

Scene Contraction. We analyze the proposed contraction technique applied on the unbounded scene (Tab. 4 w/o Contract.).
Compared with *Full*,
we get inferior performance ($0.8$ PSNR drop), and tend to have training instability issues because the points occasionally move outside the Axis-Aligned Bounding box.

## 6. Conclusion

This paper introduces predictive 3D Gaussian splats, a lightweight representation that dramatically reduces storage for large-scale scenes compared to 3DGS, while maintaining high-fidelity rendering results. To build the framework, we introduce several new techniques. For example, we propose an efficient *parent*-*children* structure that only requires saving *parent* points. The *children* points and most Gaussian attributes can be estimated during rendering by utilizing *parent* points. Additionally, we leverage a hash grid and self-attention on aggregated features to enforce connectivity for *parent* and *children* nodes. We conduct extensive experiments on benchmark datasets to validate our design and demonstrate the our advantages of storage saving and high-quality novel view synthesis.

## Appendix A Implementation Details

In this section, we provide more details for our training. We first provide the learning rates used during the traing in A.1. Next we show the visual illustration and the implementation details of the contraction in Sec. A.2. Then, we analyze the effectiveness of the warm-up training strategy employed in our method in Sec. A.3. Lastly, we discuss the pre-filtering of *parent* points and its implementation in Sec. A.4.

### A.1. Settings of Learning Rate

We employ different learning schedules for different modules. For the hash grid, we start with a learning rate of $2e^{-3}$ and end with a rate of $2e^{-5}$. For opacity, we start with $1e^{-3}$ and end with $2e^{-5}$. The scale and rotation parameters utilize a constant learning rate of $1e^{-4}$. Additionally, we maintain a constant learning rate of $2e^{-4}$ for the attention module. We apply a standard exponential decay scheduling [Kerbl et al., ; Sara Fridovich-Keil and Alex Yu et al., ] to all modules.

### A.2. Details for Contraction

Figure: Figure 7. Illustration for our implemented contraction.
Refer to caption: x7.png

We illustrate the process of contraction (described in Sec. 3.5 of the main paper) in Fig. 7 and Alg. 1. We calculate the inscribed and circumscribed spheres (*i.e*., $S_{inner}$ and $S_{outer}$) with radius $R_{inner}$ and $R_{outer}$ of the initialized Axis-Aligned Bounding Box (AABB), which is estimated from the point cloud generated from COLMAP [Schönberger and Frahm, ]. The estimated AABB is the circumscribed cube of the outer sphere $S_{outer}$. Points falling outside of the outer sphere are brought back to $S_{outer}$.

Figure: Algorithm 1 AABB Estimation and Contraction

Figure: Figure 8. Analysis of warm-up. *First row*:training without warm-up. *Second row*: training with warm-up. *Left column*: points distribution. *Right column*: rendered images with the point cloud.
Refer to caption: x8.png

### A.3. Analysis of Warm-up

We run two experiments on the Garden [Barron et al., ] scene in 10K steps to show the effectiveness of the warm-up in our method.
We have found that using warm-up in training with low resolution images at early stages helps the points populate the empty areas, especially when the COLMAP [Schönberger and Frahm, ] initialization is poor.
Fig. 8 shows the point cloud and corresponding rendered images from different training approaches. As can be seen,
the warm-up training (second row) has a better reconstruction and rendering quality for the background scene, which is poorly initialized from COLMAP [Schönberger and Frahm, ].

### A.4. View Frustum Culling

We apply pre-filtering on the *parent* points before querying features for attributes prediction by culling the view frustum with depth, leading to the computation reduction and the training speedup.
We empirically observe that $15\%$ - $25\%$ points are removed across scenes. The implementation is shown in Alg. 2.

Figure: Algorithm 2 View Frustum Culling

## Appendix B Inference Phase Optimization

Predicting the attributes brings overheads during the inference. However, we note that only the attribute color is view-dependent and the rest remain the same for all frames. Therefore, to minimize the computation cost, we opt to run the color prediction (small MLPs) only and the rest of the attributes can be retrieved from the first frame. This technique enables us method the real-time rendering on the mobile phones.

## Appendix C Per-scene Quantitative Results

We provide the per-scene results on the benchmark datasets. Tab. 5 shows the results on the Mip-NeRF $360^{\circ}$ dataset [Barron et al., ]. Tab. 6 demonstrates the results on the Tank&Temples dataset [Knapitsch et al., ] and the Deep Blending dataset [Hedman et al., ].
We report the per-scene storage (in MB), the number of *parents* and *children*, and the metrics for image quality evaluation.

**Table 5. Per-scene metrics for our approach on the Mip-NeRF $360^{\circ}$ dataset [Barron et al., ].**
| Compact GS | Metric | Garden | Bicycle | Stump | Room | Counter | Kitchen | Bonsai |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| PSNR | 26.81 | 24.77 | 26.46 | 30.88 | 28.71 | 30.480 | 32.08 |  |
| SSIM | 0.832 | 0.723 | 0.757 | 0.919 | 0.902 | 0.919 | 0.939 |  |
| LPIPS | 0.161 | 0.286 | 0.278 | 0.209 | 0.205 | 0.131 | 0.193 |  |
| Storage (MB) | 62.78 | 62.99 | 54.66 | 34.21 | 34.34 | 44.45 | 35.44 |  |
| Light GS | PSNR | 26.73 | 24.96 | 26.70 | 31.27 | 28.11 | 30.40 | 31.01 |
| SSIM | 0.836 | 0.738 | 0.768 | 0.926 | 0.893 | 0.914 | 0.944 |  |
| LPIPS | 0.155 | 0.265 | 0.261 | 0.220 | 0.218 | 0.147 | 0.204 |  |
| Storage (MB) | - | - | - | - | - | - | - |  |
| Scaffold GS | PSNR | 27.17 | 24.50 | 26.27 | 31.93 | 29.34 | 31.30 | 32.70 |
| SSIM | 0.842 | 0.705 | 0.784 | 0.925 | 0.914 | 0.928 | 0.946 |  |
| LPIPS | 0.146 | 0.306 | 0.284 | 0.202 | 0.191 | 0.126 | 0.185 |  |
| Storage (MB) | 271.00 | 248.00 | 493.00 | 133.00 | 194.00 | 173.00 | 258.00 |  |
| 3D GS | PSNR | 27.25 | 25.10 | 26.66 | 31.50 | 29.11 | 31.53 | 32.16 |
| SSIM | 0.856 | 0.747 | 0.756 | 0.925 | 0.914 | 0.932 | 0.946 |  |
| LPIPS | 0.122 | 0.244 | 0.243 | 0.198 | 0.184 | 0.117 | 0.181 |  |
| Storage (MB) | 1331.33 | 1350.78 | 1073.60 | 350.14 | 276.52 | 411.76 | 295.08 |  |
| Ours-C1 | PSNR | 27.17 | 24.32 | 25.75 | 31.62 | 28.54 | 30.47 | 31.32 |
| SSIM | 0.832 | 0.672 | 0.768 | 0.913 | 0.889 | 0.910 | 0.923 |  |
| LPIPS | 0.169 | 0.355 | 0.312 | 0.229 | 0.226 | 0.147 | 0.206 |  |
| Storage (MB) | 26.90 | 25.71 | 34.84 | 16.85 | 17.92 | 23.68 | 17.74 |  |
| Ours-C2 | PSNR | 27.38 | 24.78 | 26.41 | 31.82 | 28.75 | 30.71 | 32.14 |
| SSIM | 0.842 | 0.701 | 0.751 | 0.916 | 0.894 | 0.913 | 0.935 |  |
| LPIPS | 0.156 | 0.325 | 0.260 | 0.224 | 0.218 | 0.146 | 0.192 |  |
| Storage (MB) | 33.01 | 31.72 | 41.09 | 23.02 | 24.04 | 30.05 | 23.61 |  |
| Ours-C3 | PSNR | 27.63 | 24.90 | 26.43 | 31.84 | 29.10 | 31.27 | 32.67 |
| SSIM | 0.847 | 0.717 | 0.753 | 0.917 | 0.900 | 0.918 | 0.941 |  |
| LPIPS | 0.147 | 0.303 | 0.267 | 0.220 | 0.212 | 0.137 | 0.186 |  |
| Storage (MB) | 39.40 | 37.81 | 47.24 | 28.95 | 30.02 | 35.92 | 29.84 |  |
| # of Parents | 1.20M | 1.06M | 1.86M | 330K | 419K | 913K | 403K |  |
|  | # of Children ($k$) | 2 | 2 | 2 | 2 | 2 | 1 | 2 |

**Table 6. Per-scene metrics for our approach on the Tank&Temples dataset [Knapitsch et al., ] and the Deep Blending dataset [Hedman et al., ].**
| Compact GS | Metric | Tank&Temples | Deep Blending |  |  |
| --- | --- | --- | --- | --- | --- |
|  | Truck | Train | Drjohnson | Playroom |  |
| PSNR | 25.070 | 21.560 | 29.260 | 30.320 |  |
| SSIM | 0.871 | 0.792 | 0.9000 | 0.902 |  |
| LPIPS | 0.163 | 0.240 | 0.258 | 0.258 |  |
| Storage (MB) | 41.57 | 37.29 | 47.98 | 38.45 |  |
| Light GS | PSNR | 24.561 | 21.095 | - | - |
| SSIM | 0.855 | 0.760 | - | - |  |
| LPIPS | 0.188 | 0.296 | - | - |  |
| Storage (MB) | - | - | - | - |  |
| Scaffold GS | PSNR | 25.77 | 22.15 | 29.80 | 30.62 |
| SSIM | 0.883 | 0.822 | 0.907 | 0.904 |  |
| LPIPS | 0.147 | 0.206 | 0.250 | 0.258 |  |
| Storage (MB) | 107.00 | 66.00 | 69.00 | 63.00 |  |
| 3D GS | PSNR | 25.350 | 22.070 | 29.060 | 29.870 |
| SSIM | 0.878 | 0.812 | 0.899 | 0.901 |  |
| LPIPS | 0.148 | 0.208 | 0.247 | 0.247 |  |
| Storage (MB) | 608.70 | 255.82 | 773.61 | 553.03 |  |
| Ours-C1 | PSNR | 24.93 | 21.44 | 28.89 | 29.75 |
| SSIM | 0.856 | 0.763 | 0.894 | 0.895 |  |
| LPIPS | 0.196 | 0.283 | 0.280 | 0.284 |  |
| Storage (MB) | 23.11 | 20.90 | 23.59 | 22.21 |  |
| Ours-C2 | PSNR | 25.22 | 21.72 | 28.93 | 30.28 |
| SSIM | 0.862 | 0.777 | 0.902 | 0.902 |  |
| LPIPS | 0.184 | 0.272 | 0.287 | 0.268 |  |
| Storage (MB) | 30.73 | 27.36 | 29.84 | 28.46 |  |
| Ours-C3 | PSNR | 25.45 | 22.18 | 29.34 | 30.44 |
| SSIM | 0.866 | 0.792 | 0.898 | 0.905 |  |
| LPIPS | 0.182 | 0.240 | 0.270 | 0.265 |  |
| Storage (MB) | 36.01 | 34.63 | 35.80 | 35.00 |  |
| # of Parents | 1M | 900K | 900K | 834K |  |
|  | # of Children ($k$) | 1 | 1 | 2 | 2 |

## Peer Reviews

confidence: 5
contribution: 2
initial_rating: 3
presentation: 3
questions: - The results of CompactGS in Table 1 seem to be not applying post-processing, such as quantization of the hash grid. As this paper reports the storage size for the quantized hash grid as described in L443, it is fair to report the post-processed results of CompactGS.

- Moreover, it seems that the tree structure and hash grid design require more training time compared to 3DGS. Please provide the training time to optimize this representation.

- Also, the detailed number of 3D Gaussians for 3DGS should be added to prove that this method requires less number of Gaussians as described in the quantitative results section.

- Furthermore, it is recommended to add more comparisons for the compact 3DGS representations as below. According to the reviewer guideline, the authors do not have to compare papers which are recently published in ECCV'24. But, as the idea this paper is significantly related to HAC [1], it could be helpful for comparing with HAC in terms of both performance and methodology.

  - Compressed3D [2] (CVPR 2024)
  - Reduced3DGS [3] (I3D 2024)
  - HAC [1] (ECCV 2024)
  - SOG [4] (ECCV 2024)
  - Compact3D [5] (ECCV 2024)

---
**Reference**
1. Chen et al., HAC: Hash-grid Assisted Context for 3D Gaussian Splatting Compression, ECCV 2024
2. Niedermayr et al., Compressed 3D Gaussian Splatting for Accelerated Novel View Synthesis, CVPR 2024
3. Papantonakis et al., Reducing the Memory Footprint of 3D Gaussian Splatting, I3D 2024
4. Morgenstern et al., Compact 3D Scene Representation via Self-Organizing Gaussian Grids, ECCV 2024
5. Navaneet et al., Compact3D: Smaller and Faster Gaussian Splatting with Vector Quantization, ECCV 2024
soundness: 3
strengths: - The proposed tree structure enables parent nodes to represent children nodes using an on-the-fly decoding pipeline. This results in the small disk usage of this representation with high rendering quality.

- The optimization schemes, ATM for growth and warm-up for initial training, lead to achieving stable and effective optimization of the proposed representation.
summary: This paper proposes a lightweight 3D Gaussian framework that models inherent spatial relationships within a hierarchical tree structure. To optimize the tree structure, the authors suggest adaptive tree manipulation (ATM), an adaptive growth and pruning strategy for parent and children nodes. Consequently, it demonstrates efficient storage size and superior rendering quality compared to compact 3D Gaussian representations, e.g., LightGS, CompactGS, and EAGLES.
weaknesses: - The main weakness of this paper is the limited technical novelty. The proposed tree structure mainly comes from the anchor-based representation, Scaffold-GS [1]. Also, the adaptive manipulation of children nodes is proposed in HAC [2], with more efficient learnable masks. Moreover, the usage of a hash grid for efficient 3DGS representation is also proposed in HAC and CompactGS [3]. 

- Also, it demonstrates a slower rendering speed compared to 3DGS, as we can see in L458. It indicates that this approach requires more 3D Gaussians splats (both parent splats and children splats) for the rendering phase and only consumes less disk usage in the storage. Therefore, the claim for **"lightweight"** representation seems overclaimed.

- Furthermore, there is limited comparison to existing compact representations. Recently, many papers have been addressing the inefficiency issue of 3DGS. However, there are too less methods in the comparison. Please refer to the question section for the related approaches.

---
**Reference**
1. Lu et al., Scaffold-GS: Structured 3D Gaussians for View-Adaptive Rendering, CVPR 2024
2. Chen et al., HAC: Hash-grid Assisted Context for 3D Gaussian Splatting Compression, ECCV 2024
3. Lee et al., Compact 3D Gaussian Representation for Radiance Field, CVPR 2024

---

confidence: 4
contribution: 4
initial_rating: 8
presentation: 4
questions: In the  text to Figure 2 "patent" should be replaced by "parent".

Maybe you comment already in section 4.1. that in your code you have chosen $K$ to be 2?
soundness: 3
strengths: The idea of deriving the positions of child splats and associated attributes – position, color, scale, etc. – from the parent using a small neural network and only storing parent splats together with the weights of the neural network is new and should definitely be published.
The paper gives a very nice introduction to the problem and a very good overview of the existing approaches.
The method itself is explained in detail and is easy to follow. Comprehensive ablation studies show the influence of the different components of the algorithm on its performance. The extensive evaluation impressively demonstrates the strength of the method compared to the state of the art. It is interesting to note that, in contrast to the original 3DGS, the quality in some of the benchmark comparisons is better despite the considerable compression of the representation.
summary: This paper presents a new representation for 3D Gaussian splats that drastically reduces disk space requirements while providing similar or improved quality compared to previous methods. The method detects inherent common features between splats in close proximity using an attention mechanism (ATM) and exploits these using a hierarchical tree structure in which only the parent splats need to be stored. As a very extensive evaluation on current benchmark data sets shows, the new representation achieves an average reduction in disk footprint of 20x compared to the original 3DGS, with improved PSNR and comparable SSIM and LPIPS, and a 2-5x reduction in storage compared to new works on more efficient representation. In addition, the method is presented to real rendering applications on mobile devices and AR glasses.
In contrast to the competing method CompactGS, it uses a combination of neural fields and self-attention layers to predict not only view-dependent colours but also geometric properties. Secondly, in contrast to competing approaches that store the position of each splat in the point cloud explicitly like ScaffoldGS, the method stores only a small subset of splats, referred to as parents, while the remaining points are predicted on-the-fly during rendering, significantly reducing memory requirements. In contrast to recently presented anchor-based methods, the hierarchical tree representation in combination with the proposed Adaptive Tree Manipulation takes into account the importance of both parent and child splats, which enables a subtree expansion strategy.
weaknesses: Unfortunately, there is no code to go with the paper, so that the procedure can be tested independently.

---

confidence: 3
contribution: 4
initial_rating: 8
presentation: 3
questions: The format of the sub-section titles is not consistent. Some of them are ended with ":", some with "." and some with " ".
soundness: 4
strengths: Storage-effective representation for 3D gaussians.  

The proposed method shown to outperform existing SOTA methods. 

The proposed method shown to run on mobile devices. 

Ablation study to show necessities of the proposed components.
summary: This paper proposes a storage-effective 3D gaussian splatting representation. It uses hierarchical tree structure use similarity between nearby 3D gaussians.  The tree structures are manipulated adaptively in the optimization processes. The method not only outperform SOTA methods, but it is also shown to run effectively in mobile devices.
weaknesses: There is only one figure (i.e., fig.2) for explaining the proposed method. Thus, it may not be easy to follow the process of the method.

Meta-review: The paper introduces a novel hierarchical 3DGS representation that effectively reduces storage requirements while maintaining similar or improved rendering quality compared to the standard 3DGS. By leveraging on inherent feature sharing among proximate splats, the method represents nearby splats using a reduced number of parent splats, leading to a more efficient and compact representation. Experimental results demonstrate that the proposed approach significantly decreases the storage footprint relative to existing or baseline 3DGS methods, without compromising rendering quality. Therefore, I recommend its acceptance.

Decision: Accept (Poster)