F-3DGS: Factorized Coordinates and Representations for 3D Gaussian Splatting

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-06 13:19:02

Abstract: The neural radiance field (NeRF) has made significant strides in representing 3D scenes and synthesizing novel views. Despite its advancements, the high computational costs of NeRF have posed challenges for its deployment in resource-constrained environments and real-time applications. As an alternative to NeRF-like neural rendering methods, 3D Gaussian Splatting (3DGS) offers rapid rendering speeds while maintaining excellent image quality. However, as it represents objects and scenes using a myriad of Gaussians, it requires substantial storage to achieve high-quality representation. To mitigate the storage overhead, we propose Factorized 3D Gaussian Splatting (F-3DGS), a novel approach that drastically reduces storage requirements while preserving image quality. Inspired by classical matrix and tensor factorization techniques, our method represents and approximates dense clusters of Gaussians with significantly fewer Gaussians through efficient factorization. We aim to efficiently represent dense 3D Gaussians by approximating them with a limited amount of information for each axis and their combinations. This method allows us to encode a substantially large number of Gaussians along with their essential attributes---such as color, scale, and rotation---necessary for rendering using a relatively small number of elements. Extensive experimental results demonstrate that F-3DGS achieves a significant reduction in storage costs while maintaining comparable quality in rendered images.

Main paper: 
 introduction: 
 The Neural Radiance Field (NeRF) [25] has demonstrated substantial success in representing 3D scenes through its differentiable volumetric rendering technique and high representation quality. This method has found widespread application in scenarios ranging from novel view synthesis [1, 2, 25, 38, 40] , 3D object generation [30] , editing [15, 16, 35] , segmentation [6, 23] , and navigation [36] , to name a few. As this representation method adopts the concept of a 'field' from physics, in which each point possesses a corresponding value, it requires sampling and processing values from numerous points for each pixel to compute the pixel's color. This can pose significant challenges, especially when attempting real-time rendering on systems with limited resources [10] .Rasterization-based methods, particularly 3D Gaussian Splatting (3DGS) [17] , have emerged as alternatives within the differentiable rendering frameworks. Distinct from NeRF-like methods using hundreds of samples per pixel for volumetric rendering, 3DGS circumvents the need to sample values from empty spaces. This efficiency stems from its primitive-based rendering approach, enabling it to achieve extremely high rendering speed without sacrificing image quality. 3DGS represents a scene with 3D Gaussians as the geometric primitives, learning each Gaussian attribute, such as color, scale, rotation, and opacity, under a fully differentiable rendering pipeline. Through millions of Gaussians, the approach has shown that it can capture detailed textures and nuances, producing exceptional quality, while enjoying the lower computational complexity of rendering via geometric primitives.Despite the fast rendering speed and quality enhancements brought by 3DGS, it comes with several significant drawbacks. A primary challenge is its dependence on an extensive number of 3D Gaussians to maintain high image fidelity. Rendering a high-quality image of a detailed real-world scene often involves several million Gaussians, adding to the system's spatial complexity. Furthermore, each Gaussian is characterized by multiple coefficients to ensure high-quality rendering results. Specifically, it entails 48 spherical harmonics coefficients for color, one for opacity, three for scale, and four quaternion coefficients for rotation. Due to the above issues, 3DGS requires large memory and storage footprints, and this limitation is especially evident in complex, unbounded scenes, such as those in Mip-NeRF360 datasets. In addition, the computational complexity of the current 3DGS rendering algorithm is proportional to the number of 3D Gaussians. Thus, a larger number of Gaussian results in reduced rendering speed.The aforementioned inefficiency stems from the inherent inability of 3DGS to utilize structural patterns or redundancies. We observed that 3DGS produces an unnecessarily large number of Gaussians even for representing simple geometric structures, such as flat surfaces. Moreover, nearby Gaussians sometimes exhibit similar attributes, suggesting the potential for enhancing efficiency by removing the redundant representations.This paper presents a novel method to overcome the limitations of 3DGS, employing structured coordinates and decomposed representations of Gaussians through factorization, as shown in Fig. 1 . Our approach took inspiration from classical tensor or matrix factorization techniques, which have also been extensively investigated in the NeRF literature [7, 9, 11, 13] . We propose a factorized coordinate scheme in which we maintain 1D or 2D coordinates in each axis or plane and generate 3D coordinates by a tensor product. This approach allows us to generate numerous 3D coordinates of Gaussians only with a small number of 1D or 2D coordinates, significantly improving spatial efficiency for the position parameters. Furthermore, the proposed factorization method extends beyond the spatial coordinates to include related attributes, such as color, scale, rotation, and opacity. By exploiting structural patterns and redundancies, the factorized Gaussian attributes can efficiently compress the model size while preserving each Gaussian's essential characteristics.Based on the factorized coordinates and attributes, some of the expanded Gaussians through de-factorization could be non-essential for rendering quality (e.g., de-factorized Gaussians in empty spaces). Motivated by a recent work [21] , we use a binary mask for removing unnecessary Gaussians to accelerate both training and rendering speed. During training, we update the mask using the straightthrough-estimator technique. Once training is complete, we only need to keep the binary values. As the mask only requires 1 bit per value, it incurs negligible storage overhead. Using the binary mask not only marginally improves representation quality but also significantly increases rendering speed, nearly doubling the frames per second (FPS) on the NeRF synthetic dataset.In this study, we conducted extensive experiments to validate the effectiveness of our approach in reducing the spatial redundancy inherent in 3DGS. These experiments demonstrate that our factorization method significantly reduces the required storage by downsizing 3DGS over 90% while achieving comparable image quality. We also provide qualitative results and analysis to support our findings and conclusions, helping to provide a comprehensive understanding of the subject matter. Finally, we prove our approach as an efficient framework for representing 3D scenes, achieving high performance, compact storage, and fast rendering. 
related work 2.1 neural rendering: 
 Neural rendering1 is an emerging approach that leverages machine learning to generate photorealistic visual objects and scenes. Central to this domain is differentiable rendering, enabling the end-to-end training and optimization of 3D scene representations directly from images [20] . Along with differentiable rendering, volumetric rendering has also been an important component in neural rendering. As addressed in 3DGS [17] , the general volumetric rendering equation to calculate the color value 𝐶 of each pixel can be represented as follows:EQUATIONwhere 𝑐 𝑖 and 𝛼 𝑖 represent the color and opacity of each sampled point, respectively. The specifics of how points are sampled, the number of samples per pixel 𝑁 , and the method for calculating opacity 𝛼 𝑖 can vary across different rendering methods. A seminal work in this field is Neural Radiance Fields (NeRF) [25] , which adopts deep neural networks and positional encoding to represent volumetric scene features for high-fidelity rendering [25] . NeRF interprets objects and scenes as fields where every point in space possesses specific values that are computationally generated by neural networks. These values are then rendered using the volumetric rendering equation (Eq. 1). While NeRF allows for detailed rendering, it necessitates dense sampling of the entire volume, including empty spaces, which markedly increases computational costs. Several strategies have been proposed to mitigate this limitation [10, 39] , yet achieving both high-quality representation and real-time rendering remains a significant challenge.Addressing the inefficiencies in NeRF's approach, 3D Gaussian Splatting (3DGS) focuses on rendering only known primitives (Gaussians), thereby eliminating the need for sampling in empty areas and greatly reducing computational costs [17] . However, this technique requires a substantial number of Gaussians to maintain high-quality scene representations, leading to increased storage usage-sometimes more than 1GB for unbounded scenes [17] . Our research, F-3DGS, builds upon 3DGS, targeting its main drawbacks: the high memory requirement and the large number of parameters needed for high-quality rendering. F-3DGS proposes a parameterefficient approach that supports fast training and real-time rendering capabilities, making high-quality neural rendering more feasible for widespread applications.To mitigate the heavy computational costs of NeRF, several works have proposed incorporating additional data structures while retaining the volumetric rendering process. Among these, Plenoxels [12] and Plenoctrees [39] proposed utilizing three-dimensional data structures. However, these approaches were highly inefficient in terms of storage requirements, as the number of required parameters increases cubically with the resolution per axis. To address this, various studies have explored more compact and efficient representations by decomposing 3D structures into combinations of lower-dimensional elements. Specifically, TensoRF [9] and Strivec [13] have implemented the canonical polyadic (CP) decomposition, which uses one-dimensional lines to approximate higherdimensional tensors, thereby enhancing the compactness of 3D representation. TensoRF further extended this approach with vectormatrix (VM) decomposition, while other studies such as EG3D [7] , Hex-Planes [5] , and K-Planes [19] have focused on matrix-only decompositions. Nevertheless, these methods have not achieved real-time rendering due to the fundamental computational limitations of the NeRF-like rendering process. In contrast, our work integrates the concept of factorization with 3DGS, thus enjoying the dual benefits of fast rendering speeds and substantially reduced storage costs. Our method is closely related to Strivec, as it decomposes a scene into a group of small areas, each represented through efficient decomposition. The principal differences lie in factorizing irregular point representations rather than using regular grid representations, which allows us to leverage the fast rasterization-based rendering pipeline. Experimental results demonstrate that our proposed method achieves rendering speeds more than five times faster than Strivec while maintaining similar or superior representation quality. Inspired by the real-time rendering and high quality of 3DGS, numerous studies have been conducted to maintain these advantages while lightweighting the method for broader applications. One of the key strategies for achieving a lighter-weight 3DGS involves minimizing the number of Gaussians used. Several studies demonstrated that techniques such as pruning [21, 28] and the use of anchors [24, 31] can significantly reduce the number of Gaussians without degrading image quality.Beyond simply reducing the number of Gaussians, some studies aim to minimize the average size required to represent each Gaussian. This includes employing lower-bit precision, codebooks, quantization, and compression algorithms like entropy encoding, which collectively make the overall size compact [14, 21, 27] .Our approach integrates both of these strategies, aiming to reduce not only the number of Gaussians needed but also the data size required for representing each Gaussian. Our research is closely related to structured 3DGS approaches [24, 31] . We hypothesize that Gaussians located in proximity can be organized into structured representations and further compacted via factorization techniques. Our method significantly lowers the storage requirements for 3DGS, enhancing its applicability in scenarios demanding both high-quality rendering and storage efficiency. 
factorized 3d gaussian splatting: 
 In this section, we present our method to make the 3D Gaussian splatting model much lighter through factorization of coordinates (Sec. 3.2) and features (Sec. 3.3). Our method integrates two distinct factorization methods: canonical polyadic (CP) and vector-matrix decompositions [9, 13] . Starting with the background, we delve into the specifics of each factorization strategy in the subsequent sections.3D Gaussian splatting is a method that employs 3D Gaussians as primitives for representing 3D scenes. The differentiability of 3D Gaussians and their ease of projection onto 2D image planes facilitate efficient 𝛼-blending in the rendering process [41] . Each Gaussian has several features, including color (represented by spherical harmonics (SH) coefficients), scale, rotation, and opacity parameters. The method uses a collection of these Gaussians to represent the whole scene. The entire rendering pipeline is differentiable, which allows end-to-end training of all Gaussian features given only the images from various views.The initialization process begins with a Structure from Motion (SfM) algorithm [33] to generate sparse point clouds. These serve as initial estimates for the Gaussian positions. The densification stage follows, where the number of Gaussians is incrementally increased through cloning and splitting, enhancing scene details. The algorithm starts with a coarse global structure and systematically integrates finer details by adding more Gaussians. Upon reaching a set number of training iterations, a final fine-tuning phase optimizes all parameters while maintaining a constant Gaussian count. This approach has demonstrated state-of-the-art image quality and exceptional rendering speeds. For further details and comprehensive information, please refer to the original paper [17] .3.2.1 CP Factorized Coordinates. Drawing inspiration from the canonical polyadic (CP) decomposition method to neural rendering [9, 13] , we introduce factorization for representing coordinates. Let 𝑝 𝑥 = {𝑥 1 , . . . , 𝑥 𝑁 }, where 𝑥 𝑖 ∈ R, be a set of factorized coordinates in 𝑥-axis. 𝑁 is the number of points on the axis (|𝑝 𝑥 | = 𝑁 ). Similarly, we can define factorized coordinates for the 𝑦 and 𝑧 axes as 𝑝 𝑦 = {𝑦 1 , . . . , 𝑦 𝑁 } and 𝑝 𝑧 = {𝑧 1 , . . . , 𝑧 𝑁 }, respectively.For these factorized coordinates, we can express the entire set of points in 3D spaces as follows:𝑝 𝑥 𝑦𝑧 = 𝑝 𝑥 × 𝑝 𝑦 × 𝑝 𝑧 = {(𝑥 1 , 𝑦 1 , 𝑧 1 ), . . . , (𝑥 𝑁 , 𝑦 𝑁 , 𝑧 𝑁 )}, (2)where '×' denotes the cartesian product of two sets. Then, the total number of points in 𝑝 𝑥 𝑦𝑧 is 𝑁 3 , and considering that each point requires three real numbers to represent its coordinate, it needs a total of 3𝑁 3 numbers. However, if those points are aligned, we can compactly represent 𝑁 3 points with only 3𝑁 numbers, using only 𝑁 points for each axis. That is, for well-aligned coordinates, factorized coordinate representations can reduce the number of parameters by 1/𝑁 2 . This can substantially alleviate the spatial complexity associated with storing position information. For example, 1,000 points per axis can cover up to 1 billion points using only 3,000 numbers. As illustrated in Fig. 1-(b ), however, this approach is only capable of constructing a very restricted range of structural shapes.Therefore, we propose a hybrid approach in which we utilize multiple sets of factorized coordinates with a small 𝑁 instead of having a single set of factorized coordinates with a large 𝑁 (Fig. 1-(c )). In this approach, we employ 𝐵 sets of factorized coordinates, where each set, denoted as 𝑝 𝑏 𝑥 𝑦𝑧 = 𝑝 𝑏 𝑥 × 𝑝 𝑏 𝑦 × 𝑝 𝑏 𝑧 , can represent a total of 𝑁 3 points, and consequently constructing 𝐵𝑁 3 number of points with 3𝐵𝑁 number of points. While the compression ratio remains the same as 1/𝑁 2 , this approach demands more parameters due to the smaller values for 𝑁 (usually 𝑁 ≤ 10 to obtain a good rendering quality). Furthermore, each set of factorized coordinates can have varying resolutions, as illustrated in Fig. 1-(d) and (e), thereby offering more flexibility in the positions, which increases the expressibility. Considering the multi-set and multiresolution factorized coordinates, we can rewrite the set of whole points constructed by the proposed method as,EQUATIONwhere𝑝 𝑏 𝑥 = {𝑝 𝑏 1 , . . . , 𝑝 𝑏 𝑁 𝑏}, and 𝑁 𝑏 denotes the number of point in each axis of 𝑏-th factorized coordinate set. That is, using only 3 𝑏 𝑁 𝑏 numbers, we can represent at maximum 𝑏 𝑁 3 𝑏 threedimensional coordinates. As shown in Fig. 1 , we can approximate dense Gaussians with a small number of factorized coordinates.Coordinates. The recently introduced planebased decomposition methods, such as TensoRF [9] and K-planes [11] , have achieved promising results, offering parameter-efficient representations in NeRF. In a similar vein, our approach incorporates a vector-matrix (VM) factorization [9] for coordinates. Let 𝑝 𝑥 𝑦 = {(𝑥 1 , 𝑦 1 ), . . . , (𝑥 𝑁 , 𝑦 𝑁 )} (we omitted the superscript 𝑏 for brevity onwards) be a set of two-dimensional coordinates in the 𝑥𝑦 plane, and 𝑝 𝑧 = {𝑧 1 , . . . , 𝑧 𝑁 } is a set of one-dimensional coordinates in the 𝑧 axis (similarly, we can also define 𝑝 𝑦𝑧 , 𝑝 𝑥 , 𝑝 𝑥𝑧 , and 𝑝 𝑦 ). Then, the set of whole points in the VM factorized coordinate scheme is defined asEQUATIONThe points on the plane, e.g., 𝑝 𝑥 𝑦 , can move freely without any constraints, increasing the flexibility of positions at the expense of the higher spatial complexity required for storing two-dimensional coordinates.3.3.1 CP Factorized Representations. The primary factor contributing to the storage requirements of 3DGS is the rich sets of parameters of each 3D Gaussian. In this section, we describe the factorized representation of those attributes for each Gaussian without compromising the rendered image quality. Similar to the CP factorized coordinate scheme, we hold the features associated with the factorized coordinates in each axis. For example, the 𝑥 axis scale parameters for each Gaussian can be obtained by the element-wise product of the features in the factorized coordinates. More formally,EQUATIONwhere • represents the outer product, 𝑠 𝑥 , 𝑠 𝑦 , 𝑠 𝑧 ∈ R 𝑁 ×3 denotes the scale parameters for 𝑁 points along the 𝑥, 𝑦, 𝑧 axis in the factorized coordinates respectively, and 𝑠 𝑥,𝑖 ∈ R 𝑁 indicates the 𝑖-th parameters in the scale parameters. 𝑠 𝑥 𝑦𝑧 ∈ R 𝑁 ×𝑁 ×𝑁 ×3 is the final scale parameters used in the rendering process, 𝑠 𝑥 𝑦𝑧,𝑖 ∈ R 𝑁 ×𝑁 ×𝑁 denotes the 𝑖-th parameters. Each element in 𝑠 𝑥 𝑦𝑧 corresponds to the scale parameter associated with the point in 𝑝 𝑥 𝑦𝑧 . We also factorize rotation parameters 𝑞 𝑥 𝑦𝑧 ∈ R 𝑁 ×𝑁 ×𝑁 ×4 in the same way.To handle view-dependent color features and opacity, we employ a Multi-Layer Perceptron (MLP) to generate coefficients for spherical harmonics and opacity. It takes low-dimensional learnable features as input, and we factorize these learnable features along each axis. We maintain a factorized feature vector for colors and opacity for each axis and compute the input feature for the MLP as follows.EQUATION𝑐 𝑥 𝑦𝑧 , 𝛼 𝑥 𝑦𝑧 ← MLP(𝑓 𝑥 𝑦𝑧 ; 𝜃 ),where 𝑓 𝑥 , 𝑓 𝑦 , 𝑓 𝑧 ∈ R 𝑁 ×𝑑 denote the learnable feature vectors for 𝑁 points along the 𝑥, 𝑦, 𝑧 axes respectively, and 𝑑 is the learnable feature dimension. Then, we obtain the features of the whole set of points 𝑓 𝑥 𝑦𝑧 ∈ R 𝑁 ×𝑁 ×𝑁 ×𝑑 , and the MLP takes the features 𝑓 𝑥 𝑦𝑧 as a batch and generates color features 𝑐 𝑥 𝑦𝑧 ∈ R 𝑁 ×𝑁 ×𝑁 ×48 (three degrees of spherical harmonics for example), and opacity 𝛼 𝑥 𝑦𝑧 ∈ [0, 1] 𝑁 ×𝑁 ×𝑁 . EQUATIONEQUATIONwhere 𝑓 𝑥 𝑦 , 𝑓 𝑦𝑧 , 𝑓 𝑥𝑧 ∈ R 𝑁 ×𝑁 ×𝑑 are the feature matrices associated with the VM factorized coordinates 𝑝 𝑥 𝑦 , 𝑝 𝑦𝑧 , 𝑝 𝑥𝑧 . In addition, 𝑓 𝑥 , 𝑓 𝑦 , 𝑓 𝑧 ∈ R 𝑁 ×𝑑 represent the feature vectors associated with 𝑝 𝑥 , 𝑝 𝑦 , 𝑝 𝑧 , respectively.The initialization scheme plays a pivotal role in achieving high rendering quality in 3D Gaussian Splatting (3DGS), especially for complex objects and scenes [17] . The original 3DGS leverages a structure from motion technique [33] , to initialize 3D Gaussian points at the onset of training.In our proposed method, we observed a similar initialization effect on the resulting models, particularly for the factorized coordinates. To address this, we propose a heuristic method to effectively determine the initial positions of 3D Gaussians. Our approach begins by generating point clouds from a pre-trained 3DGS model of a target scene.First, we identify the boundary values of the point cloud as 𝑥 𝑚𝑎𝑥 , 𝑥 𝑚𝑖𝑛 , 𝑦 𝑚𝑎𝑥 , 𝑦 𝑚𝑖𝑛 and 𝑧 𝑚𝑎𝑥 , 𝑧 𝑚𝑖𝑛 . After multiplying the bound value by 1.2, we divide it into multiple sets of bins by average interval. For example, we set 0.026 in the nerf-synthetic dataset, and the number of bins along the x-axis will be 1.2 * (𝑥 𝑚𝑎𝑥 -𝑥 𝑚𝑖𝑛 )/0.026. After calculating a 3D histogram from the point cloud coordinates for the set of bins, we construct a set of factorized coordinates (each set of factorized coordinates has 𝑁 3 points, 𝑁 points along each axis, we omitted the superscript '𝑏') in the histogram bin which contains several points exceeding the threshold 𝜆. For example, we set 5 for synthetic-NeRF dataset [25] , and discard the bins that have no more than 5 points. Furthermore, for bins that have more than 𝑁 3 points, more factorized coordinates are required, so we introduce an additional set of factorized coordinates for every additional 𝑁 3 points.This heuristic method has proven effective in practice. We will present a comparison of our initialization scheme with random initialization in Tab. 4, highlighting its practical efficacy.As addressed in Sec. 3.2.1 and 3.3.2, factorized coordinates and representations can be highly efficient in terms of compactness. However, using all 𝑁 3 is computationally expensive since the current rasterization pipeline in 3DGS involves linear sorting and 𝛼-blending to render an image; hence, the computational complexity is proportional to the number of Gaussians. Furthermore, we also observed many Gaussians in the factorized coordinates do not contribute to the final rendering results (e.g., when the opacity 𝛼 < 0.001). Therefore, pruning the less important 3D Gaussians can readily accelerate both training and rendering processes. Inspired by the recent works [21, 32] in compact representations for NeRFs and 3D Gaussian Splatting, we adopt binary masks to prune out non-used coordinates while exploiting the compactness of factorized representations. To train binary masks, we used the straight-throughestimator method [4] to update mask values. Having a trainable mask for each coordinate requires total 𝑁 3 values. However, once the binary mask is trained, a single bit per coordinate is sufficient. Therefore, the spatial overhead of using binary masks is negligible.We applied masks to variables that are directly related to visibility: both colors 𝑐 𝑥 𝑦𝑧 and opacities 𝛼 𝑥 𝑦𝑧 . Since there are total 𝑁 3 points that can be represented using factorized coordinates (Sec. 3.2.1), we also need 𝑁 3 binary masks M ∈ R 𝑁 ×𝑁 ×𝑁 for each coordinate set (To avoid the notational clutter, we omitted the superscript '𝑏', which denotes a set of factorized coordinates.). During training, we use floating points for masks, as we cannot properly calculate gradients for binary variables. We use the following equation for the binarized masks M during training:EQUATIONwhere sg, H , 𝜏, and 𝜎 denote the stop gradient function, Heaviside function, the threshold, and the element-wise sigmoid function, respectively. Then, we element-wisely multiplied this binarized mask M to scales 𝑠 𝑥 𝑦𝑧,𝑖 ∈ R 𝑁 ×𝑁 ×𝑁 and opacities 𝛼 𝑥 𝑦𝑧 ∈ [0, 1] 𝑁 ×𝑁 ×𝑁 .To increase the sparsity of the mask for more efficient representations, we add a regularizing loss term to the objective function so that mask values lean towards being zeroed out. The loss is defined as follows:EQUATIONwhere M 𝑖 𝑗𝑘 ∈ R denotes an element of the mask. 
experiment 4.1 proof-of-concept experiment: 
 In this section, we introduce a proof-of-concept experiment designed to demonstrate the feasibility of our method using point clouds. The experiment involves initializing sets of factorized coordinates and optimizing them using the Chamfer distance function.The primary objective here is to showcase the effectiveness and efficiency of factorized coordinates in approximating ground truth point clouds. For this experiment, we selected a sample point cloud of an object from ShapeNetCorev2 [8] .Table 1 : Comparison of our method with the previous and concurrent novel view synthesis methods on synthetic-NeRF dataset [25] . All scores for the baseline methods are directly taken from their respective published papers, whenever available. We show the training time, model size, PSNR, and rendering FPS (Frames per second). As illustrated in Fig. 3 , utilizing factorized coordinates led to a significant reduction in the number of parameters required for representing 3D coordinates. Remarkably, it requires less than 10% of the original point cloud parameters to depict the contour of the object. However, this method introduced a certain level of blockiness in the approximation, which is inherent to the approach. To address this, we suggest the integration of learnable opacities (Secs. 3.3.1 and 3.3.2) and masks (Sec. 3.5), which could effectively mitigate this blockiness.We implemented our F-3DGS model in PyTorch [29] framework with the original 3DGS [17] CUDA kernels for rasterization. In experiments, we used the default training hyperparameters of 3DGS [17] and added additional hyperparameters required for the F-3DGS model. To achieve the coarse-to-fine scene geometry reconstruction, we first jointly train factorized coordinates and representations. However, we found that joint training often leads to unstable optimization after a certain number of iterations. Therefore, after 20K iterations, we fixed the coordinates and only optimized the factorized attributes of Gaussians. For CP decomposition, we employed MLPs with a single hidden layer consisting of 128 hidden units. For VM decomposition, we utilized MLPs with two hidden layers. We used the Adam optimizer with initial learning rates of 0.02 for 3D Gaussians' tensor factors and 0.001 for the MLP decoder. As for scale coefficients, we did not use any activation function.In this subsection, we evaluated F-3DGS using two real-world datasets (objects from Tanks&Temples [18] and indoor scenes from Mip-NeRF 360 [3] ) as well as one synthetic dataset (synthetic-NeRF [25] ).Qualitative Result. For a qualitative evaluation, we selected two distinct scenes from the synthetic-NeRF dataset. As Fig. 5 shows, we obtain similar visual quality on the Ship object only with 4-7MB. Similarly, for the Mic object, our VM-48 model requires 16MB storage, while the original 3DGS costs 40-50MB storage. 3.9 27.59 0.897 0.144 <20 TensoRF-VM [9] 71.8 28.56 0.920 0.125 <20 Strivec-48 [13] 54.08 28.70 0.924 0.113 <20 3DGS [17] 105.15 To analyze more deeply, we selected six objects from the synthetic-NeRF dataset to illustrate the distribution of Gaussians and rendered images using both our method and the original 3DGS (Fig. 4 ). Our F-3DGS only requires 10% of storage while performing accurate reconstruction. For example, on the Drum surface, our F-3DGS points distribute an aligned axis in each set of factorization coordinates, while the original 3DGS points are unordered. This comparison reveals that our factorized representation enables an aligned and denser Gaussian distribution. By employing the factorization of 3D Gaussians, our model captures and represents the flat surfaces of objects such as the Hotdog and Drums efficiently, thereby achieving compact model sizes without compromising on fidelity. Attributing to the learnable scale parameters of Gaussian points, our model can also obtain high fidelity on smooth surfaces, such as the Ship object. The density of F-3DGS can reconstruct more detailed objects without increasing the number of parameters, showcasing the efficacy of our method in achieving high-quality renderings with a reduced storage requirement.Quantitative Result. Tab. 1 shows the quantitative results evaluated on the synthetic-NeRF dataset. Our approach consistently reduces the storage requirements while maintaining fast rendering speed. In particular, the approach obtained the equivalent standard or even surpassed TensoRF [9] and Strivec [13] . With 6.06 MB storage costs, our CP model can achieve 32.42 on PSNR for synthetic-NeRF dataset [25] .In addition, further experiments are conducted using Tanks&Temples and 360 indoor scenes datasets. Note that we achieve a 10× compression ratio and the same rendering speed on the Tanks&Temples dataset in Tab.2. And Tab.3 shows our quantitative result on 360 indoor scenes.Initialization. As addressed in Sec. 3.4, the initialization is an important factor in achieving high rendering quality in both 3D Gaussian Splatting (3DGS) and our proposed method. As shown in Tab. 4, using random initialization significantly deters the quality down to 26.06, which demonstrates the significance of our initialization method.Factorization of Non-coordinate Attributes. The original 3DGS produces a significant amount of Gaussians, and moreover, some nearby Gaussians share similar attributes. To address this problem, the proposed factorization of color and opacity achieves a reduction in storage requirements by approximately 70%, and factorization of scale and rotation by 25% while maintaining the rendered quality. As Tab. 4 shows the storage could be compressed to 14MB with our F-3DGS.Learnable Masking. It is crucial that densification scheme in 3DGS [17] is not feasible in our structured factorization coordinates approach. However, the redundant Gaussians in our factorized coordinates also need to be eliminated. Considering real-time rendering and training speed, we utilize the binary mask to prune out redundant and unessential 3D Gaussians to accelerate while only increasing a little storage to store mask parameters. The rendering speed is nearly 90% increase than the method without masking in Tab. 4. The Size of Learnable Feature Vectors. As mentioned in Secs. 3.3.1 and 3.3.2, our method introduces a hyperparameter 𝑑 that controls the number of learnable feature dimensions. We evaluate our Factorize-3DGS on the synthetic-NeRF dataset [25] using both CP and VM decompositions with different numbers of 𝑑.As shown in Tab. 5, the larger 𝑑 enhances the 3D reconstruction performance, simultaneously increasing the overall storage requirements. Both Factorize-CP and Factorize-VM achieve consistently better rendering quality with more feature dimensions. Factorize-CP achieves a more compact model with high performance by using our factorized method. Ours-CP-16 achieves 32.43 PSNR with 6.06 MB, outperforming TensoRF-CP (see Tab. 1). Also, F-3DGS using VM-16 achieves the visual quality of TensoRF-VM (33.14 PSNR) but requires significantly less storage only 28.75 MB, compared to 71.80 MB needed by TensoRF-VM. 
conclusion: 
 In this paper, we have proposed a novel Factorized 3D Gaussian Splatting (F-3DGS), effectively addressing the computational and resource constraints in neural rendering applications. Our method integrates tensor factorization techniques, significantly reducing the storage requirements while maintaining image quality compared to 3DGS, as evidenced by our extensive experiments across various datasets. We believe this newly introduced representation opens a new direction for further research. Compared to traditional matrix and tensor factorization, the proposed approach does not involve any pre-defined grid representations. This will enable us to model very large and unbounded scenes more efficiently since we can ignore large empty parts in the scene, as opposed to grid-based approaches that require additional sparsification techniques. 
 
 Appendix: We use the "neural rendering" term to refer to a general rendering technique that exploits differentiable computational pipelines to render images, which includes both NeRF and 3DGS. 


Decision: This paper was reviewed by four experts in the field. The recommendations are Weak Accept, Borderline Accept, Borderline Accept, Weak Reject. The authors have addressed majorities of the concerns from reviewers about the experimental comparison. Based on this, the decision is to recommend the paper for acceptance to ACM Multimedia 2024.

Still, the reviewers did raise some valuable concerns, including 1) comparison to other lightweight models, and 2) discussing the trade-off of increasing training time and smaller model size. We recommend that the authors carefully read all reviewers' final feedback, and revise the manuscript as needed. We congratulate the authors on the acceptance of their paper!

Reviews: review: paper_summary: paper_summary: This paper addresses the computational costs associated with 3D Gaussian Splatting (3DGS), which offers fast rendering but requires substantial storage. To mitigate this, the authors propose Factorized 3D Gaussian Splatting (F-3DGS), which reduces storage requirements while preserving image quality. F-3DGS approximates dense clusters of Gaussians with fewer Gaussians through efficient factorization techniques. This approach represents dense 3D Gaussians by encoding them with limited information along each axis, reducing storage costs while maintaining comparable quality in rendered images.
strength_weakness: strength_weakness: - This paper is written well and very easy to follow.
- The proposed idea is inspired by TensoRF and Strivec, the authors apply their idea to decompose the parameters of 3DGS, including color, scale, and rotation. This approach effectively reduces storage requirements while having a minimal impact on PNSR performance. 
- The validation is comprehensive, and the source codes are provided by the authors.
limitations: limitations: - 3DGS is an explicit representation; unlike TensoRF, the memory footprint in 3DGS may not be able to be reduced. I would like to see more results about VRAM memory footprint during the training, inference, and rendering of the proposed method instead of only storage memory.
- The qualitative results make me think of recent works in 3DGS-based inverse rendering and surface reconstruction, for example, GaussianShader, GIR, 2D Gaussian Splatting, etc., by aligning the axis of 3D/2D Gaussian towards the screen and improving the efficiency of 3D/2D Gaussians. The authors are encouraged to discuss those works in the literature review.
- I would like to understand why the authors choose not to decompose positions (i.e., means) of 3D Gaussians, the authors could discuss it a bit.
paper_summary: paper_summary: This paper proposes a novel framework based on factorized representation to reduce the storage requirements of 3D Gaussian splatting. Specifically, 3D coordinates are decomposed as the Cartesian product of three axes or unions of the Cartesian product of a plane and an axis. Then, scale, rotation, and learnable features are factorized by the outer product of each axis and utilized to further generate attributes of 3D Gaussian primitives. Experimental results demonstrate that the proposed method achieves around 10 times size reduction with comparable quality when compared to the original 3DGS.
strength_weakness: strength_weakness: 1.	The introduction is well-written and easy to follow.
2.	The illustration of the factorized representation is interesting and technically sound.
3.	The experimental results show that the proposed method can significantly reduce the size of 3D Gaussian primitives without sacrificing rendering quality.
limitations: limitations: 1.	In the experiment section, authors should compare other mentioned methods that lightweight 3DGS. 
2.	What is the difference between TensoRF [9] and the proposed method, in addition to applying CP and VM factorization [9] to 3DGS? It would be better to explicitly state the special design in terms of 3DGS to highlight the novelty.
3.	Table 2 shows that 3DGS has 30.88 dB with 105.15MB. However, 3DGS presents 23 dB with 411MB on the same dataset in their paper. Please explain this discrepancy.
4.	What is the data type of factorized coordinates, integers or floating-point numbers? Could the masking strategy remove pseudo coordinates generated by the Cartesian product? Please explain why the size without masking is the same as the full proposed method in Table 4 in the ablation study.
5.	The authors generated point clouds from a pre-trained 3DGS model as initialization. It would be better to compare initialization from the original SfM points and the pre-trained points in the ablation study.
6.	It would be better to improve the illustration of the initialization part.
paper_summary: paper_summary: This method proposes a 3D scene representation that factorizes both coordinates and 3D Gaussian Splatting (3DGS) features. By introducing structured coordinate and feature factorization, it significantly reduces storage requirements. Additionally, the method proposes an initialization approach that generates point clouds from a pre-trained 3DGS model, factorizes these point clouds, and then initializes the 3DGS. The method also introduces an adaptive mask to remove redundant Gaussians in empty spaces.
strength_weakness: strength_weakness: Experimental results demonstrate the effectiveness and superiority of this method by using structural similarity index (SSIM) and learned perceptual image patch similarity (LPIPS) metrics. This method makes it easier for 3DGS to fit structured objects and reduces a certain model size
limitations: limitations: Although this method can compress the model and improve structural attributes, its indicator improvement is not very competitive, and the training time is much longer than that of 3DGS.
paper_summary: paper_summary: This paper proposes a compact representation for 3D Gaussians, namely F-3DGS. In the proposed method, 3D Gaussians are compactly represented by factorized coordinates and latent embeddings associated with the coordinates. Then, 3D Gaussian attributes are generated from latent embeddings of given factorized coordinates. Meanwhile, learnable masks are applied to coordinate sets to remove redundant 3D Gaussians.
strength_weakness: strength_weakness: 1. The proposed factorized coordinate representation of 3D Gaussians is novel and interesting. This is because factorized representation has been proven effective in NeRF slimming, while its application in 3D Gaussian splatting has not been fully studied. This paper successfully uses spatial decomposition to compactly represent 3D Gaussian primitives as factorized grids and effectively recover primitive parameters from grid features.
2. The motivation of the proposed method is reasonable. This is because there are redundant primitives in the model of vanilla 3DGS, which can be removed by the proposed masking mechanism. In addition, there are correlations between spatially adjacent Gaussian primitives, which can be implicitly captured by the factorized grid to improve compactness.
3. The performance of the proposed method well demonstrates its effectiveness. That is, the proposed method is able to achieve 10x size reduction compared to vanilla 3DGS.
limitations: limitations: 1. Hyper-parameters in Sec 3.4 should be specified for each dataset. For example, the hyper-parameter lambda should be specified for each dataset, and the average interval of bins are also should be specified.
2. I would like to know why you did not report the performance of VM decomposition on the Tanks & Temples dataset. Besides, which factorization paradigm do the results on Mip-360 correspond to?
3. I have some concerns about the experimental results. For example, the size of 3DGS on the Tanks&Temples dataset exceeds 400MB, as reported in existing work, while the size reported in this paper is only 105.15MB. Therefore, the authors should explain the model size calculation method.
4. I would like to know why the model size does not decrease after adding the mask, as shown in Table 4. I would like to see more analysis of this phenomenon.