Tangram-Splatting: Optimizing 3D Gaussian Splatting Through Tangram-inspired Shape Priors

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-01 09:35:43

Abstract: As the growth of VR and AR industry, 3D reconstruction has become a more and more important topic in multimedia. Although 3D Gaussian Splatting is the state-of-the-art method of 3D Reconstruction, it needs a large number of Gaussians to fit a 3D scene due to the Gibbs Phenomenon. The pursuit of compressing 3D Gaussian Splatting and reducing memory overhead has long been a focal point. Embarking on this trajectory, our study delves into this domain, aiming to mitigate these challenges. Inspired by tangram, a Chinese ancient puzzle, we introduce a novel methodology (Tangram-Splatting) that leverages shape priors to optimize 3D scene fitting. Central to our approach is a pioneering technique that diversifies Gaussian function types while preserving algorithmic efficiency. Through exhaustive experimentation, we demonstrate that our method achieves a remarkable average reduction of 62.4\% in memory consumption used to store optimized parameters and decreases the training time by at least 10 minutes, with only marginal sacrifices in PSNR performance, typically under 0.3 dB, and our algorithm is even better on some datasets. This reduction in memory burden is of paramount significance for real-world applications, mitigating the substantial memory footprint and transmission burden traditionally associated with such algorithms. Our algorithm underscores the profound potential of Tangram-Splatting in advancing multimedia applications.

Main paper: 
 introduction: 
 In recent years, 3D reconstruction and novel view synthesis tasks have gained increasing popularity in multimedia, driven by the growing demand for Virtual Reality (VR) and Augmented Reality (AR) applications [13, 31] which is an important topic in multimedia [1, 15, 20, 22, 36] . Traditional methods for such tasks, e.g., Structure from Motion (SfM) [26] and Multi-View Stereo (MVS) [12, 32, 33, 35] , have yielded impressive results. Additionally, recent advancements such as Neural Radiance Field (NeRF) [21] and 3D Gaussian Splatting (3DGS) [14] have further pushed the boundaries of performance in these tasks. However, despite its advancements, 3DGS [14] still exhibits inherent limitations. The Gibbs phenomenon [7] inherent in the Gaussian Transform implies that accurately representing a scene requires an infinite number of Gaussians, which is impractical to be realized. Consequently, Gaussian-based representations have an upper bound of its performances.In this study, we introduce Tangram-Splatting, a novel approach aimed at providing a more precise and efficient representation for 3D scenes. This method exhibits promising potential for enhancing 3D reconstruction tasks and may serve as inspiration for future research in the field. Inspired by previous works [4, 9, 17, 29] , we recognize the diverse range of Gaussian-based functions, such as Gaussian Mixture functions, Difference of Gaussian functions, and Generalized Exponential Function, each offering unique advantages. For instance, Generalized Exponential Functions are better to converge on rectangular waves, Difference of Gaussian functions are better to converge on triangular waves and others are better to converge on a different shape. Tangram, an ancient Chinese puzzle which aims at filling a specific kind of shape by using different shapes of geometry, inspires us to leverage the advantages of those functions and combine them together to better fit the 3D scene. According to [10] , directly changing the orginal Gaussian representation to a new Gaussian-based function is non-trivial, especially for the rasterization process. In order to reserve the original rasterization process, we use maximum likelihood estimation to represent different functions with an adaptive multiplication matrix, instead of directly changing the original Gaussian rasterization process. Different parts of a 3D scene require different kinds of shape priors, which means that we set strong condition to apply certain kind of Gaussian function to reconstruct a certain space based on the prior information about the eigenvalues of vanilla Gaussian. In order to search for the best fit of the 3D scene, we change the clone procedures of 3DGS [14] and set a brand new criterion to allocate the types of Gaussians in 3DGS [14] .Extensive experiments show that our approach improves 3DGS [14] approximately 62.4% in terms of memory efficiency while maintaining comparable image quality and training speed. In fact, our method performs even faster than 3DGS [14] for at least 10 minutes in several cases. Furthermore, we conducted experiments to validate the proposed theories.The main contributions are summarized as follows:(1) To the best of our knowledge, we are the first to leverage various types of Gaussians for 3D scene reconstruction with promising performance.(2) By harnessing a new cloning technique, we successfully allocate the types of Gaussians while not sacrificing the speed of running the procedures which has saved almost 62.4% of the points that 3DGS [14] originally needed and decreasing at least 10 minutes of training time.Figure 1 : Illustration of the proposed method. We have shown the big picture of our method in the figure. We argue that only using one particular type of Gaussian function is not a compact representation. As a result, we leverage 3 types of Gaussian-based functions, including vanilla Gaussian function (GS), Generalized Exponential function (GEF), Difference of Gaussian (DoG) to reconstruct a 3D scene. However, directly using 2 new functions as basis function is non-trivial, particularly we may face a drawback in the speed of rasterization process. As a result, we use the maximum likelihood estimation method to diversify the Gaussian basis function while maintaining the original rasterization methods. Our method can reduce the memory cost for about 62.4% compared to 3DGS [14] , decrease the training time for about 10 minutes while maintaining the equivalent PSNR performance.(3) We have introduced a signal estimation framework aimed at reconstructing a 3D scene using various types of base functions, significantly enhancing the practical application of our work. 
related work 2.1 3d gaussian splatting: 
 3D reconstruction and rendering has gained increasing attention. Algorithms include Neural Radiance Field [21, 34, 37] , Structure from Motion [28] and other works [14, 30] have tremendously improve the performance of 3D reconstruction. Among them, 3DGS [14] is the state-of-the-art algorithm in 3D rendering and reconstruction field which has gained great popularity recently. It represents a 3D scene by lots of Gaussians and leverages rasterization methods and an adaptive control method to aprroximate a 3D scene. However, a lot of works [5, 18, 19, 24, 25] have claimed its bottleneck in memory cost and claim that it is not a compact representation. However, those works mainly focus on how to prune the trivial Gaussian and how to compress the parameters. In this work, we are going to refine the representation part. We claim that Gaussian function itself is not compact enough due to the Gibbs Phenomenon and we then design a brand new mechanism which can better represent a realistic 3D scene with less points needed.Inspired by lots of works [3, 8, 27] , especially Generalized Exponential Splatting (GES) [10] , we are interested in leveraging signals and systems perspectives to analyze the properties of 3DGS [14] . Gaussian Transform is a long studied signal processing field especially in the last century, which aimed at analyzing the properties when trying to use Gaussian functions to approximate a signal. We have found out that the last century's Gaussian Transform is so useful for analyzing 3DGS [14] and that, there is nothing new under the sun. 
theoretical analyses: 
 In this section, we present theoretical analyses of our framework. As we present in Fig. 1 , we try to leverage different types of Gaussians to approximate the 3D scene in order to have a more compact representaion. The theoretical analyses involved in our approach will be described in detail below.To better understand the rationale behind employing diverse basis functions as opposed to relying solely on a single Gaussian function, as in the 3DGS algorithm [14] , or a singular Generalized Exponential Function, as utilized in the GES algorithm [10] , it is imperative to delve into the distinctive properties inherent in various basis functions. This strategic approach enables the reconstruction of scenes in accordance with localized conditions. To streamline our analysis, we embark on an exploration of the properties exhibited by various basis functions in a one-dimensional context. Gaussian Function. The spatial domain expression and Fourier transform of a Gaussian function with a mean of 𝜇 and a variance of 𝜎 2 is shown below:EQUATIONAs depicted in Fig. 2 , the Fourier transform of a Gaussian signal preserves its Gaussian nature. Notably, the temporal energy distribution of a Gaussian signal is predominantly concentrated within a range offset by 3𝜎 from its mean 𝜇. Conversely, in the frequency domain, the energy of the signal is concentrated within a range offset by 3 𝜎 , centered around 0. Due to its inherent low-pass filtering characteristics, the Gaussian function necessitates an increase in the number of Gaussian components and the adoption of smaller Gaussian spheres by 3DGS [14] to effectively capture scene details. However, this approach inevitably leads to redundancy, inefficient storage utilization, and diminished computational efficiency.Generalized Exponential Function (GEF). The one-dimensional expression of the Generalized Exponential Function (GEF), as described in [10] , is 1√ 2𝜋𝜎 𝑒 - (𝑥 -𝜇) 𝛽 2𝜎 2. However, it is worth noting that the Fourier transform of GEF might lack an analytical solution, contingent upon the parameter 𝛽 within the exponential term. This feature may contribute to the instability observed during the reconstruction process, as noted in [10] . For simulation purposes, specific parameter values were selected, and the outcomes are delineated in Fig. 2 .The one-dimensional simulation results from [10] indicate that GEF exhibits favorable performance when applied to smooth signals (e.g., Parabolic Function, Gaussian Function), requiring fewer instances (N ) to minimize fitting errors. However, when being confronted with sharp signals (e.g., Triangle Function, Exponential Function), GEF tends to exhibit non-convergence. This observation implies that while GEF is adept at fitting smoother regions in 3D reconstruction, it may not yield optimal results for abrupt regions. Consequently, the introduction of an additional basis function, such as the Difference of Gaussians, becomes imperative.Difference of Gaussians (DoG). The spatial domain expression and Fourier transform of a DoG with a mean of 𝜇, a variance of 𝜎 2 and a step size 𝜈 of the variance is shown below:EQUATIONAs depicted in (2), DoG serves as a band-pass filter, with the center frequency being contingent upon 𝜈, as illustrated in Fig. 2 for simulation. Leveraging DoG as the basis function facilitates the effective reconstruction of scene details at the desired frequency by adjusting 𝜈. The Gaussian function excels in reconstructing low-frequency information, while the GEF demonstrates proficiency in capturing smooth regions with minimal loss, even with a reduced number of functions. On the other hand, the DoG proves effective in reconstructing scene details. Leveraging the distinctive strengths of these three basis functions, we introduce Tangram-Splatting. By harnessing the low-pass property of Gaussians for large-scale scene reconstruction, coupled with the shape adaptability of GEF and the band-pass nature of DoG for fine-tuning, we achieve a remarkable reduction in memory usage by 62.4% and a 44.2% increase in computational speed compared to 3DGS [14] . Notably, this improvement is achieved without resorting to a plethora of small Gaussian spheres, resulting in a more compact scene reconstruction.To enhance the computational efficiency of 3DGS [14] , we adopt a novel approach in Tangram-Splatting, eschewing the construction of three distinct models for scene reconstruction. Instead, we frame the problem of scene reconstruction with different basis functions as a signal estimation task.Given that fast rasterization in 3DGS [14] is tailored for Gaussian functions, we conceptualize the field space 𝑁 𝑗=1 𝜔 𝑗 𝑒 -1 2 𝑥 T Σ -1 𝑥 , formed by Gaussian spheres, as the observation space Z. Although we receive a sample signal of 𝑁 Gaussian spheres from Z, it often contains noise, such as neural network performance fluctuations or inherent dataset noise, compromising the reconstruction quality. To enhance the accuracy of estimating characteristic parameters from noise-affected samples within the observation space Z, we introduce a parameter space 𝜃 , where 𝜃 0 = 𝛽 and 𝜃 1 = 𝜈. Here, 𝛽 dynamically adjusts the shape of the GEF, and 𝜈 adaptively tunes the center frequency of the band-pass filter.Utilizing maximum likelihood estimation, we refine β and ν from the noise-contaminated samples of Z. Subsequently, we adjust the Gaussian spheres within Z to compose the estimation space Y, comprising three distinct basis functions: Gaussian, GEF, and DoG. This process, illustrated in Fig. 1 , enables more accurate estimation and modification of the observation space, leading to enhanced scene reconstruction in low memory consumption.The maximum likelihood estimate of GEF is discussed in work [4] , and this estimate is approximated in work [10] to a smoother function:EQUATIONwhere 𝛽 ∈ [0, +∞), (3) is a monotone increasing function.(3) gives an estimate of the covariance in the estimation space, i.e., σ = 𝜙 (𝛽)𝜎.For the maximum likelihood estimation of DoG, we analyzed as follows.Theorem 3.1. (Maximum Likelihood Estimation of DoG) The maximum likelihood estimate of the DoG function with respect to the covariance:EQUATIONProof. The likelihood function of DoG is as follows, and cross terms in the formula are removed for easy calculation:EQUATIONThen set 𝜇 = 0 and take the logarithm of both sides:EQUATION)Take the derivative of both sides with respect to 𝜎:EQUATIONLet 𝜕 ln 𝐿 (𝑥;𝜎 ) 𝜕𝜎 = 0, we have:EQUATIONSince√︃ 𝑁 𝑖=1 𝑥 2𝑖 is a statistic of 𝜎, we have the estimation formula:EQUATION)Since the domain of 𝜙 (𝜈) does not contain 0 and is not differentiable at 1 2 , in order to make the formula smooth, we let 𝜈 ∈ [2, +∞). Through 𝜙 (𝛽) and 𝜙 (𝜈), we can use these two formulas to modify the covariance of the Gaussian function in the observation space Z, and obtain the covariance under GEF and DoG respectively, so as to construct the estimation space Y of the span of the three basis functions. Using an approach similar to work [10] , we set Ŝ (𝜃 ) = 𝜙 (𝜃 )𝑆, which has the same benefit as directly modifying the covariance matrix Σ of 3DGS [14] . Because the covariance matrix Σ is a real symmetric semidefinite matrix. Σ = 𝑅𝑆𝑆 T 𝑅 T , where 𝑅 is the identity orthogonal matrix, 𝑆 is the diagonal matrix, and the elements in 𝑆 are the root of the eigenvalues. The scaling matrix 𝑆 controls the degree of anisotropy of the covariance matrix. 
methods 4.1 preliminary: 
 The geometry of 3DGS [14] , begins with the mathematical expression of a 3D Gaussian function:EQUATIONIt then describes how 2D projection is achieved through a viewing transformation (𝑊 ) and the Jacobian (𝐽 ) of an affine projective transformation: Additionally, it explains the positive semi-definite covariance of the system, where the covariance matrix (Σ) is decomposed into the product of scale (𝑆) and rotation (𝑅) matrices, denoting positive semi-definite covariance:EQUATIONEQUATION3DGS [14] leverages the 3-dimensional Gaussian functions, rasterization methods and adaptive control (including cloning, splitting and pruning methods) to approximate a 3D scene.The densify and clone operation in 3DGS [14] relies on thresholds for position gradient and maximum value of the scaling matrix 𝑆. 3DGS [14] suggests that large position gradients indicate poor geometric reconstruction, while small maximum values of 𝑆 trigger cloning of small Gaussian spheres, potentially leading to excessive memory usage and reduced computational efficiency. This highlights the need to enhance the densify and clone operation to address underreconstruction issues.In order to make the representation more compact, we allocate the Gaussian form based on the shape priors. For the underreconstruction region, we avoid cloning a large number of small Gaussian spheres by adding two basis functions, GEF and DoG, on the basis of Gaussian function. The introduction of these two basis functions with different properties can make our Tangram-Splatting algorithm more compact in reconstruction like a Tangram puzzle, thus compressing memory. We define the DoG formula for position 𝑥 and the symmetric positive definite covariance matrix Σ in three-dimensional space as follows:EQUATIONwhere 𝑅 is the rotation matrix of Σ and 𝑆 is the scaling matrix of Σ. 𝜈 is the step length between covariance of two Gaussians, always larger than 2, and is updated by the neural network, while 𝜈 also controls the anisotropic properties and center frequency of the DoG. The GEF formula for position 𝑥 and covariance matrix Σ in three-dimensional space is defined in [10] :EQUATION)where 𝛽 is the shape parameter of GEF and updated by network. Thus, Eq. ( 10), (13) , and ( 14) constitute the three basis function representations of Tangram-Splatting.Inspired by the signal estimation theory in Sec. 3.2 and [10], we do not need to explicitly construct three models for Eq. ( 10), (13) , and (14) . We only estimate and modify the parameters of the covariance matrix of the observation space Z through the maximum likelihood theorem in the estimation space Y. This method not only saves the training time, but also looks like the three basis functions spanning the estimation space. Since the essence of the modified covariance matrix 𝜎 is to correct the anisotropy property of the Gaussian function, and the anisotropy property is reflected by the eigenvalues of the matrix, we only need to correct the scaling matrix 𝑆 to achieve this purpose (because 𝑆 is a diagonal matrix formed by the square root of the eigenvalues). The correction method is defined by the following two formulas:EQUATIONand Ŝ (𝜈) = 𝜙 (𝜈)𝑆. (16) Considering that it is necessary to increase the number of points appropriately to achieve a better reconstruction effect in complex scenes, we perform densify and clone operations in the first n iterations. This allows the algorithm to use the original Gaussian spheres to reconstruct the scene over a wide coarse-grained range. Then, in order to avoid the scene being filled with a lot of small Gaussian spheres, instead of densify and clone operations, we replace the Gaussian spheres that satisfy the clone condition with GEF or DoG which are optimized by the network. The choice of GEF or DoG depends on the degree of anisotropy of the Gaussian spheres satisfying the cloning condition. When the largest eigenvalue of the matrix 𝑆 is larger than k times of the second largest eigenvalue (we set 𝑘 = 5), we consider its anisotropy property to be significant, so the DoG with adaptive center frequency is used to replace the Gaussian spheres. If not, GEF is used instead. 
experiments 5.1 datasets and metrics: 
 To assess the efficacy of our approach in memory compression and training speed enhancement, we utilized the eigenvalues of the Gaussian spheres as shape priors to construct three types of basis functions and conducted experiments utilizing a diverse array of real-world scene datasets. Specifically, we focused on two scenes sourced from the Tanks&Temples dataset [16] , nine scenes from the Mip-NeRF360 dataset [2] , and two scenes provided in Deep Blending [11] . These datasets were deliberately selected to align with those utilized in the evaluation of 3DGS [14] , ensuring comparability and facilitating a comprehensive assessment of our method's performance.We compared the average training time and memory consumption required to store optimized parameters after 40,000 training iterations. It is important to note that the stored parameters of point clouds remain consistent with those used in 3DGS [14] . Furthermore, we employed standard metrics such as PSNR, L-PIPS, and SSIM to quantitatively evaluate the reconstruction quality of test views. To maintain consistency and avoid confusion regarding the current state-of-the-art (SOTA) methods, we extracted numerical results from previous works [2, 10, 14] and included them in Table 1 .We utilized an A100 GPU for the majority of our tests. The learning rate for the step parameter of the covariance of the DoG was set to 0.00025, with a step reset interval of 1000 iterations. The learning rate for the shape parameter of the GEF, as well as the shape reset interval and the density gradient threshold, were set according to the specifications in GES [10] . Other hyperparameters and design choices were consistent with those of 3DGS [14] . Additionally, to Ground Truth 3DGS GES OURS Mip-NeRF360Figure 3 : Qualitative Results. We compare our method to GES [10] , 3DGS [14] and Mip-NeRF360 [2] . From the top to bottom are the scenes BICYCLE, GARDEN, STUMP, COUNTER, ROOM from the Mip-NeRF360 dataset [2] , PLAYROOM and DRJOHNSON from the Deep Blending [11] , TRUCK and TRAIN from the Tanks&Temples dataset [16] . The reconstruction performances of Tangram-Splatting are comparable to GES [10] and 3DGS [14] . [23] and Mip-NeRF360 [2] , have limited memory as they rely on slower neural networks for decoding. Notably, superior performance is highlighted in red.𝑆𝑆𝐼 𝑀 ↑ 𝑃𝑆𝑁 𝑅 ↑ 𝐿𝑃𝐼 𝑃𝑆 ↓ Train ↓ Mem ↓ 𝑆𝑆𝐼 𝑀 ↑ 𝑃𝑆𝑁 𝑅 ↑ 𝐿𝑃𝐼 𝑃𝑆 ↓ Train ↓ Mem ↓ 𝑆𝑆𝐼 𝑀 ↑ 𝑃𝑆𝑁 𝑅 ↑ 𝐿𝑃𝐼 𝑃𝑆 ↓ Train ↓ Mem ↓ Plenoxels 0.accommodate differences in texture details among images from various datasets, we allowed densify and clone operations for different scenes with fewer iterations, as elaborated in the Appendix. 
quantitative results: 
 We compared our method with 3DGS [14] , GES [10] , Mip-NeRF360 [2] , InstantNGP [23] and Plenoxels [6] on the Tanks&Temples [16] dataset, the Mip-NeRF360 dataset [2] and two scenes provided in Deep Blending [11] . It is important to note that our Tangram-Splatting approach primarily focuses on leveraging basis functions with distinct properties to achieve a more compact representation.As such, we only adopted the GEF basis functions from the GES algorithm [10] , and did not incorporate other innovative enhancements introduced in GES [10] into our algorithm. The quantitative results are presented in Table 1 .Compactness. From the perspective of compact representation, we compared our Tangram-Splatting algorithm, utilizing three different types of anisotropic basis functions, with 3DGS [14] using solely anisotropic Gaussian spheres, and GES [10] employing only GEF as the basis function. As illustrated in Table 1 , for the Tanks&Temples dataset, our method incurs a memory cost after 40,000 iterations that is 0.365 times that of the memory produced by 3DGS [14] , and 0.676 times that of GES [10] . Our PSNR is increased by 0.08 compared to 3DGS [14] while being decreased by 0.13 as compared to GES [10] , the difference is marginal.Given the finer texture of scenes in the Mip-NeRF360 dataset [2] , we allowed certain iterations of density and clone operations for certain scenes based on our experiences (e.g., BICYCLE, STUMP, and GARDEN). Even with this adjustment, the memory cost after 40,000 iterations remains significantly lower, at 0.406 times and 0.790 times that of 3DGS [14] and GES [10] , respectively, albeit with a slight decrease in PSNR by about 0.26 compared to 3DGS [14] and an increase of 0.04 compared to GES [10] .Regarding the scenes from Deep Blending [11] , the PSNR is increased by 0.18 compared to 3DGS [14] and only experiences an average decrease of 0.09 compared to GES [10] , while the memory cost after 40,000 iterations is merely 0.349 times and 0.591 times that of 3DGS [14] and GES [10] , respectively.In Table 1 , it is evident that our algorithm significantly reduces memory consumption while maintaining reconstruction quality. Comparatively, our compression ratio indicates a remarkable reduction of 62.4% in memory usage compared to 3DGS [14] , with GES [10] achieving a slightly lower reduction of 43.2%. In terms of PSNR, our algorithm's performance falls between that of 3DGS [14] and GES [10] , with marginal deviations being observed.Computational efficiency. By reducing the number of point clouds, minimizing the computation of numerous small Gaussian spheres, and streamlining densify and clone operations, our method significantly decreases training time across the three aforementioned datasets. Additionally, in the replacement operations introduced in Tangram-Splatting between different basis functions, we optimize the process by replacing the computation of eigenvalues of the covariance matrix Σ with direct sorting of the scaling matrix 𝑆, leveraging the mathematical properties of the covariance matrix. This enhancement further contributes to the reduction in training time. As indicated in Table 1 , the training time for Tangram-Splatting is decreased by an average of 22 minutes, 11 minutes, and 13 minutes, respectively, compared to 3DGS [14] .While compressing memory, we also need to consider the reconstruction effect of Tangram-Splatting. The qualitative results on the Tanks&Temples dataset [16] , the Mip-NeRF360 dataset [2] and the two scenes provided in Deep Blending [11] are shown in Fig. 3 . The results show that the overall effect of Tangram-Splatting is comparable to the reconstruction effect of previous work, which is hard to tell the differences of rendered images between Tangram-Splatting and previous work. Only in some details (such as reflective Windows in the GARDEN scene from the Mip-NeRF360 dataset [2] , shadows in the sky in the TRAIN scene from the Tanks&Temples dataset [16] ) the performances are lower than the Ground Truth.We show the point clouds of the scene FLOWERS, COUNTER, KITCHEN and ROOM from the Mip-NeRF360 dataset [2] from top to bottom, as shown in Fig. 4 . We showcase the comparisons of point clouds produced by our Tangram-Splatting and 3DGS [14] . We observe that both Tangram-Splatting and 3DGS [14] effectively [14] . We can infer from the images that our method can reduce the number of Gaussians needed in reconstruction while maintaining the PSNR performance.capture the outline contour of the 3D scene. However, Tangram-Splatting achieves this with a significantly smaller number of Gaussian counts, achieved by reducing redundant Gaussians that are unnecessary for 3D reconstruction. We observe that our point clouds are more sparsely distributed in specific areas: the white wall behind the bicycle in the FLOWERS scene and the white wall behind the sofa in the ROOM scene. This suggests that Tangram-Splatting tends to reconstruct flat areas with fewer functions but larger volumes.We re-run the GES [10] algorithm with the same parameter settings as Tangram-Splatting to highlight the robustness of our algorithm by comparing the reconstruction performance with approximately the same memory. To ensure memory consistency and accurately We do the ablation study on the Deep Blending dataset [11] . We try to make the Gaussian count of GES [10] to our algorithm to see whether our representation is better than GES's [10] (whose paper has already shown it is better than 3DGS's [14] ). We can indicate in the table that our algorithm is a better representative method, when our performance is slightly better and still reduces a lot of memory cost.Ground Truth GES OURS assess the reconstruction performance of both algorithms, we maintain the density iterations of GES [10] in line with those of Tangram-Splatting, while leaving the remaining parameters of GES [10] unchanged. We evaluate the entire scenes from the Deep Blending dataset [11] . The quantitative results presented in Table 2 showcase that, despite our lower memory consumption, our performance consistently excels. Furthermore, the qualitative results, illustrated in Fig. 5 , highlight the proficiency of our method in reconstructing intricate details, such as straight lines. We observe that the lines in the magnified region of GES [10] appear more blurred compared to ours, and there is a pronounced white artifact present in the lower left corner, as shown in Fig. 5 . This enhancement can be attributed to the inherent anisotropy of GEF and DoG. 
conclusion and future work: 
 In this paper, we put forward a new method for a more compact representation for 3D reconstruction. We leverage signals and systems perspective to make the base function more compact than vanilla Gaussian function. However, our method somewhat revisits the problem from the last century of designing base functions in the field of signal processing. In the future work, we may borrow the idea of implicit representation from Neural Radiance Field [21] to use networks to represent a base function (while maintaining the same speed) instead of manually designing a new compact base function.  
 
 Appendix:  


Decision: The reviewers unanimously voted to accept this paper. Therefore, we are happy to recommend the acceptance of this paper.

Reviews: review: paper_summary: paper_summary: The paper introduces a novel approach to 3D reconstruction, incorporating shape priors inspired by the Tangram puzzle, an ancient Chinese puzzle, to optimize the 3D Gaussian Splatting method to improve memory efficiency and reduce computational overhead in 3D scene reconstruction. The approach integrates different Gaussian functions to refine scene fitting, achieving notable reductions in memory usage and training time. The paper presents a comprehensive evaluation of the proposed method, Tangram-Splatting, against existing techniques using various datasets and metrics.
strength_weakness: strength_weakness: 1. The paper presents an innovative approach to 3D reconstruction by introducing shape priors inspired by tangram, which is a unique perspective in the field.
2. The paper provides a thorough empirical evaluation, demonstrating significant memory reduction and improved training time, which are crucial for practical applications.
limitations: limitations: Insufficient evaluation:
1. The rationale for incorporating both the Gaussian function and GEF, along with DoG, into the proposed Tangram-Splatting approach is not entirely clear. Given that the Gaussian function is a specific instance of GEF,  it raises the question of why both are utilized concurrently. Ideally, GEF should encompass the functional capabilities of the Gaussian function by adjusting $\beta$  accordingly.  The paper could benefit from a deeper exploration or justification for the simultaneous use of these specific functions, particularly focusing on whether the Gaussian function introduces any unique benefits that a variable $\beta$ in GEF does not already provide.

2. Clarification of Step Size:
    1). The justification provided for restricting the parameter $\nu$ to the range $[2, +\infty)$ is somewhat ambiguous. The paper states that this range is chosen because the function $\phi(\nu)$ is not differentiable at $\nu = \frac{1}{2}$ and does not contain 0. However, the connection between these mathematical properties and the practical implications for choosing $\nu \geq 2$ needs to be more clearly articulated. 
    2). Additionally, the impact of varying step size $\nu$ on the reconstruction quality and efficiency is a critical aspect that seems underexplored. 

3. Proportion of Each Basis Function
    The manuscript could benefit from including detailed information about the proportion in which each basis function is utilized. Detailed statistical or visual representations of their usage across various datasets or scenes would provide clearer insights into the method’s operational dynamics.

4. Necessity of DoG in Ablation Study:
 An ablation study that methodically examines the reconstruction performance with and without the DoG component would offer a clearer perspective on its efficacy. The baseline method GES does not include Gaussian function and DoG, and thus the comparison between GES can not directly demonstrate the efficacy of DoG.

Lack of novelty:
5. Given the weaknesses highlighted in points 1 and 4, the method may not sufficiently demonstrate new contributions over existing work. (Compared to the baseline method GES)

Minor:
1. Figure 3 would benefit from the addition of visual quality metrics directly on each rendered image. This would provide a clearer, immediate understanding of the performance differences between the presented methods.
2. The captions for figures should be more concise. Descriptive details of results should be relocated to the main text to allow captions to focus solely on describing the visual content.
paper_summary: paper_summary: This paper proposes a hybrid primitive representation to improve the compactness of 3D Gaussian splatting. Specifically, in 3D Gaussians, and generalized exponential functions (GEF) are leveraged to represent low-frequency signals of the scene and differences of Gaussians (DoG) are used as band-pass filters to effectively represent objects with sharp shapes. Experimental results demonstrate that the proposed hybrid primitive representation can effectively reduce the number of primitives required for scene modeling.
strength_weakness: strength_weakness: 1. The proposed hybrid primitive representation is technically reasonable, as this representation can leverage the advantage of three types of primitives to handle the distinct properties of different objects.
2. The theoretical analysis in Sec.3 is comprehensive and detailed.
3. Experimental results show that the proposed hybrid representation can significantly reduce the model size and training time, which demonstrates the effectiveness of the proposed method.
limitations: limitations: 1. The smoother function of GEF shown in Eq. 3 is different from the form in work [10], wherein the function in work [10] has the hyper-parameter $/rho$
2. It would be better to add more discussion and details about the derivation of Eq. 4. For example, authors should discuss the rationale for ignoring cross terms in the likelihood function of DoG.
3. Authors should explain the definition of the field space $\sum_{j=1}^{N}\omega_j e^{-\frac{1}{2}\boldsymbol{x}^T\Sigma^{-1}\boldsymbol{x}$. For example, the definition of $\omega_j$ should be clarified.
4. It would be better to identify which primitives belong to DoG and which primitives belong to other types of primitives in Fig. 4.
paper_summary: paper_summary: This paper introduces Tangram-Splatting, an approach for 3D scene reconstruction using different basis functions. By leveraging the strengths of Gaussian, Generalized Exponential, and Difference of Gaussians functions, Tangram-Splatting achieves a more compact representation of the scene with reduced memory usage and increased computational speed compared to existing methods. The article also provides theoretical analyses of the properties of these basis functions and highlights the advantages of using diverse functions for scene reconstruction.
strength_weakness: strength_weakness: The advantages of this paper can be summarized as follows: 

1. Innovation: The paper proposes a novel approach by using different types of base functions to optimize the 3D Gaussian Splatting algorithm. This method allows for a more compact representation of 3D scenes compared to traditional Gaussian function representations. 

2. Theoretical analysis: The paper provides theoretical analyses of different types of base functions, revealing their properties in the time and frequency domains. This analysis helps to better understand the selection and usage of base functions and their role in the reconstruction process.
limitations: limitations: There are some comments for this paper:
1. The authors claim that the proposed method performs well in detail reconstruction. It seems difficult to distinguish between "GES" and "ours" in Figure 3. It would be beneficial to enhance the visualization by zooming in on specific details in Figure 3.
2. The authors do not mention the loss function. For GES, it gives the design of the loss function and conducts ablation studies for it.
3. I would like to know how many training iterations were used for the other comparison experiments, as I understand that training iterations can have an impact on the experimental results. Additionally, I'm interested in understanding the experimental settings of the comparative methods. Could you please provide me with this information?
4. The author claims that this method can reduce memory usage and, compared to GES-4K, there might be a slight performance drop. Would it be possible to compare the memory usage at the same performance level?

suggestion:
1. The author should provide some explanations about the models "3D Gaussians-7K" and "3D Gaussians-30K" for better understanding. Different numbers represent different iterations number？
2.  The authors should provide explanations for all the colors used in Table 1.
paper_summary: paper_summary: The method proposes a tangram-like Gaussian Splatting strategy to represent the target scene more efficiently by using three different types of Gaussians simultaneously. This strategy can reduce the training time and storage overhead without losing the rendering quality.
strength_weakness: strength_weakness: 1. By using Gaussians of different types, redundancy can be reduced when representing a scene, making the representation more compact and efficient.
2. For the training time, since it optimizes the number of Gaussians based on the shape prior，
the method does not increase the training time but rather reduces it, although it introduces other computations.
limitations: limitations: This hybrid representation is very interesting and could be a future trend, but there are still some problems in terms of this paper:
1. This paper does similar work as the Gaussian lightening methods with the same purpose of removing redundant Gaussians or their parameters, so I think it would be better to compare the lightening methods in the experiments to emphasize the advantages of the method to the lightening methods in terms of the training time or the rendering quality.

2. It can be understood that the GEF is used to express the areas with smooth color change, but as long as it changes its β, it can also be turned into a Gaussian distribution with a sharp peak. So, wouldn't it already be able to express the two different cases (smooth color change and dramatic color change) by itself? So why is the original Gaussian still needed, which is missing some analysis?
In addition, the paper claims that GEF loses its Gaussian property in the frequency domain, which could lead to non-convergence, and therefore, GEF is problematic. So why does the paper still use GEF and doesn't modify it, which lacks an explanation?

3. The paper didn't show the effect of DoG in ablation. As the authors said, DoG is more likely to express scene details. However, a Gaussian can only carry one color, so it still can't express too complex textures even if it has a more complex spatial domain distribution. Maybe it can express repetitive textures better. Thus, I doubt whether this DoG is useful or not. 
In addition, in the ablation study, the authors should give some visual results to show more intuitively what scenarios different types of Gaussians are more inclined to express and whether they are consistent with what the authors analyzed in section 3.

4. Also, the paper doesn't prove that the types of Gaussians chosen are reasonable. While the idea of using different types of Gaussians simultaneously is correct, I don't think the artificial selection of the types of Gaussians is reasonable.

5. Fig.3 shows little difference. It would be better to modify this figure to reflect the advantages of the proposed method.