Effective Rank Analysis and Regularization for Enhanced 3D Gaussian Splatting

Authors: junha  hyung
susung  hong
sungwon  hwang
jaeseong  lee
jaegul  choo
jin-hwa  kim
  kaist
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-11 15:47:41

Abstract: 3D reconstruction from multi-view images is one of the fundamental challenges in computer vision and graphics. 
Recently, 3D Gaussian Splatting (3DGS) has emerged as a promising technique capable of real-time rendering with high-quality 3D reconstruction. This method utilizes 3D Gaussian representation and tile-based splatting techniques, bypassing the expensive neural field querying. Despite its potential, 3DGS encounters challenges, including needle-like artifacts, suboptimal geometries, and inaccurate normals, due to the Gaussians converging into anisotropic Gaussians with one dominant variance.
We propose using effective rank analysis to examine the shape statistics of 3D Gaussian primitives, and identify the Gaussians indeed converge into needle-like shapes with the effective rank 1. To address this, we introduce effective rank as a regularization, which constrains the structure of the Gaussians. Our new regularization method enhances normal and geometry reconstruction while reducing needle-like artifacts. The approach can be integrated as an add-on module to other 3DGS variants, improving their quality without compromising visual fidelity.

Main paper: 
 introduction: 
 Creating 3D models from multiple images is a central challenge in computer vision and graphics. Neural Radiance Fields (NeRF) [20] have revolutionized this area by demonstrating remarkable capabilities in novel view synthesis through implicit neural fields and differentiable rendering techniques. Despite their impressive 3D reconstruction quality, the training and rendering processes of NeRFbased methods are computationally intensive, posing significant challenges for real-time applications. To improve training and rendering efficiency, various acceleration techniques, such as baking with shell [12, 32] and grid representations [5, 21] , have been introduced. While these solutions enhance efficiency to some extent, there are still limitations for real-time interactive scenarios.Recently, 3D Gaussian Splatting (3DGS) has emerged as a promising technique capable of realtime rendering with high-quality results. This method utilizes 3D Gaussian representations and tile-based splatting techniques instead of expensive neural field querying, making it feasible to apply the technique in practical applications. This opens up new possibilities in areas that require faster rendering, such as virtual and augmented reality, gaming, and real-time avatars.However, despite its potential, 3DGS encounters several challenges in terms of geometry reconstruction, including noisy rendering results with needle-like artifacts, especially in novel and extreme views far from the training images. These issues stem from the primitive-based nature of 3DGS, where individual primitives lack geometric constraints.For accurate geometry reconstruction, it is well known that the density field should be concentrated near the surface [30] . To this end, previous efforts, such as SuGaR [10] , have focused on regularizing the 3D Gaussians to be flatter, i.e., regularizing the primitives into anisotropic Gaussians with one of its variance very small. Similarly, 2DGS [13] utilizes 2D Gaussians instead of 3D Gaussians to force this effect.However, while the flatness of Gaussians is necessary to make them align well with the surface, we argue that flatness alone is not sufficient for accurately representing surface geometry. Specifically, we observe that the majority of Gaussians converge into anisotropic forms with one dominant variance in 3DGS, effectively becoming needle-like with small scales along two of their axes. We identify this phenomenon as an important factor hindering accurate reconstruction, as needle-like Gaussians cover a negligible portion of the surface and create spiky artifacts. Disk-like Gaussians that cover non-negligible areas are actually needed for reconstructing the surface. However, previous methods do not properly distinguish between disk-like and needle-like Gaussians, as both have one of their scales near or exactly zero. In fact, we observe that in previous works, the majority of Gaussians converge into needle-like shapes.To directly examine the shape statistics (whether their geometries are disk-like or needle-like) of 3D Gaussian primitives and understand their structural changes during training in a differentiable manner, we first propose performing effective rank analysis on the covariance matrices of Gaussians. The effective rank [25] , which is a real-valued and differentiable extension of integer rank, can be utilized to monitor the training dynamics and structural transformations of Gaussian primitives. Indeed, our analysis reveals that the effective ranks of Gaussians approach an effective rank of 1 (erank-1), resulting in needle-like shapes in 3DGS and other methods, such as SuGaR [10] and 2DGS [13] .Additionally, we propose using effective rank as a regularization term to constrain the structure of the Gaussians. The differentiable nature of effective rank, with its concave logarithmic term providing stable gradients, makes it directly applicable to continuous optimization problems. Our new regularization method enhances normal and geometry reconstruction while reducing needle-like artifacts, particularly in novel view scenarios. Furthermore, our effective rank regularization can be applied as an add-on module to other 3DGS variants, improving their quality.The main contributions of our work are as follows:• We are firstly analyzing the dynamics of Gaussian primitive structures using effective rank in the optimizing process, discovering that Gaussians converge into anisotropic forms with one dominant variance.• We propose an effective rank regularization method that alleviates needle-like artifacts in 3DGS rendering and improves geometric reconstruction.• Our approach is an add-on module that can be integrated with other 3DGS variants, and demonstrate that our method enhances 3D geometry reconstruction without compromising visual quality. 
related work: 
 Novel view synthesis Neural Radiance Fields (NeRF) [20] have revolutionized photo-realistic rendering from novel viewpoints by introducing a neural implicit representation of 3D scenes. This approach uses high-frequency positional encoding and differentiable volume rendering to achieve unprecedented realism. Enhancements to NeRF address challenges like anti-aliasing [1, 3] , parameterizing unbounded scenes [2, 37] , and training from in-the-wild images [19, 8, 29] through probabilistic transience modeling. Further improvements reduce training time and enhance rendering quality by incorporating low-rank tensor components [5] .Other research efforts have aimed for real-time rendering using alternative implicit models that do not rely on MLPs. Notable examples include sparse voxel grids [9] and multi-resolution hash encoding [21] . Despite these advancements, ray tracing methods are inherently slower than rasterization.To address this, 3D Gaussian Splatting (3DGS) [16] introduced a point-based rasterization technique for real-time, high-fidelity view synthesis. Inspired by EWA Volume Splatting [39] , 3DGS uses a fully differentiable pipeline, representing 3D scenes with 3D Gaussians and performing volume splatting to known camera poses for rasterization.Surface reconstruction Surface reconstruction is a critical area in computer vision and graphics, aiming to recreate 3D shapes and structures from 2D images or other data forms. Among recent innovations, NeuS [30] leverages volume rendering and signed distance functions (SDF) for highfidelity reconstructions. NeuS2 [31] significantly improves training speed and extends modeling capacity to dynamic scenes. UNISURF [22] integrates implicit surface models and radiance fields for both surface and volume rendering. VolSDF [33] models volume density as a function of geometry, achieving high-quality geometry reconstructions. Neuralangelo [18] uses multi-resolution hash grids and neural surface rendering to recover detailed structures. BakedSDF [34] introduces a hybrid neural volume-surface representation optimized for mesh extraction.Recent advancements in 3D Gaussian Splatting (3DGS) have further propelled surface reconstruction.NeuSG [6] refines surface details using 3D Gaussian Splatting and neural implicit models. SuGaR [10] focuses on mesh extraction with SDF-based regularization and Poisson reconstruction. 2DGS [13] collapses 3D volumes into 2D Gaussian disks for view-consistent geometry and detailed mesh reconstruction. GaussianShader [15] enhances rendering quality in reflective surfaces using a shading function on 3D Gaussians. GOF [36] utilizes ray-Gaussian intersection for density estimation and geometric regularization. GIR [28] employs 3D Gaussians for inverse rendering, enabling accurate estimation of material properties, illumination, and geometry. These advancements showcase the potential of 3DGS for high-speed, detailed, and versatile surface reconstructions. 
preliminaries: 
 3.1 3D Gaussian splatting 3D Gaussian Splatting (3DGS) [16] represents a scene with a set of learnable 3D Gaussian primitives{G k | k = 1, • • • , K}, where each 3D Gaussian G k consists of mean µ k ∈ R 3×1 , covariance Σ k ∈ R 3×3 , point opacity α k ∈ [0, 1] and view-dependent color c k in spherical harmonics. Covariance matrix Σ k = R k S k S ⊤ k R ⊤ k is positive semi-definite, where S k = diag(s k) is a scaling matrix, s k = (s k1 ; s k2 ; s k3 ) ∈ R 3×1 is a scale parameter, and R k ∈ R 3×3 is a rotation matrix parameterized by a quaternion. A 3D Gaussian primitive can be represented in 3D space as:EQUATIONThe primitives are then rasterized via differentiable volume splatting. Specifically, a 3D Gaussian is projected to 2D screen space asΣ ′ k = JWΣ k W ⊤ J ⊤ ,where W is a world-to-camera transform and J is the Jacobian of the affine approximation of the projection matrix [39] . The covariance and mean of the projected Gaussian G 2D k (x) are then obtained by removing the third column and row of Σ ′ k and simply projecting µ k to screen space, respectively. Finally, the Gaussians are alpha-blended in the order of depth as:EQUATIONwhere u is a screen space coordinate. The rendered images are supervised with photometric loss L for 3D primitive optimization similar to NeRF [20] .As Gaussians are initialized by sparse SfM points, Adaptive Density Control (ADC) is designed for densification during optimization. Specifically, ADC subsamples and splits Gaussians that satisfy the condition:EQUATIONwhere P and p i denote a set of pixel indices and the i-th pixel, respectively, and τ is a predefined threshold. The intuition behind Eq. 3 is that regions not yet well reconstructed exhibit large viewspace positional gradients. This occurs because the optimization process attempts to move the Gaussians to correct these areas, so densifying such Gaussians can effectively increase expressibility.Consider a real-valued non-all-zero M × N matrix A. The singular value decomposition (SVD) of A can be expressed as A = UDV, where U and V are unitary matrices of sizes M × M and N × N respectively, and D is a diagonal matrix of size M × N containing the real positive singular values in descending order:EQUATIONwhere L = min{M, N }. The singular value distribution is then defined asEQUATIONwhereσ = (σ 1 , σ 2 , • • • , σ L ) T , and ∥ • ∥ 1 denotes ℓ 1 -norm.Definition 1 (Effective rank). The effective rank of the matrix A is concisely defined as erank(A) = exp{H(q 1 , q 2 , • • • , q L )}, where H(q 1 , q 2 , • • • , q L ) is the Shannon entropy given byEQUATION 
method: 
 In Section 4.1, we introduce effective rank analysis to inspect the geometries of Gaussians of 3DGS and its variants, shedding light on their underlying structures. Based on the findings from our effective rank analysis, we propose a novel effective rank regularization method in Section 4.2.We propose to analyze the effective rank to investigate the structural dynamics of individual 3D Gaussians by calculating the effective rank of the covariance matrix of the Gaussians. The covariance matrix of the 3D Guassians is defined asΣ k = R k S k S T k R T k, and the diagonal matrix after SVD is D = S k S T k , with real positive singular values in a descending order as follows:EQUATIONwhere we omit subscript k of s k for brevity.Accordingly, we can derive the effective rank of a 3D Gaussian G k with the covariance matrix Σ k . The entropy term is H(G k ) := H(q 1 , q 2 , q 3 ) := -EQUATION, where S =EQUATIONand the effective rank of a 3D Gaussian G k with covariance matrix Σ k is defined as follows:EQUATIONThe effective rank, being a differentiable extension of integer rank, is a suitable tool for geometric analysis of 3D Gaussians since it jointly considers all of the scale parameters and can identify the relative scales of the three axes. The advantage of effective rank becomes more apparent when compared to recent works that only analyze individual or pair-wise variances of the 3D Gaussians [15] . Such approaches do not fully represent the geometry of Gaussians, potentially leading to planar and needle-like Gaussians being categorized together. For better understanding, we visualize effective ranks of sphere and 2D disks in Fig. 3 .With the distinct advantage of our approach, we can differentiate between needle-like Gaussians, which have effective ranks close to 1, and planar disk-like Gaussians. To reconstruct a scene with an accurate surface, we need Gaussians that represent a plane that aligns and concentrates well with the surface [30] . Ideally, 3D Gaussians with erank(G k ) ≈ 2 are preferred, but Gaussians with effective rank smaller than 2 are also required for representing thin and elongated objects and patterns. Needle-like Gaussians with erank(G k ) ≈ 1 are undesirable because they account for a negligible region of the surface and produce degenerate results in novel views.The first row of Fig. 2 (green graph) shows the effective rank histogram for 3DGS during training.As the model converges, the number of 3D Gaussians with erank(G k ) ≈ 1 increases, indicating overfitting without improvements in PSNR and Chamfer distance metrics (metrics are provided in the Appendix A.5, Table 5 ). This indicates that the majority of "flat" Gaussians (singular values close to 0) are actually needle-like (erank(G k ) ≈ 1), rather than disk-like (erank(G k ) ≈ 2). It is also interesting to note that 3DGS naturally forms a small mode at erank(G k ) = 2, indicating a observed preference that can be further strengthened with our regularization.Despite having different geometric constraints on the Gaussians, SuGaR [10] (the second row in Fig. 2 ) and 2DGS [13] (the third row in Fig. 2 ) also exhibit a similar tendency to have a large amount of needle-like Gaussians with a single dominant variance along an axis. Notice that all Gaussians in 2DGS start with an effective rank of exactly 2, but the majority still fail to remain disk-shaped and instead become needle-like 2D Gaussians.The real-valued and differentiable nature of the effective rank allows us to utilize it as a regularization objective to impose structural constraints on 3D Gaussians. Specifically, our goal is to keep the effective rank of 3D Gaussians below 2, thereby promoting planar shapes, while penalizing Gaussians with an effective rank close to 1 to minimize needle-like artifacts. Although disk-like Gaussians with erank(G k ) ≈ 2 are preferred, shapes with erank(G k ) < 2 are also essential for representing complex geometries. We propose an effective rank regularization term that increases exponentially as the effective rank nears 1, strongly penalizing such Gaussians:EQUATIONwhere ϵ = 1 × 10 -5 ensures numerical stability, and s 3 is the smallest scale parameter of G k . The regularization effectively constrains the effective rank of Gaussian primitives when added to the baselines, as shown in the purple graphs of Fig. 2 . Also, the regularization is scheduled to be applied from 7000-iteration, adhering to the coarse-to-fine training paradigm, which enables stable training upon early iterations with erank(G k ) > 2 Gaussians.We adopt the revised version of the densification algorithm presented in [4, 36] , which densifies Gaussians based on the summation of norms instead of the norm of the summation in Eq. 3 (further details in Appendix A.4). This change is particularly important for our regularization method. Unlike thin, needle-like Gaussians, disk-like Gaussians often do not meet the splitting criterion set by Eq. 3. This is because a disk-like Gaussian does not have a second axis with a much smaller variance than the axis with the largest variance. As a result, the gradient signals from each pixel are generally smaller compared to those from needle-like Gaussians. Furthermore, since disk-like Gaussians typically cover more pixel space, unaligned signals tend to cancel each other out. In contrast, the revised densification algorithm facilitates the splitting of Gaussians with our regularization. However, note that due to the efficacy of disk-like Gaussians in reconstructing the surface compared to needle-like ones, our method still requires about 10% fewer Gaussians than the baseline [16] . 
experiments: 
 We evaluate the effective rank regularization, comparing its performance as an add-on to baseline models. Additionally, we analyze the contributions of different components of the method.The regularization hyperparameter λ erank = 0.01 is used for all training. For other components belonging to the baselines, we use the same settings as described in the corresponding papers. All experiments are conducted on a Tesla V100 GPU. For mesh extraction, truncated signed distance function (TSDF) fusion with Open3D [38] is used, with details in the Appendix A.3.Dataset We evaluate our model on the DTU [14] and Mip-NeRF360 [2] datasets. The DTU dataset consists of 15 forward-facing bounded scenes with a resolution of 1600 × 1200. Following prior standards [13, 36] , we downsample the images to a resolution of 800 × 600. The DTU dataset is used for evaluating both geometry reconstruction (using Chamfer distance) and novel view synthesis. The Mip-NeRF360 dataset comprises 9 indoor and outdoor scenes with images at a resolution of 1600 × 1050 and is used exclusively for novel view synthesis evaluation. For novel view synthesis, the images are split into training and test sets, while the entire set of images is used for geometry reconstruction. COLMAP [26, 27] is used to initialize point clouds for the baselines.Baselines Our method is applicable to other baselines as an add-on term. Therefore, we compare baselines with and without our regularization. We choose SuGaR, 2DGS, and GOF as our baselines, works that focus on better geometry reconstruction, along with the original 3D Gaussian Splatting. All of the experiments are performed with the proposed setting of the original paper. Geometry reconstruction Table 1 presents the quantitative results of geometry reconstruction on the DTU dataset. We report the Chamfer distance for each scene, along with the mean Chamfer distance and mean PSNR. The "+e" symbol indicates the addition of effective rank regularization (with fixed densification) to the baseline methods.The results show that methods enhanced with our add-on term outperform the baselines. Notably, applying our regularization to 3DGS (3DGS+e) results in a significant improvement in geometry reconstruction, demonstrating the effectiveness of the regularization. This supports our hypothesis that reducing needle-like Gaussians and achieving flatness as in Fig. 2 improves performance. Additionally, the figure shows that SuGaR contains both needle-like and non-planar Gaussians with effective ranks greater than 2. By attaining flatness and removing spikes through effective rank regularization, we achieve a substantial performance gain for SuGaR (SuGaR+e).GOF and 2DGS already incorporate well-designed regularization terms, such as depth distortion loss [13, 2] , to align Gaussians with surfaces and enhance geometry reconstruction. Furthermore, 2DGS explicitly uses 2D Gaussians as their primitive, inherently achieving planarity. Nonetheless, our method prevents Gaussians from converging into needles in both approaches (and enforces flatness in GOF), resulting in performance gains.Figure 4 shows mesh reconstruction results, where baseline methods often exhibit empty holes in the reconstructed meshes. Our regularization term enforces disk-like Gaussians, reducing such holes and proving advantageous for surface reconstruction.Figure 5 and the first row of Figure 1 display normal reconstruction results. In Fig. 5 , the resulting image from GOF shows spiky artifacts and a hollow surface on the pear. Similarly to the mesh results, needle-like Gaussians often fail to cover the entire area, leaving empty holes or transparent regions, resulting in hollow or incomplete reconstructions. The effective rank regularization mitigates these noisy artifacts, leading to a more accurate reconstruction of the underlying geometry.Novel view synthesis Since 3D reconstruction from 2D images is an ill-posed problem, Gaussians tend to overfit to the training views, converging into needle-like shapes and causing spiky artifacts in test views, as shown in Fig. 1 (b), (c), and Fig. 6 . For better understanding, we visualize Gaussians Furthermore, as seen in Fig. 7 , our method adaptively preserves some elongated Gaussians when necessary, allowing the representation of slender structures. The results indicate that while 3DGS heavily relies on needle-like Gaussians to represent the scene, our method limits their use to only when required, leading to improved novel view synthesis performance.We also provide quantitative results in Table 1 , where we report the average PSNR for the DTU dataset. Results for Mip-NeRF360 are reported in Table 3 in the Appendix A.5. While many geometry regularization techniques degrade visual quality, our method does not exhibit this trade-off and actually shows slight improvements by properly constraining the shape of the Gaussians.Efficiency As shown in Fig. 7 and Table 4 , efficacy of disk-like Gaussians in 3D reconstruction, compared to needle-like Gaussians, leads to a better memory footprint. The average storage usage for the DTU and Mip-NeRF360 datasets is reported in Table 4 .Our method comprises two key components: (a) the fixed densification (ADC) algorithm and (b) effective rank regularization. We performed an ablation study on these components to observe their performance gains compared to the naive 3DGS method. Table 2 shows the Chamfer distance and PSNR measured on the DTU dataset. The results indicate that both components contribute to performance gains in geometry reconstruction and novel view synthesis tasks. Additionally, incorporating techniques such as depth distortion loss [13, 36] can further enhance the best-performing model (row +a+b+c). These techniques are discussed in Appendix A.2. 
conclusion: 
 Limitations Our regularization term constrains individual Gaussians but does not account for the local and global structure of the scene. Thus, it may be beneficial to pair our method with structure-aware regularizations, such as the depth distortion loss [13] , which considers the Gaussians along the ray collectively. Another limitation is the manual selection of the hyperparameter λ erank . While our chosen hyperparameter works well for the scenes used in our evaluation, it may not be optimal for extreme scenes dominated by thin objects and structures.A Appendix / supplemental material A.1 Broader impactThe broader impact of our work on 3D reconstruction lies in its potential to advance various fields such as virtual and augmented reality, medical imaging, and digital content creation by enabling more efficient and high-quality 3D model generation. However, like any powerful technology, it also presents potential risks and avenues for misuse. For instance, enhanced 3D reconstruction techniques could be exploited to create deepfakes or unauthorized reproductions of proprietary designs, posing ethical and legal challenges. To mitigate these risks, we propose implementing strict usage guidelines to ensure the integrity and rightful use of 3D models. We aim to maximize the positive impact of our research while minimizing potential negative consequences. 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: We discuss the societal impacts in the Appendix.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [Yes]Justification: We discuss such information in the Appendix.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: We are the original owners of code and the models.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix: For rendering normals, we add other regularization terms, such as depth distortion loss [13] and normal regularization, as proposed in [13, 36] . (We do not utilize these regularization terms for evaluating effective rank regularization as an add-on module in Table . 1.) Depth distortion loss, which concentrates splats on a surface and mitigates floater artifacts, is given asEQUATIONwhereω i = α i G i (x) i-1 k=1 (1 -α k G k (x))) and z i is the blending weight of the i-th Gaussian, and i, j are indexes over Gaussians contributing to a certain ray.Normal regularization minimizes difference between the rendered normal map n of the splats and the gradient normals n derived from the rendered depth map,EQUATIONwhich locally aligns the 3D Gaussians with the actual surfaces. Since effective rank regularization does not account for the local and global structure of the scene, it isbeneficial to pair our method with these structure-aware regularizations.We utilize the Truncated Signed Distance Function (TSDF) fusion for mesh extraction. The algorithm encodes the distance of any point in the voxel grid to the nearest surface, with the distance being truncated to a maximum value to limit the influence of faraway points. The sign of the distance function indicates whether the point is inside (negative) or outside (positive) the object. Multiple TSDFs are combined from different viewpoints to create a more accurate and complete 3D reconstruction, forming a coherent and comprehensive 3D model. The Marching Cubes algorithm is then used for triangulation.We adopt the revised version of the densification algorithm presented in [4, 36] , which densifies Gaussians based on the summation of the norm instead of the norm of the summation in Eq. 3:EQUATIONAs discussed in the main paper, this approach is crucial with our regularization because disk-like Gaussians typically cover more screen space and receive gradient signals from various pixels, which can cancel out when summed. The revised algorithm ensures effective splitting of Gaussians with our regularization. However, due to the efficiency of disk-like Gaussians in surface reconstruction, our method still requires about 10% fewer Gaussians compared to the baseline [16] . We report novel view synthesis results on Mip-NeRF360 dataset in Table 3 . The results show that our add-on regularization term improves visual quality of 3DGS in terms of PSNR, SSIM, and LPIPS. Also the method even shows comparable, or slightly better performance compared to the NeRF variants with slow and computationally intensive rending.We report the training time of our method in Table 4 . The training time for 3DGS on the DTU [14] dataset averages 11.2 minutes per scene. Adding effective rank regularization with the densification fix incurs no overhead, since the additional computation is compensated with reduced number of Gaussians. Total training time is in average 11.1 minutes for DTU dataset and 40 minutes for Mip-NeRF360 dataset, on a single V100 GPU, reported in Table 4 .Also with reduced number of Gaussians, our method requires less memory and storage for scene representation, as in Table 4 . While being more compact, our method outperforms baselines in terms of Chamfer distance and PSNR.Table 5 demonstrates Chamfer distance and PSNR changes during the course of training, for the baselines shown in Fig. 2 . Results are reported for scene 37 of DTU dataset. Needle-like Gaussians increase, but the performance plateaus, indicating overfitting. Additionally, different Gaussian structures with similar metrics suggest the heterogeneous nature of Gaussians in 3DGS and its variants. Also, the reported "Number of needles" correspond to Gaussians with effective rank smaller than 1.04. The results suggest that our regularization term effectively minimizes the number of needles without visual quality trade-off.We present per scene PSNR on DTU dataset in Table 6 . The mean PSNR is already shown in Table 1 and Table 2 of the main paper.While not directly related to our methodology, we investigate some reasons for the convergence of 3D Gaussians into anisotropic Gaussians with one dominant variance.First, the scale of the 3D Gaussians is not properly constrained due to the dilation operation, which adds a small constant to screen space Gaussians [16] to ensure a minimum scale, as noted in Mip- Splatting [35] . Combined with the inherent implicit shrinkage bias of 3D Gaussian Splatting [16, 35] , this results in the underestimation of the scale parameters during the optimization process.Second, densification along the longer axis does not occur effectively since the longer axes, or axes with large variance, have smaller gradients. When Gaussians move in the direction of the shorter axis, pixel values change abruptly. In contrast, there are only small changes in pixel values when moving along the longer axis. Specifically, when ∂pi ∂u aligns with the direction of the longest axis, the gradient values are typically small. Consequently, the norm of the final gradient often falls below the densification threshold ∥ ∂L ∂x ∥ 2 < τ x , preventing effective densification. We visualize ∂G k ∂x in arrows in Fig. 8 (a), which is proportional to ∂pi ∂u , for better understanding. Therefore, the splats are biased towards adjusting its scale parameters (Fig. 8 (b) ) rather than splitting along the longer axis, converging into needle-like Gaussians.Third, scale parameters are kept the same after splitting, so needles are not shortened after densification.It will be an interesting future work to delve deeper into these reasons and address the problem with other approaches.We present normal rendering of our method results. Fig. 9 are results of the scene 122, with depth distortion and normal regularization loss used together. Fig. 10 shows the results of scene 55. Fig. 11 shows rendering results of Mip-NeRF360 dataset of our method. We visualize Gaussians with effective rank smaller than 1.02 in red. Effective rank regularization is adaptive to the scene, reducing the number of needle-like Gaussians, while effectively representing the required regions. Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).† Corresponding authors 


Decision: This paper was reviewed by four experts in the field.  Based on the reviewers' feedback, the decision is to recommend the paper for acceptance.  The reviewers did raise some valuable concerns that should be addressed in the final camera-ready version of the paper. The authors are encouraged to make the necessary changes to the best of their ability.    We congratulate the authors on the acceptance of their paper!

Reviews: review: paper_summary: paper_summary: The paper identifies a common problem in 3D Gaussian splatting where Gaussians converge to needle-like shapes, with its variance mostly contained in one axis. These needle-shaped Gaussians can cause artifacts and inaccurate surface reconstruction. 
To quantify this phenomenon, the paper uses the concept of effective rank, a continuous generalization of the rank of a matrix.  During scene reconstruction, while 3D gaussians may begin with an effective rank of 3, they tend to converge to an effective rank of 1 after several thousand iterations of training.  Based on effective rank, the paper proposes a regularization method to reduce needle-shaped Gaussians. On the DTU dataset, the authors show that 3DGS with rank regularization causes better chamfer distance and PSNR metrics.
strength_weakness: strength_weakness: While the paper introduces a new way to quantify needle-like Gaussians, this is not the first paper to propose a regularization term for them.  PhysGaussian [A] introduces an anisotropic regularizer to prevent skinny Gaussians. This regularization term has already been implemented in Nerfstudio’s Splatfacto method [B]. This seems like the most relevant and important comparison to the proposed method, and is completely missing from the paper. This missing comparison is the main reasoning behind my score. 

In addition, I wish that the paper did a deeper investigation into what causes spikey Gaussians. I found the discussion in Appendix A.4. very fascinating, and think it could be very valuable to the community if elaborated on. Maybe a simple experiment validating the hypothesis could strengthen the paper. 

Minor Comments: 
I found that Fig. 1 was hard to parse. I think it would help to include labels for the figure’s images. 

[A] https://arxiv.org/pdf/2311.12198
[B] https://docs.nerf.studio/nerfology/methods/splat.html
questions: questions: What happens if you do not regularize erank(G) < 2? Do Gaussians with erank(G) > 2 hurt reconstruction quality? 

What are x and o in Fig. 8?
limitations: limitations: Yes the authors have discussed the limitations and potential societal impacts.
paper_summary: paper_summary: This paper performs a statistical analysis on 3DGS for its effective rank distribution of the learned Gaussians. It claims that most of the Gaussian learned are close to rank 1 effectively, giving needle-like artifacts in novel view synthesis and reconstruction. Hence, this paper proposes a regularization loss to discourage low range Gaussians from forming, showing improved reconstruction results.
strength_weakness: strength_weakness: I do not doubt the motivation and effectiveness of the proposed method. In fact, needle-like artifacts have been a well-known issue for 3DGS in the community. However, I feel reluctant to accept this paper due to the following reasons:
1. Although the phenomenon is well known by many, the root cause of this issue is not clear. This paper tries to explain the cause in its appendix, and gives two hypotheses: a) incorrect dilation observed by MipGaussian, b) Gaussians are more likely to densify along their short axis instead of their long axis. Based on this, I would like to point out the following issues:
- This part should be put into the main paper instead of the statistical analysis. The causes of the low-rank Gaussians would be much more valuable than reiterating the phenomenon that many people already know.
- This cause analysis needs to be more rigorous. The version provided in the appendix is more of a hypothesis than an analysis. 
- If it is because of the dilation error, is it already fixed by MipGaussian? What is the rank distribution like for MipGaussian? Can the proposed method improve the MipGaussian performance as well? If yes, then something else is still causing the issue. If not, why shouldn't MipGaussian be used instead of the proposed method?
- If it is because of the densification bias, does this problem still occur without densification at all? In fact, I have personally observed many cases of needle-like artifacts that are so long that extend across the entire image. In this case, the rendering loss should be enough to prevent this from happening without anything to do with the densification. But it is not happening, why? Is there something wrong with the rendering loss supervision we have with 3DGS? I am not saying that the densification bias is not a cause, but it requires more evidence and experiment results to be proved.

2. Continuing from the previous point, the method proposed does not address the cause hypothesized. It is not fair to say that regularization methods are just bad, but there have been even simpler regularization methods used by other papers (e.g. PhyGaussian). The most common method is to regularize the ratio between the long axis and the short axis of the 3DGS. It takes only a few lines of code and works reasonably well. This paper does not mention this simpler regularization and does not compare with it as a baseline. However, I think it is very important to justify the more complicated method proposed in this paper, with either rigorous analysis as well as empirical results (preferably both). 

In summary, I believe that this paper lacks a rigorous analysis of the cause of the low-rank Gaussian formation, and the proposed method lacks novelty and performance comparison against the simple naive regularization used in other papers.
questions: questions: To summarize my opinions mentioned in the weakness section into questions, it would be:
1. what is really the cause for the low-rank Gaussian formation and how does the proposed method address it?
2. why is the proposed method better than the simple regularization on the ratio of the long/short gaussian axis?
limitations: limitations: The paper does not have potential negative societal impact and there is no significant limitation of the method that needs to be addressed other than the ones mentioned in the previous sections.
paper_summary: paper_summary: 3D Gaussian Splatting is a remarkable technique in novel view synthesis. However, it usually degenerates into noodle-like shapes which sometimes bring visual artifacts and inaccurate geometry. This paper analyzes the phenomenon and proposes an effective rank loss to regularize the Gaussians. The technique seems to be a plug-and-play module for 3DGS variants. Extensive experiments suggest the effectiveness.
strength_weakness: strength_weakness: 1. There is no supplemental video, which is essential for nvs tasks. 
2. The improvement of e-rank for Novel view synthesis is limited. It would be beneficial to highlight how this could improve novel view synthesis.
3. Although the paper analyzes that the noodle-like artifacts are due to the training bias, they can also be created to represent high-frequency details. The paper should highlight the PSNR in Figure.1 to give readers a sense that such regularization indeed improves novel view synthesis.
4. It would be beneficial to add the number of Gaussians since enforcing the Gaussian to 2D from 1D should reduce the number of Gaussians in representing a scene.
questions: questions: 1. How the normal is constructed for 3D Gaussian and how it is visualized?
2. What is the effect of densification on geometry reconstruction in 3DGS? In Table 1. Does 3DGS+e indicate with ERANK only or plus densification? It should be clearly ablated where the performance comes from.
limitations: limitations: Overall, the paper did a good job with minor issues required to clarify. The limitation of tuning the hyperparameter is also discussed and the potential negative effect is reasonable. 

**minors:**

The normal visualization seems not to be normalized witn [0,1]
paper_summary: paper_summary: The authors present a new regularization for Gaussian splatting (GS) method that increases the Shannon entropy of the scale parameters (it was named erank as while rendering the scale is presented as a diagonal matrix, and the aforementioned calculation turns into the effective rank), to better reconstruct 3d mesh out of the GS point cloud. The paper argues that a majority of the gaussians in previous methods are close to erank of 1, which ruins the 3d reconstruction because of needle-like gaussians that effectively add noise to the reconstructions. Given that regularization as an add-ons to other methods it improves the 3D reconstruction, as shown in the experiments on DTU dataset. Additionally, it shows in the appendix that the image rendering abilities of the GS is not impaired because of the added regularization.
strength_weakness: strength_weakness: There are several concerns regarding this paper, first it should include more benchmarks for 3d reconstruction, for example Tanks and Temples dataset which was used as benchmark in many other works in the field, including the works that was cited in the paper. Secondly, there are two works that the authors did not mention [1,2] and should be compared to in the DTU experiment. The first [1] has the same idea as 2DGS and was already published. It might be that their regularization solves part of the issue of the GS that this paper presents. The second [2], does not use the gaussian splatting directly for reconstruction and instead renders stereo images for depth estimate. Thus, it does not share the reconstruction error because of the needle-like gaussians. Comparing both of them will strengthen this paper. Lastly, there is no adequate justification for the erank regularization in comparison to other possibilities, e.g. linear, parabolic, or exponential loss on the size of the smallest magnitude scale. Finally, an ablation study showing why erank-regularization is superior compared to other losses is missing. 
[1] Dai, P., et al. "High-quality surface reconstruction using gaussian surfels." arXiv preprint arXiv:2404.17774 (2024).
[2] Wolf, Y., et al.. "Surface Reconstruction from Gaussian Splatting via Novel Stereo Views." arXiv preprint arXiv:2404.01810 (2024).
questions: questions: see weaknesses
limitations: limitations: see weaknesses

Comments: title: Comparison with recent results would improve the paper.
Thanks
Official Comment by Reviewer 7LDG
comment: The authors provided answers to our concerns regarding the T&T dataset, and the justification for using the entropy like regularization compared to other possibilities. Regarding the recent papers to compare with, such evaluations would definitely improve the paper.
Thank you for your response. I appreciate the new experiments on PhysGaussian, and I see how their method can still cause low-rank Gaussians from forming. Along with the other reviewers, I agree that a deeper investigation into the cause of spiky Gaussians is important for the paper, and I hope that the new analysis is included in a revised version of the paper. 

Based on the rebuttal, I will change my score.
Thank you for your response. I appreciate that most of my questions have been addressed. After reading the author rebuttal and considering the other reviews, I’d like to share my thoughts.

The proposed method is simple yet effective and can serve as a plug-and-play module for various GS variants, potentially broadening its impact.

I also agree with Reviewer w1Sn that there should be more analysis on the root cause of the needle-like artifacts, as this could enhance the paper’s impact. I encourage the authors to include some of this analysis in the main paper. Additionally, it would be beneficial to emphasize that the regularization, while reducing needle-like artifacts, does not compromise the reconstruction of high-frequency details.

I also noticed, as pointed out by Reviewer biCn, that some important baselines like PhysGaussian were missing in my initial feedback. I appreciate that the authors have since provided these comparisons.

Overall, while some minor revisions are needed, I believe the authors are well-equipped to address these issues, as demonstrated in their rebuttal, including the comments and tables presented. Therefore, I have no concerns with the majority of the work and would like to maintain my original rating.
I appreciate the effort put in by the authors to conduct the toy experiment and the comparison against scaling loss from PhyGaussian. I am convinced that the low dL/du gradient problem demonstrated in the toy example is partly the reason why the Gaussians tend not to split along the long axis. Hence, I would like to raise my score. I hope the additional experiments and justifications can be added to the final version.

However, I encourage the authors to look deeper into the scaling optimization of Gaussians as well. Other than the reason shown, there should be something about the scaling optimization of Gaussians also causing the spiky Gaussians, because we often see very long and thin Gaussians that should have been penalized heavily on rendering loss even from training views. This phenomenon can hardly be explained by splitting bias alone. I believe that finding out the reason behind this can be very benefitial to the research community.