Spec-Gaussian: Anisotropic View-Dependent Appearance for 3D Gaussian Splatting

Authors: ziyi  yang
xinyu  gao
yang-tian  sun
yi-hua  huang
xiaoyang  lyu
wen  zhou
shaohui  jiao
xiaojuan  qi
†  xiaogang
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-07 10:59:00

Abstract: The recent advancements in 3D Gaussian splatting (3D-GS) have not only facilitated real-time rendering through modern GPU rasterization pipelines but have also attained state-of-the-art rendering quality. Nevertheless, despite its exceptional rendering quality and performance on standard datasets, 3D-GS  frequently encounters difficulties in accurately modeling specular and anisotropic components. This issue stems from the limited ability of spherical harmonics (SH) to represent high-frequency information. To overcome this challenge, we introduce Spec-Gaussian, an approach that utilizes an anisotropic spherical Gaussian (ASG) appearance field instead of SH for modeling the view-dependent appearance of each 3D Gaussian. Additionally, we have developed a coarse-to-fine training strategy to improve learning efficiency and eliminate floaters caused by overfitting in real-world scenes. Our experimental results demonstrate that our method surpasses existing approaches in terms of rendering quality. Thanks to ASG, we have significantly improved the ability of 3D-GS to model scenes with specular and anisotropic components without increasing the number of 3D Gaussians. This improvement extends the applicability of 3D GS to handle intricate scenarios with specular and anisotropic surfaces.

Main paper: 
 introduction: 
 High-quality reconstruction and photorealistic rendering from a collection of images are crucial for a variety of applications, such as augmented reality/virtual reality (AR/VR), 3D content production, and art creation. Classic methods employ primitive representations, like meshes [39] and points [4, 67] , and take advantage of the rasterization pipeline optimized for contemporary GPUs to achieve real-time rendering. In contrast, neural radiance fields (NeRF) [37, 5, 38] utilize neural implicit representation to offer a continuous scene representation and employ volumetric rendering to produce rendering results. This approach allows for enhanced preservation of scene details and more effective reconstruction of scene geometries. Figure 1 : Our method not only achieves real-time rendering but also significantly enhances the capability of 3D-GS to model scenes with specular and anisotropic components. Key to this enhanced performance is our use of ASG appearance field to model the appearance of each 3D Gaussian, which results in substantial improvements in rendering quality for both complex and general scenes.Despite its exceptional performance, 3D-GS struggles to model specular components within scenes (see Fig. 1 ). This issue primarily stems from the limited ability of low-order spherical harmonics (SH) to capture the high-frequency information in these scenarios. Consequently, this poses a challenge for 3D-GS to model scenes with reflections and specular components, as illustrated in Fig. 1 .To address the issue, we introduce a novel approach called Spec-Gaussian, which combines anisotropic spherical Gaussian (ASG) [60] for modeling anisotropic and specular components, an effective training mechanism to eliminate floaters and improve learning efficiencies, and anchorbased 3D Gaussians for acceleration and storage reduction. Specifically, the method incorporates two key designs: 1) A new 3D Gaussian representation that utilizes an ASG appearance field instead of SH to model the appearance of each 3D Gaussian. ASG with a few orders can effectively model high-frequency information that low-order SH cannot. This new design enables 3D-GS to more effectively model anisotropic and specular components in static scenes. 2) A coarse-to-fine training scheme specifically tailored for 3D-GS is designed to eliminate floaters and boost learning efficiency. This strategy effectively shortens learning time by optimizing low-resolution rendering in the initial stage, preventing the need to increase the number of 3D Gaussians and regularizing the learning process to avoid the generation of unnecessary geometric structures that lead to floaters.By combining these advances, our approach can render high-quality results for specular highlights and anisotropy as shown in Fig. 4 while preserving the efficiency of Gaussians. Furthermore, comprehensive experiments reveal that our method not only endows 3D-GS with the ability to model specular highlights but also achieves state-of-the-art results in general benchmarks.In summary, the major contributions of our work are as follows:• A novel ASG appearance field to model the view-dependent appearance of each 3D Gaussian, which enables 3D-GS to effectively represent scenes with specular and anisotropic components. • A coarse-to-fine training scheme that effectively regularizes training to eliminate floaters and improve the learning efficiency of 3D-GS in real-world scenes. • An anisotropic dataset has also been made to assess the capability of our model in representing anisotropy.2 Related Work 
implicit neural radiance fields: 
 Neural rendering has attracted significant interest in the academic community for its unparalleled ability to generate photorealistic images. Methods like NeRF [37] utilize Multi-Layer Perceptrons (MLPs) to model the geometry and radiance fields of a scene. Leveraging the volumetric rendering equation and the inherent continuity and smoothness of MLPs, NeRF achieves high-quality scene reconstruction from a set of posed images, establishing itself as the state-of-the-art (SOTA) method for novel view synthesis. Subsequent research has extended the utility of NeRF to various applications, including mesh reconstruction [53, 27, 58, 34] , inverse rendering [48, 72, 31, 62] , optimization of camera parameters [29, 55, 54, 41] , few-shot learning [12, 61, 57] , and anti-aliasing [2, 1, 3] .However, this stream of methods relies on ray casting rather than rasterization to determine the color of each pixel. Consequently, every sampling point along the ray necessitates querying the MLPs, leading to significantly slow rendering speed and prolonged training convergence. This limitation substantially impedes their application in large-scene modeling and real-time rendering.To reduce the training time of MLP-based NeRF methods and improve rendering speed, subsequent work has enhanced NeRF's efficiency in various ways. Structure-based techniques [68, 14, 43, 17, 7] have sought to improve inference or training efficiency by caching or distilling the implicit neural representation into more efficient data structures. Hybrid methods [30, 49] increase efficiency by incorporating explicit voxel-based data structures. Factorization methods [5, 18, 8, 16] apply a lowrank tensor assumption to decompose the scene into low-dimensional planes or vectors, achieving better geometric consistency. Compared to continuous implicit representations, the convergence of individual voxels in the grid is independent, significantly reducing training time. Additionally, Instant-NGP [38] utilizes a hash grid with a corresponding CUDA implementation for faster feature querying, enabling rapid training and interactive rendering of neural radiance fields. Spec-NeRF [35] achieves high-quality specular reflection modeling by introducing Gaussian directional encoding.Despite achieving higher quality and faster rendering, these methods have not fundamentally overcome the substantial query overhead associated with ray casting. As a result, a notable gap remains before achieving real-time rendering. In this work, we build upon the recent 3D-GS [23] , a pointbased rendering method that leverages rasterization. Compared to ray casting-based methods, it significantly enhances both training and rendering speed.Point-based representations, similar to triangle mesh-based methods, can exploit the highly efficient rasterization pipeline of modern GPUs to achieve real-time rendering. Although these methods offer breakneck rendering speeds and are well-suited for editing tasks, they often suffer from holes and outliers, leading to artifacts in the rendered images. This issue arises from the discrete nature of point clouds, which can create gaps in the primitives and, consequently, in the rendered image.To address these discontinuity issues, differentiable point-based rendering [67, 15, 24, 25] has been extensively explored for fitting complex geometric shapes. Notably, Zhang et al. [71] employ differentiable surface splatting and utilize a radial basis function (RBF) kernel to compute the contribution of each point to each pixel.Despite achieving SOTA results on commonly used benchmark datasets, 3D-GS still struggles to model scenes with specular and reflective components, which limits its practical application in real-time rendering at the photorealistic level. In this work, by replacing spherical harmonics (SH) with an anisotropic spherical Gaussian (ASG) appearance field, we have enabled 3D-GS to model complex specular scenes more effectively. 
method: 
 The overview of our method is illustrated in Fig. 2 . The input to our model is a set of posed images of a static scene, together with a sparse point cloud obtained from SfM [46] . The core of our method is to use the ASG appearance field to replace SH in modeling the appearance of 3D Gaussians (Sec. 3.2). Moreover, we introduce a simple yet effective coarse-to-fine training strategy to reduce floaters in real-world scenes (Sec. 3.3). To further reduce the storage overhead and rendering speed pressure introduced by ASG, we combine a hybrid Gaussian model that employs sparse anchor Gaussians to facilitate the generation of neural Gaussians (Sec. 3.4) to model the 3D scene.3D Gaussian splatting. 3D-GS [23] is a point-based method that employs anisotropic 3D Gaussians to represent scenes. Each 3D Gaussian is defined by a center position x, opacity σ, and a 3D covariance matrix Σ, which is decomposed into a quaternion r and scaling s. The view-dependent appearance of each 3D Gaussian is represented using the first three orders of spherical harmonics (SH). This method not only retains the rendering details offered by volumetric rendering but also achieves real-time rendering through a CUDA-customized differentiable Gaussian rasterization process. Following [75] , the 3D Gaussians can be projected to 2D using the 2D covariance matrix Σ ′ , defined as:EQUATIONwhere J is the Jacobian of the affine approximation of the projective transformation, and V represents the view matrix, transitioning from world to camera coordinates. To facilitate learning, the 3D covariance matrix Σ is decomposed into two learnable components: the quaternion r, representing rotation, and the 3D-vector s, representing scaling. The resulting Σ is thus represented as the combination of a rotation matrix R and scaling matrix S as:Σ = RSS T R T .(EQUATIONThe color of each pixel on the image plane is then rendered through a point-based volumetric rendering (alpha blending) technique:EQUATIONwhere p denotes the pixel coordinate, T i is the transmittance defined by Π i-1 j=1 (1 -α j ), c i signifies the color of the sorted Gaussians associated with the queried pixel, and µ i represents the coordinates of the 3D Gaussians when projected onto the 2D image plane.Anisotropic spherical Gaussian. Anisotropic spherical Gaussian (ASG) [60] has been designed in the traditional rendering pipeline to efficiently approximate lighting and shading. Different from spherical Gaussian (SG), ASG has been demonstrated to effectively represent anisotropic scenes with a small number. In addition to retaining the fundamental properties of SG, ASG also exhibits rotational invariance and can represent full-frequency signals. The ASG function is defined as:EQUATION)where ν is the unit direction serving as the function input; x, y, and z correspond to the tangent, bi-tangent, and lobe axis, respectively, and are mutually orthogonal; λ ∈ R 1 and µ ∈ R 1 are the sharpness parameters for the xand y-axis, satisfying λ, µ > 0; ξ ∈ R 2 is the lobe amplitude; S is the smooth term defined as S(ν; z) = max(ν • z, 0).Inspired by the power of ASG in modeling scenes with complex anisotropy, we propose integrating ASG into Gaussian splatting to join the forces of classic models with new rendering pipelines for higher quality. For N ASGs, we predefined orthonormal axes x, y, and z, initializing them to be uniformly distributed across a hemisphere. During training, we allow the remaining ASG parameters, λ, µ, and ξ, to be learnable. We use the reflect direction ω r as the input to query ASG for modeling the view-dependent specular information. Note that we use N = 32 ASGs for each 3D Gaussian.Anchor-based Gaussian splatting. Anchor-based Gaussian splatting was first proposed by Scaffold-GS [32] . Unlike the attributes carried by each entity in 3D-GS, each anchor Gaussian carries a position coordinate P v ∈ R 3 , a local feature f v ∈ R 32 , a displacement factor η v ∈ R 3 , and k learnable offsets O v ∈ R k×3 . They use the COLMAP [46] point cloud to initialize each anchor 3D Gaussian, serving as the voxel centers to guide the generation of neural Gaussians. The position P v of the anchor Gaussian is initialized as:EQUATION)where P is the point cloud position, ϵ is the voxel size, and {•} denotes removing duplicated anchors.Then anchor Gaussians can guide the generation of neural Gaussians, which have the same attributes as vanilla 3D-GS. For each visible anchor Gaussian within the viewing frustum, we spawn k neural Gaussians and predict their attributes. The positions x of neural Gaussians are calculated as:EQUATIONwhere P v represents the position of the anchor Gaussian corresponding to k neural Gaussians. The opacity σ is calculated through a tiny MLP:EQUATIONwhere δ cv denotes the distance between the anchor Gaussian and the camera, and d cv is the unit direction pointing from the camera to the anchor Gaussian. The rotation r and scaling s of each neural Gaussian are derived similarly using the corresponding tiny MLP F r and F s .ASG appearance field for 3D Gaussians.Although SH has enabled view-dependent scene modeling, the low frequency of low-order SH makes it challenging to model scenes with complex optical phenomena such as specular highlights and anisotropic effects. Therefore, instead of using SH, we propose using an ASG appearance field based on Eq. ( 4) to model the appearance of each 3D Gaussian. However, the introduction of ASG increases the feature dimensions of each 3D Gaussian, raising the model's storage overhead. To address this, we employ a compact learnable MLP Θ to predict the parameters for N ASGs, with each Gaussian carrying only additional local features f ∈ R 24 as the input to the MLP:EQUATIONTo better differentiate between high and low-frequency information and further assist ASG in fitting high-frequency specular details, we decompose color c into diffuse and specular components:EQUATIONwhere c d represents the diffuse color, modeled using the first three orders of SH, and c s is the specular color calculated through ASG. We refer to this comprehensive approach to appearance modeling as the ASG appearance field.Although ASG theoretically enhance the ability of SH to model anisotropy, directly using ASG to represent the specular color of each 3D Gaussian still falls short in accurately modeling anisotropic and specular components, as demonstrated in Fig. 6 . Inspired by [16] , we do not use ASG directly to represent color but instead employ ASG to model the latent feature of each 3D Gaussian. This latent feature, containing anisotropic information, is then fed into a tiny feature decoding MLP Ψ to determine the final specular color: where κ is the latent feature derived from ASG, denotes the concatenation operation, γ represents the positional encoding, d is the unit view direction pointing from the camera to each 3D Gaussian, n is the normal of each 3D Gaussian, and ω r is the unit reflect direction. This strategy significantly enhances the ability of 3D-GS to model scenes with complex optical phenomena, whereas neither pure ASG nor pure MLP can achieve anisotropic appearance modeling as effectively as our approach.EQUATIONNormal estimation. Following [22, 47] , we use the shortest axis of each Gaussian as its normal. This approach is based on the observation that 3D Gaussians tend to flatten gradually during the optimization process, allowing the shortest axis to serve as a reasonable approximation for the normal.The reflect direction ω r can then be derived using the view direction and the local normal vector n as:EQUATIONwhere ω o = -d is a unit view direction pointing from each 3D Gaussian in world space to the camera. We use the reflect direction ω r to query ASG, enabling better interpolation of latent features containing anisotropic information. Experimental results show that although this unsupervised normal estimation cannot generate physically accurate normals aligned with the real world, it is sufficient to produce relatively accurate reflect direction to assist ASG in fitting high-frequency information.We observed that in many real-world scenarios, 3D-GS tends to overfit the training data, leading to the emergence of numerous floaters when rendering images from novel viewpoints. One important reason is that the COLMAP point cloud is too sparse. Poor initialization makes it difficult for 3D-GS to compensate for overly sparse areas through densification during optimization, leading to floaters in the rendering images. Moreover, 3D-GS accumulates gradients from each pixel to the GS:dL dx =dL dpi dpi dx , and the densification occurs when the accumulated amount exceeds a threshold τ g = 0.0002. However, having positive and negative gradients can cause GSs that should be densified to be ignored due to the large negative gradient. Thus, to mitigate the occurrence of floaters in real-world scenes, we propose a coarse-to-fine training mechanism. We first impose an L1 constraint on the gradients from pixels to GS: dL dx = ∥ dL dpi dpi dx ∥ 1 , accumulating the numerical contribution from pixels to GS rather than gradients. This idea is similar to the concurrent works [65, 70] . Next, to avoid overfitting caused by excessive growth of 3D-GS during the early stages of optimization, we decide to train 3D-GS progressively from low to high resolution in real-world scenes:EQUATIONwhere r(i) is the image resolution at the i-th training iteration, r s is the starting image resolution, r e is the ending image resolution (the full resolution we aim to render), and τ is the threshold iteration, empirically set to 5k.This training method allows 3D-GS to densify correctly and prevents excessive growth of 3D-GS in the early stages. Additionally, due to the lower resolution training in the initial phase, this mechanism reduces training time by approximately 10%. In our experiments, we offer a performance version with τ g = 0.0005 and light version with τ g = 0.0006. Table 2 : Quantitative comparison of on real-world datasets. We report PSNR, SSIM, LPIPS (VGG) and color each cell as best , second best and third best . Our method has achieved the best rendering quality, while striking a good balance between FPS and the storage memory.While the ASG appearance field significantly improves the ability of 3D-GS to model specular and anisotropic features, it introduces additional computational overhead due to the additional local features f associated with each Gaussian. Inspired by [32] , we employ anchor-based Gaussian splatting to reduce storage overhead and accelerate the rendering.Since the anisotropy modeled by ASG is continuous in space, it can be compressed into a lowerdimensional space. Thanks to the guidance of the anchor Gaussian, the anchor feature f v can be used directly to compress N ASGs, further reducing storage pressure. To make the ASG of neural Gaussians position-aware, we introduce the unit view direction to decompress ASG parameters. Consequently, the ASG parameters prediction in Eq. ( 8) is revised as follows:EQUATIONwhere d cn denotes the unit view direction from the camera to each neural Gaussian. Additionally, we set the diffuse part of the neural Gaussian c d = ϕ(f v ), directly predicted through an MLP ϕ, to ensure the smoothness of the diffuse component and reduce the difficulty of convergence.We optimize the learnable parameters and MLPs using the same loss function as 3D-GS [23] . The total supervision is given by:EQUATIONwhere the λ D-SSIM = 0.2 is consistently used in our experiments. 
experiments: 
 In this section, we present both quantitative and qualitative results of our method. To evaluate its effectiveness, we compared it to several state-of-the-art methods across various datasets. We color with τ g = 0.0006. Our method demonstrates superior performance in modeling complex specular and anisotropic features, as evidenced by comparisons on the NeRF, NSVF, and our "Anisotropic Synthetic" datasets. Additionally, we showcase its versatility by comparing its performance in diffuse scenarios, further proving the robustness of our approach.We implemented our framework using PyTorch [42] and modified the differentiable Gaussian rasterization to include depth visualization. For the ASG appearance field, the decoupling MLP Ψ consists of 3 layers, each with 64 hidden units, and the positional encoding for the view direction is of order 2. Regarding coarse-to-fine training, which is applied only to real-world scenes to remove floaters, we start with a resolution r s that is 4x downsampled. To further accelerate rendering, we prefilter and allow only those Gaussians with opacity σ n > 0 to pass through the ASG appearance field and Gaussian rasterization pipelines. All experiments were conducted on an NVIDIA RTX 3090.Synthetic bounded scenes. We used the NeRF, NSVF, and our "Anisotropic Synthetic" datasets as the experimental datasets for synthetic scenes. Our comparisons were made with the most relevant state-of-the-art methods, including 3D-GS [23] , Scaffold-GS [32] , GaussianShader [22] , and several NeRF-based methods such as NSVF [30] , TensoRF [5] , NeuRBF [8] , and Tri-MipRF [18] .As shown in Fig. 4 (with PSNR and LPIPS), and Tabs. 3-4, our method achieved the highest performance with fewer Gaussians compared to vanilla 3D-GS. It also improved upon the issues that 3D-GS faced in modeling high-frequency specular highlights and complex anisotropy as shown in Tab. 1 with fewer Gaussians and better metrics. See more in the supplementary materials.Real-world unbounded scenes. To verify the versatility of our method in real-world scenarios, we used the Mip360 [2] dataset, which contains indoor scenes with specular highlights. As shown in Tab. 2, our method surpasses state-of-the-art methods on Mip-NeRF 360. Furthermore, our method effectively balances FPS, storage, and rendering quality. It enhances rendering quality without increasing storage or significantly reducing FPS. As illustrated in Fig. 5 and Fig. 7 , our method has also significantly improved the visual effect. It removes a large number of floaters in outdoor scenes and successfully models the high-frequency specular highlights in indoor scenes. This demonstrates that our approach is not only adept at modeling complex specular scenes but also effectively improves rendering quality in general scenarios.ASG feature decoupling MLP. We conducted an ablation study to evaluate the key components of the ASG appearance field, which include ASG features, decoupling MLP, and the separation of diffuse and specular colors. As demonstrated in Fig. 6 (with PSNR and LPIPS), directly using ASG to output color results in the inability to model specular and anisotropic components. In contrast to directly using an MLP for color modeling, as in Scaffold-GS [32] , separately modeling diffuse and specular color can enhance the fitting ability for high-frequency information. ASG can encode higherfrequency anisotropic features. With the help of ASG's ability to encode high-frequency anisotropic features, the decoupling MLP can fit complex optical phenomena, leading to more accurate rendering results. We also demonstrated that higher-order SH (6-order) and more MLP layers (4-layers) do not help 3D-GS and Scaffold-GS achieve satisfactory results, highlighting the importance of ASG.Coarse-to-fine training. We conducted an ablation study to assess the impact of coarse-to-fine (c2f) training. As illustrated in Fig. 7 (with LPIPS and number of Gaussian), both 3D-GS and Scaffold-GS exhibit a large number of floaters in the novel view synthesis. Coarse-to-fine training effectively reduces the number of floaters, alleviating the overfitting issue commonly encountered by 3D-GS in real-world scenarios. Applying an L1 constraint to the gradients used for 3D-GS densification further reduced the number of floaters and Gaussians. See more in the supplementary materials. 
conclusion: 
 In this work, we introduce Spec-Gaussian, a novel approach to 3D Gaussian splitting that features an anisotropic view-dependent appearance. Leveraging the powerful capabilities of ASG, our method effectively overcomes the challenges encountered by vanilla 3D-GS in rendering scenes with specular highlights and anisotropy. Additionally, we innovatively implement a coarse-to-fine training mechanism to eliminate floaters in real-world scenes. Both quantitative and qualitative experiments demonstrate that our method not only equips 3D-GS with the ability to model specular highlights and anisotropy but also enhances the overall rendering quality of 3D-GS in general scenes.Limitations. Although our method enables 3D-GS to model complex specular and anisotropic features, it still faces challenges in handling reflections. Specular and anisotropic effects are primarily influenced by material properties, whereas reflections are closely related to the environment and geometry. Due to the lack of explicit geometry in 3D-GS, we cannot differentiate between reflections and materials using constraints like normals, as employed in Ref-NeRF [52] and NeRO [31] . We plan to explore solutions for modeling reflections with 3D-GS in future work. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: The paper poses no such risks. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: Yes, they do. Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA] Justification: The paper does not release new assets. Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: This supplementary material provides more results that accompany the paper.• Section A provides more ablations.• Section B provides additional results, including more visualizations and quantitative results on complete datasets.In this section, we present the complete quantitative ablations on the key components of our method.We first evaluate the role of each component of the ASG appearance field in NeRF synthetic scenes as shown in Tab. 5. The introduction of ASG improves the ability to model specular highlights and reduces the number of 3D Gaussians. The inclusion of normals did not significantly increase computational overhead, but it did enhance rendering metrics and visual quality. More importantly, we achieve better rendering quality with fewer Gaussians than vanilla 3D-GS, a characteristic that can be further explored in the future.Next, we evaluated our full method on the Mip360 dataset in Tab. 6. It is important to note that the Mip360 dataset is divided into indoor and outdoor scenes. Indoor scenes have more specular highlights, while outdoor scenes contain a large number of floaters. The coarse-to-fine approach itself improves the quality of 3D-GS in real-world scenes, mainly by eliminating a significant amount of floaters in outdoor settings. Although the introduction of the ASG appearance field significantly increases rendering overhead, it did greatly enhance the modeling of specular highlights in indoor scenes. Under the constraints of the coarse-to-fine mechanism, our complete method combines the advantages of both, achieving the best rendering quality. To further improve rendering speed, we also implement a light version and a mini version based on Scaffold-GS. These versions offer a trade-off between rendering quality and speed and can be used as needed. The quality of the Mip360 scenes demonstrates that our method is not only capable of handling scenes with specular highlights but is also robust in real-world diffuse scenarios.In this section, we present the complete quantitative results of our experiments. We report PSNR, SSIM, LPIPS (VGG), and color each cell as best , second best and third best .As shown in Tabs. 7-9, our method demonstrates the best rendering quality metrics in almost every scene. It's important to note that the experimental setup for Tri-MipRF [18] differs from other methods. It uses both the training and validation sets as training data, expanding the scale of the model's data. When its training data is limited to the training set, its metrics suffer a noticeable drop. Nevertheless, to ensure that the experimental results fully reflect the highest performance of each method, and to prevent significant drops in metrics due to differences in experimental environments, we still present the metrics from the Tri-MipRF official paper. Our method achieved more prominent metrics in scenes with notable specular reflection and anisotropy, such as Drums, Lego, and Ship. This demonstrates that our method not only improves the overall rendering quality but also has a more significant advantage in complex specular scenarios.The NSVF [30] dataset, in comparison to NeRF, features more noticeable metallic specular reflection, as presented in the Wineholder, Steamtrain, and Spaceship scenes. It is important to note that Tri-MipRF fails to converge in the Steam scene with the official code, so we did not report metrics for that scenario. As shown in Tabs. 10-12, we present the per-scene experimental results of PSNR, SSIM, and LPIPS in the supplementary material. The experimental results indicate that compared to other methods based on 3D-GS [23] , our method has significant advantages in metallic highlights "Anisotropic Synthetic" is a synthetic dataset we rendered ourselves, which includes 8 scenes with significant anisotropy. We tested some existing 3D-GS-based methods on "Anisotropic Synthetic." As shown in Tabs. 13-15, our method achieved a very significant improvement in rendering metrics. Fig. 13 shows the comparison between our method and 3D-GS across all eight scenes. Qualitative experiments also demonstrate the significant visual advantages of our method, highlighting the substantial improvement our method brings to anisotropic parts, thereby enhancing the overall rendering quality.The MipNeRF-360 scenes include five outdoor and four indoor scenarios. There are several scenes rich in specular reflections, such as bonsai, room, and kitchen. As shown in Tabs. 16-18, our method achieved significant advantages in the four indoor scenes. This reflects our method's strengths in modeling specular reflections and anisotropy. In outdoor scenes, our method also achieved rendering metrics comparable to the SOTA methods. Furthermore, with the help of the coarse-to-fine training mechanism, our method significantly reduced the number of floaters as shown in Fig. 11 , resulting in a substantial improvement in visual effects.Answer Justification: There is no societal impact of the work performed. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 


Decision: This paper received positive scores of weak accept, weak accept, borderline accept and borderline accept. The AC concurs with the reviewers and recommends acceptance. The authors have promised changes in their rebuttal:

- Slightly tone down their claims wrt being the "first to address specular highlights modeling in GS."
- Incorporate suggestions from reviewer 8TqS regarding section 3, as well as address additional questions from 8TqS re: problems with the text
- Incorporate additional experiments provided in the rebuttal

The AC urges the authors to uphold their promises.

Reviews: review: paper_summary: paper_summary: This paper presents an approach for reconstruction and view synthesis of scenes that exhibit strong specular/view dependent appearance. In particular, the authors extend the framework of Gaussian Splatting [Kerbl et al. 2023] and Scaffold-GS [Lu et al. 2023], replacing spherical harmonics for parameterizing view-dependent appearance with anisotropic spherical Gaussians [Xu et al. 2013], as well as leveraging a coarse-to-fine training strategy to generally improve performance. The authors carry out qualitative/quantitative evaluation on a variety of datasets, and report improved quantitative performance over nearly every other baseline.
strength_weakness: strength_weakness: * I felt that the writing quality could be improved. In section 3, I couldn't tell which of the described components were part of the base method, and which were part of method variants (e.g. ours-light, ours w/ anchor). Method training and architecture details were somewhat sparse (only a few details were provided in section 4.1). I was also confused about how the word anisotropy is being used throughout the paper. Typically, it's used to mean view dependence that is not isotropic (e.g. rotationally invariant) -- but it does necessarily imply *high frequency* view dependence. Spherical harmonics are, in fact, anisotropic spherical functions. 
* I'm not quite sure how to assess the novelty of this paper, and feel that some claims (e.g. this is "the first work to address the specular highlights modeling in 3D-GS") are not quite fair (other works, such as GaussianShader [Jiang et al. 2024] at CVPR 2024 attempt to improve modeling of view dependent appearance in Gaussian splatting). 
* For many of the real scenes, it's hard to judge the qualitative improvement of this method over baselines (e.g. Figure 8). The video comparisons are nice, but are not provided for all baselines -- I feel that a webpage would've been more effective for showcasing comparisons/improvements.
* As far as I can tell, this work does not make many changes on top of existing methods -- the two main changes being using ASGs to parameterize view-dependent appearance, and implementing a coarse-to-fine training strategy. Perhaps I'm missing something, but I'm not sure why these changes should lead to such a large improvement in view-dependent appearance modeling and quantitative performance. For example, are ASGs really responsible for modeling the reflection in the CD in Figure 9? Or is the ability to model this reflection due to Scaffold-GS's view-dependent decoding of Gaussian parameters? When the removal of floaters is shown in the video for the Bonsai scene, is this due to the coarse-to-fine training strategy, or something else (hard to say, because only the baseline 3DGS is shown in the comparison)? In general I don't feel that the relationship between quantitative/qualitative gains and method design are fully justified, but I acknowledge that I could be in the minority here.
questions: questions: See above.
limitations: limitations: Limitations are discussed.
paper_summary: paper_summary: This paper proposes using Anisotropic Spherical Gaussians (ASGs) as view encoding to enhance the modeling of specular reflections in 3D Gaussian splatting. In addition to Spherical Harmonics (SH) encoded colors, the method additionally queries reflection direction with multiple ASGs to generate a view encoding. Since ASGs can potentially model higher frequency signals, the proposed method improves reconstruction quality in scenarios with strong view-dependence. The paper also introduces minor contributions, such as coarse-to-fine training and ASG compression, to further enhance quality and efficiency. The results demonstrate that the method surpasses all baselines.
strength_weakness: strength_weakness: • Contribution: I would not describe this work as "the first to address specular highlights modeling in GS," as some earlier work has also attempted this. It would be better to tone down this claim.

• Sum of L1 Norm of Gradients: One improvement in the paper is achieved by adding L1 norm to the gradient accumulation for GS densification, as illustrated in lines 202-214. I don't really follow the intuition behind this design. The original design for densification involves duplicating Gaussians and moving along the gradient direction to reduce image loss from all views. So when different views suggest moving the Gaussian in different directions, these directions cancel out and we don't do densification. This makes sense because it means that the particular gaussian is centered at an optimal location that won't sacrifice the quality of any view. However, I cannot find any physical meaning for using the sum of L1 norms. Although experiments show better performance, it would be helpful to provide an explanation, possibly with a toy example. Could this performance improvement be because the sum of L1 norms tends to be larger and thus more likely to exceed the threshold? Would reducing the threshold have a similar effect?

• Lack ablation on the Number of ASGs: An important hyperparameter is the number of ASGs used. Theoretically, this is crucial for balancing efficiency and quality.

• Discussion on Shape-Radiance Ambiguity: An important aspect of reflection modeling in radiance fields is dealing with shape-radiance ambiguity. I assume the coarse-to-fine training helps with this ambiguity, but it is not discussed in the methods or experiments.

• Missing Comparison with Inverse Rendering GS: Although inverse rendering has a slightly different task than view-dependent appearance modeling, it is a valid approach to solving this problem. Therefore, it is necessary to compare this method with at least one GS inverse rendering method to demonstrate the benefits of using ASG over IR.

• Rendering Time Breakdown: It would be helpful to show a breakdown of rendering time for each sub-step to identify the bottleneck preventing the model from achieving a similar FPS as the original GS.

• Additional Visualizations: I would like to see visualizations of the learned ASGs and normal maps.

• Performance on Ref-NeRF's In-the-Wild Data: I am curious to see how this method performs on Ref-NeRF's in-the-wild data, which contains more challenging mirror-like reflections. 

• Missing References: Some important citations are missing. Here are a few:
		○ Reflection Modeling in 3DGS:
			§ "3D Gaussian Splatting with Deferred Reflection"
		○ Reflection Modeling in Point Cloud:
			§ "Neural Point Catacaustics for Novel-View Synthesis of Reflections"
		○ Inverse Rendering in NeRF:
			§ "PhySG: Inverse Rendering With Spherical Gaussians for Physics-Based Material Editing"
		○ Reflection Modeling in NeRF:
			§ "NeRF-Casting: Improved View-Dependent Appearance with Consistent Reflections"
			§ "SpecNeRF: Gaussian Directional Encoding for Specular Reflections"
questions: questions: Please see the weakness.
limitations: limitations: The modeling of complex reflections (e.g., self-reflections, mirror-like reflections) could be further discussed.
Also it would be helpful to demonstrate some failure cases.
paper_summary: paper_summary: Spherical harmonics-based 3D Gaussian splatting (3DGS) struggles with specular and anisotropic components. To address this problem, the paper proposes adopting anisotropic spherical Gaussians (ASG). However, directly adopting ASG does not demonstrate superior performance in representing specular and anisotropic parts. Therefore, the paper proposes separating diffuse and specular components from color representations and using a feature decoupling MLP to generate colors from ASG features. Through experiments, the paper demonstrates improved ability to represent highly specular parts.
strength_weakness: strength_weakness: **Related Works**

The related works section could better highlight the differences and advantages of this paper compared to other studies. The paper does not mention other works on specular scenes and objects. For instance, GaussianShader is another 3D Gaussian splatting-based method for specular scenes and objects, yet it is not mentioned in the related works section, even though it is referenced elsewhere in the paper. The related works section should compare this method with prior works, such as SpecNeRF (CVPR 2024), and highlight their limitations and how the proposed method overcomes them, or emphasize the novelty of this paper.





**Mathematical Notations**

The mathematical notations could be improved.
The inner product is represented both by $\cdot$ (Line 145, Eq. 11) and $\langle \rangle$ (Eq. 10).
Additionally, $\cdot$ denotes element-wise multiplication (Eqs. 5 and 6), inner product (Line 145, Eq. 11—inside the parenthesis), scalar-vector multiplication (Eq. 11—outside the parenthesis), and scalar-scalar multiplication (Eqs. 4 and 12).
At least element-wise multiplication and inner product should use different notations.
And using the same notation for different operations within a single equation (Eq. 11) should also be avoided.





**Coarse-to-fine Training**

Coarse-to-fine training is not a novel approach within the 3DGS framework.
For example, “EAGLES: Efficient Accelerated 3D Gaussians with Lightweight EncodingS” already proposed a similar coarse-to-fine training approach.
questions: questions: Line 144: Is $\xi$ really $\mathcal{R}^2$ or is this a typo of $\mathcal{R}$?



Lines 185: Could you specifically state what is decoupled through $\psi$.

Line 191: There are no equations for pure ASG nor pure MLP. Could you state the equations or clarify how they work? For example, for pure MLP, is $\kappa$ removed from Eq. 10 and only $\Psi(\gamma(d), \langle n, -d\rangle)$ used?



Lines 232-234: This seems contradictory to lines 178-179. Could you clarify which one was actually used during the experiments.



Lines 242-244 and Tabs. 3 and 4: Based on the explanation, Ours-light refers to 3DGS + ASG, while Ours-w/ anchor refers to Scaffold-GS + ASG. However, it is unclear what the performance version (Ours) refers to. As stated in lines 220 and 221, is the performance version the same as the light version (3DGS + ASG) with a lower threshold $\tau_g$? If so, clarifying this in the experimental section would help readers understand what the performance version is.



Fig 6: Lines 181-183 and 277 state that directly using ASG leads to an inability to represent specular and anisotropic components. If “Ours w/o MLP” is the one, clarifying this in the caption could help with understanding. In addition, the meaning of “MLP” in “Ours w/o MLP” is ambiguous. It is unclear whether MLP in Fig. 6 refers only to $\Psi$ (Eq. 10) or both $\Psi$ and $\phi$ (line 233).




**Color separation**

As shown in Eq. 9, the proposed method separates diffuse and specular components.
What happens without color separation? Additionally, what happens if the diffuse part is also represented through ASG? Based on the ASG paper, replacing SH with ASG can improve performance. Therefore, could you provide a reason why the diffuse part still uses SH?
limitations: limitations: The paper stated its limitation.
paper_summary: paper_summary: This paper introduces an anisotropic spherical Gaussian (ASG) appearance field into 3D Gaussian splatting for modeling the view-dependent appearance of each 3D Gaussian, which increases the ability of 3D Gaussian in representing high-frequency information. The key idea of this paper is combining ASG and SH to model the color of each 3D Gaussian. The experiments show the effectiveness of the method in modeling specular highlights and rendering quality.
strength_weakness: strength_weakness: 1. The method of this paper includes several MLPs, however, the authors didn't report the training time of their model when compared with other methods. I think the training time should be considered for a more fair comparison. If I missed this, please correct me. 
2. The structure of 3D Gaussian in this paper is mainly based on Scaffold-GS, anchor-based Gaussian splatting. Therefore, the performance of the method in this paper may be degraded when the vanilla Scaffold-GS cannot perform well, such as the scene dominated by large texture-less regions.
questions: questions: 1. It would be better if the authors could include the training time of their model when comparing it with other methods.
2. If we use RGB color as the diffuse color directly, instead of using SH to model, whether there will be a better or a worse result, since the high order SH is used in 3D-GS to represent high-frequency information.
limitations: limitations: Yes. The authors discussed the limitations of their work.

Comments: title: Please review author rebuttal!
Please review author rebuttal!
Please review author rebuttal!
Official Comment by Authors
comment: We thank the reviewer for the valuable comments and are glad to hear that our previous answers helped you better understand our work. We are keen to follow up on the provided suggestions:

- We will incorporate our response to reviewer `8TqS` regarding section 3 into the paper to enhance clarity.
- We will slightly tone down our claims and retain only the statement: "An anisotropic dataset has also been created to assess the capability of our model in representing anisotropy."
I thank the authors for the rebuttal. It addressed many of my questions, e.g.: (1) what kinds of effects the ASGs are supposed to model, and (2) which qualitative improvements each contribution (ASGs, coarse-to-fine training) is responsible for. 

I think, perhaps, I judged the work a bit too harshly on my first pass. Although the changes made by the authors are not huge, together they comprise a very effective system for reconstruction/view synthesis of objects with strong view dependent appearance (supported by strong qualitative/quantitative results in the paper, and additional results provided in the rebuttal). 

I would still suggest that the authors focus on improving clarity (e.g. by incorporating their response to reviewer 8TqS about section 3 into the paper), and also slightly tone down their claims. While I appreciate the distinction between specular highlights and reflections, specular highlights are a subset of effects caused by strong reflections -- so I would say that GaussianShader/NeRFCasting does attempt to model such affects, although they do not *specifically* focus on these effects.
Thanks for providing the details of the training time of methods and the results using RGB color directly, which answers my questions. I also have read other reviewer's comments. I will keep my initial rating.
Thanks for the additional explanation and experiments. I have no further comments.
Thanks so much for the constructive feedback. We hope that our response below will address your concerns.

**Q1: Why increasing the number of ASGs does not lead to increasing of rendering qualities.**

As shown in Eq. 10, the number of ASGs affects the input dimension of the decoding MLP. While increasing the number of ASGs theoretically enhances the encoding capability of specular features, it may also lead to overfitting in the decoding MLP. This can reduce the model's generalization ability, resulting in a decline in rendering metrics on the test set.

**Q2: More explanations about L1-normed gradients.**

Thanks to the reviewer for the in-depth analysis. We would like to explain the rationale behind using the L1-norm from two perspectives.
- From a theoretical standpoint, L1-norm gradients can alter the distribution of densification. This allows regions that need densification to be correctly densified without producing floaters, while regions that do not need densification can avoid excessive growth, thereby reducing memory overhead. This is something that simply lowering the densification threshold cannot achieve.

- From an experimental perspective, we conducted experiments where we lowered the threshold, with $\tau_g=0.0002$ being the densification threshold for vanilla 3D-GS.
| Method                               | PSNR  | SSIM  | LPIPS | Mem  | FPS  |
| ------------------------------------ | ----- | ----- | ----- | ---- | ---- |
| Ours (w/ L1 & $\tau_g=0.0005$)       | 28.18 | 0.835 | 0.176 | 848  | 33   |
| Ours-light (w/ L1 & $\tau_g=0.0006$) | 28.07 | 0.834 | 0.183 | 684  | 44   |
| Ours (w/o L1 & $\tau_g=0.0001$)      | 28.12 | 0.831 | 0.187 | 1619 | 18   |
| Ours (w/o L1 & $\tau_g=0.0002$)      | 28.05 | 0.828 | 0.194 | 1044 | 26   |

The experimental results show that our method can improve rendering metrics without increasing memory usage, demonstrating that the L1 norm is more effective than simply lowering the threshold. Beyond the improvement in metrics, the more important observation is that lowering the threshold still results in visual floaters, whereas the L1-norm can effectively remove them.
I appreciate the effort authors put into the rebuttal. Overall I'm satisfied with the rebuttal and it addresses most of my concerns. However, here are a few further comments:

1. It would be helpful to explain why further increasing the number of ASGs does not lead to increasing of rendering qualities, which is somehow counter-intuitive

2. Regarding the L1 norm of gradients, I believe it's totally reasonable to accumulate negative gradients both for optimization and densification, as the goal of the densification is to aid in optimization. My understanding is that accumulating L1-normed gradient tends to results in larger accumulation and could trigger densification more easily. So one experiment that may worth trying is to lower the threshold for densification, which could have a similar effect. One scenario where using L1-norm may be beneficial is in cases of view inconsistency, where a 3D Gaussian may have gradient in different directions. In this situation, using L1-norm will tends to densify and help to explain the view inconsistency with more gaussians. But in any case, simply stating that the original method is unreasonable without further explanation seems vague and potentially confusing.
I appreciate you for further addressing my concern. Thank you.
Dear Reviewer,

I wanted to gently remind you to please review the rebuttal provided by the authors. Your feedback is invaluable to the decision-making process, and if you feel that the rebuttal addresses any of your concerns, please consider updating your score accordingly.

Thank you for your continued dedication to ensuring a fair and thorough review process!

Best, Your AC
Dear Reviewer,

I wanted to gently remind you to please review the rebuttal provided by the authors. Your feedback is invaluable to the decision-making process, and if you feel that the rebuttal addresses any of your concerns, please consider updating your score accordingly.

Thank you for your continued dedication to ensuring a fair and thorough review process!

Best, Your AC
Dear Reviewer,

I wanted to gently remind you to please review the rebuttal provided by the authors. Your feedback is invaluable to the decision-making process, and if you feel that the rebuttal addresses any of your concerns, please consider updating your score accordingly.

Thank you for your continued dedication to ensuring a fair and thorough review process!

Best, Your AC
Thank you for the prompt response and your suggestions. 

As mentioned in the global response, our method has three variants: the 3D-GS-based `Ours` and `Ours-light`, and the Scaffold-GS-based `Ours-w/ anchor`. 

In Section 3, 
- Sec 3.1 is the Preliminaries, where 3D Gaussian splatting and Anchor-based Gaussian splatting are introduced as explanations of 3D-GS and Scaffold-GS, respectively. The Anisotropic Spherical Gaussian is introduced to explain the ASG formula, which will be applied to each of our variants to encode specular features.
- Sec 3.2 is the modeling of view-dependent appearance for `Ours` and `Ours-light`.
- Sec 3.4 is the modeling of view-dependent appearance for `Ours-w/ anchor`, which is slightly different from Sec 3.2. 
- Sec 3.3 is a general part used for all three variants of our method, aimed at removing floaters in real-world scenes.

We hope our explanation can resolve your confusion about Section 3. And we fully agree that clarifying these details will strengthen the readability of the paper. We will include more detailed explanations for Section 3 in the paper.
I appreciate the authors for their detailed rebuttal. It has addressed concerns to some extent. However, I agree with reviewer anMP that the writing could be improved. Specifically, the rebuttal did not address the following point: "In section 3, it is unclear which components belong to the base method and which are part of the method variants." Clarifying these details could strengthen the paper.