Spec-Gaussian: Anisotropic View-Dependent Appearance for 3D Gaussian Splatting
Authors: Ziyi Yang
Xinyu Gao
Yang-Tian Sun
Yi-Hua Huang
Xiaoyang Lyu
Wen Zhou
Shaohui Jiao
XIAOJUAN QI
Xiaogang Jin
Venue: NeurIPS 2024 poster (2024)
Keywords: Gaussian Splatting
specular highlights modeling
real-time rendering
Primary area: machine_vision
TL;DR: We employ ASG appearance field to enhance the 3D-GS for modeling scenes with specular highlights.

Abstract: The recent advancements in 3D Gaussian splatting (3D-GS) have not only facilitated real-time rendering through modern GPU rasterization pipelines but have also attained state-of-the-art rendering quality. Nevertheless, despite its exceptional rendering quality and performance on standard datasets, 3D-GS  frequently encounters difficulties in accurately modeling specular and anisotropic components. This issue stems from the limited ability of spherical harmonics (SH) to represent high-frequency information. To overcome this challenge, we introduce Spec-Gaussian, an approach that utilizes an anisotropic spherical Gaussian (ASG) appearance field instead of SH for modeling the view-dependent appearance of each 3D Gaussian. Additionally, we have developed a coarse-to-fine training strategy to improve learning efficiency and eliminate floaters caused by overfitting in real-world scenes. Our experimental results demonstrate that our method surpasses existing approaches in terms of rendering quality. Thanks to ASG, we have significantly improved the ability of 3D-GS to model scenes with specular and anisotropic components without increasing the number of 3D Gaussians. This improvement extends the applicability of 3D GS to handle intricate scenarios with specular and anisotropic surfaces.

Spec- Anisotropic View-Dependent Appearance for 3D Gaussian Splatting
Ziyi Yang
Xinyu Gao
Yang-Tian Sun?
Yi-Hua Huang²
Xiaoyang Lyu?
Wen Zhou
Shaohui Jiao?
Xiaojuan Qi
Xiaogang Jin
State Key Lab of CAD&CG, Zhejiang University
Thh University of Hong Kong
BByteDance Inc.
Abstract
The recent advancements in 3D Gaussian splatting (3D-GS) have not only facil- itated real-time rendering through modern GPU rasterization pipelines but have also attained state-of-the-an rendering quality. Nevertheless, despite its excep- tional rendering quality and performance standard datasets, 3D-GS frequently encounters difficulties accurately modeling specular and anisotropic compo- nents. This issue stems from the limited ability of spherical harmonics (SH) represent high-frequency information. To overcome this challenge, we introduce Spec-Gaussien an approach that utilizes an anisotropic spherical Gaussian (ASG) appearance field instead of SH for modeling the view-dependent appearance of each 3D Gaussian. Additionally, we have developed coarse-to-fine training strategy to improve learning efficiency and eliminate floaters caused by overfit- ting in real-world scenes. Our experimental results demonstrate that our method surpasses existing approaches in terms of rendering quality. Thanks to ASG, we have significantly improved the ability of 3D-GS model scenes with specular and anisotropic components without increasing the number of 3D Gaussians. This improvement extends the applicability of 3D GS to handle intricate scenarios with specular and anisotropic surfaces. Our codes and datasets are available at https:://nggra 4m. wtwp://oeeeeeessssaaaaeeeeeee
Introduction
High-quality reconstruction and photorealistic rendering from collection of images are crucial for variety of applications, such augmented reality/virtual reality (AR/VR), 3D content production. and art creation. Classic methods employ primitive representations, like meshes (39 and points 4 67], and take advantage of the rasterization pipeline optimized for contemporary GPUs to achieve real-time rendering. In contrast, neural radiance fields (NeRF) [37 5 38 utilize neural implicit representation to offer a continuous scene representation and employ volumetric rendering to produce rendering results. This approach allows for enhanced preservation of scene details and more effective reconstruction of scene geometries.) |
Recently, 3D Gaussian Splatting (3D-GS) (2 has emerged as leading technique, delivering state- of-the-art quality and real-time speed. This method optimizes a set of 3D Gaussians that capture the appearance and geometry of a 3D scene simultaneously, offering a continuous representation that preserves details and produces high-quality results. Besides, the CUDA-customizes differentiable rasterization pipeline for 3D Gaussians enables real-time rendering even at high resolution.
Corresponding Authors.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

30-65 (27.0AdB)
Scaffold (32.46d8)
Ours 50dB)
Ground
Ours 21_6
Our method not only achieves real-time rendering but also significantly enhances the capability of 3D-GS to model scenes with specular and anisotropic components. Key to this enhanced performance is our use of ASG appearance field to model the appearance of each 3D Gaussian, which results in substantial improvements in rendering quality for both complex and general scenes.
Figure 1:
Despite its exceptional performance, 3D-GS struggles to model specular components within scenes (see F. This issue primarily stems from the limited ability of low-order spherical harmonics (SH) to capture the high-frequency information in these scenarios. Consequently, this poses challenge for 3D-GS to model scenes with reflections and specular components, as illustrated in Fig. ||
To address the issue, we introduce novel approach called Spec-Gaussiant which combines anisotropic spherical Gaussian (ASG) [60] for modeling anisotropic and specular components, an effective training mechanism to eliminate floaters and improve learning efficiencies, and anchor- based 3D Gaussians for acceleration and storage reduction. Specifically, the method incorporates two key designs: 1) A new 3D Gaussian representation that utilizes an ASG appearance field instead of SH to model the appearance of each 3D Gaussian. ASG with a few orders can effectively model high-frequency information that low-order SH cannot. This new design enables 3D-GS more effectively model anisotropic and specular components in static scenes. 2) A coarse-to- to-fine training scheme specifically tailored for 3D-GS is designed to eliminate floaters and boost learning efficiency, This strategy effectively shortens learning time by optimizing low-resolution rendering in the initial stage, preventing the need increase the number of 3D Gaussians and regularizing the learning process to avoid the generation of unnecessary geometric structures that lead to floaters.
By combining these advances, our approach can render high quality results for specular highlights and anisotropy as shown in Fig. 4 while preserving the efficiency of Gaussians. Furthermore, comprehensive experiments reveal that method not only endows 3D-GS with the ability model specular highlights but also achieves wate-of-the-an results in general benchmarks.
In summary, the major contributions of our work are as follows:
A novel ASG appearance field to model the view-dependent appearance of each 3D Gaussian, which enables 3D-GS to effectively represent scenes with specular and anisotropic components. coarse-to-fing training scheme that effectively regularizes training to eliminate floaters and improve the learning efficiency of 3D-GS in real-world scenes. An anisotropic dataset has also been made to assess the capability of our model in representing anisotropy.
2
Related Work
2.1 Implicit Neural Radiance Fields
Neural rendering has attracted significant interest in the academic community for its unparalleled ability to generate photorealistic images. Methods like NeRF (37] utilize Multi-Layer Perceptrons (MLPs) to model the geometry and radiance fields of a scene. Leveraging the volumetric rendering equation and the inherent continuity and smoothness of MLPs, NeRF achieves high-quality scene reconstruction from a set of posed images, establishing itself as the state-of-the-an (SOTA) method for novel view synthesis. Subsequent research has extended the utility of NeRF to various applications, including mesh reconstruction
53
27
58 34), inverse rendering [8]] 72 31]62] 62), optimization of
camera parameters 29 55 54
few-shot learning [12 61 57), and anti-alian anti-aliasing [2 13.
However, this stream of methods relies on ray casting rather than rasterization to determine the color of each pixel. Consequently, every sampling point along the ray necessitates querying the MLPs,

---

Σ
SH.
ASGJ
SIM Points
Gaussian Densification
View-dependent Appearance Modeling
Figure 2: Pipeline of -pus-Gaus The optimization process begins with SfM points derived from
COLMAP or generated randomly, serving as the initial state for the 3D Gaussians. To address the limitations of low-order SH and pure MLP in modeling high-frequency information, we additionally employ ASG in conjunction with feature decoupling MLP to model the view-dependent appearance of each 3D Gaussian. Then, 3D Gaussians with opacity > 0 are rendered through differentiable Gaussian rasterization pipeline, effectively capturing specular highlights and anisotropy in the scene.
leading to significantly slow rendering speed and prolonged training convergence. This limitation substantially impedes their application in large-scene modeling and real-time rendering.
To reduce the training time of MLP-based NeRF methods and improve rendering speed, subsequent work has enhanced NeRF's efficiency in various ways. Structure- techniques [68]
14
43
17 7
have sought to improve inference training efficiency by caching distilling the implicit neural
representation into more efficient data structures. Hybrid methods [30 49 increase efficiency by
incorporating explicit voxel-based data structures. Factorization methods 55 18
g 16) apply low-
rank tensor assumption to decompose the scene into low-dimensional planes or vectors, achieving better geometric consistency. Compared continuous implicit representations, the convergence of individual voxels in the grid is independent, significantly reducing training time. Additionally, Instant-NGP 38 utilizes hash grid with corresponding CUDA implementation for faster feature querying, enabling rapid training and interactive rendering of neural radiance fields. Spec-NeRF [35] achieves high-quality specular reflection modeling by introducing Gaussian directional encoding.
Despite achieving higher quality and faster rendering, these methods have not fundamentally over- come the substantial query overhead associated with ray casting. As result, a notable gap remains before achieving real-time rendering. In this work, we build upon the recent 3D-GS [23]. point- based rendering method that leverages rasterization. Compared to ray casting-based methods, significantly enhances both training and rendering speed.
2.2 Point-based Neural Radiance Fields
Point-based representations, similar triangle mesh-based methods, exploit the highly efficient rasterization pipeline modern GPUs to achieve real-time rendering. Although these methods offer breakneck rendering speeds and well-suited for editing tasks, they often suffer from holes and outliers, leading to artifacts in the rendered images. This issue arises from the discrete nature of point clouds, which can create gaps in the primitives and, consequently, in the rendered image.
To address these discontinuity issues, differentiable point-based rendering [67 15 24 25 has been
extensively explored for fitting complex geometric shapes. Notably, Zhang et al. employ differentiable surface splatting and utilize radial basis function (RBF) kernel to compute the contribution of each point to each pixel.
71
Recently, 3D-GS (23] has employed anisotropic 3D Gaussians, initialized from Structure from Motion (SfM), to represent 3D scenes. The innovative densification mechanism and CUDA-customizae differentiable Gaussian rasterization pipeline of 3D-GS have not only achieved state-of-the--ar (SOTA) rendering quality but also sigmificantly surpassed the threshold of real-time rendering. Many concurrent works have rapidly extended 3D-GS variety of downstream applications, including
dynamic scenes 33
63
64
20
26
50
text-to-3D generation 28 51 66 I0,, avatars 774
73 21
45 40
scene editing [59
quality enhancement
36 and mesh reconstruction
34

---

Despite achieving SOTA results on commonly used benchmark datasets, 3D-GS still struggles to model scenes with specular and reflective components, which limits its practical application in real-time rendering the photorealistic level. In this work, by replacing spherical harmonics (SH) with an anisotropic spherical Gaussian (ASG) appearance field, we have enabled 3D-GS to model complex specular scenes more effectively.
3
Method
The overview of our method is illustrated in Fig. The input to our model is a set of posed images of static scene, together with sparse point cloud obtained from SIM [46]. The core of our method to use the ASG appearance field to replace SH in modeling the appearance of 3D Gaussians (Sec. (.2). Moreover, we introduce simple yet effective coarse-to-fine training strategy to reduce floaters in real-world scenes (Sec. B.3). To further reduce the storage overhead and rendering speed pressure introduced by ASG, we combine hybrid Gaussian model that employs sparse anchor Gaussians to facilitate the generation of neural Gaussians (Sec. 3.4 to model the 3D scene.
3.1 Preliminaries
3D Gaussian splatting. 3D-GS [23 is point-based method that employs anisotropic 3D Gaussians to represent scenes. Each 3D Gaussian is defined by a center position æ, opacity o, and 3D covariance matrix Σ, which is decomposed into a quaternion r and scaling The view-dependent appearance each 3D Gaussian is represented using the first three orders of spherical harmonics (SH). This method not only retains the rendering details offered by volumetric rendering but also achieves real-time rendering through a CUDA-custominn process. Following (7). the 3D Gaussians can be projected to 2D using the 2D covariance matrix Σ', defined as:
differentiable Gaussian rasterization
=' JVEV JT.
(1)
where J is the Jacobian of the affine approximation of the projective transformation, and V represents the view matrix, transitioning from world to camera coordinates. To facilitate learning, the 3D) covariance matrix E is decomposed into two learnable components: the quaternion r, representing rotation, and the 3D-vector 8, representing scaling. The resulting Σ is thus represented as the combination of rotation matrix It and scaling matrix S as:
Σ = RSST
The color of each pixel on the image plane is then rendered through a point- volumetric rendering (alpha blending) technique:
C(p) "ETac, VEN
a; 0,C -DP- ((p-1.)
where p denotes the pixel coordinate, T; is the transmittance defined by I1 (1 - az), signifies the color of the sorted Gaussians associated with the queried pixel, and μ4 represents the coordinates of the 3D Gaussians when projected onto the 2D image plane.
Anisotropic spherical Gaussian. Anisotropic spherical Gaussian (ASG) 60 has been designed in the traditional rendering pipeline efficiently approximate lighting and shading. Different from spherical Gaussian (SG), ASG has been demonstrated to effectively represent anisotropic scenes with small number. In addition to retaining the fundamental properties of SG, ASG also exhibits rotational invariance and can represent full frequeeeyy signals. The ASG function defined as:
ASG((v (x 2), [A, p),
5(::2) A(uxx) -(uxx)"=((x>?
where v is the unit direction serving as the function input; x, y, and z correspond to the tangent, bi- bi-tangent, and lobe axis, respectively, and are mutually orthogonal; À R and μ R are the sharpness parameters for the x- and y-xxis, satisfying X, f > 0; & E R is the lobe amplitude; is the smooth term defined as S(uzz) max(y z, 2).
Inspired by the power of ASG in modeling scenes with complex anisotropy, we propose integrating ASG into Gaussian splatting to join the forces of classic models with new rendering pipelines for

---

higher quality. For N ASGs, we predefined orthonormal axes x, y. and z, initializing them to be uniformly distributed across hemisphere. During training, we allow the remaining ASG parameters. the input to query ASG for modeling
X. 14, and Ç. to be learnable. We use the reflect direction the view-dependent specular information. Note that we use N 32 ASGs for each 3D Gaussian.
Anchor-based Gaussian splatting. Anchor-based Gaussian splatting was first proposed by Scaffold-GS [32] Unlike the attributes carried by each entity in 3D-GS, each anchor Gaussian carries position coordinate P. E IR$ local feature f, E R displacement factor 7/o R and learnable offsets O, E R4x3 They use the COLMAP [46] point cloud to initialize each anchor 3D Gaussian, serving as the voxel centers to guide the generation of neural Gaussians. The position P. of the anchor Gaussian is initialized as:
P P. = (1- + 0.5 1.0......
(5)
where P is the point cloud position, is the voxel size, and {-} denotes removing duplicated anchors.
Then anchor Gaussians can guide the generation of neural Gaussians, which have the same attributes as vanilla 3D-GS. For each visible anchor Gaussian within the viewing frustum, we spawn neural Gaussians and predict their attributes. The positions x of neural Gaussians are calculated as:
[xo ................................ x P. fO...... .......,...1) 0x-1) Te
(6)
where P. represents the position of the anchor Gaussian corresponding to k neural Gaussians. The opacity a calculated through a tiny MLP:
(oo,
0k-1) ;
Fo (fo, dew, dow),
where 6.v denotes the distance between the anchor Gaussian and the camera, and d., is the unit direction pointing from the camera to the anchor Gaussian. The rotation and scaling s of each neural Gaussian are derived similarly using the corresponding tiny MLP F, and F..
3.2 Anisotropic View-Dependent Appearance
ASG appearance field for 3D Gaussians. Although SH has enabled view-dependent scene modeling, the low frequency of low-order SH makes it challenging to model scenes with complex optical phenomena such as specular highlights and anisotropic effects. Therefore, instead of using SH, we propose using an ASG appearance field based on Eq. (4) to model the appearance of each 3D Gaussian. However, the introduction of ASG increases the feature dimensions of each 3D Gaussian, raising the model's storage overhead. To address this, we employ a compact learnable MLP θ to predict the parameters for N ASGs, with each Gaussian carrying only additional local features f E 24 as the input to the MLP: ((f) - (,,
To better differentiate between high and low- fow-frequeccy information and further assist ASG in fitting nigh-frequency specular details, we decompose color into diffuse and specular components:
Cd + Cs.
where ça represents the diffuse color, modeled using the first three orders of SH, and c, the specular color calculated through ASG. We refer to this comprehensive approach appearance modeling as the ASG appearance field.
Although ASG theoretically enhance the ability of SH model anisotropy, directly using ASG to represent the specular color of each 3D Gaussian still falls short in accurately modeling anisotropic and specular components, demonstrated in Fig. 6 Inspired by [16), do not use ASG directly to represent color but instead employ ASG to model the latent feature each 3D Gaussian. This latent feature, containing anisotropic information, is then fed into tiny feature decoding MLP v to determine the final specular color:
V (N, (d), (n, -d) C..
(10)
ASG(wr [x,Y, De.
5

---

Table 1: Quantitative Comparison on anisotropic synthetic dataset.
baisset PSNR
Amisonepii Synihetic
Mathod
SSIM
1PIPS |
FPS
Men
Nam.(k)
3D:GS
01506
01062
47MB
201
Scaffeld-GS
0.052
27MBB
Ours-wl w/ anchor
36.76
0.976
0.016
180
25MB
Oun-liihtt
37:42
0.979
0.044
159
45MB
146
Figure 3: Using a coarse-to-fine strategy, our approach can eliminate the floaters without increasing the number of GS.
Ours 37.70
0.980
0.042
145
57MB
183
where is the latent feature derived from ASG, ① denotes the concatenation operation, " represents the positional encoding, d is the unit view direction pointing from the camera to each 3D Gaussian, is the normal of each 3D Gaussian, and w, is the unit reflect direction. This strategy significantly enhances the ability of 3D-GS to model scenes with complex optical phenomena, whereas neither pure ASG nor pure MLP can achieve anisotropic appearance modeling as effectively as our approach.
the shortest axis of each Gaussian its normal. This approach is based on the observation that 3D Gaussians tend to flatten gradually during the optimization process, allowing the shortest axis to serve reasonable approximation for the normal.
Normal estimation. Following 22 47
The reflect direction wi can then be derived using the view direction and the local normal vector n as:
W/ = 2(w n) Was
(11)
where w, = -d is a unit view direction pointing from each 3D Gaussian in world space to the camera. We the reflect direction w, query ASG, enabling better interpolation of latent features containing anisotropic information. Experimental results show that although this unsupervised normal estimation cannot generate physically accurate normals aligned with the real world, sufficient to produce relatively accurate reflect direction to assist ASG in fitting high-frequency information.
3.3 Coarse-to-fine Training
We observed that in many real-world scenarios, 3D-GS tends overfit the training data, leading to the emergence of numerous floaters when rendering images from novel viewpoints. One important reason is that the COLMAP point cloud is too sparse. Poor initialization makes it difficult for 3D-GS to compensate for overly sparse areas through densification during optimization, leading to floaters in the rendering images. Moreover, 3D-GS accumulates gradients from each pixel to the GS: dP Σ dL dp. and the densification occurs when the accumulated amount exceeds threshold dx
dp, dx Tg 0.0002. However, having positive and negative gradients can cause GSs that should be densified to be ignored due to the large negative gradient.
Thus, mitigate the occurrence of floaters in real-world scenes, we propose a coarse-to- fine training mechanism. We first impose an Ll constraint on the gradients from pixels GS: de accumulating the numerical contribution from pixels GS rather than gradients. This idea similar to the concurrent works [65 70). Next, to avoid overfitting caused by excessive growth of 3D-GS during the early stages of optimization, we decide train 3D-GS progressively from low high resolution in real-world scenes:
ax Σ dL dp. dp. dx ||
r(i) min((rs + (re rr)) i/7l,re),
(12)
where r(i) is the image resolution at the i-th training iteration, T. is the starting image resolution, Te the ending image resolution (the full resolution we aim to render), and 7 is the threshold iteration, empirically set to 5k.
This training method allows 3D-GS to densify correctly and prevents excessive growth of 3D-GS in the early stages. Additionally, due to the lower resolution training in the initial phase, this mechanism reduces training time by approximately 10%. In our experiments, we offer a performance version with Tg 0.0005 and light version with Ta 0.0006.

---

24si/0.61
20.26/0.234
38.5k/0.077
20.75/0.334
Tri-MipRE
3D-GS
Scaffold-GS
GS-Shader
Ours
Figure 4: Visualization NeRF dataset. Our method has achieved specular highlights modeling,
which other 3D--GS-based methods fail to accomplish, while maintaining fast rendering speed.
Dataset
Mip-NeRI-
Mip-NeRI
Mip-NeRF 360
Method Metrics
PSNR 1
SSIM 1
LPIPS
FPS
Mcm
PSNR 1
SSIM f
LPIPS
PSNR
SSIM |
LPIPS
Plenoxels
2.IGB
21.68
0.491
0.426
INGP
25.59
0.699
0.331
9.43
48MB
22.75
0.567
0.403
29.14
0.863
0.242
Mip-NeRF360
27.69
0.792
0.237
8.6MB
24.47
31.72
0.180
3D-GS
27.79
0.826
0.202
115
748MB
25.02
0.742
0.232
31.25
0.931
0.164
Seaffold-GS
27.98
0.824
0,207
203MB
25.07
0.736
0.243
31.61
0.933
0.162
Ours anchor
28.14
0.824
0.196
70
260MB
24.98
0.735
0.223
32,09
0.935
0.161
Ours-light
28.07
0.834
0.183
684MR
25.09
0.752
0.203
31.80
0.936
0.158
Ours
28.18
0.835
0.176
33
847MB
25.11
0.754
0.195
32.01
0.937
0.153
Table 2: Quantitative comparison of on real-world datasets. We report PSNR, SSIM, LPIPS
(VGG) and color each cell as best, second best and third best Our method has achieved the best rendering quality, while striking a good balance between FPS and the storage memory.
3.4 Adaption for Machor-Based Gaussian Splatting
While the ASG appearance field significantly improves the ability of 3D-GS model specular and anisotropic features, introduces additional computational overhead due to the additional local features f associated with each Gaussian. Inspired by [32], employ anchor-based Gaussian splatting to reduce storage overhead and accelerate the rendering.
Since the anisotropy modeled by ASG is continuous in space, it can be compressed into lower- dimensional space. Thanks to the guidance the anchor Gaussian, the anchor feature f, can be used directly to compress N ASGs, further reducing storage pressure. To make the ASG of neural Gaussians position-awarr we introduce the unit view direction to decompress ASG parameters. Consequently, the ASG parameters prediction in Eq. (8) is revised as follows:
(f., den) + {A, E.EN.
(13)
where den denotes the unit view direction from the camera to each neural Gaussian. Additionally, we set the diffuse part of the neural Gaussian Cu = o(f.), directly predicted through an MLP ó. to ensure the smoothness of the diffuse component and reduce the difficulty of convergence.
3.5 Losses
We optimize the learnable parameters and MLPs using the same loss function as 3D-GS 23 The total supervision is given by:
L= Ap CDSSSMMC
(14)
where the Àp Ap.SSIM 0.2 is consistently used in our experiments.
4 Experiments
In this section, we present both quantitative and qualitative results of our method. To evaluate its effectiveness, compared it to several state-of-the- art methods across various datasets. We color
7

---

Table 3: Results on NeRF synthetic dataset.
Table 4: Results on NSVF synthetic dataset.
Datneet
Nomf-Symbmii
Doraset
SyVF-Syntetic
Method Metnes
PSNR
SSIM
LPIPS 4
FPS
Mem
Mechod Meriis
PSNK
SSIM
LPIFS L
FFS
Mem
INGP base
33.18
0.56.3
0.035
10
BMB
Tensolel
36,52
0,952
0,026
1.3
65MB
Mip-NoRtF
33.09
0.961
0.943
10MB
Ti-MigRF
34,58
0,973
0430
<
c4MB
Tr-MippF
33665
0.963
0042
<.5
60MB
NeuRBF
37.80
0,986
0.019
580MB
15.52
Seaffold-CIS
33.68
Scuffoli-GS
17MB
Ous wt unchor
33.96
0.969
0032
172
19MB
Our- w/ anchoe
37.21
0.957
0.015
152
16MB
Ouns-lihtt
14.08
0.970
01029
148
SAMB
Oun-light
38.25
0.953
0.013
124
70MB
Ours
34.19
0.971
0.028
121
72MB
Ours
38.40
0.963
0,012
106
S9MD
Mip-NeRF 360
3D-GS
Scaffold-GS GS
Ours
GT
Figure 5: Visualization Mip-NeRF 360 indoor scenes. Our method achieves superior recovery of specular effects compared to SOTA methods.
each cell best second best and third best Our method includes three versions, each based on different foundational methods with distinct hyperparameter settings. The performance version (Ours) is based 3D-GS (23] with Tg 0.0005: the light version (Ours- (Ours-light), also based 3D-GS, has Tg = 0.0006; and the mini version (Ours-w/ anchor) is based on Scaffold-GS [32], with Tg 0.0006. Our method demonstrates superior performance in modeling complex specular and anisotropic features, as evidenced by comparisons on the NeRF, NSVF, and our "Anisotropic Synthetic" datasets. Additionally, showcase its versatility by comparing its performance in diffuse scenarios, further proving the robustness of our approach.
4.1 Implementation Details
We implemented our framework using PyTorch 42 and modified the differentiable Gaussian rasteri- zation to include depth visualization. For the ASG appearance field, the decoupling MLP / consists of layers, each with 64 hidden units, and the positional encoding for the view direction is of order 2. Regarding coarse-to-fine training, which is applied only to real-world scenes to remove floaters, we start with a resolution rs that is 4x downsampled. To further accelerate rendering, we prefilter and allow only those Gaussians with opacity
0 to pass through the ASG appearance field and Gaussian rasterization pipelines. All experiments were conducted on an NVIDIA RTX 3090.
4.2 Results and Comparisons
Synthetic bounded scenes. We used the NeRF, NSVF, and our "Anisotropic Synthetic" datasets as the experimental datasets for synthetic scenes. Our comparisons were made with the most relevant state-of-the-an methods, including 3D-GS (23), Scaffold-GS [32]. GaussianShader [22]. and several
NeRF-based methods such as NSVF 300 TensoRF 5) NeuRBF
[ and Tri-MipRF 18)
As shown in Fig. (with PSNR and LPIPS), and Tabs. 3 our method achieved the highest performance with fewer Gaussians compared to vanilla 3D- It also improved upon the issues that 3D-GS faced in modeling figh-frequency specular highlights and complex anisotropy as shown in Tab. with fewer Gaussians and better metries. See more the supplementary materials.
Real-world unbounded scenes. To verify the versatility of our method in real-world scenarios, we used the Mip360 2 dataset, which contains indoor scenes with specular highlights. As shown in Tab. 2 our method surpasses state-of-the-art methods on Mip-NeRF 360. Furthermore, our method effectively balances FPS, storage, and rendering quality. It enhances rendering quality without increasing storage or significantly reducing FPS. As illustrated in Fig. 5 and Fig. 7 our method has

---

28 $9/0.99
28 6846//
27 990019
25 88/00/0868
28-7800077
26.52/0.041
30.66/0.035
33.92/0.023
24.95/0.023
37.50/0.018
3D-GS w/ 6-SH
Scaffold w/ 4-layers
Ours w/o MLP
Ours w/o ASG
Ours
GT
Figure 6: Ablation ASG appearance field. We show that directly using ASG to model color leads to the failure in modeling anisotropy and specular highlights. By decoupling the ASG features through MLP, we can realistically model complex optical phenomena.
0.180//4477
3D-GS
Scaffold
Ours c2f'
norm
GT
Figure 7: Ablation on coarse-to-fine training. Experimental results demonstrate that our simple yet effective training mechanism can effectively remove floaters without increasing the number of 3D Gaussians, thereby alleviating the overfitting problem prevalent in 3D-GS-based methods.
also significantly improved the visual effect. It removes a large number of floaters in outdoor scenes and successfully models the high- fighffreuenccy specular highlights in indoor scenes. This demonstrates that our approach is not only adept at modeling complex specular scenes but also effectively improves rendering quality in general scenarios.
4.3 Ablation Study
ASG feature decoupling MLP. We conducted an ablation study to evaluate key components of the ASG appearance field, which include ASG features, decoupling MLP, and the separation of diffuse and specular colors. As demonstrated in Fig. 6 (with PSNR and LPIPS), directly using ASG to output color results in the inability model specular and anisotropic components. In contrast to directly using an MLP for color modeling. as in Scaffold-GS (32, separately modeling diffuse and specular color enhance the fitting ability for high-frequency information. ASG encode higher- frequency anisotropic features. With the help of ASG's ability to encode high-frequency anisotropic features, the decoupling MLP can fit complex optical phenomena, leading more accurate rendering results. We also demonstrated that higher-order SH (6-order) and more MLP layers (4-layers) do not help 3D-GS and Scaffold-GS GS achieve satisfactory results, highlighting the importance of ASG.
Coarse-to-fine training. We conducted an ablation study to assess the impact coarse-to- fone (c2f) training. As illustrated in Fig. 7 (with LPIPS and number of Gaussian), both 3D-GS and Scaffold-Gr exhibit a large number of floaters in the novel view synthesis. Coarse-to-fing training effectively reduces the number of floaters, alleviating the overfitting issue commonly encountered by 3D-GS in real-world scenarios. Applying an LI constraint to the gradients used for 3D-GS densification further reduced the number of floaters and Gaussians. See more in the supplementary materials.
5 Conclusion
In this work, we introduce Spec-Gaussian, novel approach to 3D Gaussian splitting that features an anisotropic view-dependent appearance. Leveraging the powerful capabilities of ASG, our method effectively overcomes the challenges encountered by vanilla 3D-GS in rendering scenes with specular highlights and anisotropy. Additionally, we innovatively implement a coarse-to-fin training mechanism to climinate floaters in real-world scenes. Both quantitative and qualitative experiments

---

demonstrate that our method not only equips 3D-GS with the ability to model specular highlights and anisotropy but also enhances the overall rendering quality of 3D-GS in general scenes.
Limitations. Although our method enables 3D-GS model complex specular and anisotropic features, it still faces challenges in handling reflections. Specular and anisotropic effects are primarily influenced by material properties, whereas reflections are closely related to the environment and geometry. Due to the lack of explicit geometry in 3D-GS, we cannot differentiate between reflections and materials using constraints like normals, as employed in Ref-NeRF (52) and NeRO [3]. We plan to explore solutions for modeling reflections with 3D-GS in future work.
6 Acknowlegements
We thank Chao Wan from Cornell University for the help during rebuttal period. This work was supported by the National Natural Science Foundation of China (Grant No. 62036010). Ziyi Yang was also supported by ByteDance MMLab.
References
[1] Jonathan T. Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin-Brualla and Pratul P. Srinivasan. Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields. ICCV, 2021.
[2] Jonathan T Barron, Ben Mildenhall, Dor Verbin, Pratul P Srinivasan, and Peter Hedman. Mip-nerf 360: Unbounded anti-aliased neural radiance fields. In Proceedings of the IEEE/CVF Conference Computer Vision and Pottern Recognition, pages 5470-5479, 2022.
[3] Jonathan T. Barron, Ben Mildenhall. Dor Verbin, Pratul P. Srinivasan, and Peter Hedman. Zip-nerf: Anti-aliased grid-based neural radiance fields. ICCV, 2023.
[4] Mario Botsch, Alexander Horuung Matthias Zwicker, and Leif Kobbelt. High-quality surface splatting on today's gpus. Proceedings Eurographica/IEEE VGTC Symposium Point-Based Graphics, 2005., pages 17-141. IERE, 2005.
[5] Anpei Chen, Zexiang Xu, Andreas Geiger, Jingyi Yu, and Hao Su. Tensorf: Tensorial radiance fields. In European Conference on Computer Vision, pages 333 350. Springer. 2022.
[6] Yiwen Chen, Zilong Chen, Chi Zhang, Feng Wang. Xiaofeng Yang. Yikai Wang. Zhongang Cai, Lei Yang.
Huaping Liu, and Guosheng Lin. Gaussianeditor: Swift and controllable 3d editing with gaussian splatting, 2023.
[7] Zhiqin Chen, Thomas Funkhouser, Peter Hedman, and Andrea Tagliasacchi. Mobilenerf: Exploiting the polygon rasterization pipeline for efficient neural field rendering on mobile architectures. arXiv preprint arXiv:2208.0 00277, 2022.
[8] Zhang Chen. Zhong Li. Liangchen Song, Lele Chen, Jingyi Yu, Junsong Yuan, and Yi Xu. Neurbf: neural fields representation with adaptive radial basis functions. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4 4182-4194, 2023.
[9] Zilong Chen, Feng Wang, and Huaping Liu. Text-to-3d using gaussian splatting. arXiv preprint ar0iv:2309. 16585, 2023.
[10] Jaeyoung Chung, Suyoung Lee. Hyeongjin Nam, Jaerin Lee, and Kyoung Mu Lee. Luciddreamer: Domain-free generation of 3d gaussian splatting scenes. arXiv preprint arXiv:2311. 13..333, 2023.
[11] Pinxuan Dai, Jiamin Xu, Wenxiang Xie. Xinguo Liu, Huamin Wang, and Weiwei Xu. High-quality surface reconstruction using gaussian surfels. In ACM SIGGRAPH 2024 Conference Papers. Association for Computing Machinery. 2024.
[12] Kangle Deng, Andrew Liu, Jun-Yan Zhu, and Deva Ramanan. supth-super NeRF: Fewer views and faster training for free. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2022.
[13] Xinyu Gao, Ziyi Yang. Bingchen Gong. Xiaoguang Han, Sipeng Yang. and Xiaogang Jin, Towards realistic example-based based modeling via 3d gaussian stitching. arXiv preprint arXiv:2408. 15708, 2024.
[14] Stephan J Garbin, Marek Kowalski, Matthew Johnson, Jamie Shotton, and Julien Valentin. Fastnerf: High-fidelity neural rendering at 200fps. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 14346-14355, 2021.
[15] Markus Gross and Hanspeter Pfister. Point- baitt-based/ae graphics. Elsevier, 2011.
[16] Kang Han and Wei Xiang. Multiscale tensor decomposition and rendering equation encoding for view synthesis. In The IEEE/ / CVF Computer Vision and Pattern Recognition Conference, pages 4232-4241 4241, 2023.
[17] Peter Hedman, Pratul P Srinivasan, Ben Mildenhall, Jonathan T Barron, and Paul Debevec. Baking neural radiance fields for real-time view synthesis. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5875-5884, 2021.
[18] Wenbo Hu, Yuling Wang. Lin Ma. Bangbang Yang, Lin Gao, Xiao Liu, and Yuewen Ma. Tri-miprf: Tri-mip representation for efficient anti- anti-aliasing neural radiance fields. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 19774 19783, 2023.
10

---

[19] Binbin Huang, Zehao Yu, Anpei Chen, Andreas Geiger, and Shenghua Gao. 2d gaussian splatting for geometrically accurate radiance fields. In ACM SIGGRAPH 2024 Conference Pupers. pages 1-11. 2024. Yi-Hua Huang, Yang- Tian Sun, Ziyi Yang. Xiaoyang Lyu, Yan-Pei Cao, and Xiaojuan Qi. Se-gs: Sparse- controlled gaussian splatting for editable dynamic scenes. pages 1-11, 2023.
[20]
[21] Yuheng Jiang. Zhehao Shen, Penghao Wang. Zhuo Su, Yu Hong. Yingliang Zhang. Jingyi Yu, and Lan
Xu. Hifi4g: High-fidelity human performance rendering via compact gaussian splatting, arXiv preprint WwXiv:2312.03461 2023.
[22] Yingwenqi Jiang. Jiadong Tu, Yuan Liu, Xifeng Gao, Xiaoxiao Long, Wenping Wang, and Yuexin Ma.
Gaussianshader: 3d gaussian splatting with shading functions for reflective surfaces. arXiv preprint
arXiv:2311.179) 17977. 2023
[23] Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics, 42(4), July 2023.
[24] Leonid Keselman and Martial Hebert. Approximate differentiable rendering with algebraic surfaces. In
European Conference on Computer Vision. pages 596-614, Springer, 2022.
[25] Leonid Keselman and Martial Hebert. Flexible techniques for differentiable rendering with 3d gaussians.
arXiv preprint arXiv:2308. 14737, 2023.
[26] Zhan Li, Zhang Chen, Zhong Li, and Yi Xu. Spacetime gaussian feature splatting for real-time dynamic
view synthesis. arXiv preprint XxXiv:2312.66822 2023.
[27] Zhaoshuo Li, Thomas Mäller, Alex Evans, Russell H Taylor, Mathias Unberath, Ming- Yu Liu, and Chen-
Hsuan Lin. Neuralangelo: High-fidelity neural surface reconstruction. In IEEE Conference on Computer
Vision and Pattern Recognition (CVPR), 2023.
[28] Yixun Liang. Xin Yang. Jiantao Lin, Haodong Li, Xiaogang Xu, and Yingcong Chen. Luciddreamer:
Towards high-fidelity text-to-3d tod generation via interval score matching, 2023.
[29] Chen-Hsuan Lin, Wei-Chiu Ma, Antonio Torralha, and Simon Lucey. Barf: Bundle-adjusting neural
radiance fields. In JEEE International Conference on Computer Vision (ICCV), 2021.
[30] Lingjie Liu. Jiatao Gu, Kyaw Zaw Lin, Tat-Seng Chua, and Christian Theobalt. Neural sparse voxel fields.
Advances in Neural Information Processing Systems, 13:15651-15661 2020.
[31] Yuan Liu, Peng Wang, Cheng Lin, Xiaoxiao Long, Jiepeng Wang, Lingjie Liu, Taku Komura, and Wenping Wang. Nero: Neural geometry and brdf reconstruction of reflective objects from multiview images. In
SIGGRAPH, 2023.
[32] Tao Lu, Mulin Yu, Linning Xu, Yuanho Xiangli, Limin Wang, Dahua Lin, and Bo Dai. Scaffold-gs: Structured 3d gaussians for view-adantive rendering. arXiv preprint arXiv:2312.0010 2023.
[33] Jonathon Luiten, Georgios Kopanas, Bastian Leibe, and Deva Ramanan. Dynamic 3d gaussians: Tracking by persistent dynamic view synthesis. In 3DV, 2024.
[34] Xiaoyang Lyu, Yang- Tian Sun, Yi-Hua Huang, Xiuzhe Wu, Ziyi Yang. Yilun Chen, Jiangmiao Pang. and Xiaojuan Qi. 3dgsr: Implicit surface reconstruction with 3d gaussian splatting. arXiv preprint
arXiv:2404. 00409, 2024.
[35] Li Ma. Vasu Agrawal. Haithem Turki, Changil Kim, Chen Gao. Pedro Sander, Michael Zollhöfer. and
Christian Richardt. Specnerf: Gaussian directional encoding for specular reflections. In Proceedings of the
IEFE/CVF Conference on Computer Vision and Pattern Recognition, pages 21 21188-211999 198, 2024.
[36] Dawid Malarz, Weronika Smolak, Jacek Tabor, Slawomir Tadeja, and Przemyslaw Spurek. Gaussian
splatting with nerf-based color and opacity.
[37] Ben Mildenhall, Pratul Srinivasan, Matthew Tancik, Jonathan Barron, Ravi Ramamoorthi, and Ren
Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In ECCV, 2020.
[38] Thomas Müller. Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives
with a multiresolution hash encoding. ACM Trans. Gruph., 41(4):102:1-102:155 July 2022.
[39] Jacob Munkberg, Jon Hasselgren, Tianchang Shen, Jun Gao, Wenzheng Chen, Alex Evans, Thomas Müller, and Sanja Fidler. Extracting triangular 3d models, materials, and lighting from images. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 8280-8290, 2022. [40] Haokai Pang. Heming Zhu, Adam Kortylewski, Christian Theobalt, and Mare Habermann. Ash: Animatable gaussian splats for efficient and photoreal human rendering. 2023.
[41] Keunhong Park, Philipp Henzler, Ben Mildenhall, Jonathan T Barron, and Ricardo Martin-Brualla. Camp: Camera preconditioning for neural radiance fields. ACM Trunsactions on Gruphics (TOG), 12(6):1-11, 2023.
[42] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, perfforerrrmcce deep learning library. Advances in Neural Information Processing Systems, 32, 2019.
[43] Christian Reiser, Songyou Peng. Yiyi Liao, and Andreas Geiger. Kilonerf: Speeding up neural radiance fields with thousands of tiny mlps. In Proceedings of the IEEE/CVF International Conference on Computer
Vision, pages 14335-14345, 2021.
[44] Kerui Ren, Lihan Jiang. Tao Lu, Mulin Yu, Linning Xu, Zhangkai Ni, and Bo Dai. Octree-gs: Towards consistent real-time rendering with lod-structured 3d gaussians. arXiv preprint arXiv:2403. 17898, 2024.
[45] Shunsuke Saito, Gabriel Schwartz, Tomas Simon, Junxuan Li, and Giljoo Nam. Relightable gaussian codec avatars. 2023.
[46] Johannes L Schonberger and Man-Mie Frahm. fimeture-from-moote motion revisited. In Proceedings of the IEEE Conference on Computer Vision and Puttern recognition, pages 4104-4113, 2016.
11

---

[47]
Yahao Shi, Yanmin Wu, Chenming Wu, Xing Liu, Chen Zhao, Haocheng Feng, Jingtuo Liu, Liangjun Zhang, Jian Zhang. Bin Zhou, al. Gir: 3d gaussian inverse rendering for relightable scene factorization. arXiv preprint 23xi::3312.0113.... 2023.
[48]
Pratul P. Srinivasan, Boyang Deng. Xiuming Zhang, Matthew Tancik, Ben Mildenhall, and Jonathan T. Barron. Nerv: Neural reflectance and visibility fields for relighting and view synthesis. In CVPR, 2021. Cheng Sun. Min Sun, and Hwann- Tzong Chen. Direct voxel grid optimization: Super-fast convergence for radiance fields reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5459-5469, 2022.
[49]
[50]
Yang-Tive Miag-TianSa Sun, Yi-Hua Huang. Lin Ma, Xiaoyang Lyu, Yan-Pei Cao, and Xiaojuan Qi. Splatter a video:
Video gaussian representation for versatile processing, arXiv preprint arXiv:2406. 13870, 2024.
[51] Jiaxiang Tang, Jiawei Ren, Hang Zhou, Ziwei Liu, and Gang Zeng. Dreamgaussian: Generative gaussian
splatting for efficient 3d content creation. arXiv preprint arXiv:2309. 16653,6666333 2023.
[52] Dor Verbin. Peter Hedman. Ben Mildenhall, Todd Zickler. Jonathan T. Barron. and Pratul P. Srinivasan.
Ref-NeRF: Structured view-dependent appearance for neural radiance fields. CVPR, 2022.
[53] Peng Wang, Lingjie Liu, Yuan Liu, Christian Thcobalt, Taku Komura, and Wenping Wang. Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. NeurlPS, 2021.
[54] Peng Wang. Lingzhe Zhao, Ruijie Ma, and Peidong Liu. BAD-NeRF: Bundle Adjusted Deblur Neural
Radiance Fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition
(CVPR), pages 4170-4179, June 2023.
[55]
Zirui Wang. Shangzhe Wu, Weidi Xie, Min Chen, and Victor Adrian Prisacariu. NeRF--; Neural radiance
fields without known camera parameters, arXiv preprint 1eXiv:2102.07000 2021.
[56] Suttisak Wizadwongsa, Pakkapon Phongthawee, Jiraphon Yenphraphai, and Supasom Suwajanakom. Nex: Real-timo view synthesis with neural basis expansion. In IEEE Conference on Computer Vision and Pattern
Recognition (CVPR). 2021.
[57] Rundi Wu, Ben Mildenhall, Philipp Henzler, Keunhong Park, Ruiqi Gao, Daniel Watson, Pratul P. Srini- vasan, Dor Verbin. Jonathan T. Barron, Ben Poole, and Aleksander Holynski. Reconfusion: 3d reconstruc- tion with diffusion priors. arXivv 2023.
[58] Tong Wu, Jiaqi Wang. Xingang Pan, Xudong Xu, Christian Thcobalt, Ziwei Liu, and Dahua Lin. Voxurf: Voxel-based efficient and accurate neural surface reconstruction. In International Conference on Learning Representations (ICLR), 2023.
[59]
Tianyi Xie, Zeshun Zong, Yuxing Qiu, Xuan Li, Yutao Feng. Yin Yang. and Chenfanfu Jiang. Physgaussian: 3d gaussians for generative dynamics. arXiv preprint 2rXiv:2311.12190 2023. [60] Kun Xu, Wci-Lun Sun, Zhao Dong, Dan- Yong Zhao, Run-Dong Wu, and Shi-Min Hu. Anisotropic spherical gaussians. ACM Transactions on Gruphics, (2(6): 112:0099:220:111 1 2013.
[61] Jiawei Yang. Marco Pavone, and Yue Wang. Freenerf: Improving few-shot neural rendering with free frequency regularization. In Proc. IEEE Conference Computer Vision and Pottern Recognition (CVPR),
2023.
[62]
Ziyi Yang. Yanzhen Chen, Xinyu Gao, Yazhen Yuan, Yu Wu, Xiaowei Zhou, and Xiaogang Jin. Sire-ir: Inverse rendering for brdf reconstruction with shadow and illumination removal in high-illuminane scenes.
arXiv preprint arXiv:2310. 1030, 2023.
2023.
[63] Ziyi Yang, Xinyu Gao, Wen Zhou, Shaohui Jiao, Yuqing Zhang, and Xiaogang Jin. Deformable 3d gaussians for high-fidelity monocular dynamic scene reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20331-20341, 2024.
[64]
Zeyu Yang, Hongye Yang. Zijie Pan, Xiatian Zhu, and Li Zhang. Real-time photorealistic dynamic scene representation and rendering with 4d gaussian splatting, arXiv preprint arXiv 10642, 2023.
[65] Zongxin Ye, Wenyu Li, Sidun Liu, Peng Qiao, and Yong Dou. Absgs: Recovering fine details in 3d gaussian splatting. In ACM Multimedia 2024, 2024.
[66]
Taoran Yi, Jiemin Fang. Junjie Wang. Guanjun Wu, Lingxi Xie, Xiaopeng Zhang, Wenyu Liu, Qi Tian, and Xinggang Wang. Gaussiandreamer: Fast generation from text 3d gaussians by bridging 2d and 3d diffusion models. arXiv preprint arXiv:23 10.08529, 2023.
[67] Wang Yifan, Felcce Serena, Shihao Wu, Cengiz Oztireli, and Olga Horkine-Horuun.. Differentiahle surface splatting for point- based geometry processing, ACM Transactions on Graphics (TOG), 38(6):1-14, 2019. Yu, Ruilong Li, Matthew Tancik, Hao Li, Ren Ng. and Angjoo Kanazawa. Plenoctrees for real-time rendering of neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer
[68]
Vision, pages 5752-5761, 2021.
[69] Mulin Yu. Tao Lu, Linning Xu, Lihan Jiang, Yuanbo Xiangli, and Bo Dai. Gsdf: 3dgs meets sdf for
improved rendering and reconstruction. arXiv preprint arXiv:2403. 16964, 2024.
[70] Zehao Yu, Torsten Sattler, and Andreas Geiger. Gaussian opacity fields: Efficient and compact surface
reconstruction in unbounded scenes. arXiv preprint arXiv:2404. /0772, 2024.
[71] Qiang Zhang. Seung-Hwan Back, Szymon Rusinkiewicz. and Felix Heide. Differentiable point-based radiance fields for efficient view synthesis. arXiv preprint arXiv:2205. /4330, 2022.
[72] Xiuming Zhang, Pratul Srinivasan, Boyang Deng, Paul Debevec, William T Freeman, and Jonathan T Barron. Nerfactor: Neural factorization of shape and reflectance under an unknown illumination. ACM Transactions on Graphics (ToG), 40(6):1-18 2021.
[73] Shunyuan Zheng, Boyao Zhou, Ruizhi Shao, Boning Liu, Shengping Zhang, I Liqiang Nie, and Yebin Liu. Gps-gaussian Generalizable pixel-wise 3d gaussian splatting for real-time human novel view synthesis.
arXiv, 2023.
12

---

[74] Wojciech Zielonka, Timur Bagautdinov, Shunsuke Saito, Michael Zollhöfer, Justus Thies, and Javier Romero. Drivable 3d gaussian avatars. 2023.
[75] Matthias Zwicker, Hanspeter Pfister, Jeroen Van Baar, and Markus Gross. Ewa volume splatting. In Proceedings Visualization, 2001. VIS O1. pages 29-538. IEEE, 2001.
13

---

Appendix
This supplementary material provides more results that accompany the paper.
Section A provides more ablations.
Section B provides additional results, including more visualizations and quantitative results on complete datasets.
A More Ablations
In this section, we present the complete quantitative ablations on the key components our method.
We first evaluate the role of each component of the ASG appearance field in NeRF synthetic scenes shown in Tab. 5 The introduction of ASG improves the ability model specular highlights and reduces the number of 3D Gaussians. The inclusion of normals did not significantly increase computational overhead, but it did enhance rendering metrics and visual quality. More importantly, we achieve better rendering quality with fewer Gaussians than vanilla 3D-GS, characteristic that can be further explored in the future.
Next, we evaluated our full method on the Mip360 dataset in Tab. 6 is important to note that the Mip360 dataset is divided into indoor and outdoor scenes. Indoor scenes have more specular highlights, while outdoor scenes contain a large number of floaters. The coarse-to-fine approach itself improves the quality of 3D-GS in real-world scenes, mainly by eliminating significant amount of floaters in outdoor settings. Although the introduction of the ASG appearance field significantly increases rendering overhead, it did greatly enhance the modeling of specular highlights in indoor scenes. Under the constraints of the coarse-to-fine -ineine mechanism, our complete method combines the advantages of both, achieving the best rendering quality. To further improve rendering speed, we also implement light version and mini version based on Scaffold-GS. These versions offer trade-off between rendering quality and speed and can be used as needed. The quality of the Mip360 scenes demonstrates that our method is not only capable of handling scenes with specular highlights but is also robust real-world diffuse scenarios.
B More Comparisons
In this section, we present the complete quantitative results of our experiments. We report PSNR, SSIM, LPIPS (VGG), and color each cell as best, second best and third best
B.1 NeRF Synthetic Scenes
As shown in Tabs. 9 method demonstrates the best rendering quality metrics in almost every scene. It's important to note that the experimental setup for Tri-MipRF [18] differs from other methods. It uses both the training and validation sets as training data, expanding the scale of the model's 's data. When its training data limited to the training set, its metrics suffer a noticeable drop. Nevertheless, ensure that the experimental results fully reflect the highest performance of each method, and to prevent significant drops in metrics due to differences in experimental environments, we still present the metrics from the Tri-MipRR official paper. Our method achieved more prominent metrics in scenes with notable specular reflection and anisotropy, such as Drums, Lego, and Ship. This demonstrates that our method not only improves the overall rendering quality but also has more significant advantage in complex specular scenarios.
B.2 NSVF Synthetic Scenes
The NSVF 30 dataset, comparison to NeRF, features more noticeable metallic specular reflection, as presented in the Wineholder, Steamtrain, and Spaceship scenes. It is important to note that Tri- MipRF fails to converge in the Steam scene with the official code, so we did not report metries for that scenario. As shown in Tabs. 10 12 we present the per-scene experimental results of PSNR, SSIM, and I.PIS in the supplementary material. The experimental results indicate that compared to other methods based on 3D-GS (23), our method has significant advantages in metallic highlights
14

---

Table Ablation on ASG appearance field.
Table 6: Ablation on full method.
MipNekF 300
Mehhod
PSNK
SSIM
LPPPS
Nun.
wão ASG
3203
01366
10300
175
271
w/o decsap-MLPP
33.85
0.970
01030
217
244
who-22
wio nomal
34.10
0.911
0.29
130
236
anchor
0.824
Full (Ligho
34.08
0,970
01029
148
186
Full (Ligh)
28.07
0.854
0.183
44
2.52
Full
35.19
0.711
00025
121
237
Pull
28.18
0383
0L176
3.00
Mip-NeRI 360
3D-GS
Scaffold-SS Ours-w/o Norm
Ours
GT
Figure 8: Visualization on Mip-NeRF 360 outdoor scenes. Our method achieves robust floater removal by coarse to fine training.
and complex transmission scenarios. Additionally, compared it with the SOTA NeRF-based methods based on NeRF. Our approach enables 3D-GS to surpass the latest SOTA NeRF, achieving nigh-frequency highlight modeling that 3D-GS couldn't realize but NeRF could, thereby achieving truly high-quality rendering as shown in Fig. 14
3D-GS
Scaffold-GS
Ours
GT
Figure More comparisons with baselines. Our method achieves robust floater removal by coarse- to-fine training.
15

---

Spexial ligligts
Reflectie
GS-Shader
Ours
GT
Figure 10: Illustration specular high- lights and reflections.
Figure 11: Ccomparison Ref-NeRF dataset.
3D-GS
Scaffold-GS
Figure 12: Visualization on Nex [56 dataset.
Chair
Drums
Ficus
Hotdog
Lego
Materials
Mic
Ship
Avg.
INGP-Base
35.00
26.02
33.51
37.40
36.39
29.78
36.22
31.10
33.18
NeRF
33.09
Tri.MipRF
36.10
26.59
34.51
38.54
36.15
30.73
37.75
28.78
33.65
GS Shader
35.83
26.36
34.97
37.85
35.87
30.07
35.23
30.82
33.38
3D-GS
35.36
26.15
34.87
37.72
35.78
30.00
35.36
30.80
31.32
Scaffold-GS
35.28
26.44
35.21
37.73
35.69
30.65
37.25
31.17
33.68
Ours w/
Ours-light
35.69
26.77
36.03
38.25
36.1
30.84
36.95
31.97
34.08
Ours
35.72
26.92
36.10
38.25
36:46
30.98
37.09
31.97
34.19
Table 7: Per-scene PSNR comparison on the NeRF dataset.
Chair
Drums
Ficus
Holdog
Materials
Mic
Ship
Avg.
INGP-Base
0.979
0.937
0.981
0.982
0.982
0.931
0.990
0.896
0.963
Mip-NeRI
0.981
0.959
Tri Miiii
0.985
0,939
0.983
0.984
0.982
0.953
0.992
0.879
0.963
Shader
0.987
0.985
3D-GS
0.987
0.955
0.987
0.985
0.983
0.960
0.992
0.907
0.969
Scaffold GS
0.985
0,950
0.985
0.983
0.980
0.960
0.992
0.898
0.967
Ours light
0.987
0.955
0.988
0.985
0.981
0.963
0.993
0.905
0.970
Table 8: Per-scene SSIM comparison the NeRF dataset.
Chair
Drums
Ficus
Hotdog
Lego
Materials
Mic
Ship
Avg.
INGP-Base
0.022
0.071
0.023
0.027
0.017
0.000
0.010
0.132
0.045
Mip-NeRE
0.021
0.065
0.020
0.027
0.021
0.040
0.009
0.138
0.043
Tri-MipRF
0.016
0.020
0.052
0.008
0.136
0.042
GS-Shader
0.012
0,040
0.013
0.019
0.014
0.033
0.006
0.103
0.030
3D-GS
0.011
0.037
0.012
0.020
0.016
0.057
0.006
0.106
0.031
Scaffold GS
0.013
0.042
0.013
0,023
0,040
0.008
0.114
0,034
Ours-w/ w/ anchor
0.013
0.038
0.012
0.022
0.016
0.037
0.007
0.112
0.032
light
Oues
0.011
0.033
0.011
0.018
0.014
0.031
0.006
0.099
0.028
Table 9: Per-scene LPIPS (VGG) comparison on the NeRF dataset.
16

---

3D-G5
Ours
GT
Teapot
Plane
Record
Ashtray
Dishes
Headphone
Jupyter
Lock
Figure 13: Visualization on our "Anisotropic Synthetic" dataset. We show the comparison
between our method and 3D-GS across eight scenes. Qualitative experimental results demonstrate the significant advantage of our method in modeling anisotropic scenes, thereby enhancing the rendering quality of 3D-GS.
3D-GS
Scaffold-GS
GS-shader
Ours
GT
Figure 14: Visualization on NSVF dataset. Our method significantly improves the ability to model metallic materials compared to other GS-based methods. At the same time, our method also demonstrates the capability to model refractive parts, reflecting the powerful fitting ability of our method.
Bike
Life
Palace
Robot
Space
Toad
Wine
NeRF
31.77
31.08
11.76
28.69
34.66
30.84
29.42
28.23
30.81
37.75
34.60
34.05
35.24
39.00
33.25
32.04
35.13
TensoRF
39.23
34.51
37.56
38.26
38.60
37.87
34.85
31.32
36.52
Tri MipRF
36.98
33.98
36.55
33.49
37.60
33.48
29.97
34.58
NeuRBF
36.08
39.13
40.44
35.73
3D-GS
40.76
33.19
38.89
39.16
36.80
37.67
37.33
32.76
37.07
Scaffold-GS
39,87
35.00
38.53
37.92
34.36
37.12
36.29
32.32
36.43
OS Shader
37.38
27.36
36.55
37.00
32.61
35.27
34.50
30.16
33.85
Ours-w/ anchor
40.63
35.56
38.95
38.52
39.47
37.98
36.55
34.04
37.71
Ours-light
Ours
41.67
36.15
39.33
39.65
40.03
38.26
37.43
34.69
38.40
Table 10: Per-scene PSNR comparison on the NSVF dataset.
17

---

Bike
Life
Palace
Robot
Space
Steam
Toad
Wine
Avg.
NoRF
NSVF
0.911
0970
0.946
0.971
0.950
0.969
0.900
0.900
0.991
0,9800
0.966
0.986
0.968
0,920
0.965
0,920
0.952
0,979
TensoRF
0.993
0.968
0.979
0.989
0.991
0.978
0.91
0.982
Tri-MipRR
0.990
0.962
0,985
NeuRBF
0.995
0.977
0.985
0.995
0.993
0.993
0.985
0.972
0.986
3D-GS
0.994
0.979
0,983
0,994
0,991
0.995
0.985
0.975
Seaffold-GSS
0.993
0.979
0,981
0.995
0,985
0.992
0,982
0.971
0,984
GS Shader
0.992
0.964
0.979
0.994
0.985
0.990
0.980
0.966
0.981
0.979
0.994
Ours-light
0.995
0.982
0.984
0.993
0.994
0.994
0.984
0.977
0.988
Oues
0.995
0.982
0.984
0.995
0.994
0.994
0.985
0.978
0.988
Table 11: Per-scene SSIM comparison on the NSVF dataset.
Bike
Life
Palace
Robot
Space
Steam
Toad
Wine
Avg.
TensoRF
0.010
0.048
0.022
0.010
0.020
0.017
0.031
0.051
0.026
Tri-MipRF
0.012
0.048
0.023
0.119
0.019
0.036
0.055
0.030
NeuRBF
0.006
0.036
0.016
0.009
0.011
0.011
0.025
0.036
0.019
3D-GS
0.005
0.028
0.017
0,006
0.009
0.007
0.018
0.025
0.015
Scaffold-GS
0.007
0.030
0.019
0.008
0.019
0.010
0.022
0.021
0.017
GS GS-Shader
0.007
0.081
0.020
0.008
0.016
0.010
0.023
0.029
0.020
Ours-light
0.005
0.024
0.015
0.006
0.007
0.007
0.018
0.022
0.013
Ours
0.004
0.022
0.014
0.005
0.007
0.007
0.017
0.021
0.012
Table 12: Per-scene LPIPS (VGG) comparison on the NSVF dataset.
Teapot
Plane
Record
Ashtray
Dishes
Headphone
Jupyter
Lock
Avg.
3D-GS
27.24
26.80
43.81
34.43
29.62
38.72
40.52
29.36
33.81
Scaffold-GS
29.14
35.66
32.12
37.19
40.04
30.13
35.34
Ours-w/ anchor
33.53
31.56
50.35
36.14
32.95
38.48
40.10
30.96
36.76
Ours-light
31.01
50.30
37.76
33.03
39.39
41.42
31.68
37.42
Ours
35.24
30.95
50.90
38.03
33.04
40.12
41.47
31.86
37.70
Table 13: Per-scene PSNR comparison on our "Anisotropic Synthetic" dataset.
Teupot
Plane
Recurd
Ashiray
Dishes
Teaadphone
Jupyter
Lock
Avg.
0.346
0.947
0.989
0.985
ScufTold GS
0,979
0.965
0.998
0.973
0.967
0.986
0.983
0.924
0.972
Ours-w/ anchor
0.985
0.973
0.999
0.974
0.973
0.988
0.984
0.930
0.976
Ours-light
0.987
0.967
0.998
0.984
0.970
0.990
0.987
0.948
0.979
Ours
0.988
0.967
0.999
0.985
0.970
0.990
0.987
0.951
0.980
Table 14: Per-scene SSIM comparison on our " "Anisotropic Synthetic" dataset.
Teapot
Plane
Record
Ashtray
Dishes
Headphone
Jupyter
Lock
3D
0,043
0.085
0.019
0.044
0.120
0.015
0.075
0.098
0.062
0.086
0.099
Ours w/ anchue
0.022
0.042
0.004
0.039
0.067
0.017
0.084
0.093
0,046
Ours: light
0.021
0.052
0.007
0.022
0.079
0.014
0.076
0.080
0.044
Ours
0.051
0.005
0.020
0.077
0.013
0.071
0.075
0.042
Table 15: Per-scene LPIPS (VGG) comparison our "Anisotropic Synthetic" dataset.
bicycle
flowers
garden
Slump
trechull
100m
counter
kitchen
bonsai
Plenoxels
21.91
20.10
23.49
20.66
27.59
23.62
24.67
INGP
22.17
20.65
25.07
23.47
22.37
29.69
26.69
29.48
30.69
Mip-NeRF360
24.37
21.73
26.98
26.40
22.87
31.63
29.55
32.23
33.46
3D.GS
25,63
21.94
27.73
27.02
22.79
31.50
29.12
31.61
32.48
Scaffold-GS
25.61
21.74
27.82
26.79
23.38
32.14
29.62
31.81
32,87
Ours-w anchor
25.44
21.36
27.97
26.91
23.23
32.27
30.12
33.91
Ours-light
25.87
21.81
28.06
27.23
22.48
32.11
29.74
32.09
33.26
Ours
25.90
21.86
28.07
27.25
22.48
32.11
30.12
32.25
33.54
Table 16: Per-scene PSNR comparison on the Mip-NeRF 360 dataset.
18

---

bicycle
flwwrss
garden
stump
trechul
2oom
counter
kitchen
bonsai
Plenoxels
0.496
0,431
0.606
0.523
0.759
0.814
INGP
0.512
0.486
0.701
0.594
0.542
0.871
0.817
0.858
0.906
Mip-NoRF360
0.685
0.583
0.813
0.744
0.632
0.913
0.894
0.920
0.941
3D-GS
0.623
0.874
0.784
Scaffold-GS
0.773
0.649
0.867
0.774
0.657
0.931
0.919
0.931
0.950
Ours-w anchor
0.775
0.611
0.809
0.775
0.645
0.932
0.920
0.934
0.953
Ours-light
0.795
0.645
0.879
0.795
0.647
0.934
0.920
0.936
0.952
Ours
0.797
0.648
0.881
0.797
0.647
0.935
0.923
0.937
0.953
Table 17: SSIM Comparison on the Mip-NeRE 360 dataset.
bicycle
flowrss
garden
slump
Plenoxels
0.506
0.521
0.386
0.303
0.540
0.419
0.441
0,447
0.398
INGP
0.446
0,441
0.257
0.421
0.450
0.261
0.306
0.195
0.205
Mip:NeRF360
0.301
0.344
0.170
0.261
0.339
0.211
0.204
0.127
0.176
3D-GS
0.204
0,328
0.103
0.207
0.318
0.191
0.178
0.113
0.173
Scaffold-GS
Ours-w anchor
0.208
0.292
0.110
0.215
0.293
0.185
0.179
0.115
0.166
Ours-light
0.173
0,279
0.097
0.190
0.275
0.182
0.173
0.111
0.168
Ours
0.166
0.263
0.092
0.184
0.269
0.177
0.166
0.108
0.162
Table 18: LPIPS Comparison on the Mip-NeRF 360 dataset.
B.3 Anisotorpic Synthetic Scenes
"Anisotropic Synthetic" is synthetic dataset rendered ourselves, which includes 8 scenes with significant anisotropy. We tested some existing 3D-GS-based methods on "Anisotropic Synthetic." As shown in Tabs. [5 our method achieved very significant improvement rendering metrics. Fig. shows the comparison between our method and 3D-GS across all eight scenes. Qualitative experiments also demonstrate the significant visual advantages of our method, highlighting the substantial improvement our method brings to anisotropic parts, thereby enhancing the overall rendering quality.
B.4 Mip-360 Scenes
The MipNeRF-360 scenes include five outdoor and four indoor scenarios. There are several scenes rich in specular reflections, such bonsai, room, and kitchen. As shown in Tabs. [6 18 our method achieved significant advantages in the four indoor scenes. This reflects our method's strengths in modeling specular reflections and anisotropy. In outdoor scenes, method also achieved rendering metrics comparable to the SOTA methods. Furthermore, with the help of the coarse-to- coarse-to-fine training mechanism, our method significantly reduced the number of floaters as shown in Fig. II resulting in a substantial improvement in visual effects.
19

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: I am sure that the abstract and introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: I am sure that the paper discuss the limitations of the work.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
20

---

Justification:
Yes, the paper does.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Yes, the paper does.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
21

---

Answer: [No]
Justification:
The code can be released upon acceptance, but now it's not a clean version.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ ccpppp.ccppulli//uuuii wwwtc//ggue////eeeeceoocc CoodeSubeiiieeee ssi wswonnolli onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The paper contains details about the training model.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We evaluate the results through PSNR, SSIM and LPIPS.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
22

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: They are presented in the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: btttp:://neurrppe http://urrs.cccppuip/p/e httppp/ssccppp..t.ttttttttteeeea
Answer: [Yes]
Justification: Yes, do.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There is societal impact of the work performed.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
23

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Yes, they do.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, wwppppswpptseeee whworee.wthhhe....c.. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's is creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
24

---

Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/coddemoodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
25


Reviews:
[{"review_id": "Yh7goPpFfY", "reviewer": "Reviewer_anMP", "summary": "This paper presents an approach for reconstruction and view synthesis of scenes that exhibit strong specular/view dependent appearance. In particular, the authors extend the framework of Gaussian Splatting [Kerbl et al. 2023] and Scaffold-GS [Lu et al. 2023], replacing spherical harmonics for parameterizing view-dependent appearance with anisotropic spherical Gaussians [Xu et al. 2013], as well as leveraging a coarse-to-fine training strategy to generally improve performance. The authors carry out qualitative/quantitative evaluation on a variety of datasets, and report improved quantitative performance over nearly every other baseline.", "questions": "See above.", "limitations": "Limitations are discussed.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 1, "contribution": 2, "strengths": "The authors show good quantitative performance on a large number of datasets -- the quantitative evaluation in particular is quite comprehensive -- and some of the qualitative results presented in the video/paper are compelling. The design of the method seems sensible, and various components are ablated to show their importance.\n\nThe largest contribution of the work (and an important one, if accurate) is assembling a system for reconstruction and view synthesis that performs well for challenging view dependent scenes.", "weaknesses": "* I felt that the writing quality could be improved. In section 3, I couldn't tell which of the described components were part of the base method, and which were part of method variants (e.g. ours-light, ours w/ anchor). Method training and architecture details were somewhat sparse (only a few details were provided in section 4.1). I was also confused about how the word anisotropy is being used throughout the paper. Typically, it's used to mean view dependence that is not isotropic (e.g. rotationally invariant) -- but it does necessarily imply *high frequency* view dependence. Spherical harmonics are, in fact, anisotropic spherical functions. \n* I'm not quite sure how to assess the novelty of this paper, and feel that some claims (e.g. this is \"the first work to address the specular highlights modeling in 3D-GS\") are not quite fair (other works, such as GaussianShader [Jiang et al. 2024] at CVPR 2024 attempt to improve modeling of view dependent appearance in Gaussian splatting). \n* For many of the real scenes, it's hard to judge the qualitative improvement of this method over baselines (e.g. Figure 8). The video comparisons are nice, but are not provided for all baselines -- I feel that a webpage would've been more effective for showcasing comparisons/improvements.\n* As far as I can tell, this work does not make many changes on top of existing methods -- the two main changes being using ASGs to parameterize view-dependent appearance, and implementing a coarse-to-fine training strategy. Perhaps I'm missing something, but I'm not sure why these changes should lead to such a large improvement in view-dependent appearance modeling and quantitative performance. For example, are ASGs really responsible for modeling the reflection in the CD in Figure 9? Or is the ability to model this reflection due to Scaffold-GS's view-dependent decoding of Gaussian parameters? When the removal of floaters is shown in the video for the Bonsai scene, is this due to the coarse-to-fine training strategy, or something else (hard to say, because only the baseline 3DGS is shown in the comparison)? In general I don't feel that the relationship between quantitative/qualitative gains and method design are fully justified, but I acknowledge that I could be in the minority here.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are glad and appreciate that you recognizes that the results of Spec-Gaussian are comprehensive and compelling. Our response to your valuable comments is below:\n\n**Q1: What makes Spec-Gaussian work: Evaluation of the different components.**\n\nThe key components that make Spec-Gaussian work includes: the ASG appearance field and the coarse-to-fine training mechanism. These components are extensively studied both quantitatively (Tables 5-6) in supplementary file and qualitatively in Figures 6-8.\n\nFirst, the ASG appearance field works by using ASG and decoding MLP to augment Gaussians capabilities. (a) With the ASG appearance field, the performance improves by ~4dB in our anisotropic synthetic dataset. Our extensive ablation study of the ASG component also demonstrates that the improvements are not from the using of MLP like in Scaffold-GS (see Figure 6 and Tables 1-4), but more fundamentally from the introduction of ASG to effectively capture the high-frequency specular colors. (b) Moreover, our empirical experiments also show that using SH or MLP to fit the entire color spectrum independently is not ideal, as color contains both high and low-frequency signals, making it difficult to fit accurately. By filtering diffuse with SH and modeling the remaining specular components with ASG, each part can operate within its fitting capability, thereby improving the overall rendering quality.\n\nSecond, the coarse-to-fine strategy is effective in resolving the floaters in real-world scenes. The reason might be that by fitting the model on coarse images, the model is encouraged to capture the low-frequency geometry instead of the high-frequency noisy details, which reduces the chance for overfitting to noisy details that are harmful for generalization causing floaters. We also introduced L1-normed gradients in the accumulation process used for densification, making the GS densification more reasonable.\n\n**Q2: Real-world comparison with Scaffold-GS.**\n\nIn this rebuttal, we have submitted more comparisons incorporating scaffold-GS for comparisons and a zoomed-in version of Fig. 8 in the PDF.  We will incorporate more comparisons in the final version.\n\n**Q3: About the term `Anisotropic`.**\n\nIn this paper, the term `anisotropic` often appears alongside `specular`. Take Figure 1's CD as an example—the anisotropic part refers to the CD's surface, which shows different specular colors when viewed from different angles (resulting in the rainbow effect). While SH is indeed an anisotropic spherical function, using low-order SH (e.g., first 3 orders in 3D-GS) struggles to model complex shading effects. In Figure 6, even the first 6 orders of SH falls far short of properly modeling specular scenes (perhaps using more than 100 orders would make a difference, but the computational cost would be far greater than that of ASG).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8YB6PAmIgF", "reviewer": "Reviewer_Ye8M", "summary": "This paper proposes using Anisotropic Spherical Gaussians (ASGs) as view encoding to enhance the modeling of specular reflections in 3D Gaussian splatting. In addition to Spherical Harmonics (SH) encoded colors, the method additionally queries reflection direction with multiple ASGs to generate a view encoding. Since ASGs can potentially model higher frequency signals, the proposed method improves reconstruction quality in scenarios with strong view-dependence. The paper also introduces minor contributions, such as coarse-to-fine training and ASG compression, to further enhance quality and efficiency. The results demonstrate that the method surpasses all baselines.", "questions": "Please see the weakness.", "limitations": "The modeling of complex reflections (e.g., self-reflections, mirror-like reflections) could be further discussed.\nAlso it would be helpful to demonstrate some failure cases.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The idea presented is neat and simple, which is good. Although it is not too surprising that introducing higher frequency view encoding could enhance appearance modeling, it is noteworthy that no one else has systematically explored this idea. This could inspire the community. Thus, there are contributions, though not significant. I also appreciate the effort to improve rendering and memory efficiency after incorporating the ASG model.", "weaknesses": "• Contribution: I would not describe this work as \"the first to address specular highlights modeling in GS,\" as some earlier work has also attempted this. It would be better to tone down this claim.\n\n• Sum of L1 Norm of Gradients: One improvement in the paper is achieved by adding L1 norm to the gradient accumulation for GS densification, as illustrated in lines 202-214. I don't really follow the intuition behind this design. The original design for densification involves duplicating Gaussians and moving along the gradient direction to reduce image loss from all views. So when different views suggest moving the Gaussian in different directions, these directions cancel out and we don't do densification. This makes sense because it means that the particular gaussian is centered at an optimal location that won't sacrifice the quality of any view. However, I cannot find any physical meaning for using the sum of L1 norms. Although experiments show better performance, it would be helpful to provide an explanation, possibly with a toy example. Could this performance improvement be because the sum of L1 norms tends to be larger and thus more likely to exceed the threshold? Would reducing the threshold have a similar effect?\n\n• Lack ablation on the Number of ASGs: An important hyperparameter is the number of ASGs used. Theoretically, this is crucial for balancing efficiency and quality.\n\n• Discussion on Shape-Radiance Ambiguity: An important aspect of reflection modeling in radiance fields is dealing with shape-radiance ambiguity. I assume the coarse-to-fine training helps with this ambiguity, but it is not discussed in the methods or experiments.\n\n• Missing Comparison with Inverse Rendering GS: Although inverse rendering has a slightly different task than view-dependent appearance modeling, it is a valid approach to solving this problem. Therefore, it is necessary to compare this method with at least one GS inverse rendering method to demonstrate the benefits of using ASG over IR.\n\n• Rendering Time Breakdown: It would be helpful to show a breakdown of rendering time for each sub-step to identify the bottleneck preventing the model from achieving a similar FPS as the original GS.\n\n• Additional Visualizations: I would like to see visualizations of the learned ASGs and normal maps.\n\n• Performance on Ref-NeRF's In-the-Wild Data: I am curious to see how this method performs on Ref-NeRF's in-the-wild data, which contains more challenging mirror-like reflections. \n\n• Missing References: Some important citations are missing. Here are a few:\n\t\t○ Reflection Modeling in 3DGS:\n\t\t\t§ \"3D Gaussian Splatting with Deferred Reflection\"\n\t\t○ Reflection Modeling in Point Cloud:\n\t\t\t§ \"Neural Point Catacaustics for Novel-View Synthesis of Reflections\"\n\t\t○ Inverse Rendering in NeRF:\n\t\t\t§ \"PhySG: Inverse Rendering With Spherical Gaussians for Physics-Based Material Editing\"\n\t\t○ Reflection Modeling in NeRF:\n\t\t\t§ \"NeRF-Casting: Improved View-Dependent Appearance with Consistent Reflections\"\n\t\t\t§ \"SpecNeRF: Gaussian Directional Encoding for Specular Reflections\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the positive review as well as the insightful suggestions for improvement. Our response to the reviewer’s comments is below:\n\n**Q1: Ablation on the Number of ASGs.**\n\nThat's an excellent question. During the implementation of our code, we explored the number of ASGs. In the table below, we present the ablation results on the `teapot` scene. ASG=32 achieves the highest overall rendering metrics without causing a significant increase in training time or a decrease in FPS. Further increasing the numbers will not result in increase of rendering qualities but decreasing the rendering speed. \n\n|    ASG Num     |   PSNR    |    SSIM    |   LPIPS    | Training Time (min) | FPS     | Mem (MB) |\n| :------------: | :-------: | :--------: | :--------: | :-----------------: | ------- | -------- |\n|       8        |   31.93   |   0.9814   |   0.0281   |       **11**        | **185** | **37**   |\n|       16       |   34.43   |   0.9861   |   0.0224   |         12          | 147     | 39       |\n| **32 (paper)** | **35.24** | **0.9876** | **0.0206** |         13          | 153     | 38       |\n|       64       |   34.81   |   0.9872   |   0.0209   |         15          | 111     | 41       |\n\n**Q2: Comparison with inverse rendering GS.**\n\nAlthough theoretically, it is reasonable to incorporate inverse rendering to improve rendering quality, this suffers difficulties and may cause burdens in practice. This is because decoupling and learning the required information for physically based rendering (PBR) from multiview images is a highly ill-posed problem. The errors will, in turn, have negative impacts on rendering. The table below compares our method with Relightable-GS on the NeRF-Synthetic dataset, and Spec-Gaussian outperforms Relightable-GS by a large margin.\n\n|         Scene |   PSNR    |   SSIM    |   LPIPS   |\n| ------------: | :-------: | :-------: | :-------: |\n|         chair |   33.19   |  0.9798   |  0.0177   |\n|         drums |   25.41   |  0.9484   |  0.0444   |\n|         ficus |   32.64   |   0.979   |  0.0195   |\n|        hotdog |   35.56   |  0.9802   |  0.0291   |\n|          lego |   34.37   |  0.9776   |  0.0216   |\n|     materials |   28.50   |  0.9488   |  0.0474   |\n|           mic |   33.94   |  0.9863   |  0.0134   |\n|          ship |   29.63   |  0.8891   |  0.1226   |\n|       Average |   31.66   |  0.9612   |  0.0394   |\n| Spec-Gaussian | **34.19** | **0.971** | **0.028** |\n\n**Q3: Missing comparison and citation about works on reflection.**\n\nThanks for mentioning this. Although reflection and specular highlights (ours) focus on two different properties of objects and environments for causing view-dependent appearances as clarified previously, we will cite these works and discuss their differences in the related work. Although our work is not directly related to addressing reflection, we have still provided some experimental results for comparison. The qualitative experimental results can be seen in the submitted rebuttal PDF. The table below shows the comparison on real-world scenes from ref-nerf.\n\n|          | PSNR      | SSIM       | LPIPS      |\n| -------- | --------- | ---------- | ---------- |\n| garden   | 23.11     | 0.6174     | 0.1677     |\n| sedan    | 26.42     | 0.7317     | 0.1442     |\n| toy      | 24.93     | 0.6539     | 0.1454     |\n| Average  | **24.82** | **0.6677** | 0.1524     |\n| Ref-NeRF | 24.45     | 0.6650     | **0.1478** |\n\nWe also provided a comparison between Spec-Gaussian and GaussianShader on the Ref-NeRF Synthetic scenes (Shiny-Blender):\n\n| Scene         | PSNR      | SSIM       | LPIPS      | FPS     |\n| ------------- | --------- | ---------- | ---------- | ------- |\n| Spec-Gaussian | **31.00** | 0.9500     | **0.0752** | **145** |\n| GS-Shader     | 30.73     | **0.9540** | 0.0798     | 87      |\n\nWe would like to emphasize once again that specular highlights and reflections are two distinct shading effects, each requiring different technical approaches to address. Improvement in one shading effect does not necessarily translate to improvement in the other. Generally speaking, enhancing the modeling of specular highlights can also improve the modeling capability for general scenes. However, improving reflections mainly enhances the rendering quality of the reflective parts, and it may lead to negative optimization for the non-reflective parts, like NeRO and GS-DR.\n\n**Q4: Missing references.**\n\nThank you for pointing out these awesome works. We will add these citations to our paper.\n\n**Q5: About the insights of the L1 norm of gradients.**\n\nDuring the optimization process, 3D-GS accumulates the gradient of each pixel ( $\\frac{d L}{d \\mathbf{x}}=\\sum \\frac{d L}{d \\mathbf p_i} \\frac{d \\mathbf p_i}{d \\mathbf{x}}$) for every GS. When the accumulated value exceeds a threshold $\\tau_g$, the GS will densify. It is important to note that this value is not the gradient of each GS position but rather the accumulated gradient sum used for densification. Our insight is that gradients can be both positive and negative, and summing them for accumulation is not reasonable because large negative gradients can decrease the accumulated value, preventing GS that should densify from doing so. While negative gradients are meaningful for position optimization, they are clearly not reasonable for accumulation to determine whether densification should occur, as large negative gradients indicate that the GS requires more refined optimization. Therefore, we decided to apply the `L1 norm` **only** to the gradients used for accumulation to determine whether densification should occur ( $\\frac{d L}{d \\mathbf{x}}=\\sum \\Vert \\frac{d L}{d \\mathbf p_i} \\frac{d \\mathbf p_i}{d \\mathbf{x}} \\Vert_1$).\n\n**Q6: Time breakdown.**\n\nWe have provided the impact of different components on FPS in the paper. It can be found in Tabs. 5-6.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "v2d5cuLUsL", "reviewer": "Reviewer_8TqS", "summary": "Spherical harmonics-based 3D Gaussian splatting (3DGS) struggles with specular and anisotropic components. To address this problem, the paper proposes adopting anisotropic spherical Gaussians (ASG). However, directly adopting ASG does not demonstrate superior performance in representing specular and anisotropic parts. Therefore, the paper proposes separating diffuse and specular components from color representations and using a feature decoupling MLP to generate colors from ASG features. Through experiments, the paper demonstrates improved ability to represent highly specular parts.", "questions": "Line 144: Is $\\xi$ really $\\mathcal{R}^2$ or is this a typo of $\\mathcal{R}$?\n\n\n\nLines 185: Could you specifically state what is decoupled through $\\psi$.\n\nLine 191: There are no equations for pure ASG nor pure MLP. Could you state the equations or clarify how they work? For example, for pure MLP, is $\\kappa$ removed from Eq. 10 and only $\\Psi(\\gamma(d), \\langle n, -d\\rangle)$ used?\n\n\n\nLines 232-234: This seems contradictory to lines 178-179. Could you clarify which one was actually used during the experiments.\n\n\n\nLines 242-244 and Tabs. 3 and 4: Based on the explanation, Ours-light refers to 3DGS + ASG, while Ours-w/ anchor refers to Scaffold-GS + ASG. However, it is unclear what the performance version (Ours) refers to. As stated in lines 220 and 221, is the performance version the same as the light version (3DGS + ASG) with a lower threshold $\\tau_g$? If so, clarifying this in the experimental section would help readers understand what the performance version is.\n\n\n\nFig 6: Lines 181-183 and 277 state that directly using ASG leads to an inability to represent specular and anisotropic components. If “Ours w/o MLP” is the one, clarifying this in the caption could help with understanding. In addition, the meaning of “MLP” in “Ours w/o MLP” is ambiguous. It is unclear whether MLP in Fig. 6 refers only to $\\Psi$ (Eq. 10) or both $\\Psi$ and $\\phi$ (line 233).\n\n\n\n\n**Color separation**\n\nAs shown in Eq. 9, the proposed method separates diffuse and specular components.\nWhat happens without color separation? Additionally, what happens if the diffuse part is also represented through ASG? Based on the ASG paper, replacing SH with ASG can improve performance. Therefore, could you provide a reason why the diffuse part still uses SH?", "limitations": "The paper stated its limitation.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "**Novelty**\n\nThe idea of adopting ASG for color representation in the Gaussian Splatting framework is novel.\n\n\n\n**Performance**\n\nThe paper demonstrates significant performance improvements across different datasets.", "weaknesses": "**Related Works**\n\nThe related works section could better highlight the differences and advantages of this paper compared to other studies. The paper does not mention other works on specular scenes and objects. For instance, GaussianShader is another 3D Gaussian splatting-based method for specular scenes and objects, yet it is not mentioned in the related works section, even though it is referenced elsewhere in the paper. The related works section should compare this method with prior works, such as SpecNeRF (CVPR 2024), and highlight their limitations and how the proposed method overcomes them, or emphasize the novelty of this paper.\n\n\n\n\n\n**Mathematical Notations**\n\nThe mathematical notations could be improved.\nThe inner product is represented both by $\\cdot$ (Line 145, Eq. 11) and $\\langle \\rangle$ (Eq. 10).\nAdditionally, $\\cdot$ denotes element-wise multiplication (Eqs. 5 and 6), inner product (Line 145, Eq. 11—inside the parenthesis), scalar-vector multiplication (Eq. 11—outside the parenthesis), and scalar-scalar multiplication (Eqs. 4 and 12).\nAt least element-wise multiplication and inner product should use different notations.\nAnd using the same notation for different operations within a single equation (Eq. 11) should also be avoided.\n\n\n\n\n\n**Coarse-to-fine Training**\n\nCoarse-to-fine training is not a novel approach within the 3DGS framework.\nFor example, “EAGLES: Efficient Accelerated 3D Gaussians with Lightweight EncodingS” already proposed a similar coarse-to-fine training approach.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the positive and detailed review as well as the suggestions for improvement. We will revise the mathematical notations in the paper based on these insightful suggestions. Our response to the reviewer’s comments is below:\n\n**Q1: Color Separation.**\n\nGreat question. It's worth noting that in Fig. 6, we've already explored the model without separation. This is because Scaffold-GS inherently uses an MLP to represent color without distinguishing between diffuse and specular components. We conducted further experiments on the teapot scene in Fig. 6, as shown in the table below:\n\n| Method | 3D-GS (SH color) | 3D-GS (MLP color) | Scaffold-GS (MLP color) | Ours (SH diffuse + ASG specular) | Ours (ASG diffuse + ASG specular) | Ours-w/ anchor (SH diffuse + ASG specular) |\n| ------ | ---------------- | ----------------- | ----------------------- | -------------------------------- | --------------------------------- | ------------------------------------------ |\n| PSNR   | 27.04            | 11.32 (Failed)    | 32.46                   | 37.50                            | 33.93                             | 35.69                                      |\n\nIt can be observed that color separation significantly improves the rendering quality of scenes with specular highlights, while using ASG to model diffuse color results in a decline in rendering quality. This experimental result reveals that clean separation of diffuse and specular components can aid in the learning of each part, thereby enhancing the overall color quality. Compared to ASG, SH serves as a better **low-frequency filter**. The inherent limitations in SH's fitting capability actually contribute to a cleaner diffuse component, allowing ASG to focus more effectively on learning the specular part.\n\n**Q2: Related work.**\n\nWe will add and discuss them in the related work. GaussianShader cannot effectively address scenes with specular highlights; it primarily attempts to enhance the capability of 3D-GS in modeling reflective scenes. \n\nWe are sorry for missing Spec-NeRF. Although we both aim to address specular highlights, the approaches to modeling specular highlights differ from each other: Spec-NeRF uses Gaussian directional encoding, while we employ anisotropic spherical Gaussian (ASG).\n\n**Q3: Coarse-to-fine training.**\n\nThank you very much for the reminder. We have taken note of the outstanding work in EAGLES, and we will cite this paper in Spec-Gaussian. It is important to note that, unlike EAGLES, our coarse-to-fine approach includes two components: 1) L1-normed gradients for GS densification, and 2) progressively training from low to high resolution. This design aims to prevent GS from becoming overly densified in the early stages of training (due to the L1 norm), which significantly reduces the number of GS.\n\n**Q4: Explanation of other questions.**\n\n> Line 144: Is $\\xi$ really $\\mathbb R^2$ or is this a typo of $\\mathbb R$?\n\n- $\\xi$ is $\\mathbb R^2$, which means it is a 2-dimensional real vector.\n\n> Lines 185: Could you specifically state what is decoupled through $\\Psi$.\n\n- Thank you for raising this question. We now believe that `decode` better conveys the meaning of $\\Psi$ compared to `decouple`. Specifically, it involves decoding the ASG-encoded features to obtain the specular color.\n\n> Line 191: There are no equations for pure ASG nor pure MLP. Could you state the equations or clarify how they work? \n\n- The ablation of pure ASG and pure MLP aims to demonstrate that both ASG and decode MLP are crucial. In the case of pure ASG, we directly employ ASG to obtain the color through $c_s = \\bigoplus_{i=1}^{N} ASG(\\omega_r \\: | \\: [\\mathbf{x}, \\mathbf{y}, \\mathbf{z}],  [\\lambda_i, \\mu_i], \\xi_i), \\text{where}\\  \\xi_i \\in \\mathbb R^3$. While for the pure MLP, we need to input features that have not been encoded by ASG to ensure a fair comparison. Therefore, the formula is shown below:  $\\Psi (\\kappa, \\gamma(\\mathbf{d}), \\langle n, -\\mathbf{d} \\rangle) \\rightarrow c_s, \\kappa = \\bigoplus_{i=1}^{N} [\\lambda_i, \\mu_i, \\xi_i].$\n\n> Lines 232-234: This seems contradictory to lines 178-179. Could you clarify which one was actually used during the experiments.\n\n- As previously mentioned, our method has three versions. `Ours` and `Ours-light` are based on 3D-GS, while `Ours-w/ anchor` is based on Scaffold-GS. For the versions based on 3D-GS, we used the approach described in Lines 178-179, and for the version based on Scaffold-GS, we adopted the color model described in Lines 232-234.\n\n> Lines 242-244 and Tabs. 3 and 4: Based on the explanation, Ours-light refers to 3DGS + ASG, while Ours-w/ anchor refers to Scaffold-GS + ASG. However, it is unclear what the performance version (Ours) refers to. \n\n- Thank you for pointing out this issue. Clarifying this in the experimental section is very important. We will include the previous explanation in the paper.\n\n> Fig 6: Lines 181-183 and 277 state that directly using ASG leads to an inability to represent specular and anisotropic components. The meaning of “MLP” in “Ours w/o MLP” is ambiguous.\n\n- As mentioned in Lines 181-183, directly using ASG can result in failure to model specular color. Therefore, in the ablation study of Fig. 6, we included `Ours-w/o MLP` to support this statement. Here, MLP refers only to the decoding MLP $\\Psi$ in Eq. (10). This is an excellent suggestion, and we will include a more detailed explanation of this in the paper.\n\nFinally, we would like to thank the reviewer once again. Many of these points were things we had not noticed before, and these suggestions will significantly improve the readability of the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dEZtIYB9Fv", "reviewer": "Reviewer_2PwK", "summary": "This paper introduces an anisotropic spherical Gaussian (ASG) appearance field into 3D Gaussian splatting for modeling the view-dependent appearance of each 3D Gaussian, which increases the ability of 3D Gaussian in representing high-frequency information. The key idea of this paper is combining ASG and SH to model the color of each 3D Gaussian. The experiments show the effectiveness of the method in modeling specular highlights and rendering quality.", "questions": "1. It would be better if the authors could include the training time of their model when comparing it with other methods.\n2. If we use RGB color as the diffuse color directly, instead of using SH to model, whether there will be a better or a worse result, since the high order SH is used in 3D-GS to represent high-frequency information.", "limitations": "Yes. The authors discussed the limitations of their work.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. An ASG appearance field is introduced to increase the ability of 3D Gaussian in representing high-frequency information.\n2. The coarse-to-fine training scheme can effectively eliminate floaters, which is verified by the corresponding ablation studies.\n3. Extensive comparison and ablation studies have been done to show the effectiveness of each component in this method.", "weaknesses": "1. The method of this paper includes several MLPs, however, the authors didn't report the training time of their model when compared with other methods. I think the training time should be considered for a more fair comparison. If I missed this, please correct me. \n2. The structure of 3D Gaussian in this paper is mainly based on Scaffold-GS, anchor-based Gaussian splatting. Therefore, the performance of the method in this paper may be degraded when the vanilla Scaffold-GS cannot perform well, such as the scene dominated by large texture-less regions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for the positive feedback and constructive suggestions. Our response to the your  concerns are incorporated below:\n\n**Q1: Training time of our method.**\n\nThis is a great question regarding the scalability of our Spec-Gaussian model. The training time of Spec-Gaussian does not significantly increase compared to the baselines. We present the comparison on NeRF-Synthetic dataset in the table below:\n\n| Method              | 3D-GS | Scaffold-GS | Ours | Ours-light | Ours-w/ anchor |\n| ------------------- | ----- | ----------- | ---- | ---------- | -------------- |\n| Training Time (min) | 8     | 9           | 15   | 14         | 11             |\n\n**Q2: Spec-Gaussian may not perform well in texture-less scenes.**\n\nIt is worth noting that our approach is generic to be incorporated into 3DGS and Scaffold-GS: both the `Ours` and `Ours-light` versions are based on 3D-GS; and `Ours-w/ anchor` is based on Scaffold-GS. The anchor-based design is only used for efficiency improvements to balance the additional overhead introduced by ASG. As shown in Tab. 4, `Ours-w/ anchor` still outperforms Scaffold-GS by a large margin, demonstrating its superior modeling ability over Scaffold-GS.  Also, we didn’t observe significant degradation in texture-less areas by using anchors in NSVF dataset.\n\n**Q3: Use RGB color as the diffuse color.**\n\nGreat thought. We also experimented with using RGB color as the diffuse color to reduce the computational and storage overhead introduced by spherical harmonics (SH). Experiments on the NeRF-Synthetic dataset showed that using RGB color slightly decreases the rendering metrics. Considering that Spec-Gaussian aims to explore the upper limits of 3D-GS rendering quality, we chose to filter the diffuse color with third-order SH. We will include a discussion on diffuse color modeling in the paper.\n\n| Scene    | PSNR      | SSIM      | LPIPS     | Time (3090) | FPS     | Mem    |\n| -------- | --------- | --------- | --------- | ----------- | ------- | ------ |\n| chair    | 35.75     | 0.9869    | 0.0111    | 24          | 81      | 64     |\n| drums    | 26.92     | 0.9552    | 0.034     | 17          | 113     | 46     |\n| ficus    | 35.75     | 0.987     | 0.0118    | 13          | 208     | 25     |\n| hotdog   | 38.25     | 0.9859    | 0.0184    | 14          | 156     | 31     |\n| Lego     | 36.48     | 0.9828    | 0.0156    | 14          | 137     | 38     |\n| material | 30.77     | 0.9626    | 0.0353    | 11          | 201     | 22     |\n| mic      | 36.95     | 0.9929    | 0.0059    | 13          | 153     | 26     |\n| ship     | 31.63     | 0.9037    | 0.1027    | 21          | 99      | 61     |\n| Average  | 34.07     | 0.9696    | 0.0294    | 15.87       | **144** | **39** |\n| Paper    | **34.19** | **0.971** | **0.028** | **15.42**   | 121     | 72     |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all the reviewers for their valuable comments. We are glad and appreciate that the reviewers recognize that our proposed ASG appearance field and coarse-to-fine training are sound, efficient, and show significant performance improvements. We will polish our paper further and release our codes.



We would first like to clarify the contributions of Spec-Gaussian and the different variants presented in our paper. Following that, we will address the specific questions posed by each reviewer.



**Contribution and differences with GaussianShader:**

Reviewers `anMP` and `Ye8M` have raised concerns regarding our claim that “this work is the first to address specular highlights modeling in GS,” suggesting that prior works like GaussianShader have also attempted to tackle this issue. We respectfully disagree with this. Although both GaussianShader and our approach aim to handle view-dependent appearances, we fundamentally differ in how we model the underlying factors that cause these appearances:

- GaussianShader, GS-DR, and NeRF-casting mainly focus on reflective scenes, referring to the phenomenon where glossy objects reflect their **surrounding objects in the environment**. These methods primarily incorporate the objects’ geometry and environment map with a rendering equation to model view-dependent appearances caused by reflective surfaces.
- In contrast, our approach and Spec-NeRF focus on scenes with specular highlights, which are the bright spots of light that appear on shiny surfaces when viewed from a specific direction. These highlights result from the interaction between the **intensity of light sources and the material properties** of the object and are independent of other objects in the environment. Our method effectively captures specular highlights, as demonstrated in our paper (see Fig. 1 and Fig. 4), while previous methods based on Gaussian splatting have indeed been unable to model sharp specular highlights.

In sum, the efforts made by GaussianShader and our method are complementary and could be combined in the future to model view-dependent appearances in complex scenes and objects. This can be an area for future research. We will emphasize these differences in our paper to avoid any confusion. If the reviewers still find our claim inappropriate, we are very open to toning down this statement. Reviewers can see the illustration of specular highlights and reflection in the submitted rebuttal PDF.



**Explanation of different variants:**

- `Ours`, a method based on 3D-GS, referred to as the performance version, has $\tau_g=0.0005$. 
- `Ours-light`, also based on 3D-GS, is called the light-version, with $\tau_g=0.0006$. 
- `Ours-w/ anchor`, based on the Scaffold-GS, is referred to as the mini-version, with $\tau_g=0.0006$.

Meta-review: This paper received positive scores of weak accept, weak accept, borderline accept and borderline accept. The AC concurs with the reviewers and recommends acceptance. The authors have promised changes in their rebuttal:

- Slightly tone down their claims wrt being the "first to address specular highlights modeling in GS."
- Incorporate suggestions from reviewer 8TqS regarding section 3, as well as address additional questions from 8TqS re: problems with the text
- Incorporate additional experiments provided in the rebuttal

The AC urges the authors to uphold their promises.
Decision: Accept (poster)