Reflective Gaussian Splatting
Authors: Yuxuan Yao
Zixuan Zeng
Chun Gu
Xiatian Zhu
Li Zhang
Venue: ICLR 2025 Poster (2025)
Keywords: Gaussain-Splatting
Physically based Rendering
Deferred-Rendering
Inter-Reflection
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Novel view synthesis has experienced significant advancements owing to increasingly capable NeRF- and 3DGS-based methods. However, reflective object reconstruction remains challenging, lacking a proper solution to achieve real-time, high-quality rendering while accommodating inter-reflection. To fill this gap, we introduce a Reflective Gaussian splatting (Ref-Gaussian) framework characterized with two components: (I) Physically based deferred rendering that empowers the rendering equation with pixel-level material properties via formulating split-sum approximation; (II) Gaussian-grounded inter-reflection that realizes the desired inter-reflection function within a Gaussian splatting paradigm for the first time. To enhance geometry modeling, we further introduce material-aware normal propagation and an initial per-Gaussian shading stage, along with 2D Gaussian primitives. Extensive experiments on standard datasets demonstrate that Ref-Gaussian surpasses existing approaches in terms of quantitative metrics, visual quality, and compute efficiency. Further, we show that our method serves as a unified solution for both reflective and non-reflective scenes, going beyond the previous alternatives focusing on only reflective scenes. Also, we illustrate that Ref-Gaussian supports more applications such as relighting and editing.

Published as a conference paper at ICLR 2025
REFLECTIVE GAUSSIAN SPLATTING
Yuxuan Yao',, Zixuan Zeng'', Chun Gu', Xiatian Zhu Li Zhang fee "chooo of Data Science, Fudan University Uniiversity of Surrey
https: / /ffuuan----t zaggannn-vv.... git.gg.ttub........ io/ ref gaussian
Glossy-Synthetic dataset
JDCS
J0GS-DI
GRase
Reigdeon
Ours
K300S
GShaddr
Circle
(Sad)Boo)
Editis
3005:
Rat-butty
Oun
G
-1
Ret NeRF
24
25
26
27
28
29
30
PSNR (dB)
Figure 1: Our Ref-Gaussian achieves superior rendering quality in novel view synthesis while en- joying fast optimization (training time) and real-time rendering (FPS), and supporting various down- stream applications such as relighting and editing. Please refer to the video in our supplementary material for more comprehensive and intuitive comparison.
ABSTRACT
Novel view synthesis has experienced significant advancements owing to increas- ingly capable NeRF- and 3DGS-based methods. However, reflective object re- construction remains challenging, lacking a proper solution to achieve real-time, high-quality rendering while accommodating inter-reflection. To fill this gap, we introduce a Reflective Gaussian splatting (Ref-Gaussian) framework character- ized with two components: (I) Physically based deferred rendering that empowers the rendering equation with pixel-level material properties via formulating split- sum approximation; (II) Gaussian- cruussian-rrounded inter-reflection that realizes the de- sired inter-reflcction function within Gaussian splatting paradigm for the first time. To enhance geometry modeling. we further introduce material-aware nor- mal propagation and an initial Ber-Gaussian shading stage, along with 2D Gaus- sian primitives. Extensive experiments on standard datasets demonstrate that Ref- Gaussian surpasses existing approaches in terms of quantitative metrics, visual quality, and compute efficiency. Further, we show that our method serves as a unified solution for both reflective and non-reflective scenes, going beyond the previous alternatives focusing on only reflective scenes. Also, we illustrate that Ref-Gaussian supports more applications such as relighting and editing.
INTRODUCTION
Tremendous progress has been recently made in 3D object reconstruction and novel view synthe- sis from multiview images. Neural radiance field (NeRF) (Mildenhall et al., 2021) represents 3D scenes with neural implicit fields, achieving high-quality, photo-realistic renderings through volume rendering. However, NeRF is expensive computationally for optimization and rendering, limiting
Co-l authorship
"Li Zhang (lizhangfd@@ fudan. edu.en) is the corresponding author.

---

Published as a conference paper at ICLR 2025
its use in real-time applications. To address this, 3D Gaussian splatting (3DGS) Kerbl et al. (2023)
instead represents a scene using a set of 3D Gaussians. Combining rasterization and alpha-blending
it achieves both real-time and high-quality rendering.
However, most NeRF and 3DGS struggle to model reffective surfaces due to the intrinsic inability of capturing high-frequency specular components. To mítigate this, Ref-NeRF (Verbin et al., 2022) leverages integrated directional encoding to smooth interpolation of reflected radiance. But this method fails to decompose the environment illumination, making it less useful for example unable to relight. A couple of NeRF methods (Verbin et al., 2022; Liang et al., 2023) are challenged by computational cost and limited flexibility. Using a simplified shading functions on each Gaussian can achieve rapid convergence, but suffering substantial noise in modeling of geometry, material, and lighting (Jiang et al., 2024). To avoid this issue, performing the shading function at the pixel level (i.e., deferred shading) following alpha-blending 2024). Nonetheless, limiting to a simplified shading function makes it ineffective to model complex reflective objects with complex structures and surfaces and the inter-reflection phenomenon. To address the latter. RelightableGausiin inferring at the cost of heavy noises and added computational overhead from Monte Carlo sampling.
smooth out the gradients is helpful (Ye al.,
(Gao et al., 2023) traces rays across Gaussians for visibility
To address the aforemenniined issues all together, we propose Reflective Gaussian splatting (Ref- Gaussian) framework, allowing for real-time high-quality rendering of reflective objects while tak- ing into account the complicated inter-reflection effect. This is made possible by two key compo- nents: (I) Physically based deferred rendering, where we empower the rendering equation with pixel-level material properties (e.g.. Bidirectional reflectance distribution function, BRDF in short), and adopt the split-sum approximation to avoid the heavy compute with Monte Carlo sam- pling. (II) Gaussian -rouinddd inter-reflection, where we realize the intricate inter-reflection fune- tion with Gaussian splatting for the very first time, despite the challenge of extracting the mesh accurately. To enhance the modeling of geometry, we choose 2D Gaussian primitives as scene representation, while introducing an initial per-Gaussian shading stage (performing shading at the Gaussian level to provide good geometric initialization), as well as a material-awarr normal propa- gation process (enlarging the scales of Gaussians with high metallic and low roughness).
We make these contributions: (I) We strive to realize real-time high-quality rendering of reflective objects with inter-reflection in 3D space. (II) We innovate generic 3D reconstruction framework, Reflective Gaussian splatting (Ref-Gaussian), characterized with physically based deferred render- ing and Caussian-grouus optimization including adopting 2D Gaussian primitives, material- normal propagation, and per- Gaussian shading initialization. (III) Extensive experiments demonstrate that Ref-Gaussian out- performs previous methods in terms of quantitative metrics, visual quality, and compute efficiency under both reflective and non-reflective scenes, whilst supporting downstream applications such as relighting and editing.
refr-reflection. This approach is further enhanced by geometry- -focusd
RELATED WORK
Novel view synthesis aims to generate new images of a scene from unseen viewpoints with a limited set of observed images. Neural radiance fields (NeRF) (Mildenhall et al., 2020) employ vol- umetric rendering with implicit neural representations to produce highly detailed novel views from input images. Subsequent works have focused on two main directions: improving rendering qual- ity (Barron et al., 2021; 2022; 2023) and accelerating both training and rendering speeds (Sun et al., 2022; Fridovich-Keil et al., 2022; Müller et al., 2022; Chen et al., 2022). More recently, 3D Gaus- sian splatting (3DGS) (Kerbl et al., 2023) proposed using 3D Gaussian primitives and tile-based differentiable rasterizer, achieving superior rendering quality and efficiency over NeRF-based meth- ods. Building on 3DGS, methods like 2DGS (Huang et al., 2024) and GOF (Yu et al., 2024) have further explored surface reconstruction with Gaussian splatting. In this paper, we adopt 2D Gaussian primitives, as proposed in 2DGS, to represent scenes for more accurate geometry reconstruction.
Inverse rendering seeks to decompose scenes into geometry, lighting, and materials, posing a significant challenge due to the high-dimensional complexity of light interaction. NeRF (Milden- hall et al., 2020) is commonly used as a 3D representation in inverse rendering due to its flexible raymarching technique. Several Nesed methods (Srinivasan et al., 2021; Boss et al., 202 Zhang et al., 2021; Boss et al., 2021b; Yao et al., 2022; Boss et al., 2022; Zhang et al., 2023; Liu et al., 2023; Jin et al., 2023) model light interaction using the explicit rendering equation with

---

Published as a conference paper at ICLR 2025
BRDF. For instance, NeRO (Liu et al., 2023) divides the training process into two stages: the first stage employs split-sum approximation to obtain detailed geometry, while the second stage focuses on accurate sampling to recover lighting and materials. In contrast, Ref-NeRF (Verbin et al., 2022) does not decompose material and environment lighting but instead shades points using a simplified function, limiting its applicability in downstream tasks such relighting and editing.
With the advent of 3D Gaussian splatting (Kerbl et al., 2023), recent methods (Jiang et al., 2024; Gao et al., 2023; Liang et al., 2024; Shi et al., 2023) have begun to address inverse rendering tasks using Gaussian primitives for much faster rendering speed than NeRF methods. GShader (Jiang et al., 2024) applies a simplified shading functions from rendering equation on each Gaussian. While achieves rapid convergence, this approach also brings substantial noise in geometry, material, and lighting. In contrast, 3DGS-DD (Ye et al., 2024) employs deferred shading by applying a sim- plified shading function at the pixel level, stabilizing the optimization process by smoothing out normal gradients. As both GShader and 3DGS-DR cannot model the inter-reflection effects, Re- lightableGaussian (Gao et al., 2023) introduces an innovative Gaussian- based ray tracing technique for visibility precomputation. However, it applies Monte Carlo sampling to evaluate the rendering equation, which slows down the rendering speed, and the inaccurate visibility also reduces the qual- ity of reconstruction. In this paper, inspired by 3DGS-DR, we adopt pixel-level deferred shading but perform the rendering function with BRDF properties to handle complex scenes. Furthermore, we propose performing ray tracing on the extracted mesh for visibility computation in the specular component of our rendering equation. Together with the split-sum approximation of the rendering equation, we model inter-reflection effects while maintaining fast rendering speeds.
METHOD
Our Reflective Gaussian splatting (Ref-Gaussin is formulated in the Gaussian splatting frame- work. The key components of Ref-Gaussian include (1) physically based deferred rendering and (2)
Gaussian-focused inter-reflection. An overview of our pipeline is shown in Figure 2. We start with the preliminary of Gaussian splatting.
In standard 3DGS (Kerbl et al., 2023), each Gaussian is modeled as an ellipsoid, characterized by 3D position pa and a covariance matrix 2, defined as:
((p) = exp - (PpP-tt"E-11 pb)
(1)
During the rendering process, 3D Gaussians are projected into 2D Gaussians in camera space. The transformed 2D covariance matrix is approximated as: >'' = JW JWEWTJ where W is the view transformation matrix and J is the Jacobian of the perspective projection. Each Gaussian is also assigned an opacity and a vieendepe color c; represented by spherical harmonics. The final pixel color is computed by blending Gaussians front-to-back order based on their depth:
C= Σ cTa, where T, T=
II aa), j=1
where o; is calculated by multiplying the opacity o, with the Gaussian weight, which is determined using the 2D covariance matrix Σ.
2D Gaussian primitive However, 3D Gaussians struggle to capture detailed geometry due to in- herent multi-view inconsistencies. We hence leverage 2D Gaussian splatting (Huang et al., 2024) where view-consistent 2D oriented planar Gaussian disks are used as rendering primitives.
Each 2D Gaussian is defined by its position p. tangential vectors tu and t,, and scaling factors (Su: sy) that control its shape and size. Its influence is given by:
g(u, u) = cxp (- u² + 12
(3)
2
where (u, v) are coordinates the local tangent space. The ray-splat intersection (u, v) in this space is obtained via:
x = (az, yz, 1)T = W H(, v, 1,1),, where H =
sttt s,t,
0 pl
E
4x44
(4)

---

Published as a conference paper at ICLR 2025
Inter-reflection
A
TSDF
Rataded
Surrice
Environment map
Diffuse
Gaussions
Properties
dircction
Query
L
A
Mesh
tracing
Visibility
Physkcllly based defanrd mandeiing
n Ind
Feature maps
Split-sum
Rendering
Specular
Splatting
May
(ea, Ni
Metallic
Roughness
Figure 2: Overview of the Ref-Gaussian framework: First, we apply the splatting process to produce feature maps and perform ray-tracing on the extracted mesh to compute visibility for the specular term in the rendering equation. Next, the pixel-level feature maps apply the rendering equation with split-sum approximation, yielding the final physically based rendering result.
Here, represents the homogeneous ray passing through pixel (z, y) and intersecting the 2D Gaus- sian disk at depth z. See more details of 2DGS in (Huang al., 2024).
3.1 PHYSICALLY BASED DEFERRED RENDERING
Our physically based deferred rendering employs a simplified version of the Disney BRDF model (Burley and Studios, 2012). Specifically, each Gaussian is associated with set of material- related properties, including albedo À E [0, 1j metallic m E [0, 1]. and roughness r E [0, 1]. The normal vector E [0, 1]s can be derived from the tangential vectors of each 2D Gaussian using tu x tv. We deploy the rendering equation on pixel-level feature maps obtained via dipha-blending:
A
A₂
N X
T.a. II a1,,,, where J=1
X
M
m
(5)
R
Ti
N
n2
Note, unlike shading directly on the Gaussians before alpha-blending (Jiang et al., 2024), our de- ferred shading treats alpha-blending smoothing filter. That allows to stabilize the optimization of features and finally produce more cohesive rendering results.
With the aggregated material maps, including albedo A, metallic M, roughness and normal N, the rendering equation is subsequently applied. The rendering equation expresses the outgoing radiance L(w,) the direction Wa as:
L(wo)= = / [,(w;)) (w) N) day,
(6)
where [wi, w.) denotes the bidirectional reflectance distribution function (BRDF). It represents the integral reflected incident light over the hemisphere.
Since BRDF consists of both diffuse and specular terms, the integral also be divided into two parts. The diffuse term is computed by querying the pre-integrated environment map using the nor- mal N and multiplying with the material terms, while the specular term requires more complex reflection calculations to account for surface reflections. Here, let us focus on the specular compo- nent, where the specular term of the BRDF is given by:
DGF
f.(u, = A(w, () N)
where D, F. and G represent the GGX normal distribution function, the Fresnel term, and the saudowing-maskingg term, respectively.
To compute the above integral term, typical approach is Monte Carlo sampling as in (Gao et al., 2023). However, that is computationally expensive to render in real time. Instead, we adopt the split-sum approximation (Munkberg et al., 2022):

---

Published as a conference paper at ICLR 2025
La(wa) /, Is (wi, wa)s N)dw; J. Li(wi) Wo) N)dws,
(8)
where the first term depends solely (wy N) and roughness R. allowing the results to be precom- puted and stored in a 2D lookup texture map. The second term represents the integral of incident radiance over the specular lobe, and can be pre-integrated the beginning of each training iteration with different roughness levels. That allows to efficiently employ series of cubemaps to represent the environment lighting by performing trilinear interpolation using the reflected direction and the roughness as parameters.
3.2 GAUSSIAN GROUNDED INTER REFLECTION
Inter-reflection plays a critical role in rendering reflective objects. To that end, further enhance the specular light component by separately modeling the direct light Lair (second term in Eq. 8) and indirect light Lind- Direct light Ldir denotes that whose reflection not blocked by any scene elements, and the rest is defined as indirect light Lind-
Specifically, we approximate the visibility V E 10.1) of incident light based on whether it is self- occluded along the reflected direction R 2(w,, -N)N w We represent the indirect light from occluded part as Lind:
['(wo)) se S// (slws, N)dws) [Lair V + Lina (1-V))
Intuitively, this indirect light component Lind introduced here aims effectively model the pertur- bation caused by occlusion in environmental lighting estimation.
Efficiently computing visibility along a ray across Gaussian primitives is non-trivial. Here, we propose ray tracing extracted mesh the fly. During optimization, we periodically extract the object's surface mesh using truncated signed distance function (TSDF) fusion. Once the mesh constructed, we calculate the intersection points between rays and the surface using ray tracing, determining whether each pixel is occluded. For efficiency, we employ a bounding volume hierarchy (BV accelerate the ray tracing process for visibility checks. As for direct lighting Lairs it just corresponds to the second term in Eq.8, handled as described in the previous section.
For the indirect lighting component, each Gaussian is assigned an additional view new-dependent color Lid modeled by spherical harmonics. During the rendering process, lind is evaluated in the reflected direction at the Gaussian level, and alpha blending is applied to aggregate the indirect lighting map follows:
N
i-1
Lind
Lind Og
II a1)a)). j=1
(10)
3.3 GEOMETRY FOCUSED MODEL OPTIMIZATION
To facilitate our physically based rendering, we further enhance the underlying geometry during optimization with the following designs.
Initial stage with per-Gaussian shading As the cost for the benefit of smoothing gradients, pixel- level shading poses convergence challenges during the initial phase, because the shading function disturbs the gradient directions of the geometry. To address this, we propose dedicated initial stage where the rendering equation is applied directly to each Gaussian primitive, using the material and geometry properties associated with it to compute the outgoing radiance. The outgoing radiance is then alpha-blended during rasterization to produce the final PBR rendering. This Gaussian-level shading design can help the gradients be more effectively transferred back to the Gaussian primi- tives and facilitate eventually the optimization process.
Material -wwrre normal propagation Positions with inaccurate normals often have difficulty cap- turing a significant specular component due to its intrinsic sensitivity to the reflected direction. We observe that in Ref-Gaussian there exists strong positive correlation between normal accuracy and high metallic, low roughness properties (a typical situation where the specular component is signif- icant). Under this insight, we propose periodically increasing the scale of 2D Gaussians with high

---

Published as a conference paper at ICLR 2025
Table 1: Per-scene image quality comparison on synthesized test views. The intensity of the red color signifies a better result.
Dataerts
Shiny Blender (Yorhin et al, 2023)
Glonsy Synthetie (Lin
sh, 2023)
Real (Verbin et al, 2023)
Sernes
ball
coffee
belmet
teapol
bell
horse
layu
potion
thell
lcapot
sedan
PSNR
Ref-NeRF
33.15
30.44
33.99
25.
45.12
25. 12
20.89
30.02
29.76
19.30
25.42
30.11
26.91
22.77
22.01
25.21
23.65
ENVIDR
30.57
52.71
30.58
28.03
32.11
26:37
21.47
24.61
22.92
1D038
27.65
27.26
32:30
28.23 22
45.71
20.99
34-49
25.11
$1.36
24.63
26.97
30.16
23.68
21.51
21.75
26.03
23.78
2DX38
25.97
26.38
32:31
27.42
20.42
26.05
34.79
25.18
26.89
29.50
23.38
21.29
22.53
26.23
23.70
(SStader
50.99
27.96
12.99
28. 52
45.86
26.28
25.08
18.07
31.81
26.56
27.18
10,00
24.48
25.58
21.74
74.89
33.76
R3DG
23.64
25.92
30.10
25.01
43.15
18,80
24.90
23.51
27.99
23.37
24.65
27.29
21,25
20.47
21.92
21.18
22.83
3DXIS-DR
94.53
26.76
78.
32.65
26.33
37.01
31.04
34.63
32.32
47.16
28.05
30.38
32.56
33.01
27.05
30,04
33.07
29.84
26.68
22.97
26,60
24.27
SSIM
Ref-NoRF
0.971
0.972
0.954
0.653
0.901
0.807
0.720
ENVIDR
0.997
0,943
0.962
0.987
0.995
0.922
0.934
0.954
0.965
0.925
0.931
0.900
0.947
0.957
0.56]
0.707
0.549
0.937
0.072
0.951
0.996
0.894
0.792
0.908
0.990
0.797
0.916
0.938
0.900
0.881
0.571
0.771
0.617
2003
0.94
0,930
0.972
0.953
0.997
0.892
0.918
0,911
0.958
0.909
0.918
0.919
0,902
0.886
0.609
0.778
0.997
GiStadder
0.065
0:092
0.971
0.951
0.966
0.929
0.914
0.919
0.961
0.913
0.914
0.936
0.808
0.001
0.576
0.728
0.637
R3DG
0.963
0.856
0.889
0.556
3DXIS-DR
0.979
0:963
0.976
0.971
0.997
0.942
0.942
0.999
0.973
0.933
0.943
0.959
0.958
0.942
0.581
0.773
0.639
Ref-Gewrvian
0.981
0.976
0.971
0.548
0.954
0.909
0.973
0.944
0.953
0.963
0.960
0.947
0.617
0.277
0.660
LPIPS
Ref NeRF
0.165
0.050
0,082
0.086
0.012
01085
0.348
0.102
0.104
0.155
0.095
0.084
0.114
0,098
0,251
0.254
0.231
ENVIDR
0.020
0.046
0,083
0.035
0.009
0.081
0.067
0.054
0.049
0,065
0.059
0.072
0.069
0.041
0,263
0.387
0.345
3DGS
0.162
0,047
0,079
0.081
0.008
0.125
0.088
0.104
0.077
0.064
0.093
0.125
0.248
0.206
0.237
2008
0.156
0.062
0.079
0.079
0.088
0.127
0:072
0.109
0.000
0.071
0.066
0.097
0.125
0.101
0.254
0.225
0.386
GShader
0.078
0.074
0,082
0.088
0,562
0.064
0.122
0.259
R3DG
0.214
0:088
0.090
0.125
0.013
11170
11:855
0.125
0.089
0
0.080
0.117
0.156
0.115
0.354
0.380
0.312
3DGS DR
0.105
0:003
0.078
0.050
0.005
0.082
0,052
0.050
0.042
0,087
0.043
0.063
0.059
0.060
0.247
0.203
Ref-Gawssinn
0,076
0.049
0.074
0,042
0,040
0,048
0.043
0.064
0.058
0.256
0.245
0.256
metallic and low roughness so that their normal information can be propagated to adjacent Gaussians for capturing more accurate geometry.
Objective loss function We design a geometry focused objective function: = Le + An Ln + Asmooth Lsmonth, where L. (1 - L) + ALD-SSIM is the RGB reconstruction loss where we set the balancing weight À = 0.2 (Kerbl et al., 2023).
The second term Ln = - NT N is the normal consistency loss. It encourages alignment of the Gaussians with the surface by minimizing the cosine difference between the rendered normal N, obtained via alpha-blending, and the surface normal N, derived from the depth map.
The third term Lemooth VV exp((II ) is an edge-aware normal smoothness loss. It aims to regularize normal variation in regions of low texture.
EXPERIMENTS
To evaluate the performance of Ref-Gaussian in novel view synthesis, well as geometry, mate- rials, and lighting reconstruction, and downstream applications, we conduct a series of quantitative and qualitative experiments. Additionally, We provide a demo video in the supplementary mate- rial, showcasing dynamic renderings of Ref- across two synthetic datasets, along with a comparative analysis.
Datasets We select two synthetic datasets Shiny Blender (Verbin et al., 2022) and Glossy Syn- thetic (Liu et al., 2023) for novel view synthesis of reflective objects, and Ref-Real dataset (Verbin et al., 2022) for real-world open scenes.
Competitors We compare several representative models in the field of reflective 3D reconstruction, including Ref-NeRF (Verbin et al., 2022), ENVIDR (Liang et al., 2023), 3DGS (Kerbl al., 2023), GaussianShader (Jiang et al., 2024), RelightableGaussian (R3DG) (Gao et al., 2023), and 3DGS- DR (Ye et al., 2024). We also compared with more NeRF-based models like NeRO (Liu et al., 2023), NeuS (Wang et al., 2021) and NDE (Wu et al., 2024), see Tables 5 and 6 in appendix.
Evaluation metrics We use three standard metrics: PSNR, SSIM (Wang et al., 2004), and LPIPS (Zhang et al., 2018).

---

Published as a conference paper at ICLR 2025
3DGS
GShader
3DGS-DR
Ours
Ground Truth
Bell
Luyu
Helmet
Figure 3: Qualitative comparisons on reflective scenes, including Bell, Luyu and Helmet.
Figure 4: Qualitative comparisons on Ref-Real dataset (Verbin et al., 2022).
Implementation details Our training procedure consists of two stages. We utilize per-Gaussian rendering for 18, 000 steps as an initial stage, followed by a deferred rendering stage training for about 40, 000 steps. We reset all color and material attributes before the second stage, retaining only the the geometry of Gaussians. The learning rates of the trainable material attributes (metallic, roughness, and albedo) are all set 0.005, while the learning rate of the environment map set to 0.01. The set of other basic trainable attributes for Gaussians like position and covariance is consis- tent with 2DGS (Huang et al., 2024). Then, our material-aware normal propagation is conducted to those Gaussians with metallic no less than 0.02 and roughness more than 0.1. Also, we adopt the metallic initial value from 3DGS-DR (Ye et al., 2024) and set the initial roughness value to be 0.1. During the implementation of physically based rendering, we discovered that spherical harmonics have a better fitting capability than the integrated diffuse lighting and we thereby use spherical har- monics to substitute it. For the loss function, A, is set 0.05 and Asmooth is 1.0. Additionally, periodically extract object's 'ss surface mesh at 3000 step intervals. All experiments are conducted on single NVIDIA A6000 GPU.

---

Published as a conference paper at ICLR 2025
Boll
Pottonn
Rendering
Dirtuse
Metallic
L-Visbilliyy
Mesh
Figure 5: Inverse rendering with extracted mesh and indirect light. Indirect: Only consider indirect light as specular component when rendering.
Scenes
Ground Truth
Ours
3DGS-DR
GShader
ENVIDR
NIVVR
HAA
JAAA
Figure Qualitative comparisons the estimated environment maps on Glossy Synthetic
dataset (Liu et al., 2023).
4.1 COMPARISONS
Novel view synthesis In Table 1, we present the quantitative results of several competitors, along with Ref- Gee--aassinn, for the novel view synthesis task across multiple datasets. Ref-Gaussian achieves state-of-the-art performance in most scenes, especially with significant improvements glossy synthetic datasets, where reflective surfaces dominate. As shown Figure 3. 3DGS (Kerbl et al., 2023) struggles handle high-frequency, view-dependent effects, resulting in noticeable blurriness on reflective surfaces in multi-view scenarios, such as Bell, GShader (Jiang et al., 2024) introduces significant noise, leading distortion. 3DGS-DR (Ye et al., 2024), which employs simplified shading function, finds challenging to accurately model complex reflective scenes, as observed Luyu and the rough tinted visor on Helmet. In contrast. Ref- Gaussian effectively overcomes these challenges, offering precise environmental lighting modeling and robust handling of complex geometries and materials. Additionally, in Figure 4, we present qualitative comparison of real- world scenes (Verbin et al., 2022), Ref-Gaussian provides more accurate normal maps and lighting, capturing subtle nuances on reflective surfaces. Specifically, the reflection phenomena on the garden sphere in the Garden and the car window in the Sedan are particularly noticeable. We demonstrate the more physically accurate reflection detail capture ability of Ref-Gaussian Figure 4.
Material, normal, and light estimation Table 2, we present a quantitative comparison against competitors in terms of normal maps, environment maps, and efficiency. Ref-Gaussian demonstrates

---

Published as a conference paper at ICLR 2025
PBR
Normal
PBR
Nomnal
GT
PBR
w/o PBR
GT
2DGS
3DGS
Figure 7: Ablation study on PBR.
Figure 8: Ablation study on usage of 2DGS.
w/o
2000 iterations
4000 iterations
6000 iterations
8000 iterations
12000
iterations
GT
Ours
w/o Inter
I-visibility
Figure 10: Effect of material-aware normal propa- gation on normal maps at various steps.
Figure Ablation study on inter-reflection.
superior geometry reconstruction, as demonstrated by the high quality of the normal maps. This im- provement is largely attributed to the carefully designed geometry optimization strategies discussed in Scction 3.3. As shown in Figure 5, our normal map is smooth, as well as other decomposed material maps. Additionally, the reflected direction. Ref-Gaussian also achieves the best results among competitors in estimating environment lighting, which is evaluated after re-scaling to eliminate ambiguity. Figure 6 presents a qualitative comparison of the estimated environment maps. Compared to previous methods, our results closely resemble the ground truth in terms of both detail and overall tone, while hibiting the least amount of noise. This improvement can be attributed to the precise geometry and comprehensive modeling employed in Ref- Gaussian.
visibility map is sharp and accurately highlights areas occluded in
Furthermore, Ref-Gaussian excels in optimization speed, demonstrating rapid convergence, and sup- ports real-time rendering. The average training time and frame-pers per-second (FPS) rendering perfor- mance are evaluated on the Shiny Blender dataset (Verbin et al., 2022).
4.2 ABLATION STUDY
We provide quantitative ablation studies for Ref- below.
Physically based rendering Assigning BRDF properties to Gaussians greatly enhances their abil- ity to represent complex scenes. Compared to the simplified shading function using only one reflec- tive direction vector query the environment map as in 3DGS-DR (Ye et al., 2024), Ref-Gaussian demonstrates substantially better performance when learning scenes with material variations, such as the rough band on the sphere (Figure 7) and the visor on the helmet (Figure 3). Also, physically based rendering contributes to the geometry reconstruction, verified by the results in Table 3.
Gaussian-grounded inter-reflection As shown in Figure 9, the impact of our inter-reflection com- ponent is particularly evident on the head part of Tbell and Bell. With ray-traced visibility, we manage to reconstruct indirect lighting for novel view synthesis. Note that, inter-reflection is the minority in the glossy synthetic dataset. Its effect cannot be fully observed from Table 4.
Deferred rendering We conduct physically based rendering at the per- per- ner-Gaussian level and observe that the modeling reflective details becomes noticeably more blurred. As shown in Table 4, without deferred rendering, all quality metrics of the rendered output drop significantly.

---

Published as a conference paper at ICLR 2025
Quantitative comparison on normal maps, environment maps, and efficiency, with all metrics averaged across datasets.
Table 2:
Table Ablation stud- ies on geometry opti- mization techniques.
Model
Normal
Envmup
Efficiency Training time (h),
Model
MAEL
SSIM
IPIPS.
[PIPS]
FPST
Ref-Gaussian
ENVIDR (Liang et al., 2023)
2.74
0.948
0.0728
0.5099
5.84
w/o 2DGS
GShuder (liang et 2024)
7.00
0.931
0.1109
0.6145
0.48
28
w/o Initial
3.53
3DGS-DR DR (Ye et al., 2024)
2.62
0.947
0.0645
0.5290
0.35
251
w/o Material-aware
3,86
Ren-Ginssinn
2.15
0.954
0.0511
0.4431
0.58
122
w/o PER
3.81
Table Ablation studies on the components Ref-Gaussian. w/o Material-aware: use original normal propagation following 3DGS-DR (Ye et al., 2024).
Model
Ref-Gaussiar
Inter-rellection
Deferred rendering
Initial stage
Material-aware
PSNR? 30.33
29.84
30.14
28.85
29.94
29.68
SSIM 0.958
0.956
0.957
0.935
0.956
0.952
LPIPS] 0.049
0.052
0.050
0.074
0.052
0.056
The results, as shown in Figure 8, Table 3 and Table 7 in the appendix, emphasize the superiority of 2D Gaussian primitives over 3D alternative for more precise geometric representation. This is because 3D Gaussian conflicts with the thin nature of surfaces for its volu- metric radiance representation. Notably, replacing 2DGS with 3DGS and keeping other components unchanged, Ref-Gaussian can still achieve outstanding performance across Shiny Blender (Verbin et al., 2022) and Glossy Synthetic (Liu et al., 2023) datasets, as shown in Table 7 the appendix.
2D Gaussian primitive
Per-Gaussian shading initialization The per- par-Gaus physically based rendering used in the initial stage offers notable advantages in both the speed and accuracy of geometric convergence. When we replace the initial stage with an equal number of steps using physically based deferred rendering, the final results show decline in rendering quality and normal reconstruction quality across various metrics.
Material-aware normal propagation The material-aware normal propagation accelerates the convergence of smooth surfaces. The quantitative comparisons presented in Tables 3 and 4 demon- strate its effectiveness in geometry reconstruction. As shown in Figure 10, smooth surfaces are prone to geometric collapse, but our material-aware normal propagation effectively addresses this issue.
4.3 APPLICATION TO RELIGHTING AND EDITING
Ref- Geefsiaassin is generic model, supporting many applications such relighting and editing. For relighting, we model the diffuse term using integrated environment lighting, allowing to be modified. Similarly, we can modify the material properties of the Gaussians to edit the reconstructed scenes. We showcase the results in Figure 11 (appendix).
5 CONCLUSION
In this paper, we have introduced Ref-Gaussian, novel Gaussian splatting framework designed for reconstruction of reflective objects/scenes in real time and high quality. This is achieved by formulating two integrated components: physically based deferred rendering and Gaussian grounded inter-reflection, along with geometry focused model optimisation. We also show the importance of choosing the basis representation model with 2DGS preferred over 3D alternative. Extensive experiments multiple datasets demonstrate the superiority of Ref-Gaussian over existing methods in terms of quantitative metrics, visual quality, and efficiency. Also, we showcase the effect of Ref- Gaussian for other applications such as relighting and editing.
REFERENCES
T. Barron, B. Mildenhall, M. Tancik, P. Hedman, R. Martin-Brualla, and P. P. Srinivasan. Mip- nerf: A multiscale representation for anti-aliasing neural radiance fields. In ICCV, 2021.

---

Published as a conference paper at ICLR 2025
J. T. Barron, B. Mildenhall, D. Verbin, P. P. Srinivasan, and P. Hedman. Mip-nerf 360: Unbounded anti-aliased neural radiance fields. In CVPR, 2022.
T. Barron, B. Mildenhall, D. Verbin, P. P. Srinivasan, and P. Hedman. Zip-nerf: Anti-aliased brid-based neural radiance fields. In ICCV, 2023.
M. Boss, R. Braun, V. Jampani, J. T. Barron, C. Liu, and H. Lensch. Nerd: Neural reflectance decomposition from image collections. In ICCV, 2021a.
M. Boss, V. Jampani, R. Braun, C. Liu, Barron, and H. Lensch. Neural heural-pil: Neural pre- integratetedd lighting for reflectance decomposition. NewrIPS, 2021b.
M. Boss, A. Engelhardt, A. Kar, Y. Li, D. Sun, J. Barron, H. Lensch, and V. Jampani. Samurai: Shape and material from unconstrained real-world arbitrary image collections. NeurIPS, 2022.
B. Burley and W. D. A. Studios. Physically-based shading at disney. In SIGGRAPH, 2012.
A. Chen, Z. Xu, A. Geiger, J. Yu, and H. Su. Tensorf: Tensorial radiance fields. In ECCV, 2022.
S. Fridovich-Keil, A. Yu, M. Tancik, Q. Chen, B. Recht, and A. Kanazawa. Plenoxels: Radiance fields without neural networks, In CVPR, 2022.
J. Gao, C. Gu, Y. Lin, H. Zhu, X. Cao, L. Zhang, and Y. Yao. Relightable 3d gaussian: Real-time point cloud relighting with brdf decomposition and ray tracing. arXiv preprint, 2023.
B. Huang, Z. Yu, A. Chen, A. Geiger, and S. Gao. 2d gaussian splatting for geometrically accurate radiance fields. SIGGRAPH, 2024.
Y. Jiang, J. Tu, Y. Liu, X. Gao, X. Long, W. Wang, and Y. Ma. Gaussianshader: 3d gaussian splatting with shading functions for reflective surfaces. In CVPR, 2024.
H. Jin, Liu, P. Xu, X. Zhang, S. Han, S. Bi, X. Zhou, Z. Xu, and H. Su. Tensoir: Tensorial inverse rendering. In CVPR, 2023.
B. Kerbl, Kopanas, T. Leimkühler, and Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Trans. Graph., 2023.
R. Liang, H. Chen, C. Li, F. Chen, S. Panneer, and N. Vijaykumar. Envidr: Implicit differentiable renderer with neural enviromment lighting. In ICCV, 2023.
Z. Liang, Q. Zhang. Y. Feng, Y. Shan, and K. Jia, Gs-ir: 3d gaussian splatting for inverse rendering. In CVPR, 2024.
Y. Liu, P. Wang, C. Lin, X. Long, J. Wang. I.. Liu, T. Komura, and W. Wang. Nero: Neural geometry and brdf reconstruction of reflective objects from multiview images. ACM Trans. Graph., 2023.
B. Mildenhall, P. P. Srinivasan, M. Tancik, J. Barron, R. Ramamoorthi, and R. Ng. Nerf: Repre- senting scenes as neural radiance fields for view synthesis. In ECCV, 2020.
B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng. Nerf: Repre- senting scenes as neural radiance fields for view synthesis. Commun. ACM, 2021.
T. Müller, A. Evans, C. Schied, and A. Keller. Instant neural graphics primitives with a multiresolu- tion hash encoding. TOG, 2022.
J. Munkberg, J. Hasselgren, T. Shen, J. Gao, W. Chen, A. Evans, T. Müller, and S. Fidler. Extracting triangular 3d models, materials, and lighting from images. In CVPR, 2022.
Y. Shi, Y. WU, C. Wu, X. Liu, C. Zhao, H. Feng, J. Liu, L. Zhang, J. Zhang, B. Zhou, et al. Gir: 3d gaussian inverse rendering for relightable scene factorization. arXiv preprint, 2023.
P. P. Srinivasan, B. Deng, X. Zhang, M. Tancik, B. Mildenhall, and J. T. Barron. Nerv: Neural reflectance and visibility fields for relighting and view synthesis. In CVPR, 2021.
C. Sun, M. Sun, and H. Chen. Direct voxel grid optimization: Super-fast convergence for radiance fields reconstruction. In CVPR, 2022.

---

Published as a conference paper at ICLR 2025
D. Verbin, P. Hedman, B. Mildenhall, T. Zickler, J. T. Barron, and P. P. Srinivasan. Ref-nerf: Struc- tured view-dependent appearance for neural radiance fields. In CVPR, 2022.
P. Wang, L. Liu, Y. Liu, C. Theobalt, T. Komura, and W. Wang. Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. arXiv preprint, 2021.
Wang, A. C. Bovik, H. R. Sheikh, and E. Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process., 2004.
L. Wu, S. Bi, Z. Xu, F. Luan, K. Zhang, I. Georgiev, K. Sunkavalli, and R. Ramamoorthi. Neural directional encoding for efficient and accurate view-dependent appearance modeling. In CVPR, 2024.
Y. Yao, Zhang, J. Liu, Y. Qu, T. Fang, D. McKinnon, Y. Tsin, and L. Quan. Neilf: Neural incident light field for physically-bass based material estimation. In ECCV, 2022.
K. Ye, Q. Hou, and K. Zhou. 3d gaussian splatting with deferred reflection. In SIGGRAPH, 2024.
Z. Yu, T. Sattler, and A. Geiger. Gaussian opacity fields: Efficient and compact surface reconstruc- tion unbounded scenes. arXiv preprint, 2024.
Zhang, Y. Yao, S. Li, J. Liu, T. Fang, D. McKinnon, Y. Tsin, and L. Quan. Neilf++: Inter- reflectable light fields for geometry and material estimation. In ICCV, 2023.
R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang. The unreasonable effectiveness of deep features as a perceptual metric. In CVPR, 2018.
X. Zhang, P. P. Srinivasan, B. Deng, P. Debevec, W. T. Freeman, and J. T. Barron. Nerfactor: Neural factorization of shape and reflectance under an unknown illumination. ACM Trans, Graph., 2021.

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1 LIMITATIONS
As Ref-Gaussian is specifically designed for reconstructing reflective scenes, might not excel for non-specular scenes. When modeling inter-reflections, we only consider the visibility in the reflection direction to approximate the overall visibility across the entire specular lobe. While this approach significantly reduces the computational complexity of rendering, might introduce some noise in material estimation due to the inherent inaccuracy with the approximation.
A.2 QUALITATIVE RESULTS OF RELIGHTING AND EDITING
We illustrate the results of both applications with realistic visual effects in Figure 11.
A.3 FURTHER QUANTITATIVE COMPARISONS ON NON-SPECULAR DATASET
By design, Ref-Gaussian excel for both reflective and non-reflective scenes, making it as unified soluton for modeling a variety of scenes with varying reflection. This because, as described in Eq. inter-reflection is incorporated just as an additional element into the specular component. Consequently, in less reflective scenes, the influence of inter-reflection diminishes naturally and ac- cordingly the metallic attribute decreases. In such cases, the diffuse component instead takes over, including the capture of any (usually substantially weak) inter-reflecttin effect to achieve top performance. This behavior is evident in the decomposition results provided in Figure and Fig- ure 14 in the appendix. In contrast, previous reflection focused alternatives such as 3DGS-DR (Ye al., 2024) designed specifically only for reflective scenes, leading narrowed applications as mentioned.
To support this claim, we have included Table 8 in the appendix, which presents a per-scene quanti- tative comparison among Ref-Gaussian 3DGS-DR (Ye et al., 2024), and 3DGS (Kerbl et al., 2023) on the NeRF-Synthetic dataset (Mildenhall et al., 2020), predominantly composed non-reflective scenes, for novel view synthesis. The results show that Ref-Gaussian still excels over both alternatives, validating its generic and unified advantages.
A.4 EXTRA BASELINES FOR NERF-BASED METHOD
We have further compared the average PSNR, SSIM, LPIPS, and FPS on the Glossy Blender dataset with NeRO (Liu et al., 2023) and NeuS (Wang et al., 2021) in Table 5, as well as with NDE (Wu al., 2024) the Shiny Blender dataset in Table 6. The quantitative results demonstrate that Ref- Raf-Ganssia significantly outperforms NeRO and remains comparable to NDE in rendering quality while achieving significantly better rendering speed and training efficiency.
Table 5: NVS quality of NeuS (Wang al., 2021) and NeRO (Liu al., 2023) the Glossy- Blender dataset (Liu et al., 2023).
Model PSNRT
NeuS
NeRO
Ref-Gaussian
27.86
29.73
30.33
SSIM
0.878
0.904
0.958
LPIPS
0.375
0.324
0.049
FPSt
0.02
0.11
122
Table 6: NVS quality of NDE (Wu et al., 2024) on the Shiny-Blender dataset (Verbin et al., 2022).
Model PSNR1
NDE
Ref-Gaussian
35.48
35.04
SSIM*
0.976
0.973
LPIPS
0.027
0.056
FPSt
66
122

---

Published as a conference paper at ICLR 2025
Table Per-scene PSNR comparison on synthesized test. w/o 2DGS: Using 3DGS the represen- tation of our Ref- Gaussian with the rest unchanged.
Datasets
Shiny Blender (Verbin 2023)
Glossy Synthetic (I.iu et et. 2023)
helmet
teapot
toaster
angel
bell
luyu
potion
teapot
ENVIDR
41.02
27.81
30.57
32.71
42.62
26.03
29.02
30.88
31.04
25.99
28.03
32.11
28.64
26.77
3DGS-DR
33.43
30.48
34.53
31.44
47.04
26.76
29.07
30.60
32,59
26.
96
32.65
29.03
25.77
30.65
30.60
32.94
26.31
Ref-Gaussian
37.01
31.04
34.63
32.32
47.16
28.05
30.38
32.86
33.01
27.05
30.04
33.07
29.84
26.68
A.S QUALITATIVE RESULTS OF INDIRECT COMPONENT ON REF-REAL DATASET
We additionally provided richer qualitative results in Figure 12 where multiple objects provide rich inter-reflection. It can be observed that, in the task of novel view synthesis, the three compo- nents-diffuse, specular, and indirect complecom each other effectively and together pro- duce excellent final rendering result. Specifically, based the metallic property, the diffuse component primarily models non-reflective objects and ground details outside the stone sphere. The specular component focuses the parts of the stone sphere where reflected light is not occluded by the ground or other objects, capturing fine reflective details, and the indirect component effectively captures the projections of the ground and adjacent objects. These decomposition results strongly demonstrate that Ref-Gaussian, even in complex real-world scenes, can accurately capture various details through its well-designed rendering approach and precise geometric reconstruction.
A.6 QUANTITATIVE RESULTS OF COMPREHENSIVE 2DGS ABLATION STUDY
To explicitly showcase the effect of 2DGS representation and to more fairly and comprehensively demonstrate the superiority of Ref- Gaussian with other techniques, over other 3DGS-base based methods, we replace the 2DGS representation of Ref-Gaussian with 3DGS while keeping the rest unchanged for comparison as shown in Table 7. The introduction of 2DGS has significantly enhanced the model's performance. And it is worth emphasizing that the Ref-Gaussian built on 3DGS, also excels in most scenarios of the Shiny Blender (Verbin et al., 2022) and Glossy Synthetic (Liu et al., 2023) datasets, further highlighting the inherent superiority of the rendering model.
A.7 QUALITATIVE RESULTS OF GEOMETRY RECONSTRUCTION
We provide further illustration of our advantage in geometry reconstruction (Figure 13, 14). The qualitative comparison Figure 13 demonstrates Ref- Gef-Gausnnn'' comprehensive grasp details over other advanced techniques (such as the tires in car and the water surface in coffee).
Luyu
Teapot
Editing GT
Relighting
Figure 11: Relighting and editing on Luyu and Teapot with ground truth on the left.

---

Published as a conference paper at ICLR 2025
Ground Truth
Rendering Diffuse lighting
Normal Indirect lighting
Metallic Visibility
Specular lighting
View
View 2
Figure 12: Qualitative results of indirect lighting on Rea- dataset (Verbin et al., 2022). Indirect lighting: Only considering indirect light as specular component when rendering a focused view.
Table 8: Quantitative comparisons on NeRF-Synthetic dataset (Mildenhall et al., 2020).
chair
drums
fieus
hotdog
lego
material
ship
PSNR
3DGS
35.03
26.04
35.29
37.57
33.71
30.04
31.43
3DGS DR
32.10
25.31
28.03
35.58
32,94
28.35
29.07
Ref-Gaussian
34.71
26.33
35.74
37.65
33.46
30.44
30.78
3DGS
0.987
0.954
0.987
0.985
0.975
0.959
0.906
3DGS-DR
0.977
0.946
0.963
0.982
0.978
0.950
0.894
Ref-Gaussion
0.981
0.955
0.988
0.985
0.964
0.898
LPIPS
3DGS
0.012
0.040
0.012
0.021
0.027
0.040
0.111
3DGS DR
0.024
0.055
0.055
0.033
0.026
0.044
0.1
Ref-Gawssin

---

Published as a conference paper at ICLR 2025
GT
Ours
3DGS-DR
GShader
3DGS
ENVIDR
Car
Coffee
Ball
Figure 13: Per-scene qualitative comparisons of normals.

---

Published as a conference paper at ICLR 2025
GT
Rendering
Diffuse
Specular
Normal
Angel
Coffee
Teapot
Toaster
Car
Luyu
Horse
Hotdog
Figure 14: Qualitative decomposition results.


Reviews:
[{"review_id": "dU88phTjU3", "reviewer": "Reviewer_147t", "summary": "Though powerful in novel view synthesis, vanilla 3DGS encounters challenges in extending to physically-based rendering or modeling inter-reflection due to lack of deterministic geometry. This work introduces a novel approach Ref-Gaussian, which achieves real-time high-quality rendering of reflective objects while also modeling inter-reflection effects.\n\nThe paper proposes several key techniques:\n(a) Geometry enhanced technique: employing 2DGS to bridge deterministic geometry with Gaussian splatting and enhancing the geometry by the novel material-aware normal propagation.\n(b) PBR optimization framework for 3DGS-based methods: using per-Gaussian PBR initialization followed with physically-based deferred rendering.\n(c) Gaussian-grounded inter-reflection: applying real-time ray-tracing to the extracted mesh from 2DGS.\n\nExtensive experiments demonstrate the effectiveness of these techniques and that the proposed method outperforms several baselines significantly.", "questions": "1. What's the main difference between the \"material-aware normal propagation\" in the paper and the \"normal propagation\" in 3DGS-DR? According to 3DGS-DR, their normal propagation are also aware of reflection strength. Is there any key improvement over their solution? Otherwise, is it an adaptation from reflection-strength-aware to PBR-attribute-aware?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. By extracting explicit geometry, the paper addresses the inter-reflection issue in Gaussian splatting, which is important in realistic PBR. Experimental results showcase its SOTA performance in novel view synthesis and decomposition on reflective cases.\n2. Instead of per-Gaussian PBR (like Relightable 3DGS or Gaussian Shader), The proposed method employs an effective PBR deferred rendering to achieve better PBR performance (similar to 3DGS-DR). Further ablation study demonstrates the superiority of such a deferred rendering technique over the per-Gaussian solutions.\n3. The proposed method employs a material-aware normal propagation. This enhances normal estimation by periodically increasing the scale of 2D Gaussians with high metallic and low roughness, which demonstrates interesting material-normal interaction in 3DGS-based PBR.", "weaknesses": "1. The effectiveness of inter-reflection technique lacks further qualitative evidence (e.g. providing indirect components in Fig.5 or showcasing indirect components in Ref-Real dataset where the multiple objects provide rich inter-reflection), as the ablation study in table 3 indicates only a slight decrease in PSNR when rendering without inter-reflection.\n2. The ablation study in table 3 only takes PSNR changes into account, while the influences on geometry may need further demonstration (e.g. normal MAE or qualitative illustration), in order to provide stronger evidence for effectiveness of each techniques.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "V9UJy2HLl7", "reviewer": "Reviewer_voQB", "summary": "This paper proposed a novel Gaussian Splatting based inverse rendering framework, which focus on reflective object reconstruction. This method rely on deferred shading to achieve more smooth and cohesive rendering results, as well as the combination of mesh-based visibility and per-Gaussian indirect lighting to model the inter-reflection. The experimenta evaluation demonstrate that this method can accurately reconstruct reflective object while maintaining real-time rendering capabilities.", "questions": "- In ablation study, the normal of 2DGS is much better than 3DGS. Since accurate normals are very important for PBR, does this mean that the performance gain of the proposed method over other methods comes largely from the better geometry reconstruction quality of 2DGS? This is important for evaluating the technical contribution of this paper. I hope the author can give quantitative data (**use 3DGS as representation and keep the rest of the pipeline unchanged to evaluate the rendered results**) to show how much performance improvement can be achieved by using 2DGS representation compared to 3DGS.\n- Since indirect lighting is modeled as an attribute of each Gaussian, which represents the inter-reflection under the lighting conditions corresponding to the training stage. Is it possible to build indirect lighting when relighting?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- Rendering quality is excellent, and reconstructed normals are accurate.\n- Training time and rendering speed are satisfactory", "weaknesses": "- The contribution of this paper may lack novelty. For the deferred shading part of the pipeline, I think many previous 3DGS-based inverse rendering methods[1][2][3] have adopted these techniques and [2][3] also use split-sum approximation to handle the intractable rendering equation. Further more, assigning each Gaussian with a new attribute to model the indirect lighting has also been proposed in previous methods[4]. The innovation of this method lies in the new visibility modeling scheme and optimization techniques. Unlike previous methods that use baked volume or Gaussian-based ray-tracing to model occlusion, the proposed method attempts to first extract the mesh using TSDF and then use mesh-based ray-tracing to obtain occlusion.\n\n- To determine the visibility, this method consider the occlusion at the reflected direction $\\boldsymbol{R}=2\\left(w_i \\cdot \\boldsymbol{N}\\right) \\boldsymbol{N}-w_i$. This means that this method only considers the indirect lighting of the specular surface. For glossy or diffuse surfaces, this estimation may not be not accurate enough, and such objects do exist in the dataset. For example, for the Potion in figure.5, The lid of the bottle is obviously a rough diffuse surface. Since the proposed method focuses on the reconstruction of the reflective object, this does not seem to be a serious problem, but I would like to know if there is a way to improve this.\n\n- In addition, since the visibility only considers the reflected direction, I am a little confused about the integral in equation 9. Because this method does not calculate the integral over the entire hemisphere $\\Omega$. So I want to know how the rendering equation is finally calculated.\n\n\n\n\n\n[1] DeferredGS: Decoupled and Editable Gaussian Splatting with Deferred Shading https://arxiv.org/abs/2404.09412 \n\n[2] GS-IR: 3D Gaussian Splatting for Inverse Rendering https://openaccess.thecvf.com/content/CVPR2024/papers/Liang_GS-IR_3D_Gaussian_Splatting_for_Inverse_Rendering_CVPR_2024_paper.pdf\n\n[3] 3DGaussian Splatting with Deferred Reflection https://dl.acm.org/doi/pdf/10.1145/3641519.3657456\n\n[4] Relightable 3D Gaussians: Realistic Point Cloud\n Relighting with BRDF Decomposition and Ray\n Tracing https://www.ecva.net/papers/eccv_2024/papers_ECCV/papers/06121.pdf", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "l8L8ojX29Y", "reviewer": "Reviewer_9fZv", "summary": "This paper proposes a Reflective Gaussian splatting (*Ref-Gaussian*) framework to achieve real-time high-level novel view synthesis for reflective objects with inter-reflection effects. The framework consists of 2 main parts: (i) physically-based deferred rendering, which associates each Gaussian with material properties and leverages physically-based surface rendering (rendering equation and split-sum approximation) to compute the rendered color; (ii) Gaussian-grounded inter-reflection, which computes the visibility of the reflected ray and model indirect view-dependent color as a per-Gaussian spherical harmonics. The proposed method also leverages several techniques to enhance the underlying geometry. This paper evaluates the proposed method with qualitative and quantitative experimental results and validates its design components by an ablation study. It shows that the method can outperform baselines.", "questions": "1. L239-244. More details are required to describe how the indirect lighting is rendered. To be specific,\n    - Does the method conducts explicit ray tracing for the reflected ray and computes ray-Gaussian intersection? Or does the method uses Gaussian splatting in the reflected ray directions?\n    - In Eq. (10), the meaning of the symbol $N$ in $i\\in N$ is unclear: does it denote the set of Gaussians intersected with the reflected ray?\n2. L254: \"To mitigate this, we apply the rendering equation at the Gaussian level to achieve geometry convergence initially.\" What does \"rendering equation at the Gaussian level\" mean? What is the difference between this and Eq. (8)?\n3. Are the training hyperparameters the same across different scenes?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The key designs proposed by the paper are rational. It is not surprising that leveraging physically-based rendering can significanty improve the performance of specular rendering.\n2. I like the idea of material-aware normal propagation. Seems like it can greatly improve the quality of surface normal reconstruction.\n3. Extensive qualitative and quantitative experiments demonstrate that the proposed method outperforms baselines.\n4. I appreciate the comprehensive ablation study that covers lots of specific design choices.", "weaknesses": "1. Missing details. L233-234: \"During optimization, we periodically extract the object’s surface mesh using truncated signed distance function (TSDF) fusion.\" The authors need to specify the number of steps as the period of mesh extraction.\n2. Unclear explanations. I have several confusions when reading the paper, and please see the \"Questions\" part. I believe that the paper writing can be improved to explain the method more clearly.\n3. Baselines. Although this method is based on 3DGS, it should also include more methods based on other representations (such as NeRF) as its baseline, such as (a) NeRO: Neural Geometry and BRDF Reconstruction of Reflective Objects from Multiview Images (SIGGRAPH 2023) (b) Neural Directional Encoding for Efficient and Accurate View-Dependent Appearance Modeling (CVPR 2024).\n4. Limitation of Gaussian-grounded inter-reflection: In Eq. (8), it is clear that the direct lighting part can capture rough specular effect. But in Eq. (10), the method only traces a single reflected ray to compute the indirect reflection, which will introduce errors for rough surfaces. This should be added as a limitation.\n5. Experiments. The method leverages the extracted mesh to compute visibility, so I think the author should also show (at least qualitative) results of the extracted mesh in the experiment section.\n\nI may consider increasing the rating if my concerns and questions are addressed in the rebuttal.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ExDoRxWpEY", "reviewer": "Reviewer_xuc5", "summary": "This paper focuses on novel view synthesis for reflective objects. It proposes a method called Reflective Gaussian Splatting, which uses Gaussian Splatting as its primary representation and introduces two additional techniques: (a) physically-based deferred shading and (b) an accurate approximation for modeling inter-reflection effects within the Gaussian Splatting framework. The effectiveness of this method is validated using various benchmark datasets. Additionally, the method is compute-efficient, making it suitable for applications such as relighting and material editing.", "questions": "1. From the paper, I assume that the geometry is important for disentangling appearance and those PBR materials. I would like to know if PBR modeling will also benefit geometry. \n\n2. Why is the LIPPS metric of the proposed method on the real dataset worse than 3DGS-DR?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The proposed method achieves state-of-the-art performance on several benchmark datasets, validating its significance. \n2. The proposed method offers a good trade-off between performance and efficiency. The introduced deferred shading technique, plus the occlusion approximation using a TSDF mesh, sounds reasonable for these goals.\n3. The paper is well-written and easy to follow. The paper has enough details for reproduction.", "weaknesses": "1. While efficient in modeling the occlusion with an extracted mesh, its performance may rely on the accuracy of the extracted mesh. Since extracting a good mesh for the reflective object is non-trivial,  such a non-differentiable approximation may lead to worse results.\n\n2. While built upon 3DGS-DR, its performance (LPIPS) on real datasets appears limited over 3DGS-DR. \n\n\n#minors:\nThe citation in L49~L50 seems misplaced.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)