GI-GS: Global Illumination Decomposition on Gaussian Splatting for Inverse Rendering
Authors: HONGZE CHEN
Zehong Lin
Jun Zhang
Venue: ICLR 2025 Poster (2025)
Keywords: Inverse rendering
Gaussian Splatting
Relighting
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: A 3DGS-based inverse rendering framework that focuses on accurate global illumination decomposition via efficient path tracing with deferred shading

Abstract: We present GI-GS, a novel inverse rendering framework that leverages 3D Gaussian Splatting (3DGS) and deferred shading to achieve photo-realistic novel view synthesis and relighting. In inverse rendering, accurately modeling the shading processes of objects is essential for achieving high-fidelity results. Therefore, it is critical to incorporate global illumination to account for indirect lighting that reaches an object after multiple bounces across the scene. Previous 3DGS-based methods have attempted to model indirect lighting by characterizing indirect illumination as learnable lighting volumes or additional attributes of each Gaussian, while using baked occlusion to represent shadow effects. These methods, however, fail to accurately model the complex physical interactions between light and objects, making it impossible to construct realistic indirect illumination during relighting. To address this limitation, we propose to calculate indirect lighting using efficient path tracing with deferred shading. In our framework, we first render a G-buffer to capture the detailed geometry and material properties of the scene. Then, we perform physically-based rendering (PBR) only for direct lighting. With the G-buffer and previous rendering results, the indirect lighting can be calculated through a lightweight path tracing. Our method effectively models indirect lighting under any given lighting conditions, thereby achieving better novel view synthesis and competitive relighting. Quantitative and qualitative results show that our GI-GS outperforms existing baselines in both rendering quality and efficiency. Project page: https://stopaimme.github.io/GI-GS-site/.

Published as a conference paper at ICLR 2025
GI-GS: GLOBAL ILLUMINATION DECOMPOSITION ON GAUSSIAN SPLATTING FOR INVERSE RENDERING
Hongze Chen, Zehong Lin', Jun Zhang The Hong Kong University of Science and Technology hchenec&connect" ust.hkk hk, eezhlin@use .kk ee sejzhangeust.
Pretrained 3D Goussians
Operation with gradient Operation without
with
Albedo
Mazilic
Roughaess
enviommet map
Opacity
Narmal
Moa
Conerianat
Mulli Tragos
Pegurcall--toecl rendering
Rendered
Resterioz
map
Sccondlggaas
Depth map
Nonmal map
Masccal map
Ras traing
Diffirzttible tay tncing
Ground truit
Rasterize G-buffer
First-pass rendering occlusion
Trace Indirect Illuminatiio
Figure 1: Overview of GI-GS. GI-GS takes input set of pretrianed 3D Gaussians, each with mal attribute. It first rasterizes the scene geometry and materials into a G-buffer. Next, it incorporates differentiable PBR pipeline obtain the rendering result under direct lighting and performs path tracing to model the occlusion. Finally, it employs differentiable ray tracing to calculate indirect lighting from the scene geometry and the previous rendering result. The final rendered image is fusion of the first-pass and second-pan pass results and uses the ground truth image for supervision.
ABSTRACT
We present GI-GS, a novel inverse rendering framework that leverages 3D Gaus- sian Splatting (3DGS) and deferred shading to achieve photo-realistic novel view synthesis and relighting. In inverse rendering, accurately modeling the shading processes of objects is essential for achieving high-fidelit results. Therefore, it is critical to incorporate global illumination to account for indirect lighting that reaches an object after multiple bounces across the scene. Previous 3DGS-bass methods have attempted to model indirect lighting by characterizing indirect illu- mination learnable lighting volumes additional attributes of each Gaussian, while using baked occlusion to represent shadow effects. These methods, how- ever, fail accurately model the complex physical interactions between light and objects, making it impossible to construct realistic indirect illumination during re- lighting. To address this limitation, we propose calculate indirect lighting using efficient path tracing with deferred shading. In our framework, we first render G-buffer to capture the detailed geometry and material properties of the scene. Then, we perform physically based rendering (PBR) only for direct lighting. With the G-buffer and previous rendering results, the indirect lighting can be calcu- lated through a lightweight path tracing. Our method effectively models indirect lighting under any given lighting conditions, thereby achieving better novel view synthesis and competitive relighting. Quantitative and qualitative results show that our GI-GS outperforms existing baselines in both rendering quality and efficiency. Project page: httpp:: stopa imme..ammmm....mmmmm mme. .areee.gitttn github. 1 Lo/GI-GS SIICCI--SS-sttee ssset/
"Corresponding author

---

Published as a conference paper at ICLR 2025
INTRODUCTION
Inverse rendering (Barrow et al. 1978) aims to estimate the physical attributes of a 3D scene, such as material properties, geometry, and lighting, from captured images. This task is crucial for various downstream applications, including novel view synthesis, relighting, and VR/AR. Similar to many inverse problems, inverse rendering often considered ill-posed due to unknown lighting conditions and the intractability of the integrals in the rendering equation. Recent research has adopted the Neural Radiance Fields (NeRF) (Mildenhall et al.) 202I) paradigm, which represents scenes as neural fields using multi-layer perceptrons (MLPs). Nonetheless, rendering image with NeRF- like methods requires dense querying of ML.ss along ray directions, which significantly affects the rendering speed. Moreover, the limited representational capacity of MLPs hinders their ability to faithfully model physical attributes of scenes and complex interactions between light and surfaces.
More recently, 3D Gaussian Splatting (3DGS) (Kerbl et al. 2023) has gained significant attention for its superior reconstruction quality and rendering speed. This method explicitly represents scenes using set of 3D Gaussians and employs tile-based rasterization pipeline for efficient rendering. Unlike NeRF, which utilizes MI.Ps to encode diewddependent colors across the scene, 3DGS assigns spherical harmonics each Gaussian point to capture these view-dependent effects. Nevertheless, 3D Gaussians trained under specific lighting conditions cannot be used to render relighted results under different lighting setups, as the view-dependent colors entangle the objects' materials with the training lighting conditions. This constraint restricts the broader applicability of 3DGS. To address this limitation, recent studies (Jiang et al.] 2024) Shi et al. 2023 Liang al.] 2024; Gao et al, 2023 Wu et al.] 2024) have proposed several inverse rendering frameworks based on 3DGS. These methods successfully decouple color into the interactions between lighting and surface materials by incorporating physically-based rendering (PBR) techniques. Some of them have also attempted to model indirect lighting, but they typically store the indirect lighting either statically in a 3D Gaussian or in additional volumes and optimize it during training. These approaches overlook the dynamic nature of light and, therefore, cannot accurately model indirect lighting during the relighting process.
To overcome these limitations, propose GI-GS, novel inverse rendering framework based on 3DGS that achieves accurate estimation of materials and geometry, as well as global illumination decomposition. Our key innovation is to calculate indirect lighting by efficient path tracing based on the deferred shading technique. Specifically, as shown in Fig. L we first rasterize the geometry and materials of the scene, storing the results in G-buffer. Then, we incorporate PBR pipeline to efficiently compute the complex rendering equations and thoroughly decompose the illumination. We employ a learnable environment map approximate direct lighting. For indirect lighting and occlusion, instead of modeling them as attributes of 3D Gaussian points or using baked volumes as in previous methods, we calculate them from the first-pass rendering results through path tracing- This approach ensures that our indirect lighting is more consistent with the rendering equation. Con- sequently, our approach can reconstruct high-fidelity geometry and materials of a complex real scene under unknown natural illumination, thereby achieving state-of-the-art rendering of novel view syn- thesis and enabling additional applications like relighting. Notably, to the best of our knowledge, are among the first to develop a 3DGS- based inverse rendering framework capable of modeling indirect lighting during the relighting process.
In summary, main contributions are follows:
We present GI-GS, novel inverse rendering framework based on 3DGS that achieves oftee-of-thearr art intrinsic decomposition and rendering results for both objects and scenes.
We propose an efficient tile-based path tracing method based on deferred shading, which enables accurate occlusion and indirect lighting calculation from previously rendered re- sults stored in the G-buffer.
We extend our path tracing- based indirect lighting from screen space world space for scene-level datasets, achieving more natural occlusion and indirect lighting.
2
RELATED WORKS
Neural Rendering and Radiance Fields. Neural rendering, exemplified by NeRF (Mildenhall
al. 2021), has received widespread attention in recent years due to its remarkable representational
2

---

Published as a conference paper at ICLR 2025
capabilities and support for end-to-end optimization. NeRF implicitly represents scene as contin- uous radiance field using MLPs that take the 3D positions and view directions as inputs to output the density and deeedeeendeent color. However, the rendering process of NeRF requires dense impor- tance sampling along rays, which involves repeated MI.P queries and consequently limits rendering speed. To address this issue, subsequent research has introduced efficient explicit data structures, such as hash grids (Müller et al. 20022) voxels (Sun et al.) 2022 Fridovich- et al.] (2022), and tri-planes (Chen et al. 2022). to encode features in 3D space, thereby accelerating both training and inference phases. In addition, some works (Barron et al. 2021 2022 2023) have focused on
enhancing the rendering quality of NeRF, while others (Wang et al. 2021 Yariv et al.) 202T] have utilized radiance fields to represent signed distance functions (SDFs) for surface reconstruction.
Recently, 3DGS (Kerbl et al. 2023) replaces MLPs with set set 3D Gaussian primitives to explicitly represent the neural field and incorporates a tile-based rasterization pipeline to achieve high-quality real-time rendering. Since 3DGS is originally designed for novel view synthesis and lacks relevant geometric constraints or priors, how to reconstruct accurate geometry from 3DGS remains im- portant problem. Subsequent methods have significantly improved the quality of the reconstructed geometry by integrating geometrically relevant regularization terms (Guédon & Lepetit (0224) or employing more geometrically accurate representations (Huang et al.) [0024] Dai et al. (0024). In our work, we extend 3DGS the inverse rendering task by combining it with PBR, thereby broadening its applicability to a wider range of downstream tasks.
Inverse Rendering. Recovering geometry, materials, and lighting from captured images has been a long-standing challenge due to the unknown lighting conditions and the complex interactions between light and objects. Most previous inverse rendering methods simplify the problem by in- troducing prior information. For instance, [Brrron & Malik (2014); Li et al. (2018) introduce plane assumptions simplify the acquisition of normals, [Bi et al. (2020b) b Nam al. (2018) use flash- light to align the light source with the camera positions, and Xia et al. (2016) capture rotating object under fixed camera and lighting conditions.
Inspired by the success of volume rendering techniques based on radiance fields, subsequent works (Bi et al.] [2200a] Srinivasan et al., [202T] [Boss et al.) [20TT] Zhang et al.) [022b] Jin et al.] [023] Zhang et al.] 202 2022. Yao et al.
(0222) have attempted to apply the NeRF paradigm to address this issue by leveraging the powerful representational capabilities of neural networks. For instance, 2020a) is the first work to introduce the bidirectional reflectance Neural Reflectance Fields (Bi et al.
distribution function (BRDF) and lighting in NeRF, which represents a scene as a reflectance field under point light source. NeRV (Srinivasan et al. 2021) uses an environment map to represent the given light condition and trains an ML.P to approximate the visibility of a given point in space from different directions, while PhySG (Zhang et al.) 2021a) models lighting as spherical Gaussians and incorporates SDFs for more accurate geometry reconstruction. Besides, NeRFactor (Zhang et al. 2021b) adopts a multi-stage training reduce the difficulty of optimization. TensoIR (Jin et al. 2023 employs a compact tri plane representation for ray tracing-based indirect lighting calculations. Moreover, Neilf (Yao al. 2022) utilizes a neural incident light field to better model the light source.
Recently, some works (Jiang et al. 2024 Wu et al. 2024h have extended 3DGS to the inverse render- ing task by binding BRDFs as attributes Gaussians. Besides, GS-IR (Liang et al., 2024) leverages baking-based volumes to store occlusion and indirect illumination. Relightable 3D Gaussian (Gao) (et al. 2023) bakes the occlusion by using point-based ray tracing and parameterizes the indirect lighting as additional spherical harmonics for each Gaussian. Nevertheless, both methods rely on baking to accelerate rendering and cannot model indirect lighting during relighting. In contrast, we propose 3DGS-based framework for inverse rendering and leverage deferred shading to realize efficient path tracing- based occlusion and indirect illumination for both rendering and relighting.
3 PRELIMINARIES
3D Gaussian Splatting. 3DGS (Kerbl et al. 2023) explicitly represents a scene as a set of 3D Gaussian primitives. Each primitive is parameterized by a Gaussian distribution as:
G(æ)) = e (=- xx-((=1))
(1)
where μ E R denotes the mean vector and Σ E R 3x3 denotes the covariance matrix. Besides, each 3DD Gaussian primitive is assigned set of spherical harmonics model the view-dependent color
3

---

Published as a conference paper at ICLR 2025
and opacity for volume rendering. During the rendering process, the 3D Gaussians projected onto the 2D image plane through EWA splatting (Zwicker et al.] 2001). Specifically, given the view transformation matrix W and the Jacobian of the affine approximation of the projective transforma- tion J, the covariance matrix E' in camera coordinates is computed as E' = JJWEW" J For each screen pixel, its color obtained by a-blending 21) Gaussians sorted by depth
C= Σ T.aa; with 7; iEN
II (1-a)). j=1
where N denotes the set of Gaussians, og is the transmittance of each Gaussian derived from its opacity and covariance matrix, and T; is the accumulated transmittance.
Physically-basee Rendering. PBR is rendering approach that models the color of an object as the interaction light and surface material. Given a surface point æ and its normal vector n, the outgoing light L, in the camera view direction w, can be calculated using the following rendering equation (Kajyy 1986):
Lo (Wo: 2z) 1 Jr (Wo, wi, a;) Li (Wi, a²) (wi n.) dwi,
where is the upper hemisphere centered at x and fr (wo, wi x) is the bidirectional reflectance distribution function (BRDF), which characterizes the relationship between the outgoing light Lo and the incident light L; coming from direction wi based on the material of the object. The BRDF can be divided into diffuse component Ja and specular component according to the Cook- Torrance BRDF model (Cook & Torrance 1982) as follows:
D(h; F(Wo h; a, m) m)GGGw Wo, h)
fr(wi, (1 m) π
(4)
4(n (ln:w:))(r wi) wo)
difluse fa
specular (wwww.)
where a E [0, 11s denotes the diffuse albedo, m E [0, 1] denotes the metallic value, P E [0. denotes the roughness, and h - is the half vector. The specular component fa wo) is further decomposed into normal distribution function (NDF) D, Fresnel term F, and geometry term G. Consequently, the outgoing irradiance Lo can be decomposed into diffuse light La and specular light L, as follows:
Lo (wo: x) = La (x) + Lo (Wor x)
(5)
where
La (xx) = 1 Sali (wi, x)) (wi n) dwss La (Wo, x)) / So [o(w;, Li (wi, at) (wo n)) dws, (6)
Global Illumination Decomposition. In real-world scenes, the incident light reaching surface is often a combination of direct illumination from light sources and indirect illumination from light reflected off other objects. This is due to the occlusion of the light sources by the environment or the object itself, as well as multiple bounces of light between objects. To achieve a more realistic rendering of these scenes, for the diffuse component La, we decompose the incident light into direct illumination and indirect illumination, and calculate them separately follows:
La (x)) [ (11mm)=- m) Li (We, x)) (we n) dw;
Ryn L (wi, x)) (wo n)) dwo; (
=11 m))
Lind (wi, (wi n) dwss)
(7)
S
IInd(x)) m) O(x) (a)) + m) π
where the two integrals calculate the irradiance of the incident direct lighting and indirect lighting in the visible area Uvis and the occluded area Poce in the upper hemisphere, respectively. Here, approximate these two
have Ry U fooe Ω and Pvvis n Roce D. Furthermore, integrals using the direct irradiance Lur and the indirect irradiance Lind- and introduce an occlusion term (x)) amproximate the visibility.
4

---

Published as a conference paper at ICLR 2025
4 METHOD
In this section, we present our GI-GS framework for inverse rendering, as depicted in Fig. [ Given a set of posed RGB images, the pipeline proceeds in three stages. In the first stage, we follow the vanilla 3DGS pipeline to reconstruct the scene geometry. We incorporate the normal as a new attribute for each Gaussian primitive and optimize it using pscudo normals derived from the depth map (see Sec. Se.. In the second stage, to recover the BRDF and lighting conditions, we incorporate a PBR pipeline with the deferred shading technique. We first render the depth map, normal map. and BRDF maps from the 3D Gaussians and store them in a G-buffer. Then, we estimate the occlusion from the depth map and normal map, and leverage differentiable PBR pipeline render an image Lair under direct lighting using a learnable environment map (see Sec. (.2).. In the final stage, given the geometry (depth and normals) and the first-pass rendered image Lair. we apply differentiable path tracing to obtain the rendering result Lind under indirect lighting (see Sec. 4.3 J. Finally, we combine the two rendering results and optimize the lighting and the Gaussians' BRDF attributes.
4.1 GROMETRY RECONSTRUCTION
To enable PBR, we first need to reconstruct reliable scene geometry from 3DGS.
Depth Rendering. We follow GS-IR (Liang et al. 2024) treat the depth as a linear interpolation of the depth of the Gaussians, i.e., Σi w,dd where w; = Σ Tia Compared di-
T,o
rectly a-blending the depth of each Gaussian, this approach reduces artifacts and ensures a smoother transition in rendered depth values between the deepest and shallowest Gaussians.
Normal Estimation. In this work, treat the normal n attribute of each Gaussian and use blending to render the normal map. Unlike existing methods (Shi et al. 2023 Chen et al. 2024) that the direction of the shortest axis of the ellipsoid the normal direction and apply regularization to force the Gaussian into a flat disk, our approach avoids the degradation in rendering quality caused by regularization. To enhance geometric consistency, we derive pseudo normal map á from the depth map during the optimization process and use it to supervise the rendered normals. Moreover, we incorporate the following total variation loss (TV-loss) from GS-IR achieve smoother results:
L = =na + An-TV L-Vimma: Ln.p = n2 nll-
4.2 DIRECT LIGHTING MODELING
To capture the illumination from various directions and simplify the calculation, employ image- based lighting (IBL. to model the direct lighting. According to the rendering equation in Eq. [ we decompose the outgoing radiation into diffuse component Tag and specular component T.g- For the diffuse component Ly, we store the incident irradiance of direct lighting Lair (see Eq. from different view directions in a prefiltered environment map.
To tackle the intractable integral of the specular component Lg, we leverage the widely used split- sum approximation (Krris & Games 2013) divide the integral into two parts as follows:
DFG
L
Li (wi, xr) (wi n)) dws
In A(n. ((n-wn) .w:)(n w.)
DFG
(9)
dlos
Le (w:) D (wo, w.) (wi n)) dusi.
4(n ww)
BRDF integral R
Pre-filtered environment map 1.
The first term R represents the integral of BRDF under a constant environment light and is only determined by the angle 9 between the surface normal n and view direction wo, as well as the roughness p. This integral can be precomputed and stored in a 2D look-up table. The second term I, represents the incident irrdiance considering the NDF D. Since the NDF D related to the surface roughness p (the rougher the surface, the larger the specular lobe), we can also store the incident irrdiance in prefiltered environment maps with different mip-levels for different roughness values. Therefore, the rendering result under direct lighting can be written as:
Lair (1 m)
O<) + RI,,
a
(10)
where (z)) is the occlusion at point obtained through path tracing, elaborated in Sec. 4.3
5

---

Published as a conference paper at ICLR 2025
Unoccluded Direction Occluded Direction
Aa
(x. y.
occlusion
More occlusion
(h)
Figure 2: (a) Occlusion quantifies the degree to which a surface point can receive ambient light. Points on flat surfaces demonstrate high visibility due less occlusion from surrounding geometry. In contrast, points located in holes, corners, adjacent to other surfaces appear darker, since they are more occluded. (b) Path tracing: For each surface point corresponding to a pixel on the depth map, we perform ray marching starting from that point to calculate the visibility in different directions.
4.3 INDIRECT LIGHTING MODELING
Indirect lighting refers to the illumination that reaches a surface after reflecting off surfaces. It is a crucial component for realistic rendering, particularly in areas occluded by other objects or by the surfaces themselves, as depicted in Fig. [(a) Typically, to calculate occlusion and indirect lighting, can sample several rays from the normal-oriented hemisphere Q of the surface and determine whether these rays intersect with other surfaces. However, 3DGS explicitly represents scenes as set of points, which complicates the use of path tracing methods. Inspired by the deferred shading techniques from the gaming industry, we propose a G-buffer based indirect lighting. Our key insight to recover the geometry from the G-buffer and implement path tracing on the recovered surface. To accelerate the speed of path tracing, we implement tile-based fast ray marching framework.
Ambient Occlusion Calculation. Given the rendered depth map and normal map from the first stage, for a surface point = (u, v) in the depth map, its position in the view space (X, Y, Z) can be obtained from its depth
ddu,, v) represents its depth and (cz, C2) is the coordinate of the optical center. The occlusion at surface point æ is expressed as the integral of the visibility function V from various directions w f2 follows:
= (xr, y, =)² z(u Cz. v Cy: 1)" where
0(x)=11 = / V(w) wdw,
(11)
where V(w)= 1 if ray starting from point with direction w intersects the surface and (ww) otherwise. Since it is difficult to perform uniform sampling directly in solid angle space, we first consider the equivalent form of the original integral in spherical coordinates and uniformly generate sampling vectors to estimate the integral. For more details, please refer to Appendix B
Adaptive Path Tracing. Path tracing is used to determine whether a ray hits the surface. As illustrated in Fig. 2(b) consider a ray starting from point æo R3x1 with direction w E R3311 any point x = (xr, y, z) R3x1 on this ray can be written as æ = 3o + wl, where is the distance from To- Then, pixel coordinate (U, V) is obtained as æ = (u, v) = ( Cz: 2 Cyy)
trace the sample points along the ray. The position of sample point in the y.
a is larger than the depth zd = d(u, v) at the corresponding coordinate on the depth map and less than za + 5, where 5 the thickness of the surface, the ray intersects surface and is occluded in the w direction. That is,
V(w) {
if 2d < 2d 5.
(12)
else.
To handle the issue of perspective projection, where the marching of ray in 3D space is not linear to the marching of ray in 2D screen space, we adaptively adjust the step length based on the distance of the current ray- ray-marching starting point. Please refer to Appendix B for more details.
Indirect Lighting Calculation. When calculating indirect lighting, other objects occluding the
surface point are regarded indirect lighting sources, shown in Fig. 3(a) According to rendering equation in Eq. 3 the incoming radiation at point x in direction w is the outgoing radiation of the intersection point & in direction-44 i.e.. (x w) L. -w). To obtain the radiation from intersection points, we first render the RGB image Iavr of the scene, considering only direct lighting (see Sec. 4.2) Then, we employ the aforementioned path tracing method to determine if ray

---

Published as a conference paper at ICLR 2025
Camera
Light Source
@pp
w
69)
00)
(a)
(b)
Figure 3: (a) When the surface point is occluded by a in direction w, a acts as an indirect light source. In this case, the light received by æ in direction w; is equivalent the light from the actual light source reflected by è in the -w; direction. (b) To obtain global geometry and lighting information, we rotate the camera to render the results of the remaining five perspectives and form cubemap. This cubemap is then utilized to restore the global geometry and facilitate path tracing.
intersects with other surfaces. Once an intersection point è is identified, the outgoing radiation obtained by indexing the corresponding pixel value Tav (ú, ú) the RGB image based on the position of the intersection point (ú, 6). Consequently, the outgoing diffuse radiation under indirect lighting has a similar form to the occlusion integral in Eq. [aas as follows:
a [ π
Lind (1 m)
L, (wi, x) n widwi, Li(,,, a) = V(w,, x Lair (ú. 6).
(13)
We use uniform sampling to calculate indirect lighting and obtain the final rendering result as:
Lo(Wo; σ) Lidir Lind (1 m) π
O(x)) RI
Lind
(14)
First-puss
Second-pass
To optimize the materials and lighting, we minimize the following decomposition loss:
La
L1
+ AM LTV
AELTViin
(15)
Cooks
Lanirral
Eighn
where the first term is the color loss, and the remaining two terms are the material TV loss Lmsttrial and lighting TV loss Light introduced in GS-IR (Liang al.] (024) to smooth the materials and lighting. Please refer to Appendix B for more details about Lmaterial and Ligght
4.4 CUBEMAP BASED INDIRECT LIGHTING
In Sec. the scene geometry is recovered from the depth map. However, this approach only cal- culates occlusion and indirect lighting in screen space, considering only the local geometry around the surface point. For complex real-world scenes, occlusion and indirect lighting may originate from areas outside the camera's 's frustum. Thus, it is necessary to model the scene's 's global geometry. To this end, we extend our path tracing-bas based indirect lighting from screen space to world space. Specifically, illustrated in Fig. 3(b]
utilize cubemap recover the global geometry and calculate indirect lighting based on it. By taking the current rendering result (depth map. normal map, and RGB image Tarr) as the front face, we change the camera's pose render the remaining five faces of the cubemap. Each face of the depth cubemap can be roughly regarded as the geometry of the scene in that direction, and the corresponding indirect illumination is contained in the RGB cubemap. Then, we calculate the indirect lighting by path tracing, as elaborated in Sec.
5 EXPERIMENTS
5.1 SETTINGS
Datasets and Metrics. We perform experiments on the TensoIR synthetic dataset (Jin et al. 2023) and the Mip-NeRF 360 (Mildenhall et al. 2021) dataset. For both datasets, evaluate the quality of novel view synthesis to compare the overall performance. Notably, the TensoIR dataset provides ground truth results for albedo and relighting under different lighting conditions, which allows us to further evaluate the performance of our estimated albedo and relighting outputs. To quantify the performance, employ three key metrics: PSNR, SSIM, and IPIPS.
7

---

Published as a conference paper at ICLR 2025
Table I: Quantitative results on the TensoIR dataset. The results show that our method surpasses all baselines in novel view synthesis and albedo reconstruction. ranks second only to TensoIR in the relighting results. Notably, our method effectively models indirect lighting when relighting.
NVS
Albedo
Relighting
Time
PSNR
SSIM
LPIPS 4
PSNR
SSIM
LPIPS 4
PSNR
SSIM t LPIPS
NeRFactor
24.68
0.922
0.120
25.13
0,940
0.109
23.38
0,908
0.131
100 hrs
InvRender
27.37
0.934
0.089
27.34
0.933
0.100
23.97
0.901
0.101
14 hrs
NVDiffrec
30.70
0.962
0.052
29.17
0.908
0.115
19.88
0.879
0.104
<1 hr
TensoIR
35.09
0.976
0.040
29.27
0.950
0.085
28.58
0.944
0.081
4.5 hrs
GS-IR
35.33
0.974
0.039
30.29
0,941
0.084
24.37
0.885
0.096
33 min
Ours
36.75
0.972
0.037
31.97
0.941
0.085
24.70
0.886
0.106
28 min
TensoIR
GS-IR
Ours
GT
TensoIR
GS-IR
Ours
GT
Rendering
Albodo
Relighting
Figure 4: Qualitative comparison on the TensoIR dataset. We visualize the rendering results, reconstructed albedo, and relighting results, respectively.
Baselines. We compare our GI-GS method against several baselines that focus on inverse rendering, including NeRRactor ZZhang et al. (2221b. InvRender (Zhang et al.] 2022). NVDiffrec (Munkberg
al. 2022). TensoIR (Jin al. (023)) well as recent state-of-the-art (SOTA) 3DGS-based meth- ods, GaussianShader (Jiang et al. 2024) and GS-IR (Liang et al. 2024.. We also compare our method with some previous methods aimed at novel view synthesis, including MeRF++(Zhange al.) 2020). Instant NGP (Müller et al., 2022). Mip-NeRF360 (Barron et al., 202T]. and 3DGS (Kerbl al. 2023). For quantitative comparisons, we report the results from their original papers, while the results of GaussianShader are from (Du et al. (024). For qualitative comparisons, we reproduce the results of TensolR and GS-IR using their released codes.
5.2 RESULTS AND COMPARISONS
TensolR Dataset. We first evaluate our approach against previous SOTA methods on the TensoIR dataset. Table I presents quantitative comparison. Our approach outperforms previous methods in both novel view synthesis and albedo estimation in terms of PSNR with the shortest training time, while maintaining comparable SSIM and LPIPS scores. This demonstrates the efficiency and effectiveness of our approach in material estimation and global illumination decomposition, In the relighting results, method ranks second only to TensoIR and surpasses all other methods. We also visualize the qualitative results in Fig. 4 which demonstrate that our rendering results are closer the ground truth overall color accuracy and have a smoother appearance occluded areas.
Mip-NeRF 360 Dataset. Table presents the quantitative results on the Mip-NeRF 360 dataset, and Fig. 5 showcases the rendering outcomes. Our method surpasses previous 3DGS-based inverse ren- dering approaches across all metrics, achieving superior performance. Notably, our method demon- rendering indoor scenes, where contain more occlusions. This highlights the superiority of our approach in modeling indirect lighting. Moreover, we extend our method from screen space to world space, as elaborated in Sec. [4.4] and refer to the extended ver- sion as Ours- Cubemap. While this extension incurs a slight decline in quantitative performance, the rendering results reveal more accurate and smoother modeling of occluded areas.
strates significant advantage
Indirect Lighting. To demonstrate the accuracy of our indirect lighting modeling, separately render the occlusion map and the image under indirect illumination, as shown in Fig. 6 GS-IR uses volumes to store occlusion and indirect lighting. However, during optimization, not all volumes are sampled, particularly those farther from the camera, resulting in missing occlusion and lighting in many areas of the scene. In contrast, our method leverages adaptive path tracing effectively
8

---

Published as a conference paper at ICLR 2025
Table 2: Quantitative results on the Mip-NeRF 360 dataset. Our GI-GS outperforms previous
inverse rendering methods and even surpasses some methods designed for novel view synthesis.
Outdoor
Indoor
PSNR t
SSIM f
LPIPS
PSNR +
SSIM f
LPIPS
NeRF
21,46
0.458
0.515
26.84
0.790
0.370
Instant NGP
22.90
0.566
0.371
29.15
0.880
0.216
Mip-NeRF
24.47
0.691
0.283
31.72
0.917
0.180
3DGS
24.64
0.731
0.234
30.41
0.920
0.189
GaussianShader
22.80
0.665
0.297
26.19
0.876
0.243
GS-IR
23.45
0.671
0.284
27.46
0.863
0.237
Ours
24.01
0.701
0.250
29.29
0.896
0.152
Ours-Cubeer Curs-Cubemap
23.81
0.695
0.251
29.07
0.892
0.161
GS-IR
Ours
Ours-Cubee Cubemap
GT
Figure 5: Qualitative comparison the Mip-NeRF 360 dataset. Our GI-GS effectively constructs high-frequency details and occluded areas in real-world scenes. Notably, the extended version, Curs-Cubemap, achieves smoother result in occluded areas. Cubemap
GS-IR
Figure 6: Qualitative comparison of indirect lighting. For indirect illumination, we increase the brightness linear space to make it easier to see the results. Our GI-GS provides more natural and smoother indirect lighting. Moreover, the indirect lighting of GI-GS covers most areas in the scene, especially when extending it to Curs-Cubemap, while the results of GS-IR have large areas missing.
Figure 7: Relighting visualization on the TensoIR dataset using different environment maps.
capture occlusion and indirect lighting across most areas. By rendering from multiple perspectives to capture world space information, extended accurate indirect lighting while maintaining multi-view consistency, especially for distant objects.
method achieves smoother and more
Relighting, Fig. visualizes the relighting results the TensoIR dataset under different light ditions. Our method achieves high-quality relighting results, which demonstrates the high accuracy of estimated materials and calculated indirect lighting. More results provided in the Appendix.

---

Published as a conference paper at ICLR 2025
Table 3: Ablation on occlusion and indirect lighting. By incorporating both occlusion and indirect
lighting. the rendering process more faithfully captures the light and surface interactions.
RensoIR
Mip-NeRF 360
PSNR t
SSIM f
LPIPS
PSNR t
SSIM f
LPIPS
Ours w/o occlusion Ours w/o indirect lighting Ours
36.37
0.970
0.041
26.24
0.788
0.210
35.60
0.965
0.047
25.89
0.778
0.212
36.75
0.972
0.037
26.35
0.788
0.206
Table 4: Ablation the number of sampled rays. Our training speed remains comparable with
GS-IR on the TensoIR dataset and is significantly faster on the Mip-NeRF 360 dataset.
TensoIR
Time (min)
Mip-NeRF 360
Time
PSNR
SSIM
LPIPS $
PSNR t
SSIM
LPIPS
(min)
GS-IR
35.33
0.974
0.039
33
25.38
0.757
0.268
Ours (N,- = 16)
36.25
0.966
0.047
27
26.19
0.773
0.222
55
Ours (N, 64)
36.75
0.972
0,037
28
26.35
0.788
0.206
57
Ours (N, -256
36.82
0.976
0.034
32
26.42
0.791
0.203
64
N, 16
N₂ 64
N₂ 256
Figure 8: Qualitative comparison of different numbers of ray samples. The accuracy of indirect lighting increases with the number of ray samples, leading smoother and less noisy results.
5.3 ABLATION STUDY
Effectiveness of Indirect Lighting. To evaluate the impact of our path tracing- rracingbbased occlusion and indirect lighting, we conduct two experiments: one considering only occlusion and the other consid- ering only indirect lighting. The quantitative results presented Table that both components significantly affect on the final rendering quality, with indirect lighting being particularly impactful.
Number of Sampled Rays. We also conduct experiment analyze the impact of the number of ray samples N.. The results in and Fig. F show that increasing the number of sampled rays N, enhances the rendering quality, especially when N, is small. Notably, thanks to efficient path tracing, increasing the number of ray samples only slightly slows down our training speed.
6 CONCLUSION
In this paper, we introduce GI-GS, a novel inverse rendering framework hased on 3D Gaussian Splatting. To extend 3DGS the inverse rendering problem, we augment each Gaussian's original attributes by incorporating surface normals and BRDFs, and optimize them through differentiable PBR pipeline. The key insight of our approach is combine deferred shading and path tracing en- able accurate global illumination decomposition while ensuring real-time rendering. Our approach successfully achieves high-fidelity geometry and material reconstruction of objects and scenes, as well effective global illumination decomposition. Extensive experiments demonstrate that GI- GS outperforms previous NeRe-based and 3DGS-based inverse rendering methods in novel view synthesis, achieving comparable relighting with superior computational efficiency.
Limitations. Our approach does not consider the specular component of indirect illumination, as it requires complex Monte Carlo sampling and is long-standing challenge in computer graphics. Moreover, for complex real-world scenes, using an environment map as the direct lighting source may fail to capture the spatially varying lighting in the environment. In addition, our approach relies on high-quality reconstruction the geometry, which suggests that the accuracy of path tracing can be improved by incorporating additional geometric constraints.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Jonathan T Barron and Jitendra Malik. Shape, illumination, and reflectance from shading, IEEE transactions on pattern analysis and machine intelligence, 3768) 16870-0-87.11000 2014.
Jonathan T Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Partin-Bruaall, and
Pratul P Srinivasan. Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields. In Proceedings of the IEEE/CVF international conference computer vision, pp. 5855-5864, 2021.
Jonathan T Barron, Ben Mildenhall, Dor Verbin, Pratul P Srinivasan, and Peter Hedman. Mip-nerf 360: Unbounded anti-aliased neural radiance fields. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 5470-5479 2022.
Jonathan T Barron, Ben Mildenhall, Dor Verbin, Pratul P Srinivasan, and Peter Hedman. Zip-nerf: Anti-aliased grid-based neural radiance fields. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pp. 19697-19705, 2023.
Harry Barrow, Tenenbaum, Hanson, and Riseman. Recovering intrinsic scene characteristics. Comput, vís. syst. 2(3-26):2, 1978.
Sai Bi, Zexiang Xu, Pratul Srinivasan, Ben Mildenhall, Kalyan Sunkavalli, Miloš Hašan, Yannick Hold-Geoffroy, David Kriegman, and Ravi Ramamoorthi. Neural reflectance fields for appearance acquisition. arXiv preprint arXiv:2008.. 200.0:0000....24. 2020a.
Sai Bi, Zexiang Xu, Pratul Srinivasan, Ben Mildenhall, Kalyan Sunkavalli, Miloš Hašan, Yannick Hold-Geoffre Geoffreorrry, David Kriegman, and Ravi Ramamoorthi. Neural reflectance fields for appearance acquisition. arXiv preprint arXiv:2008. 20Xv:20008.0884, 2020b.
Mark Boss, Raphael Braun, Varun Jampani, Jonathan Barron, Ce Liu, and Hendrik Lensch. Nerd: Neural reflectance decomposition from image collections. In Proceedings of the IEEE/CVF In- ternational Conference on Computer Vision, pp. 12684-12694, 2021.
Anpei Chen, Zexiang Xu, Andreas Geiger, Jingyi Yu, and Hao Su. Tensorf: Tensorial radiance fields. In European conference on computer vision, pp. 333-350. Springer, 2022.
Danpeng Chen, Hai Li. Weicai Ye, Yifan Wang, Weijian Xie, Shangjin Zhai, Nan Wang. Haomin Liu, Hujun Bao, and Guofeng Zhang. Pgsr: Planar-based gaussian splatting for efficient and high-fidelity surface reconstruction. arXiv preprint arXiv:2406.06521, 2024.
Robert L Cook and Kenneth E. Torrance. A reflectance model for computer graphics. ACM Trans- actions on Graphics (ToG), 1(1):7-24, 1982.
Pinxuan Dai, Jiamin Xu, Wenxiang Xie, Xinguo Liu, Huamin Wang, and Weiwei Xu. High-quality surface reconstruction using gaussian surfels. In ACM SIGGRAPH 2024 Conference Papers, pp. 2024.
Kang Du, Zhihao Liang, and Zeyu Wang. Gs-id: Illumination decomposition on gaussian splatting via diffusion prior and parametric light source optimization. arXiv preprint 0rXiv:2408.08524, 2024.
Ainaz Eftekhar, Alexander Sax, Jitendra Malik, and Amir Zamir. Omnidata: A scalable pipeline for making multi-task mid-level vision datasets from 3d scans. In Proceedings of the IEEE/CVE International Conference Computer Vision, pp. 10786-10796. 2021.
Sara Fridovich-Keil, Alex Yu, Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angjoo Kanazawa. Plenoxels: Radiance fields without neural networks. In Proceedings the IEEE/CVF conference on computer vision and pattern recognition, pp. 5501-5510, 2022.
Jian Gao, Chun Gu, Youtian Lin, Hao Zhu, Xun Cao, Li Zhang, and Yao Yao. Relightable 3d gaus- sian: Real-time point cloud relighting with brdf decomposition and ray tracing. arXiv preprint arXiv:2311.16044 2023.
11

---

Published as a conference paper at ICLR 2025
Antoine Guédon and Vincent Lepetit. Sugar: Surface-aligned gaussian splatting for efficient 3d mesh reconstruction and high-quality mesh rendering. In Proceedings of the IEEE/CVE Confer- on Computer Vision and Pattern Recognition, pp. 1354-5363, 2024.
Binbin Huang, Zehao Yu, Anpei Chen, Andreas Geiger, and Shenghua Gao. 2d gaussian splatting for geometrically accurate radiance fields. In ACM SIGGRAPH 2024 Conference Papers, pp- 1-11, 2024.
Yingwenqi Jiang, Jiadong Tu, Yuan Liu, Xifeng Gao, Xiaoxiao Long, Wenping Wang. and Yuexin Ma. Gaussianshader: 3d gaussian splatting with shading functions for reflective surfaces. In Pro- ceedings of the IEEF/CVF Conference on Computer Vision and Patlern Recognition, pp. 5322- 5332, 2024.
Haian Jin, Isabella Liu, Peijia Xu, Xiaoshuai Zhang, Songfang Han, Sai Bi, Xiaowei Zhou, Zex- iang Xu, and Hao Su. Tensoir: Tensorial inverse rendering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 165-174, 2023.
James T Kajiya. The rendering equation. In Proceedings of the 13th annual conference on Computer graphics and interactive techniques, pp. 143-150, 1986.
Brian Karis and Epic Games. Real shading in unreal engine 4. Proc. Physically Based Shading
Theory Practice, 1(3):1, 2013.
Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splat- ting for real-time radiance field rendering. ACM Trans. Graph., 12 2023.
Zhengfei Kuang, Yunzhi Zhang, Mong-Xing Yu, Samir Agarwala, Elliott Wu, Jiajun Wu, et al. Stanford-orb: a real-world 3d object inverse rendering benchmark, Advances in Neural Informa- fion Processing Systems, 36, 2024.
Zhengqin Li. Zexiang Xu, Ravi Ramamoorthi, Kalyan Sunkavalli, and Manmohan Chandraker. Learning to reconstruct shape and spatially-varying reflectance from a single image. ACM Trans- actions Graphics (TOG), 37(6):1-11, 2018.
Zhihao Liang, Qi Zhang, Ying Feng, Ying Shan, and Kui Jia. Gs-ir: 3d gaussian splatting for inverse rendering. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 21 21644-21653, 2024.
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes neural radiance fields for view synthesis. Communications of the ACM, 65(1):99-106, 2021.
Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics prim- itives with a multiresolution hash encoding. ACM transactions on graphics (TOG), (14: 2022.
Jacob Munkberg, Jon Hasselgren, Tianchang Shen, Jun Gao, Wenzheng Chen, Alex Evans, Thomas Müller, and Sanja Fidler. Extracting triangular 3d models, materials, and lighting from images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8280-8290, 2022.
Giljoo Nam, Joo Ho Lee, Diego Gutierrez, and Min H Kim. Practical svbrdf acquisition of 3d objects with unstructured flash photography. ACM Transactions on Graphics (ToG), 37(6) 2018.
Yahao Shi, Yanmin Wu, Chenming Wu, Xing Liu, Chen Zhao, Haocheng Feng, Jingtuo Liu, Liangjun Zhang, Jian Zhang. Bin Zhou, et al. Gir: 3d gaussian inverse rendering for relightable scene factorization. arXiv preprint arXiv:2312.05133. 2023.
Pratul P Srinivasan, Boyang Deng, Xiuming Zhang, Matthew Tancik, Ben Mildenhall, and Jonathan T Barron. Nerv: Neural reflectance and visibility fields for relighting and view synthe- sis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7495-7504, 2021.
12

---

Published as a conference paper at ICLR 2025
Cheng Sun, Min Sun, and Hwann-Tzong Chen. Direct voxel grid optimization: Super-fast conver- gence for radiance fields reconstruction. In Proceedings of the IEEE/CVF conference on computer viston and pattern recognition, pp. 5459-5469 2022.
Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang. Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. arXiv preprint arXiv:2106. 10689, 2021.
Tong Wu, Jia-Mu Sun, Yu-Kun Lai, Yuewen Ma, Leif Kobbelt, and Lin Gao. Deferredgs: Decoupled and editable gaussian splatting with deferred shading, arXiv preprint arXiv:2404.09412, 2024.
Rui Xia, Yue Dong, Pieter Peers, and Xin Tong. Recovering shape and spatially-varying surface flectance under unknown illumination. ACM Transactions on Graphics (ToG), 35(6):1-12, 2016.
Yao Yao, Jingyang Zhang, Jingbo Liu, Yihang Qu, Tian Fang. David McKinnon, Yanghai Tsin, and Long Quan. Neilf: Neural incident light field for physically-basee material estimation. In European Conference on Computer Vision, pp. 700-716. Springer, 2022.
Lior Yariv, Jiatao Gu, Yoni Kasten, and Yaron Lipman. Volume Volume of neural implicit surfaces. Advances in Neural Information Processing Systems, 34:4805-4815, 2021.
Zehao Yu, Torsten Sattler, and Andreas Geiger. Gaussian opacity fields: Efficient and compact
surface reconstruction in unbounded scenes. arXiv preprint arXiv:2404.10772. 2024.
Kai Zhang, Gernot Riegler, Noah Snavely, and Vladlen Koltun. Nerf++: Analyzing and improving
neural radiance fields. arXiv preprint arXiv:2010.07492, 2020.
Kai Zhang, Fujun Luan, Qianqian Wang. Kavita Bala, and Noah Snavely. Physg: Inverse rendering with spherical gaussians for physics-based material editing and relighting. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 5453-5462, 2021a.
Xiuming Zhang, Pratul P Srinivasan, Boyang Deng, Paul Debevec, William T Freeman, and Jonathan T Barron. Nerfactor: Neural factorization of shape and reflectance under an unknown illumination. ACM Transactions Graphics (ToG), 40(6):1-18, 2021b.
Yuanqing Zhang, Jiaminn Sun, Xingyi He, Huan Fu, Rongfei Jia, and Xiaowei Zhou. Modeling indi- rect illumination for inverse rendering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2022.
Matthias Zwicker, Hanspeter Pfister, Jeroen Van Baar, and Markus Gross. Ewa volume splatting. In Proceedings Visualization, 2001. VIS'O1., pp. 29-538. IEEE, 2001.
13

---

Published as a conference paper at ICLR 2025
A IMPLEMENTATION DETAILS
Training Details. The first stage of our training process follows the vanilla 3DGS with the addition of the normal loss term. The training iterations vanilla 3DGS. Besides, we use the improved densification strategy introduced in GOF (Yu et al, 2024) to reduce blurred areas. The optimization of the materials and the lighting takes 10,000 and 5,000 iterations on the Mip-NeRF 360 and TensoIR datasets, respectively. For the BRDF attributes, we adopt the continuous learning rate decay strategy in (ennoxes(Fridovich--ee et al. 2022), where the initial learning rate is set to 0.05 and decays to final learning rates of 0.005. All training is conducted on single NVIDIA A5000 GPU.
30K, with the same learning rate as the
Loss Functions. For the first stage, we initialize set of 3D Gaussians using the following loss function:
Linit L1 + LSSTM + Ln.
(16)
where
= Cn. + An-TV LVVamat
(17)
and CrV.sm is the total variation (TV) loss conditioned by the predicted normal map N and the given reference image I as follows:
LTY
AN
(18)
N
and
A exp (- 11. 1-11i))) (Ni, Ni-1,j²²
(19)
+ exp [T.j Iil- (Ni, Ni.-1)²
For the optimization of the materials and lighting, the decomposition loss La includes Ceolors CTV... and LrV.
La
L1 + AM DECuv
(20)
Lastraa
Light
where LiV.n is a smooth term similar to LiVVem Given the material map M, LrV... defined as:
1
LTV...
A
(21)
M
where
AM exp (- [1 1 (--ii) (Mi. M;-1.)²
(22)
+ eexp (- ||ا 11) (M.j - M,,-1)²
Besides, LTVi is used Eq. 20 obtain a smoother environment light, which defined as:
TVight
(23)
E
kj*
i.j
where
4 = (E,
E.-1,)) (Eij E
(24)
and E is learnable environment light map. Following GS-IR, we set An-TV, Am and Ag to 5.0, 1.0, and 0.01, respectively.
B TECHNICAL DETAILS
Pseudo Normal. We derive the pseudo normals from the depth map Given a point d(u, v) in the depth map, we consider the local 3 x 3 window of depth values centered on this point and project these 9 depth points into the correpsonding 3D points in the scene. By calculating the tangent vectors between these 3D points, tangent vectors. The final pseudo normal for the central pixel corresponding to (u, v) is computed the average of all these derived normals.
derive the normals through the cross-product of these
14

---

Published as a conference paper at ICLR 2025
Uniform Sampling. Since it is difficult to perform uniform sampling directly in solid angle space, we first consider the equivalent form of the original integral (see Eq. II in spherical coordinates as follows:
0(+)= - 1 -0=0 V(Ø, 0) cos(O) sin(O)dédo.
(25)
To estimate the integral, we uniformly generate sampling vectors spread inside the normal-oriented hemisphere by dividing each spherical coordinate into n1 and n2 samples to calculate
Ex 2j V (os, 0j) cos (0,) sin (0,)
O 1
(26)
Σ Ej= cos (8,) sin (f,)
Moreover, some previous inverse rendering works, such NeILF Yao al. 2022 use Fibonacci sampling over the hemisphere to generate uniform sampling points on the sphere, which effective and easy implement.
Adaptive Ray Tracing. Due to the perspective projection, the marching of a ray in 3D space is not linear to the marching of the ray in 2D) screen space. For areas of scene that far from the image plane, traversing sigmificant distance in 3D may only cover few pixels on the screen. This can lead to poor sampling accuracy, we are oversampling several pixels. To address this issue, adaptively adjust the step length based on the distance of the current ray-marching starting point. Considering a ray that originates from a surface point Zo (Io, yo. 2n) R3x1 with direction E 333x1 the proposed adaptive ray tracing is expressed as follows:
2o + wkt, k E.
(27)
where
20
t to
(28)
2Jar
Znear
Here, to is the pre-defined step length, 2far is the distance from the camera to the far clipping plane, and Znear the distance the near clipping plane. This adaptive approach enhances the accuracy of ray tracing by ensuring that step sizes are more appropriate for varying distances in the scene.
C ADDITIONAL RESULTS ON THE MIP-NERF 360 DATASET
We provide the quantitative results of novel view synthesis the Mip-NeRF 360 dataset for each scene, as detailed Tables 5 7 Our method outperforms all the previous 3DGS-ban based inverse ren- dering approaches and even some previous methods that focus novel view synthesis. Importantly, our method achieves SOTA performance in terms of PSNR among all 3DGS-based inverse render- ing approaches across all scenes, except for negligible decline in the garden scene. In addition, visualize the rendering results, decomposition results (normal, occlusion, indirect illumination), and relighting results Fig. 9 Notably, our calculated occlusion and indirect illumination model the shadow effect and the bounces of lights across the scene with high fidelity. We also provide more qualitative comparison results with the previous SOTA 3DGS-based baseline GS-IR in Fig. TO and
Fig.
Table 5: PSNR scores for Mip-NeRF360 scenes.
method
bicycle
flowers
garden
stump
trechill
room
counter
kitchen
bonsai
NeRF++
22.64
20.31
24.32
24.34
22.20
28.87
26.38
27.80
29.15
Plenoxels
21.91
20.10
23.49
20.66
22.25
27.59
23.62
23.42
24.67
INGP-Base
22.19
20.35
24.60
23.63
22.36
29.27
26.44
28.55
INGP-Big
22.17
20.65
25.07
23.47
22.37
29.69
26.69
29.48
30.69
Mip-NeRF 360
24.37
21.73
26.98
26.40
22.87
31.63
29.55
32.23
33.46
3DGS
25.25
21.52
27.41
26.55
22.49
30.63
28.70
30.32
31.98
GaussianShader
23.12
20.34
26.44
23.92
20.17
24.27
26.35
27.72
28.09
GS-IR
23.80
20.57
25.72
25.37
21.79
28.79
26.22
27.99
28.18
Ours
24.32
20,90
26.42
26.08
22.31
29.89
27.18
29.94
30.15
Ours Ours-Cubeman Cubemap
24.21
20.85
26.05
25.89
22.05
29.60
27.03
29.60
30.05
15

---

Published as a conference paper at ICLR 2025
Table 6: SSIM scores for Mip-NeRF360 scenes.
method
bicycle
flowers
garden
stump
treehill
room
counter
kitchen
bonsai
NcRF++
0.526
0.453
0.635
0.594
0.530
0.530
0.802
0.816
0.876
Plenoxels
0.496
0.431
0.606
0.523
0,509
0.842
0.759
0.648
0.814
INGP-Base
0.491
0.450
0.649
0.574
0.518
0.855
0.798
0.818
0.890
INGP-Big
0.512
0.486
0.701
0.594
0,542
0.871
0.817
0.858
0.906
Mip-NeRF 360
0.685
0.583
0.813
0.744
0.632
0.913
0.894
0.920
0.941
3DGS
0.771
0.605
0.868
0.775
0.638
0.914
0.905
0.922
0.938
GaussianShader
0.700
0.542
0.842
0.667
0.572
0.847
0.874
0.887
0.893
GS-IR
0.706
0.543
0.804
0.716
0.586
0.867
0.839
0.867
0.883
Ours
0.740
0.574
0.830
0.751
0,608
0,901
0.860
0.908
0.917
Ours-Culee Cuss-Cubemap
0.737
0.574
0.817
0.749
0.598
0.892
0.857
0.901
0.917
Table 7: LPIPS scores for Mip-NeRF360 scenes.
method
bicycle
flowers
garden
stump
trechill
counter
kitchen
bonsai
NcRF++
0.455
0.466
0.331
0.416
0.466
0.335
0.351
0,260
0.291
Plenoxels
0.506
0.521
0.386
0.503
0.540
0.419
0,441
0.447
0.398
INGP-Base
0.487
0.481
0.312
0.450
0.489
0.301
0.342
0.254
0.227
INGP-Big
0.446
0.441
0.257
0.421
0.450
0.261
0.306
0.195
0.205
Mip-NeRF 360
0.301
0.344
0.170
0.261
0.339
0.211
0.204
0.127
0.176
3DGS
0.205
0.336
0.103
0,210
0.317
0.220
0.204
0.129
0.205
GaussianShader
0.274
0.377
0.130
0.297
0.406
0.304
0.242
0.167
0.257
GS-IR
0.259
0.371
0.158
0.258
0.372
0.279
0.260
0.188
0.264
Ours
0.213
0.313
0.140
0.222
0.363
0.175
0.172
0.111
0.150
Ours-Cubentt Cubemap
0.213
0.310
0.149
0.224
0.360
0.191
0.179
0.120
0.152
Table 8: Per-scene results on the TensoIR synthetic dataset. For albedo reconstruction results, we
follow NeRFactor (Zhang et al. 20216b and scale each RGB channel by global scalar.
NVS
Albedo
Relighting
Scene
Method
PSNR
SSIM
LPIPS
PSNR
SSIM t LPIPS
PSNR t
SSIM t
LPIPS
3DGS
39.68
0.985
0.015
N/A
N/A
N/A
N/A
N/A
N/A
NeRFactor
26.08
0,881
0.151
25.44
0.937
0.112
23.25
0.865
0.156
InvRender
24.39
0.883
0.151
21.43
0.882
0.160
20.12
0.832
0.171
Lego
NVDiffrec
30.06
0.945
0.059
21.35
0.849
0.166
20.09
0.844
0.114
TensoIR
34.70
0.968
0.037
25.24
0.900
0.145
27.60
0.922
0.095
GS-IR
34.38
0.968
0.036
24.96
0.889
0.143
23.26
0.842
0.117
Ours
35.74
0.966
0.031
24.68
0.883
0.148
23.42
0.843
0.119
3DGS
39.99
0,986
0.017
N/A
N/A
N/A
N/A
N/A
N/A
NeRFactor
24.50
0,940
0.141
24.65
0,950
0.142
22.71
0.914
0.159
InvRender
31.83
0.952
0.089
27.03
0.950
0.094
27.63
0.928
0.089
Hotdog
NVDiffrec
34.90
0.972
0.054
26.06
0.920
0.116
19.07
0.885
0.118
TensolR
36.82
0.976
0.045
30.37
0.947
0.093
27.93
0.933
0.115
GS-IR
34.12
0.972
0.049
26.75
0.941
0.088
21.57
0.888
0.140
Ours
35.77
0.968
0.056
26.59
0.937
0.094
21.62
0.889
0.144
3DGS
46.50
0.991
0.023
N/A
N/A
N/A
N/A
N/A
N/A
NeRFactor
26.48
0,947
0.095
28.01
0.946
0.096
26.89
0.944
0.102
InvRender
31.12
0.968
0.057
35.57
0.959
0.076
27.81
0.949
0.069
Armadillo
NVDiffree
33.66
0.983
0.031
38.84
0.969
0.076
23.10
0.921
0.063
TensoIR
39.05
0.986
0.039
34.36
0.989
0.059
34,50
0.975
0.045
GS-IR
39.29
0.980
0.039
38.57
0.986
0.051
27.74
0.918
0.091
Ours
40.37
0,980
0.036
43.71
0.983
0.053
28.10
0.917
0.080
3DGS
41.11
0.994
0.007
N/A
N/A
N/A
N/A
N/A
N/A
NeRFactor
21.66
0.919
0.095
22.40
0.928
0.085
20.68
0.907
0.107
InvRender
22.13
0.934
0.057
25.33
0.942
0.072
20.33
0.895
0.073
Ficus
NVDiffrec
22.13
0.946
0.064
30.44
0.894
0.101
17.26
0.865
0.073
TensoIR
29.78
0.973
0.041
27.13
0.964
0.044
24.30
0.947
0.068
GS-IR
33.55
0.976
0.031
30.87
0.948
0.053
24.93
0.893
0.081
Ours
35.13
0.976
0.023
32.90
0.961
0.047
25.65
0.895
0.081
D ADDITIONAL RESULTS ON THE TENSOIR SYNTHETIC DATASET
We provide the quantitative results of novel view synthesis, albedo estimation, and relighting on the TensolR synthetic dataset for each object, as reported in Table 8. Our method outperforms all previ-
16

---

Published as a conference paper at ICLR 2025
Bicycle
Garden
Stump
Room
Kitchen
Counter
Bonsai
Normal
oo!
Rendering
Occlusion
Indirect
Relighting]
Relighting2
Figure 9: Visualization our inverse rendering, indirect lighting and relighting results on the Mip-
NeRF 360 dataset.
GS-IR
Ours
GT
Figure 10: More qualitative comparison results on the Mip-NeRF 360 dataset.
inverse rendering approaches in novel view synthesis. For albedo estimation and relighting, method surpasses the previous SOTA 3DGS-based baseline GS-IR. Besides, we visualize the ren- dering results, decomposition results (normal, albedo, roughness, occlusion, indirect illumination), and relighting results Fig. 12 Moreover, Fig. [3 provides the material editing results on the TensoIR dataset.
17

---

Published as a conference paper at ICLR 2025
GS-IR
Ours
GT
Figure 11: More qualitative comparison results on the Mip-NeRF 360 dataset.
18

---

Published as a conference paper at ICLR 2025
Normal
Rendering
Albedo
Roughness
Occlusion
Indirect
Relighting
Relighting?
Lego
Holdog
Armadillo
Ficus
Figure 12: Visualization of our inverse rendering, indirect lighting and relighting results on the TensoIR synthetic dataset.
Figure 13: Visualization of material editing on the TensoIR synthetic dataset.
19

---

Published as a conference paper at ICLR 2025
E ANALYSIS OF NORMAL ESTIMATION
In this section, explore the impact of normal estimation quality on relighting. The accuracy of normals crucial to the quality of PBR results. However, our method does not introduce additional supervision normals and relies only the geometry from the vanilla 3DGS. Previous experi- mental results show that inadequate normal estimation will greatly degrade the quality of relighting, because the split-sum approximation relies the normals determine lighting shading points. To verify this viewpoint, we use the relatively accurate normals predicted by the Omnidata model (Eftekhar al., 2021I supervise the rendered normals, which provides additional geometric pri- ors. As illustrated in Fig. [4] with the guidance of Omnidata model, the quality of normal estimation greatly improved. Consequently, the relighting results are also significantly enhanced, thanks to more accurate normal estimation, as shown in Fig. 15
Ours W/ Omnidata
Ours
Figure 14: Qualitative comparison of normal estimation before and after the supervision by the Omnidata model.
Ours Omnidata
Ours
Ours w/ Omnidata
Ours
Figure 15: Qualitative comparison of relighting before and after the supervision by the Omnidata model.
20

---

Published as a conference paper at ICLR 2025
F ANALYSIS OF GLOBAL ILLUMINATION
In this section, we evaluate the quality, controllability, and impact of global illumination on the final rendering result. We also add some implementation details of global illumination.
F.1 COMPARISON ON OCCLUSION
We compare the quality of estimated occlusion between our method and two baselines. As illustrated in Fig. 16 method accurately estimates occlusion at both object and scene levels. In contrast, GS-IR faces challenges in geometrically complex regions and struggles to accurately reconstruct the occlusion of distant objects at the scene level, often leading incorrect estimations of occlusion in certain areas. Although TensolR estimates high-quality occlusion, is limited to the object level and cannot be applied to scene-level situations. Besides, for unoccluded areas, TensoIR may produce some degree of occlusion. This is likely because the visibility of TensoIR is estimated by an MLP instead of querying whether the ray intersects the surface.
Ours
GS-IR
TensolR
Figure 16: Qualitative comparison of occlusion.
F.2 INDIRECT ILLUMINATION CALCULATION
introduce how calculate indirect illumina- tion from the first-pass rendering result. In this section, We will explain that this method of calculating indirect illumi- nation is feasible approximation with high efficiency. As illustrated Fig. 17 for surface point æ, the indirect light- ing (x, w) in direction is equal to the outgoing radiance Lo -w) of surface point & direction -w. Besides, RGB value [u,, v) corresponding to & in the first-pass ren- dering result is equal to the outgoing radiance Lo w1) in direction W1. Since the diffuse component has direction- ality, the difference between (ăt ww) and Lo (#, w1) is the specular component:
In Sec. 4.3
Figure 17: Indirect illumination cal- culation.
AT,, = L0, - -w) - Lo wi) (La( -w) + Lo -w)) (Lale, Lo wI))
(29)
L,(Ø, -w) Lg(ă, W1).
21

---

Published as a conference paper at ICLR 2025
In most practical scenarios, the specular component AL, is relatively negligible compared to the diffuse component. Therefore, we believe that substituting [u, v) to replace ( -w) serves as a practical upproximation that avoids the need for additional calculations. The visualization results in Fig. 19 also support this claim, where the tone and details of indirect illumination are satisfactory.
F.3 THICKNESS IN PATH TRACING
In this section, we provide a detailed explanation of the thickness parameter introduced in Eq. 12 Consider scenario where ray does not intersect with surface. If we compare the depth of the sampling point to the corresponding value zd in the depth map, a situation arises where 2d < 2. In this case, the ray would incorrectly be treated as intersecting with the surface, illustrated on the left side of Fig. T8
With the introduction of the thickness parameter 8, we refine this condition. As shown in the middle of Fig. 18 if = falls outside the range defined by za and zd + ō, the ray does not satisfy the criteria for intersection with the surface. Consequently, we conclude that the ray does not intersect with the surface. Only when the depth of the sampling point lies within the interval [zd, 2d + 6]. as depicted on the right side of Fig. 18 we consider the ray be intersecting with the surface.
Zd
Z
Z
d + 8
Z
Zd + Ö
Z
Z
Figure 18: Illustration of the determination of intersection.
F.4 CONTROLLABILITY OF GLOBAL ILLUMINATION
In this section, demonstrate the impact of different path tracing settings global illumination. As illustrated in Fig. [9] changing the path tracing parameters, such as the step size and the number of steps produces different global illumination results can be obtained. This controllability allows our method to accommodate a wide range of objects and scenes.
to 0.05 steps
tu 0.1
to 0.05
to 0.1
16 steps
8 steps
16 steps
Figure 19: Visualization of occlusion and indirect illumination under different settings.
22

---

Published as a conference paper at ICLR 2025
F.5 SHARP SHADOW UNDER NATURAL LIGHT SOURCE
The occlusion in our method is obtained by integrating the visibility function over the hemisphere. Therefore, for natural light sources with directionality, this method struggles to accurately approx- imate sharp shadows. To address this issue, we can employ shadow map, which is widely used in computer graphics, to obtaín accurate shadows. Specifically, by rendering a depth map at the loca- tion of the light source and comparing the depth of the shaded point relative to the light source and the value in the depth map, is possible to determine whether the shaded point shadow. In Fig. 20 present the result of relighting under a point light using the shadow map, from which sharp shadows can be observed.
Figure 20: Relighting under point light source with shadow map.
G LIGHTING DECOMPOSITION
In this section, we visualize the different components the rendering result. The decomposed dif- fuse and specular components are illustrated in Fig. 21 It can be seen that our method successfully decouples the diffuse and specular components. Besides, in the top of Fig. 22 we visualize the di- rect lighting during relighting. from which for the quality of direct lighting. Moreover, at the bottom of Fig. 22 we show the contribution of indirect lighting the rendering results, which is especially important for the correct rendering of occluded areas.
the importance of accurate normal estimation
H ABLATION STUDY ON INVERSE RENDERING PERFORMANCE
To validate the effectiveness of the proposed path tracing- based occlusion and indirect illumina- tion, we compare the performance of material estimation and relighting in three settings: with both occlusion and indirect illumination, with occlusion only, and with indirect illumination only. The quantitative results in Table 9 show that both occlusion and indirect illumination have significant impact on the performance of inverse rendering. In addition, we provide a qualitative comparison in Fig. 23 The results indicate that the absence of either occlusion or indirect illumination leads to degradation in reconstruction quality of occluded areas. Specifically, without the help of occlu- sion, the rendering of occluded areas tends to be be brighter than the actual scene, leading more shadows in the estimated albedo. Moreover, without the compensation from indirect illumination, occluded areas appear darker and bright spots emerge in the albedo.
23

---

Published as a conference paper at ICLR 2025
Rendering
Diffuse
Specular
Figure 21: Visualization of decomposed diffuse and specular components.
Ours w/ Omnidata
Ours
Rendering
Direct Lighting
Rendering
Direct Lighting
Figure 22: Top: Direct lighting during relighting. result and only direct lighting.
Bottom: Comparison between the rendering
I MORE RESULTS
In this section, we provide additional experiment results on the Stanford-ORB (Kuang et al. and Synthetic4Relight ZZhang al. 2022) datasets.
2024)
I.1 RESULTS ON THE SYNTHETIC4RELIGHT DATASET
To further compare the novel-view synthesis, material estimation, and relighting performance, we conduct experiments on the Synthetic4Relight dataset (Zhang et al.) (022). Table T0 presents guantitative comparison. Our approach outperforms previous methods both novel view synthe-
sis and relighting, achieving higher PSNR and lower LPIPS scores while maintaining comparable SSIM. In the material estimation results, our method is only slightly inferior to InvRender (Zhang
ett al. 2022) and surpasses all other basclines. We also provide qualitative results in Fig. 24 Fig.
25 and Fig. 26 which demonstrate that our method achieves more accurate material estimation and relighting results than the previous GS-based inverse rendering method.
24

---

Published as a conference paper at ICLR 2025
Table 9: Ablation on material estimation and relighting.
Albedo
Relighting
PSNR
SSIM
LPIPS
PSNR
SSIM t LPIPS
Ours w/o occlusion
31.39
0.928
0.093
24.38
0.879
0.111
Ours w/o indirct lighting Ours
31.46
0.929
0.089
24.30
0.877
0.113
31.97
0.939
0.085
24.70
0.886
0.106
w/o occlusion
w/o ind. lighting
Ours
GT
Figure 23: Top: Comparison of rendering results on the Mip-Nerf 360 dataset. Bottom: Comparison of albedo estimation on the TensolR dataset.
1.2 RESULTS ON THE STANFORD-ORB DATASET
To further evaluate the inverse rendering performance on real-world data, we conduct comparison with GS-IR (Liang et al. 2024) on four scenes from the Stanford-ORd dataset. Table [1 presents the quantitative results. Our method outperforms GS-IR in novel-view synthesis, material estimation and relighting. Compared with GS-IR, our method estimates occlusion more accurately, thereby achieving more effective decoupling of lighting and materials. Therefore, the estimated albedo contains fewer shadow and highlight components, resulting in better relighting quality. This is also supported by the qualitative comparisons in Fig. 27 and Fig. 28
We also visualize the relighting
results under real-world environment map in Fig. 29
J ANALYSIS ON RELIGHTING
In this section, we analyze the impact of different settings on relighting results. is well known that the accuracy of material estimation, especially the accuracy of albedo. is crucial the quality of the relighting. Therefore, applying different transformations to the albedo will produce different re- lighting results. We find that some previous methods, such as TensoIR, will rescale the albedo using the ground truth in the dataset and normalize the energy of the environment map before relighting, then perform color gamut correction on the relighting results. Therefore, we conduct experiments on relighting under this setting and use Monte Carlo sampling to calculate the rendering equation. As illustrated in Fig. 30 compared to our original relighting results, using the settings in TensoIR can significantly improve the relighting performance.
25

---

Published as a conference paper at ICLR 2025
Table 10: Quantitative results on the Synthetic4Relight dataset.
NVS
Albedo
Relighting
Roughness
PSNR 1
SSIM
LPIPS
PSNR
SSIM
LPIPS
PSNR
SSIM
LPIPS 4
MSE J
NeRFactor
22.80
0.917
0.151
22.96
0.906
0.162
21.54
0.875
0.171
PhySG"
23.42
0.987
0.068
21.80
0.973
0.185
22.63
0,973
0.073
0.268
Inv InvRende**
26.19
0.991
0.044
25.25
0.983
0.058
25.59
0.984
0.041
0.072
GS IR
34.79
0.973
0.042
23.20
0.918
0.091
26.30
0.945
0.071
0.214
Ours
35.42
0.973
0.042
24.68
0.931
0.085
27.36
0.945
0.070
0.128
Table 11: Quantitative results on the Stanford- dataset.
Scene
Method
NVS
Albedo
Relighting
PSNR
SSIM
LPIPS
PSNR t
SSIM
LPIPS +
PSNR
SSIM
LPIPS J
Gnome
GS-IR
37.73
0.984
0.023
22.57
0.886
0.101
24.83
0.905
0.069
Ours
36.99
0.984
0.026
26.54
0.920
0.085
30.79
0.947
0.052
Car
GS IR
36,61
0.990
0.010
26.21
0.958
0.033
24.91
0.959
0,031
Ours
37.16
0,992
0.009
30.13
0.966
0.027
25.39
0.961
0.028
Cactus
GS IR
35.91
0,987
0.016
32.81
0.962
0.038
30.42
0.974
0.018
Ours
37.52
0,990
0.013
34.70
0.968
0.035
34.54
0.981
0.020
Teapot
GS-IR
35.10
0.987
0.013
28.46
0.956
0.037
21.07
0.962
0.024
Ours
35.71
0.989
0.011
30.76
0.965
0.031
23.45
0.970
0.021
Avg.
GS-IR
36.34
0.987
0.015
27.51
0.940
0.052
25.31
0.950
0.036
Ours
36.85
0,989
0.015
30.53
0.955
0.045
28.54
0.965
0.030
Rendering
Albedo
Roughness
Relighting
Occlusion
GS-IR
Ours
GT
Figure 24: Qualitative comparison on Chair from the Synthetic4Relight dataset.
26

---

Published as a conference paper at ICLR 2025
Rendering
Albedo
Roughness
Relighting
Occlusion
GS-IR
Ours
GT
Figure 25: Qualitative comparison on Balloon from the Synthetic4Relight dataset.
Rendering
Albedo
Roughness
Relighting
Occlusion
CSAR
Ours
GT
Figure 26: Qualitative comparison on Jugs from the Synthetic4Relight dataset.
27

---

Published as a conference paper at ICLR 2025
Rendering
Albedo
Occlusion
Relighting
Relighting2
Relighting3
GS-IR
Ours
GT
N/A
Figure 27: Qualitative comparison on Gnome from the Stanford-ORB dataset. The asterisk indicates that the GT relighting result is available in the original dataset but obtained by relighting using the albedo in the original dataset and the remaining material properties and normals estimated by the proposed method.
Rendering
Albedo
Occlusion
Relighting
Relighting2
Relighting3
GS-IR
Ours
GT
N/A
Figure 28: Qualitative comparison on Cactus from the Stanford- -ORo dataset. The asterisk indicates is available in the original dataset but obtained by relighting using the albedo in the original dataset and the remaining material properties and normals estimated by the proposed method.
that the GT relighting result
28

---

Published as a conference paper at ICLR 2025
GS-IR
Ours
GT
GT with background
Rendering
Relighting]
Relighting2
Rendering
Relighting
Relighting
Figure 29: Qualitative comparison on relighting with real-world environment map.
Env map
Ours
Ours w/ Rescale
GT
Figure 30: Qualitative comparison on relighting with different settings.
29

---

Published as a conference paper at ICLR 2025
K GROUND TRUTH RELIGHTING RESULTS FROM THE TENSOIR DATASET
Fig. 31 provides the ground truth relighting results from the TensoIR dataset. The results demon- strate that the environment map used in Relighting contains yellow light sources and dim blue ambient light, so the result of relighting is dominated by yellow tones. In contrast, when the envi- ronment map used is filled with bright white light sources and blue ambient light like in Relighting 2, the relighting result will have a cool tone.
Rendering
Albedo
Relightingl
Relighting2
Figure 31: Ground truth relighting results from the TensolR dataset.
30


Reviews:
[{"review_id": "8Sb3uUQuKA", "reviewer": "Reviewer_zyyD", "summary": "The paper present a novel inverse rendering framework, GI-GS, to achieve scene decomposition and photo-realistic relighting.\n\nExisting 3DGS-based methods, due to their nature of non-deterministic geometry, encounter challenges in modeling inter-reflection and self-occlusion, which prevent them from occlusion-aware/indirect-light-aware decomposition or relighting. While recent works like relightable 3DGS use ray tracing to estimate occlusion, preventing shadow from being baked into materials, many of them are still unable to process inter-reflection, or only employ a learnable residual term to model indirect light.\n\nTo address this, the paper propose a novel framework with deferred rendering technique, which employs path tracing to achieve indirect illumination computation. Experimental results are provided to demonstrate the effectiveness of the framework, highlighting the promising potential of deferred rendering in efficiently modeling indirect geometry-lighting interactions. In object-level cases, the proposed method outperforms several baselines in both novel view synthesis and decomposition, while achieving comparable performance in relighting. In scene-level cases, the method also demonstrates meaningful results in both decomposition and relighting.", "questions": "1. The occlusion map and indirect light map in Figs. 6 and 8 are quite impressive. However, I am curious as to why Fig. 4 shows that the proposed method still retains a significant amount of shadow in the albedo, particularly in the case of the Hotdog.\n\n2. What does the direct light look like? The incorrect direct lighting reconstruction also lead to poor relighting performance.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The paper propose a novel framework where the deferred rendering technique enables light-weight path tracing on the deferred rendering buffer, which highlights the promising potential of deferred rendering in efficiently modeling indirect geometry-lighting interactions.\n2. The method is extended to scene level and successfully produces meaningful decomposition and relighting results.\n3. In novel view synthesis tasks, the method achieves best performance over inverse rendering baselines.", "weaknesses": "1. The novel indirect illumination module may require further experimentation to validate its effectiveness. Theoretically, this module has the potential to achieve superior relighting performance compared to solutions that rely on learnable parameters for modeling indirect light, as the fixed inter-reflection in those methods is often inaccurate when relighted under another illumination.\n\nHowever, the quantitative results presented in Table 1 indicate that the relighting performance is subpar, even when achieving best albedo estimation, which undermines the strength of the paper's key claim. I believe the relighting performance is likely impacted by the inadequate normal estimation, as illustrated in Fig. 9 (Garden, Kitchen, Counter, and Bonsai), rather than suggesting that path tracing is ineffective, especially since the impressive qualitative results in Fig. 6 indicate otherwise.\n\nTherefore, to quantitatively validate the effectiveness of the method, it would be beneficial to conduct an ablation study focused on albedo estimation or relighting. The current ablation study in Table 3 only examines performance changes related to novel view synthesis quality, which is insufficient to support the claims made in the paper.\n\n2. In the absence of quantitative results, the qualitative findings suggest that normal estimation may be inadequate, as demonstrated in Fig. 9 (Garden, Kitchen, Counter, and Bonsai). This deficiency could negatively impact the overall performance of inverse rendering.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SnhVNKo1hY", "reviewer": "Reviewer_tAKv", "summary": "This works introduces GI-GS, a novel gaussian-splatting based inverse rendering pipeline to extract intrinsics from multi-view images. As a follow-up of existing GS-based inverse rendering,  GS-IR, this paper focuses on taking the indirect illumination into account, to solve more accurate material and environment map from photorealistic images.", "questions": "See Weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The paper is intuitive and easy to follow.\n\nIntegrating indirect illumination calculation in a GS-based inverse rendering pipeline is a challenging movement.\n\nIn the experiments, the model shows comparable results in the task of novel view synthesis, decent performance in relighting, outperforming the GS-based baseline.", "weaknesses": "The way of processing indirect illumination in the pipeline is similar to InvRender: It uses the direct illumination (represented as a radiance function) of 3D points to calculate the indirect illumination. How is the indirect illumination calculation in this paper distincted from the one in InvRender? \n\nTab. 1: TensoIR performs much better than the proposed method in relighting task. and in figure 4 it is hard to see the differences between GS-IR and proposed method. This harms the soundness of the method. Please add more explanation of the performance.\n\nThe usage of I_{dir}(u, v) in the eq. 13 might be inaccurate, since the image RGB value is the combination of diffuse and specular lighting, which is not equal to the illumination from x_hat to x. One way to avoid this to re-apply the first rendering pass with new look-at direction. Is there any specific reason for directly using the RGB values? Please clarify.\n\nWhile the main focus of the work is on decomposition and relighting, no relighting evaluation for real-world inputs is shown. This challenges the effectiveness of the method in more realistic and complex cases. An experiment conducted on real-world benchmarks (such as Stanford-ORB) can be very helpful to support the claims of the paper.\n\nOther minor issues.\nAdaptive Path Tracing happens in the projected camera only, which is not able to take the occluded objects into account.\nEq. (8): $\\mathcal_{L}_{n-p}$ looks confusing (might be interpreted as n minus p), $\\mathcal_{L}_{n,p}$ might be better.\nTab. 1: Only relighting methods are compared in the NVS column. Please consider adding reconstruction methods as reference.\n\nBased on the above points, I’m on the fence but slightly leaning to reject this paper. I’ll carefully look at the authors’ responses and happy to change my score if the concerns are addressed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EWxNeHLGAs", "reviewer": "Reviewer_SvCz", "summary": "This paper propose a efficient path tracing techinque that can calculate indirect lighting. They first render a G-buffer to capture the detailed geometry and material properties of the scene. Then, they perform physically-based rendering only for direct lighting. With the G-buffer and previous rendering results, the indirect lighting can be calcuated through the proposed path tracing. The training is decomposed into three stages: 1. vanilla 3dgs, 2.deferred rendering with split-sum approximation with occlusion calculated from the proposed path tracing, 3.add indirect lighting term on top of stage2.", "questions": "see weakness.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The paper propose a novel path tracing strategy that can query visibility in screen space.", "weaknesses": "1. in line303-305, I think authors can refer to the uniform sampling in hemi-sphere used by NeILF, which should be better than  sampling in spherical coordinate\n2. The authors argure that they are the frist to incorperate indrect illuminance in relighting, however, the proposed lighting model can not model the indirect lighting in specular term, also, the approximation in Equation7 makes it impossible to model sharp shadows as visbility is considered in the integration of direct lighting.\n3. The path tracing part is not well introduced. I can't figure out how exactly is the procedure of sampling point on ray. I guess the authors mean they sample many points step by step until the depth of a point is smaller than the rendered depth.\n4. The calculation of visibility by comparing the depth of sampled point on the ray with the rendered depth is not comprehensive. There is another condition: the ray (start from the surface) is not occluded by the object, but the sampled points on the ray has a bigger depth value than the rendered depth map. For example, a ray that has a consistent direction as the orientation of the camera, then whether it is occluded is not determined by the rendered depth.\n5. I'm also confused about the calculation of indirect light, in line338-341. Is the outgoing radiation indicates $L_p(\\hat{x},-\\omega_)$? If so, why can it be obtained by indexing the RGB map. And in some conditon, indirect light of a surface point can not be seen by the camera. Equation13 is also confusing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6UsFvYTtjL", "reviewer": "Reviewer_uXjo", "summary": "This work introduces a novel inverse rendering framework designed to achieve photo-realistic novel view synthesis and relighting by leveraging 3D Gaussian Splatting (3DGS) and deferred shading. Inverse rendering aims to accurately model scene materials, geometry, and lighting from images, a task complicated by the need to account for indirect lighting, which prior methods often model poorly. GI-GS improves upon these methods by incorporating efficient path tracing and deferred shading to compute indirect lighting, which leads to better handling of complex light-object interactions. The framework first uses a G-buffer to capture scene details and perform physically-based rendering (PBR) for direct lighting. Then, it applies lightweight path tracing to compute indirect lighting based on previous results, enabling more realistic global illumination. The authors demonstrate that GI-GS outperforms previous techniques in rendering quality and efficiency, making it a state-of-the-art method for novel view synthesis and relighting.", "questions": "1. How does the method validate the effectiveness of its decomposed diffuse and specular components, given that these are crucial for its lighting model but are not evaluated?\n\n2. Given the importance of material properties for accurate reflection, why is there no analysis of the decomposed material map components (albedo, specular, roughness), especially since benchmarked methods include accessible material evaluations?\n\n3. How does the paper substantiate its claim that calculated occlusion effectively models shadowing, considering that benchmarked methods provide visibility and occlusion data but no strong evaluation is presented for this feature?\n\n4. Without ground truth for indirect lighting, how can the accuracy of modeled interreflection and bounce lighting be verified, and might these effects reflect residuals of reflection optimization?\n\n5. In what ways does the proposed method’s relighting evaluation demonstrate superiority over existing methods, especially given that occlusion and indirect lighting calculations are also present in baseline methods?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The paper is well-written and easy to follow, and it includes comprehensive relighting works from the past.\n\n1. GI-GS uses path tracing with deferred shading to accurately model indirect lighting, achieving realistic relighting and view synthesis.\n\n2. By combining 3D Gaussian Splatting with a physically-based pipeline, GI-GS efficiently reconstructs detailed geometry and materials for high-quality results.\n\n3. The framework decouples direct and indirect illumination, enabling good-quality relighting across various conditions.", "weaknesses": "1. This method leverages image-based lighting (IBL) to simulate illumination from multiple directions, addressing both diffuse and specular reflections as outlined in Section 4.2. However, the decomposed diffuse and specular components are not evaluated, even in the supplementary material, which I believe is crucial for your lighting model.\n\n2. The paper benchmarks against the TensoIR, Gaussian Shader and GS-IR, where these methods, including the relightable 3D Gaussian, offer accessible material evaluations. While the proposed method includes a material map (albedo, specular, roughness, etc.) in its pipeline, there should be an analysis or evaluation of the decomposed material, as material properties are central to reflection.\n\n3. The paper claims that the calculated occlusion models shadowing, a key contribution similar to that of benchmarked methods. However, the experiments do not include a strong evaluation of this feature, despite visibility and occlusion information being available in the benchmarked methods.\n\n4. Additionally, for indirect lighting, which accounts for bouncing and interreflection, there is no ground truth to verify the accuracy of the modeled indirect illumination. This could also be interpreted as the residual of your reflection during optimization.\n\nI have to challenge the novelty and quality of the proposed method, as both occlusion and indirect lighting are also addressed and calculated similarly in these baseline methods and relighting evaluation didn't prove the proposed method achieved better quality than the previous method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)