GI-GS: Global Illumination Decomposition on Gaussian Splatting for Inverse Rendering

## Abstract

Abstract We present GI-GS , a novel inverse rendering framework that leverages 3D Gaussian Splatting (3DGS) and deferred shading to achieve photo-realistic novel view synthesis and relighting. In inverse rendering, accurately modeling the shading processes of objects is essential for achieving high-fidelity results. Therefore, it is critical to incorporate global illumination to account for indirect lighting that reaches an object after multiple bounces across the scene. Previous 3DGS-based methods have attempted to model indirect lighting by characterizing indirect illumination as learnable lighting volumes or additional attributes of each Gaussian, while using baked occlusion to represent shadow effects. These methods, however, fail to accurately model the complex physical interactions between light and objects, making it impossible to construct realistic indirect illumination during relighting. To address this limitation, we propose to calculate indirect lighting using efficient path tracing with deferred shading. In our framework, we first render a G-buffer to capture the detailed geometry and material properties of the scene. Then, we perform physically-based rendering (PBR) only for direct lighting. With the G-buffer and previous rendering results, the indirect lighting can be calculated through a lightweight path tracing. Our method effectively models indirect lighting under any given lighting conditions, thereby achieving better novel view synthesis and relighting. Quantitative and qualitative results show that our GI-GS outperforms existing baselines in both rendering quality and efficiency. Project page: https://stopaimme.github.io/GI-GS/ .

## 1 Introduction

Inverse rendering (Barrow et al., ) aims to estimate the physical attributes of a 3D scene, such as material properties, geometry, and lighting, from captured images. This task is crucial for various downstream applications, including novel view synthesis, relighting, and VR/AR. Similar to many inverse problems, inverse rendering is often considered ill-posed due to unknown lighting conditions and the intractability of the integrals in the rendering equation. Recent research has adopted the Neural Radiance Fields (NeRF) (Mildenhall et al., ) paradigm, which represents scenes as neural fields using multi-layer perceptrons (MLPs). Nonetheless, rendering an image with NeRF-like methods requires dense querying of MLPs along ray directions, which significantly affects the rendering speed. Moreover, the limited representational capacity of MLPs hinders their ability to faithfully model physical attributes of scenes and complex interactions between light and surfaces.

More recently, 3D Gaussian Splatting (3DGS) (Kerbl et al., ) has gained significant attention for its superior reconstruction quality and rendering speed. This method explicitly represents scenes using a set of 3D Gaussians and employs a tile-based rasterization pipeline for efficient rendering. Unlike NeRF, which utilizes MLPs to encode view-dependent colors across the scene, 3DGS assigns spherical harmonics to each Gaussian point to capture these view-dependent effects. Nevertheless, 3D Gaussians trained under specific lighting conditions cannot be used to render relighted results under different lighting setups, as the view-dependent colors entangle the objects’ materials with the training lighting conditions. This constraint restricts the broader applicability of 3DGS. To address this limitation, recent studies (Jiang et al., ; Shi et al., ; Liang et al., ; Gao et al., ; Wu et al., ) have proposed several inverse rendering frameworks based on 3DGS. These methods successfully decouple color into the interactions between lighting and surface materials by incorporating physically-based rendering (PBR) techniques. Some of them have also attempted to model indirect lighting, but they typically store the indirect lighting either statically in a 3D Gaussian or in additional volumes and optimize it during training. These approaches overlook the dynamic nature of light as it continually bounces between objects and, therefore, cannot accurately model indirect lighting during the relighting process.

To overcome these limitations, we propose *GI-GS*, a novel inverse rendering framework based on 3DGS that achieves accurate estimation of materials and geometry, as well as global illumination decomposition. Our key innovation is to calculate indirect lighting by efficient path tracing based on the deferred shading technique. Specifically, as shown in Fig. 1, we first rasterize the geometry and materials of the scene, storing the results in a G-buffer. Then, we incorporate a PBR pipeline to efficiently compute the complex rendering equations and thoroughly decompose the illumination. We employ a learnable environment map to approximate direct lighting. For indirect lighting and occlusion, instead of modeling them as attributes of 3D Gaussian points or using baked volumes as in previous methods, we calculate them from the first-pass rendering results through path tracing. This approach ensures that our indirect lighting is more consistent with the rendering equation. Consequently, our approach can reconstruct high-fidelity geometry and materials of a complex real scene under unknown natural illumination, thereby achieving state-of-the-art rendering of novel view synthesis and enabling additional applications like relighting. Notably, to the best of our knowledge, we are among the first to develop a 3DGS-based inverse rendering framework capable of modeling indirect lighting during the relighting process.

In summary, our main contributions are as follows:

- •
We present GI-GS, a novel inverse rendering framework based on 3DGS that achieves state-of-the-art intrinsic decomposition and rendering results for both objects and scenes.
- •
We propose an efficient tile-based path tracing method based on deferred shading, which enables accurate occlusion and indirect lighting calculation from previously rendered results stored in the G-buffer.
- •
We extend our path tracing-based indirect lighting from screen space to world space for scene-level datasets, achieving more natural occlusion and indirect lighting.

Figure: Figure 1: Overview of GI-GS. GI-GS takes input a set of pretrianed 3D Gaussians, each with a normal attribute. It first rasterizes the scene geometry and materials into a G-buffer. Next, it incorporates a differentiable PBR pipeline to obtain the rendering result under direct lighting and performs path tracing to model the occlusion. Finally, it employs differentiable ray tracing to calculate indirect lighting from the scene geometry and the previous rendering result. The final rendered image is a fusion of the first-pass and second-pass results and uses the ground truth image for supervision.
Refer to caption: x1.png

## 2 Related Works

### 2.1 Neural Rendering and Radiance Fields

Neural rendering, exemplified by NeRF (Mildenhall et al., ), has received widespread attention in recent years due to its remarkable representational capabilities and support for end-to-end optimization. NeRF implicitly represents a scene as a continuous radiance field using MLPs that take the 3D positions and view directions as inputs to output the density and view-dependent color. However, the rendering process of NeRF requires dense importance sampling along rays, which involves repeated MLP queries and consequently limits rendering speed. To address this issue, subsequent research has introduced efficient explicit data structures, such as hash grids (Müller et al., ), voxels (Sun et al., ; Fridovich-Keil et al., ), and tri-planes (Chen et al., ), to encode features in 3D space, thereby accelerating both training and inference phases. In addition, some works (Barron et al., ; ; ) have focused on enhancing the rendering quality of NeRF, while others (Wang et al., ; Yariv et al., ) have utilized radiance fields to represent signed distance functions (SDFs) for surface reconstruction.

Recently, 3DGS (Kerbl et al., ) replaces MLPs with a set of 3D Gaussian primitives to explicitly represent the neural field and incorporates a tile-based rasterization pipeline to achieve high-quality real-time rendering. Since 3DGS is originally designed for novel view synthesis and lacks relevant geometric constraints or priors, how to reconstruct accurate geometry from 3DGS remains an important problem. Subsequent methods have significantly improved the quality of the reconstructed geometry by integrating geometrically relevant regularization terms (Guédon & Lepetit, ) or employing more geometrically accurate representations (Huang et al., ; Dai et al., ).
In our work, we extend 3DGS to the inverse rendering task by combining it with PBR, thereby broadening its applicability to a wider range of downstream tasks.

### 2.2 Inverse Rendering

Recovering geometry, materials, and lighting from captured images has been a long-standing challenge due to the unknown lighting conditions and the complex interactions between light and objects. Most previous inverse rendering methods simplify the problem by introducing prior information. For instance, Barron & Malik (); Li et al. () introduce plane assumptions to simplify the acquisition of normals, Bi et al. (); Nam et al. () use a flashlight to align the light source with the camera positions, and Xia et al. () capture a rotating object under fixed camera and lighting conditions.

Inspired by the success of volume rendering techniques based on radiance fields, subsequent works (Bi et al., ; Srinivasan et al., ; Boss et al., ; Zhang et al., ; Jin et al., ; Zhang et al., ; ; Yao et al., ) have attempted to apply the NeRF paradigm to address this issue by leveraging the powerful representational capabilities of neural networks. For instance, Neural Reflectance Fields (Bi et al., ) is the first work to introduce the bidirectional reflectance distribution function (BRDF) and lighting in NeRF, which represents a scene as a reflectance field under a point light source. NeRV (Srinivasan et al., ) uses an environment map to represent the given light condition and trains an MLP to approximate the visibility of a given point in space from different directions, while PhySG (Zhang et al., ) models lighting as spherical Gaussians and incorporates SDFs for more accurate geometry reconstruction. Besides, NeRFactor (Zhang et al., ) adopts a multi-stage training to reduce the difficulty of optimization. TensoIR (Jin et al., ) employs a compact tri-plane representation for ray tracing-based indirect lighting calculations. Moreover, Neilf (Yao et al., ) utilizes a neural incident light field to better model the light source.

Recently, some works (Jiang et al., ; Wu et al., ) have extended 3DGS to the inverse rendering task by binding BRDFs as attributes of Gaussians. Besides, GS-IR (Liang et al., ) leverages baking-based volumes to store occlusion and indirect illumination. Relightable 3D Gaussian (Gao et al., ) bakes the occlusion by using point-based ray tracing and parameterizes the indirect lighting as additional spherical harmonics for each Gaussian. Nevertheless, both methods rely on baking to accelerate rendering and cannot model indirect lighting during relighting. In contrast, we propose a 3DGS-based framework for inverse rendering and leverage deferred shading to realize efficient path tracing-based occlusion and indirect illumination for both rendering and relighting.

## 3 Preliminaries

### 3.1 3D Gaussian Splatting

3DGS (Kerbl et al., ) explicitly represents a scene as a set of 3D Gaussian primitives. Each primitive is parameterized by a Gaussian distribution as:

$$ $G({\bm{x}})=e^{-\frac{1}{2}(\bm{x}-\bm{\mu})^{T}\bm{\Sigma}^{-1}(\bm{x}-\bm{ \mu})},$ (1) $$

where $\bm{\mu}\in\mathbb{R}^{3}$ denotes the mean vector and $\bm{\Sigma}\in\mathbb{R}^{3\times 3}$ denotes the covariance matrix. Besides, each 3D Gaussian primitive is assigned a set of spherical harmonics to model the view-dependent color and opacity $\alpha$ for volume rendering. During the rendering process, the 3D Gaussians are projected onto the 2D image plane through EWA splatting (Zwicker et al., ). Specifically, given the view transformation matrix $\bm{W}$ and the Jacobian of the affine approximation of the projective transformation $\bm{J}$, the covariance matrix $\bm{\Sigma}^{\prime}$ in camera coordinates is computed as $\bm{\Sigma}^{\prime}=\bm{J}\bm{W}\bm{\Sigma}\bm{W^{T}}\bm{J^{T}}$.

For each screen pixel, its color is obtained by $\alpha$-blending 2D Gaussians sorted by depth as:

$$ $C=\sum_{i\in\mathcal{N}}T_{i}c_{i}\alpha_{i}\text{ with }T_{i}=\prod_{j=1}^{i- 1}\left(1-\alpha_{j}\right),$ (2) $$

where $\mathcal{N}$ denotes the set of Gaussians, $\alpha_{i}$ is the transmittance of each Gaussian derived from its opacity and covariance matrix, and $T_{i}$ is the accumulated transmittance.

### 3.2 Physically-based Rendering

PBR is a rendering approach that models the color of an object as the interaction of light and surface material. Given a surface point ${\bm{x}}$ and its normal vector ${\bm{n}}$, the outgoing light $L_{o}$ in the camera view direction $\bm{\omega}_{o}$ can be calculated using the following rendering equation (Kajiya, ):

$$ $L_{o}\left(\bm{\omega}_{o},{\bm{x}}\right)=\int_{\Omega}f_{r}\left(\bm{\omega} _{o},\bm{\omega}_{i},{\bm{x}}\right)L_{i}\left(\bm{\omega}_{i},{\bm{x}}\right) \left(\bm{\omega}_{i}\cdot{\bm{n}}\right)d\bm{\omega}_{i},$ (3) $$

where $\Omega$ is the upper hemisphere centered at ${\bm{x}}$ and $f_{r}\left(\bm{\omega}_{o},\bm{\omega}_{i},{\bm{x}}\right)$ is the bidirectional reflectance distribution function (BRDF), which characterizes the relationship between the outgoing light $L_{o}$ and the incident light $L_{i}$ coming from direction $\bm{\omega}_{i}$ based on the material of the object. The BRDF can be divided into a diffuse component $f_{d}$ and a specular component $f_{s}$ according to the Cook-Torrance BRDF model (Cook & Torrance, ) as follows:

$$ $f_{r}(\bm{\omega}_{i},\bm{\omega}_{o})=\underbrace{\vphantom{\frac{F(\bm{ \omega}_{o},{\bm{h}};{\bm{a}},m)}{4(\bm{n}\cdot\bm{\omega}_{i}}}(1-m)\frac{\bm {a}}{\pi}}_{\text{diffuse}\;f_{d}}+\underbrace{\frac{D({\bm{h}};\rho)F(\bm{ \omega}_{o},{\bm{h}};{\bm{a}},m)G(\bm{\omega}_{i},\bm{\omega}_{o},{\bm{h}}; \rho)}{4(\bm{n}\cdot\bm{\omega}_{i})(\bm{n}\cdot\bm{\omega}_{o})}}_{\text{ specular}\;f_{s}(\bm{\omega}_{i},\bm{\omega}_{o})},$ (4) $$

where ${\bm{a}}\in[0,1]^{3}$ denotes the diffuse albedo, $m\in[0,1]$ denotes the metallic value, $\rho\in[0,1]$ denotes the roughness, and $\bm{h}=\frac{\bm{\omega}_{i}+\bm{\omega}_{o}}{\|\bm{\omega}_{i}+\bm{\omega}_{o
}\|}$ is the half vector. The specular component $f_{s}(\bm{\omega}_{i},\bm{\omega}_{o})$ is further decomposed into a normal distribution function (NDF) $D$, a Fresnel term $F$, and a geometry term $G$. Consequently, the outgoing irradiance $L_{o}$ can be decomposed into diffuse light $L_{d}$ and specular light $L_{s}$ as follows:

$$ $L_{o}\left(\bm{\omega}_{o},{\bm{x}}\right)=L_{d}\left({\bm{x}}\right)+L_{s} \left(\bm{\omega}_{o},{\bm{x}}\right),$ (5) $$

where

$$ $L_{d}\left({\bm{x}}\right)=\int_{\Omega}f_{d}L_{i}\left(\bm{\omega}_{i},{\bm{x }}\right)\left(\bm{\omega}_{i}\cdot{\bm{n}}\right)d\bm{\omega}_{i},\;L_{s} \left(\bm{\omega}_{o},{\bm{x}}\right)=\int_{\Omega}f_{s}(\bm{\omega}_{i},\bm{ \omega}_{o})L_{i}\left(\bm{\omega}_{i},{\bm{x}}\right)\left(\bm{\omega}_{i} \cdot{\bm{n}}\right)d\bm{\omega}_{i}.$ (6) $$

### 3.3 Global Illumination Decomposition

In real-world scenes, the incident light reaching a surface is often a combination of direct illumination from light sources and indirect illumination from light reflected off other objects. This is due to the occlusion of the light sources by the environment or the object itself, as well as multiple bounces of light between objects. To achieve a more realistic rendering of these scenes, for the diffuse component $L_{d}$, we decompose the incident light into direct illumination and indirect illumination, and calculate them separately as follows:

$$ $\displaystyle L_{d}\left({\bm{x}}\right)$ $\displaystyle=\int_{\Omega}(1-m)\frac{\bm{a}}{\pi}L_{i}\left(\bm{\omega}_{i},{ \bm{x}}\right)\left(\bm{\omega}_{i}\cdot{\bm{n}}\right)d\bm{\omega}_{i}$ (7) $\displaystyle=(1-m)\frac{\bm{a}}{\pi}\left(\int_{\Omega_{\mathrm{Vis}}}L^{dir} _{i}\left(\bm{\omega}_{i},{\bm{x}}\right)\left(\bm{\omega}_{i}\cdot{\bm{n}} \right)d\bm{\omega}_{i}+\int_{\Omega_{\mathrm{Occ}}}L^{ind}_{i}\left(\bm{ \omega}_{i},{\bm{x}}\right)\left(\bm{\omega}_{i}\cdot{\bm{n}}\right)d\bm{ \omega}_{i}\right)$ $\displaystyle\approx(1-m)\frac{\bm{a}}{\pi}O({\bm{x}})I_{dir}({\bm{x}})+(1-m) \frac{\bm{a}}{\pi}I_{ind}({\bm{x}}),$ $$

where the two integrals calculate the irradiance of the incident direct lighting and indirect lighting in the visible area $\Omega_{\mathrm{Vis}}$ and the occluded area $\Omega_{\mathrm{Occ}}$ in the upper hemisphere, respectively. Here, we have $\Omega_{\mathrm{Vis}}\cup\Omega_{\mathrm{Occ}}=\Omega$ and $\Omega_{\mathrm{Vis}}\cap\Omega_{\mathrm{Occ}}=\emptyset$. Furthermore, we can approximate these two integrals using the direct irradiance $I_{dir}$ and the indirect irradiance $I_{ind}$, and introduce an occlusion term $O({\bm{x}})$ to approximate the visibility.

## 4 Method

In this section, we present our GI-GS framework for inverse rendering, as depicted in Fig. 1. Given a set of posed RGB images, the pipeline proceeds in three stages. In the first stage, we follow the vanilla 3DGS pipeline to reconstruct the scene geometry. We incorporate the normal as a new attribute for each Gaussian primitive and optimize it using pseudo normals derived from the depth map (see Sec. 4.1). In the second stage, to recover the BRDF and lighting conditions, we incorporate a PBR pipeline with the deferred shading technique. We first render the depth map, normal map, and BRDF maps from the 3D Gaussians and store them in a G-buffer. Then, we estimate the occlusion from the depth map and normal map, and leverage a differentiable PBR pipeline to render an image $I_{dir}$ under direct lighting using a learnable environment map (see Sec. 4.2). In the final stage, given the geometry (depth and normals) and the first-pass rendered image $I_{dir}$, we apply differentiable path tracing to obtain the rendering result $I_{ind}$ under indirect lighting (see Sec. 4.3). Finally, we combine the two rendering results and optimize the lighting and the Gaussians’ BRDF attributes.

### 4.1 Geometry Reconstruction

To enable PBR, we first need to reconstruct reliable scene geometry from 3DGS.

Depth Rendering. We follow GS-IR (Liang et al., ) to treat the depth as a linear interpolation of the depth of the $N$ Gaussians, i.e., $d=\sum_{i=1}^{N}w_{i}d_{i}$, where $w_{i}=\frac{T_{i}\alpha_{i}}{\sum_{i=1}^{N}T_{i}\alpha_{i}}$.

Compared to directly $\alpha$-blending the depth of each Gaussian, this approach reduces artifacts and ensures a smoother transition in rendered depth values between the deepest and shallowest Gaussians.

Normal Estimation. In this work, we treat the normal ${\bm{n}}$ as an attribute of each Gaussian and use $\alpha$-blending to render the normal map. Unlike existing methods (Shi et al., ; Chen et al., ) that use the direction of the shortest axis of the ellipsoid as the normal direction and apply regularization to force the Gaussian into a flat disk, our approach avoids the degradation in rendering quality caused by regularization. To enhance geometric consistency, we derive a pseudo normal map $\hat{{\bm{n}}}$ from the depth map during the optimization process and use it to supervise the rendered normals. Moreover, we incorporate the following total variation loss (TV-loss) from GS-IR to achieve smoother results:

$$ $\mathcal{L}_{n}=\mathcal{L}_{n-p}+\lambda_{n-TV}\mathcal{L}_{TV_{\text{normal} }},\;\mathcal{L}_{n-p}=\left\|\bm{n}-\hat{\bm{n}}\right\|.$ (8) $$

### 4.2 Direct Lighting Modeling

To capture the illumination from various directions and simplify the calculation, we employ image-based lighting (IBL) to model the direct lighting. According to the rendering equation in Eq. 3, we can decompose the outgoing radiation into a diffuse component $L_{d}$ and a specular component $L_{s}$. For the diffuse component $L_{d}$, we store the incident irradiance of direct lighting $I_{dir}$ (see Eq. 7) from different view directions in a prefiltered environment map.

To tackle the intractable integral of the specular component $L_{s}$, we leverage the widely used split-sum approximation (Karis & Games, ) to divide the integral into two parts as follows:

$$ $\displaystyle L_{s}$ $\displaystyle=\int_{\Omega}\frac{DFG}{4(\bm{n}\cdot\bm{\omega}_{i})(\bm{n} \cdot\bm{\omega}_{o})}L_{i}\left(\bm{\omega}_{i},{\bm{x}}\right)\left(\bm{ \omega}_{i}\cdot{\bm{n}}\right)d\bm{\omega}_{i}$ (9) $\displaystyle\approx\underbrace{\int_{\Omega}\frac{DFG}{4(\bm{n}\cdot\bm{ \omega}_{o})}d\bm{\omega}_{i}}_{\text{BRDF integral}\;R}\underbrace{\int_{ \Omega}L_{i}\left(\bm{\omega}_{i}\right)D\left(\bm{\omega}_{i},\bm{\omega}_{o} \right)\left(\bm{\omega}_{i}\cdot\bm{n}\right)d\bm{\omega}_{i}}_{\text{Pre- filtered environment map}\;I_{s}}.$ $$

The first term $R$ represents the integral of BRDF under a constant environment light and is only determined by the angle $\theta$ between the surface normal ${\bm{n}}$ and view direction $\bm{\omega}_{o}$, as well as the roughness $\rho$. This integral can be precomputed and stored in a 2D look-up table. The second term $I_{s}$ represents the incident irrdiance considering the NDF $D$. Since the NDF $D$ is related to the surface roughness $\rho$ (the rougher the surface, the larger the specular lobe), we can also store the incident irrdiance in prefiltered environment maps with different mip-levels for different roughness values. Therefore, the rendering result under direct lighting can be written as:

$$ $L_{dir}=(1-m)\frac{\bm{a}}{\pi}O({\bm{x}})I_{dir}+RI_{s},$ (10) $$

where $O({\bm{x}})$ is the occlusion at point ${\bm{x}}$ obtained through path tracing, as elaborated in Sec. 4.3.

### 4.3 indirect lighting Modeling

Indirect lighting refers to the illumination that reaches a surface after reflecting off surfaces. It is a crucial component for realistic rendering, particularly in areas occluded by other objects or by the surfaces themselves, as depicted in Fig. LABEL:fig2a. Typically, to calculate occlusion and indirect lighting, we can sample several rays from the normal-oriented hemisphere $\Omega$ of the surface and determine whether these rays intersect with other surfaces. However, 3DGS explicitly represents scenes as a set of points, which complicates the use of path tracing methods. Inspired by the deferred shading techniques from the gaming industry, we propose a G-buffer based indirect lighting. Our key insight is to recover the geometry from the G-buffer and implement path tracing on the recovered surface. To accelerate the speed of path tracing, we implement a tile-based fast ray marching framework.

Figure: Figure 3: (a) Occlusion quantifies the degree to which a surface point can receive ambient light. Points on flat surfaces demonstrate high visibility due to less occlusion from surrounding geometry. In contrast, points located in holes, corners, or adjacent to other surfaces appear darker, since they are more occluded. (b) Path tracing: For each surface point corresponding to a pixel on the depth map, we perform ray marching starting from that point to calculate the visibility in different directions.

Ambient Occlusion Calculation. Given the rendered depth map and normal map from the first stage, for a surface point ${\bm{x}}=(u,v)$ in the depth map, its position in the view space $(X,Y,Z)$ can be obtained from its depth as ${\bm{x}}={(x,y,z)}^{T}=z{(u-c_{x},v-c_{y},1)}^{T}$, where $z=d(u,v)$ represents its depth and $(c_{x},c_{y})$ is the coordinate of the optical center. The occlusion at surface point ${\bm{x}}$ is expressed as the integral of the visibility function $V$ from various directions $\bm{\omega}\in\Omega$ as follows:

$$ $O({\bm{x}})=1-\frac{1}{\pi}\int_{\Omega}V(\bm{\omega}){\bm{n}}\cdot\bm{\omega} d\bm{\omega},$ (11) $$

where $V(\bm{\omega})=1$ if a ray starting from point ${\bm{x}}$ with direction $\bm{\omega}$ intersects the surface and $V(\bm{\omega})=0$ otherwise. Since it is difficult to perform uniform sampling directly in solid angle space, we first consider the equivalent form of the original integral in spherical coordinates and uniformly generate sampling vectors to estimate the integral. For more details, please refer to Appendix B.

Adaptive Path Tracing. Path tracing is used to determine whether a ray intersects the surface. As illustrated in Fig. LABEL:fig2b, consider a ray starting from a surface point $\bm{x_{0}}\in\mathbb{R}^{3\times 1}$ with direction $\bm{\omega}\in\mathbb{R}^{3\times 1}$, then any point $\bm{x}=(x,y,z)\in\mathbb{R}^{3\times 1}$ on this ray can be written as $\bm{x}=\bm{x_{0}}+\bm{\omega}t$, where $t$ is the distance from $\bm{x_{0}}$. To approximate the marching of the ray, we trace the sample points along the ray. The position of sample point $\bm{x}$ in the pixel coordinate $(U,V)$ is obtained as $\bm{x}=(u,v)=\left(\frac{x}{z}-c_{x},\frac{y}{z}-c_{y}\right).$

If the depth $z_{d}=d(u,v)$ at the corresponding coordinate on the depth map is less than $z$, the ray intersects a surface and $\bm{x}$ is occluded in the $\bm{\omega}$ direction. That is,

$$ $V(\bm{\omega})=\begin{cases}1&\text{ if }z_{d}<z,\\ 0&\text{ else. }\end{cases}$ (12) $$

To handle the issue of perspective projection, where the marching of ray in 3D space is not linear to the marching of ray in 2D screen space, we adaptively adjust the step length based on the distance of the current ray-marching starting point. Please refer to Appendix B for more details.

Figure: Figure 5: (a) When the surface point ${\bm{x}}$ is occluded by $\bm{\hat{x}}$ in direction $\bm{\omega}_{i}$, $\bm{\hat{x}}$ acts as an indirect light source. In this case, the light received by ${\bm{x}}$ in direction $\bm{\omega}_{i}$ is equivalent to the light from the actual light source reflected by $\bm{\hat{x}}$ in the $-\bm{\omega}_{i}$ direction. (b) To obtain global geometry and lighting information, we rotate the camera to render the results of the remaining five perspectives and form a cubemap. This cubemap is then utilized to restore the global geometry and facilitate path tracing.

Indirect Lighting Calculation. When calculating indirect lighting, other objects occluding the surface point are regarded as indirect lighting sources, as shown in Fig. LABEL:fig3a. According to rendering equation in Eq. 3, the incoming radiation at point ${\bm{x}}$ in direction $\bm{\omega}$ is the outgoing radiation of the intersection point $\bm{\hat{x}}$ in direction$-\bm{\omega}$, i.e., $L_{i}(\bm{x},\bm{\omega})=L_{o}(\bm{\hat{x}},-\bm{\omega})$.

To obtain the radiation from intersection points, we first render the RGB image $I_{dir}$ of the scene, considering only direct lighting (see Sec. 4.2). Then, we employ the aforementioned path tracing method to determine if a ray intersects with other surfaces. Once an intersection point is identified, the outgoing radiation is obtained by indexing the corresponding pixel value $I_{dir}(u,v)$ in the RGB image based on the position of the intersection point $(u,v)$. Consequently, the outgoing diffuse radiation under indirect lighting has a similar form to the occlusion integral in Eq. 11 as follows:

$$ $L_{ind}=(1-m)\frac{\bm{a}}{\pi}\int_{\Omega}L_{i}\left(\bm{\omega}_{i},{\bm{x} }\right)n\cdot\bm{\omega}_{i}d\bm{\omega}_{i},\quad L_{i}(\bm{\omega}_{i},{\bm {x}})=V(\bm{\omega}_{i},{\bm{x}})I_{dir}(u,v).$ (13) $$

We use uniform sampling to calculate indirect lighting and obtain the final rendering result as:

$$ $L_{o}(\bm{\omega}_{o},{\bm{x}})=L_{dir}+L_{ind}=\underbrace{(1-m)\frac{\bm{a}} {\pi}O({\bm{x}})I_{dir}+RI_{s}}_{\text{First-pass}}+\underbrace{\vphantom{ \frac{\bm{a}}{\pi}}L_{ind}}_{\text{Second-pass}}.$ (14) $$

To optimize the materials and lighting, we minimize the following decomposition loss:

$$ $\mathcal{L}_{d}=\underbrace{\mathcal{L}_{1}}_{\mathcal{L}_{\text{color}}}+ \underbrace{\lambda_{M}\mathcal{L}_{TV_{\text{mat}}}}_{\mathcal{L}_{\text{ material }}}+\underbrace{\lambda_{E}\mathcal{L}_{TV_{\text{light}}}}_{\mathcal {L}_{\text{light }}},$ (15) $$

where the first term is the color loss, and the remaining two terms are the material TV loss $\mathcal{L}_{\text{material }}$ and lighting TV loss $\mathcal{L}_{\text{light }}$ introduced in GS-IR (Liang et al., ) to smooth the materials and lighting. Please refer to Appendix B for more details about $\mathcal{L}_{\text{material }}$ and $\mathcal{L}_{\text{light }}$.

### 4.4 Cubemap based indirect lighting

In Sec. 4.3, the scene geometry is recovered from the depth map. However, this approach only calculates occlusion and indirect lighting in screen space, considering only the local geometry around the surface point. For complex real-world scenes, occlusion and indirect lighting may originate from areas outside the camera’s frustum. Thus, it is necessary to model the scene’s global geometry. To this end, we extend our path tracing-based indirect lighting from screen space to world space. Specifically, as illustrated in Fig. LABEL:fig3b, we utilize a cubemap to recover the global geometry and calculate indirect lighting based on it. By taking the current rendering result (depth map, normal map, and RGB image $I_{dir}$) as the front face, we change the camera’s pose to render the remaining five faces of the cubemap. Each face of the depth cubemap can be roughly regarded as the geometry of the scene in that direction, and the corresponding indirect illumination is contained in the RGB cubemap. Then, we calculate the indirect lighting by path tracing, as elaborated in Sec. 4.3.

## 5 Experiments

### 5.1 Settings

Datasets and Metrics. We perform experiments on the TensoIR synthetic dataset (Jin et al., ) and the Mip-NeRF 360 (Mildenhall et al., ) dataset. For both datasets, we evaluate the quality of novel view synthesis to compare the overall performance. Notably, the TensoIR dataset provides ground truth results for albedo and relighting under different lighting conditions, which allows us to further evaluate the performance of our estimated albedo and relighting outputs. To quantify the performance, we employ three key metrics: PSNR, SSIM, and LPIPS.

**Table 1: Quantitative results on the TensoIR dataset. The results show that our method surpasses all baselines in novel view synthesis and albedo reconstruction. It ranks second only to TensoIR in the relighting results. Notably, our method effectively models indirect lighting when relighting.**
|  | NVS | albedo | relight |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | PSNR $\uparrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$ | PSNR $\uparrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$ | PSNR $\uparrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$ |
| NeRFactor | 24.68 | 0.922 | 0.120 | 25.13 | 0.940 | 0.109 | 23.38 | 0.908 | 0.131 |
| InvRender | 27.37 | 0.934 | 0.089 | 27.34 | 0.933 | 0.100 | 23.97 | 0.901 | 0.101 |
| NVDiffrec | 30.70 | 0.962 | 0.052 | 29.17 | 0.908 | 0.115 | 19.88 | 0.879 | 0.104 |
| TensoIR | 35.09 | 0.976 | 0.040 | 29.27 | 0.950 | 0.085 | 28.58 | 0.944 | 0.081 |
| GS-IR | 35.33 | 0.974 | 0.039 | 30.29 | 0.941 | 0.084 | 24.37 | 0.885 | 0.096 |
| Ours | 36.75 | 0.972 | 0.037 | 31.97 | 0.941 | 0.085 | 24.70 | 0.886 | 0.106 |

Figure: Figure 6: Qualitative comparison on the TensoIR dataset. We visualize the rendering results, reconstructed albedo, and relighting results, respectively.
Refer to caption: x2.png

Baselines. We compare our GI-GS method against several baselines that focus on inverse rendering, including NeRFactor (Zhang et al., ), InvRender (Zhang et al., ), NVDiffrec (Munkberg et al., ), TensoIR (Jin et al., ), as well as recent state-of-the-art (SOTA) 3DGS-based methods, GaussianShader (Jiang et al., ) and GS-IR (Liang et al., ). We also compare our method with some previous methods aimed at novel view synthesis, including NeRF++(Zhang et al., ), Instant NGP (Müller et al., ), Mip-NeRF360 (Barron et al., ), and 3DGS (Kerbl et al., ). For quantitative comparisons, we report the results from their original papers, while the results of GaussianShader are from (Du et al., ). For qualitative comparisons, we reproduce the results of TensoIR and GS-IR using their released codes.

### 5.2 Results and Comparisons

TensoIR Dataset. We first evaluate our approach against previous SOTA methods on the TensoIR dataset. Table 1 presents a quantitative comparison. Our approach outperforms previous methods in both novel view synthesis and albedo reconstruction in terms of PSNR, while maintaining comparable SSIM and LPIPS scores. This demonstrates the effectiveness of our approach in material reconstruction and global illumination decomposition. In the relighting results, our method ranks second only to TensoIR and surpasses all other methods. We also visualize the qualitative results in Fig. 6, which demonstrate that our rendering results are closer to the ground truth in overall color accuracy and have a smoother appearance in occluded areas.

**Table 2: Quantitative results on the Mip-NeRF 360 dataset. Our GI-GS outperforms previous inverse rendering methods and even surpasses some methods designed for novel view synthesis.**
|  | Outdoor | Indoor |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
|  | PSNR $\uparrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$ | PSNR $\uparrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$ |
| NeRF | 21.46 | 0.458 | 0.515 | 26.84 | 0.790 | 0.370 |
| Instant NGP | 22.90 | 0.566 | 0.371 | 29.15 | 0.880 | 0.216 |
| Mip-NeRF 360 | 24.47 | 0.691 | 0.283 | 31.72 | 0.917 | 0.180 |
| 3DGS | 24.64 | 0.731 | 0.234 | 30.41 | 0.920 | 0.189 |
| GaussianShader | 22.80 | 0.665 | 0.297 | 26.19 | 0.876 | 0.243 |
| GS-IR | 23.45 | 0.671 | 0.284 | 27.46 | 0.863 | 0.237 |
| Ours | 24.01 | 0.701 | 0.250 | 29.29 | 0.896 | 0.152 |
| Ours-Cubemap | 23.81 | 0.695 | 0.251 | 29.07 | 0.892 | 0.161 |

Figure: Figure 7: Qualitative comparison on the Mip-NeRF 360 dataset. Our GI-GS effectively reconstructs high-frequency details and occluded areas in real-world scenes. Notably, the extended version, Ours-Cubemap, achieves a smoother result in occluded areas.
Refer to caption: x3.png

**Table 3: Ablation on occlusion and indirect lighting. By incorporating both occlusion and indirect lighting, the rendering process more faithfully captures the light and surface interactions.**
|  | TensoIR | Mip-NeRF 360 |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
|  | PSNR $\uparrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$ | PSNR $\uparrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$ |
| Ours w/o occlusion | 36.37 | 0.970 | 0.041 | 26.24 | 0.788 | 0.210 |
| Ours w/o indirect lighting | 35.60 | 0.965 | 0.047 | 25.89 | 0.778 | 0.212 |
| Ours | 36.75 | 0.972 | 0.037 | 26.35 | 0.788 | 0.206 |

Figure: Figure 8: Qualitative comparison of indirect lighting. For indirect illumination, we increase the brightness in linear space to make it easier to see the results. Our GI-GS provides more natural and smoother indirect lighting. Moreover, the indirect lighting of GI-GS covers most areas in the scene, especially when extending it to Ours-Cubemap, while the results of GS-IR have large areas missing.
Refer to caption: x4.png

Figure: Figure 9: Relighting visualization on the TensoIR dataset using different environment maps.
Refer to caption: x5.png

Mip-NeRF 360 Dataset. Table 2 presents the quantitative results on the Mip-NeRF 360 dataset, and Fig. 7 showcases the rendering outcomes. Our method surpasses previous 3DGS-based inverse rendering approaches across all metrics, achieving SOTA performance. Notably, our method demonstrates a significant advantage in rendering indoor scenes, where contain more occlusions. This highlights the superiority of our approach in modeling indirect lighting. Moreover, we extend our method from screen space to world space, as elaborated in Sec. 4.4, and refer to the extended version as Ours-Cubemap. While this extension incurs a slight decline in quantitative performance, the rendering results reveal more accurate and smoother modeling of occluded areas.

Indirect Lighting. To demonstrate the accuracy of our indirect lighting modeling, we separately render the occlusion map and the image under indirect illumination, as shown in Fig. 8. GS-IR uses volumes to store occlusion and indirect lighting. However, during optimization, not all volumes are sampled, particularly those farther from the camera, resulting in missing occlusion and lighting in many areas of the scene. In contrast, our method leverages adaptive path tracing to effectively capture occlusion and indirect lighting across most areas. By rendering from multiple perspectives to capture world space information, our extended Ours-Cubemap method achieves smoother and more accurate indirect lighting while maintaining multi-view consistency, especially for distant objects.

Relighting. Fig. 9 visualizes the relighting results on the TensoIR dataset under different light conditions. Our method achieves high-quality relighting results, which demonstrates the high accuracy of estimated materials and calculated indirect lighting. More results are provided in the Appendix.

### 5.3 Ablation Study

Effectiveness of Indirect Lighting. To evaluate the impact of our path tracing-based occlusion and indirect lighting, we conduct two experiments: one considering only occlusion and the other considering only indirect lighting. The quantitative results presented in Table 3 show that both components significantly affect on the final rendering quality, with indirect lighting being particularly impactful.

**Table 4: Ablation on the number of sampled rays. Our training speed remains comparable with GS-IR on the TensoIR dataset and is significantly faster on the Mip-NeRF 360 dataset.**
|  | TensoIR | Time | Mip-NeRF 360 | Time |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | PSNR $\uparrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$ | (min) | PSNR $\uparrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$ | (min) |
| GS-IR | 35.33 | 0.974 | 0.039 | 33 | 25.38 | 0.757 | 0.268 | 114 |
| Ours ($N_{s}=16$) | 36.25 | 0.966 | 0.047 | 27 | 26.19 | 0.773 | 0.222 | 55 |
| Ours ($N_{s}=64$) | 36.75 | 0.972 | 0.037 | 28 | 26.35 | 0.788 | 0.206 | 57 |
| Ours ($N_{s}=256$) | 36.82 | 0.976 | 0.034 | 32 | 26.42 | 0.791 | 0.203 | 64 |

Figure: Figure 10: Qualitative comparison of different numbers of ray samples. The accuracy of indirect lighting increases with the number of ray samples, leading to smoother and less noisy results.
Refer to caption: x6.png

Number of Sampled Rays. We also conduct experiment to analyze the impact of the number of ray samples $N_{s}$. The results in Table 4 and Fig. 10 show that increasing the number of sampled rays $N_{s}$ enhances the rendering quality, especially when $N_{s}$ is small. Notably, thanks to our efficient path tracing, increasing the number of ray samples only slightly slows down our training speed.

## 6 Conclusion

In this paper, we introduce GI-GS, a novel inverse rendering framework based on 3D Gaussian Splatting. To extend 3DGS to the inverse rendering problem, we augment each Gaussian’s original attributes by incorporating surface normals and BRDFs, and optimize them through a differentiable PBR pipeline. The key insight of our approach is to combine deferred shading and path tracing to enable accurate global illumination decomposition while ensuring real-time rendering. Our approach successfully achieves high-fidelity geometry and material reconstruction of objects and scenes, as well as precise global illumination decomposition. Extensive experiments demonstrate that GI-GS achieves state-of-the-art performance in inverse rendering, novel view synthesis, and relighting tasks, and outperforms previous NeRF-based and 3DGS-based methods.

Limitations. Our approach does not consider the specular component of indirect illumination, as it requires complex Monte Carlo sampling and is a long-standing challenge in computer graphics. Moreover, for complex real-world scenes, using an environment map as the direct lighting source may fail to capture the spatially varying lighting in the environment. In addition, our approach relies on high-quality reconstruction of the geometry, which suggests that the accuracy of path tracing can be improved by incorporating additional geometric constraints.

## Appendix A Implementation Details

Training Details. The first stage of our training process follows the vanilla 3DGS with the addition of the normal loss term. The training iterations are set to 30K, with the same learning rate as the vanilla 3DGS. Besides, we use the improved densification strategy introduced in GOF (Yu et al., ) to reduce blurred areas. The optimization of the materials and the lighting takes 10,000 and 5,000 iterations on the Mip-NeRF 360 and TensoIR datasets, respectively. For the BRDF attributes, we adopt the continuous learning rate decay strategy in Plenoxels(Fridovich-Keil et al., ), where the initial learning rate is set to 0.05 and decays to a final learning rates of 0.005. All training is conducted on a single NVIDIA A5000 GPU.

Loss Functions.
For the first stage, we initialize a set of 3D Gaussians using the following loss function:

$$ $\mathcal{L}_{\text{init }}=\mathcal{L}_{1}+\mathcal{L}_{SSIM}+\mathcal{L}_{n},$ (16) $$

where

$$ $\mathcal{L}_{n}=\mathcal{L}_{n-p}+\lambda_{n-TV}\mathcal{L}_{TV_{\text{normal} }},$ (17) $$

and $\mathcal{L}_{TV_{\text{normal}}}$ is the total variation (TV) loss conditioned by the predicted normal map $\bm{N}$ and the given reference image $\bm{I}$ as follows:

$$ $\mathcal{L}_{TV_{\text{normal}}}=\frac{1}{|\bm{N}|}\sum_{i,j}\triangle_{ij}^{ \bm{N}},$ (18) $$

and

$$ $\displaystyle\triangle_{ij}^{\bm{N}}=$ $\displaystyle\exp\left(-\left|\bm{I}_{i,j}-\bm{I}_{i-1,j}\right|\right)\left( \bm{N}_{i,j}-\bm{N}_{i-1,j}\right)^{2}$ (19) $\displaystyle+\exp\left(-\left|\bm{I}_{i,j}-\bm{I}_{i,j-1}\right|\right)\left( \bm{N}_{i,j}-\bm{N}_{i,j-1}\right)^{2}.$ $$

For the optimization of the materials and lighting, the decomposition loss $\mathcal{L}_{d}$ includes $\mathcal{L}_{\text{color}}$, $\mathcal{L}_{TV_{\text{mat}}}$ and $\mathcal{L}_{TV_{\text{light}}}$:

$$ $\mathcal{L}_{d}=\underbrace{\mathcal{L}_{1}}_{\mathcal{L}_{\text{color}}}+ \underbrace{\lambda_{M}\mathcal{L}_{TV_{\text{mat}}}}_{\mathcal{L}_{\text{ material }}}+\underbrace{\lambda_{E}\mathcal{L}_{TV_{\text{light}}}}_{\mathcal {L}_{\text{light }}},$ (20) $$

where $\mathcal{L}_{TV_{\text{mat}}}$ is a smooth term similar to $\mathcal{L}_{TV_{\text{normal}}}$. Given the material map $\bm{M}$, $\mathcal{L}_{TV_{\text{mat}}}$ is defined as:

$$ $\mathcal{L}_{TV_{\text{mat}}}=\frac{1}{|\bm{M}|}\sum_{i,j}\triangle_{ij}^{\bm{ M}},$ (21) $$

where

$$ $\displaystyle\triangle_{ij}^{\bm{M}}=$ $\displaystyle\exp\left(-\left|\bm{I}_{i,j}-\bm{I}_{i-1,j}\right|\right)\left( \bm{M}_{i,j}-\bm{M}_{i-1,j}\right)^{2}$ (22) $\displaystyle+\exp\left(-\left|\bm{I}_{i,j}-\bm{I}_{i,j-1}\right|\right)\left( \bm{M}_{i,j}-\bm{M}_{i,j-1}\right)^{2}.$ $$

Besides, $\mathcal{L}_{TV_{\text{light}}}$ is used in Eq. 20 to obtain a smoother environment light, which is defined as:

$$ $TV_{\text{light }}=\frac{1}{|\bm{E}|}\sum_{i,j}\triangle_{ij}^{\bm{E}},$ (23) $$

where

$$ $\triangle_{ij}^{\bm{E}}=\left(\bm{E}_{i,j}-\bm{E}_{i-1,j}\right)^{2}+\left(\bm {E}_{i,j}-\bm{E}_{i,j-1}\right)^{2},$ (24) $$

and $\bm{E}$ is a learnable environment light map. Following GS-IR, we set $\lambda_{n-TV}$, $\lambda_{M}$, and $\lambda_{E}$ to 5.0, 1.0, and 0.01, respectively.

## Appendix B Technical Details

Pseudo Normal. We derive the pseudo normals from the depth map $d$. Given a point $d(u,v)$ in the depth map, we consider the local $3\times 3$ window of depth values centered on this point and project these $9$ depth points into the correpsonding 3D points in the scene. By calculating the tangent vectors between these 3D points, we can derive the normals through the cross-product of these tangent vectors. The final pseudo normal for the central pixel corresponding to $(u,v)$ is computed as the average of all these derived normals.

Uniform Sampling.
Since it is difficult to perform uniform sampling directly in solid angle space, we first consider the equivalent form of the original integral (see Eq. 11) in spherical coordinates as follows:

$$ $O(\bm{x})=1-\frac{1}{\pi}\int_{\phi=0}^{2\pi}\int_{\theta=0}^{\frac{\pi}{2}}V( \phi,\theta)\cos(\theta)\sin(\theta)d\phi d\theta.$ (25) $$

To estimate the integral, we uniformly generate sampling vectors spread inside the normal-oriented
hemisphere by dividing each spherical coordinate into $n_{1}$ and $n_{2}$ samples to calculate

$$ $O(\bm{x})=1-\frac{\sum_{i=0}^{n_{1}}\sum_{j=0}^{n_{2}}V\left(\phi_{i},\theta_{ j}\right)\cos\left(\theta_{j}\right)\sin\left(\theta_{j}\right)}{\sum_{i=0}^{n _{1}}\sum_{j=0}^{n_{2}}\cos\left(\theta_{j}\right)\sin\left(\theta_{j}\right)}.$ (26) $$

Adaptive Ray Tracing.
Due to the perspective projection, the marching of a ray in 3D space is not linear to the marching of the ray in 2D screen space. For areas of scene that are far from the image plane, traversing a significant distance in 3D may only cover a few pixels on the screen. This can lead to poor sampling accuracy, as we are oversampling several pixels. To address this issue, we adaptively adjust the step length based on the distance of the current ray-marching starting point. Considering a ray that originates from a surface point $\bm{x_{0}}=(x_{0},y_{0},z_{0})\in\mathbb{R}^{3\times 1}$ with direction $\bm{\omega}\in\mathbb{R}^{3\times 1}$, the proposed adaptive ray tracing is expressed as follows:

$$ ${\bm{x}}={\bm{x}}_{0}+\bm{\omega}kt,\;k\in\mathbb{Z},$ (27) $$

where

$$ $t=t_{0}\left(1+\frac{z_{0}}{z_{\text{far}}-z_{\text{near}}}\right)^{2}.$ (28) $$

Here, $t_{0}$ is the pre-defined step length, $z_{\text{far}}$ is the distance from the camera to the far clipping plane, and $z_{\text{near}}$ is the distance to the near clipping plane. This adaptive approach enhances the accuracy of ray tracing by ensuring that step sizes are more appropriate for varying distances in the scene.

## Appendix C Additional Results on the Mip-NeRF 360 Dataset

We provide the quantitative results of novel view synthesis on the Mip-NeRF 360 dataset for each scene, as detailed in Tables 5 – 7. Our method outperforms all the previous 3DGS-based inverse rendering approaches and even some previous methods that focus on novel view synthesis. Importantly, our method achieves SOTA performance in terms of PSNR among all 3DGS-based inverse rendering approaches across all scenes, except for a negligible decline in the garden scene. In addition, we visualize the rendering results, decomposition results (normal, occlusion, indirect illumination), and relighting results in Fig. 11. Notably, our calculated occlusion and indirect illumination model the shadow effect and the bounces of lights across the scene with high fidelity. We also provide more qualitative comparison results with the previous SOTA 3DGS-based baseline GS-IR in Fig. 12 and Fig. 13.

**Table 5: PSNR scores for Mip-NeRF360 scenes.**
| method | bicycle | flowers | garden | stump | treehill | room | counter | kitchen | bonsai |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| NeRF++ | 22.64 | 20.31 | 24.32 | 24.34 | 22.20 | 28.87 | 26.38 | 27.80 | 29.15 |
| Plenoxels | 21.91 | 20.10 | 23.49 | 20.66 | 22.25 | 27.59 | 23.62 | 23.42 | 24.67 |
| INGP-Base | 22.19 | 20.35 | 24.60 | 23.63 | 22.36 | 29.27 | 26.44 | 28.55 | 30.34 |
| INGP-Big | 22.17 | 20.65 | 25.07 | 23.47 | 22.37 | 29.69 | 26.69 | 29.48 | 30.69 |
| Mip-NeRF 360 | 24.37 | 21.73 | 26.98 | 26.40 | 22.87 | 31.63 | 29.55 | 32.23 | 33.46 |
| 3DGS | 25.25 | 21.52 | 27.41 | 26.55 | 22.49 | 30.63 | 28.70 | 30.32 | 31.98 |
| GaussianShader | 23.12 | 20.34 | 26.44 | 23.92 | 20.17 | 24.27 | 26.35 | 27.72 | 28.09 |
| GS-IR | 23.80 | 20.57 | 25.72 | 25.37 | 21.79 | 28.79 | 26.22 | 27.99 | 28.18 |
| Ours | 24.32 | 20.90 | 26.42 | 26.08 | 22.31 | 29.89 | 27.18 | 29.94 | 30.15 |
| Ours-Cubemap | 24.21 | 20.85 | 26.05 | 25.89 | 22.05 | 29.60 | 27.03 | 29.60 | 30.05 |

**Table 6: SSIM scores for Mip-NeRF360 scenes.**
| method | bicycle | flowers | garden | stump | treehill | room | counter | kitchen | bonsai |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| NeRF++ | 0.526 | 0.453 | 0.635 | 0.594 | 0.530 | 0.530 | 0.802 | 0.816 | 0.876 |
| Plenoxels | 0.496 | 0.431 | 0.606 | 0.523 | 0.509 | 0.842 | 0.759 | 0.648 | 0.814 |
| INGP-Base | 0.491 | 0.450 | 0.649 | 0.574 | 0.518 | 0.855 | 0.798 | 0.818 | 0.890 |
| INGP-Big | 0.512 | 0.486 | 0.701 | 0.594 | 0.542 | 0.871 | 0.817 | 0.858 | 0.906 |
| Mip-NeRF 360 | 0.685 | 0.583 | 0.813 | 0.744 | 0.632 | 0.913 | 0.894 | 0.920 | 0.941 |
| 3DGS | 0.771 | 0.605 | 0.868 | 0.775 | 0.638 | 0.914 | 0.905 | 0.922 | 0.938 |
| GaussianShader | 0.700 | 0.542 | 0.842 | 0.667 | 0.572 | 0.847 | 0.874 | 0.887 | 0.893 |
| GS-IR | 0.706 | 0.543 | 0.804 | 0.716 | 0.586 | 0.867 | 0.839 | 0.867 | 0.883 |
| Ours | 0.740 | 0.574 | 0.830 | 0.751 | 0.608 | 0.901 | 0.860 | 0.908 | 0.917 |
| Ours-Cubemap | 0.737 | 0.574 | 0.817 | 0.749 | 0.598 | 0.892 | 0.857 | 0.901 | 0.917 |

**Table 7: LPIPS scores for Mip-NeRF360 scenes.**
| method | bicycle | flowers | garden | stump | treehill | room | counter | kitchen | bonsai |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| NeRF++ | 0.455 | 0.466 | 0.331 | 0.416 | 0.466 | 0.335 | 0.351 | 0.260 | 0.291 |
| Plenoxels | 0.506 | 0.521 | 0.386 | 0.503 | 0.540 | 0.419 | 0.441 | 0.447 | 0.398 |
| INGP-Base | 0.487 | 0.481 | 0.312 | 0.450 | 0.489 | 0.301 | 0.342 | 0.254 | 0.227 |
| INGP-Big | 0.446 | 0.441 | 0.257 | 0.421 | 0.450 | 0.261 | 0.306 | 0.195 | 0.205 |
| Mip-NeRF 360 | 0.301 | 0.344 | 0.170 | 0.261 | 0.339 | 0.211 | 0.204 | 0.127 | 0.176 |
| 3DGS | 0.205 | 0.336 | 0.103 | 0.210 | 0.317 | 0.220 | 0.204 | 0.129 | 0.205 |
| GaussianShader | 0.274 | 0.377 | 0.130 | 0.297 | 0.406 | 0.304 | 0.242 | 0.167 | 0.257 |
| GS-IR | 0.259 | 0.371 | 0.158 | 0.258 | 0.372 | 0.279 | 0.260 | 0.188 | 0.264 |
| Ours | 0.213 | 0.313 | 0.140 | 0.222 | 0.363 | 0.175 | 0.172 | 0.111 | 0.150 |
| Ours-Cubemap | 0.213 | 0.310 | 0.149 | 0.224 | 0.360 | 0.191 | 0.179 | 0.120 | 0.152 |

**Table 8: Per-scene results on the TensoIR synthetic dataset. For albedo reconstruction results, we follow NeRFactor (Zhang et al., ) and scale each RGB channel by a global scalar.**
| Scene | Method | NVS | albedo | relight |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| PSNR $\uparrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$ | PSNR $\uparrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$ | PSNR $\uparrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$ |  |  |
| Lego | NeRFactor | 26.08 | 0.881 | 0.151 | 25.44 | 0.937 | 0.112 | 23.25 | 0.865 | 0.156 |
| InvRender | 24.39 | 0.883 | 0.151 | 21.43 | 0.882 | 0.160 | 20.12 | 0.832 | 0.171 |  |
| NVDiffrec | 30.06 | 0.945 | 0.059 | 21.35 | 0.849 | 0.166 | 20.09 | 0.844 | 0.114 |  |
| TensoIR | 34.70 | 0.968 | 0.037 | 25.24 | 0.900 | 0.145 | 27.60 | 0.922 | 0.095 |  |
| GS-IR | 34.38 | 0.968 | 0.036 | 24.96 | 0.889 | 0.143 | 23.26 | 0.842 | 0.117 |  |
| Ours | 35.74 | 0.966 | 0.031 | 24.68 | 0.883 | 0.148 | 23.42 | 0.843 | 0.119 |  |
| Hotdog | NeRFactor | 24.50 | 0.940 | 0.141 | 24.65 | 0.950 | 0.142 | 22.71 | 0.914 | 0.159 |
| InvRender | 31.83 | 0.952 | 0.089 | 27.03 | 0.950 | 0.094 | 27.63 | 0.928 | 0.089 |  |
| NVDiffrec | 34.90 | 0.972 | 0.054 | 26.06 | 0.920 | 0.116 | 19.07 | 0.885 | 0.118 |  |
| TensoIR | 36.82 | 0.976 | 0.045 | 30.37 | 0.947 | 0.093 | 27.93 | 0.933 | 0.115 |  |
| GS-IR | 34.12 | 0.972 | 0.049 | 26.75 | 0.941 | 0.088 | 21.57 | 0.888 | 0.140 |  |
| Ours | 35.77 | 0.968 | 0.056 | 26.59 | 0.937 | 0.094 | 21.62 | 0.889 | 0.144 |  |
| Armadillo | NeRFactor | 26.48 | 0.947 | 0.095 | 28.01 | 0.946 | 0.096 | 26.89 | 0.944 | 0.102 |
| InvRender | 31.12 | 0.968 | 0.057 | 35.57 | 0.959 | 0.076 | 27.81 | 0.949 | 0.069 |  |
| NVDiffrec | 33.66 | 0.983 | 0.031 | 38.84 | 0.969 | 0.076 | 23.10 | 0.921 | 0.063 |  |
| TensoIR | 39.05 | 0.986 | 0.039 | 34.36 | 0.989 | 0.059 | 34.50 | 0.975 | 0.045 |  |
| GS-IR | 39.29 | 0.980 | 0.039 | 38.57 | 0.986 | 0.051 | 27.74 | 0.918 | 0.091 |  |
| Ours | 40.37 | 0.980 | 0.036 | 43.71 | 0.983 | 0.053 | 28.10 | 0.917 | 0.080 |  |
| Ficus | NeRFactor | 21.66 | 0.919 | 0.095 | 22.40 | 0.928 | 0.085 | 20.68 | 0.907 | 0.107 |
| InvRender | 22.13 | 0.934 | 0.057 | 25.33 | 0.942 | 0.072 | 20.33 | 0.895 | 0.073 |  |
| NVDiffrec | 22.13 | 0.946 | 0.064 | 30.44 | 0.894 | 0.101 | 17.26 | 0.865 | 0.073 |  |
| TensoIR | 29.78 | 0.973 | 0.041 | 27.13 | 0.964 | 0.044 | 24.30 | 0.947 | 0.068 |  |
| GS-IR | 33.55 | 0.976 | 0.031 | 30.87 | 0.948 | 0.053 | 24.93 | 0.893 | 0.081 |  |
| Ours | 35.13 | 0.976 | 0.023 | 32.90 | 0.961 | 0.047 | 25.65 | 0.895 | 0.081 |  |

Figure: Figure 11: Visualization of our inverse rendering, indirect lighting and relighting results on the Mip-NeRF 360 dataset.
Refer to caption: x7.png

Figure: Figure 12: More qualitative comparison results on the Mip-NeRF 360 dataset.
Refer to caption: x8.png

Figure: Figure 13: More qualitative comparison results on the Mip-NeRF 360 dataset.
Refer to caption: x9.png

## Appendix D Additional Results on the TensoIR Synthetic Dataset

We provide the quantitative results of novel view synthesis, albedo estimation, and relighting on the TensoIR synthetic dataset for each object, as reported in Table 8. Our method outperforms all previous inverse rendering approaches in novel view synthesis. For albedo estimation and relighting, our method surpasses the previous SOTA 3DGS-based baseline GS-IR. Besides, we visualize the rendering results, decomposition results (normal, albedo, roughness, occlusion, indirect illumination), and relighting results in Fig. 14. Moreover, Fig. 15 provides the material editing results on the TensoIR dataset.

Figure: Figure 14: Visualization of our inverse rendering, indirect lighting and relighting results on the TensoIR synthetic dataset.
Refer to caption: x10.png

Figure: Figure 15: Visualization of material editing on the TensoIR synthetic dataset.
Refer to caption: x11.png

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 8
presentation: 4
questions: 1. The occlusion map and indirect light map in Figs. 6 and 8 are quite impressive. However, I am curious as to why Fig. 4 shows that the proposed method still retains a significant amount of shadow in the albedo, particularly in the case of the Hotdog.

2. What does the direct light look like? The incorrect direct lighting reconstruction also lead to poor relighting performance.
soundness: 3
strengths: 1. The paper propose a novel framework where the deferred rendering technique enables light-weight path tracing on the deferred rendering buffer, which highlights the promising potential of deferred rendering in efficiently modeling indirect geometry-lighting interactions.
2. The method is extended to scene level and successfully produces meaningful decomposition and relighting results.
3. In novel view synthesis tasks, the method achieves best performance over inverse rendering baselines.
summary: The paper present a novel inverse rendering framework, GI-GS, to achieve scene decomposition and photo-realistic relighting.

Existing 3DGS-based methods, due to their nature of non-deterministic geometry, encounter challenges in modeling inter-reflection and self-occlusion, which prevent them from occlusion-aware/indirect-light-aware decomposition or relighting. While recent works like relightable 3DGS use ray tracing to estimate occlusion, preventing shadow from being baked into materials, many of them are still unable to process inter-reflection, or only employ a learnable residual term to model indirect light.

To address this, the paper propose a novel framework with deferred rendering technique, which employs path tracing to achieve indirect illumination computation. Experimental results are provided to demonstrate the effectiveness of the framework, highlighting the promising potential of deferred rendering in efficiently modeling indirect geometry-lighting interactions. In object-level cases, the proposed method outperforms several baselines in both novel view synthesis and decomposition, while achieving comparable performance in relighting. In scene-level cases, the method also demonstrates meaningful results in both decomposition and relighting.
weaknesses: 1. The novel indirect illumination module may require further experimentation to validate its effectiveness. Theoretically, this module has the potential to achieve superior relighting performance compared to solutions that rely on learnable parameters for modeling indirect light, as the fixed inter-reflection in those methods is often inaccurate when relighted under another illumination.

However, the quantitative results presented in Table 1 indicate that the relighting performance is subpar, even when achieving best albedo estimation, which undermines the strength of the paper's key claim. I believe the relighting performance is likely impacted by the inadequate normal estimation, as illustrated in Fig. 9 (Garden, Kitchen, Counter, and Bonsai), rather than suggesting that path tracing is ineffective, especially since the impressive qualitative results in Fig. 6 indicate otherwise.

Therefore, to quantitatively validate the effectiveness of the method, it would be beneficial to conduct an ablation study focused on albedo estimation or relighting. The current ablation study in Table 3 only examines performance changes related to novel view synthesis quality, which is insufficient to support the claims made in the paper.

2. In the absence of quantitative results, the qualitative findings suggest that normal estimation may be inadequate, as demonstrated in Fig. 9 (Garden, Kitchen, Counter, and Bonsai). This deficiency could negatively impact the overall performance of inverse rendering.

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 3
questions: See Weaknesses.
soundness: 2
strengths: The paper is intuitive and easy to follow.

Integrating indirect illumination calculation in a GS-based inverse rendering pipeline is a challenging movement.

In the experiments, the model shows comparable results in the task of novel view synthesis, decent performance in relighting, outperforming the GS-based baseline.
summary: This works introduces GI-GS, a novel gaussian-splatting based inverse rendering pipeline to extract intrinsics from multi-view images. As a follow-up of existing GS-based inverse rendering,  GS-IR, this paper focuses on taking the indirect illumination into account, to solve more accurate material and environment map from photorealistic images.
weaknesses: The way of processing indirect illumination in the pipeline is similar to InvRender: It uses the direct illumination (represented as a radiance function) of 3D points to calculate the indirect illumination. How is the indirect illumination calculation in this paper distincted from the one in InvRender? 

Tab. 1: TensoIR performs much better than the proposed method in relighting task. and in figure 4 it is hard to see the differences between GS-IR and proposed method. This harms the soundness of the method. Please add more explanation of the performance.

The usage of I_{dir}(u, v) in the eq. 13 might be inaccurate, since the image RGB value is the combination of diffuse and specular lighting, which is not equal to the illumination from x_hat to x. One way to avoid this to re-apply the first rendering pass with new look-at direction. Is there any specific reason for directly using the RGB values? Please clarify.

While the main focus of the work is on decomposition and relighting, no relighting evaluation for real-world inputs is shown. This challenges the effectiveness of the method in more realistic and complex cases. An experiment conducted on real-world benchmarks (such as Stanford-ORB) can be very helpful to support the claims of the paper.

Other minor issues.
Adaptive Path Tracing happens in the projected camera only, which is not able to take the occluded objects into account.
Eq. (8): $\mathcal_{L}_{n-p}$ looks confusing (might be interpreted as n minus p), $\mathcal_{L}_{n,p}$ might be better.
Tab. 1: Only relighting methods are compared in the NVS column. Please consider adding reconstruction methods as reference.

Based on the above points, I’m on the fence but slightly leaning to reject this paper. I’ll carefully look at the authors’ responses and happy to change my score if the concerns are addressed.

---

confidence: 4
contribution: 2
initial_rating: 8
presentation: 3
questions: see weakness.
soundness: 2
strengths: The paper propose a novel path tracing strategy that can query visibility in screen space.
summary: This paper propose a efficient path tracing techinque that can calculate indirect lighting. They first render a G-buffer to capture the detailed geometry and material properties of the scene. Then, they perform physically-based rendering only for direct lighting. With the G-buffer and previous rendering results, the indirect lighting can be calcuated through the proposed path tracing. The training is decomposed into three stages: 1. vanilla 3dgs, 2.deferred rendering with split-sum approximation with occlusion calculated from the proposed path tracing, 3.add indirect lighting term on top of stage2.
weaknesses: 1. in line303-305, I think authors can refer to the uniform sampling in hemi-sphere used by NeILF, which should be better than  sampling in spherical coordinate
2. The authors argure that they are the frist to incorperate indrect illuminance in relighting, however, the proposed lighting model can not model the indirect lighting in specular term, also, the approximation in Equation7 makes it impossible to model sharp shadows as visbility is considered in the integration of direct lighting.
3. The path tracing part is not well introduced. I can't figure out how exactly is the procedure of sampling point on ray. I guess the authors mean they sample many points step by step until the depth of a point is smaller than the rendered depth.
4. The calculation of visibility by comparing the depth of sampled point on the ray with the rendered depth is not comprehensive. There is another condition: the ray (start from the surface) is not occluded by the object, but the sampled points on the ray has a bigger depth value than the rendered depth map. For example, a ray that has a consistent direction as the orientation of the camera, then whether it is occluded is not determined by the rendered depth.
5. I'm also confused about the calculation of indirect light, in line338-341. Is the outgoing radiation indicates $L_p(\hat{x},-\omega_)$? If so, why can it be obtained by indexing the RGB map. And in some conditon, indirect light of a surface point can not be seen by the camera. Equation13 is also confusing.

---

confidence: 5
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. How does the method validate the effectiveness of its decomposed diffuse and specular components, given that these are crucial for its lighting model but are not evaluated?

2. Given the importance of material properties for accurate reflection, why is there no analysis of the decomposed material map components (albedo, specular, roughness), especially since benchmarked methods include accessible material evaluations?

3. How does the paper substantiate its claim that calculated occlusion effectively models shadowing, considering that benchmarked methods provide visibility and occlusion data but no strong evaluation is presented for this feature?

4. Without ground truth for indirect lighting, how can the accuracy of modeled interreflection and bounce lighting be verified, and might these effects reflect residuals of reflection optimization?

5. In what ways does the proposed method’s relighting evaluation demonstrate superiority over existing methods, especially given that occlusion and indirect lighting calculations are also present in baseline methods?
soundness: 2
strengths: The paper is well-written and easy to follow, and it includes comprehensive relighting works from the past.

1. GI-GS uses path tracing with deferred shading to accurately model indirect lighting, achieving realistic relighting and view synthesis.

2. By combining 3D Gaussian Splatting with a physically-based pipeline, GI-GS efficiently reconstructs detailed geometry and materials for high-quality results.

3. The framework decouples direct and indirect illumination, enabling good-quality relighting across various conditions.
summary: This work introduces a novel inverse rendering framework designed to achieve photo-realistic novel view synthesis and relighting by leveraging 3D Gaussian Splatting (3DGS) and deferred shading. Inverse rendering aims to accurately model scene materials, geometry, and lighting from images, a task complicated by the need to account for indirect lighting, which prior methods often model poorly. GI-GS improves upon these methods by incorporating efficient path tracing and deferred shading to compute indirect lighting, which leads to better handling of complex light-object interactions. The framework first uses a G-buffer to capture scene details and perform physically-based rendering (PBR) for direct lighting. Then, it applies lightweight path tracing to compute indirect lighting based on previous results, enabling more realistic global illumination. The authors demonstrate that GI-GS outperforms previous techniques in rendering quality and efficiency, making it a state-of-the-art method for novel view synthesis and relighting.
weaknesses: 1. This method leverages image-based lighting (IBL) to simulate illumination from multiple directions, addressing both diffuse and specular reflections as outlined in Section 4.2. However, the decomposed diffuse and specular components are not evaluated, even in the supplementary material, which I believe is crucial for your lighting model.

2. The paper benchmarks against the TensoIR, Gaussian Shader and GS-IR, where these methods, including the relightable 3D Gaussian, offer accessible material evaluations. While the proposed method includes a material map (albedo, specular, roughness, etc.) in its pipeline, there should be an analysis or evaluation of the decomposed material, as material properties are central to reflection.

3. The paper claims that the calculated occlusion models shadowing, a key contribution similar to that of benchmarked methods. However, the experiments do not include a strong evaluation of this feature, despite visibility and occlusion information being available in the benchmarked methods.

4. Additionally, for indirect lighting, which accounts for bouncing and interreflection, there is no ground truth to verify the accuracy of the modeled indirect illumination. This could also be interpreted as the residual of your reflection during optimization.

I have to challenge the novelty and quality of the proposed method, as both occlusion and indirect lighting are also addressed and calculated similarly in these baseline methods and relighting evaluation didn't prove the proposed method achieved better quality than the previous method.

Meta-review: This paper presents a new 3D Gaussian Splatting method that takes into account the indirect illumination to achieve photo-realistic novel view synthesis and relighting. The strength of the paper is the introduction of a new path tracing strategy that can query visibility in the screen space for indirect illumination handling. This merit is something reviewers consistently agreed on. On the other hand, there were concerns about the unclear evaluation, particularly diffuse and specular reflections. This stemmed from the lack of the ground truth for quantitative evaluation. This respect has been thoroughly discussed during the author-reviewer discussion phase, and the reviewers were convinced about the additional results. The AC agreed with the reviewers' positive opinions and rendered this recommendation.

Decision: Accept (Poster)