Subsurface Scattering for Gaussian Splatting
Authors: Jan-Niklas Dihlmann
Arjun Majumdar
Andreas Engelhardt
Raphael Braun
Hendrik Lensch
Venue: NeurIPS 2024 poster (2024)
Keywords: gaussian splatting
inverse rendering
relighting
subsurface scattering
computer graphics
brdf decomposition
pbr
differentiable rendering
nerf
Primary area: machine_vision
TL;DR: Realtime relighting of subsurface scattering objects in 3D gaussian splatting scenes

Abstract: 3D reconstruction and relighting of objects made from scattering materials present a significant challenge due to the complex light transport beneath the surface. 3D Gaussian Splatting introduced high-quality novel view synthesis at real-time speeds. While 3D Gaussians efficiently approximate an object's surface, they fail to capture the volumetric properties of subsurface scattering. We propose a framework for optimizing an object's shape together with the radiance transfer field given multi-view OLAT (one light at a time) data. Our method decomposes the scene into an explicit surface represented as 3D Gaussians, with a spatially varying BRDF, and an implicit volumetric representation of the scattering component. A learned incident light field accounts for shadowing. We optimize all parameters jointly via ray-traced differentiable rendering. Our approach enables material editing, relighting, and novel view synthesis at interactive rates. We show successful application on synthetic data and contribute a newly acquired multi-view multi-light dataset of objects in a light-stage setup. Compared to previous work we achieve comparable or better results at a fraction of optimization and rendering time while enabling detailed control over material attributes.

Subsurface Scattering for 3D Gaussian Splatting
Jan-Niklas Dihlmann
Arjun Majumdar
Andreas Engelhardt
Raphael Braun
Hendrik P.A. Lensch
University of Tübingen
Abstract
3D reconstruction and relighting of objects made from scattering materials present a significant challenge due to the complex light transport beneath the surface. 3D Gaussian Splatting introduced high-quality novel view synthesis at real-time speeds. While 3D Gaussians efficiently approximate an object's 's surface, they fail to capture the volumetric properties of subsurface scattering. We propose framework for optimizing an object's shape together with the radiance transfer field given multi- view OLAT (one light time) data. Our method decomposes the scene into explicit surface represented as 3D Gaussians, with a spatially varying BRDF, and an implicit volumetric representation of the scattering component. A learned incident light field accounts for shadowing. We optimize all parameters jointly via ray- traced differentiable rendering. Our approach enables material editing, relighting and novel view synthesis at interactive rates. We show successful application on synthetic data and introduce a newly acquired multi-view multi-light dataset of objects in a light-stage setup. Compared to previous work we achieve comparable better results fraction of optimization and rendering time while enabling detailed control over material attributes. Project page: https://sss, http://sss.jiihiaie jdihlmann com
SSS Dataset
3D Reconstruction
PBR Decomposition
Dafeve
ligtestnge
Editable
IBL Relighting
OLAT betaset
PPS PONR
Novel Light Synthesis
950
Figure 1: SSS GS We propose photorealistic real-time relighting and novel view synthesis of subsurface scaltering objects. We learn reconstruct shape and translucent appearance of an object within Gaussian Splatting framework. To so we leverage our newly created multi-vien view-view multi-lig light dataset of synthetic and real-world objects acquired in a stghe setup. The object is decomposed in a PBR fashion allowing for easy material editing and relighting. For trailer visit project page at https///ss.jihllmmnn.ooo
Introduction
Subsurface scattering (SSS) is an important aspect of our visual reality and therefore an indispensable part of realistic rendering. It is the process by which light penetrates surface and scattered
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

beneath it before being reflected back out. This phenomenon is responsible for the soft and diffuse appearance of materials such as wax, marble, skin and many other organic substances. Modeling SSS is challenging because it requires capturing the complex interactions between light and matter between different points on the surface. Approximating SSS by a simple surface reflection model where the light is reflected directly at the point of incidence typically leads to unnatural appearance of those objects. As a result, efforts in computer graphics to explicitly model SSS are either computationally expensive or approximate for interactive rendering. Additionally, capturing the complexity of spatially varying real-world scattering properties of an object using conventional computer vision techniques is difficult as they are often focused on reconstructing only visible surfaces and their BRDF.
In recent years, modeling SSS using neural networks has been topic of interest, e.g. [6]22 use Neural SSS materials as part of Monte Carlo global illumination rendering pipeline. These methods are trained in a supervised fashion using synthetic datasets. contrast, Neural Radiance Fields can learn the volumetric properties of SSS under varying lighting conditions implicitly and achieve photorealistic novel view synthesis and relighting. However, NeRFs are slow in training and inference due to the volumetric rendering requiring large or multiple MLPs to be evaluated for many point samples along each ray. Recently, 3D Gaussian Splatting (3D GS) [88] has been introduced as method for 3D reconstruction with high-quality novel view synthesis. It achieves real-time speeds by avoiding costly volume rendering, which, however, also limits the representation of volumetric effects. In this work, we introduce subsurface scattering to the 3D GS framework.
(NeRFs) [27 49. 25
Specifically, we propose the first method based on 3D GS for capturing detailed SSS effects of single objects, allowing for rendering and relighting in real-time.
At the core, we propose hybrid representation that extends 3D Gaussian Splatting with PBR material parameters [I0] and deferred shading [12] with light-weight residual prediction network to learn a subsurface scattering (SSS) shading component not modeled by the surface shader. We constrain the network predicting the outgoing SSS radiance by jointly predicting the incident radiance used for the PBR rendering step enforcing a neural representation of the local and global light transport in the scene. To overcome the inherent resolution limit of 3D Gaussians we perform the shading in image space. This improves the representation of specularity in particular.
We further introduce a newly acquired OLAT (one-light-at-a-time) dataset of SSS objects. It comprises object-centric 360" multi-view image collections of synthetic objects rendered in Blender using the Cycles PBR renderer g as well as real-world examples we acquired using a light stage and motorized camera.
Our method provides object based decomposition for PBR material components and SSS effects, which allows for editing and novel material synthesis. We show that our method has improved training time and faster rendering speed compared to previous SSS approaches based on NeRFs while achieving comparable or better results.
Possible applications include Medical Imaging and Visualization for Tissue Rendering and Surgical Simulation (35 40 Entertainment for visual effects and animation 30 VR and AR by providing a more realistic and immersive experience.
2 Related Work
Scene Representation Neural Radiance Fields (NeRF) introduced by Mildenhall et al. 27 have started trend in synthesizing novel views of complex 3D scenes with high fidelity, by representing the scene as a continuous 5D function using neural networks. While NeRFs have been widely adopted, they are computationally expensive due to the volumetric evaluation. There have been several works to accelerate NeRFs [28 38 33 one of which KiloNeRF [31] that uses multiple small MLPs to represent different parts of the scene. Kerbl et al. [88 introduced representing the scene set of explicit learnable 3D Gaussians. The proposed method 3D Gaussian Splatting (3D GS) is more efficient than NeRFs due to splatting i.e. rasterization of the Gaussians. However, both these reprexentations are limited to static scene representations and do not support religting or material decomposition.
Material Decomposition and Relighting There have been prior works accomplishing 3D shape and material reconstruction for relighting: NeRD [3] was of the first works to extend NeRF [27

---

to decompose the imput images into shape, BRDF and illumination for relighting. NeRFactor 47 and NeRV 7 aim for similar goals with slightly different network architectures. Neural-PIL 4 added a split-sum pre-integrated illumination network for more accurate and faster optimization. Likewise, NVDiffrec [13] use a pre-integrated illumination representation to optimize materials together with the object's shape.
In the realm of Gaussian Splatting, there have been several works focusing on relighting and BRDF With Relightable 3D Gaussians (R3DGS) [00 the authors decompose the scene into explicit metallic, roughness, color, and normal components, which can be composed and relit in real-time. The method models light with a neural incident light field [44] and a local learnable representaion as Spherical Harmonies. The illumination and shading is optimized per scene with differentiable rendering. Gaussian Shader [17] uses a similar decomposition but predicts the environmental illumination to not only model diffuse but also reflective surfaces.
decomposition of static scenes 100 17 23
Further, works have utilized deffered shading [12 with Gaussian Splatting [39 23 [9] to improve specular reflections, by focusing on the blending and propagation of normal directions between overlapping 3D Gaussians. Additionaly DeferredGS [39] train SDF in parallel to improve the surface geometry. We identify that the areas of the 3D Gaussians are a key factor for the representation of high frequency details in the scene and propose deferred shading to improve the specular reflections without the need of normal optimization or additional SDF training. We base our work of R3DGS [10] and augment the reflectance calculation to accommodate for subsurface scattering.
Subsurface Scattering There have been a multitude of prior works in SSS. Jensen et al. [66
presents practical model for subsurface light transport in translucent materials that captures effects beyond BRDF models. Donner et al. [9] further explore light diffusion in multi-layered translucent materials using a variant of the Kubelka-Munk theory. Lensch et al. [22 introduce a rendering method for translucent diffuse objects, in which viewpoint and illumination can be modified at interactive rates. and filtered using the precomputed kernels. Vicini et al. (37] introduce a new shape-adaptive BSSRDF model that is based on a conditional variational autoencoder which learns to sample from a reference distribution produced by brute-force volumetric path tracer. The distribution is conditional both material properties and a set of features characterizing geometric variations in the neighborhood of the incident location. Zheng et (8) learn neural representations for participating media with complete simulation of global illumination. They estimate direct illumination via ray tracing and compute indirect illumination with Spherical Harmonics. Zhu al. (49 propose novel framework for learning the radiance transfer field via volume rendering and utilizing various appearance cues to refine the geometry end-to-end. They extend relighting and reconstruction prospects to tackle wider range of materials in data-driven manner. They use a NeRF-like architecture to represent subsurface scattering based on known incident light directions from controlled acquisition. Similarly, Neural Radiance Transfer Fields (25) use one light at time (OLAT) data as supervision to learn global light transport. This data representation is very similar to ours, however, their dataset of translucent objects has not been released so far. Due to using NeRF-only approach they lack editing capabilities and have longer run times. Object-Centric Neural Scattering Functions (OSF) [46) introduce a framework for representing and rendering objects under varying lighting conditions and from arbitrary viewpoints using object-centric neural scattering functions. OSFs allow for flexible composition of scenes with multiple objects, each maintaining realistic interactions with light and shadows but they also suffer from long training time, even their variant based on KiloNeRF [31]. Further, in the realm of Gaussian Splatting, there have been works on human avatars incorperating some modeling of subsurface scattering with Spherical Harmonics [32]
Our method merges the implicit based shader representation for SSS with the efficiency of 3D GS. Although this is not the first time that 3D GS has been augmented with an implicit component, e.g. previously used to simulate view-dependent effects [4] 26). ours is the first to adapt it for modeling SSS.
3 Method
Our goal is to reconstruct photorealistic 3D objects with strong subsurface scattering (SSS) effects from multi-view, multi-light image sets and to render them in real-time. We propose novel method that extends 3D Gaussian Splatting (3D GS) with an explicit surface appearance model and combines it with an implicit SSS model (Fig. 2.
3

---

SSS Prediction
Deferred Shading
Nomelay
Surbecaderrgg
Rusterizo
Saughe11
Scattering
SD Scane
3D Causslen Space
Image Space
Figure 2: Subsurface Scattering Pipeline Our method implicitly models the subsurface scattering appearance of object and combines it with explicit surface appearance model. The object is represented as of 3D Gaussians, consisting of geometry and appearence properties. We ultilize small MLP evaluate the subsurface scattering residual given the view and light direction and a subset of properties for each Gaussian, Further, we evaluate the incident light for each Gaussian as a joint task within the same MLP given the visibility supervised by ray-tracing. Based on the computed properties in deferred shading pipeline. We evaluate the diffuse and specular color with BRDF model for every pixel in image space combine with the SSS residual to get the final color the object.
accumulate and rasterize each property the image plane
3.1 Background
3D Gaussian Splatting represents scene geometry and appearance using a set of 3D Gaussians [8] Rasterizing these Gaussians with efficiently designed splatting technique allows for fast 3D reconstruction and novel view synthesis. A single 3D Gaussian defined by its mean μl, i.e. the center position in 3D space, and the covariance matrix E expressing its rotation and scale following the Gaussian function
G(x) = exp (- - (2-
(1)
Further, each Gaussian associated with a color, modeled as Spherical Harmonics (SH) coefficients c,, represent view-dependent effects and an opacity for transparency. Consequentially, a of 3D Gaussians is defined as g = [(14, D3, Ci, o:) Rendering a scene represented by 3D Gaussians is done in the first step by projecting the Gaussians onto the image plane [50), where J is the Jacobian of the affine approximation of the projective transformation and W is the viewing transformation matrix. The covariance matrix transformed as follows:
(2) The second step is to accumulate and rasterize the Gaussians onto the image plane, which is done by alpha blending the splatted colors and opacities of the Gaussians. The final color C for pixel (u, v) is given by the summation of the set of layered sequenced Gaussians Gi contributing to the pixel:
Σ' JW SWEWFJ. Jr
i-1 II
Cu. Σ
(1-1aa))
The a term is constructed by multiplying the opacity with the 2D covariance contribution Σ' at given pixel position. To facilitate optimization the covariance E is parameterized as 3D vector for scale and unit quaternion g for rotation.
Relightable 3D Gaussians (R3D GS) 100 the authors decompose the appearance of 3D Gaussians into explicit material properties, which can be relit in real-time. The method models light with a global neural incident light field (45) and local learnable incident light representation based on SHs. The illumination and shading optimized for static scene with differentiable rendering.
Relightable 3D Gaussians
Each Gaussian receives additional physically based rendering (PBR) parameters, such as basecolor b, a roughness r. a metalness m, and a normal n. The approach adopts the Disney BRDF model [6] with the diffuse and specular terms follows:
m. b,
and Sxpecutar (Wos wi) D(h, F) h b, m)G wi, h,
fastusee
(4)
(wo (ww .

---

Here D is the microfacet distribution function, F is the Fresnel term, is the geometry term and h the half vector between the view and light direction. The method first optimizes the geometry of the scene and then the shading parameters such as the incident light field and the PBR parameters. Further, the normal derived from the geometry based on the depth in the scene. The authors additionally build efficient ray-tracing for 3D GS to supervise a learnable visibility SH term per Gaussian, which guides the incident light field optimization.
While showing impressive relighting results on a variety of objects, it fails for translucent objects featuring SSS. The method limited to learning from scenes with a single static illumination setting, Consequently, as SSS is not explicitly modeled, the SSS effect is baked-in into the basecolor parameter, preventing SSS effects from being correctly rendered during relighting (Fig. (5).
3.2 Subsurface Scattering for 3D Gaussian Splatting
By building upon the 3D GS framework including R3D GS, our approach extends it to capture the SSS effects of objects. An implicit neural SSS model is jointly trained with the explicit surface BRDF model. We utilize global neural network to estimate the SSS effects for each Gaussian in the scene and therefore approximate the internal light transport. At the same time, the neural network also takes care of the incident illumination on the object including local visibility to allow for fast evaluation.
SSS Modeling Our core contribution is the modeling of SSS effects in the 3D Gaussian representa- tion. For photorealistic rendering, the internal scattering of light can be modeled with a BSSRDF [[S] as simulated the rendering equation as follows:
S [ss((Xin Xout; Win Woul Tiin Zinn(Xmn, (Xm Win (win n. dlwi drin:
Tooul (Xuu Wout)
Here Sass is the BSSRDF, I. is the radiance, x is the position, are the directions, A is the surface area of the object, and 5 is the hemisphere at Xout The light transport through the object sketched on the left side of Figure 2 To exactly evaluate the scattered radiance due to the BSSRDF one would need to integrate all incoming light directions over the entire surface, which is expensive.
Rather than modeling the BSSRDF directly or explicitly evaluating the light transport by integrating over the surface we propose an implicit SSS shader. For small surface area, in our case 3D Gaussian, neural network learns to estimate the outgoing radiance due to SSS when the entire scene illuminated from single direction. This global implicit network (Fig. 2 is formulated as follows:
(Lot Lin) fe E, n, Win Wout, v),
(6)
where fe is the neural network, μl and Σ are the mean and covariance of a 3D Gaussian, is the normal, Win and Wout are the incident and outgoing light directions, and v is the shadowing term (see below). We apply Fourier encoding to μ in (27 which is omitted in Equation 6 The network is not only estimating the outgoing SSS radiance Tow (SSS) but at the same time the incident light Tan is predicted which will later be used to evaluate the direct reflection at p. We want to note that Li is a prediction of our model that is close to the true physical quantity Lin but might be slightly offset to compensate for limitations in the model. By forcing the network to predict the incident as well as the outgoing SSS radiance we let the network implicitly learn about the local and global light transport in the scene. Lout (SSS) and Lin predictions share the same MLP but apply separate output heads (Fig. 2).
Besides this general modeling of the outgoing radiance, we introduce a local parameter sss per 3D Gaussian to control the ratio of SSS vs. direct reflection.
In order to constrain the network to predict physically plausible results, we combine it with the previously described BRDF model and optimize jointly with the incident light field prediction. We formulate the combination of the BRDF and the SSS network as follows:
Lout = sss Lou + (1 sss) (fspecular + faituss) Ln (n wwn);
(7)
where sss subsurfaceness, which is optimized during training to balance the direct reflection and the SSS effects as formulated in OpenPBR (1.
Incident Light We assume a single light source at a time (OLAT) setup, where the position of the light source is known. Similar to [10) we trace rays using BVH to quickly estimate the visibility for
5

---

each Gaussian and learn this per Gaussian SH visibility term v. As explained above we model the incident light field with a neural network that takes the scene geometry. the light positions and the visibility into account. This network is jointly evaluated with our SSS radiance. We optimize the incident light field as one additional component of our differentiable rendering pipeline.
Per-Pixel Deferred Shading Instead of the direct illumination model of the R3DGS [IO] we introduce deferred shading [12] capture sharper highlights. We noticed that computing the BRDF model in Gaussian space is insufficient to capture high-free freuefre details such as specular highlights. This because some Gaussians represent large surface area as seen Figure 5 A single point-wise evaluation of the BRDF model at the Gaussian's center as done by [10] is too sparse. Therefore, we propose deferred shading approach for Gaussians, where we evaluate the BRDF model in image space after the rasterization of the Gaussians. The image space surface position is projected back to details such
the 3D space to evaluate the BRDF model at this surface point. This way, specular highlights are properly reproduced. In addition, a large range of editing capabilities is enabled as pixel operations can be applied to the shading buffers.
With this formulation, we achieve joint learning of geometry and the direct and global appearance of SSS objects. See Appendix [F for more details the architecture and the training. Further, the choice of the 3D GS representation and our lightweight MI. combined with explicit PBR shading allow for real-time rendering speeds. Due the deferred shading approach, we can capture high-frequency details such specular highlights. Furthermore, the decomposition into explicit distinct appearance components allows for high degree of editability, even controlling the degree of SSS effects.
Figure 3: Results of Decomposition showing different views with different light directions. Further, the decomposition PBR parameters is shown. The first two objects shown are synthetic while the lower two are scanned real-world world objects.
Experiments
Our proposed method facilitates real-time rendering of SSS objects and enables relighting and material editing. In the following, we present results of our method which is evaluated on synthetic and real-world datasets.
4.1
Experimental setup
While other NeRF-based SSS reconstruction and novel view relighting methods exist [46] 49 none of them provide public dataset. We created new OLAT dataset from synthetically rendered objects and real-world captured objects that capture various effects of SSS materials. In total, our datasets consist of 20 distinct objects from translucent material categories such plastic, wax, marble, jade, and liquids (Sec. B
Dataset
For the synthetic dataset we created synthetic light stage setup in Blender [8] that follows the formulation of [49]. It models 112 fixed light positions on the upper hemisphere divided into rings of 16 lights each. For training, we render 100 random camera views of each object with fixed distance to the object. For testing, we use the NeRF synthetic camera path proposed by Mildenhall et al. (27]. which consists of 200 camera views positioned outside the light stage hemisphere. In total,

---

we have 11.200 train and 22.400 test images of 800 x 800 resolution for each object. We rendered datasets for 5 distinct objects with Blender's [ Cycles renderer and the Principled BSDF shader. The 3D models are sourced from BlenderKit library (2]
The real-world dataset was captured in a light stage with turntable supporting the object and camera mounted on a motorized sled which can move on the vertical main are of sphere. Currently, the dataset includes 15 objects selected for diverse representation of geometric detail and materials exhibiting SSS based on local availability and suitability for the acquisition setup in terms of rigidness and size. We captured 158 positions per object with 167 light positions each. Additionally, one image with uniform illumination was captured for camera pose optimization and object masking, The relative camera positions were optimized with COLMAP (34] resulting in an inward inward-facing facing 360 multi-view dataset. The cameras were aligned to the known light source positions in joined reference frame. Similar to the camera reconstruction, object masks are generated from the uniformly lit images using automatic image matting based on 442 Depending on the object we use text prompt or points from the SfM stage as query to first generate pseudo trimaps leveraging SAM (2] and an open-vocabulary model (24]. The matting is then performed by ViTMatte [43 yielding masks with transparency. See appendix section for more information on the image processing pipeline. We will release more details when publishing the dataset.
total of approximately 25,000 images are split evenly into a train and test set by uniform sampling from the camera and light positions. We exclude frames with strong light flares or incomplete or wrong masks based on heuristies and some manual annotation. In summary, we discard roughly 10 % of the dataset. We don't it always use the full size of the datasets for training, as shown in Table | Our method can also be trained sparsely using only 500 images.
Implementation Details SSS GS builds upon the 3D GS framework [18 and its extension Re- lightable 3D Gaussians (R3DGS) [00] that we in turn extend to capture the subsurface scattering effects of an object. For our implicit representation of the scattering component and the incident light prediction, we use a shallow MI. with 3 layers with Leaky-Rel.U activations [1]]. The whole
pipeline is implemented with Pytorch using the provided eustom CUDA kernels from [99 10 for rendering. For more details on the training setup see Appendix F
4.2 Qualitative Results
Our qualitative results are shown in Figure and Figure However, relighting and novel view synthesis are best experienced in the supplementary video. In we show results rendered from our test set of objects from the synthetic (first two rows) and real-world (bottom two rows) part of our newly created dataset. In addition to novel view synthesis (view 0, view 1) we can freely change the light direction which we show for both views. We present the decomposition of the object into the surface appearance parameters, SSS effects and incident light map as they are used to render the final views on the left. Our method captures the SSS effect well using the SSS residual and adds the object's specularity from the PBR shading. Together with the predicted incident light the model achieves a plausible decomposition into basecolor, roughness and metalness material parameters. Note how the volumetric component can represent the complex light transport inside the entire volume. This is particularly observable in the case of the Tupperware object while the normals show a detailed representation of the surface. Find further quantitative analysis regarding the decomposition in section
Comparison In Figure 5 we show that Relightable 3D Gaussian (R3DGS) 10 fail to capture the subsurface scattering effects of the object. As they only allow training on a static scene without any dynamic lighting the scattering component baked into the basecolor which will fail to represent the SSS for a new light position (see also Sec. Our method can capture the subsurface scattering effects and the specular highlights of the object. Achieving clearer results than R3DGS [TO] in that regard due to the formulation of shading in image space. Figure 3 also visualizes the difference between shading in world space vs. our deferred shading approach in image space. As visualized on the left the Gaussians occupy different, sometimes very large areas along the surface limiting the rendering of nigh-frequency effects like specular highlights. NeRF-based methods have a lot more training capacity due to their large MI.ss that can represent the complex light transport connected to SSS. For fair comparison we select KiloOSF a variant of OSF (46) that optimized for real-time rendering. Compared to KiloOSF (6) in Figure i becomes apparent that also small MLP paired
7

---

with our explicit shading approach can achieve higher quality results after shorter training time. Even after 20 hours of training. there are still some artifacts visible in the geometry that stem from the voxelization performed by the underlying KiloNeRF [31. While the Stanford Bunny object overall works well, the soap bar and car toy object are lacking in the representation of the surface reflectivity and show frequent errors in the geometry.
4.3 Quantitative Results
We evaluate our method on the test split of the synthetic SSS dataset and the real-world SSS dataset on the task of novel view synthesis using the following metrics: PSNR, SSIM, and LPIPS. The results are shown in Table [ Our method can successfully render novel view synthesis and relighting of subsurface scattering objects at real-time speeds, with high-quality results. Each component of our method is crucial to achieving such results, we show the ablation of the components in the appendix (Sec. C
Method
Category
PSNRT
SSIME
LPIPS.
FPS
Train T.
Res.
Data
Ours
Synthetic
37.35
2.13
0.986 +: 0.006
0.03 + 0.01
16..2:411.5
2h
800"
11.200
Ours
Reul World
31.12 = 2.11
0.96 0.02
0,042 - 0.03
155.25 +: 22.38
2h
800"
=s 12.000
Ours
Synthetic
35.01 = 1.01
0.972 0.01
0.040 + 0.01
154.8 + 28.26
1h
256"
500
KiloOSE
Synthetic
25.91 1.88
0.93 0.02
0.097 + 0.03
14.4
20h
256
500
Ours
Real World 26.61
0.09
0.93 + 0.003
0.08 + 7e-4
94.5
3.54
1h
256
500
KiloOSF
Reul World 23.24
1.58
0.83 + 0.09
0.21 +: 0.07
14.4
20h
256
500
Table 1: Quantitative results test views of mehod on large images and bigger dataset (top)
and comparison against KiloOSF [46] within their setting (bottom). We excluded runs of KiloOSF that couldn't reconstruct anything for more comparable results. The best results are highlighted in
bold. All experiments where timed and run on a single NVIDIA RTX 4090. *: cropped images with one side length of 800.
Comparison We compare our method against the state-of-the-art NeRF method KiloOSF [46] which claims to achieve rendering real-time speeds (< 14 FPS). We use single NVIDIA RTX 4090 GPU per run on a compute server with total of 512 GB of RAM. We use the provided code and default configuration for the experiments, converting our datasets to their dataset specification. To be comparable and to fit within the framework of [46] we downscale our images to a resolution of 256 x 256 and only use 500 images of our synthetic dataset. We want to point out, that our method achieves similar results at real-time speeds on images of 800 x 800 resolution. In comparison, our method is faster to train since we can earry over the efficiency of the 3D GS framework by carefully designing our pipeline around a small MLP and an explicit BRDF decomposition. Similarly, we can save on the number of parameters per Gaussian since we directly store color instead of the SH coefficients, for example.
Up until now we only compare within the domain of real-tim sil-timme rendering and novel view synthesis, for SSS reconstruction and relighting, as our method is optimized for speed and editability through decomposition. Comparing the numbers of Tabel [ with the results reported in subsurface NeRF- based methods that aim for quality [49, (5 we still achieve similar results within the domain of synthetic scenes, and are en-par on real-world scenes. However, the datasets on which they reported the measurements are not publicly available. Therefore, we cannot directly compare.
4.4 Applications
In Figure 4 we provide a detailed demonstration of the editing applications enabled by our approach. Our method facilitates adjustments to the base color and SSS residual, allowing for seamless color changes. Additionally, it supports the editing of material parameters, enabling modifications that make the appearance more metallic, shinier, or rougher. We also demonstrate the ability to alter the opacity and intensity of the SSS residual, and show single light illumination beyond the training domain. Moreover, our approach introduces editing capabilities, powered by the SSS residual, that surpass those available in previous methods [10.
8

---

02
Blue
Diue
0.5 05
Green
Grean
Sidn
Mediam
0. B Rec
Givce
FAT
Roughtess
Light
Figure 4: Editing results, showcasing PBR based edits such (roughness metalness/ base color) as well as method specific properties (subsurfaceness residual color). The latter highlights editing only possible with this method. The rightmost column shows light positions not sampled from the light stage.
Find details further qualitative and quantitative analysis of the application possibilities of our method such as image based relighting (Sec. D in the appendix.
Relightable 3D Gaussian Splatting
Deferred Shading
Subourface Scottelin Tirpah
For Caussion Sparting
Vinw Syntheds
Relighing
R2005
R3DGS
Our
Figure Limits Relightable Gaussians (left) While Relightable Gaussians can reproduce view couuiinatiin-dependeen reflections they fail to properly relight subsurface scattering objects. Deferred Shading (right) allows us to evaluate the surface reflectance for each rendered pixel instead of per Gaussian. This way, specular highlights are rendered with crisper detail.
Kicorr
Figure 6: Comparison against the KiloOSF method. The top two objects real-world objects and the bottom two synthetic objects. Note the qualitative improvement shape and appearance compared to KiloOSF.
Limitations
Our approach reproduces SSS by modeling the apparent effect the Gaussian where the light leaves the object. To avoid the costly integration over the whole object surface, this information predicted by an MLP. Intricate variation due to strongly heterogeneous materials or angularly dependent SSS effects might only be roughly approximated. Furthermore, altering the geometry or the BSSRDF of the object would require retraining the SSS representation. While the screen space shading improves specular rendering significantly the shadowing is still limited by the low resolution

---

of the 3D Gaussians in the currently used rendering scheme. Additionally, our screen space shading complicates the modeling of transparent refracting surfaces as it would require multiple passes to correctly warp the buffers. We do not explicitly account for this in our method. The white light assumption helps the severely under-constrained decomposition task. Still, while the decomposition is constrained to be physically plausible, it might not always match the ground truth there are multitude of possible explanations given only the appearance and light position. Moving forward we would like to also enable reconstruction under less constrained illumination setting. Although not the intended use case and scope of this work our method could potentially be used to improve the scanning of humans SSS is an integral property of the appearance of human skin. This has implications on personal rights and privacy and potential misuse that need to be addressed these cases.
Conclusion
We present Subsurface Scattering (SSS) for 3D Gaussian Splatting (3D GS), a method to reconstruct translucent objects from OLAT multi-view image sets. By decomposing light transport into explicit PBR materials and an implicitly represented scattering component we enable novel view synthesis and relighting, as well as light and material editing in real-time. A per 3D Gaussian SSS parameter is learned to merge the two components. Our formulation enables small MLP to reason about local and global light transport in the scene and predict incident light in addition to the SSS radiance. By evaluating the BRDF in image space we can achieve high-quality specular shading independent of the resolution of the 3D Gaussians. Compared to 3D GS enable high-quality reconstructions for new class of objects and achieve faster optimization and rendering speed than previous NeRF-based methods with similar improved quality. Some limitations connected to the SSS representation and the rendering scheme remain which open up an interesting trajectory for future work. We also plan to release our dataset as the first OLAT SSS dataset including real-world translucent objects to the community.
Acknowledgements
Funded by the Deutsche Forschunsggemiinccaat (DFG, German Research Foundation) under Germany's ' Excellence Strategy - EXC number 2064/1 - Project number 390727645. This work was supported by the German Research Foundation (DFG): SFB 1233, Robust Vision: Inference Principles and Neural Mechanisms, TP 02, project number: 276693517. This work was supported by the Tübingen AI Center. The authors thank the International Max Planck Research School for Intelligent Systems (IMPRS-IS) for supporting Jan-Niklas Dihlmann and Arjun Majumdar. In cooperation with Sony Semiconductor Solutions -Europe, the authors would like to thank Mr. Zoltan Facius and Mr. Prasan Ashok Shedligeri.
References
[1] Zap Andersson, Paul Edmondson, Julien Guertault, Adrien Herubel, Alan King, Peter Kutz, Andréa Machizaud, Jamie Portsmouth, Frédéric Servant, and Jonathan Stone. Openpbr surface. Technical report, Academy Software Foundation (ASWF), May 2024. URL https:
trguttuu...a. 1o/ /o//pp/P////////
[2] BlenderKit. BlenderKit Download 0,3 FREE 3D models, textures and other Blender assets
blenderkit.com. https://www http:://ww.bleedeerkt.cc com 2024. [Accessed 22-05-2024].
[3] Mark Boss, Raphael Braun, Varun Jampani, Jonathan T Barron, Ce Liu, and Hendrik Lensch. Nerd: Neural reflectance decomposition from image collections. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 12684-12694, 2021.
[4] Mark Boss, Varun Jampani, Raphael Braun, Ce Liu, Jonathan Barron, and Hendrik Lensch. Neural-pil: Neural pre-integrated lighting for reflectance decomposition. Advances in Neural Information Processing Systems, 34: 10691-107044 2021.
[5] G. Bradski. The OpenCV Library. Dr. Dobb's 's Journal of Software Tools, 2000.
[6] Brent Burley. Physically-baser shading disney, 2012.
10

---

[7] Hao Chen, Bo He, Hanyu Wang. Yixuan Ren, Ser-Nam Lim, and Abhinav Shrivastava. NeRV: Neural representations for videos. In NeurlPS, 2021.
Blender Online Community. Blender 3D modelling and rendering package. Blender Foundation, Stichting Blender Foundation, Amsterdam, 2024. URL/htpp://wer blender
org
[9]
Craig Donner and Henrik Wann Jensen. Light diffusion in multi-layered translucent materials. ACM SIGGRAPH 2005 Papers, 2005.
[10] Jian Gao, Chun Gu, Youtian Lin, Hao Zhu, Xun Cao, Li Zhang, and Yao Yao. Relightable
3d gaussian: Real-time point cloud relighting with brdf decomposition and ray tracing, arXiv preprint arXiv:231 /60443,, 2023.
[11] Richard Hahnloser, Rahul Sarpeshkar, Misha Mahowald, Rodney Douglas, and H. Seung. Digital selection and analogue amplification coexist a cortex-inspired silicon circuit. Nature, 405:947-51, 07 2000. doi: 10. 1038/35016072.
[12] Shawn Hargreaves and Mark Harris. Deferred shading. In Game Developers Conference. volume 2, page 31, 2004.
[13] Jon Hasselgren, Nikolai Hofmann, and Jacob Munkberg. Shape, light, and material decom- position from images using monte carlo rendering and denoising. In Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS '22 Red Hook, NY,
USA, 2024. Curran Associates Inc. ISBN 9781713871088.
[14] Poly Haven. Poly Haven Poly Haven - polyhaven.com. httpss:/poolyhavan. com/ 2024. [Accessed 22-08-2024).
[15] Henrik Wann Jensen, Stephen R Marschner, Mare Levoy, and Pat Hanrahan. A practical model for subsurface light transport. In ACM SIGGRAPH, Proceedings of the 28th annual conference on Computer graphics and interactive techniques, pages 51 511-518, 2001.
[16) Henrik Wann Jensen, Steve Marschner, Marc Levoy, and Pat Hanrahan. A practical model for subsurface light transport. Proceedings of the 28th annual conference on Computer graphics and interactive techniques, 2001.
[17] Yingwenqi Jiang, Jiadong Tu, Yuan Liu, Xifeng Gao, Xiaoxiao Long, Wenping Wang, and Yuexin Ma. Gaussianshader: 3d gaussian splatting with shading functions for reflective surfaces, 2023.
[18] Bernhard Kerbl, Georgios Kopanas, Thomas Leimküüher, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics, 42(4):1-14, 2023.
[19] Ye Keyang, Hou Qiming, and Zhou Kun. 3d gaussian splatting with deferred reflection. In ACM SIGGRAPHI Conference Proceedings, Denver, CO, United States, July 28- August 1, 2024, 2024.
[20] Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR), San Diega, CA, USA, 2015.
[21] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan- Wan-Yen!! Lo, et al. Segment anything. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4026. 2023.
[22] H.P.A. Lensch, M. Goesele, P. Bekaert, J. Kautz, M.A. Magnor, Jochen Lang, and H.-P. Seidel. Interactive rendering of translucent objects. 10th Pucific Conference on Computer Graphics and Applications, 2002.
[23] Zhihao Liang, Qi Zhang, Ying Feng, Ying Shan, and Kui Jia. Gs-ir: 3d gaussian splatting for inverse rendering. arXiv preprint arXiv:2311.1644 2023.
[24] Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al. Grounding dino: Marrying dino with grounded pre- pre-training for open-set object detection. arXiv preprint arXiv:2303.. 00099, 2023.
[25] Linjie Lyu, Ayush Tewari, Thomas Leimkuehler, Mare Habermann, and Christian Theobalt. Neural radiance transfer fields for relightable novel-view synthesis with global illumination. In
ECCV, 2022.
11

---

[26] Dawid Malarz, Weronika Smolak, Jacek Tabor, Stawomir Tadeja, and Przemyslaw Spurek. Gaussian splatting with nerf-based color and opacity, 2024.
[27] Ben Mildenhall, Pratul Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi. and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In ECCV, 2020.
[28] Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with multiresolution hash encoding. Transactions on Graphics, 2022.
[29] Krishna Mullia, Fujun Luan, Xin Sun, and Milos Hasan. Relightable neural assets. ArXiv, abs/2312.093999 2023.
[30] OTOY. LightStage OTOY home.otoy.com. https://home.otan otoy. cow capture/ lightstage/ 2024. [Accessed I 11-08-2024).
[31] Christian Reiser, Songyou Peng. Yiyi Liao, and Andreas Geiger. Kilonerf: Speeding up neural radiance fields with thousands of tiny mlps. In ICCV, pages 14315-14325, 2021. doi:
10.1 00.110//CCC44892.22222 2021.0107..
[32] Shunsuke Saito, Gabriel Schwartz, Tomas Simon, Junxuan Li, and Giljoo Nam. Relightable gaussian codec avatars. In CVPR, 2024.
[33] Sara Fridovich- Keil and Alex Yu, Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angjoo Kanazawa. Plenoxels: Radiance fields without neural networks. CVPR, 2022.
[34] Johannes Lutz Schönberger and Jan-Michael Frahm. Structure-from-mottie revisited. In Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
[35] Johannes Schüle, Franziska Krauss, Carina Veil, Stefan Kunkel, Peter Somers, Cristina Tarin, and Oliver Sawodny. Multi Muyticl tissue modeling of human urinary bladder. 2021 43rd Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC), pages 4297-4302, 2021.
[36] TG Thomson, Jeppe Revall Frisvad, Ravi Ramamoorthi, and Henrik Wann Jensen. Neural BSSRDF: Object appearance representation including heterogeneous subsurface scattering. ArXiv, abs/2312.157111 2023.
[37] Delio Vicini, Vladlen Koltun, and Wenzel Jakob. A learned adape subsurface scattering model. Transactions on Graphics (Proceedings of SIGGRAPH), 38(4):15, July 2019. doi:
01114553306346.32229744
[38] Qianqian Wang, Zhicheng Wang, Kyle Genova, Pratul Srinivasan, Howard Zhou, Jonathan T. Barron, Ricardo Martin-Brualla, Noah Snavely, and Thomas Funkhouser. Ibrnet: Learning multi-view image-based rendering. CVPR, 2021.
[39] Tong Wu, Jia-Mu Sun, Yu-Kun Lai, Yuewen Ma, Leif Kobbelt, and Lin Gao. Deferredgs: Decoupled and editable gaussian splatting with deferred shading, 2024.
[40] Zhuoyue Yang, Ju Dai, and Junjun Pan. 3d reconstruction from endoscopy images: A survey. Computers biology and medicine, 175:108546, 2024.
[41] Ziyi Yang. Xinyu Gao, Yangtian Sun, Yihua Huang, Xiaoyang Lyu, Wen Zhou, Shaohui Jiao, Xiaojuan Qi, and Xiaogang Jin. Spec-gaussian Anisotropic view-dependent appearance for 3d gaussian splatting, 2024.
[42] Jingfeng Yao. Xinggang Wang, Lang Ye, and Wenyu Liu. Matte anything: Interactive natural image matting with segment anything models. arXiv preprint arXiv:2306. 04121, 2023.
[43] Jingfeng Yao. Xinggang Wang, Shusheng Yang, and Baoyuan Wang. Vitmatte: Boosting
image matting with pre-trained plain vision transformers. Information Fusion, 102:10 2024. ISSN 1566-2535. doi: https://doi.or/// 10. 101 20f. URL https
7/ww. www.ssciencediret.com com/ http///e/aaaattt////////////////
[44] Yao Yao. Jingyang Zhang. Jingbo Liu, Yihang Qu, Tian Fang. David McKinnon, Yanghai Tsin, and Lo @ pegrrceddigsmmtttibbaall22220000
author Martin-Brualla, Ricardo and Radwan, Noha and Sajjadi, Mehdi M. and Barron, Jonathan and Dosovitskiy, Alexey and Duckworth, Daniel, title NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collections, booktitle = CVPR, year=2021 ng Quan. Neilf: Neural incident light field for material and lighting estimation. In ECCV, 2022. URL hoLthtts::/ : /awpiv..ivv////////////a////aai org/ abs 2203. 07182
12

---

[45] Yao Yao, Jingyang Zhang. Jingbo Liu, Yihang Qu, Tian Fang. David McKinnon, Yanghai Tsin, and Long Quan. Neilf: Neural incident light field for physically based
based material estimation. In ECCV, page 700-716, Berlin, Heidelberg, 2022. Springer- Verlag ISBN 978-3-031- doi: 111077997833-03--1982222222 [40
[46] Hong-Xing Yu, Michelle Guo, Alireza Fathi, Yen- Yu Chang, Eric Ryan Chan, Ruohan Gao, Thomas Funkhouser, and Jiajun Wu. Learning object-centric neural scattering functions for free-viewpoint relighting and scene composition. Transactions on Machine Learning Research, 2023.
[47] Xiuming Zhang, Pratul P. Srinivasan, Boyang Deng, Paul Debevec, William T. Freeman, and Jonathan Barron. Nerfactor: Neural factorization of shape and reflectance under an unknown illumination. In ACMISIGGRAPII Asia Technical Papers, 2021.
[48] Quan Zheng, Gurprit Singh, and Hans-Peter Seidel. Neural relightable participating media rendering. In NIPS, 2021.
[49] Shizhan Zhu, Shunsuke Saito, Aljaz Bozic, Carlos Aliaga, Trevor Darrell, and Christoph Lassner. Neural relighting with subsurface scattering by learning the radiance transfer gradient. ArXiv, 0tw/2306.09322.. 2023.
[50] M. Zwicker, H. Plister, van Baar, and M. Gross. Ewa splatting. IEEE Transactions Visualization and Computer Graphics, 8(3):223-238, 2002. doi: 10.1 10.109/7VCC..20020101115576
13

---

A Appendix
A Camera Preprocessing Pipeline
For capturing the real-world datasets (Fig. 7 we use an Oryx 123S6C-C camera by Teledyne/FLIR. The camera has resolution of 4092 x 3000 pixels and uses a Bayer pattern to capture color. The white balance settings for the camera and light sources were estimated using an X-Rite Color Checker. We capture raw, single-channel 8-bit images to render the data transfer from the camera as fast as possible. To reduce noise we always capture 5 frames instead of one and compute the pixel-wise median. We remove fix- ix-pattern noise via dark- frame subtraction. We capture dark frame for every object the dataset. The median-filtered and dark frame corrected images are demosaiced using OpenCV [S] For every view direction, capture one image with all light sources the Light Stage active. We run a structure from motion (SfM) algorithm from COLMAP 34 on those uniformly lit images to get the precise camera positions. The objects were placed box with a random noise texture to aid feature detection.
median-filtered
Real-world Setup
Synthetic Setup
5X
Toiniights
Buah word Light Stago
Skander Light Stage
Preprocessing and Light Stage showing our real-world and synthetic data acquisition pipeline. For the real-world data, we present a sample of the all lights on images used for COLMAP (34) reconstruction. Further show segement anything [1] based automatic masking of the data (with failure case that we filter out for training). Note that all lights images not used during the training process. On the right, show the synthetic data acquisition pipeline and train and test split also applied to the weal-world data.
Figure 7:
B Translucent Object Dataset
Our datasets consist of 20 distinct objects from translucent material categories such as plastic, wax, marble, jade, and liquids. We captured 15 object in total with our light stage setup and processed them as described in Section AA] Further, we constructed 5 synthetic scenes using Blender [8] and rendered them synthetic light stage setup. Figure shows our entire dataset. Find the full dataset on our project page
Figure 8: Translucent Captured Objects highlighting subset of images from various camera and light poses
with all 20 objects of our datasct. The top left 5 are synthetic and the remainder shows real-world captures.
14

---

C Ablation
We conducted an ablation study to evaluate the importance each component of our method (Tbl. 2). The first variant of our method removes the SSS residual and solely relies our PBR shading model. As clearly visible (Tbl. 2 results degrade significantly, highlighting the impracticability relying the PBR shading model for translucent objects. We also compare the results of our full method against variant without the PBR shading model and variant without the deferred shading stage. The results show that the PBR shading stage together with deferred shading is crucial for the generation of high-quality images, as it enables the representation of specular highlights, Please note that the metrics used only have limited capability express the fine details in visual quality that the improved representation of specular highlights adds (Fig.
solely
Method
Category
PSNR
SSIM
LPIPS!
Ours
Full
36.16 + 3.76
0.982 + 0.010
0.027 + 0.006
Ours
w/o Deferred
35.61 + 2.28
0.981 0.009
0.027 0.004
Ours
w/o PBR
35.00 + 1.98
0.981 0.009
0.029 0.005
Ours
w/o Joint MLP
31.68 = 0.66
0.971 + 0.002
0.041 = 0.008
Ours
w/o Residual
30.23 = 2.28
0.954 0.015
0.056 = 0.026
R3DGS 10
w/ Inc. Light F.
24.81 = 5.29
0.926 + 0.055
0.087 = 0.044
Table 2: Component Ablation we show ablations on a subset the dataset consisting of two synthetic and two real world scenes, "Full" refers to the method used in the paper. We trained four variants of our method without crucial components "w/o Deferred", "w/o PBR". "w/o Joint MLP" and "w/o Residual". The last adds comparison of variant of R3DGS [0] which combines R3DGS with our incident light field.
Further ablations highlight importance the joint MLP. If we split the MLP for residual and incident light, inputting only the relevant physical properties, it leads to results. We believe this because, with the joint parameterization, the main MLP can learn about the global light transport, thereby providing valuable insights for the two output heads predicting the SSS residual and the incident illumination, respectively.
Full Mothod
Full Method
Ground
PBR
Figure 9: Shading Ablation highlighting the importance of each component of method and their difference compared against ground truth. First (left to right) "w/o PBR" (only residual and incident light field), second adding PER "w/o Deferred" and last showcasing full specular highlights and the least error by employing our full method "PBR=Deffered". We show results from individual training runs (not post-training toggling of components).
We also run experiments against variant of R3DGS 10 with known illumination. Please note that the original R3DGS approach only supports single illumination setting, therefore differing from our OLAT setting and not capable of learning to disentangle SSS and surface reflection. To still work within the OLAT setting we attached the incident light field prediction from our method to the R3DGS pipeline. The results in Table 2 indicate the limitations of the base model in our experiment setting. In summary, the ablation study demonstrates the importance of each component of our method and the necessity of the joint MLP for the prediction of the SSS residual and incident light field.
15

---

D Image Based Lighting
In Figure 10 we show that our method achieves image based lighting (IBL) with high visual quality. First, we sample the HDR maps representing distant environment illumination. The samples don't t need to correspond to OLAT samples used in training and could be placed much denser. Using this approach we can generate a relit frame about 20 seconds for a medium resolution. To speed up generation we can precompute a reflectance field assuming white light. We compute the relit view as the sum over the reflectance field scaled by the environment illumination before applying tone mapping for display. This runs in fraction of second even with naive implementation. As be seen in Figure T0 an object can be rendered in different illumination settings [4] yielding consistent photorealistic results.
Indone
Woodde
Snow
Environment
Frant
Dunny
Stcck
Poont
Soap
Back
Fromt
Cande
Blakk
Front
Car
Slack
Figure 10: Image Based Lighting (IBL) results of our method, with an OLAT sample in the top row for comparison and three different environment maps used on two synthetic (bunny soap) and two real-world (candle/ / car) objects.
We also compared quantitatively against Blender renders for chosen environment maps. For fairness, we used the same sampling of the environment map as point lights as in our method. As can be seen in 33 the relighting results closely resemble the ground truth. Note, that the PSNR metric is affected by denoising artifacts from Blender and the noise residual from the Monte-Carlo path tracing in the ground truth data.
Environment Map
PSNR
SSIM
LPIPS
Average
31.62 + 2.31
0.974 0.012
0.049 0.018
Indoor
30.24 + 1.06
0.971 0.0133
0.052 0.019
Woods
34.36 1.37
0.978 + 0.009
0.044 + 0.017
Snow
30.28 + 1.31
0.971 = 0.013
0.050 + 0.019
Table 3: IBI. Quantitative comparison of relighting our five synthetic scenes against Blender S renders using environment maps from Figure above.
Intrinsics
We further evaluated the intrinsic properties (Fig. referring to albedo and illumination as common properties of intrinsic image decomposition. While our base color parameter can be understood as the albedo we also output additional material parameters that are needed for our physically based rendering model. We show base color, roughness, metalness, normal, sss residual, specular diffuse components and again the final render. Specular and diffuse illumination are intermediate results in our rendering pipeline during the deferred shading stage representing the diffuse and specular illumination components (before multiplication with the base color), respectively.
16

---

Centar
Figure 11: Intrinsics Maps rendered from Blender compared to ours with absolute difference at the bottom. Blender does not provide SSS intrinsics. Therefore, the residual here is the difference of the diffuse light rendered with SSS and off. Find in Tab. 4 results for all synthetic scenes.
Type
Base Color
Roughness
Metalness
Normal
Residual
Specular
Diffuse
Render
Bunny
0.152
0.037
0.019
0.018
0.018
0.017
0.163
0.055
Dragon
0.077
0.141
0.045
0.366
0.039
0.022
0.074
0.190
Statue
0.119
0.038
0.026
0.049
0.016
0.013
0.159
0.065
Soup
0.164
0.047
0.031
0.203
0.012
0.013
0.081
0.061
Candle 0.080
0.042
0.014
0.015
0.032
0.013
0.090
0.065
Average 0.119
0.061
0.027
0.130
0.023
0.016
0.113
0.087
Table 4: Intrinsic Comparison of Blender renders against ours. We report RMSE for all synthetic measurable scenes, subset of 25 renders with 5 different camera and light poses, and "Average" over all of our synthetic scenes.
All selected properties are compared to ground truth properties obtained using the Cycles renderer in Blender [8] for our synthetic scenes. We want to note that Blender uses different shading model, such that some of these properties are not directly equivalent. Most notably, the SSS residual cannot be retrieved and is calculated by as the difference of diffuse reflection from rendering with SSS turned on and SSS turned off. In Figure II we plotted the absolute difference and properties. For all of these intrinsics, we calculated the RMSE, see Table 4 While achieving overall good results, especially for the illumination, the value of the quantitative evaluation is limited by the fact that the optimization can generate multiple plausible solutions for given appearance due to the under-constrained problem space. This is also the reason why base colors might differ.
Training Details
In this section, provide details the training of model, including the loss terms, network architecture, and optimization details. For training, we use the PyTcch framework and train on a single NVIDIA RTX 4090 GPU with 24GB of memory. Our code is build upon the 3D Gaussian Splatting (3D GS) [8] and Relightable 3D Gaussians (R3DGS) (10) codebase. For further details on the training pipeline, we refer the original papers and our codebase, which you can access via our project page
Losses We have multiple loss terms in our training pipeline that are mainly adapted from R3DGS [10] that we will briefly outline them and their weighting here. As in 3D GS (18), we utilize a L1 loss and perceptual loss LISIM comparing the predicted and ground truth images. Both of those are combined with Ling (1.0 Adssim L2 + A1ssim LSSIM) where Assim = 0.2. We further optimize for the perceptual quality of the images by employing a LPIPS loss that is weighted with Atpaps = 0.2. Taken from R3DGS [10] we utilize MSE between pseudo normals calculated from the depth map and the predicted normals, scaled with Apammats 0.02. Special to our work we constrain the clamped predicted incident light to be close to the evaluated visibility with an L loss and Aincident = 0.02. This helps to avoid evaluating to zero in rare cases and thus only utilizing the ML. for the prediction. As in R3DGS penalize the creation of Gaussians outside the masked original image
17

---

space by using mean entropy loss Lmask mmaaak log( opacity) + (1 - mask) ) log 1 - opacity) ) with Amask 0.1. We further employ R3DGS bilateral smoothing losses for the predicted metalness, roughness, subsurfaceness and base color with Asmooth [0.002, 0.002, 0.002, 0.006). Additionally, we use the highlight and shadow enhancement loss from R3DGS with Aenhance 0.005 and the raytraced visibility learning that is weighted with Amytrace = 0.01.
Network As outlined in Figure 2 we use shallow MLP to predict the subsurface properties and
incident light of the Gaussians. Our MLP accepts an input having (16 24) feature dimensions,
with normals (3), rotation (2), scale (3), light direction (3), view direction (3), light distance (1) and visibility (1). The position (3) is encoded using positional Fourrier encoding [27] resulting in features. The input is fed into a 3-layer MLP with [64, 32, 32] neurons in them followed by Leaky-Rell activation functions. The output is fed into two output heads. One of them predicts the incident illumination with another layer of 32 neurons and a Leaky-ReLU and ReLU output layer, while the other one computes the SSS component of the radiance with sigmoid output layer.
Optimization The per-Gaussian position μa, covariance as rotation q and scale s, opacity o, basec- olor Cbase, metalness m, roughness normal n, light visibility as Spherical Harmonics and the newly introduced subsurfaceness 888 are optimized together with the network weights for the base MLP and the two output heads for SSS radiance and incident radiance, respectively. We use the ADAM 20 optimizer with default parameters and a learning rate 0.001 with an exponential decay of 0.99 every 1000 steps. We train for 60k steps although we observe that the model already receives good results after 30k steps. Further, we follow the default splitting and pruning schedule proposed by the original 3D GS.
Scheduling We schedule the training of the individual 3D Gaussian properties with the original provided exponential learning rate decay function. For our MLP we also use an exponential decay function with gamma of 0.9999. Further, we schedule the incident light by linearly incrementing the incident light up until 7k steps and then removing the constraint. This helps to stabilize the training and prevent the MLP from predicting zero incident light in the beginning. We also freeze the optimization of the roughness and set value of 0.5 for the first 10k steps stabilize the training.
18

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: To the best of our knowledge we present the first method to add support for SSS to Gaussian Splatting. The abstract and introduction introduces all relevant tasks and the key contributions. Our evaluation shows that our method design achieves all introduced tasks, relighting, novel view synthesis and material editing, and reaches the claimed performance variety of tested metrics for view synthesis. Also the runtime has been compared to previous works and the numeric results support the claims.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: In Section 5 we discuss limitations of the proposed method in terms of editing capabilities (e.g. fixed geometry) due the chosen representation as well quality limitations from the choice of rendering algorithms. Also potential ethical and privacy concerns are mentioned.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best
19

---

judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: In Sec. 3 we introduce the background of methods we build upon and derive our rendering model based on the rendering equation. All components and their connections are laid out in the individual subsections.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
All steps of the proposed pipeline described in Sec. B and the experimental setup introduced in detail Sec. To exactly reproduce the results some prior knowledge about the methods in the background subsection is assumed and some hyper parameters either need to be fine-tuned based on the data or looked up in the appendix section.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
20

---

reproducible, they (c) If the contribution is new model (e.g.. a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: At this point in time we cannot provide open access to code and data. However, it is our clear intend to release the data acquired for this work together with all codes needed reproduce the results and use the data for future work. At the time of writing suitable hosting infrastructure for the dataset needs to be established which would disclose the authors.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://niis.cc)
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
//nips. //nips.c/ppul
swssssiiicciiiii for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
The scheme to create dataset splits as well as the optimization parameters are outlined in Sec. [ We also specify the parameters used for the different comparisons, e.g. when using resized or subsampled versions of the dataset.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
21

---

7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We add standard deviation for the results on novel view synthesis over the objects in the datasets for all metrics reported in the quantitative results section. We also ensure comparability of the methods by carefully matching the experiment conditions in terms of data sample size and resolution. We make sure the same hardware to measure timings.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: As part of Sec. 4 we extensively compare runtimes of method against previous works. GPU resources are also specified. The needed system memory might change from dataset to dataset but is in the range of the mentioned baseline methods.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: /ttn:://neriiie httppppc.ccppllccttthccccceeeee.
Answer: [Yes]
22

---

Justification: The authors have read the NeurIPS code of ethics. We mention anticipated risks that follow up work could potentially encounter as part of Sec. 5 Our experiments do not include humans and the content of the dataset generation is licensed accordingly. We acknowledge that there bias in the selection of objects included the dataset based on geographic availability etc. but try to compile a diverse presentation of material attributes independently object semantics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We mention some positive impacts like better quality entertainment, photore- alistic representation for education in Sec. Tas well as potential limitations and negative societal impacts in Sec.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative socictal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploymeet of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: Our scope of application does not have a high risk of potential misuse as it is aimed image rendering household objects. For the dataset generation we hand selected objects and 3DD models taking into account potential ethical implications and connotations.
Guidelines:
The answer NA means that the paper poses such risks.
23

---

Released models that have high risk for misuse or dual-use use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (c.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: In addition to the related works mentioned in Sec. 2 we mention all methods and modules we build our approach on as part of Sec. and Sec. In Sec. we cite the sources of the 3D models used in our synthetic dataset.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version the asset used and, possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. cow/coo////oooooo datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-par se-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: As there is no dataset or code submission part of the submission at this point this is not relevant. Still, we document the process of dataset acquisition that will be released at a later time to level of detail that does not disclose the authorship in Sec. 4
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/moodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?

---

Answer:
[NA]
Justification: No crowdsourcing experiments or research with human subjects has been conducted in the context of this work.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: No research with Human Subjects has been conducted in the context of this work.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such as the institution conducting the review.
25


Reviews:
[{"review_id": "Bf5tmQXxvX", "reviewer": "Reviewer_pNBB", "summary": "This paper proposes an algorithm to reconstruct relightable objects with subsurface scattering (SSS) effects. It proposes to model SSS as a residual to surface PBR using a neural network. It also proposes to perform shading in image space (i.e. deferred shading) to improve specularity. Lastly, the SSS network takes ray-trace light visibility as input, accounting for shadowing effects for the SSS component.", "questions": "1. What prevents the method from working without knowing point-light?\n\n2. Do the results in Tab. 1 include novel lighting conditions? How are light intensity set in both synthetic and real dataset? How would the method work if the light intensity changes drastically, i.e. can the model handle light intensity changes from e.g. 5 (training) to 20 (testing)?\n\n3. Only point light is demonstrated in the results. Would the model still work under natural illumination (e.g. image-based lighting)? According to Eq. 6 it seems to be designed specifically for a single point light, which is quite limiting. Would be good to see both qualitative and quantitative results on synthetic datasets with image-based lighting", "limitations": "Limitation is discussed, though I think a crucial part (not supporting image-based lighting) is missing. Please see questions.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Modeling SSS is an under-explored research area in inverse rendering, and the demonstrated results in this paper look promising.\n\n2. Using a neural network to predict residuals to surface-based PBR is straightforward yet effective, as demonstrated in the paper.", "weaknesses": "1. The approach assumes known point-light, which limits its applicability in real-world scenarios\n\n2. The novelty is overclaimed: GS-IR ([19] in the paper) has already introduced deferred/pixel-space shading to Gaussian splatting. There are also several concurrent works reaching the same conclusion as this paper (e.g. deferred shading improves specularity) [1, 2], please cite them and discuss the relations between them and this paper in the final version of this paper. However, I do acknowledge that this paper is one of the first papers that argues deferred shading improves specularity.\n\n3. Insufficient ablation: only qualitative results on ablation are presented. It would be preferable to have quantitative results on the ablation baselines. Also, the comparison to relightable 3D Gaussians is not fair. A more fair comparison is to remove the SSS part of the proposed method, which becomes a variant of relightable 3D Gaussians that can handle known illumination.\n\n\n[1] Ye et. al. 3D Gaussian Splatting with Deferred Reflection, arXiv 2024\n\n[2] Wu et. al. DeferredGS: Decoupled and Editable Gaussian Splatting with Deferred Shading, arXiv 2024", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Optimization under Unknown Illumination \n> Question 1  \nOptimization with unknown point light locations adds additional dimensions to an already under-constrained optimization problem which we consider out of scope of this work as we focus on the geometry representation and rendering part.  \nAs of interest for multiple reviewers we discussed the advantages of OLAT data for the reconstruction of subsurface scattering (SSS) in the global response. Opening up the illumination to more complex patterns and natural illumination eventually is an interesting research trajectory that we plan to pursue in future work. To successfully reconstruct the subsurface scattering component of the light transport it is necessary to understand the global light transport in the scene including the heterogeneous volume structure of an object. In theory our method supports any representation of the incident light field. Assuming an unknown illumination this light field would need to be estimated in addition to the light transport, though. This adds additional complexity to the optimization problem. New priors on either illumination or geometry would be needed here which could be potentially obtained with the help of our acquired dataset. However, given the overall complexity of the problem we decided to focus on the geometry and material representation aspect first and consider improvements of the illumination representation beyond the incident light prediction as out of the scope of this work.  \nStill, our work is robust to deviations from the actual light position and potentially could also optimize an offset on the point light positions without much additional effort.\n\n### Concurrent Works on Deferred Shading for 3DGS\n> Weakness 2  \nWe thank the reviewer for hinting at existing deferred shading techniques in the realm of Gaussian Splatting and we will gladly incorporate these in the related work section. The motivation to use a deferred shading pipeline is the limited capability of the shading approach of R3DGS [9] to reproduce specular highlights. See Fig. [3] for a visual comparison.\nWhile Ye et al. [44] and Wu et al. [45] focus on the blending and propagation of normal directions between overlapping 3D Gaussians, our analysis based on the 3D Gaussian areas adds to the understanding of these effects. [45] train a SDF in parallel to improve the surface geometry which we do not need to. Also our shading model differs from the existing ones as we add the explicit SSS Residual term to the PBR material parameters and predict detailed incident light. \n\n\n### Novel Lighting in Tab. 1\n> Question 2  \nThe values shown in Tab. 1 (Paper) include novel lighting positions, we build a train and test set with a 50/50 split for the real world and synthetic dataset (see Sec. 4.1 Paper). Still, these light positions are part of the lightstage, therefore, have the same distance to the object. For truly new light positions we would like to refer to the videos in the supplementary material and Fig. 2 in the rebuttal. Varying intensity or colored light was not available within the test set but we now added a comparison for the synthetic scenes in Tab. 1 and Fig. 1. Illumination editing is still possible as described next.  \n\n### Illumination Editing including Light Intensity\n> Question 3  \nIn Fig. 2 we present various editing modes our method enables. Light intensity can be changed by scaling the incident light field and residual prediction or the linear output of the deferred shading stage. As we only train with a fixed light spectrum and intensity extreme edits will result in inaccurate results for the residual component, though. Extending the setting to a spectral subsurface scattering model e.g. by training on different spectral responses in addition to the white lights is an interesting idea for future work. We thank the reviewer for the inspiring comment.\n\n### Image Based Lighting \n> Question 3  \nIndeed our method is capable of achieving image based lighting. As this was of interest for multiple reviewers we would like to refer to the global comment as well as Fig. 1 for qualitative and Tab. 1 for quantitative results. As can be seen the relighting results closely resemble the ground truth renders regarding color, light direction and subsurface scattering. Note, that the PSNR metric is affected by denoising artifacts and the noise residual from the Monte-Carlo path tracing in the ground truth data.\n\n### Insufficient Ablation\n> Weakness 3  \nAs introduced in the global response on “Comprehensive Evaluation and Ablation” we now provide qualitative and quantitative results of multiple ablations. In Tab. 2  and Fig. 3 can be seen that the residual prediction, the PBR rendering and the deferred shading component all contribute to the quality of the final results. Our model is capable of representing the SSS component and specular materials and also enables intuitive editing of the material parameters as shown in Fig. 2.\n\n### Comparison against Relightable 3D Gaussian Splatting (R3DGS)\n> Weakness 3  \nIn Tab. 2 we also compare against a variant of R3DGS [9] with known illumination. Please note that the original R3DGS approach only supports a single illumination setting, therefore differing from our OLAT setting and not capable of learning to disentangle SSS and surface reflection.   \nCompared to R3DGS we use a different light representation that models the changing illumination with explicit point light locations compared to the NeILF [37] based approach in R3DGS. The results in Tab. 2 clearly indicate the limitations of the base model in our experiment setting.  Our approach together with the OLAT data enables higher frequency illumination and more direct control of the lighting compared to the Spherical Harmonics based representation of R3DGS.\n\n> 44: Ye et. al. 3D Gaussian Splatting with Deferred Reflection, arXiv 2024  \n> 45: Wu et. al. DeferredGS: Decoupled and Editable Gaussian Splatting with Deferred Shading, arXiv 2024", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QnHvja1aa5", "reviewer": "Reviewer_wfKX", "summary": "This paper proposes the 3D Gaussian Splatting for subsurface scattering objects by decomposing the scene into subsurface scattering, diffuse and specular reflections, and object shape. By using multi-view OLAT (one light at a time) data of translucent objects, the proposed method optimizes BRDF parameters attributed to Gaussians and a small MLP that outputs subsurface scattering radiance and incident light. For accurate rendering, this paper proposes to incorporate deferred shading into 3DGS for specular highlights and explicitly handle shadowing by considering the visibility of Gaussians. The experimental results demonstrate the successful decomposition of the PBR parameters and the effectiveness of using 3DGS compared with the NeRF-based method.", "questions": "- A clearer presentation of the key idea and its superiority beyond the just combination of the existing works (NeRF-based method and relightable 3DGS) is expected.\n- Why does the MLP in Eq. 6 take the covariance and normal of a 3D Gaussian as input? The outputs may not depend on them.", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": 4, "presentation": 2, "contribution": 3, "strengths": "+ The first method for handling subsurface scattering objects with 3DGS and enabling the decomposition of subsurface scattering, diffuse and specular reflections, and object shape of translucent objects.\n+ Introduce deferred shading into 3DGS to enable accurate rendering of specular highlights.\n+ Experimentally show that the proposed method successfully decomposes the PBR parameters and significantly outperforms the NeRF-based method for novel view synthesis, training time, and rendering speed.", "weaknesses": "- The representation of subsurface scattering is almost the same as the existing NeRF-based work [39], which takes points, viewing direction, and light direction as input of an MLP. The main difference is to simultaneously estimate incident light and use it for the physically-based rendering of specular and diffuse reflections like [9]. Although this difference improves the quality of rendering and enables the decomposition of PBR parameters, this method seems a combination of the existing works.\n- The incident light $L_{in}$ can be view-dependent, which is physically inaccurate.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Limited Novelty due to Recombination of Recent Techniques and Similarity to NeRF-based Approaches\n> Weakness 1 & Question 1  \nTo enable the first method for reconstruction of translucent objects that enables relighting and material editing we \n> - have performed multiple substantial modifications on the R3DGS baseline (Tab. 2),\n> - contribute a novel hybrid representation that has an architectural bias for decomposition and global light transport understanding,\n> - created and will release an open dataset of OLAT scenes featuring diverse translucent objects.\n>\n>Compared to the NeRF based approaches [42, 39] our hybrid representation features explicit surfaces modeled by the 3D Gaussians and a novel parameterization of a global MLP for the volumetric part.  \nOur reasoning for the residual setup is presented in “Clearer Presentation of Key Idea and Contribution” of the global response.\nThe key insight here is that a single MLP backbone together with two heads for volumetric residual and incident illumination prediction, respectively, performs best in our data setting (Tab. 2). Please also refer to the following paragraphs on the parameterization of the MLP and the incident light prediction. We will rework the relevant sections in the paper to make this reasoning clearer.  \nIn addition to the speed up that 3DGS [15] brings through point based rasterization, we identify the advantage of the strong surface prior from 3DGS for our decomposition task as a key difference compared to existing NeRF based approaches. Our formulation of a surface based shading with a volumetric residual to represent the subsurface scattering component is well suited for many editing applications and leads to a better geometry reconstruction compared to NeRF based approaches as well as a subsurface scattering representation much closer to the physical model than the plain R3DGS [9] (see the examples in section 4.2 comparison of the paper and the new ablation results as part of this rebuttal, specifically Fig. 4 and Tab. 2 and 3).  \nWe find that R3DGS’s capability to render specular reflections is limited by the size of the 3D Gaussians and find a straightforward solution by using a deferred shading pipeline.  \nFurthermore, we adapt the framework for the OLAT setting (also see “Choice of OLAT Data and Relighting” in the global response), replacing the global environment map and the local illumination representation with a point light based representation. This enables higher frequency illumination and more direct control of the lighting compared to the Spherical Harmonics based representation of R3DGS.  \nFinally we abandon the two stage optimization approach from R3DGS and propose a single optimization stage without the need of initialization with a sparse point cloud that might be hard to obtain for translucent objects. \nSince we could not find a suitable open dataset we also contribute a collection of OLAT scenes with translucent objects as part of this work in the hope that this will enable follow up work in this field.\n\n### Parameterization of MLP (Eq. 6)\n> Question 2  \nEq. 6 accurately reflects the implementation, meaning that the MLP does indeed take covariance (rotation and scale parameter) and normal of a 3D Gaussian as input. The original intent was to condition the MLP on all surface information we have from the 3D Gaussian and to possibly receive an additional gradient update on the normal parameter.\nInspired by the reviewer’s comment we ran additional experiments and it turns out that conditioning without normals and covariance  parameters is sufficient to achieve the reported quality. \n\n### Incident Light Prediction is Physically Inaccurate\n> Weakness  2  \nThanks to the reviewer’s feedback we realize that the notation $L_{in}$ in Equation 6 in the paper might be a bit misleading in the context of physically-based rendering. We will update the section in the paper to make it clearer that $L_{in}$ is a prediction of our model that is close to the true physical quantity but might be slightly offset to compensate for limitations in the model.  \nWhile the incident light is independent of the view direction, the residual prediction is not. Using a single MLP backbone together with two heads for volumetric residual and incident illumination prediction, respectively, performs best in our experiments (see Tab. 2 row “w/o Joint MLP”). The hypothesis here is that the processing inside the MLP includes reasoning about the global light transport inside the object that is inherently constrained by the prediction of the incident light field (which needs to work for the PBR renderer). By relaxing the strictly physical model a little we gain better quality output, potentially compensating limitations of our rendering model.\nPlease also see the global response on “Clearer Presentation of Key Idea and Contribution” for further notes. \n\n> We want to thank the reviewer again for the thorough analysis and hope that the key ideas of our method became clearer through this rebuttal. As mentioned above we will rework the introduction and method section of the paper to reflect the reviewer’s feedback. We will also add the additional analysis provided in the accompanying PDF to the supplementary section. We invite the reviewer to also take a look into it as it illustrates the effects of the explanations above.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "P1PCCzYaXH", "reviewer": "Reviewer_F9dx", "summary": "This paper presents a method for recovering the shape and radiance transfer field (RTF) of an object from multi-view, OLAT data; placing an emphasis on translucent objects that exhibit subsurface scattering (SSS). In particular, the authors extend the framework of Relightable 3D Gaussian [Gao et al. 2024] in two ways: (1) instead of estimating per-Gaussian physically-based BRDF parameters, shading is performed in image space with a neural field that produces (potentially distinct) BRDF parameters for every pixel (backprojected into 3D); (2) in addition to predicting an incoming light field, an MLP predicts residual radiance due to SSS (i.e. a radiance transfer field that accounts for SSS alone), conditioned on 3D point, viewing direction, light source direction, and light source visibility.\n\nIn order to validate the efficacy of their framework, the authors collect a large multi-view OLAT dataset of translucent objects. With quantitative/qualitative comparisons on this dataset, the authors show favorable performance against existing state of the art works for shape and RTF estimation.", "questions": "* Why do the results of KiloOSF look so bad compared to their results on their own dataset? What is the source of the voxel artifacts in the rendered images?\n* Can you go into more detail on the conditioning of the outgoing SSS component and the incoming light? As written, it seems like the incoming light is conditioned on the outgoing view direction, which I don't think should be the case?", "limitations": "The authors adequately discuss the method's limitations and potential negative social impact.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "One strength of the work is the multi-view, OLAT dataset, which the authors say they plan to release. Containing 15 objects, with 100s of views and light positions (over 25000 images per object), this dataset seems like it took considerable effort to collect. I imagine it will prove quite useful in benchmarking approaches that perform shape/material estimation and relighting, esp. for translucent objects. \n\nThe quality of the results appears to be good (the qualitative results in the paper and supplement look reasonable), and quantitative results are strong compared to existing methods.", "weaknesses": "While the paper is, for the most part, well-structured and easy to follow, I found parts of the presentation to be slightly substandard. The pipeline figure is essentially just a collection of text boxes. A clearer delineation between network input, encoding, network architecture, network output (as well as an illustration of the purpose of each output), would help improve the clarity of this figure. I also don't think that the model design shown in this figure is fully accurate -- roughness, base color, and metalness should be position-dependent quantities only and should not depend on incoming/light direction. \n\nI would also have liked it if the qualitative results were slightly better organized. I thank the authors for providing a number of renders of each object in the dataset under novel lighting/view. Collecting the results in a web page with labels for each video would have made the results easier to navigate. \n\nI was surprised that so few visual comparisons to existing approaches are shown (Fig. 5 only).The KiloOSF results look far worse than I would expect -- the voxel artifacts seem to indicate a bug in evaluation/training. \n\nFinally, while I don't doubt that the authors have built an effective system for shape and RTF recovery, I question the novelty of the contributions that enable this system. Performing deferred shading in pixel space for forward rendering/inverse rendering is hardly a new idea. Leveraging neural fields to predict RTFs (even residual RTFs) has also been done before in the literature (e.g. in Neural Fields for Structured Lighting [Shandilya et al. 2023]). Self-supervised visibility, the third listed contribution, is part of Relightable 3D Gaussian. Given, then, that the argument seems to be that a novel *combination* of these tools enables a better-performing system I would've hoped to see more comprehensive / higher quality evaluation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Improvements of Pipeline Overview (Fig. 2)\n> Weaknesses  \nThanks for the valuable feedback. We realize that the clarity of Fig. 2 in the paper can be improved. Please see Fig. 5 in the PDF for an updated version. We now tried to make clearer that base color, roughness and metalness are properties of each 3D Gaussian independent of incoming and light direction.  \nSee below as well as the paragraph on “Clearer Presentation of Key Idea and Contribution” in the global response for more information on the conditioning of the residual MLP.\n\n### Conditioning of the SSS Residual and Incident Light Prediction\n> Question 2  \nWhile the incident light is independent of the view direction, the residual SSS prediction might depend on it. \nIn our proposed setup we jointly optimize the SSS Residual and incident light prediction.  \nWe find that a single lightweight MLP jointly conditioned on the 3D Gaussian parameters as well as the view direction and the ray traced visibility yields high quality results at interactive rates for the given task. See Tab. 2 “w/o Joint MLP” for comparison, where we split the MLP for residual and incident light inputting only relevant physical properties, leading to worse results. Our reasoning is that with this parameterization the main MLP is able to learn about the global light transport and therefore is capable of providing useful hints for the two output heads predicting the SSS Residual and the incident illumination, respectively. Both predictions are inherently constrained by the prediction of the incident light field which needs to drive the PBR rendering. However, the residual prediction can compensate for some limitations of the rendering models based on the provided input. As can be seen in the results in Fig. 4  the output is still a physically plausible material parameterization that enables multiple downstream applications like relighting or material editing.\n\n### Combination of Tools\n> Weaknesses  \nOur goal is to represent objects featuring translucent materials to be able to relight them or change individual material parameters. The focus is on enabling correct rendering of specular highlights and decomposition of the subsurface scattering component. This cannot be achieved with the existing methods and is a unique application at the time of writing.  \nWhile we build on existing tools like 3DGS [15] and R3DGS [9] and ideas like deferred shading [11] we argue that our method is more than a combination of existing ideas.  \nOur hybrid representation introduces a MLP for residual prediction of the subsurface scattering component which is constrained by the incident light prediction as architectural bias (as outlined above). While works like [46] show that MLPs can be used to represent the light transport also in complex scenes, our particular parameterization has not been proposed before as far as we know.\nPlease also refer to “Clearer presentation of key idea and contribution” in the global section for further discussion of the residual prediction and MLP parameterization.  \nMoreover, our work includes multiple modifications of the R3DGS framework like the support for OLAT illumination and the single stage optimization. The results in Tab. 2 also state that only the combination of all our proposed components achieve such high quality results.  \nFinally, we also contribute a collection of OLAT scenes featuring translucent objects together with a preprocessing pipeline in the hope that this will enable follow up work in this field. \n\n### More Comprehensive / Higher Quality Evaluation\n> Weaknesses  \nPlease find additional ablations in the newly added Fig. 3 and 4 as well as Tab. 2. We also demonstrate the editing applications our approach enables in more detail in Fig. 2. Please also refer to the global response on “Comprehensive evaluation” for more notes on the evaluation results. Given the novelty of the proposed editing applications there is no method to directly compare against. For a fair comparison we select KiloOSF [39] as the only real-time method that also enables relighting and novel view synthesis. We assume that the NeRF based methods can achieve similar quality, however at the cost of a much higher optimization and inference cost that makes comparisons on the full dataset unattractive.\n\n### KiloOSF Results\n> Question 1  \nWhile we did not directly analyze the voxel artifacts, we speculate that they are the results of the voxelization technique used in KiloNeRF and the complexity of the task gives the sparse views and light positions. The results shown in their paper [39] are produced by the default OSF NeRF approach training for multiple days and only achieving 0.27 FPS during inference, for fairness we choose to compare against their KiloOSF variant. There is no visual analysis shown for their real-time approach KiloOSF which achieves 14 FPS, that we could compare against. Therefore we don’t have a qualitative ground truth of the KiloOSF results. We utilized their official implementation and documentation at https://github.com/yuyuchang/KiloOSF and also saw similar artifacts using their own dataset that got worse when using non opaque objects. Our quantitative analysis shows similar results as provided within their work hinting these are the correct results. \n\n### Webpage as Mode of Presentation\n> Weakness  \nWe thank the reviewer for the valuable feedback and taking the time to look into the supplementary material. We will launch a webpage including interactive viewing of the results together with the publication of the work and the dataset. \n\n> Again, we thank the reviewer for the valuable feedback. We hope that through our rebuttal it becomes clearer how our approach  uniquely enables accurate rendering and material parameterization for translucent objects as evidenced by new evaluations that will be added to the final paper.\n\n\n> 46: Shandilya et al. Neural Fields for Structured Lighting, ICCV 2023", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FBkMJcmhsp", "reviewer": "Reviewer_uYWZ", "summary": "This paper aims to model the subsurface scattering (SSS) effects under the 3D Gaussian Splatting framework, which is an efficient 3D representation for novel view synthesis. The challenge of this SSS modeling is the complicated light path in the final rendering output. The proposed framework is based on Relightable 3DGS but proposes an MLP-based module to capture SSS effects for 3DGS. It improves the representation of specularity by performing shading in image space. The experimental results achieve comparable or better results and fast optimization and rendering on synthetic and real-world data compared with NeRF-based approaches.", "questions": "1. What is the performance of Relightable 3DG / 3DGS?\n2. Eq. 6 overfits the experiment setting. Thus, it is hard to say the representation capabilities of the MLP. If the incident lighting in Eq. 6 is predicted and fixed by the MLP, how could the proposed method achieve relighting, like changing the intensity?", "limitations": "The authors have mentioned the limitations and potential negative societal impacts of their framework in Sec.5.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. Introduce SSS residual to the 3DGS pipeline, which achieves comparable results and fast rendering speed.\n2. Propose an image-space deferred rendering to capture specularity.", "weaknesses": "1. Some contributions may be overclaimed:\na. Editability. It was not demonstrated in the paper. Moreover, some editability comes with the Relightable 3DG, thus not the contribution of this paper.\nb. The relighting effects seem only to change w.r.t the light direction.\nc. Shadowing effects are handled by Relightable 3DG, thus not the contribution of this paper.\nd. It is unclear how to make applications to other fields as mentioned in line 60, since this paper makes various assumptions like a single light source and multi-view captures.\n2. Insufficient evaluation. As the method is based on Relightable 3DGS, it would be better to compare the proposed framework with Relightable 3DGS and 3DGS in quantitative results on novel view synthesis in Table 1.\n3. The experiment settings are constrained: a single, known light source, object-centric, and given object masks. Moreover, the backbone model Relightable 3DG also requires estimated depth, which is not discussed in the paper.\n4. Some typos and unclear sentences: line 232 about the number of images; Table 1 caption \"non quadratic image\". Line 298: sentence fragment. line 22 unclear sentence, etc.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Generalization of MLP / Representational Power of MLP\n> Question 2  \nWe add application examples that show interpolation capabilities in both the viewpoint as well as illumination domain. Results on the test set (Fig. 5 in the paper, for example)  are outside of  the training setting in terms of light and camera position. This can be best observed in the videos included in the supplementary material where also the light and camera distances are varied compared to the training.  \nThe global MLP is chosen specifically for its power to interpolate on the trained manifold similar to how NeRF [24] uses it. \n\n### Additional Relighting Examples\n> Question 2, Weakness 1b  \nPlease find qualitative and quantitative results for the image based lighting application in Fig. 1 and Tab. . We also show additional relighting results together with the other editing capabilities in Fig. 2. As can be seen the object can be rendered in different, both indoor and outdoor illumination settings yielding consistent photorealistic results. Please also refer to the global response for additional notes on the relighting examples and the method of computing them.\n\n### Editability \n> Weakness 1a  \nWe specifically focus on enabling intuitive and fast editing of the illumination and material properties of a reconstructed scene. In Fig. 2 we show examples of the material editing modes in detail. Fig. 1 shows new examples of relighting using environment lighting.\nAs this topic was of interest for multiple reviewers we added a paragraph on “Editing Applications” in the global response that we invite the reviewer to take a look at, too.\n\n\n### Limited Applicability to Other Domains due to Constraint Acquisition Setting\n> Weakness 1d, Weakness 3  \nThere are multiple application settings where a constrained acquisition setting like the OLAT approach might be tolerable and materials with subsurface scattering are relevant.  \nIn “Choice of OLAT Data and Relighting” of the global response we discuss the choice of the experiment setting. For example, in the games and movie industry a large effort is put into asset generation, often employing custom build scanning devices or light stages [https://home.otoy.com/capture/lightstage/] very similar to the one used in our dataset acquisition. Scans of individual objects that can then be composed and combined into larger scenes. Our relighting capability enables easy integration of objects into new settings. For scenes with multiple objects only the visibility attributes need to be updated as has been shown in [39].  \nAlso in the medical imaging domain it is feasible to perform a scan of e.g. an organ in a controlled environment to construct a detailed model that is then used for supervision during surgery, for example [47]. In this context also an endoscope with a point light source could be well represented using our model [48].\n\n### Comparison against R3DGS and 3DGS\n> Question 1 & Weakness 2  \nAdditional results can be found in the newly added Tab. 2 as R3DGS (with incident light field). The results are obtained from a base configuration of our model without residual prediction and without deferred shading which comes closest to R3DGS.  \nThe original R3DGS approach only supports a single illumination setting which is fundamentally different from the OLAT setting with many different illumination configurations. See the paragraph on “Choice of OLAT Data and Relighting” in the global response for more information about the setting and its relevance to subsurface scattering estimation. We use a light representation that models the changing illumination with explicit point light locations compared to the NeILF [37] based approach in R3DGS. The results in Tab. 2 clearly indicate the limitations of the base model in our experiment setting. \nPlain 3D Gaussian Splatting (3DGS) also assumes a static illumination. Hence, we choose to not compare against 3DGS as these differences would overall limit the value of the comparison.\n\n\n### Shadowing Already Handled by R3DGS\n> Weakness 1c  \nAs the reviewer correctly points out, the novelty here is not the handling of shadows but the way they are handled. Detailed shadowing effects can be predicted by our incident light prediction that is conditioned on the ray-traced visibility map on the 3D Gaussians.  \nCompared to R3DGS we use a different light representation that models the changing illumination with explicit point light locations compared to the NeILF [37] based approach in R3DGS. Our approach together with the OLAT data enables higher frequency illumination and more direct control of the lighting compared to the static Spherical Harmonics based representation of R3DGS.\n\n### Mask & Depth Input and Initialization Scheme\n> Weakness 3  \nOur automatic mask generation only adds minimal overhead during preprocessing and generates masks based on either text prompts or control points (see section 4.1 real-world dataset for additional examples composited on black). Depth input is not needed for our pipeline. As we have a larger number of input images available the quality of the normals is sufficient without depth supervision. Normal estimation is additionally regularized through the deferred physically based renderer in agreement with [44]. Similarly, we don’t use the sparse point cloud from the SfM reconstruction as initialization (as is being done for R3DGS). Finally, we do not require a two staged training scheduling, we only use a single stage optimization initialized with random positions.\n\n### Typos and Text Quality\n> Weakness 4  \nThanks for the pointers. We will fix the typos and rework the text for improved readability in section 1, 4.1 and 4.3. \n\n\n> 44: Ye et. al. 3D Gaussian Splatting with Deferred Reflection, arXiv 2024  \n> 47: J. Schüle et al. Multi-Physical Tissue Modeling of a Human Urinary Bladder, EMBC 2021  \n> 48: Yang et al. 3D reconstruction from endoscopy images: A survey, Computers in Biology and Medicine 2024", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dVyF6HYyrz", "reviewer": "Reviewer_uk4p", "summary": "This paper proposes a framework for capturing the geometry, specular, and subsurface scattering appearance of 3D objects using a captured dataset composed of multi-view OLAT images. \nThe appearance of the object is decomposed into two different models. \nFirst, a 3D Gaussian representation with a spatially varying BRDF model for explicit surface representation. Secondly, an implicit volumetric representation is used for subsurface scattering appearance. \nThis framework enables material editing, relighting, and novel view synthesis in real-time.", "questions": "1. The real-world dataset images look over-exposed in the specular regions. In the experimental setup. In the paper, same images were captured five times to reduce image noise. Did you change the exposure of the camera while capturing the raw images? Moreover, why did you choose 8-bit instead of 16-bit images?\n2. This doesn't need to be addressed in the rebuttal (Minor), but it would be better to add some information about the name of each video. Without clear explanations, some notations can be misleading to the reader.", "limitations": "The author adequately addressed the limitations, including:\n1. Difficulty in optimizing strongly heterogeneous materials.\n2. Limitations in screen space shading.\nI might add that this paper is also limited to static scenes only. Human skin, for example, is constrained not only by personal rights but also by the inability to handle movement during capturing. This will cause blurry results that the current framework cannot handle.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. I believe that the authors will provide a multi-view OLAT dataset for subsurface scattering objects in the future, and this dataset will be a great contribution to the vision and graphics community.\n2. Screen space shading algorithm for 3D Gaussian representation improves the view-dependent appearance quality of the reconstructed object.\n3. The use of two different appearance models enables the reconstruction of the geometry, BRDF, and subsurface scattering of 3D objects.", "weaknesses": "1. Missing references for diffusion-based SSS approximation models:\n\n[1] A Practical Model for Subsurface Light Transport, Jensen et al., 2001.\n\n[2] Light Diffusion in Multi-Layered Translucent Materials, Donner et al., 2005.\n\n2. Lack of validation on the intrinsic properties. \n\nTo generate realistic novel relit scenes and material edits, it is important to correctly acquire the intrinsic properties. For the synthetic data, each intrinsic property can be directly compared with the reconstructed one (RMSE). Although reconstructed properties can differ due to the different models from Blender or the limitations of the method, the author should address this. Currently, it is unclear if this method reconstructs each intrinsic property well because there is no ground truth intrinsic property information in the paper or supplemental material, even though the rendered results look realistic and similar to the ground truth data. For example, in the dragon dataset in the supplemental material, there are many specularity changes in the ground truth, but these are not observed in the renders.\n\n3. The relighting results only include novel views with single-light images. \n\nThere are no results for environmental lighting or changing light colors. Since the framework does not support multiple light sources, we could generate multiple light sources or environment-relighting results similar to the previous method [3]. Simply adding multiple OLAT images in screen space would provide much more powerful relighting results, which I would like to see.\n\n[3] Neural Light Transport for Relighting and View Synthesis, Zhang et al., 2020.\n\n4. It would be powerful to show the change in intrinsic properties between objects, for example, from bunny to dragon and vice versa.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Additional Relighting Results with Image Based Lighting\n> Weakness 3  \nPlease find qualitative and quantitative results for the image based lighting in Fig. 1 and Tab. 1 in the PDF accompanying this rebuttal. We also show additional relighting results together with the other editing capabilities in Fig. 2. Using our approach as outlined in the global response in  “Image Based Relighting” we can generate a relit frame in a fraction of a second after a one-time pre-computation of the reflectance field which takes 20 seconds for a reasonable resolution. Alternatively, we can perform importance sampling on a given environment map which reduces combined processing time but cannot be reused for a different illumination setting. \n\n### Evaluation of Intrinsic Properties\n> Weakness 2  \nWe provide a small qualitative evaluation in Fig. 4 and an extensive quantitative evaluation in Tab. 3 on the prediction of intrinsic properties of our model.  \nWe assume the reviewer refers to albedo and illumination as common properties of intrinsic image decomposition. While our base color parameter can be understood as the albedo we also output additional material parameters that are needed for our physically based rendering model. We think it is interesting to also compare these properties. Therefore, we decided to showcase base color, roughness, metalness, normal, sss residual, specular & diffuse component and again the final render. Specular and diffuse illumination are intermediate results in our rendering pipeline during the deferred shading stage representing the diffuse and specular illumination components (before multiplication with the base color), respectively.\n\n> All selected properties are compared to ground truth properties obtained using the Cycles renderer in Blender [8] for our synthetic scenes. We want to note that Blender uses a different shading model, such that some of these properties are not directly equivalent. Most notably, the SSS Residual cannot be retrieved and is calculated by us as the difference of diffuse reflection from a rendering with SSS turned on and SSS turned off. In Fig. 4 we plotted the absolute difference and properties for one example and will add more to the supplementary material. For all of these intrinsics we calculated the RMSE. While achieving overall good results, especially for the illumination, the value of the quantitative evaluation is limited by the fact that the optimization can generate multiple plausible solutions for a given appearance due to the under-constrained problem space. This is also the reason why base colors might differ. \nAs correctly noticed by the reviewer, the dragon scene does not capture all specularity changes and is also the one performing the worst in our analysis. It is the most complex scene with heterogeneous subsurface material and a lot of geometric details. Our method does not perfectly represent the geometric detail of the dragon scales in the scene resulting in incorrect specular highlights that lead the optimization to a rougher material. To improve our method for such tough scenes and also to better handle anisotropic scattering are things we want to explore in future work. \n\n### Editing of Intrinsic Properties\n> Weakness 4  \nFor the editing of intrinsic properties we want to reference Fig. 2 that highlights free editing of intrinsics and shows results for various settings of those properties. Surely, intrinsic properties could be transferred between models also shown by edit examples in Fig. 2 transforming the statue into various different material types. Note, that some intrinsics such as the residual are position dependent and could hardly be transferred, though. \n\n### Multi-exposure Acquisition\n> Question 1  \n> We acknowledge HDR images in the dataset could be beneficial, especially for fitting BRDFs. We still decided to capture our real-world dataset in LDR for three main reasons, which all boil down to ease of use:\n> - In deep learning working with LDR images is much simpler than handling the unconstrained value ranges of linear HDR images.\n> - Competing methods are also designed for and trained on LDR datasets.\n> - The disk space needed for capturing, processing and later hosting and downloading such a big HDR dataset would be substantially higher than for a compressed LDR dataset. The png-compressed, single channel, 8-Bit Bayer images still need 2.2 TB of storage, which increases to 5.2TB when adding color, rectification and object masks.\n>\n> While some specular highlights will be unavoidably clipped the overall exposure is carefully set for an optimal SNR for single light images as can also be seen in Fig. 6 of the rebuttal PDF.\n\n### Additional References on Diffusion Based SSS Estimation\n>  Weakness 1  \nWe reworked the related works section also including the suggested literature on diffusion based estimation of scattering parameters.\n\n### Presentation and Labeling of Videos\n> Question 2  \nWe thank the reviewer for the additional feedback regarding the presentation of the results. We will take this into account when redesigning the supplementary material and accompanying website.\n\n\n> We want to thank the reviewer again for the thorough analysis of our work. We hope that our new evaluation of intrinsic properties as well as the relighting results add insights into the method’s capabilities and that we could clarify the reasoning for some of our choices for the data acquisition.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
R1 pNBB - R2 wfKX - R3 F9dx - R4 uYWZ - R5 uk4p  
Find cited works in the original paper, Fig. numbers refer to rebuttal PDF

We thank the reviewers for their constructive feedback and for recognizing our effort to advance an “under-explored research area in inverse rendering” [R1]. 

We propose a method to “reconstruct relightable objects with subsurface scattering (SSS) effects” [R1] by “decomposing the scene into subsurface scattering, diffuse and specular reflections, and object shape” [R2]. This decomposition in combination with the Gaussian Splatting (GS) “framework enables material editing, relighting, and novel view synthesis in real-time” [R4].  
We appreciate the acknowledgement of the reviewers, that the “proposed method successfully decomposes the PBR parameters and significantly outperforms the NeRF-based method for novel view synthesis, training time, and rendering speed” [R2]. Further, that the “quantitative results are strong compared to existing methods” [R3], overall achieving “comparable results and fast rendering speed” [R4]. Additionally, the created dataset can be a “great contribution to the vision and graphics community” [R5], enabling e.g. benchmarking of “approaches that perform shape/material estimation and relighting, esp. for translucent objects” [R3]. 

We hope to clarify open questions and show new insights to our method (please also see attached PDF) that underline its contribution to the research field. In the following sections we address questions and comments raised by multiple reviewers.  

## Clearer Presentation of Key Idea and Contribution [R2, R3]
To the best of our knowledge the proposed method is the first to show joint reconstruction and decomposition of translucent objects for high quality relighting and material editing in real-time.  
We will rework the introduction section of the paper to make the key components of our contribution clearer as outlined below.
At the core we propose a hybrid representation that extends 3D Gaussian Splatting with PBR material parameters [9] and deferred shading [11] with a light-weight residual prediction network to learn shading components not modeled by the surface shader. We focus on the subsurface scattering component for translucent objects which is underrepresented in recent research on neural inverse rendering.  
We constrain the network predicting the outgoing subsurface scattering (SSS) radiance by jointly predicting the incident radiance used for the PBR rendering step. The incident light prediction is effectively regularized as it is used in a physically based renderer together with the independently optimized material parameters. We highlight this in Tab. 2 of our newly added ablations showing the difference in performance between a joint MLP and two independent ones. By relaxing the physical definition of the residual prediction a little we gain better quality output, potentially compensating limitations of our rendering model.

## Choice of OLAT Data and Relighting [R1, R4]
One Light at A Time (OLAT) data makes it possible to disentangle the subsurface scattering effects and reflectance at the surface that would need additional priors otherwise.  
Inverse rendering of translucent objects is a severely ill-posed problem. The OLAT setting is common here [42, 39] as it helps to recover the complex global lighting in the scene by providing an impulse response of the system; every image in the dataset is only illuminated by a single light of known position. This leads to a more accurate reproduction of highlights and SSS than if illuminated by an environment light. We acknowledge the limitations such an acquisition setup imposes and, therefore, want to make our acquired data available. Once reconstructed, our model can work in arbitrary lighting settings.

## Image Based Relighting [R1, R3, R4, R5]
In Fig. 1 we show that our method achieves image based lighting with high visual quality. First we sample the HDR maps representing distant environment illumination. The samples don’t need to correspond to OLAT samples used in training and could be placed much denser. Using this approach we can generate a relit frame in about 20 seconds for a medium resolution. To speed up generation we can precompute a reflectance field assuming white light. We compute the relit view as the sum over the reflectance field scaled by the environment illumination before applying tone mapping for display. This runs in a fraction of a second. As can be seen in Fig. 1 an object can be rendered in different illumination settings yielding consistent photorealistic results. 

## Editing Applications [R4, R5]
We demonstrate the editing applications our approach enables in more detail in Fig. 2. We show:
- Relighting with environment map based lighting 
- Single light illumination outside the training domain
- Color changes of base color or SSS Residual
- Editing of material parameters to make the appearance more metallic, shinier or rougher
- Changing opacity and intensity of the SSS  

Further, our SSS Residual adds editing capabilities that are not available in previous methods [9].

## Comprehensive Evaluation and Ablation [R1, R3, R4]
Please find the additional ablations in Tab. 2 and visual examples in Fig. 3. The provided evidence underlines the pipeline choices presented in our method. Please note that the metrics used only have limited capability to capture the fine details in visual quality that the improved representation of specular highlights adds (see Fig. 3). Compared to previous methods we allow for detailed editing of the reflectance parameters at the surface independently of the volumetric residual while keeping and extending the relighting functionality. The analysis of the material prediction and the intermediate shading buffers highlights the physical plausibility of our results (see Fig. 4).

We would like to thank all reviewers again for their valuable feedback that already led to an improved submission.

Meta-review: This paper received 5 mixed, but positive leaning reviews ---- three 6s (weak accept), one 5 (borderline accept), one 4 (borderline reject). 

There was general consensus around the novelty of problem being addressed (incorporating subsurface scattering in 3DGS for volumetric reconstruction of translucent objects), the quality of results and the usefulness of the proposed dataset. 

There were several concerns raised - from evaluation and experimental standpoints, and also regarding technical novelty. The authors provided thorough responses which assuaged the evaluation concerns to a large extent. One of the reviewers had lingering doubts about the technical novelty arguing that the paper mostly builds upon existing approaches. On the other hand, most other reviewers appreciated the principled approach the paper takes for handling subsurface scattering. 

Given these, on balance, an accept decision was reached. 

The authors are strongly encouraged to incorporate the reviewer comments -- especially around new validation content that was promised in the rebuttal -- in the camera-ready version.
Decision: Accept (poster)