BecomingLit: Relightable Gaussian Avatars with Hybrid Neural Shading
Authors: Jonathan Schmidt
Simon Giebenhain
Matthias Nießner
Venue: NeurIPS 2025 poster ()
Keywords: Head Avatars
Relighting
OLAT Dataset
Neural Rendering
Intrinsic Decomposition
Primary area: applications
TL;DR: Photorealistic relightable head avatars with a new OLAT dataset of faces.

Abstract: We introduce *BecomingLit*, a novel method for reconstructing relightable, high-resolution head avatars that can be rendered from novel viewpoints at interactive rates. Therefore, we propose a new low-cost light stage capture setup, tailored specifically towards capturing faces. Using this setup, we collect a novel dataset consisting of diverse multi-view sequences of numerous subjects under varying illumination conditions and facial expressions. By leveraging our new dataset, we introduce a new relightable avatar representation based on 3D Gaussian primitives that we animate with a parametric head model and an expression-dependent dynamics module. We propose a new hybrid neural shading approach, combining a neural diffuse BRDF with an analytical specular term. Our method reconstructs disentangled materials from our dynamic light stage recordings and enables all-frequency relighting of our avatars with both point lights and environment maps. In addition, our avatars can easily be animated and controlled from monocular videos. We validate our approach in extensive experiments on our dataset, where we consistently outperform existing state-of-the-art methods in relighting and reenactment by a significant margin.

BecomingLit: Relightable Gaussian Avatars with Hybrid Neural Shading
Jonathan Schmidt
Simon Giebenhain Technical University of Munich
Matthias Nießner
jonathsch, jontthsch.github. io/becomiiggli
Multi- View OLAT Dataset
Intrinsically Decomposed Avatar
Relighting Animation
BecomingLit: Our approach effectively reconstructs detailed human head avatars that can be animated from videos and relighted real-time using our hybrid neural shading approach. Besides our method, we introduce new high-quality, multi-view OLAT dataset of faces.
Figure 1:
Abstract
We introduce BecomingL.t, a novel method for reconstructing relightable, high- resolution head avatars that can be rendered from novel viewpoints interactive rates. Therefore, we propose a new low-cost light stage capture setup, tailored specifically towards capturing faces. Using this setup, collect novel dataset consisting of diverse multi-view sequences of numerous subjects under varying illumination conditions and facial expressions. By leveraging our new dataset, we introduce new relightable avatar representation based on 3D Gaussian primitives that we animate with parametric head model and an expression-depender dy- namics module. We propose new hybrid neural shading approach, combining a neural diffuse BRDF with an analytical specular term. Our method reconstructs disentangled materials from our dynamic light stage recordings and enables all- frequency relighting of our avatars with both point lights and environment maps. addition, our avatars can easily be animated and controlled from monocular videos. We validate our approach in extensive experiments on our dataset, where we con- sistently outperform existing state-of-thed the-hhh-rtt methods in relighting and reenactment by a significant margin.
Introduction
The creation of photorealistic, relightable 3D head avatars from real-world data is core problem of computer vision with applications across wide range of graphics tasks, such cinematography, virtual reality, or the metaverse in general. Traaiitionlll, this requires professional, room-scale capture setups that only handful of institutions afford 334 23 24 as the joint estimation of geometry, intrinsic material parameters, and lighting is an extremely under-constrained problem.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

At the same time, with the progressing growth of virtual reality applications at the consumer level, creating photorealistic avatars is becoming more important than ever. While there has been immense progress over the recent years in terms of geometric representations [15 and rendering speed thanks to the availability of custom datasets [T6]. most 3D avatars do not have a disentangled representation of the material properties and bake the radiance properties of the training environment into the avatar, which makes relighting impossible. As a result, placing the avatar in a novel virtual environment dramatically lowers the visual quality of the renderings. In comparison, research on relightable avatars is scarce. One of the major reasons for this is the lack of publicly available and free-to-use datasets that come with controlled light captures in order to broadly study the reconstruction of facial appearance.
30
18 25 visual quality 10
To this end, we introduce an OLAT dataset and propose BecomingLit, novel approach to reconstruct photorealistic, relightable head avatars from short multi-view light stage sequences. We represent the head with supression-dependeen
Gaussian primitives and model the complex reflection behavior of faces by learning a hybrid neural BRDF. Thanks to our efficient parameterization and regularization, our method requires capture setup that is an order of magnitude more economical compared to pre- vious work, and outperforms state-of-the-at methods in self-reenactment under novel illuminations. To address the lack of data, we introduce a new multi-view video dataset of different participants in a light stage setting, which we will make publicly available for research purposes. Overall, our contributions are two-fold:
We introduce novel, publicly available dataset, combining high-resolution, high-framerate. multi-vien recordings of different subjects in a calibrated light stage setting.
We propose a relightable. photorealistic avatar representation based on 3D Gaussian primi- tives and hybrid neural shading, which can be relighted and rendered from novel viewpoints in real-time and animated from monocular videos.
2 Related Work
Human Head Modeling addresses the problem of representing and modeling the geometry and appearance of human heads. Traditional methods learn morphable models from head scans via PCA (5)
28 [7]. While being strong in generalization, PCA-based 3DMMs have a limited expressiveness and can fail to represent fine geometric details such skin wrinkles or hair. As alternative, [. 20 propose to learn the geometry and appearance space with autoencoders. More recently, volumetric approaches based on NeRF (25), represent heads with more detailed appearance, despite
not requiring explicit input geometry 16
54
18
57
32). Another line of work uses 3D Gaussian some of them in combination with a 3D morphable
primitives [15] to model human heads 10
19
47
model 30 34].
Facial Appearance Capture. Capturing the appearance of human faces is long-standing problem in computer vision. Debevec et al. 7 introduced the light stage and demonstrated how the reflectance field of human face can be reconstructed from oig-tight-at-aa aight-at-a-time captures, and relighted using image- image-based rendering 7 43 Subsequent work leveraged polarized light to decompose specular 2).
and diffuse reflectance 21
33
335 46) approach the intrinsic decomposition problem with propose a learnable, data-driven appearance model that learns avatar
radiance fields 225 4
49
relighting in an end-to-end manner with neural lighting model. In constrast, 34 40
propose to
learn radiance transfer properties of 3D Gaussian primitives (15).
Neural Shading is concerned with learning light reflectance functions instead of using analytical models developed in computer graphics. This has been successfully applied to static scenes [35]
46 48
31 and dynamic objects [23 24). image-based methods enable relighting of single 26 38 511 but fail to synthesize novel views and struggle with temporal consistency, which are key requirements for head avatars. While 34 learns the coefficients of an explicit precomputed radiance transfer (PRT) function, [I] proposes to learn the PRT function with a neural network. In contrast, we propose hybrid neural shading approach, combining implicitly learned diffuse radiance transfer with a well-established analytical specular term.
53
portrait 12
3 Multi-View OLAT Dataset of Faces
Capturing human faces under known, calibrated illumination enables efficient estimation of skin properties such as reflectance 7 35 and pore-level normals (2I]. We therefore introduce a novel dataset, which consists of multi-view recordings of different subjects in light stage setting.
2

---

Table Existing light stage datasets of human heads. ICT-3DRFE 37 and no raw footage or calibration data.
contains only processed data
Dataset
IDs
Views
# Lights
FPS
Resolution
Setup Cost
3DRFE 37
23
only processed data
S$
Goliath
22
144
460
(90
1334x2048
S$$
Ours
10
16
40
72
2200x3208
$
(a) Capture Rig
(b) Dataset Samples
Figure 2: OLAT Dataset: (a) Our custom light-stage rig we used to capture (b) our dataset consisting of high-resolution, high frame rate, multi-view recordings of faces under both OLAT and fully-lit conditions.
The dataset offers an unprecedented combination of high- fighrreoluutin, high high-frame-re frame-rate multi-view recordings of many sequences under numerous calibrated lighting conditions.
3.1 Capture Setup
As our primary capture target are human faces, we build a light stage setup that covers the frontal hemisphere of the subject's head. The setup consists of 16 machine vision cameras and 40 custom- built LED modules that are driven by microcontrollers. The cameras cover a field view 93 horizontally and 32" vertically. The lights placed uniformly around the subject, covering range of 180 horizontally and 60% vertically. All cameras and lights face towards the subject's 's face. See Figure 2 for visualization of the capture rig. Each LED emits enough luminance one-light-at-a-ting (OLAT) and more complex light patterns, while maintaining low shutter speed of 3ms, thus reducing motion blur minimum. We use high-quality LEDs with Color Rendering Index (CRI) of over 98, which closely approximates natural white light.
both
We control the LEDs using microcontrollers that synchronize with the cameras using vendor- specific logic. Our capture rig equipped with 16 machine vision cameras, which we internally synchronize using the Precision Time Protocol (PTP), leading multi-view frames captured with a deviation of less than one microsecond. Each of the cameras records images with a resolution of 2,200x3 .08 pixels at 72 frames per second, sufficient capture specular reflections the skin at pore-level detail as depicted Figure 2b
3.2 Data Acquisition
Using our light stage setup, we capture several sequences of different participants for a few minutes in total. During the capture sessions, each participant performs predefined set of facial expressions, emotions, and reads out several sentences. Please refer to the supplementary for more details about our capture script. In total, we record around 150 seconds for each subject, which divided into blocks. In addition, we capture another sequence where every participant free to perform arbitrary expressions for 20 seconds. For each frame, we activate new light from the set of available OLAT configurations. To enable tracking, we follow previous work [43] 34 and interleave our cycle of light patterns with fully-lit tracking frames. More specifically, every third frame a tracking frame, which
190 FPS is only available for short test segment.

---

multi-vie view capture of the mirror sphere. To account for differences results in tracking sequences captured at effectively 24 frames per second. See the rightmost image of Figure 2b for an example of tracking frame.
3.3 Data Processing
The camera poses and intrinsic parameters are obtained using a checkerboard and bundle adjustment. Both the position and the intensity of the LEDs are calibrated using mirror sphere whose shape and reflection properties are known. We follow the procedure of [45] and find the 3D position of each light source using ray- ray-tracing in colors among the camera sensors, we use a color checker board and compute a color correction
matrix for each camera. We use BiRefNet [55] for obtaining high-resolution foreground masks and obtain semantic segmentation with Facer [56
3.4 Data Privacy
Our dataset contains highly personal information, which requires distributing it with extreme caution. We will only share the data with approved academic institutions and exclusively for non non-commercial research purposes. All participants signed an agreement for publication, yet retain the right have their data deleted at any time in the future, which we will enforce when distributing the dataset.
Method
Our method reconstructs relightable avatars from multi-view light stage sequences. F
overview of our method. After preliminary information (Sec. 4.1 we describe our geometry (Sec.
4.2) and appearance (Sec. 4.3) model. In Sec. [nd 4.5 we provide details about the optimization strategy and implementation details, respectively.
4.1 Preliminaries
3D Gaussian Splatting 15 introduces a point-based radiance field representation, that defines 3D scene with set of anisotropic 3D Gaussians parameterized by mean μe, covariance Σ and opacity a. In addition, each Gaussian can hold an arbitrary number of features. Unlike continuous representations such NeRF 25 that require ray marching for rendering, 3D Gaussians can be efficiently projected onto the image plane and rasterized in real-tima on consumer-graad GPUs. We refer the original paper of (15) for more thorough overview.
Physically- based Rendering aims at synthesizing images by simulating the physical transport of light from the emitter to the camera sensor. The core is the rendering equation [3] that is defined as follows:
Lo( / (i, w.) (L, Lax
n))dw,
where L. is the outgoing radiance observed by the camera, L; is the incident radiance at point æ from direction w., and f. is the BRDF. Our goal is to recover the BRDF f, from data observations, such that the resulting avatar can be integrated with novel illuminations.
4.2 Geometry
We model the geometry of our avatar with fixed set of anisotropic Gaussians 15 that we define on the UV map of tracked template mesh. Inspired by 440 [0], we employ an expression-dependee module F, to model fine-grained
dynamics module Fg, and a view and iupression-dependdeee geometric expression details beyond the scope of the template mesh.
As our base geometry, we use the parametric head model FLAME deformations over time. Given the fully-lit tracking frames of our dataset, we obtain shape, expression, and pose parameters using the photometric tracker VHAP 30 29). For the remaining OLAT frames, we linearly interpolate the FLAME parameters the nearest tracking frames. To get the proxy geometry for the Gaussian primitives, we obtain the posed mesh . = (V,F) from the FLAME parameters and compute tangents L,, bitangents by and normals n& for every texel & on the UV map. In addition, we obtain the interpolated 3D position of texel k denoted fåg.
17 which models coarse

---

(fee
yk. 8k. kke Cer C
P
Hocal
rk: Oe
M4p.
Render
FLAME Tracking
Fd
Diffiee Reflectance
Specuaar Shading
Trge
Stotic Amtrbutte
Lightigg
F v
F 9
W
FLLAME
Expecssion Dependent
Dependent Attributes
View
Gi
Attributes
Figure 3: Method Overview:
Given estimated FLAME coefficients, obtain posed 3D Gaussian primitives with our expression-dependen dynamics module Fg. To render photorealistic appearance, combine the neural diffuse BRDF Fa with analytical specular shading term. The parameters for the specular shading are predicted by the view-dependent Fu network. The avatar is optimized from light stage sequences using a photometric loss term.
Given the tracked FLAME expression parameters FFLAME. define Fg as convolutional neural network which predicts per per-gaussian attributes in UV-space:
tô q, 8, 0, feavey M
===1 F,(0FFLME)
The final Gaussian center is defined as ft RR Jocall +õμ, where BN = [bx, L. n& the orientation of the shading frame of texel k and palocal is a parameter learned statically for each gaussian. The purpose of pooa is to define most of the offsets regularize the expression-dependen offsets 8j4 to be small, which avoids artifacts when synthesizing novel expressions. The remaining Gaussian parameters qk. 8k, d% are directly predicted by Fg, feap an expression-dependent feature vector for shading, which describe in Sec. 4.3
such that
4.3 Material
Modeling the reflectance properties of faces with common analytical models from computer graphics inevitably leads to insufficient quality due their lack of modeling global illumination effects, such as subsurface scattering, which is omnipresent on human skin. We observe that such global illumination effects primarily affect the low-frequency, propose hybrid shading scheme, which learns diffuse light transport implicitly with a small neural network, while modeling specular reflectance with a well-established analytical model. this end, we decompose the reflectance function fr(w,, w:) from Eq. (I) into view-independent diffuse term fa(w,) and view-dependent specular term f. (W... w.).
-ew-indeenneent diffuse part. Thus, we
Diffuse. The view-independent diffuse term models subsurface scattering and self-shadowing effects. At its core is tiny neural network Fa, shared among primitives, and jointly trained with the avatar. The final diffuse color ed is computed by multiplying statically learned albedo ag with the predicted reflectance of Fa:
& as Fa(SH (
where SH,, (Lş) are the coefficients from the spherical harmonics parameterization of the incident light of degree m, and fo are the expression-depenae degree m in all experiments.
feature vectors. We empirically set SH
Fa is parameterized as a monochrome BRDF function, mapping single-channel incident light to scalar reflectance value. This parameterization is necessary the model only sees white light during
5

---

training. yet must also handle colored illumination at inference time. Here, we evaluate Fa separately for each color channel and concatenate the results into a single reflectance vector, which we multiply
element-wise with the albedo. The architecture of Fa is detailed in the supplementary.
Specular. Our specular term is based the Cook-Torrance model 6 which is generally defined as
fa fa k. D(wo: w.,r) G(wo; w;) F(wo, w.)
(4)
4(n wo) wi)
D(-) a-12()) + +1 (s((
(5)
where k, is the specular intensity, D is the Normal Distribution Function (NDF), and G is the masking and shadowing term, which is derived from the NDF [39]. models the Fresnel effect, for which we use Schlick's approximation [36). As the NDE, we use the 2-Blinn-Phong-hee
mix introduced by Riviere et al. [33]. The advantage of this NDF representation is that roughness r is a linear parameter, which is beneficial during the optimization.
Due to the ellipsoidal shape of 3D Gaussian primitives, it is non-trivial to associate a single normal vector to them. As suggested by Saito et al. [34), we observe that the normal varies with the viewing direction. We, therefore, use a second, smaller CNN F.. which takes expression features feupr and the viewing direction W,, and predicts specular intensity 8g and normal offsets ón. While the general idea behind F, is similar to 34 our U-Net architecture requires fewer network parameters, improving performance on consumer-level hardware. The final shading normals are obtained by adding the normal offsets ong to the mesh normals, followed by normalization. During training, we evaluate the specular term with the point light pattern of the current frame. For environment map relighting, we use the split-sum approximation [14]. We provide details the supplementary.
4.4 Optimization
Given calibrated multi-view sequences from our dataset and corresponding estimated FLAME parameters, we jointly optimize Fg. Fd, Fo. and static parameters uloral ag. T'k with the following loss term:
L = Lrgh + Freg
(6)
Freg = Anormal nnormal + Nalpha Lalpha + Ascate + Apos Coos
(7)
where Ligh An Lu + ASSIMLSSIM is the photometric loss term consisting of L1 and SSIM term as proposed by [15]. We set (An, ASSIM) to (1.0. 0.2) in all experiments. Our regularization |[ôra]]. which encourages the predicted normal
loss Greg consists of the normal loss Lnorma! offsets to be small, and thus, be close to the normals of the FLAME mesh. Our capture rig only contains lights on the frontal hemisphere, which would lead to artifacts when we render the avatars with lights from the rear or environment maps. We find that simple L2 loss Lalpha between the rendered alpha maps and the alpha masks from background matting prevents the avatar from becoming too transparent. The scale loss is adapted from (34) and promotes the primitive scales to remain in a reasonable range. Lpos is another L2 term which drives F, predict small delta means. We set (atlhha, Asente, Apna} to (2e-2) 2, 2e 2, le- in all experiments.
4.5 Implementation Details
We implement all networks and optimization logic in PyTorch 27 and write custom GPU kerness for the specular shading using the SLANG.D shading language [3]. For rendering the Gaussian primitives, use gsplat [50]. We use texture resolution of 512² 2 in all experiments, which results in 202k primitives after masking out texels from the FLAME UV map that are not assigned to any surface point. We use the 2023 version of FLAME [17] with the manually added teeth from Qian et al. 300. We train our avatars at 1 100x 604 resolution for 250k iterations with a batch size of 4, which takes approximately 30 hours single NVIDIA RTX A6000 GPU.
4.6 Differences to RGCA
uses a variational autoencoder, which learns a personalized expression space and predicts the parameters of the precomputed radiance transfer function. In contrast, our method builds directly on top of FLAME [17], which has a shared expression space across identities, enabling applications
RGCA 34

---

Table 2: Quantitative results on held-out lights on both the training and held-out segments.
Relighting
Relighting Self-Reenactment
Method
PSNR
SSIM
LPIPS|
PSNRT
SSIMT
LPIPS|
RGCA
29.21
0.8462
0.1659
26.31
0.8206
0.1917
RGCAFLAME
29.78
0.8464
0.1444
26.91
0.8282
0.1667
Ours
31.38
0.8956
0.1040
28.08
0.8730
0.1317
such as cross reosstrennat..ent. Generally speaking, our avatars can be animated with FLAME parameters from any source without the need for a personalized encoder. Further, model diffuse light transport with a small MLP and use Cook-Torranee 6 33 variant for specular reflection.
Experiments
We evaluate our method on 4 subjects from our dataset, where our focus lies on relighting and self-reenactment. From the 16 available camera views, we use 15 for training, and hold out the center camera for testing. We further hold out light patterns from training altogether. From the available sequences, we use all scripted sequences for training and use the free sequence for testing. As the test metrics, we use the Peak-Sige Peak-Signal-te Signal-to-Non -o-ooise Ratio (PSNR), Structural-Similarityy -immiarity--nndee Mamnlarrty-ndee-MMaauuuu MeauueMe (SSIM) 42) and the Learned Perceptual Image Patch Similarity (LPIPS) (52)
Baselines. Our main baseline is Relightable Gaussian Codec Avatars (RGCA) that builds head avatars by decoding learned expression codes to 3D Gaussian attributes for geometry and intrinsic radiance transfer. The resulting avatars can be relighted by integrating the predicted intrinsic radiance properties with novel light sources. The input to RGCA are the vertices of a coarse template mesh together with unwrapped average textures, for which we use the FLAME meshes and textures from the VHAP 22 30 tracking. Since the expression space of RGCA is learned per identity, it requires comprehensive training sequences, while our method can leverage the existing FLAME expression space. Therefore, introduce a second baseline denoted RGCAFLAME, where we replace the learned expression latent space with FLAME expression coefficients.
54 recent method
5.1 Relighting and Self-Reenactment
Our primary target application is reenactment under novel illuminations. Therefore, we animate our trained avatars with the FLAME parameters of the held-out sequence and select those frames with light pattern not seen during training. We then render the avatars from the held-out camera view. Table 2 we report the quantitative results of relighting for both a training and test sequence. Figure 4 presents the qualitative results of our avatars rendered from the test camera, with unseen lighting condition and expression. A comparison under environment map relighting is shown in Figure Our rendered avatars match the target appearance more closely in terms of the color and fine geometric details, which enables more realistic specular reflections. Notably, we observe that RGCA conditioned FLAME parameters performs strictly better than the original version with the personalized expression space. We hypothesize that learning an expression space per subject is suboptimal for reenactment tasks.
In Figure 5| we qualitatively compare the intrinsic decomposition performed by our approach to the baselines. Our method recovers cleaner albedo and sharper specular highlights from the data observations and faithfully decomposes the diffuse and specular parts of the material. addition, we recommend to watch our supplementary video for more results, which allow for more complete comparison, including the temporal axis.
5.2 Ablation Study
We verify the key components of our method with ablation experiments, which conduct with the same subjects. qualitative and quantitative comparison is presented in Figure and Table 3 respectively.
PBR. We compare our full model to a version where we replace the hybrid neural shading with a classic PBR shading model using Lambertian term for diffuse, and a reflection. This simple appearance model cannot reproduce the complex appearance of skin since subsurface scattering not modeled, which results in synthetically looking renderings.
(6) for specular
7

---

RGCA
RGCAFLAME
Ours
Figure 4: Relighting and Self- Reff-Reenattment: Qualitative comparison held-out segments and held-out illuminations.
PRT Diffuse. We compare our neural diffuse component to the learned precomputed radiance transfer (PRT) model introduced in 34 The respective SH coefficients are directly predicts by Fg. Although, observe good results on the training frames, PRT struggles to generalize to novel illuminations, which aligns with our findings from Section 5.1
SG. To justify the choice of the specular term, we ablate our specular BRDF to a simple Spherical Gaussian term. Compared our Cook Torrance term, we observe slightly better PSNR and LPIPS scores as well as marginal sharper details under point light illumination. However, depicted in Fig. we observe significantly more natural and detailed lore reflection with Cook-Torrane based specular model.
Alpha Loss. The key component prevent the avatars from becoming transparent is the alpha loss using estimated foreground segmentation masks. Since our capture setup only has lights and cameras on the frontal hemisphere. we observe artifacts with environment map relighting when using no regularization. We want to highlight that this simple regularization scheme effectively reduces the complexity of our capture setup, the resulting dataset, and computational cost during training by half.
Expression Features. We compare our expression features against static feature vectors as proposed by Giebenhain et al. [10]. Here, we static features for the diffuse BRDF network Fa, and use the FLAME parameters as condition for F... We observe that without the sxpression-dependae features, the model fails accurately reproduce pore-level details and specular highlights. We can further notice worse color and reflections compared to the full model.
5.3 Application
Once trained, the only inference parameters are FLAMME expression and pose parameters, which can be obtained from monocular videos [29 30 We demonstrate this by animating avatars with short

---

ROCA
RGCAFLAME
Ours
(a) Albedo
(b) Normals
(c) Diffuse
(d) Specular
(c) Full
(f) GT
Figure 5: Comparison of Intrinsic Decomposition: We compare the recovered albedo (a) and normals (b), as well as the diffuse (c) and specular (d) contributions on a training frame that sum up to the final rendering (c). Note that the reference image (f) is identical in all rows.
Table 3: Ablations: We conduct ablations with the same 4 subjects and report relighting and reenactment results on the training and test expressions.
Relighting
Relighting + Self-Reenactment
Method
PSNR
SSIM
LPIPS
PSNR
SSIM
LPIPS
w/ PBR shading
29.42
0.8719
0.1344
26.31
0.8448
0.1665
PRT diffuse
29.23
0.8374
0.1577
25.47
0.8074
0.1918
w/ SG
31.55
0.8953
0.1031
28.09
0.8729
0.1310
w/o alpha loss
31.34
0.8955
0.1043
28.07
0.8729
0.1328
w/o expr. features
31.23
0.8928
0.1071
28.13
0.8717
0.1332
Ours (full)
31.38
0.8956
0.1040
28.08
0.8730
0.1317
PBR
w/o alpha
w/o ex. feat.
PRT
SG
Ours
GT
w/ PBR
w/o alpha
w/o ex. feat.
PRT
SG
Ours
Target Expression
Figure 6: Ablation Study: With only PBR shading, the avatar has a synthetic, plastic-like appearance. Without the alpha loss, we observe artifacts when rendering with environment maps. The expression- dependent features further improve both appearance and fine geometric details.

---

RGCA
RGCA
Ours
Driving Sequence
Target Actors
Figure 7: Left: Qualitative comparison on environment map relighting. Right: Animation using
monocular videos.
Table 4: Runtime comparison: We report the component wise inference time in milliseconds.
Method
CNNs
Diffuse Shading
Specular Shading
Splatting
Total
RGCA
9ms
Ims
Ims
9ms
20ms
Ours
4ms
3ms
Ims
9ms
17ms
video sequences from the VFHQ dataset 44 We obtain the FLAME parameter with the monocular version of the VHAP tracker [29]. and relight our avatars with environment maps collected from PolyHaven [[]. We present the results in Figure 7 and highly encourage the reader to watch the accompanying video for temporal results.
5.4 Runtime
In Table 5, we summarize the runtime of the components of our method and compare with RGCA 34 We conducted all measurements
single NVIDIA RTX A6000 GPU using a UV resolution of 5122, which results 202k primitives (due to masking of texels not assigned to any surface location). We render images at a resolution of 1100x1604, corresponding to the training resolution of our avatars.
5.5 Discussion
Limitations. While method delivers state-of-the-ate results and enables more practical animation than previous methods, our approach is still not without limitations. While our capture setup
is order of magnitude more economical than existing setups 34 4 12), avatar training still requires several thousand frames and a diverse set of training expressions. Obtaining photorealistic avatars from causal phone captures with uncalibrated lighting remains open challenge for future work. The FLAME base geometry is limited in its expressiveness, is sensitive to tracking failures (particularly with respect gaze direction), and does not model the mouth interior. These limitations are consequently inherited by our avatars. As of now, the neural diffuse shading model is trained from scratch jointly with the avatar. Using dataset learn appearance prior of human faces and heads is an interesting direction for future work.
Ethical Considerations. Creating photorealistic, relightable avatars entails the potential for various malicious use cases, such as identity theft, deepfakes, and privacy violations. This a particular concern when avatars can be driven from simple video sequences, in case. However, to create an avatar with our method, the respective subject must first be scanned in our capture setup, which is only applied to limited number of consenting individuals. Further, will be restrictive with access to our dataset as outlined in Section 3.4
6 Conclusion
We have presented BecomingLit, novel framework for reconstructing photorealistic, relightable avatars from capture setup, orders of magnitude more economical than previous state-of sfate-of-the-art -he-ttrrtt methods. We have proposed new hybrid shading approach for 3D Gaussian primitives, which enables better generalization to novel illuminations and expressions. Our relightable avatars can be animated from simple videos and relighted with both point lights and environment maps. Along with our method, we will publish new dataset of faces under OLAT conditions, which is unprecedented in terms of resolution and frame rate. We believe that this will democratize research on facial appearance modcling and serve valuable contribution the community.
10

---

Acknowledgments and Disclosure of Funding
This work was supported by the ERC Consolidator Grant Gen3D (101171131 and the German Research Foundation (DFG) Research Unit "Learning and Simulation in Visual Computing". Addi- tionally, we would like to thank Angela Dai for the video voice-over.
References
[1] Poly haven hdris, 2023. URL MtLLLttts://pplyyaaaar com/ndris
[2] Dejan Azinović, Olivier Maury, Christophe Hery, Matthias Nießner, and Justus Thies. High-res facial appearance capture from polarized smartphone images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages
2023.
[3] Sai Praveen Bangaru, Lifan Wu, Tzu-Mao Li, Jacob Munkberg, Gilhert Bernstein, Jonathan Segan-Kelley, Frédo Durand, Aaron Lefohn, and Yong He. Slang. d: Fast, modular and differentiable shader programming. ACM Transactions Graphics (TOG), 42(6):1-28, 2023.
[4] Sai Bi, Stephen Lombardi, Shunsuke Saito, Tomas Simon, Shih-En Wei, Kevyn Mcphail, Ravi Ramamoor- thi, Yaser Sheikh, and Jason Saragih. Deep relightable appearance models for animatable faces. ACM Transactions on Graphics (ToG), 1054) 2021.
Volker Blanz and Thomas Vetter. morphable model for the synthesis of faces. Seminal Graphics Papers: Pushing the Boundaries, Volume 2, pages 157-164, 2023.
[6] Robert Cook and Kenneth Torrance. reflectance model for computer graphics. ACM Transactions on Graphics (ToG), 1(1):-224, 1982.
[7] Paul Debevec, Tim Hawkins, Chris Tchou, Haarm-Pieter Duiker, Westley Sarokin, and Mark Sagar. Acquiring the reflectance field of human face. In Proceedings of the 27th annual conference on Computer graphics and interactive techniques, pages 145-156, 2000.
[8] Stephan J Garbin, Marek Kowalski, Virginia Estellers, Stanislaw Szymanowicz, Shidch Rezacifar, Jingjing Shen, Matthew A Johnson, and Julien Valentin. Voltemorph: Real-time, controllable and generalizable animation of volumetric representations. Computer Graphics Forum, volume 43, page e15117. Wiley Online Library, 2024.
[9] Abhijeet Ghosh, Graham Fyffe, Borom Tunwattanapong, Jay Busch, Xueming Yu. and Paul Debevec. Multiview face capture using polarized spherical gradient illumination. Proceedings of the 2011 SIGGRAPH Asia Conference, pages 1-10, 2011.
[10] Simon Giebenhain, Tobias Kirschstein, Martin Rünz, Lourdes Agapito, and Matthias Nicßner. Npga: Neural parametric gaussian avatars. In SIGGRAPH Asia 2024 Conference Papers, pages 1-11, 2024.
[11] Kaiwen Guo, Peter Lincola, Philip Davidson, Jay Busch, Xueming Yu, Matt Whalen, Geoff Harvey, Sergio Orts-Escolano, Rohit Pandey, Jason Dourgarian, et al. The relightables: Volumetric performance capture of humans with realistic relighting, ACM Transactions Graphicx (ToG), 38(6):1-19, 2019.
[12] Mingming He, Pascal Clausen, Ahmet Levent Tayel, Li Ma, Oliver Pilarski, Wenqi Xian, Laszlo Rikker, Xueming Yu, Ryan Burgert, Ning Yu, al. Diffrelight: Diffusion-based facial performance relighting. arXiv preprint arXiv:2410.08188. 2024.
[13] James T Kajiya. The rendering equation. In Proceedings of the 13th annual conference on Computer graphics and interactive techniques, pages
1986.
[14] Brian Karis and Epic Games, Real shading unreal engine Proc. Physically Based Shading Theory Practice, 4(3):1, 2013.
[15] Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. gaussian splatting for real-time radiance field rendering. ACM Trans. Graph., 42(4): 139-1, 2023.
[16] Tobias Kirschstein, Shenhan Qian, Simon Giebenhain, Tim Walter, and Matthias Nicßner. Nersemble: Multi-view radiance field reconstruction of human heads. ACM Transactions on Graphics (TOG), 42(4): 1-14, 2023.
[17] Tianye Li. Timo Bolkart, Michael Black, Hao Li, and Javier Romero. Learning model of facial shape and expression from 4d scans. ACM Trans, Graph., 1646):194-1,20 2017.

---

[18] Stephen Lombardi, Tomas Simon, Gabriel Schwartz, Michael Zollhoefer, Yaser Sheikh, and Jason Saragih. Mixture of volumetric primitives for efficient neural rendering. ACM Trunsactions on Gruphics (ToG), 40 (4):1-13, 2021.
[19] Shengjie Ma. Yanlin Weng, Tianjia Shao, and Kun Zhou. 3d gaussian blendshapes for head avatar animation. In ACM SIGGRAPH 2024 Conference Papers, pages 1-10, 2024.
[20] Shugao Ma, Tomas Simon, Jason Saragih, Dawei Wang. Yuecheng Li, Fernando De La Torre, and Yaser Sheikh. Pixel codec avatars. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 64-73, 2021.
[21] Wan- Man-Chun Ma, Tim Hawkins, Pieter Peers, Charles-Fett Chabert, Malte Weiss, Paul E Debevec, et al. Rapid acquisition of specular and diffuse normal maps from polarized spherical gradient illumination. Rendering Téchniques, 9(10)22, 2007.
[22] Julicta Martinez, Emily Kim, Javier Romero, Timur Bagautdinov, Shunsuke Saito, Shoou-I Yu, Stuart Anderson, Michael Zollhöfer, Te-Li Wang. Shaojie Bai, Chenghui Li, Shih-En En Wei, Rohan Joshi, Wyatt Borsos, Tomas Simon, Jason Saragih, Paul Theodosis, Alexander Greene, Anjani Josyula, Silvio Mano Maeta, Andrew I. Jewett, Simon Venshtain, Christopher Heilman, Yuch-Tung Chen, Sidi Fu, Mohamed Ezzeldin A. Elshaer, Tingfang Du, Longhua Wu, Shen-Chi Chen, Kai Kang. Michael Wu, Youssef Emad, Steven Longay, Ashley Brewer, Hitesh Shah, James Booth, Taylor Koska. Kayla Haidle, Matt Andromalos, Joanna Hsu, Thomas Dauer, Peter Selednik, Tim Godisart, Scott Ardisson, Matthew Cipperly, Ben Humberston, Lon Farr, Bob Hansen, Peihong Guo, Dave Braun, Steven Krenn, He Wen, Lucas Evans, Natalia Fadeeva, Matthew Stewart, Gabriel Schwartz, Divam Gupta, Gyeongsik Moon, Kaiwen Guo, Yuan Dong, Yichen Xu, Takaaki Shiratori, Fabian Prada, Bernardo R. Pires, Bo Peng, Julia Buffalini, Autumn Trimble, Kevyn McPhail, Mclissa Schoeller, and Yaser Sheikh. Codec Avatar Studio: Paired Human Captures for Complete, Driveable, and Generalizable Avatars. NewrlPS Truck on Dutasets and Benchmarks, 2024.
[23] Abhimitra Mcka, Christian Haene, Rohit Pandey, Michael Zollhöfer, Sean Fanello, Graham Fyffe, Adarsh Kowdle, Xueming Yu, Jay Busch, Jason Dourgarian, et al. Deep reflectance fields: high-quality facial reflectance field inference from color gradient illumination. ACM Transactions Graphicx (TOG), 38(4): 1-12, 2019.
[24] Abhimitra Meka, Rohit Pandey, Christian Haene, Sergio Osts-Esco,ano, Peter Barnum, Philip David-Son, Daniel Erickson, Yinda Zhang, Jonathan Taylor. Sofien Bouaziz. et al. Deep relightable textures: volumetric performance capture with neural rendering. ACM Transactions on Graphics (TOG), 39(6):1221, 2020.
[25] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes neural radiance fields for view synthesis. Communications of the ACM, 65 (1):99-106, 2021.
[26] Rohit Pandey, Sergio Orts-Escolann Escolano, Chloe Legendre, Christian Haene, Sofien Bouaziz, Christoph Rhe- mann, Paul E Debevec, and Sean Ryan Fanello. Total relighting: learning to relight portraits for background replacement. ACM Trans. Graph., 40(4):43-1, 2021. 2021.
[27] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high- gesh--peffomance deep learning library. In Advances in Neural Information Processing Systems, volume 32, 2019.
[28] Pascal Paysan, Reinhard Knothe, Brian Amberg. Sami Romdhani, and Thomas Vetter. A 3d face model for pose and illumination invariant face recognition. In 2009 sixth IEEE international conference on advanced video and signal based surveillance, pages 296 301. Icee, 2009.
[29] Shenhan Qian. Vhap: Versatile head alignment with adaptive appearance priors, sep 2024. URL https
/VHAP
[30] Shenhan Qian. Tobias Kirschstein, Liam Schoneveld, Davide Davoli, Simon Giebenhain, and Matthias Nießner. Gaussianavatars: Photorealistic head avatars with rigged 3d gaussians. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20299-20300 2024.
[31] Gilles Rainer. Adrien Bousseau, Tobias Ritschel, and George Drettakis. Neural precomputed radiance transfer. In Computer graphics forum, volume 41, pages 365-378. Wiley Online Library, 2022.
[32] Pramod Rao, Mallikarjun BR, Gereon Fox, Tim Weyrich, Bernd Bickel, Hans- Seidel, Hanspeter Pfister, Wojciech Matusik, Ayush Tewari, Christian Theobalt, et al. Vorf: Volumetric relightable faces. In 33rd British Machine Vision Conference, 2022.
12

---

[33] Jérémy Riviere. Paulo FU Gotardo, Derek Bradley, Abbijeet Ghosh, and Thabo Beeler. Single-shot high-quality facial geometry and skin appearance capture. ACM Trans. Graph., 39(4):81, 2020.
[34] Shunsuke Saito, Gabriel Schwartz, Tomas Simon, Junxuan Li, and Giljoo Nam. Relightable gaussian codec avatars. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 130-141, 2024.
[35] Kripasindhu Sarkar, Marcel C Bühler, Gengyan Li, Daoye Wang, Delio Vicini, Jérémy Riviere, Yinda Zhang. Sergio for high-quality view synthesis and relighting of faces. In SIGGRAPH Asia 2023 Conference Papers, pages
Paulo Gotardo, Thabo Beeler. et al. Litnerf: Intrinsic radiance decomposition
1-11. 2023.
[36] Christophe Schlick. An inexpensive brdf model for physically based rendering. In Computer graphics forum, volume 13, pages 233-246. Wiley Online Library, 1994.
[37] Giota Stratou, Abhijeet Ghosh, Paul Debevec, and Louis-PPillipe Morency. Effect of illumination on automatic expression recognition: a novel 3d relightable facial database. In 2011 IEEE International Conference on Automatic Face & Gesture Recognition (FG), pages 611-618. IEEE, 2011.
[38] Tiancheng Sun, Jonathan T Barron, Yun-Ta Tsai, Zexiang Xu, Xueming Yu, Graham Fyffe, Christoph Rhemann, Jay Busch, Paul E Debevec, and Ravi Ramamoorthi. Single image portrait relighting. ACM Trans. Graph., 38(4):79- 20199.
[39] Bruce Walter, Stephen R Marschner, Hongsong Li, and Kenneth E Torrance. Microfacet models for refraction through rough surfaces. Rendering techniques, 2007:18th, 2007.
[40] Shaofei Wang, Tomas Simon. Igor Santesteban, Timur Bagautdinov, Junxuan Li. Vasu Agrawal, Fabian Prada, Shoou-1 Yu, Pace Nalbone, Matt Gramlich, Roman Lubachersky, Chenglei Wu, Javier Romero, Jason Saragih, Michael Zollhoefer. Andreas Geiger. Siyu Tang, and Shunsuke Saito. Relightable full-body gaussian codec avatars. arXiv.org, 2501. 14726, 2025.
[41] Zhibo Wang, Xin Yu, Ming Lu, Quan Wang, Chen Qian, and Feng Xu. Single image portrait relighting via explicit multiple reflectance channel modeling. ACM Transactions on Graphics (ToG), 39(6):1133, 2020.
[42] Zhou Wang, Alan Bovik, Hamid R Sheikh, and Eero Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing, 13(4))600- 612, 2004.
[43] Andreas Wenger, Andrew Gardner. Chris Tchou, Jonas Unger, Tim Hawkins, and Paul Debevec. Perfor- mance relighting and reflectance transformation with multipleeeee illumination. ACM Transactions on Graphics (TOG), 24(3) 764, 2005.
[44] Liangbin Xie, Xintao Wang, Honglun Zhang, Chao Dong, and Ying Shan. Vfhq: A high-quality dataset and benchmark for video face super-resolution. The IEEE Conference on Computer Vision and Puttern Recognition Workshops (CVPRW), 2022.
[45] Yingyan Xu, Jérémy Riviere, Gaspard Zoss. Prashanth Chandran, Derek Bradley, and Paulo Gotardo. Improved lighting models for facial appearance capture. EG (022-Short Papers, pages 5-8. 2022.
[46] Yingyan Xu, Gaspard Zoss, Prashanth Chandran, Markus Gross, Derek Bradley, and Paulo Gotardo. Renerf: Relightable neural radiance fields with nearfield lighting. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 22581- -221112221112222 2023.
[47] Yuelang Xu, Benwang Chen, Zhe Li, Hongwen Zhang, Lizhen Wang, Zerong Zheng, and Yebin Liu. Gaussian head avatar: Ultra high- fidelity head avatar via dynamic gaussians. In Proceedings of the IEEE/CVF conference on computer vision and patern recognition, pages 1931-1941, 2024.
[48] Zexiang Xu, Kalyan Sunkavalli, Sunil Hadap, and Ravi Ramamoorthi. Deep image-based relighting from optimal sparse samples. ACM Transactions Graphics (ToG), 37(4):1-33, 2018
[49] Haotian Yang, Mingwu Zheng, Wanquan Feng, Haibin Huang, Yu-Kun Lai, Pengfei Wan, Zhongyuan Wang, and Chongyang Ma. Towards practical capture of high-fidelity relightable avatars. In SIGGRAPH Asia 2023 Conference Papers, pages 1-11, 2023.
[50] Vickie Ye, Ruilong Li, Justin Kerr, Matias Turkulainen, Brent Yi, Zhuoyang Pan, Otto Seiskari, Jianbo Ye, Jeffrey Hu, Matthew Tancik, and Angioo Kanazawa. gsplat: An spercsource library for Gaussian splatting. arXiv preprint 2xX1v:2409.06765 2024. URL htt.pstss:///axii v org/ abs/ 2409 06765
[51] Yu- Ying Yeh, Koki Nagano, Sameh Khamis, Jan Kautz, Ming Yu Liu, and Ting- Chun Wang. Learning to relight portrait images via virtual light stage and synthetic- centhetic-to-ree re-l adaptation. ACM Transactions on Graphics (TOG), (1(6):1-21.1 2022.
13

---

[52] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features perceptual metric. In Proceedings of the IEEE conference computer vision and puttern recognition, pages 586-595, 2018.
[53] Xiuming Zhang, Sean Fanello, Yun-Ta Tsai, Tiancheng Sun, Tianfan Xue, Rohit Pandey, Sergio Orts- Escolano, Philip Davidson. Christoph Rhemann, Paul Debevec, et al. Neural light transport for relighting and view synthesis. ACM Transactions on Graphics (TOG), 40(1):1-17. 2021.
[54] Xiaochen Zhao, Lizhen Wang. Jingxiang Sun, Hongwen Zhang, Jinli Suo, and Yebin Liu. Havatar: High- fidelity head avatar via facial model conditioned neural radiance field. ACM Transactions on Graphics, 43 (1):1 16, 2023.
[55] Peng Zheng. Dehong Gao. Deng-Ping Fan, Li Liu, Jorma Laaksonen, Wanli Ouyang, and Nicu Sebe. Bilateral reference for high-resolution dichotomous image segmentation. CAAI Artificial Intelligence Research, 3:9150038, 2024.
[56] Yinglin Zheng. Hao Yang. Ting Zhang. Jianmin Bao, Dongdong Chen, Yangyu Huang, Lu Yuan, Dong Chen, Ming Zeng, and Fang Wen. General facial representation learning in a visual-linguistic manner. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18697- 18709, 2022.
[57] Wojciech Zielonka, Timo Bolkart, Justus Thies. Instant volumetric head avatars. Proceedings of IEEF/CVF conference on computer vision and pattern recognition, pages 4574-4584, 2023.
14

---

NeurIPS Paper Checklist
1. Claims
Question: the main claims made the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our main contributions from the abstract are a novel dataset of faces and an avatar representation with a new hybrid shading scheme. The dataset is introduced in Sec. 3 the avatar in
Sec. We validate our contributions with experiments in Sec. 5
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can he expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
A discussion of limitations can be found in the respective paragraph in Sec. 5.5
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper limitations, those are not discussed the paper.
The authors encouraged create separate" "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results these assumptions (c-g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
violations of
The authors should reflect on the scope of the claims made, e.g.. if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should he articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting.
speech-to- peech-to-texx text system might not used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms how they scale with dataset size.
applicabll, the authors should discuss possible limitations their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification: The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and wross-referenced.
15

---

All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper the extent that it affects main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: We list implementation details including hyperparameters Sec. 4.5 and Sec. [4.4 Detailed network architectures are included in the supplementary. In addition, code and data will be publicly released upon acceptance.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, contribution novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (c.g., in the case of a large language model), relcasing of model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) contribution a new model (e.g.. . a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g.. with open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may tricky in some cases, which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-soure source models, it may be that access to the model is limited in some way (c.g.. registered users), but it should be possible for other rescarchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open faithfully reproduce main experimental results, as described supplemental material?
the data and code, with sufficient instructions
Answer: [Yes]
Justification: Code and data will be released upon acceptance, Following GDPR, we will set up download form to manage the distribution. Unfortunately, this is not possible in an anonymous form.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines
guides/Codd Codee/bdeeSSiiccn asionPolicyy for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
16

---

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines https:://nip.cc/puabbbb for more details.
Suadde//DodeSubmiissioooooo
The authors should provide instructions data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: See Sec. 4.5 the main paper. Detailed network architectures and hyperparameters are provided in the supplementary. Data splits will be released with the dataset upon acceptance.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Dnes the paper report error hars suitably and correctly defined or other appropriate informa- tion about the statistical significance of the experiments?
Answer: [No]
Justification: The quantitative results do not contain statistical significance tests due to limited computational resources.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanicd by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful symmetric error bars would yield results that are out of range (e.g. negative error rates). error bars reported in tables plots, The authors should explain in the text how they calculated and reference the corresponding figures or tables the text.
show in tables figures
8. Experiments compute resources
Question: For each experiment, does the paper provide sufficient information the computer resources (type of compute workers, memory, time execution) needed reproduce the experiments?
Answer: [Yes]
Justification:
Details are listed in Section of the main paper. resources are the same for all experiments.
Guidelines:
17

---

The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn 't make into the paper).
9. Code of ethics
Does the research conducted the paper conform, every respect, with the NeurIPS Code of Ethics httppe//neerrss..c/pollttt/ttt
Question:
Answer: [No]
Justification: deviate from the Code of ethics with respect to human research subjects to whom we cannot provide financial compensation. All captured subjects participated on a voluntary basis.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
the authors answer No, they should explain the special circumstances that roquire a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
A discussion of ethical concerns can be found in Sec. 5.5 in the respective paragraph.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deploy- ment technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it not needed point out that generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative socictal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitor- ing misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility MI.).
11. Safeguards
Does the paper describe safeguards that have been put in place responsible release of data or models that have a high risk for misuse (e.g.. pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: We discuss the sensitivity of our dataset in Sec. 3.4 and outline how minimize the risk for misuse when distributing it.
Guidelines:
The answer NA means that the paper poses such risks.
18

---

Released models that have high risk for misuse or dual-use should be released with necessary safeguards allow for controlled use of the model, for example by requiring that adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We cited all used code in the main paper (Sec. 4.5) Detailed licence information included in the supplementary.
Guidelines:
answer means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g.. CC-BC 4.0) should be included for each asset. For scraped data from particular source (c.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms of use the package should he provided. For popular datasets, paapersai snperrawtthode..c thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should he provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Our dataset is thoroughly documented in Sec. Additional documentation will be made available with the release of the datasct upon acc
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details of the dataset/code/moodl as part of their sub- missions via structured templates. This includes details about training, license, limitations,
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and sereenshots, if applicable, as well as details about compensation (if any)?
Answer: [Yes]
Justification: The script given the participants of our dataset is provided in the supplementary material. Data collection and curation was handeled exclusively by the authors. All captured subjects participated on a voluntary basis.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
19

---

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other
labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [Yes]
Justification:
We confirmed that our data collection is compliant with local privacy regulations.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval. you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
initial submissions, not include any information that would break anonymity applica- ble), such as institution conducting the review.
16. Declaration of LLL usage
Question: Does the paper describe the usage of LLMs it an important, original, or non-standes aon-standard component of the core methods in this research? Note that if the LLM used only for writing, editing. formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is required.
Answer: [NA]
Justification:
Guidelines:
The answer NA that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer our LLM policy (https: htteee////ouuggeccc//oooorrrrrrr for what should or should not described.
20

---

BecomingLit: Relightable Gaussian Avatars with Hybrid Neural Shading
Supplementary Material
Jonathan Schmidt
Simon Giebenhain
Matthias Nießner
Technical University of Munich
jonathsch github 1athub...o/beccmnnggl..
A Network Architecture
Our geometry module F, maps FLAME [7] expression, jaw and eyes pose coefficients, predicts per-texel attributes {ōfτ, q, frape] ( A where fx has a dimension of 32 in all experiments. We use the first 100 principal components for the expression parameter, and rodrigues parameterization for jaw and both eye rotations. Hence, our input shape 109 is transformed by linear layer and then reshaped to 256 set of transposed convolutional layers then gradually upsamples the feature maps to the final output of shape 43 x 512 x 512. We use leaky-Rel.U as activation function for all layers except for the final output. For all convolutional layers, we adopt untied hias
Fu takes input the per- Ger-Gausiia feature map ferp" and the view direction, which is encoded using single linear layer (8-dim output shape) and then expanded the height and with dimension of the feature map. We concatenate feature map and encoded view direction and feed through single convolutional layer which downsamples the input by half. Finally, a transposed convolutional layer maps the latent feature map back to its original resolution with 4 output channels.
Our diffuse BRDF network Fa is a 3-layer MLP with hidden dimension 64 and ReLU activation in every layer, except the last onc. The input the concatenation of fe and the spherical harmonics coefficients of the incident light.
B Environment Map Rendering
In this section, we provide further details on how we render our avatars with all-frequency continuous illumination in the form of environment maps. While our diffuse BRDF trivially adopts to continuous illumination due to the spherical harmonies parameterization, we need to adopt the specular shading of the primitives.
For the specular pre-integration, follow the split-sum approximation [1), Karis al. 14 propose assume that the view direction w. and the surface normal are identical. With that assumption, the specular reflection longer view-dependent, and we can pre-integrate environment map different roughness values using a mipmap. In each mipmap level, we numerically integrate L, with importance sampling using the Blinn-Phong distribution:
DGF
L speccalar w.. = L.(we)) rwws nidwer k. 4(ww.)) (nw))
dwss
The incoming illumination L, (w,) stored in the pre-integrated environment map Lapeculaa r). During rendering, we linearly interpolate the mip levels to obtain the final radiance value the roughness parameter. Hence, the new specular term becomes:
I sppccul (xc, wo) Ro Isspecuter /
DGF ((w.))(mm))))
dwi,
(9)
The remaining integral is essentially the integration of the BRDF with completely white environment light. We substitute the Fresnel term F(Wo, h) with the Schlick approximation (36) and factor out Fo:

---

RGCA
RGCA (subset)
Ours
Ours (subset)
GT
RGCA
RGCA (subset)
Ours
Ours (subset)
Target Expression
Qualitative comparison on held-out frames and held-out light patterns from an unseen viewpoint.
Figure 8: Goliath-4 Evaluation:
Jo fo folw;; wo)(n wi)dw; Fo S falwe, (1 (1 wo 1t)") (wi ne)dw;
FWooh h
L La(we, wa) F Flwo.t h (1 wo h (wi n)dws (10)
These integrals depend the two inputs (w n) and the roughness parameter and act scale and bias to Fo. We pre-integrate both terms for all possible input combinations in [0, 1]² and store the two outputs in 2D texture map fo (w,
During rendering, can now compute the shaded color by evaluating the following terms:
w. -w. - -2(-w. 12)
(11)
b fo((u, n))r)
(12)
ci(w)) (a kis B) (wr,r)
(13)
C
Goliath-4 Dataset
demonstrate that our method also generalize well to data domains beyond our new light stage dataset, we perform additional experiments on the Goliath-4 dataset (22). We train both our method and RGCA 34] in two different configurations. (1) With the full available set of cameras, holding out 10 random views for evaluation. and (2) with random subset of 16 train cameras and 4 cameras for evaluation. We limit ourselves to one of the subjects (QZX685) and the provided train/test split for evaluating unseen expressions and hold random subset of 10% of the available light patterns to evaluate relighting capabilities. Our method consistently outperforms RGCA on the SSIM and LPIPS metrics. We report quantitative results in Table and a qualitative comparison in Figure We want to point out, that the publicly released dataset (22) is subsampled to every 10th frame, and the provided images are heavily compressed, which inevitably leads drop in quality compared to the results shown
D Capture Script
For each participant of our dataset, we record sequences in total. The first consist of a predetined set of facial expressions, emotions and sentences that we ask the subjects to perform and read out. In the 7th sequence the participant free perform any facial expression for 20s. The instructions given via screen that is placed

---

Table Goliath-4 Dataset: Quantitative results on RelfaReenattmet and Relighting. We evaluate relf-reenactment on the validation camera views.
Full Cam Set
Random Cam Subset (10%)
Method
PSNR
SSIM
LPIPS
PSNR1
SSIM
LPIPS
RGCA
29.89
0.8869
0.1392
28.74
0.8753
0. 0.1468
Ours
29.70
0.9080
0.1165
28.68
0.8966
0. 0.1298
in front of the subject. In the following provide list of the single components, which during the capture sessions accompanied with images.
Expressions-I:
Head rotation with mouth open closed
Eyes blink
Eyes squint
Eyebrows up down
Puffed Cheeks
Mouth Vacuum
Nose Wrinkle
Lip bite
Expressions-2
Grin (multiple variations)
Jaw movement
Lip licking
Tongue
Emotions
Shout
Laugh
Surprise
Fear
Angry
Disgust
Happy
Confusion
Amazement
Embarrassment
Sentences-1
A cramp is no small danger swim.
He said the same phrase thirty times.
Pluck the bright rose without leaves.
Two plus seven less than ten.
The glow deepened in eyes of the sweet girl.
By eating yogurt you may live longer.
Sentences-2
Bring your problems the wise chief.
Write fond note the friend you cherish.
Clothes and lodging are free to new men.
We frown when events take bad turn.
Port is a strong wine with smoky taste.
They had slapped their thighs.
Sentences-3

---

She always jokes about too much garlic in his food.
Why put such a high value on being top dog.
All your wishful thinking won't I change that.
Take charge of choosing her bridesmaids gowns.
Why buy oil when you always use mine.


Reviews:
[{"review_id": "JUcs3Up8ye", "reviewer": "Reviewer_Awo4", "summary": "This paper proposes a method to reconstruct a relightable head avatar from a light stage, which contains multiple calibrated cameras and controllable lights. In the method section, the model learns static material properties and decodes Gaussian attributes using a CNN decoder. One of the contributions proposed by the author is to separate the shading into neural diffuse part and  analytical specular part, which can better recover the materials and obtain improved rendering results. In the experiments, compared to the previous state-of-the-art method RGCA, the proposed method can render images with more realistic lighting and finer details. Additionally, the author's claim about the release of the dataset and code is fascinating, and I believe it can further advance the development of the field of relightable head avatars.", "questions": "1. It seems that the quantitative results regarding the rendering speed are missing. It would be better to provide the speed data and a comparison with RGCA, since real-time rendering is claimed in the main paper. The proposed method includes many components, such as CNNs and MLPs, as well as shading computation. It would be valuable to provide the running time for each module.\n2. I want to confirm whether the shadow effect is learned from the tiny neural network Fd, as mentioned in line 168 regarding \"self-shadowing effects.\" If so, would it be possible to provide a visual result of the shadow map learned from Fd?", "limitations": "yes.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Advantages:\n1. The results of the paper and video are realistic and vivid. The proposed method can not only recover reasonable face materials from the captured data, but it is also able to render the results under challenging lighting conditions, such as OLAT light and real-world environmental lighting. Even when the avatar is driven by novel expressions and head poses, the details of the face, such as small wrinkles and hair, as well as shadows, can be correctly rendered as the environmental lighting changes.\n2. The author claims to release the code and dataset, which are essential for this area. Several previous papers related to head avatar relighting captured their own datasets. However, they either do not open source their data, or the data they release is of poor quality. In this paper, the captured data includes many subjects with OLAT lights and high resolution, and the promise to release code that can reproduce the results is, in my opinion, a great contribution.\n3. Although the previous method, RGCA, can render realistic relighting results, the proposed method can relight the avatar with finer details and fewer artifacts. In the experiment section, the proposed method outperforms RGCA both qualitatively and quantitatively.\n\nWeaknesses:\n1. The data capturing setup is still somewhat complicated, and the training time for a single subject is a bit long. Although these issues are well discussed in the limitations section, they are still worth mentioning, as the complicated process can prevent other users from utilizing the method. Training a prior and fine-tuning the model may help reduce the requirements for the captured data and the training time.", "was_revised": true, "final_justification": "The rebuttal have well addressed my main concerns. I will keep my initial rating and lean toward acceptance for this submission.", "rebuttal": ">The data capturing setup is still somewhat complicated, and the training time for a single subject is a bit long\n\nOur hardware and space requirements are significantly lower compared to existing setups (see Table 1 of the main paper). We acknowledge that our method does indeed not work with casually captured data. However, we are releasing our dataset to make future research accessible to everyone.\n\nThe training time (30h on a single GPU)  is comparable to concurrent and faster than previous work, which used NeRF-based representations. The primary goal of our work is to get high-quality reconstructions and run inference, i.e., relighting, animation, and novel-view synthesis, in real-time.\n\n>Training a prior and fine-tuning the model may help reduce the requirements for the captured data and the training time.\n\nRelightable Avatar reconstruction from casually captured images or videos is the ultimate goal of research in this direction. We discussed in Sec. 5.4 that learning an appearance prior of human heads is an exciting direction for future work, and we expect that the release of our dataset opens up research in this direction for a broader community.\n\n>It seems that the quantitative results regarding the rendering speed are missing. It would be better to provide the speed data and a comparison with RGCA, since real-time rendering is claimed in the main paper. The proposed method includes many components, such as CNNs and MLPs, as well as shading computation. It would be valuable to provide the running time for each module.\n\nIn the table below, we summarize the runtime of the components of our method and compare it with RGCA [34]. We conducted all measurements on a single NVIDIA RTX A6000 GPU  using a UV resolution of $512^2$, which results in 202k primitives. We render images at a resolution of 1100x1604, corresponding to the training resolution of our avatars.\n\n\n| Method | CNNs | Diffuse Shading | Specular Shading | Splatting / Rasterization | Total |\n|--------|:----:|:---------------:|:----------------:|:-------------------------:|:-----:|\n| RGCA   | 9ms  | 1ms             | 1ms.             | 9ms                       | 20ms  |\n| Ours   | 4ms  | 3ms             | 1ms.             | 9ms                       | 17ms  |\n\nWe will add this table to the final version of the paper. Both methods use the same gsplat splatting/rasterization backbone. Hence, we only report the rasterization speeds for completeness.\n\n>I want to confirm whether the shadow effect is learned from the tiny neural network Fd, as mentioned in line 168 regarding \"self-shadowing effects.\" If so, would it be possible to provide a visual result of the shadow map learned from Fd?\n\nYes, all global illumination effects (e.g., shadows, indirect illumination, etc.) are learned implicitly by the neural diffuse BSDF, since a physically-based approach would not allow for real-time \nrendering and would further increase the training time of the avatars. Since the diffuse BSDF learns all global illumination effects jointly, we cannot visualize the shadow map only. We do, however, depict the output of the diffuse BSDF in Figure 3, right to the $F_d$ network, and will add more vivid visualizations to the final paper. Further, we want to point to Figure 5, where the learned shadow can be observed in columns (c) and (e), demonstrating that it closely matches the shadow in the corresponding reference image in column (f)."}, {"review_id": "uyyGK5snX7", "reviewer": "Reviewer_rxBB", "summary": "Authors build a pipeline for relightable avatars acquired with a low cost light stage setup. They collect subjects under multiple viewpoints, illuminations and different facial expressions. Their model is based on a head parametric model (FLAME), which is further optimized to capture the real subject geometry. Their shading model is hybrid, using a neural diffuse BRDF combined with an analytical specular term. They prove high quality results under different conditions. Their dataset is publicly available to accelerate the research in that direction.", "questions": "My main question for authors would be to better clarify the differences with [34], since their explanation is simply “we prefer a hybrid architecture for the rendering”, and also why their comparisons with [34] do not seem to match what authors showed in the original paper.\n\nFinally I’d like them to discuss more around the limitations I pointed out in the next section.", "limitations": "Authors discuss some limitations but they did not mention the following:\n\nDensity of the lighting setup: while the setup is indeed less expensive than a full light stage, the light density seems to cause issues, in particular OLAT like reflections can be observed when the subjects are relit (see video) or when predicting albedo (Fig 5). Authors should discuss this limitation and how they plan to address it.\n\nAlbedo prediction is a simpler problem when starting with OLAT images, but in practice, this is not the actual problem to solve. Indeed a more challenging case that authors did not address is how to build an avatar from any arbitrary lighting condition, which would mimic real-world scenarios. Also note that the albedo results show issues specifically around the eyes, where OLAT reflections can be observed (Figure 5)", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 2, "strengths_and_weaknesses": "[Strength] Relevant problem, well explained by authors and how they solve it\n[Strength] Dataset release to accelerate the research in relightable avatars\n[Strength] The method is simple yet effective and authors do a good job at explaining all the details.\n[Strength] Well presented video results\n\n[Weakness] My main criticism is around novelty and specifically comparing it with [34], while I understand the difference in terms of architecture (explicit pre-computed radiance transfer function vs hybrid), I did not find enough evidence by authors why this is needed.  Additionally, the comparisons authors show here do not seem to align with the results presented in [34] (see my next point)\n\n[Weakness] Video results show limitations of the method, in particular around eyes. HDRI results show “OLATs” like reflections. The comparisons with Relightable Gaussian Codec Avatars [34] do not seem to match the original results in the paper. Indeed RGCA demonstrates high quality HDRI results, whereas authors mention in their video that the previous method “struggles” in that scenario.", "was_revised": true, "final_justification": "I thank the authors for the additional experiments and clarification, I am ok raising my rating to Borderline Accept. I would recommend add in the Limitations sections the ones I have suggested, in particular the challenges with real-world captures where users do not have access to an OLAT setup when building an avatar.", "rebuttal": ">My main question for authors would be to better clarify the differences with [34], since their explanation is simply “we prefer a hybrid architecture for the rendering”, and also why their comparisons with [34] do not seem to match what authors showed in the original paper.\n\n### Differences to RGCA\n\nRGCA is based on a variational Autoencoder (VAE), which learns a **personalized expression space** from dense multi-view captures of a light stage setting. The input to the encoder are the vertices of a **non-rigidly registered** template mesh, with corresponding texture maps. The decoders predict the geometric properties of Gaussian primitives along with the parameters for the precomputed radiance transfer function and specular shading.\n\nIn contrast, our method builds directly on top of the **parametric** morphable model FLAME, which has a **shared expression space** across different identities. This key difference enables applications such as cross-reenactment, where expressions can be transferred between subjects, or generally speaking, an avatar can be animated with FLAME parameters from any source without the need for a personalized encoder. Therefore, we show animation from monocular videos, which is not possible with RGCA’s VAE approach.\n\nGiven these FLAME parameters, we predict expression-dependent offsets for Gaussian primitives rigged to the base mesh, and a set of expression features for our appearance model, which computes diffuse reflectance with a small MLP and specular reflectance using view-dependent normals and visibility in a Cook-Torrance model. As stated in the main paper, we acknowledge that obtaining the parameters for the specular shading is inspired by RGCA [34]. We will add these clarifications in the related work section of the final paper.\n\n### Ablation / Justification of Our Appearance Model\n\nBelow, we show two additional ablations to quantify the contribution of our new appearance model. The results are extensions to the ablation study in Table 3 of the main paper and follow the same evaluation protocol. For completeness, we also evaluate novel-view synthesis on the training frames.\n\n#### NVS\n\n| Method             | PSNR  | SSIM   | LPIPS  |\n|--------------------|:-----:|:------:|:------:|\n| Ours (Full)        | 32.12 | 0.9266 | 0.0699 |\n| Spherical Gaussians | 31.86 | 0.9254 | 0.0700 |\n| PRT Diffuse        | 30.70 | 0.9031 | 0.0987 |\n\n#### NVS + Relighting\n\n| Method             | PSNR  | SSIM   | LPIPS  |\n|--------------------|:-----:|:------:|:------:|\n| Ours (Full)        | 31.38 | 0.8956 | 0.1040 |\n| Spherical Gaussians | 31.55 | 0.8953 | 0.1031 |\n| PRT Diffuse        | 29.23 | 0.8374 | 0.1577 |\n\n#### NVS + Relighting + Self-Reenactment\n\n| Method             | PSNR  | SSIM   | LPIPS  |\n|--------------------|:-----:|:------:|:------:|\n| Ours (Full)        | 31.38 | 0.8956 | 0.1040 |\n| Spherical Gaussians | 31.55 | 0.8953 | 0.1031 |\n| PRT Diffuse        | 29.23 | 0.8374 | 0.1577 |\n\n**Spherical Gaussians Specular**: In this experiment, we replace our Cook-Torrance specular term with the spherical Gaussians formulation from RGCA [34], leaving the rest of our method untouched. While we observe slightly better results in almost all metrics, we still decided in favor of the Cook-Torrance model since we observed slightly sharper specular highlights under environment relighting. \n\n**Precomputed Radiance Transfer (PRT)**: Here, we replace our neural diffuse BRDF with the PRT model from RGCA [34]. Therefore, we modify our dynamics model $F_g$ to also predict the spherical harmonics coefficients for the PRT function of the primitives, which we then integrate with the incident lighting. While we observe reasonable results under training illumination, PRT struggles to generalize to novel illuminations, which aligns with our observations from the comparisons with RGCA in Table 2 and Figures 4 and 7.\n\nWe will add the qualitative comparisons of both ablations to the final version of the paper.\n\n### Our RGCA Results Compared to the Original Evaluation\n\nRGCA results from the original paper were obtained using a capture setup with 160 cameras and 460 lights densely distributed on a 360-degree dome structure. Our comparisons were conducted with data from our capture setup, consisting of (only) 16 cameras and 40 lights, and covering only the frontal hemisphere around the head.\n\nAnother difference is the evaluation protocol. While we test and visualize novel-view synthesis, unseen illuminations, and reenactment jointly in Table 2 and Figure 4, RGCA evaluates each modality separately, e.g., Figure 9 of [34] shows novel-view synthesis on a training frame and Figure 10 in [34] shows relighting on a training view and training expression which is a much less challenging setting than in our evaluation. Also, while we hold out a set of lights from training altogether, RGCA only excludes a set of light patterns, which, due to the additive nature of light, is a less meaningful test. Still, we indeed acknowledge that RGCA can produce higher-fidelity results when trained with more camera views and light patterns.\n\nTo quantify these claims, we present the results of an additional evaluation on the Goliath-4 dataset, which is similar to the original training setting of RGCA. Note, however, that the publicly released dataset is subsampled to every 10th frame, and the provided images are heavily compressed. The required FLAME tracking we computed for the Goliath-4 dataset is also not perfectly accurate due to the limited time of the rebuttal period. We train both our method and RGCA in three different configurations. (1) With the full available set of cameras, holding out 10 random views for evaluation, (2) with a random subset of 16 train cameras and 4 cameras for evaluation, and (3), with a subset of 16 train and 4 test cameras sampled from the frontal region to mimic a similar setting than in our capture setup.\n\nDue to the limited time, we limit ourselves to one of the subjects (QZX685) and use the provided train/test split for evaluating unseen expressions and hold out a random subset of 10% of the total light patterns to evaluate relighting capabilities. Our method consistently outperforms RGCA on the SSIM and LPIPS metrics. We will add a qualitative comparison to the final version of the paper.\n\n#### Full Set of Cameras\n\n| Method             | PSNR  | SSIM   | LPIPS  |\n|--------------------|:-----:|:------:|:------:|\n| RGCA               | 29.89 | 0.8869 | 0.1392 |\n| Ours               | 29.70 | 0.9080 | 0.1165 |\n\n#### Random Camera Subset (10%)\n\n| Method             | PSNR  | SSIM   | LPIPS  |\n|--------------------|:-----:|:------:|:------:|\n| RGCA               | 28.74 | 0.8753 | 0.1468 |\n| Ours               | 28.68 | 0.8966 | 0.1298 |\n\n#### Frontal Camera Subset (10%)\n\n| Method             | PSNR  | SSIM   | LPIPS  |\n|--------------------|:-----:|:------:|:------:|\n| RGCA               | 29.88 | 0.8922 | 0.1326 |\n| Ours               | 29.82 | 0.9137 | 0.1139 |\n\n>Density of the lighting setup: while the setup is indeed less expensive than a full light stage, the light density seems to cause issues, in particular OLAT like reflections can be observed when the subjects are relit (see video) or when predicting albedo (Fig 5). Authors should discuss this limitation and how they plan to address it.\n\nThe mentioned artifacts are also visible when training on the Goliath-4 dataset, which consists of a setup with an order of magnitude more lights than our setup. Thus, we conclude that this is a general limitation of our method, and is not caused by our dataset.\n\nThe intrinsic decomposition is not perfect in the eyes region, since we use only a single specular model for all parts of the face, and eyeballs should show mirror-like reflections. Potential solutions would be an explicit eye model, such as in [Li 22, Schwartz 20], or a heterogeneous optimization/material model for the eyeballs based on the FLAME eye geometry. We will add this discussion to the final version of the paper.\n\n*Li et al. 2022 \"Eyenerf: a hybrid representation for photorealistic synthesis, animation and relighting of human eyes.\"*\n\n*Schwartz et al. 2020 \"The eyes have it: An integrated eye and face model for photorealistic facial animation.\"*\n\n>Albedo prediction is a simpler problem when starting with OLAT images, but in practice, this is not the actual problem to solve. Indeed a more challenging case that authors did not address is how to build an avatar from any arbitrary lighting condition, which would mimic real-world scenarios.\n\nWe agree that having OLAT image data from a calibrated light stage setup simplifies the problem of recovering the geometry and material properties, and we also believe that the ultimate goal is to build relightable avatars from casually captured video data.\n\nHowever, there remain numerous open research questions regarding the geometry representation used to model facial expressions and hair, and also how to represent the heterogeneous materials present on human heads in a realistic and easily optimizable manner.\nWe believe that finding new solutions to those open questions can legitimately be conducted in a more controlled setting, i.e., with a light stage and calibrated OLAT images, and that the generated insights will eventually also lead to research progress of relightable avatars from casually captured images. Moreover, we want to highlight our dataset contribution, which we believe will accelerate research in this direction."}, {"review_id": "kenjGDmJQU", "reviewer": "Reviewer_CBwD", "summary": "The paper  proposes a method for reconstructing photo-realistic and relightable avatars from a ‘lightweight’ capture setup equipped with OLAT. First FLAME fitting is obtained from fully lit images of the lightstage. Then an expression dependent network is trained to predict gaussian attributes and an additional features vector. This feature vector is used to learn an implicit diffuse radiance with an explicit analytical specular BRDF. \nCompared to Codec Avatars from Meta (RGCA) on the proposed dataset, the method outperforms RGCA especially in relighting.", "questions": "My major concerns are the following:\n\n1. It is on the scientific contribution and the positioning of the paper with respect to RGCA. It is clear that this work is built on top of RGCA, however how this differs (other than using the implicit diffuse component) needs to be stated clearly. And it seems that this implicit diffuse term is the only difference which makes the scientific contribution limited to me. Also, reading the related work does not help positioning the paper and this needs to be improved. Also the claims provided in the introduction needs to be more specific to highlight the paper contribution \n\n2. Using an hybrid reflectance field (with diffuse term learned implicitly) may violate energy conservation and the method does not seem to handle this which may lead to some saturation or artefacts. This need to be discussed as well\n\n3. The RGCA works show their method still performs well with a low number of cameras. However results shown here for RGCA (in relighting) look completely off.  Especially that fig 8 in supp shows fair decomposition from RGCA. This needs to be discussed as well.\n\n4. The predicted normals are tight to FLAME normals by design using a soft constraint. Is this enough to capture fine grained details such as wrinkles/folds in the normal map ?showing predicted normals map can be very insightful.  \n\nI like the results shown in the paper, also releasing a dataset with OLAT if of great value for the community.  However I am more concerned about the contributions itself. I would like to hear from authors and see other reviewers before making the final decision", "limitations": "yes", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "The paper reads very well and the problem addressed here is well sound and very active. The release of OLAT dataset is highly appreciated by the community. The idea of using a hybrid BRDF material with only diffuse  components learned implicitly seems effective and allows the network to complete the cook-torrance model by learning global illumination effects.", "was_revised": true, "final_justification": "I would like to thank the authors for addressing my concerns in the rebuttal. While I agree that the scientific contribution is a bit incremental with respect to RCGA. However, given the extended experimental protocol provided in the rebuttal, good paper flow, nice results and the release of the OLAT dataset (which is very valuable for the community), I vote for accepting this paper.", "rebuttal": ">It is on the scientific contribution and the positioning of the paper with respect to RGCA. It is clear that this work is built on top of RGCA, however how this differs (other than using the implicit diffuse component) needs to be stated clearly. And it seems that this implicit diffuse term is the only difference which makes the scientific contribution limited to me.Also, reading the related work does not help positioning the paper and this needs to be improved. Also the claims provided in the introduction needs to be more specific to highlight the paper contribution\n\nOur high-level goal is to reconstruct photorealistic, relightable head avatars that can be animated with novel expressions in real-time. In contrast to previous work, we focus on achieving this with a much more affordable capture setup, which we use to collect a dataset for the research community. Below, we detail the differences between our method and RGCA.\n\nRGCA is based on a variational Autoencoder (VAE), which learns a **personalized expression space** from dense multi-view captures of a light stage setting. The input to the encoder are the vertices of a **non-rigidly registered** template mesh, with corresponding texture maps. The decoders predict the geometric properties of Gaussian primitives along with the parameters for the precomputed radiance transfer function and specular shading.\n\nIn contrast, our method builds directly on top of the **parametric** morphable model FLAME, which has a **shared expression space** across different identities. This key difference enables applications such as cross-reenactment, where expressions can be transferred between subjects, or generally speaking, an avatar can be animated with FLAME parameters from any source without the need for a personalized encoder. Therefore, we show animation from monocular videos, which is not possible with RGCA’s VAE approach.\n\nGiven these FLAME parameters, we predict expression-dependent offsets for Gaussian primitives rigged to the base mesh, and a set of expression features for our appearance model, which computes diffuse reflectance with a small MLP and specular reflectance using view-dependent normals and visibility in a Cook-Torrance model. As stated in the main paper, we acknowledge that obtaining the parameters for the specular shading is inspired by RGCA [34].\n\nWe will clarify the differences to RGCA as stated above in the final version of the paper, and also update the introduction to state our contributions more comprehensively.\n\n>Using an hybrid reflectance field (with diffuse term learned implicitly) may violate energy conservation and the method does not seem to handle this which may lead to some saturation or artefacts. This need to be discussed as well\n\nWe decided to use a learned reflectance model in favor of a physically-based model since human skin consists of several different layers, each of which must be described with a subsurface scattering model. This would not only be hard to optimize but also impossible to render in real time. Further, our appearance model must not only cope with skin, but also with hair, eyeballs, the mouth interior, and teeth. Our ablation study in Table 3 and Figure 6 shows that a simple PBR shading model, as used in real-time graphics today, is not sufficient for representing these materials realistically.\n\nOur neural diffuse BRDF model does indeed not guarantee energy conservation. However, since it is trained on several thousand image observations, we conclude that fundamental characteristics of reflectance models, such as energy conservation, are learned implicitly to an acceptable level. We will point this out in the final version of the paper.\n\nLearning an appearance prior of human heads is an exciting direction for future work, and we expect that the release of our dataset opens up research in this direction for a broader community.\n\n>The RGCA works show their method still performs well with a low number of cameras. However results shown here for RGCA (in relighting) look completely off. Especially that fig 8 in supp shows fair decomposition from RGCA. This needs to be discussed as well.\n\nThe RGCA results from the original paper were obtained using a capture setup with 160 cameras and 460 lights densely distributed on a 360-degree dome structure. As you mentioned, their ablation shows that it also works reasonably well with a small random subset of cameras (Fig 9 of [34]). However, our comparisons were conducted with data from our capture setup, consisting of 16 cameras and 40 lights, covering **only the frontal hemisphere** around the head.\n\nAnother difference is the evaluation protocol. While we test and visualize novel-view synthesis, unseen illuminations, and reenactment jointly in Table 2 and Figure 4, RGCA evaluates each modality separately, e.g., Figure 9 of [34] shows novel-view synthesis on a training frame and Figure 10 in [34] shows relighting on a training view and training expression which is a much less challenging setting than in our evaluation. Also, while we hold out a set of lights from training altogether, RGCA only excludes a set of light patterns, which, due to the additive nature of light, is a less meaningful test. Still, we indeed acknowledge that RGCA can produce higher-fidelity results when trained with more camera views and light patterns.\n\nFigure 8 shows that RGCA fairly recovers albedo and normals. However, in our experiments, we found that the shown artifacts primarily arise from the learned diffuse PRT function, which tends to overfit to the training expressions and illuminations.\n\nBelow, we present the results of an additional evaluation on the Goliath-4 dataset, which is closer to the original training setting of RGCA. Note, however, that the publicly released dataset is subsampled to every 10th frame, and the provided images are heavily compressed. The required FLAME tracking we computed for the Goliath-4 dataset is also not perfectly accurate due to the limited time of the rebuttal period. We train both our method and RGCA in three different configurations. (1) With the full available set of cameras, holding out 10 random views for evaluation, (2) with a random subset of 16 train cameras and 4 cameras for evaluation, and (3), with a subset of 16 train and 4 test cameras sampled from the frontal region to mimic a similar setting than in our capture setup.\n\nDue to the limited time, we limit ourselves to one of the subjects (QZX685) and use the provided train/test split for evaluating unseen expressions and hold out a random subset of 10% of the total light patterns to evaluate relighting capabilities. Our method consistently outperforms RGCA on the SSIM and LPIPS metrics. We will add a qualitative comparison to the final version of the paper.\n\n#### Full Set of Cameras\n\n| Method             | PSNR  | SSIM   | LPIPS  |\n|--------------------|:-----:|:------:|:------:|\n| RGCA               | 29.89 | 0.8869 | 0.1392 |\n| Ours               | 29.70 | 0.9080 | 0.1165 |\n\n#### Random Camera Subset (10%)\n\n| Method             | PSNR  | SSIM   | LPIPS  |\n|--------------------|:-----:|:------:|:------:|\n| RGCA               | 28.74 | 0.8753 | 0.1468 |\n| Ours               | 28.68 | 0.8966 | 0.1298 |\n\n#### Frontal Camera Subset (10%)\n\n| Method             | PSNR  | SSIM   | LPIPS  |\n|--------------------|:-----:|:------:|:------:|\n| RGCA               | 29.88 | 0.8922 | 0.1326 |\n| Ours               | 29.82 | 0.9137 | 0.1139 |\n\n>The predicted normals are tight to FLAME normals by design using a soft constraint. Is this enough to capture fine grained details such as wrinkles/folds in the normal map ?showing predicted normals map can be very insightful.\n\nSimilar to normal maps in mesh rendering, our method uses the coarse FLAME mesh normals as base normals, and predicts expression and view-dependent offsets for the individual primitives. In addition, the $F_g$ CNN also predicts expression-dependent offsets for the primitive positions. Both combined can faithfully model wrinkles and folds of the skin beyond the capabilities of an ordinary normal map. Examples of the rendered normals can be found in Figure 5b. We see that the example shown is not ideal for highlighting the method’s capabilities with respect to modeling the respective effects, and we will include a more vivid example in the final version of the paper."}, {"review_id": "5Z09CI6Kzi", "reviewer": "Reviewer_B4Yt", "summary": "The paper introduces a novel dataset and a method that utilizes it, achieving state-of-the-art results. The introduced dataset is captured using a lightweight capture system consisting of 16 cameras and 40 lights, which is significantly smaller compared to, for instance, the RGCA dataset. Despite its smaller size, the quality provided by the neural appearance representation is very high, producing sharp and detailed avatars.", "questions": "*RGCA* and *BecomingLit* use different image formation models. From the initial evaluation, it appears that using the classical Cook-Torrance microfacet model for specular reflection gives superior results. However, I would like to see whether this holds true when both methods are trained on the same dataset. This is an interesting aspect that I believe could be better investigated. Specifically, the comparison between RGCA's linear radiance transfer and the small neural BRDF function used in BecomingLit to model light transport, without the need to store it per Gaussian primitive, deserves further analysis.\n\nI would suggest performing two additional evaluations: one on the full RGCA dataset using all 144 cameras, and another on a subsampled set of cameras. The drop in quality for RGCA likely also depends on the tracking. However, for the Goliath experiments, I would recommend using the original setup (with the provided tracking) and only modifying the number of views used. There may still be some inconsistencies, since I assume BecomingLit would need to use VHAP tracking on Goliath to remain compatible, but this would still provide more insight into the underlying model expressiveness.\n\n**Questions**\n1) Do you use any regularizers on the prediction of $F_v$ for Cook–Torrance formula?\n2) What is the motivation behind using the classical Cook–Torrance microfacet model for specular?", "limitations": "The quality is still lower compared to RGCA trained on the Goliath dataset, especially in terms of teeth and hair quality. However, it remains unclear how much of this issue is due to the dataset or the image formation model used.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Weaknesses**\n\n1) The provided evaluation is not sufficient. While I can imagine that RGCA does not perform well under few-view settings, I still have doubts about the presented quality; it appears significantly worse than in the original paper. I would suggest running BecomingLit on the RGCA-released dataset as well, to enable a fair comparison. \n2) There are still artifacts, especially when relighting dynamic facial parts such as the teeth.\n\n**Strenghts**\n\n1) Very high-quality results are obtained on a dataset that is orders of magnitude smaller compared to other state-of-the-art methods. The dataset will be publicly available, which is a valuable contribution to the community and will support further research on relightable avatars.\n2) The presented appearance model is an interesting alternative to the fully linear RGCA approach. RGCA uses a fully linear model for both specular and diffuse lighting, utilizing spherical harmonics for diffuse and spherical Gaussians for specular components, which allows for explicit light transfer. In the context of this work, the diffuse color (BRDF) is represented by a small neural network conditioned on lighting and expressions. The specular component, on the other hand, uses the Cook-Torrance microfacet model, where $k_s$ is predicted by a neural network $F_v$, along with normal perturbations $\\delta n_k$. The roughness $r_k$ is learned per Gaussian as a variable. This is an elegant formulation based on a widely used specular model from computer graphics. However, the effectiveness of this formulation remains to be seen, as I believe the presented results are not sufficient to determine which aspects contribute most to the performance.", "was_revised": true, "final_justification": "After reading the rebuttal, most of my concerns were clarified, and a new ablation was provided. I will maintain my previous score of Weak Accept. However, I would like to ask authors to attach the new findings in the final version of the paper.", "rebuttal": ">The provided evaluation is not sufficient. While I can imagine that RGCA does not perform well under few-view settings, I still have doubts about the presented quality; it appears significantly worse than in the original paper.\n\nRGCA results from the original paper were obtained using a capture setup with 160 cameras and 460 lights densely distributed on a 360-degree dome structure. Our comparisons were conducted with data from our capture setup, consisting of (only) 16 cameras and 40 lights, and covering only the frontal hemisphere around the head.\n\nAnother difference is the evaluation protocol. While we test and visualize novel-view synthesis, unseen illuminations, and reenactment jointly in Table 2 and Figure 4, RGCA evaluates each modality separately, e.g., Figure 9 of [34] shows novel-view synthesis on a training frame and Figure 10 in [34] shows relighting on a training view and training expression which is a much less challenging setting than in our evaluation. Also, while we hold out a set of lights from training altogether, RGCA only excludes a set of light patterns, which, due to the additive nature of light, is a less meaningful test. Still, we indeed acknowledge that RGCA can produce higher-fidelity results when trained with more camera views and light patterns.\n\n>I would suggest performing two additional evaluations: one on the full RGCA dataset using all 144 cameras, and another on a subsampled set of cameras. The drop in quality for RGCA likely also depends on the tracking. However, for the Goliath experiments, I would recommend using the original setup (with the provided tracking) and only modifying the number of views used. There may still be some inconsistencies, since I assume BecomingLit would need to use VHAP tracking on Goliath to remain compatible, but this would still provide more insight into the underlying model expressiveness.\n\nWe present the results of the additional evaluation on the Goliath-4 dataset below. Note, however, that the publicly released dataset is subsampled to every 10th frame, and the provided images are heavily compressed. The required FLAME tracking we computed for the Goliath-4 dataset is also not perfectly accurate due to the limited time of the rebuttal period.  We train both our method and RGCA in three different configurations. (1) With the full available set of cameras, holding out 10 random views for evaluation, (2) with a random subset of 16 train cameras and 4 cameras for evaluation, and (3), with a subset of 16 train and 4 test cameras sampled from the frontal region to mimic a similar setting than in our capture setup.\n\nDue to the limited time, we limit ourselves to one of the subjects (QZX685) and use the provided train/test split for evaluating unseen expressions and hold out a random subset of 10% of the total light patterns to evaluate relighting capabilities. Our method consistently outperforms RGCA on the SSIM and LPIPS metrics. We will add a qualitative comparison to the final version of the paper.\n\n#### Full Set of Cameras\n\n| Method             | PSNR  | SSIM   | LPIPS  |\n|--------------------|:-----:|:------:|:------:|\n| RGCA               | 29.89 | 0.8869 | 0.1392 |\n| Ours               | 29.70 | 0.9080 | 0.1165 |\n\n#### Random Camera Subset (10%)\n\n| Method             | PSNR  | SSIM   | LPIPS  |\n|--------------------|:-----:|:------:|:------:|\n| RGCA               | 28.74 | 0.8753 | 0.1468 |\n| Ours               | 28.68 | 0.8966 | 0.1298 |\n\n#### Frontal Camera Subset (10%)\n\n| Method             | PSNR  | SSIM   | LPIPS  |\n|--------------------|:-----:|:------:|:------:|\n| RGCA               | 29.88 | 0.8922 | 0.1326 |\n| Ours               | 29.82 | 0.9137 | 0.1139 |\n\n>RGCA and BecomingLit use different image formation models. From the initial evaluation, it appears that using the classical Cook-Torrance microfacet model for specular reflection gives superior results. However, I would like to see whether this holds true when both methods are trained on the same dataset. This is an interesting aspect that I believe could be better investigated. Specifically, the comparison between RGCA's linear radiance transfer and the small neural BRDF function used in BecomingLit to model light transport, without the need to store it per Gaussian primitive, deserves further analysis.\n\nFirst, we want to clarify that both methods were trained on the same dataset for all comparisons we made in the paper.\n\nIn addition to the evaluations on the Goliath-4 dataset shown above, we also present the results of two additional ablations to further justify the design of our appearance model. The results are extensions to the ablation study in Table 3 of the main paper and follow the same evaluation protocol. For completeness, we also evaluate novel-view synthesis on the training frames.\n\n#### NVS\n\n| Method             | PSNR  | SSIM   | LPIPS  |\n|--------------------|:-----:|:------:|:------:|\n| Ours (Full)        | 32.12 | 0.9266 | 0.0699 |\n| Spherical Gaussians | 31.86 | 0.9254 | 0.0700 |\n| PRT Diffuse        | 30.70 | 0.9031 | 0.0987 |\n\n#### NVS + Relighting\n\n| Method             | PSNR  | SSIM   | LPIPS  |\n|--------------------|:-----:|:------:|:------:|\n| Ours (Full)        | 31.38 | 0.8956 | 0.1040 |\n| Spherical Gaussians | 31.55 | 0.8953 | 0.1031 |\n| PRT Diffuse        | 29.23 | 0.8374 | 0.1577 |\n\n#### NVS + Relighting + Self-Reenactment\n\n| Method             | PSNR  | SSIM   | LPIPS  |\n|--------------------|:-----:|:------:|:------:|\n| Ours (Full)        | 31.38 | 0.8956 | 0.1040 |\n| Spherical Gaussians | 31.55 | 0.8953 | 0.1031 |\n| PRT Diffuse        | 29.23 | 0.8374 | 0.1577 |\n\n**Spherical Gaussians Specular**: In this experiment, we replace our Cook-Torrance specular term with the spherical Gaussians formulation from RGCA [34], leaving the rest of our method untouched. While we observe slightly better results in almost all metrics, we still decided in favor of the Cook-Torrance model since we observed slightly sharper specular highlights under environment relighting. \n\n**Precomputed Radiance Transfer (PRT)**: Here, we replace our neural diffuse BRDF with the PRT model from RGCA [34]. Therefore, we modify our dynamics model F_g to also predict the spherical harmonics coefficients for the PRT function of the primitives, which we then integrate with the incident lighting. While we observe reasonable results under training illumination, PRT struggles to generalize to novel illuminations, which aligns with our observations from the comparisons with RGCA in Table 2 and Figures 4 and 7.\n\nWe will extend Table 3 with the results from above and also add qualitative comparisons of both ablations in Figure 6 in the final version of the paper.\n\n>There are still artifacts, especially when relighting dynamic facial parts such as the teeth.\n\nReconstructing the mouth interior is particularly challenging, due to its highly non-rigid deforming geometry. Further, the mouth interior is only observed under a limited number of expressions and camera views due to occlusions. We tackled these challenges by adding explicit teeth geometry to the FLAME template mesh, which spawns additional Gaussian primitives for this. As stated in the main paper, we suspect that the described artifacts mainly arise from the limited expressiveness of the FLAME model. We believe that our method would deliver even better results when built with a more expressive parametric model. We will state this limitation more clearly in the final version of the paper.\n\n>Do you use any regularizers on the prediction of  for Cook–Torrance formula?\n\nIn the initial phase of the optimization, we regularize the predicted normal offsets to be small, thus the primitive’s normal remains close to the mesh normal. We gradually relax this constraint during the first 100k iterations of training. The roughness and visibility parameters are not regularized explicitly, but we use a sigmoid activation to ensure their values stay within the valid range.\n\n>What is the motivation behind using the classical Cook–Torrance microfacet model for specular?\n\nUsing the Cook-Torrance model, we can leverage the specular normal distribution function (NDF) from [33], which gives superior results in the environment map relighting setting compared to a simple Gaussian distribution. Please also refer to our additional ablation on the appearance model from above."}]

Author rebuttal:

Meta-review: This paper was reviewed by 4 experts in the field. After authors’ feedback and internal discussion, all reviewers agreed that this is solid work and should accept it (4, 4, 4, 5).

Specifically, reviewers agreed that the proposed solution of this work generates high-quality results. Particularly, the results are vivid and contain fine details. The proposed solution is interesting and novel.

Still, there are some remaining issues. For example, some ideas are from RGCA [34] and authors should further clarify that. Also, the results of [34] do not match the original results. Finally, the evaluation is limited and more results should be included. Although they should not hurt the novelty of this work, we think the authors must address them in the revision.

Considering all these, the decision is to recommend the paper for acceptance to NeurIPS 2025. We recommend the authors carefully read all reviewers’ final feedback and revise the manuscript as suggested in the final camera-ready version. We congratulate the authors on the acceptance of their paper!
Decision: Accept (poster)