Title: BecomingLit: Relightable Gaussian Avatars with Hybrid Neural Shading
Venue: NeurIPS 2025
Accepted: yes
Keywords: Head Avatars

Relighting

OLAT Dataset

Neural Rendering

Intrinsic Decomposition
Abstract: We introduce *BecomingLit*, a novel method for reconstructing relightable, high-resolution head avatars that can be rendered from novel viewpoints at interactive rates. Therefore, we propose a new low-cost light stage capture setup, tailored specifically towards capturing faces. Using this setup, we collect a novel dataset consisting of diverse multi-view sequences of numerous subjects under varying illumination conditions and facial expressions. By leveraging our new dataset, we introduce a new relightable avatar representation based on 3D Gaussian primitives that we animate with a parametric head model and an expression-dependent dynamics module. We propose a new hybrid neural shading approach, combining a neural diffuse BRDF with an analytical specular term. Our method reconstructs disentangled materials from our dynamic light stage recordings and enables all-frequency relighting of our avatars with both point lights and environment maps. In addition, our avatars can easily be animated and controlled from monocular videos. We validate our approach in extensive experiments on our dataset, where we consistently outperform existing state-of-the-art methods in relighting and reenactment by a significant margin.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a novel dataset and a method that utilizes it, achieving state-of-the-art results. The introduced dataset is captured using a lightweight capture system consisting of 16 cameras and 40 lights, which is significantly smaller compared to, for instance, the RGCA dataset. Despite its smaller size, the quality provided by the neural appearance representation is very high, producing sharp and detailed avatars.\n\nSTRENGTHS AND WEAKNESSES:\n**Weaknesses**\n\n1) The provided evaluation is not sufficient. While I can imagine that RGCA does not perform well under few-view settings, I still have doubts about the presented quality; it appears significantly worse than in the original paper. I would suggest running BecomingLit on the RGCA-released dataset as well, to enable a fair comparison. \n2) There are still artifacts, especially when relighting dynamic facial parts such as the teeth.\n\n**Strenghts**\n\n1) Very high-quality results are obtained on a dataset that is orders of magnitude smaller compared to other state-of-the-art methods. The dataset will be publicly available, which is a valuable contribution to the community and will support further research on relightable avatars.\n2) The presented appearance model is an interesting alternative to the fully linear RGCA approach. RGCA uses a fully linear model for both specular and diffuse lighting, utilizing spherical harmonics for diffuse and spherical Gaussians for specular components, which allows for explicit light transfer. In the context of this work, the diffuse color (BRDF) is represented by a small neural network conditioned on lighting and expressions. The specular component, on the other hand, uses the Cook-Torrance microfacet model, where $k_s$ is predicted by a neural network $F_v$, along with normal perturbations $\\delta n_k$. The roughness $r_k$ is learned per Gaussian as a variable. This is an elegant formulation based on a widely used specular model from computer graphics. However, the effectiveness of this formulation remains to be seen, as I believe the presented results are not sufficient to determine which aspects contribute most to the performance.\n\nQUESTIONS:\n*RGCA* and *BecomingLit* use different image formation models. From the initial evaluation, it appears that using the classical Cook-Torrance microfacet model for specular reflection gives superior results. However, I would like to see whether this holds true when both methods are trained on the same dataset. This is an interesting aspect that I believe could be better investigated. Specifically, the comparison between RGCA's linear radiance transfer and the small neural BRDF function used in BecomingLit to model light transport, without the need to store it per Gaussian primitive, deserves further analysis.\n\nI would suggest performing two additional evaluations: one on the full RGCA dataset using all 144 cameras, and another on a subsampled set of cameras. The drop in quality for RGCA likely also depends on the tracking. However, for the Goliath experiments, I would recommend using the original setup (with the provided tracking) and only modifying the number of views used. There may still be some inconsistencies, since I assume BecomingLit would need to use VHAP tracking on Goliath to remain compatible, but this would still provide more insight into the underlying model expressiveness.\n\n**Questions**\n1) Do you use any regularizers on the prediction of $F_v$ for Cook–Torrance formula?\n2) What is the motivation behind using the classical Cook–Torrance microfacet model for specular?\n\nLIMITATIONS:\nThe quality is still lower compared to RGCA trained on the Goliath dataset, especially in terms of teeth and hair quality. However, it remains unclear how much of this issue is due to the dataset or the image formation model used.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-21"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper  proposes a method for reconstructing photo-realistic and relightable avatars from a ‘lightweight’ capture setup equipped with OLAT. First FLAME fitting is obtained from fully lit images of the lightstage. Then an expression dependent network is trained to predict gaussian attributes and an additional features vector. This feature vector is used to learn an implicit diffuse radiance with an explicit analytical specular BRDF. \nCompared to Codec Avatars from Meta (RGCA) on the proposed dataset, the method outperforms RGCA especially in relighting.\n\nSTRENGTHS AND WEAKNESSES:\nThe paper reads very well and the problem addressed here is well sound and very active. The release of OLAT dataset is highly appreciated by the community. The idea of using a hybrid BRDF material with only diffuse  components learned implicitly seems effective and allows the network to complete the cook-torrance model by learning global illumination effects.\n\nQUESTIONS:\nMy major concerns are the following:\n\n1. It is on the scientific contribution and the positioning of the paper with respect to RGCA. It is clear that this work is built on top of RGCA, however how this differs (other than using the implicit diffuse component) needs to be stated clearly. And it seems that this implicit diffuse term is the only difference which makes the scientific contribution limited to me. Also, reading the related work does not help positioning the paper and this needs to be improved. Also the claims provided in the introduction needs to be more specific to highlight the paper contribution \n\n2. Using an hybrid reflectance field (with diffuse term learned implicitly) may violate energy conservation and the method does not seem to handle this which may lead to some saturation or artefacts. This need to be discussed as well\n\n3. The RGCA works show their method still performs well with a low number of cameras. However results shown here for RGCA (in relighting) look completely off.  Especially that fig 8 in supp shows fair decomposition from RGCA. This needs to be discussed as well.\n\n4. The predicted normals are tight to FLAME normals by design using a soft constraint. Is this enough to capture fine grained details such as wrinkles/folds in the normal map ?showing predicted normals map can be very insightful.  \n\nI like the results shown in the paper, also releasing a dataset with OLAT if of great value for the community.  However I am more concerned about the contributions itself. I would like to hear from authors and see other reviewers before making the final decision\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nAuthors build a pipeline for relightable avatars acquired with a low cost light stage setup. They collect subjects under multiple viewpoints, illuminations and different facial expressions. Their model is based on a head parametric model (FLAME), which is further optimized to capture the real subject geometry. Their shading model is hybrid, using a neural diffuse BRDF combined with an analytical specular term. They prove high quality results under different conditions. Their dataset is publicly available to accelerate the research in that direction.\n\nSTRENGTHS AND WEAKNESSES:\n[Strength] Relevant problem, well explained by authors and how they solve it\n[Strength] Dataset release to accelerate the research in relightable avatars\n[Strength] The method is simple yet effective and authors do a good job at explaining all the details.\n[Strength] Well presented video results\n\n[Weakness] My main criticism is around novelty and specifically comparing it with [34], while I understand the difference in terms of architecture (explicit pre-computed radiance transfer function vs hybrid), I did not find enough evidence by authors why this is needed.  Additionally, the comparisons authors show here do not seem to align with the results presented in [34] (see my next point)\n\n[Weakness] Video results show limitations of the method, in particular around eyes. HDRI results show “OLATs” like reflections. The comparisons with Relightable Gaussian Codec Avatars [34] do not seem to match the original results in the paper. Indeed RGCA demonstrates high quality HDRI results, whereas authors mention in their video that the previous method “struggles” in that scenario.\n\nQUESTIONS:\nMy main question for authors would be to better clarify the differences with [34], since their explanation is simply “we prefer a hybrid architecture for the rendering”, and also why their comparisons with [34] do not seem to match what authors showed in the original paper.\n\nFinally I’d like them to discuss more around the limitations I pointed out in the next section.\n\nLIMITATIONS:\nAuthors discuss some limitations but they did not mention the following:\n\nDensity of the lighting setup: while the setup is indeed less expensive than a full light stage, the light density seems to cause issues, in particular OLAT like reflections can be observed when the subjects are relit (see video) or when predicting albedo (Fig 5). Authors should discuss this limitation and how they plan to address it.\n\nAlbedo prediction is a simpler problem when starting with OLAT images, but in practice, this is not the actual problem to solve. Indeed a more challenging case that authors did not address is how to build an avatar from any arbitrary lighting condition, which would mimic real-world scenarios. Also note that the albedo results show issues specifically around the eyes, where OLAT reflections can be observed (Figure 5)", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a method to reconstruct a relightable head avatar from a light stage, which contains multiple calibrated cameras and controllable lights. In the method section, the model learns static material properties and decodes Gaussian attributes using a CNN decoder. One of the contributions proposed by the author is to separate the shading into neural diffuse part and  analytical specular part, which can better recover the materials and obtain improved rendering results. In the experiments, compared to the previous state-of-the-art method RGCA, the proposed method can render images with more realistic lighting and finer details. Additionally, the author's claim about the release of the dataset and code is fascinating, and I believe it can further advance the development of the field of relightable head avatars.\n\nSTRENGTHS AND WEAKNESSES:\nAdvantages:\n1. The results of the paper and video are realistic and vivid. The proposed method can not only recover reasonable face materials from the captured data, but it is also able to render the results under challenging lighting conditions, such as OLAT light and real-world environmental lighting. Even when the avatar is driven by novel expressions and head poses, the details of the face, such as small wrinkles and hair, as well as shadows, can be correctly rendered as the environmental lighting changes.\n2. The author claims to release the code and dataset, which are essential for this area. Several previous papers related to head avatar relighting captured their own datasets. However, they either do not open source their data, or the data they release is of poor quality. In this paper, the captured data includes many subjects with OLAT lights and high resolution, and the promise to release code that can reproduce the results is, in my opinion, a great contribution.\n3. Although the previous method, RGCA, can render realistic relighting results, the proposed method can relight the avatar with finer details and fewer artifacts. In the experiment section, the proposed method outperforms RGCA both qualitatively and quantitatively.\n\nWeaknesses:\n1. The data capturing setup is still somewhat complicated, and the training time for a single subject is a bit long. Although these issues are well discussed in the limitations section, they are still worth mentioning, as the complicated process can prevent other users from utilizing the method. Training a prior and fine-tuning the model may help reduce the requirements for the captured data and the training time.\n\nQUESTIONS:\n1. It seems that the quantitative results regarding the rendering speed are missing. It would be better to provide the speed data and a comparison with RGCA, since real-time rendering is claimed in the main paper. The proposed method includes many components, such as CNNs and MLPs, as well as shading computation. It would be valuable to provide the running time for each module.\n2. I want to confirm whether the shadow effect is learned from the tiny neural network Fd, as mentioned in line 168 regarding \"self-shadowing effects.\" If so, would it be possible to provide a visual result of the shadow map learned from Fd?\n\nLIMITATIONS:\nyes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-06"}