RelitLRM: Generative Relightable Radiance for Large Reconstruction Models
Authors: Tianyuan Zhang
Zhengfei Kuang
Haian Jin
Zexiang Xu
Sai Bi
Hao Tan
He Zhang
Yiwei Hu
Milos Hasan
William T. Freeman
Kai Zhang
Fujun Luan
Venue: ICLR 2025 Spotlight (2025)
Keywords: Relightable reconstruction
Inverse Rendering
Generative Relighting
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: We propose \emph{\method}, a Large Reconstruction Model (LRM) for generating high-quality Gaussian splatting representations of 3D objects under novel illuminations from sparse (4-8) posed images captured under unknown static lighting.

Abstract: We propose RelitLRM, a Large Reconstruction Model (LRM) for generating high-quality Gaussian splatting representations of 3D objects under novel illuminations from sparse (4-8) posed images captured under unknown static lighting. Unlike prior inverse rendering methods requiring dense captures and slow optimization, often causing artifacts like incorrect highlights or shadow baking, RelitLRM adopts a feed-forward transformer-based model with a novel combination of a geometry reconstructor and a relightable appearance generator based on diffusion. The model is trained end-to-end on synthetic multi-view renderings of objects under varying known illuminations. This architecture design enables to effectively decompose geometry and appearance, resolve the ambiguity between material and lighting, and capture the multi-modal distribution of shadows and specularity in the relit appearance. We show our sparse-view feed-forward RelitLRM offers competitive relighting results to state-of-the-art dense-view optimization-based baselines while being significantly faster. Our project page is available at: https://relit-lrm.github.io/.

Published as a conference paper at ICLR 2025
RELITLRM: GENERATIVE RELIGHTABLE RADIANCE FOR LARGE RECONSTRUCTION MODELS
Tianyuan Zhang' Zhengfei Kuang²
Haian Jin
Zexiang Xu+
Sai Bi
Hao Tan'
He Zhang' Yiwei Hu* Milos Hasan
William T. Freeman'
Kai Zhang** Fujun Luan
1 "Massachusetts Institute of Technology *Cornell University
2Stanford University
4 *Adobe Research
ABSTRACT
We propose RelitLRM, Large Reconstruction Model (LRM) for generating high- quality Gaussian splatting representations of 3D objects under novel illuminations from sparse (4-8) posed images captured under unknown static lighting. Un- like prior inverse rendering methods requiring dense captures and slow optimiza- tion, often causing artifacts like incorrect highlights or shadow baking, RelitL adopts feed-forward transformer-bas based model with a novel combination of ge- ometry reconstructor and relightable appearance generator based on diffusion. The model trained end-to-end on synthetic multi-view renderings of objects under varying known illuminations. This architecture design enables to effec- tively decompose geometry and appearance, resolve the ambiguity between ma- terial and lighting, and capture the multi-modaa distribution of shadows and spec- ularity the relit appearance. We show our sparse-view feed-forward RelitLRM offers competitive relighting results to state-of-the-art dense-view optimization- based baselines while being significantly faster. Our project page is available at: bttps://rell rm. gi github.ii. .io/.
INTRODUCTION
Reconstructing high-quality, relightable 3D objects from sparse images is a longstanding challenge in computer vision, with vital applications in gaming, digital content creation, and AR/VR. This problem is typically resolved by inverse rendering systems. However, most existing inverse render- ing approaches suffer from limitations ranging from requiring dense captures under controlled light- ing, slow per-scene optimization, using analytical BRDF hence not capable of modeling complex light transport, lacking data prior hence failure to resolve the ambiguity between shading and light- ing in the case of static unknown lighting (Ramamoorthi & Hanrahan, 2001; Zhang et al., 2021b;a; 2022a; Jin et al., 2023; Kuang et al., 2024).
To overcome these limitations, we present RelitLRM, a generative Large Reconstruction Model (LRM) (Zhang et al., 2024a) that efficiently reconstructs high-quality, relightable 3D objects from as few as 4-8 posed images captured under unknown lighting conditions. Unlike traditional inverse rendering techniques that explicitly decompose appearance and shading. RelitLRM introduces an end-to-end relighting model directly controlled by environment maps. RelitI employs a feed- forward transformer-based architecture integrated with denoising diffusion probabilistic model trained on large-scale relighting dataset. This architecture models a multi-modal distribution of possible illumination decompositions, enabling deterministic reconstruction of object geometry and probabilistic relighting. In the case of glossy objects and challenging high-frequency novel lighting, the relit object appearance is inherently multi-modal with possibly many small specular regions; our relit appearance generator can well capture such high- figh-frequency appearance, while regression-baas based approaches tend to be dominated by large smooth non-specular or weakly specular areas, and hence ignore the small strong specular highlights (see Fig. 5).
As shown in Fig. 1, our method can accurately reconstruct and relight the object's appearance under diverse lighting conditions and viewpoints, demonstrating its capability for photorealistic rendering
"Co-advise the project

---

Published as a conference paper at ICLR 2025
Lignn #E
Light #2
(c) Object Insertion
Ours
Ground Truth
Ours
Ground Truth
(a) Input Views
(b) Relit Appearance
(d) Objects Relighting
Figure 1: Demonstration of RelstLRM's relighting capabilities. (a) Sparse posed images captured
under unknown lighting conditions serve as the input for our method. (b) RelitI accurately reconstructs and relight the 3D object in the form of 3DGS under novel target lightings (either outdoor or indoor), with renderings closely matching the ground truth. (c) Object insertion into virtual 3D scene. The red arrows indicate the objects relighted by RelitLRM using the scene illumination, demonstrating its ability to capture surrounding illumination and seamlessly harmonize with the scene background. (d) Objects relighting results showcase the robustness of our method under challenging lighting conditions, such as strong directional lighting. Our method faithfully models complex lighting effects, including removing shadow and highlights in input images and also casting strong shadows and glossy specular highlights under target lighting.
even with limited input views. By processing tokenized input images and target lighting conditions through transformer blocks, RelitLRM decodes 3D Gaussian (Kerbl et al., 2023) primitive parame- ters within approximately one second on a single A 100 GPU. This approach disentangles geometry, appearance, and illumination, resolving material-lightinn ambiguities more effectively than prior models and enabling photorealistic rendering from novel viewpoints under novel lightings.
We conduct extensive experiments synthetic and real-world datasets to evaluate RelitLRM. The results demonstrate that our method matches state-of-the-ar inverse rendering approaches while us- ing significantly fewer input images and requiring much less processing time (seconds v.s. hours). Additionally, RelitLMM does not rely on fixed analytic reflectance models, allowing greater flexi- bility in handling diverse material properties. Its scalability and efficiency make well-suited for real-world applications where flexibility in lighting conditions and rapid processing are essential. In summary, our key contributions the field of relightable 3D reconstruction are:
Novel feed-forward generative 3D relighting architecture. We end-to-end learn gener- ative 3D relightable reconstruction model with deterministic geometry reconstruction and probabilistic radiance generation. Our approach bypasses the explicit appearance decom- position and shading, and directly generates relighted radiance, producing realistic view- dependents appearances and self-shadows.
State-of-the-ard performance with practical efficiency. Trained large-scale 3D ob- ject dataset, RelitLRM matches or surpasses the performance leading inverse rendering methods while using significantly fewer input images and less processing time, demon- strating superior generalization and practicality for real-world applications.
2
RELATED WORK
2.1 INVERSE RENDERING
Inverse rendering aims to recover intrinsic scene properties - geometry. material reflectance, and lighting from images, enabling tasks like relighting and novel view synthesis. Traditional meth- ods optimize through physically-bar rendering equations (Kajiya, 1986) using dense image cap- tures (typically 100-200 views) under controlled lighting (Debevec al., 2000; Ramamoorthi &
2

---

Published as a conference paper at ICLR 2025
Hanrahan, 2001; Marschner, 1998). Propelled by the recent advances in Neural Radiance Fields (NeRF) (Mildenhall et al., 2020) and 3D Gaussian Splatting (3DGS) (Kerbl et al., 2023), recent leaming-based inverse rendering methods have emerged to decompose the scene properties with these neural representations (Bi et al., 2020b;a; Zhang et al., 2021a;b; Boss et al., 2021 Luan al., 2021; Srinivasan al., 2021), 1,, by leveraging the differentiable rendering for their optimiza- tion and training (Boss et al., 2022; Zhang et al., 2022a;b; Kuang et al., 2022a;b; Jin et al., 2023; Shi et al., 2023; Yang et al., 2023; Zhang et al., 2023b; Gao al., 2023; Liang et al., 2024; Jiang et al., 2024). However, these methods typically require dense inputs and computationally intensive optimization, limiting scalability and struggling with complex lighting and materials.
Our method addresses these challenges by efficiently reconstructing relightable 3D objects from sparse images (4-8 views) without per-scene optimization. Utilizing a transformer-basee diffusion model integrated with the 3DGS (Kerbl et al., 2023), our method bypasses explicit appearance de- composition and shading, directly generates relighted radiance, enabling high-quality relighting and rendering under unknown lighting conditions with sparse views, and offering advantages in scala- bility and practicality
2.2 LARGE RECONSTRUCTION MODEL
Advancements in 3D reconstruction have been driven by neural implicit representations like NeRF (Mildenball et al., 2020) and 3DGS (Kerbl et al., 2023), enabling high-quality novel view synthesis. Built upon it, the Large Reconstruction Model (LRM) series has further enhanced 3D reconstruction using transformer-based architectures for feed-forward processing. Notable mod- els such as Single-view LRM (Hong et al.). Instant3D (Li et al., 2023). DMV3D (Xu al.), PF- LRM (Wang et al., 2023), and GS-LRM (Zhang et al., 2024a), along with others (Wei et al., 2024; Xie et al., 2024; Anciukevičius et al., 2023; Szymanowicz al., 2023), have introduced scalable methods for multi-view data, enabling detailed reconstructions even from sparse inputs.
Despite these improvements, integrating relighting capabilities remains challenging. Our ap- proach, RelitLRM, extends the LRM series by incorporating relightable reconstruction, combin- ing transformer-based architecture with neural implicit radiance representations within the 3DGS framework. This enables robust 3D reconstruction with relighting capabilities from sparse inputs, overcoming scalability and practicality issues of previous methods.
2.3 IMAGE-BASEE BASED RELIGHTING AND DIFFUSION MODELS
Diffusion models have become prominent in visual content generation, excelling in tasks like text- to- to-image synthesis and image editing (Ho et al., 2020; Nichol & Dhariwal, 2021). Models such as Stable Diffusion (Rombach al., 2022) and ControlNet (Zhang al., 2023a) adapt diffusion frameworks for controlled image synthesis. While primarily 2D, recent works like 7ero-1-to-3 (Liu at., 2023b) demonstrate diffusion models' latent understanding of 3D structures, relevant for view synthesis and relighting.
Single-image relighting is challenging due to the limited view of input data. Early neural network- based approaches (Ren et al., 2015; Xu et al., 2018) laid the groundwork, with recent portrait- focused methods (Sun et al., 2019; Zhou et al., 2019; Pandey et al., 2021; Mei et al., 2024; Ren et al., 2024) enhancing realism and lighting control. However, these methods are often domain- specific or rely on explicit scene decomposition, limiting adaptability. Recent methods like Neural Gaffer (Jin et al., 2024), DiLhtttNet (Zeng et al., 2024), and IC-Light (Zhang et al., 2024b) leverage diffusion models for precise lighting control, achieving high-quality relighting. However, All theses works are constrained to single-view inputs.
Our approach integrates generative relighting and 3D reconstruction into an end-to-end transformer- based architecture. It produce high-quality renderings of objects from novel viewpoints and under arbitrary lighting, given sparse input images.
3 METHOD
Our RelitLRM is built on top of the prior GS-LRM (Zhang et al., 2024a) model, but differs in crucial aspects: 1) aim for relightability of the output assets; 2) achieve this, we propose
3

---

Published as a conference paper at ICLR 2025
Progressive Relit-view Denoising loop
Discarded
Noised Relit Views
Denoised Relit Views
Input Views
Render
Appearance Transfornet Blocks (xL,)
Geometry Transformer Blocks (x1.1)
Enviromment Map
Dilfusion Timestop
Appeanunce
Gaussian
Gaussian Geometry
Appeurance
Geometry Tokens
(4) Geometry Regression
(b) Appearance Generation via Rellt-view
Overview of RelitLRM for sparse-view relightable 3D reconstruction. Our pipeline consists of geometry regressor and a relit appearance generator, both implemented as transformer blocks and trained jointly end-to-end. We implicitly bake the relit appearance generation in relit- view diffusion process. During inference, we first extract geometry tokens from sparse input images and regress geometry parameters for per-pixel 3D Gaussians (3DGS). Conditioning on novel target lighting and extracted geometry features, we denoise the relit views by first predicting 3DGS appearance, then render (along with 3DGS geometry that stays fixed in the diffusion denoising loop) into the denoising viewpoints. This iterative process produces the relit 3DGS radiance byproduct while denoising the relit views. The generative appearance and deterministic geometry blocks trained end-to-end, ensuring scalability.
Figure 2:
novel generative relighting radiance component to replace GS-LRM's deterministic appearance part. In this section, we first briefly review the model architecture GS-LRM, then describe in detail our relit 3D appearance generation algorithm based on relit-view diffusion.
3.1 REVIEW OF GS-LRM MODEL.
GS-LRM (Zhang et al., 2024a) uses a transformer model for predicting 3DGS parameters from sparse posed images. They demonstrated instant photo-realistic 3D reconstruction results. We pro- vide an overview of their method below.
Given a of N images = 1, 1,...., N and corresponding camera Plücker rays P.. GS-LRM concatenates I,,P, channel-wise first, then converts the concatenated feature map into tokens using patch size p. The multi-view tokens are then processed by a sequence of L1 transformer blocks for predicting 3DGS parameters Giy: one 3DGS at each input view pixel location.
(T. 1- 1,2_=,,,...,///// Linear (Patchiff,, (Patchify,,Connaa((,
(Concat(I,, P.))).
(Tr)) = (Ta),
(2)
(T.)] 1 TransformerBlock/ ((T,)) 1= 1,2,,..., LT,
(3)
[G.} Linear((TT,))*))
(4)
The whole model is trained end-to-end using novel-view rendering loss on multi-view posed images.
3.2 GENERATING RELIT 3DGS APPEARANCE VIA RELIT VIEW DIFFUSION
The relighting procedure involves the removal source lighting from input images, and relighting the objects under target lighting, which are inherently ambiguous in the case of sparse view observa- tions and unknown lighting conditions, especially in regions of shadows and specularity. Moreover, when relighting glossy objects using high-frequency novel illuminations, the relit object appearance inherently multi-modal with possibly many small specular regions; regression baasss approaches tend be dominated by large diffuse weakly specular areas, ignoring the small strong specu-

---

Published as a conference paper at ICLR 2025
lar highlights. Hence, to boost relighting quality, need a generative model to better handle the uncertainty in relighting and multi multi-modality in relit radiance.
We design a new approach to generate relit 3DGS appearance under novel lighting. Related to prior work (Xu et al.; Szymanowicz et al., 2023; Anciukevičius et al., 2023), we perform diffusion, in which we force 3D representation a bottleneck. However, the key algorithmic innovation is that our appearance predictor is a relighting diffusion model, conditioned on the novel target illumination and the predicted geometry features. The relit-view denoiser uses zn prediction objective (Ramesh et al., 2022); at each denoising step, it outputs the denoised relit views by predicting and then rendering the relit 3D Gaussian representation.
Formally. we first predict 3DGS geometry parameters {G using the same approach (see Eq. 1.2,3,4) as GS-LRM (Zhang al., 2024a); we also obtain the last transformer block's out- put (Toom) ) of this geometry network to use as input to our relit-view denoiser. To produce the relit 3DGS apperance under target lighting, we then concatenate the noisy relit view tokens, tar- get lighting tokens, denoising timestep token, and predicted 3DGS geometry tokens, passing them through a second transformer stack with L2 layers. The process is described as follows:
3T 3
{ (T f @ (I E (T ) (Trimescop)
(5)
(T TransformerBlockk- ((T. 1 1,2, 1,2..... L2.
(6)
The output tokens {T.) j are used to decode the appearance parameters of the 3D Gaussians, while the other tokens, e.g., i4
(eltdised-rlittvivwes j are discarded on the output end. In detail, we predict the 3DGS spherical harmonics (SH) coefficients via linear layer:
[G Linear ((Tg) L2),
where Gagler E 75p represents the 4-th order SH coefficients for the predicted per-pixel 3DGS in a pxp patch.
At each denoising step. we output the denoised relit views (using prediction objective) by ren- dering the predicted 3DGS representation:
[I Idlenoised relit views] Render ((G ), (Ctplor)
The interleaving of 3DGS appearance prediction and rendering in the relit-view denoising process allow to generate relit 3DGS appearance side product during inference time. We output the 3DGS at final denoising step as our model's final output: a 3DGS relit by target lighting.
Tokenizing HDR environment maps. To make it easier for networks ingest HDR environ- ment maps E E R M.xW, x3 we convert each E into two feature maps through two different tone- mappers: one E, emphasizing dark regions and one Eg for bright regions. We then concatenate E1, Eg with the ray direction D, creating a 9-channel feature for each environment map pixel. Then we patchify it into light tokens, (Tligh)) 1j==122...,,.W.. H.W./p3 with MLP. See Appendix 9 for details.
3.3 TRAINING SUPERVISION
In each training iteration, we sample sparse set of input images for an object under random lighting and another set of images of the same object under different lighting (along with its ground- truth environment map). The input posed images are first passed through the geometry transformer to extract object geometry features. As described in Sec. 3.2, we then perform relighting diffusion with the Zi prediction objective; our denoiser a transformer network translating predicted 3DGS geometry features into 3DGS appearance features, conditioning the sampled novel lighting and noised multi- view images under this novel lighting (along with diffusion timestep).
We apply lo and perceptual loss (Chen & Koltun, 2017) to the renderings both the diffusion viewpoints and another two novel viewpoints in the same novel lighting. The deterministic geometry predictor and probabilistic relit view denoiser are trained jointly end-to-end from seratch.
3.4 SAMPLING RELIT RADIANCE
At inference time, we first reconstruct 3DGS geometry from the user- urer-provided sparse images. Then sample relit radiance in the form of 3DGS spherical harmonics given target illumination. Our
5

---

Published as a conference paper at ICLR 2025
relit radiance is implictly generated by sampling the rill diffusion model. We use the DDIM sampler (Song et al., 2020) and classifier-fis free guidance technique (Ho & Salimans, 2022). Since the denoised relit views are rendered from the 3DGS geometry and predicted relit radiance each denosing step, the relit-view denoising process also results a chain generated relit radiances. We only keep the relit radiance last denoising step final predicted 3DGS appearance.
4 DATASET AND IMPLEMENTATION
Training dataset. Our training dataset is constructed from a combination of 800K objects sourced from Objaverse (Deitke et al., 2023) and 210K synthetic objects from Zeroverse (Xic et al., 2024). The inclusion of Zeroverse data, which features a higher proportion of concave objects, assists in training the model to handle complex shadowing effects. Additionally, We also modify the metallic properties and roughness of some Zeroverse objects to introduce more specular surfaces, adding variety and challenge to the dataset.
For lighting diversity, we gathered over 8,000 HDR enviromment maps from multiple sources, in- cluding Polyhaven', Laval Indoor (Gardner et al., 2017), Laval Outdoor (Hold-Geoffroy et al., 2019), internal datasets, and a selection of randomly generated Gaussian blobs. We augmented these maps with random horizontal rotations and flips, increasing their number 16-fold. Each object is rendered from 10 to 32 randomly selected viewpoints under 2 to 12 different lighting conditions.
Model details. Our model comprises 24 layers of transformer blocks for the geometry reconstruc- tion stage and an additional layers for the appearance diffusion (denoising) stage, with a hidden dimension of 1024 for the transformers and 4096 for the MLPs, totaling approximately 0.4 billion trainable parameters. We employ Pre-Layer Normalization (Xiong et al., 2020) and GeLU activa- tions (Hendrycks & Gimpel, 2016).
Input images and environment maps are tokenized using an 8 x 8 patch size, while denoising views tokenized with 16 x 16 patch size to optimize computational efficiency. Tokenization involves a simple reshape operation followed by a linear layer, with separate weights for the input and target image tokenizers. The diffusion timestep embedding processed via an MLP similar to Peebles & Xie (2023), is appended to the input token set to the diffusion transformer.
Training details. The initial training phase employs four input views, four target denoising views (under target lighting, used for computing the diffusion loss), and two additional supervision views (under target lighting), all resolution of 256 256, with the environment map set to 128 x 256. The model is trained with a batch size of 512 for 80K iterations, introducing the perceptual loss after the first 5K iterations to enhance training stability. Following this pretraining at the 256-resolution, we fine-tume tune the model for a larger context by increasing to six input views and six denosing target views at a higher resolution of 512 512. This fine-tuning expands the context window up to 31K tokens. For diffusion training, we discretize the noise into 1,000 timesteps, adhering to the method described in Ho al. (2020), with variance schedule that linearly increases from 0.00085 to 0.0 To enable classifier-free guidance, environment map tokens are randomly masked to zero with a probability of 0.1 during training. For more details, please refer to Appendix.
Sampling details. We performed an ablation study on the key hyperparameters, specifically the number of denoising steps and the dlssifier-ffre guidance (CFG) weight, detailed in Table 6. We found our model can produce realistic and diverse results with five sampling steps, and CFG of 3.0 gives the best result for our res-256 model.
EXPERIMENTS
We refer the readers to our project page for video results and interactive visualizations.
hhaps:/pooyhhavenncom/httt

---

Published as a conference paper at ICLR 2025
Envmap
Ground Truth
Ground Truth
Ground Truth
Ours
Ours
Ours
Misuha+NNus
InvRender
InvRender
NVDifRoo-MC
TensoriR
TensoriR
(a) Stanford-ORB (Real World Data)
Objects-with- Lighttng (Real World Data)
(c) Syathei-Sytthttic
(Synthetic Data)
Figure Comparison with optimization- inverse rendering baselines on STANFORD-
ORB (Kuang et al., 2024), OBJECTS- BEEECTT-WTT--LIHHTTT (Ummenhofer et al., 2024), and TENSOIR- (a) On STANFORD-ORI, our method captures realistic specular highlights and geometric details, outperforming NYDiffRce-MC(Hasee
SYNTHETIC (Jin et al., 2023) datasets.
et al., 2022) and InvRender (Zhang et al., 2022b). (b) On OBJECTS-WE SEAHCTSS--T----- WITH-LIGHTING, our results closely match the ground truth, while baselines show over-specularity artifacts. (c) For TENNOOR-SSYTTETIIC our method achieves comparable relighting with significantly fewer views. Notably, our method views and completes relighting 2-3 seconds, while baselines need 50 views and hours of processing. No color rescaling is performed for visualization.
requires only
COMPARISON WITH OPTIMIZATION BASED METHODS
Current optimization- based inverse rendering algorithms achieve state-of-the-arr 3D relighting re- sults, but typically require dense view captures (over 40 views) and long optimization times, often taking several hours, using traditional rendering techniques like rasterization, path tracing, or neural rendering.
We evaluate our method against these approaches on three publicly available datasets: STANFORD- (Ummenhofer et al., 2024), and TENSoIR-
ORB (Kuang et al., 2024), OBJECTS- OBJECTS-WITH- SYNTHETIC (Jin et al., 2023). The STANFORD- ORB dataset comprises 14 objects captured under three lighting conditions, with around 60 training views and 10 test views per lighting setup per object. The -WIECCT-WWTH-LIIGE
dataset contains 7 objects with dense views captured under one training lighting condition and views for two additional lighting conditions for testing. The TENSOIR-S -SSOIR--YYTIETII dataset consists of objects with 100 training views under one lighting condition and 200 test views for each of five lighting conditions.
Our method requires only six input views for STANFORD-ORB and OBJECTS-WITH-LIGHEE and eight input views for ing process in just 2 to 3 seconds. In contrast, state-of th-the-artt art optimization-bant based methods typically rely on all available training views (around views per object for STANFORD-ORE 42-67 views object for OBJECTS-WIY WIACTS--WWH-LIGHHINN
completing the entire reconstruction and relight-
and 100 views per object for TEESOIRR-SYTHEETE -SYNTHETIC and take several hours to optimize. Despite using significantly fewer input views and processing time, our method achieves comparable reconstruction and relighting results, demonstrated in Ta- ble I and Table 2, showcasing its remarkable efficiency. Qualitative comparisons in Figure 3 further highlight our method's superior performance across various scenarios. Optimization-baas methods struggle with high uncertainties when decomposing object appearances under input lighting and re- lighting, leading baked highlights (Column-6 Figure 3) and poor specular reflections (Columns

---

Published as a conference paper at ICLR 2025
1 and 3). In contrast, our generative approach handles these challenges effectively. Relighted images is rescaled when evaluating, following each benchmark's 's protocol for Table 1, 2.
Table I: Real world experiments. We compare method with state-of-the-art optimization approaches on the STANFORD-ORB (Kuang et al., 2024) and menhofer et al.. 2024) datasets. Our method offers competitive relighting results, but uses only six input views and runs within seconds on a single A100 GPU, whereas other methods use ~~ views for STANFORD-ORB and 42 to 67 views for Objects-Wiine With-Ligghinn, taking hours to complete.
(Um-
STANFORD-ORE
ORR
OBJECTS WITH- LIGHTIGG
Method
PSNR-I 11 1
PSNR-L
SSIM
LPIPS
PSNR
SSIM 1 LPIPS
Impul views
Rumtime
Ours
24.67
31.52
0.969
0.032
23.08
0.79
0.284
2-2 seconds
Neural-PRIR
26.01
33.26
0.979
0.023
NIA
N/A
N/A
42-67
hours
Mitsubu-Neus
N/A
NIA
N/A
N/A
26.24
0.227
42.67
hours
IllumiNeRF
25.56
32.74
0.976
0,027
NIA
N/A
NIA
42-67
bours
NVDIFTREC.O MC
31.60
0,036
20.24
0.73
hours
Lav Render
23.76
30.83
0.970
0,046
23.45
0.77
0.374
42-67
bours
TensoIR
N/A
NIA
N/A
N/A
0.77
0.378
42-67
hours
NeRFactor
23.54
30.38
0.969
0,048
20.62
0.72
0.486
42-67
hours
NVDIFFREC
22.91
29.72
0.963
0.039
22. 22.60
0.72
0.406
42-67
hours
PhySG
21.81
28.11
0.960
0.055
22.77
0.82
0.375
42-67
hours
Table Comparison with dense-view optimization-based methods on TynselR-S This dataset from (Jin et al., 2023) includes 4 scenes, each having 100 training views and 5 lighting conditions with 200 test views. We report per-scene and average metrics. Baselines use 100 input views; our method uses only eight. indicates rescaling in pixel space instead of albedo.
Mothod
Armadillo PSNR
SSIM
PSNR
Ficus
SSIM
PSNR
Hotdgg
SSIM
Lego PSNR
SSIM
PSNR
Average SSIM
LPIPS
# Imput views
Runtime
Ouns"
33.25
26.18
0.952
29.32
0.938
27.44
0.886
29.05
0.936
0.082
8
<33 second
FlashCache
34.81
0.959
26.29
0,960
29.21
0.941
28.56
0.917
29.72
0.914
0.080
100
hours
IllumiNeRF"
38.36
0.974
27.36
0.959
27.95
0.939
28.14
0.915
29.71
0.947
0.072
100
hours
TensolR.
34.75
0.974
25.66
0.950
28.78
0.932
29.37
0.932
29.64
0.947
0.068
100
hours
TensolR
34.50
24.30
0.947
27.93
0.933
0.922
28.58
0.081
100
hours
InvRender
27.81
0.949
20.33
0.895
27.63
0.928
20.12
0.832
23.97
0.901
0.101
100
hours
NeRFactor
26.89
0.944
20.68
0.907
22.71
0.914
23.25
0.865
23.38
0,908
0.131
100
heurs
5.2 COMPARISON WITH DATA-DRIVEN BASED RELIGHTING METHODS
We also compare our method with ofte-offtheearr image relighting approaches (Jin et al., 2024; Zeng et al., 2024), which fine-tune text-to-image diffusion models for single-image methods, however, only support relighting from the input views and do not handle novel view re- lighting. Real-world benchmarks like STANFORD- ORB and OBJECTS WITH-LIGHTING focus on novel view relighting, making these methods unsuitable for direct evaluation those datasets.
These
To ensure a fair comparison, we construct a held-out set from filtered Objaverse (unseen during training), comprising 7 highly specular objects and 6 concave objects with shadows. Each object is rendered under five environment maps with four viewpoints, resulting in a total of 260 images for evaluation. We compute PSNR, SSIM, and LPIPS metries, after applying per-channel RGB scaling to align predictions with the ground truth, following (Jin et al., 2024). Our method outperforms the baselines, showing better image quality metries in Table 3.
5.3 OBJECT SCENE INSERTION
RelitLRM's capability to accurately reconstruct and relight 3D objects makes particularly effec- tive for object scene insertion tasks, where the goal is to seamlessly integrate objects into existing environments, as demonstrated in Fig. I(c) and Fig. 1(d). In Fig. I(c), we demonstrate the in- sertion of two objects into indoor living-room scene, where they harmonize naturally with the surrounding lighting and shadows. In Fig. 1(d), we present that eight objects are relit under strong directional sunlight with method, showcasing their ability to cast realistic shadows and exhibit accurate specular highlights, demonstrating RelitLRM's effectiveness in handling diverse lighting conditions.
8

---

Published as a conference paper at ICLR 2025
Input
Ground Truth
Neural Gaffer
Ours
Taget Lighting
Figure 4: Comparison with image- based relighting baseline on our held-out evaluation set shows that our model produces better visual quality, with improved shadow removal and highlight. Our model processes four imput images jointly, while the baseline relights each image independently.
Input images
Ground Truth
Our--rrbbsbiissicc
Ours-Deterministicc
Targel lighting
Figure 5: Our probabilistic design yields significantly better results on specular highlights com- pared the deterministic counterpart. The radiance function of specular objects under challeng- ing lighting is highly multi-modal with long tails. Our denoising diffusion approach models this distribution more effectively, while the deterministic design fails to mode such complex distribution and produce overly smooth specular highlights.
5.4 ABLATION STUDY
Deterministic vs. probabilistic relighting. Decomposing object's appearance from input light- ing and relighting is inherently ambiguous. Moreover, the radiance function of object under high- frequency lighting extremely complex and highly multi-modal, c.g. containing multiple small sharp highlights. To address this, our model reconstructs geometry deterministically while relight- ing the object's ss radiance probabilistically using a diffusion approach. To evaluate this design, we introduce deterministic baseline that relights objects directly, without the diffusion step, using the same architecture, amount parameters, and training iterations. We assess both models on our hold-out validation set, with metrics presented in Table 4 and visual comparisons in Fig. 5. Although the quantitative results are similar, the probabilistic approach significantly improves visual quality for specular objects. This supports the idea that our probabilistic design captures the complex multi- modal radiance function more effectively than the deterministic counterpart, which tends to produce over-smooth results.
Effects of more input views. We evaluate our model with gradually increased number of input images up 16 (70K input tokens) our held-out and TensolR-SSnthen We observe a positive correlation between our relighting quality and the number of input images.
shown in Table 5.
in diffusion. In our model, the number of denoising views is independent of the number of input views. After all denoising steps, the virtual denoising views discard, leaving only the 3D Gaussians for relighting at arbitrary views. A key question how the denoising views affect the results. Additionally, two other parameters significantly impact relighting performance:

---

Published as a conference paper at ICLR 2025
Table 3: Comparison with image-based re- lighting models on held-out validation set. The prediction scaled per-channel to align with the target following (Jin et al., 2024), The PSNR only computed masked foreground regions. Our method takes four input views, and baseline methods take each view independently.
Table 4: Ablation on probabilistic design. We design a deterministic counterpart by remov- ing the noisy image tokens and keeping the model architecture and training configuration the same. We compare our probabilistic design with the deterministic counterpart on held-out evaluation set. See Figure 5 for visual results.
PSNR f SSIM f LPIPS NFE Purameters
Input Views PSNR
SSIM
LPIPS
Ours
28,50
0.93
0.058
0.4B
NeuralGaffer
24.51
0.90
0.072
50
0.9B
Ours- Probabilistic
27.63
0.922
0.064
DiLightNet 20.02
0.85
0.132
20
0.9B
Ours Deterministic
27.10
0.912
0.076
Table 5: Impact of more input views: We evaluate the Res-512 model by incrementally adding more input views and compare its performance on our hold-out evaluation set and TENSOIR- SYNTHETIC. The Res-512 model fine-tuned from the Res-256 model using six input views and six denoising views, and fixed through this evaluation.
HOLDOUT VALIDATION SET
TENSOIR SYNTHETIC
Input Views PSNR
SSIM
LPIPS 4
Input Views
PSNR
SSIM f
LPIPS
23.58
0.896
0.113
4
27.83
0.926
0.191
26.14
0.915
0.086
8
29.04
0.936
0.082
27.71
0.926
0.072
12
29.27
0.938
0.081
28.08
0.070
16
29.31
0.938
0.080
Table 6: Ablation on the parameter space of diffusion with four input views at 256 x 256 resolu- tion in our holdout evaluation Default setting is marked gray.
(a) Number of denoising views.
(b) Classifier-fres free guidance.
(c) Number of denoising steps.
views
PSNR
SSIM
LPIPS
CFG
PSNR
SSIM
LPIPS |
steps
PSNR
SSIM
LPIPS
1
27.05
0,913
0.075
1.0
27.00
0.915
0.072
2
23.81
0.884
0.096
26.87
0.0745
3,0
27.63
0.922
0.064
5
27.63
0.922
0.064
27.63
0.064
6.0
23.64
0.891
0.0974
27.00
0.915
0.073
the classifier free guidance weight and number of denoising steps. We present qualitative ablation results in Table 6. For classifier- free guidance, we visualize its effect in Fig. 6, where the uncondi- tional predictions looks like relighting under average lighting in our dataset, and higher CFG weights making the object appear more specular.
Target lightng
Target
GT'
CFG-D
CHje
CK333
CRG=6
lighting
GT
CPG-O
CHG
CHG
CKG
Figure 6: Effect of classffer-frre-guidaae (CFG). We show novel-view relighting results with different CFG weight. Zero weight denotes unconditional relighting, which resembles relighting with average dataset lighting. Higher CFG weight makes the object more specular.
5.5 LIMITATIONS
Despite our model reconstructs and relights objects from sparse-view images, it requires camera parameters for the input views, which can be impractical in some cases. Also our model does not support material editing applications, due to lack of explicit appearance decompositions. Although the model improves output quality with more input images, performance saturates around 16 views (as shown in Table 5). Improving scalability with additional input views and higher resolutions remains an area for future exploration. Additionally. our model uses environment maps as the target lighting representation, which cannot accurately represent near-field lighting.
10

---

Published as a conference paper at ICLR 2025
6 CONCLUSION
we presented RelitLRM, a generative Large Reconstruction Model for reconstructing high-quality, relightable 3D objects from sparse input images using diffusion transformer. Unlike traditional methods, our approach bypasses explicit appearance decomposition and shading, instead generating relighted radiance directly through deterministic geometry reconstructor followed by a proba- bilistic appearance generator based on relit-view diffusion. Trained on extensive relighting datasets, RelitLRM captures complex lighting effects such as shadows and specularity, often surpassing state- of-the-art baselines while requiring significantly fewer input images. Moreover, it achieves this in just 2-3 seconds, compared to the hours needed by per-scene optimization methods, demonstrating its remarkable efficiency and potential for real-world applications in relightable 31) reconstruction.
Ethics Statement. This work proposes a large reconstruction model with a generative relightable radiance component, which can be used to reconstruct relightable 3D assets from sparse images. The proposed model tries best to preserve the identity of objects in the captured images in the reconstruction process; hence it might be used for reconstructing 3D from images with humans or commerical IPs.
Reproducibility Statement. We have included enough details in this main paper (Sec. 4) and appendix (Sec. A.1) to ensure reproducibility, which includes our training and testing data, model architecture, training setup, etc. We are also happy to address any questions regarding the imple- mentation details of our paper, any.
This work was started when Tianyuan Zhang, Zhengfei Kuang and Haian Jin were intern at Adobe Research. This research was in part sponsored by the Department of the Air Force Artificial Intelligence Accelerator and was accomplished under Cooperative Agreement Number FA8750-199 100--19----000-- the NSF PHY-2019786 (The NSF AI Institute for Artificial Intelligence and Fundamental Interactions, http: /ttp::///aafffooof Directional Resolution in Computational Imaging).
Acknowledgements.
and NSF CIF 1955864 (Occlusion and
REFERENCES
Titas Anciukevičius, Zexiang Xu, Matthew Fisher, Paul Henderson, Hakan Bilen, Niloy J Mitra, and Paul Guerrero. Renderdiffusion: Image diffusion for 3d reconstruction, inpainting and generation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.
12608-12618, 2023.
Lucas Beyer, Andreas Steiner, André Susano Pinto, Alexander Kolesnikov, Xiao Wang, Daniel Salz, Maxim Neumann, Ibrahim Alabdulmohsin, Michael Tschannen, Emanuele Bugliarello, et al. Paligemma: A versatile 3b vlm for transfer. arXiv preprint arXiv:2407.07726. 2024.
Sai Bi, Zexiang Xu, Pratul Srinivasan, Ben Mildenhall, Kalyan Sunkavalli, Miloš Hašan, Yannick Hold Good-Geoffroy, David Kriegman, and Ravi Ramamoorthi. Neural reflectance fields for appearance acquisition. arXiv preprint arXiv:2008. 03824, 2020a.
Sai Bi. Zexiang Xu, Kalyan Sunkavalli, Miloš Hašan, Yannick Hold-Geoffroy, David Kriegman, and Ravi Ramamoorthi, Deep reflectance volumes: Relightable reconstructions from multi-view photometric images. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part III 16, pp. 294-311. Springer, 2020b.
Mark Boss, Raphael Braun, Varun Jampani, Jonathan Barron, Ce Liu, and Hendrik Lensch. Nerd: Neural reflectance decomposition from image collections. In Proceedings of the IEEE/CVF In- ternational Conference on Computer Vision, pp. 12684- 2021a.
Mark Boss, Varun Jampani, Raphael Braun, Ce Liu, Jonathan Barron, and Hendrik Lensch. Neural- pil: Neural pre-integrated lighting for reflectance decomposition. Advances in Neural Information Processing Systems, 34: 101100199000- 2021b.
Mark Boss, Andreas Engelhardt, Abhishek Kar, Yuanzhen Li, Deqing Sun, Jonathan Barron, Hen- drik Lensch, and Varun Jampani. Samurai: Shape and material from unconstrained real-world bitrary image collections. Advances in Neural Information Processing Systems, 95:26389-26400 2022.
11

---

Published as a conference paper at ICLR 2025
Qifeng Chen and Vladlen Koltun. Photographic image synthesis with cascaded refinement networks. In Proceedings of the IEEE international conference on computer vision, pp. 1511-1520, 2017.
Paul Debevec, Tim Hawkins, Chris Tchou, Haarm- Duiker, Westley Sarokin, and Mark Sagar. Acquiring the reflectance field of a human face. In Proceedings of the 27th annual conference on Computer graphics and interactive techniques, pp. 145-156, 2000.
Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: A universe of anno- tated 3d objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13142-13153, 2023.
Jian Gao, Chun Gu, Youtian Lin, Hao Zhu, Xun Cao, Li Zhang, and Yao Yao. Relightable 3d gaus- sian: Real-time point cloud relighting with brdf decomposition and ray tracing. arXiv preprint arXiv:2311.16045 2023.
Marc-André Gardner, Kalyan Sunkavalli, Ersin Yumer, Xiaohui Shen, Emiliano Gambaretto, Chris- tian Gagné, and Tean-Françoni Lalonde. Learning to predict indoor illumination from a single image. arXiv preprint vrXiv:11044 00090, 2017.
Jon Hasselgren, Nikolai Hofmann, and Jacob Munkberg. Shape, light, and material decomposi- tion from images using monte carlo rendering and denoising. Advances in Neural Information Processing Systems, 01222866-22661 2022.
Dan Hendrycks and Kevin Gimpel. arXiv: 08415, 2016.
Gaussian error linear units (gelus). arXiv preprint
Jonathan Ho and Tim Salimans. arXiv:2207. 12598, 2022.
Classifier-free diffusion guidance.
arXiv preprint
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-688 6851,, 2020.
Yannick Hold-Ges Gold-Geeffroy, Akshaya Athawale, and François Lalonde. Deep sky modeling for sin- gle image outdoor lighting estimation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 6927-6935, 2019.
Yicong Hong, Kai Zhang, Jiuxiang Gu, Sai Bi, Yang Zhou, Difan Liu, Feng Liu, Kalyan Sunkavalli, Trung Bui, and Hao Tan. Lrm: Large reconstruction model for single image 3d.
Yingwenqi Jiang, Jiadong Tu, Yuan Liu, Xifeng Gao, Xiaoxiao Long, Wenping Wang, and Yuexin Ma. Gaussianshader: gaussian splatting with shading functions for reflective surfaces. Pro- ceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 5322- 5332, 2024.
Haian Jin, Isabella Liu, Peijia Xu, Xiaoshuai Zhang, Songfang Han, Sai Bi, Xiaowei Zhou, Zex- iang Xu, and Hao Su. Tensoir: Tensorial inverse rendering. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 165-174, 2023.
Haian Jin, Yuan Li, Fujun Luan, Yuanbo Xiangli, Sai Bi, Kai Zhang, Zexiang Xu, Jin Sun, and Noah Snavely. Neural gaffer: Relighting any object via diffusion. arXiv preprint arXiv:2406.075 075200, 2024.
James T Kajiya. The rendering equation. In Proceedings of the 13th annual conference Computer graphics and interactive techniques, pp. 143-150, 1986.
Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splat- ting for real-time radiance field rendering. ACM Trans. Graph., 12
Zhengfei Kuang, Fujun Luan, Sai Bi, Zhixin Shu, Gordon Wetzstein, and Kalyan Sunkavalli. Palettenerf: Palette-bare based appearance editing of neural radiance fields. arXiv preprint arXiv:2212. 10699, 2022a.
12

---

Published as a conference paper at ICLR 2025
Zhengfei Kuang, Kyle Olszewski, Menglei Chai, Zeng Huang, Panos Achlioptas. and Sergey Tulyakov. Neroic: Neural rendering of objects from online image collections. ACM Transac- tions Graphics (TOG), 41(4):1-12, 2022b.
Zhengfei Kuang, Yunzhi Zhang, Long-XXing Yu, Samir Agarwala, Elliott Wu, Jiajun Wu, et al. Stanford-orb: real-world 3d object inverse rendering benchmark. Advances in Neural Informa- fion Processing Systems, 36, 2024.
Jiahao Li, Hao Tan, Kai Zhang, Zexiang Xu, Fujun Luan, Yinghao Xu, Yicong Hong. Kalyan Sunkavalli, Greg Shakhnarovich, and Sai Bi. Instant3d: Fast text-to-3d with sparse-view gen- eration and large reconstruction model. arXiv preprint arXiv:231 1.0223111.02111 14, 2023.
Zhihao Liang, Qi Zhang, Ying Feng, Ying Shan, and Kui Jia. Gs-ir: 3d gaussian splatting for inverse rendering. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pp. 21644-21653, 2024.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36:34892-34916, 2023a.
Haotian Liu, Chunyuan Li. Yuheng Li, and Yong Jae Lee, Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion, pp. 26296-26306, 2024.
Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick. Zero-I-to-3; Zero-shot one image 3d object. In Proceedings of the IEEE/CVE international conference on computer vision, pp. 999989939,,, 2023b.
Fujun Luan, Shuang Zhao, Kavita Bala, and Zhao Dong. Unified shape and svbrdf recovery using differentiable monte carlo rendering. In Computer Graphics Forum, volume 40, pp. 101-113. Wiley Online Library, 2021.
Stephen Robert Marschner. Inverse rendering for computer graphics. Cornell University, 1998.
Yiqun Mei, Yu Zeng, He Zhang, Zhixin Shu, Xuaner Zhang, Sai Bi, Jianming Zhang, HyunJoon Jung, and Vishal M. Patel. Holo-relighting: Controllable volumetric portrait relighting from single image. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4263-4273, June 2024.
Ben Mildenhall, Pratul P. Srinivasan, Matthew Taneik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes neural radiance fields for view synthesis. In ECCV, 2020.
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusson probabilistic models. In International conference machine learning, pp. 8162-8171. PMLR, 2021.
Rohit Pandey, Sergio Orts-Escolano, Chloe Legendre, Christian Haene, Sofien Bouaziz, Christoph Rhemann, Paul E Debevec, and Sean Ryan Fanello. Total relighting: learning to relight portraits for background replacement. ACM Trans. Graph., 40(4):43-1, 2021.
William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pp. 4195-4205, 2023.
Ravi Ramamoorthi and Pat Hanrahan. A signal-processing framework for inverse rendering. In Proceedings of the 28th annual conference on Computer graphics and interactive techniques, pp- 117-128, 2001.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint 2711v:2204.06125, 1(2):3, 2022.
Mengwei Ren, Wei Xiong, Jae Shin Yoon, Zhixin Shu, Jianming Zhang, HyunJoon Jung, Guido Gerig, and He Zhang. Relightful harmonization: Lighting- aware portrait background replace- ment. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 6452-6462, 2024.
13

---

Published as a conference paper at ICLR 2025
Peiran Ren, Yue Dong. Stephen Lin, Xin Tong, and Baining Guo. Image based relighting using neural networks. ACM Transactions on Graphics (ToG), 34(4):1-12, 2015.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image syuthesis with latent diffusion models. In Proceedings of the IEEE/CVE confer- ence on computer vision and pattern recognition, pp. 10684-10695, 2022.
Yahao Shi, Yanmin Wu, Chemming Wu, Xing Liu, Chen Zhao, Haocheng Feng, Jingtuo Liu, Liangjun Zhang. Jian Zhang. Bin Zhou, al. Gir: 3d gaussian inverse rendering for relightable scene factorization. arXiv preprint arXiv:2312.05133, 2023.
Jiaming Song. Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02500 2020.
Pratul P Srinivasan, Boyang Deng, Xiuming Zhang, Matthew Tancik, Ben Mildenhall, and Jonathan T Barron. Nerv: Neural reflectance and visibility fields for relighting and view synthe- sis. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 7495-7504, 2021.
Tiancheng Sun, Jonathan T Barron, Yun-Ta Tsai, Zexiang Xu, Xueming Yu, Graham Fyffe, Christoph Rhemann, Jay Busch, Paul Debevec, and Ravi Ramamoorthi. Single image portrait relighting. ACM Transactions Graphics (TOG), 38(4):1--2, 2019.
Stanislaw Szymanowicz, Christian Rupprecht, and Andrea Vedaldi. Viewset diffusion:(O-) image- conditioned 3d generative models from 2d data. In Proceedings the IEEE/CVF International Conference on Computer Vision, pp. 1863-8873, 2023.
Benjamin Ummenhofer, Sanskar Agrawal, Rene Sepulveda, Yixing Lao. Kai Zhang, Tianhang Cheng, Stephan Richter, Shenlong Wang, and German Ros. Objects with lighting: A real-world dataset for evaluating reconstruction and rendering for object relighting. In 2024 International Conference on 3D Vision (3DV), pp. 137-147. IEEE, 2024.
Peng Wang, Hao Tan, Sai Bi, Yinghao Xu, Fujun Luan, Kalyan Sunkavalli, Wenping Wang, Zexi- ang Xu, and Kai Zhang. Pf-Irm: Pose-free large reconstruction model for joint pose and shape prediction. arXiv preprint arXiv:2311.12022 2023.
Xinyue Wei, Kai Zhang, Sai Bi, Hao Tan, Fujun Luan, Valentin Deschaintre, Kalyan Sunkavalli, Hao Su, and Zexiang Xu. Meshlrm: Large reconstruction model for high-quality mesh. arXiv preprint arXiv:2404.1 12385, 2024.
Desai Xie, Sai Bi, Zhixin Shu, Kai Zhang, Zexiang Xu, Yi Zhou, Sören Pirk, Arie Kaufman, Xin Sun, and Hao Tan. Lem-zero: Training large reconstruction models with synthesized data. arXiv preprint arXiv:2406.09:3 2024.
Ruibin Xiong, Yunchang Yang, Di He, Kai Zheng, Shuxin Zheng, Chen Xing, Huishuai Zhang, Yanyan Lan, Liwei Wang, and Tieyan Liu. On layer normalization in the transformer architecture. In International Conference on Machine Learning, pp. 10524-10533. PMLR, 2020.
Yinghao Xu, Hao Tan, Fujun Luan, Sai Bi, Peng Wang, Jiahao Li, Zifan Shi, Kalyan Sunkavalli, Gordon Wetzstein, Zexiang Xu, et al. Dmv3d: Denoising multi-view diffusion using 3d large reconstruction model. In The Twelfth International Conference on Learning Representations.
Zexiang Xu, Kalyan Sunkavalli, Sunil Hadap, and Ravi Ramamoorthi. Deep image- based relighting from optimal sparse samples. ACM Transactions on Graphics (ToG), 37(4):1-13, 2018.
Ziyi Yang, Yanzhen Chen, Xinyu Gao, Yazhen Yuan, Yu Wu, Xiaowei Zhou, and Xiaogang Jin. Sire-ir: Inverse rendering for brdf reconstruction with shadow and illumination removal in high- illuminance scenes. arXiv preprint arXiv:2310 10. 13030, 2023.
Chong Zeng, Yue Dong, Pieter Peers, Youkang Kong, Hongzhi Wu, and Xin Tong. Dilightnet: Fine-grained lighting control for diffusion- based image generation. In ACM SIGGRAPH 2024 Conference Papers, pp. 1-12, 2024.
14

---

Published as a conference paper at ICLR 2025
Kai Zhang, Fujum Luan, Qianqian Wang, Kavita Bala, and Noah Snavely. Physg: Inverse rendering with spherical gaussians for physics-baye material editing and relighting. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp- 5453-5462, 2021a.
Kai Zhang, Fujun Luan, Zhengqi Li, and Noah Snavely. Iron: Inverse rendering by optimizing neural sdís and materials from photometric images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5565-5574, 2022a.
Kai Zhang. Sai Bi, Hao Tan, Yuanbo Xiangli, Nanxuan Zhao, Kalyan Sunkavalli, and Zexiang Xu. Gs-Irm: Large reconstruction model for 3d gaussian splatting. arXiv preprint 1xXix:2404119700 2024a.
Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings the IEEE/CVF International Conference Computer Vision, pp. 3836-3847, 2023a.
Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Ic-light github page, 2024b.
Xiuming Zhang, Pratul P Srinivasan, Boyang Deng, Paul Debevec, William T Freeman, and Jonathan T Barron. Nerfactor: Neural factorization of shape and reflectance under an unknown illumination. ACM Transactions Graphics (ToG), 40(6))1-18, 2021b.
Youjia Zhang, Teng Xu, Junqing Yu, Yuteng Ye, Yanging Jing. Junle Wang, Jingyi Yu, and Wei Yang. Nemf: Inverse volume rendering with neural microflake field. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 22919-229231 2023b.
Yuanqing Zhang, Jiaming Sun, Xingyi He, Huan Fu, Rongfei Jía, and Xiaowei Zhou. Modeling indi- rect illumination for inverse rendering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 18643-18652, 2022b.
Hao Zhou, Sunil Hadap, Kalyan Sunkavalli, and David W Jacobs. Deep single-image portrait re- lighting. In Proceedings of the IEEE/CVF international conference computer vision, pp. 7194- 7202, 2019.
15

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1 PSEUDO CODE
Algorithm RelitLRM pseudo code.
one training iter of RolitIRK Iaput list:
x_inp: [b, a, h, w, 31, n is number of views; h and w are the height and width; seap_rey_ds buth [b, h, v, 3]. per-pixel ray direction and ray origins for input images.
[b, nl, h, w, 3), nl is the number of denosing views. excra_train_inaee i [b, n2, h, w),
sarget_relit_immee
grourd truth relighted image for
superviaion.
envmap: [b, 3, h_e, w_el, . sarget lighting environment map
Ousput: training lass
Step-1: goometry taken extraction from input images
x ini censet ([x_nn, x_inp. plackeer,,, dim-1) + [b, a, h, w. 9]
out-d, kernel-8, stride-8) + patchfy: (b, h/8, w/8, d] x i_iap kLing.reshape ib, -1, d) [o, n_imp_tokens, d]
inp convi
x_inp_geo = transformerl (IN(x_inp))
step-2: tokenize envmap
c_darker log (envmap) / max (log(maa)))))))
e t_brighter HIG (eavaap) lybrid 1og-gamma transfer funcsion
l_gtt = cancat [e_ [e_darker, e_brighter,
ray dimm-1) + [b, bee, aLe,
x lightt conv2 [x_light, out-ú, kernel-8, stride-8)
x light x
(2, -1, di # [b, a_-ight_ token, d)
1_githt linear
out-d) [b, n_light_token, d]
step-3: add noise to targer_relit__aeee__________e___ randint (0, 1000, Ix ix_ino, [0],
[b,]
aipha_cumprod - ddpm_ ddppnschhcdulertt)
noise
x_t_ralight
relit_imago sqrt (alphaacumprod) + noise * sqrt
sipha_cumprod]
t_token TLmeStepEnbedderrts [b, 1,
step-4: tokenize noisy
images
concate with plucker ray of denoaing views to [b, n', h, H, 9]
a_trraligtt concate [x_t_relight target_relitt ray.aage.aaplcckeeee x x_t_relight conv3 (x_F_relight out-d, kernel-16, stride-16) X_t_relight x_t_relight. x_s_rellihhtrrehhape (b, -1, d) [b, n_relight_token, d]
step-5: feed every tokens so the denosing transformer
everything tokans
x_*11 censat ([__inp_____ inp_gee, x_light, x_t_relight, t_token), dim-l)
[b, n_inp_ Lone_tokens a a_light_tokens n_relight__ooees d]
x_all trrnnsormerr (IN(x (x_p_ [1], dim-1) + [b, inine_ d]
racianee
step-6: decode pixel-alqined 3D Saussian parameters geometry paramaters
gee "_ing_geo reshrreshppe lb, n, h//8, w//8, ou:=9, kernel=8, stride=8) umpatchify to pixai-a1 iineed GS ousput: [b, n, h, x,
geo (IN
gee xeee. reshape (b, -1, 9) # all geometry parama [b, n h 9] mnnerrcel-hsement
up to order of x inp_p_e rediance radiannne reshape (b, n, k//8, w//8, d)
apperance_parenn aataanno_ppramttrrr.
sh_weight deconv (ad___ianedara out-?5, kernel-8, stride-8) 75-dim sh
GS parameterisation
distance, scaling, rotation, opacity x_geo. ([1, 4, 1), dim--1) * = (distandca)
xyz x_inp_ray_o x_imp_ray_d (near (1 w) far w) 2.3), 0.3)
scaling zin (exp
rotacion rotation
koepdim-True)
epacity sigmoid/opacity 2.0)
step-7:
training loss
3dga (xyz, rgb, scaling, sh_weight, retation, opacity)
prod_x0, pred_oxtra_vis ccmm=e])
Render (3dgs,
[targotrrallt_imege itteerrralt_immm..ccmmr... extra_train_immg
12 loss 12 [pred x0, extrrextra... ipips_los loss_ [PIPS([p:ed_x0, exte seazget_relit_Lne extra_train image
exttr_vveee) view], [target_ relit image, axtra_train_image
lass l2ss 0.5 losss_loss
16

---

Published as a conference paper at ICLR 2025
A.2 MODEL DETAILS
Tokenizing HDR environment maps. To make it easier for networks to ingest HDR environ- ment maps E E xx,xW,x3, we convert each E into two feature maps through two different tone- mappers: one E₁ emphasizing dark regions and one Eg for bright regions.
logio(F) max(log1o (E))
E1 =
E2 = HLG
(9)
where HI.G(.) represents the hybrid log- log-gamma transfer function. We then concatenate E1, F3 with the ray direction D, creating 9-channel feature for each environment map pixel. Then we patchify and apply a two layer MLP with to turn it into light tokens, (Tylight) 1 H.W....
A.3 IMAGE AND LIGIITING PREPROCESSING
To address scale ambiguity in relighting, we normalize both the rendered images and the environ- ment map lighting in the dataset. For rectangular target environment map, compute its total energy. weighted by the sine of each pixel's elevation angle. We then normalize the environment map to ensure the weighted total energy matches predefined constant. The same scaling factor is applied to the HDR rendered images. The detailed process is outlined in pseudo code 2.
Algorithm Normalize environment map and rendered images.
normalize the rendered images and environment maps Input: Hist:
x: [n, hy, wy 31, n is number of views rendered for this scene under this lighting. h image widht and is image height.
envmap: [h_e, m_e, 3] environment map used to render this scane.
constant_ _energyy_eeegy pre-define total energy. precomputed as dataset median number output : s_normalized, envmap_norma
Step-1: compute weighted total energy of the environmens map
H, W envmag envmap.shape [:2]
theta = 1inspace (0.5, R 0.5, H) / H np.p* sin_theta stheta)
envmap + sin_theta[:: [+, None, Nona] weighted_cnerge sum lenvmap weightedd)
nnymap_weightts
step-2: 2: compute scaling factor
scale constant_energgy weighted_eeergy
step-3: normalize envmap
envmap = envmap scale
step-4: normalia ize and tonemap rendered images x scale t [E, h, v, 3]
x flatten x.reshhpe t-1,
la_total_pixel, 3]
x s_min, x_max parcentile (e_ (x_flatton, [1, 99))
(x -_mial (x_mex x_min)
(0, 1) tonemapping (1.3 2.21
return x, envmap
A.4 TRAINING HYPERPARAMETIE
We begin by training the model from scratch with four input views. four denoising views, and two additional supervision views under target lighting. at a resolution of 256 x 256. The model is trained for 80K iterations with a batch size of 512, using the AdamW optimizer with a peak learning rate of 4e - 4 and weight decay of 0.05. The $1, B2 are set to 0.9 and 0.95 respectively. We use 2000 iterations warmup and start to introduce perceptual loss after 5000 iterations for training stability. We then finetune the model at resolution of 512 512 with six input views, six denoising views, and two supervision views, increasing the number of input tokens to 31K. This finetuning runs for 20K iterations with batch size of 128, using the AdamW optimizer with reduced peak learning rate 4e 5 and 1000 warmup steps. Throughout training, we apply gradient clipping at 1.0 and skip steps where the gradient norm exceeds 20.0.
17

---

Published as a conference paper at ICLR 2025
A.5 TRAINING COMPUTE
Our transformer model has total of 400M parameters. Initially, train it resolution of 256 x 256 over 270B input tokens, which requires four days on 32 NVIDIA A100 GPUs (40GB VRAM each). Subsequently, we fine-tune the model higher resolution of 512 x 512 with six input views, processing an additional 79B input tokens over two days.
A.6 GEOMETRY EVALUATIONS
We evaluate the geometry quality of our method the STANFORD-ORBE ((Kung al., 2024) dataset using our Res-512 model with six input views. For evaluation, we filter out Gaussian points with opacity below 0.05 and randomly sample 30,000 points. The Chamfer distance is computed against the scanned ground-truth mesh using the official Stanford-ORB script, with results scaled by 2 10s, as per the benchmark protocol. Table summarizes the geometry and relighting results. Our method achieves the best geometry score while requiring only six input images and seconds of processing time, in contrast to other methods that rely on dense view inputs and require hours for final relighting.
Table 7:
Geometry evaluation on real world dataset. We evaluate the geometry quality of our method against state-of-the-art optimization-bastt relighting approaches on the STANFORD- ORB (Kuang et al., 2024) dataset. Our method achieves superior geometry quality with comparable relighting results, using only six input views and running within seconds single A100 GPU. In contrast, competing methods require approximately 60 views for STANFORD-ORB.
GEOMETRY Chamfer distance
NOVEL VIEW RELIGHTING
Method
PSNR-H H
PSNR-L
# Input views
Runtime
Ours
0,39
24.67
31.52
6
2 seconds
Neural-PBIl
0.43
26.01
33.26
~60
hours
IllumiNeRE
N/A
25.56
32.74
~60
hours
NVDIFFREC-MC
0.51
24.43
31.60
~60
hours
InvRender
0.44
23.76
30.83
~0
hours
NeRFactor
9,53
23.54
30.38
~60
hours
NVDIFFREC
0.62
22.91
29.72
~60
hours
PhySG
9.28
21.81
28.11
~60
hours
A.7 ATTENTION MAP VISUALIZATION
We visualize the attention map between appearance tokens and environment map tokens. Our re- lighting diffusion transformer has eight transformer layers, has typically five inference steps. We visualize the attention map of the last transformer layer of the last denoising timestep.
Each transformer layer employs multi-head self-attention with 16 heads, each of dimension 64, resulting in a total hidden dimension 1024. For visualization, we concatenate the key and value vectors from all heads and compute attention using the aggregated keys and values.
We use our Res-256 model at four-view input setup. for visualization. We show visualized results for two objects, each with two input lighting setups and two target lightings. For each input setup, we visualize the attention map for the image patch at the center of the first and second input image. For each target lighting, horizontally shift by 1/4,1/2, 3/4 /4 to create four target environment map. For each target lighting, we horizontally shift the environment map by 1/4, 1/2, and 3/4, creating four variations per lighting. In total, visualize 2 2 x 4 32 attention maps per object. The results are shown in Figures/ and 8.
For the visualized attention, we summarize some empirical patterns, though not super consistent.
Lighting rotation consistency. The attention map between appearance token and envi- ronment map tokens is quite consistent rotations of the environment map. Note that environment map is tokenized by concating the color and ray direction for each pixel. This consistency implies strong dependence directional cues in environment map.
18

---

Published as a conference paper at ICLR 2025
Horizonially shitting the environment mup
Horzootally shifting the environment map
Inpor Lightnng
Impun Lightng
Input Lightng
Lightng
Ingut Imagee
Attentoo Maps for Target Ligiting
Input Images
Attcution Maps for Target Lighting
Figure 7: Attention maps between appearance tokens and environment map tokens. The red dot in the input images indicates the corresponding appearance token. Visualized attention maps are shown for two target lighting, each horizontally shifted by 1/4, 1/2, and 3/4, creating four variations per lighting. Results from two sets of input images are displayed, captured under different lighting conditions but from the same of viewpoints. This figure features a specular object.
Input lighting stability. While less consistent than rotation, attention maps relatively stable across changes in input lighting. This might suggest, that the attention map learned something about the appearance models of the objects.
Contrary to intuition, attention maps for specular objects (Figure 7) are not noticeably sharper sparser compared those for diffuse objects,
in Figure 8.
Highlights attention maps do consistently align with directions where target lighting most affects object appearance. For instance, in Figure 7 (first row), appearance tokens corresponding the object's bottom aggregate lighting from the top of the environment map, contrary to expectation.
A.8 SCALING THE NUMBER OF INPUT VIEWS DURING INFERENCE
Our model demonstrates context-length generalization, where increasing the number of input views leads improved results (quantitatively shown in Table 5). To better illustrate this effect, incre- mentally visualize relighting results for the "Lego" object with 4, 8, and 12 input views in Figure 9. The highlighted region clearly shows improvement from 4 12 views due enhanced scene cov- erage.
We observed that increasing input views tends to results in brighter outputs. To investigate this phe- nomenon, we conducted an experiment (Figure 10) using an 8-view input but selectively visualizing Gaussians predicted from only the first few views, without modifying the transformer's forward pass.
To interpret these results, recall our model details: each input image predicts pixel-aligned 3D Gaus- sians, and directly concatenate predicted Gaussians from all input views as the final output. Thus, increasing the number of input views leads to more overlapping Gaussians.
In overlapping regions between views, some Gaussians appear masked out. However, remember that we predict a Gaussian at each pixel, and these white regions indeed visible to the first four viewpoints, these Gaussians in the white region is not explicitly masked out, but learned a low opacity scores. We also visualize the predicted Gaussians from the first 2, input views to better understand this effect(still the 8 input view setting.) in Figure-12.
Key observations from Figure 10:
19

---

Published as a conference paper at ICLR 2025
Horizonially shitting the environment mup
Horioottally shifting the environment map
Iopor Lightng
Impun Lightng
Inpur Lightng
Lighung
Ingut Images
Attentio Maps for Target Ligiting
Input Images
Attcution Maps for Target Lighting
Figure 8: Attention maps between appearance tokens and environment map tokens. The red dot in the input images indicates the corresponding appearance token. Visualized attention maps are shown for two target lighting, each horizontally shifted by 1/4, 1/2, and 3/4, creating four variations lighting. Results from two sets of input images are displayed, captured under different lighting conditions but from the same of viewpoints. This figure features a diffuse object.
Our hypothesis: We conjecture that with more and more input views, there will be more and more floaters with low opacity around the objects, especially in overlapping regions between input views. And these floaters makes the object appear brighter and brighter. We also want to clarify that these floaters might not be meaningless, they should have learned to also add some finner details to final results.
To validate this hypothesis, we visualized the results after pruning Gaussians with opacity scores below 0.25. For 8-view and 12-view inputs, this reduced the object's brightness while maintaining visually high-quality renderings, as in the updated version of the Figure (last two columns of the bottom half of the figure). However, pruning also removed some high-frequency details, suggesting that these low-opacity floaters are a mechanism learned by the model to enhance detail with more input views, with side-effect of changing object's 's brightness.
This observation also highlights a key challenge for scaling feod-forward 3D models like GS-LRM and RelitL_R .RM to handle large number of input views (e.g., 100). Improved methods for aggregating predictions across viewpoints, or entirely new paradigms, may be required to address this issue effectively.
20

---

Published as a conference paper at ICLR 2025
Input Images
Relighting Results
Ground Truth
Results input views
Results 8 input views
imput views
with imput views
with 12 ingut views
Figure 9: Results with more input views. The top part of the figure displays 12 input images, with the first four images in the first row corresponding to the 4-view setup. The bottom part shows results with progressively more input views (4, 8. and 12 views). In the region highlighted by the red rectangle, the 12-view result demonstrates significant improvement over the 4-view result due better coverage of the scene from input images. The last two column of the bottom part shows relighting results after pruning Gaussian with opacity below 0.25.
Ground Truth
3DGS from first image
3DGS from first two images
3DGS from first three images
3DGS from first four images
3DGS from all eight images
Figure 10: We feed eight input images to our model, instead of concating predicted pixel-aligned Gaussians from all viewpoints, we gradually visualize the predicted Gaussian from the first few viewpoints.
A.9 ADDITIONAL RESULTS FOR FIGURE 3
For quantitative evaluation in the relighting task, the standard practice is to channel-wise rescale relighted images before computing metrics. However, the qualitative results shown in the main paper did not apply this rescaling. To address this, provide additional rescaled results for the "Pepsi" object from the Stanford- ORB dataset in Figure 7, where channel wise scaling is applied to better match the ground-truth image.
21

---

Published as a conference paper at ICLR 2025
Original Results
Rescaled Results
Ground Truth
Ours
InvRender
NVDitRee-MC
Figure 11: Results after rescaling for "Pepsi" We present the original results from the first col- umn of Figure (first column) alongside rescaled versions for better visualization. Rescaling is applied whse using a factor computed from the median values of the ground truth and the corresponding relighting results.
Figure 12: Input images of our method for the "Hotdog". While baseline methods require 100 dense-view input images, our approach uses only these eight views. These input images shows shadows at the bottom part of the hotdog, which the baseline methods shown in Figure fail to remove effectively.
Additionally, Figure 8 shows input views used by our method for the "Hotdog" object (Figure 3(c)). Our vierse-view approach effectively removes shadows from input images, outperforming baseline methods like InvRender and TensoIR, which produce overly dark results at the object's 's bottom due poor shadow removal. While shadows softer and sometimes less accurate compared to vay-marching-base based methods (e.g., TensolR), our method achieves these results without explicit shadow-specific inductive biases.
A.10 ARCHITECTURE ANALYSIS
Our model processes input tokens from multiple modalities, including posed input images, target environment maps, noisy relit images, and denoising timestep tokens. We designed our architecture as a uni-stream model, where we use lightweight MLP layers to map each modality (environment map tokens, input image tokens, noise image tokens, and timestep tokens) into a shared feature space. Then, full self-attention is applied to extract and aggregate features across all tokens.
natural question is why we chose self-attention instead of cross-attention. To address this concern, we offer the following rationale:
Simplicity: Self-attention simplifies the architecture by eliminating the need for modality- specific encoders (e.g., separate encoders for images and environment maps) and a decoder. Traditional approaches introduce complexity, such as determining how to allocate parame- ters across modalities. In contrast, we use lightweight MLPs to project tokens into a shared space, followed by self-attention, allowing the model to learn how allocate capacity dy- namically across modalities and tasks.
Our design is inspired by large-scale vasion-language models like LLava Liu et al. (2023a; 2024) and PaliGemma Beyer et al. (2024), which treat image and text tokens equivalently after projection via lightweight MLPs, followed by self-attention layers. Similarly, our model treats all input modalities equivalently after projection, avoiding complex inductive biases.
22

---

Published as a conference paper at ICLR 2025
To compare the impact of self-attention versus cross-attention, replaced the self-attention lay- ers in the relighting transformer with cross- layers while keeping the number of trainable parameters constant. Both models were trained with input views at Res-256 resolution for 80k iterations using the same setup and data. The results this comparison on the Stanford-ORB, Objects-with-Lightingg and Held-out Evaluation Sets are shown in Table 8:
Table 8: Self-Attention vs. Cross-Attention Performance Comparison
Dataset
Method
PSNR-HI
PSNR-L f SSIM
Cross-Attention
21.06
26.70
0.943
Stanford-ORB
Self-Aticntion
22.97
29.42
0.967
Cross-Attention
N/A
13.46
0.491
Objects-with-iighieee
Self-Attention
N/A
20.62
0.756
Cross-Attention Self-Attention
N/A
26.34
0.906
Held-out Evaluation Set
N/A
27.63
0.922
23


Reviews:
[{"review_id": "tuFckqjNCU", "reviewer": "Reviewer_GDsv", "summary": "The paper presents a method to generate a relighable 3D representation using 3DGS for the geometry and a diffusion based model to get the illumination dependent appearance parameters for those Gaussians.\nThe geometry is predicted in form of per pixel Gaussians from the sparse input views (4 - 16) in a single Transformer forward step. The tokens of the geometric representation is concatenated with HDR features extracted from the illumination given as environment map and the noise target views. After denoising the tokens for everything except the input gaussians are discarded. The remaining tokens are decoded into the appearance (SH) of the Gaussians. The Gaussians are then rasterized into any novel view. The diffusion model is trained such that the lighting of the rendered Gaussians should match the lighting of the input environment map. During inference this environment map and the target view camera pose can be arbitrarily be chosen, thus a scene can be reconstructed from sparse views and then relit.", "questions": "* In the tables with the numbers for metrics please highlight the best numbers (bold)\n* What hardware was used for training the model? Training time, memory requirements. Add to A.4\n* In theory the method should work for objects that traditionally have challenging reflectance properties such as hair or fur. I am not sure if hair and fur were part of the training dataset, but it still might be interesting to see if it works.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* Pseudocode in A.1clarifies all misunderstandings from the text descriptions. I can not stress enough how much I like this way of communicating the pipeline.\n* The approach uses a diffusion process instead of a multi-view optimization, which is exceptionally fast in direct comparison.\n* Trained on a rich dataset with a huge variety of lighting conditions.\n* Light representation (input) is a simple environment map. Changing lighting is therefore very simple (no complicated neural light representation)", "weaknesses": "* Knowing only even a fraction of the research and work that has gone into disentangling the ambiguity between lighting and shading it rubs me the wrong way to read something that suggests it solved it without actually addressing the core problem. The method does not really decompose a scene into geometry, lighting and shading and is not usable if the use case would require extracting or editing reflectance properties. The way I see it this paper does relighting by decomposing a scene into geometry and appearance, however this is very different to what methods which explicitly extract reflectance and illumination do. The problem statement is profoundly different if you have to produce a explicit, reflectance representation which represents some underlying physical property of the surface, compared to just estimating the product of light and reflectance. I don't think much has to be changed to show respect for this difference: In 2.1 it should be mentioned that the method models the appearance under arbitrary illumination without an explicit reflectance representation. In the introduction the claim that the ambiguity between shading and lighting, is overcome should be phrased more carefully or be clarified. As far as I understand this paper estimates appearance as the integrated product between the unknown shading and a given lighting in form of a view dependent SH. This is really great work, but should not be confused with reflectance decomposition.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Rry9eboTYm", "reviewer": "Reviewer_kqbi", "summary": "This paper tackles the task of reconstructing relightable 3D objects from sparse images. For this, the authors extend the idea of GS-LRM to incorporate the generation of a relightable appearance. Specifically, they feed the geometry feature from GS-LRM through a diffusion process based on the transformer architecture. The transformer attends to the target illumination and outputs appearance tokens. The combination of the appearance tokens from the newly added transformer and the geometry tokens from the original GS-LRM transformer concludes the generation of Gaussian Splats. Experiments on the Stanford-ORB and Objects-with-Lighting real-world benchmark as well as the TensoIR synthetic benchmark demonstrate the effectiveness of the proposed approach.", "questions": "See \"weakness\".", "limitations": "", "rating": 8, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- originality-wise: I appreciate the proposed end-to-end transformer-based architecture for relightable 3D object reconstruction. \n- quality-wise: when taking into the efficiency of the proposed approach, I think quantitatively the performance is good.\n- clarity-wise: the paper is well-written and easy to follow.\n- significance-wise: the task of relightable 3D object reconstruction is vital for many downstream applications, e.g., AR/VR and robotics.", "weaknesses": "## Concerns about the architecture\n\nI think the authors missed an important question to answer: why do we choose the current architecture? Essentially, the mechanism of discarding many tokens (L231) is wasting the model's capability. \n\nThe root question is why do we need to use self-attention instead of cross-attention? Why do the environment maps and the denoised images need to be treated the same as the appearance tokens, especially since only the appearance tokens will be used for the Gaussian Splats rendering?\n\n## Not enough understanding of the current model\n\nEven though with the current architecture, I do not think the authors provide enough analysis. Specifically, have the authors visualized the attention maps of the transformer? What does the transformer learn? Does it attend to some specific areas in the environment map that cause the specular effects in the rendering? How does the transformer attend to those denoised images?\n\n## Concerns about the qualitative results\n\nIn Fig. 3.(a), the produced Pepsi can's color is quite different from the ground truth. A similar thing happens to the gnome's colors. Further, the characters on the Pepsi can are quite blurry compared to NVDiffRec-MC / ground-truth. Additionally, in Fig. 3.(c), the RelitLRM produces quite different shows from the ground truth. However, the shadows are correctly predicted by both InvRender and TensoIR. \n\nWhy is this the case? Have the authors carefully studied the causes? Whether increasing the number of views will help? Will 16 views mitigate these issues as the authors state that \"performance saturates around 16 views\" (L525)? If even 16 views cannot resolve the issue, what are the intrinsic shortcomings of the proposed model?\n\nI hope the authors can provide a systematic analysis for a good understanding of the model.\n\n## Missed important baselines\n\nI think the authors missed several quite related as well as important baselines, e.g., [a, b]. They both use the idea of diffusion-based relighting model to tackle the task of relightable reconstruction.\n\nEspecially IllumiNeRF [b], which directly tackles the task of relightable object reconstruction and competes on the benchmark of Stanford-ORB and TensoIR. Frankly speaking, [b] outperforms the proposed approach on both benchmarks quantitatively:\n- PSNR-H / PSNR-L / SSIM / LPIPS: 24.67 / 31.52 / 0.969 / 0.032 (RelitLRM) vs 25.42 / 32.62 / 0.976 / 0.027 ([b] on Stanford-ORB)\n- PSNR / SSIM / LPIPS: 29.05 / 0.936 / 0.082 (RelitLRM)) vs 29.709 / 0.947 / 0.072 ([b] on TensoIR)\n\n[a] A Diffusion Approach to Radiance Field Relighting using Multi-Illumination Synthesis. EGSR 2024.\n\n[b] IllumiNeRF: 3D Relighting without Inverse Rendering. ArXiv 2024.\n\n## Concerns about the novelty claim\n\nOn L088, the authors claim the first contribution as \"a regression-based geometry reconstructor followed by a diffusion-based appearance synthesizer\". Though I appreciate the end-to-end transformer architecture, I may not be convinced that the idea is entirely novel since the above-mentioned IllumiNeRF has already proposed an almost exact idea. I would recommend modifying it to correctly position the work.\n\n## Missed training details\n\nWhat is the hardware setup required to train the model? How long does it take to complete the training, hours or days?\n\n## Incorrect description about the benchmark\n\nIn L356, the authors state that the Stanford-ORB benchmark has \"60 training views and 10 test views per lighting setup\". This is not true. Please correct it.\n\n## Typos\n\nL290: \"is involves\" -> \"involves\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gpIPiWckD5", "reviewer": "Reviewer_Sueo", "summary": "This paper proposes a method trained end-to-end on synthetic multi-view renderings of objects under varied, known illuminations. The approach is able to generate high-quality Gaussian splatting representations of 3D objects under novel illuminations from sparse input images (4-8 views) captured in unknown, static lighting conditions.", "questions": "- In the paper, the authors provide the rendering performance. However, I cannot find the training time. Please provide more specifics on the training setup, such as the number of GPUs used and total training hours. \n- A more thorough analyais and discussion of why performance plateaus between 8 to 16 views would enhance the paper's quality.\n- Please provide quantitative evaluations of the extracted geometries.\n- In the object insertion image (Figure 1(c)), how is illumination in the scene accounted for? Did you sample on the position of the object to capture the surrounding environment and incorporate the environment map into your model? Additionally, how do you account for the indirect light from another object produced by your RelitLRM in the scene?\n- The method still takes 2 to 3 seconds to render. In contrast, the geometry and textures obtained from other methods can be rendered using many real-time rendering techniques. Moreover, in current industrial applications, it is challenging to abandon triangle meshes and textures. Therefore, this method cannot be considered a viable solution for 3D digital assets. However, if this approach could be extended to scene relighting, its applicability could be significantly broadened.\n- Is the number of Gaussians predicted by the model sufficient for capturing different topologies and structures across diverse objects?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The method models geometry and rendering independently, akin to NeRF’s modeling architecture. It first constructs geometry from multi-view images, then generates the rendering by combining geometry features with environmental maps.\n- The method is practical as it only requires 4-8 views as input. It is able to effectively reconstruct relightable objects without per-scene optimization.\n- On both synthetic and real-world datasets, the method offers competitive relighting results, requiring much fewer input views than other methods.", "weaknesses": "- In the paper, the authors claim that performance plateaus at around 16 views. However, as shown in Table 5, there is only a marginal improvement in image quality on the TensoIR-Synthetic dataset when increasing from 8 views to 16 views. \n- The novelty of the approach needs clearer articulation. The authors state that their method differs from Neural Gaffer (Jin et al., 2024) by not being limited to single-view inputs. However, this advantage seems to stem from the use of GS-LRM (Zhang et al., 2024a). It is important to clarify how their application of diffusion for relighting distinguishes their method from existing techniques.\n- This method separates geometry from rendering, but the paper does not show results of the decomposed geometry. It is unclear how good or bad the quality of the reconstructed geometries is.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)