RelitLRM: Generative Relightable Radiance for Large Reconstruction Models

## Abstract

Abstract We propose RelitLRM , a Large Reconstruction Model (LRM) for generating high-quality Gaussian splatting representations of 3D objects under novel illuminations from sparse (4-8) posed images captured under unknown static lighting. Unlike prior inverse rendering methods requiring dense captures and slow optimization, often causing artifacts like incorrect highlights or shadow baking, RelitLRM adopts a feed-forward transformer-based model with a novel combination of a geometry reconstructor and a relightable appearance generator based on diffusion. The model is trained end-to-end on synthetic multi-view renderings of objects under varying known illuminations. This architecture design enables to effectively decompose geometry and appearance, resolve the ambiguity between material and lighting, and capture the multi-modal distribution of shadows and specularity in the relit appearance. We show our sparse-view feed-forward RelitLRM offers competitive relighting results to state-of-the-art dense-view optimization-based baselines while being significantly faster. Our project page is available at: https://relit-lrm.github.io/ .

## 1 Introduction

Reconstructing high-quality, relightable 3D objects from sparse images is a longstanding challenge in computer vision, with vital applications in gaming, digital content creation, and AR/VR. This problem is typically resolved by inverse rendering systems. However, most existing inverse rendering approaches suffer from limitations ranging from requiring dense captures under controlled lighting, slow per-scene optimization, using analytical BRDF hence not capable of modeling complex light transport, lacking data prior hence failure to resolve the ambiguity between shading and lighting in the case of static unknown lighting (Ramamoorthi & Hanrahan, ; Zhang et al., ; ; ; Jin et al., ; Kuang et al., ).

To overcome these limitations, we present *RelitLRM*, a generative Large Reconstruction Model (LRM) (Zhang et al., ) that efficiently reconstructs high-quality, relightable 3D objects from as few as 4–8 posed images captured under unknown lighting conditions. RelitLRM employs a feed-forward transformer-based architecture integrated with a denoising diffusion probabilistic model trained on a large-scale relighting dataset. This architecture models a multi-modal distribution of possible illumination decompositions, enabling deterministic reconstruction of object geometry and probabilistic relighting. In the case of glossy objects and challenging high-frequency novel lighting, the relit object appearance is inherently multi-modal with possibly many small specular regions; our relit appearance generator can well capture such high-frequency appearance, while regression-based approaches tend to be dominated by large smooth non-specular or weakly specular areas, and hence ignore the small strong specular highlights (see Fig. 5).

As shown in Fig. 1, our method can accurately reconstruct and relight the object’s appearance under diverse lighting conditions and viewpoints, demonstrating its capability for photorealistic rendering even with limited input views.
By processing tokenized input images and target lighting conditions through transformer blocks, RelitLRM decodes 3D Gaussian (Kerbl et al., ) primitive parameters within approximately one second on a single A100 GPU. This approach disentangles geometry, appearance, and illumination, resolving material-lighting ambiguities more effectively than prior models and enabling photorealistic rendering from novel viewpoints under novel lightings.

We conduct extensive experiments on synthetic and real-world datasets to evaluate RelitLRM. The results demonstrate that our method matches state-of-the-art inverse rendering approaches while using significantly fewer input images and requiring much less processing time (seconds v.s. hours). Additionally, RelitLRM does not rely on fixed analytic reflectance models, allowing greater flexibility in handling diverse material properties. Its scalability and efficiency make it well-suited for real-world applications where flexibility in lighting conditions and rapid processing are essential.

In summary, our key contributions to the field of relightable 3D reconstruction are:

- •
Novel transformer-based generative relighting architecture. We propose to use a regression-based geometry reconstructor followed by a diffusion-based appearance synthesizer (both modeled by transformers and trained end-to-end), to disentangle geometry from appearance, allowing better modeling of the uncertainty in relighting.
- •
State-of-the-art performance with practical efficiency. Trained on a large-scale 3D object dataset, RelitLRM matches or surpasses the performance of leading inverse rendering methods while using significantly fewer input images and less processing time, demonstrating superior generalization and practicality for real-world applications.

Figure: Figure 1: Demonstration of *RelitLRM*’s relighting capabilities. (a) Sparse posed images captured under unknown lighting conditions serve as the input for our method. (b) *RelitLRM* accurately reconstructs and relight the 3D object in the form of 3DGS under novel target lightings (either outdoor or indoor), with renderings closely matching the ground truth. (c) Object insertion into a virtual 3D scene. The red arrows indicate the objects relighted by *RelitLRM* using the scene illumination, demonstrating its ability to capture surrounding illumination and seamlessly harmonize with the scene background. (d) Objects relighting results showcase the robustness of our method under challenging lighting conditions, such as strong directional lighting. Our method faithfully models complex lighting effects, including removing shadow and highlights in input images and also casting strong shadows and glossy specular highlights under target lighting.
Refer to caption: x1.png

## 2 Related Work

### 2.1 Inverse Rendering

Inverse rendering aims to recover intrinsic scene properties — geometry, material reflectance, and lighting — from images, enabling tasks like relighting and novel view synthesis. Traditional methods optimize through physically-based rendering equations (Kajiya, ) using dense image captures (typically 100–200 views) under controlled lighting (Debevec et al., ; Ramamoorthi & Hanrahan, ; Marschner, ). Propelled by the recent advances in Neural Radiance Fields (NeRF) (Mildenhall et al., ) and 3D Gaussian Splatting (3DGS) (Kerbl et al., ), recent learning-based inverse rendering methods have emerged to decompose the scene properties with these neural representations  (Bi et al., ; ; Zhang et al., ; ; Boss et al., ; ; Luan et al., ; Srinivasan et al., ), by leveraging the differentiable rendering for their optimization and training  (Boss et al., ; Zhang et al., ; ; Kuang et al., ; ; Jin et al., ; Shi et al., ; Yang et al., ; Zhang et al., ; Gao et al., ; Liang et al., ; Jiang et al., ).
However, these methods typically require dense inputs and computationally intensive optimization, limiting scalability and struggling with complex lighting and materials.

Our method addresses these challenges by efficiently reconstructing relightable 3D objects from sparse images (4–8 views) without per-scene optimization. Utilizing a transformer-based architecture integrated with neural implicit radiance representations within the 3DGS framework (Kerbl et al., ), we effectively disentangle geometry, appearance, and illumination, enabling high-quality relighting and rendering under unknown lighting conditions, surpassing previous methods in scalability and practicality.

### 2.2 Large Reconstruction Model

Advancements in 3D reconstruction have been driven by neural implicit representations like NeRF (Mildenhall et al., ) and 3DGS (Kerbl et al., ), enabling high-quality novel view synthesis. Built upon it, the Large Reconstruction Model (LRM) series has further enhanced 3D reconstruction using transformer-based architectures for feed-forward processing. Notable models such as Single-view LRM (), Instant3D (Li et al., ), DMV3D (), PF-LRM (Wang et al., ), and GS-LRM (Zhang et al., ), along with others (Wei et al., ; Xie et al., ; Anciukevičius et al., ; Szymanowicz et al., ), have introduced scalable methods for multi-view data, enabling detailed reconstructions even from sparse inputs.

Despite these improvements, integrating relighting capabilities remains challenging. Our approach, RelitLRM, extends the LRM series by incorporating relightable reconstruction, combining transformer-based architecture with neural implicit radiance representations within the 3DGS framework. This enables robust 3D reconstruction with relighting capabilities from sparse inputs, overcoming scalability and practicality issues of previous methods.

### 2.3 Image-based Relighting and Diffusion Models

Diffusion models have become prominent in visual content generation, excelling in tasks like text-to-image synthesis and image editing (Ho et al., ; Nichol & Dhariwal, ). Models such as Stable Diffusion (Rombach et al., ) and ControlNet (Zhang et al., ) adapt diffusion frameworks for controlled image synthesis. While primarily 2D, recent works like Zero-1-to-3 (Liu et al., ) demonstrate diffusion models’ latent understanding of 3D structures, relevant for view synthesis and relighting.

Single-image relighting is challenging due to the limited view of input data. Early neural network-based approaches (Ren et al., ; Xu et al., ) laid the groundwork, with recent portrait-focused methods (Sun et al., ; Zhou et al., ; Pandey et al., ; Mei et al., ; Ren et al., ) enhancing realism and lighting control. However, these methods are often domain-specific or rely on explicit scene decomposition, limiting adaptability.
Recent methods like Neural Gaffer (Jin et al., ), DiLightNet (Zeng et al., ), and IC-Light (Zhang et al., ) leverage diffusion models for precise lighting control, achieving high-quality relighting. However, All theses works are constrained to single-view inputs.

Our approach integrates generative relighting and 3D reconstruction into an end-to-end transformer-based architecture.
It produce high-quality renderings of objects from novel viewpoints and under arbitrary lighting, given sparse input images.

## 3 Method

Our RelitLRM is built on top of the prior GS-LRM (Zhang et al., ) model, but differs in crucial aspects: 1) we aim for relightability of the output assets; 2) to achieve this, we propose a novel generative relighting radiance component to replace GS-LRM’s deterministic appearance part. In this section, we first briefly review the model architecture of GS-LRM, then describe in detail our relit 3D appearance generation algorithm based on relit-view diffusion.

Figure: Figure 2: Overview of *RelitLRM* for sparse-view relightable 3D reconstruction. Our pipeline consists of a geometry regressor and a relit appearance generator, both implemented as transformer blocks and trained jointly end-to-end. We implicitly bake the relit appearance generation in relit-view diffusion process. During inference, we first extract geometry tokens from sparse input images and regress geometry parameters for per-pixel 3D Gaussians (3DGS). Conditioning on novel target lighting and extracted geometry features, we denoise the relit views by first predicting a 3DGS appearance, then render it (along with 3DGS geometry that stays fixed in the diffusion denoising loop) into the denoising viewpoints. This iterative process produces the relit 3DGS radiance as a byproduct while denoising the relit views. The generative appearance and deterministic geometry blocks are trained end-to-end, ensuring scalability.
Refer to caption: x2.png

### 3.1 Review of GS-LRM model

GS-LRM (Zhang et al., ) uses a transformer model for predicting 3DGS parameters from sparse posed images. They demonstrated instant photo-realistic 3D reconstruction results. We provide an overview of their method below.

Given a set of $N$ images $I_{i}\in\mathcal{R}^{H\times W\times 3},i=1,2,...,N$ and corresponding camera Plücker rays $\mathbf{P}_{i}$, GS-LRM concatenates $I_{i}$,$\mathbf{P}_{i}$ channel-wise first, then converts the concatenated feature map into tokens using patch size $p$. The multi-view tokens are then processed by a sequence of $L_{1}$ transformer blocks for predicting 3DGS parameters $\mathbf{G}_{ij}$: one 3DGS at each input view pixel location.

$$ $\displaystyle\{\bm{T}_{ij}\}_{j=1,2,...,HW/p^{2}}=\text{Linear}\big{(}\text{ Patchify}_{p}\big{(}\text{Concat}(I_{i},\bm{P}_{i})\big{)}\big{)},$ (1) $\displaystyle\{\bm{T}_{ij}\}^{0}=\{\bm{T}_{ij}\},$ (2) $\displaystyle\{\bm{T}_{ij}\}^{l}=\text{TransformerBlock}^{l}(\{\bm{T}_{ij}\}^{ l-1}),\quad l=1,2,...,L_{1},$ (3) $\displaystyle\{\mathbf{G}_{ij}\}=\mathrm{Linear}(\{\bm{T}_{ij}\}^{L_{1}}).$ (4) $$

The whole model is trained end-to-end using novel-view rendering loss on multi-view posed images.

### 3.2 Generating relit 3DGS appearance via relit view diffusion

The relighting procedure involves the removal of source lighting from input images, and relighting the objects under target lighting, which are inherently ambiguous in the case of sparse view observations and unknown lighting conditions, especially in regions of shadows and specularity. Moreover, when relighting glossy objects using high-frequency novel illuminations, the relit object appearance is inherently multi-modal with possibly many small specular regions; regression-based approaches tend to be dominated by large diffuse or weakly specular areas, ignoring the small strong specular highlights. Hence, to boost relighting quality, we need a generative model to better handle the uncertainty in relighting and multi-modality in relit radiance.

We design a new approach to generate relit 3DGS appearance under a novel lighting. Related to prior work (; Szymanowicz et al., ; Anciukevičius et al., ),
we perform diffusion, in which we force a 3D representation as a bottleneck. However, the key algorithmic innovation is that our appearance predictor is a relighting diffusion model, conditioned on the novel target illumination and the predicted geometry features. The relit-view denoiser uses $x_{0}$ prediction objective (Ramesh et al., ); at each denoising step, it outputs the denoised relit views by predicting and then rendering the relit 3D Gaussian representation.

Formally, we first predict 3DGS geometry parameters $\{\mathbf{G}_{ij}^{\text{geom}}\}$ using the same approach (see Eq. 1,2,3,4) as in GS-LRM (Zhang et al., ); we also obtain the last transformer block’s output $\{\bm{T}_{ij}^{\text{geom}}\}$ of this geometry network to use as input to our relit-view denoiser. To produce the relit 3DGS apperance under a target lighting, we then concatenate the noisy relit view tokens, target lighting tokens, denoising timestep token, and predicted 3DGS geometry tokens, passing them through a second transformer stack with $L_{2}$ layers. The process is described as follows:

$$ $\displaystyle\{\bm{T}_{i}^{\text{all}}\}^{0}$ $\displaystyle=\left\{\{\bm{T}_{ij}^{\text{noised-relit-views}}\}^{L_{1}}\oplus \{\bm{T}_{ij}^{\text{light}}\}\oplus\{\bm{T}_{ij}^{\text{geom}}\}\oplus\{\bm{T }^{\text{timestep}}\}\right\},$ (5) $\displaystyle\{\bm{T}_{i}^{\text{all}}\}^{l}$ $\displaystyle=\text{TransformerBlock}^{L_{1}+l}(\{\bm{T}_{i}^{\text{all}}\}^{l -1}),\quad l=1,2,\ldots,L_{2}.$ (6) $$

The output tokens $\{\bm{T}_{ij}^{\text{geom}}\}^{L_{2}}$ are used to decode the appearance parameters of the 3D Gaussians, while the other tokens, e.g., $\{\bm{T}_{ij}^{\text{noised-relit-views}}\}^{L_{2}}$ are discarded on the output end. In detail, we predict the 3DGS spherical harmonics (SH) coefficients via a linear layer:

$$ $\{\bm{G}_{ij}^{\text{color}}\}=\text{Linear}\left(\{\bm{T}_{ij}^{\text{geom}} \}^{L_{2}}\right),$ (7) $$

where $\bm{G}_{ij}^{\text{color}}\in\mathbb{R}^{75p^{2}}$ represents the 4-th order SH coefficients for the predicted per-pixel 3DGS in a $p\text{x}p$ patch.

At each denoising step, we output the denoised relit views (using $x_{0}$ prediction objective) by rendering the predicted 3DGS representation:

$$ $\{\bm{I}_{i}^{\text{denoised-relit-views}}\}=\text{Render}\left(\{\bm{G}_{ij}^ {\text{geom}}\},\{\bm{G}_{ij}^{\text{color}}\}\right).$ (8) $$

The interleaving of 3DGS appearance prediction and rendering in the relit-view denoising process allow us to generate a relit 3DGS appearance as a side product during inference time. We output the 3DGS at final denoising step as our model’s final output: a 3DGS relit by target lighting.

#### Tokenizing HDR environment maps.

To make it easier for networks to ingest HDR environment maps $E\in\mathcal{R}^{H_{e}\times W_{e}\times 3}$, we convert each $E$ into two feature maps through two different tone-mappers: one $E_{1}$ emphasizing dark regions and one $E_{2}$ for bright regions.
We then concatenate $E_{1},E_{2}$ with the ray direction $D$, creating a 9-channel feature for each environment map pixel. Then we patchify it into light tokens, $\{\bm{T}_{ij}^{\text{light}}\}_{j=1,2,\ldots,H_{e}W_{e}/p_{e}^{2}}$, with a MLP. See Appendix 9 for details.

### 3.3 Training Supervision

In each training iteration, we sample a sparse set of input images for an object under a random lighting and another set of images of the same object under different lighting (along with its ground-truth environment map). The input posed images are first passed through the geometry transformer to extract object geometry features. As described in Sec. 3.2, we then perform relighting diffusion with the $x_{0}$ prediction objective; our denoiser is a transformer network translating predicted 3DGS geometry features into 3DGS appearance features, conditioning on the sampled novel lighting and noised multi-view images under this novel lighting (along with diffusion timestep).

We apply $\ell_{2}$ and perceptual loss  (Chen & Koltun, ) to the renderings at both the diffusion viewpoints and another two novel viewpoints in the same novel lighting. The deterministic geometry predictor and probabilistic relit view denoiser are trained jointly end-to-end from scratch.

### 3.4 Sampling relit radiance

At inference time, we first reconstruct 3DGS geometry from the user-provided sparse images. Then we sample a relit radiance in the form of 3DGS spherical harmonics given a target illumination. Our relit radiance is implictly generated by sampling the relit-view diffusion model. We use the DDIM sampler (Song et al., ) and classifier-free guidance technique (Ho & Salimans, ). Since the denoised relit views are rendered from the 3DGS geometry and predicted relit radiance at each denosing step, the relit-view denoising process also results in a chain of generated relit radiances. We only keep the relit radiance at last denoising step as our final predicted 3DGS appearance.

## 4 Dataset and implementation

#### Training dataset.

Our training dataset is constructed from a combination of 800K objects sourced from Objaverse (Deitke et al., ) and 210K synthetic objects from Zeroverse (Xie et al., ). The inclusion of Zeroverse data, which features a higher proportion of concave objects, assists in training the model to handle complex shadowing effects. Additionally, We also modify the metallic properties and roughness of some Zeroverse objects to introduce more specular surfaces, adding variety and challenge to the dataset.

For lighting diversity, we gathered over 8,000 HDR environment maps from multiple sources, including Polyhaven(^1^11https://polyhaven.com/hdris), Laval Indoor (Gardner et al., ), Laval Outdoor (Hold-Geoffroy et al., ), internal datasets, and a selection of randomly generated Gaussian blobs. We augmented these maps with random horizontal rotations and flips, increasing their number 16-fold. Each object is rendered from 10 to 32 randomly selected viewpoints under 2 to 12 different lighting conditions.

#### Model details.

Our model comprises 24 layers of transformer blocks for the geometry reconstruction stage and an additional 8 layers for the appearance diffusion (denoising) stage, with a hidden dimension of 1024 for the transformers and 4096 for the MLPs, totaling approximately 0.4 billion trainable parameters. We employ Pre-Layer Normalization (Xiong et al., ) and GeLU activations (Hendrycks & Gimpel, ).

Input images and environment maps are tokenized using an $8\times 8$ patch size, while denoising views are tokenized with a $16\times 16$ patch size to optimize computational efficiency. Tokenization is involves a simple reshape operation followed by a linear layer, with separate weights for the input and target image tokenizers. The diffusion timestep embedding is processed via an MLP similar to Peebles & Xie (), is appended to the input token set to the diffusion transformer.

#### Training details.

The initial training phase employs four input views, four target denoising views (under target lighting, used for computing the diffusion loss), and two additional supervision views (under target lighting), all at a resolution of $256\times 256$, with the environment map set to $128\times 256$. The model is trained with a batch size of 512 for 80K iterations, introducing the perceptual loss after the first 5K iterations to enhance training stability. Following this pretraining at the 256-resolution, we fine-tune the model for a larger context by increasing to six input views and six denosing target views at a higher resolution of $512\times 512$. This fine-tuning expands the context window to up to 31K tokens. For diffusion training, we discretize the noise into 1,000 timesteps, adhering to the method described in Ho et al. (), with a variance schedule that linearly increases from 0.00085 to 0.0120. To enable classifier-free guidance, environment map tokens are randomly masked to zero with a probability of 0.1 during training. For more details, please refer to Appendix.

#### Sampling details.

We performed an ablation study on the key hyperparameters, specifically the number of denoising steps and the classifier-free guidance (CFG) weight, as detailed in Table 6. We found our model can produce realistic and diverse results with five sampling steps, and CFG of 3.0 gives the best result for our res-256 model.

## 5 Experiments

We refer the readers to [our project page](https://relit-lrm.github.io/) for video results and interactive visualizations.

Figure: Figure 3: Comparison with optimization-based inverse rendering baselines on Stanford-ORB (Kuang et al., ), Objects-with-Lighting (Ummenhofer et al., ), and TensoIR-Synthetic (Jin et al., ) datasets. (a) On Stanford-ORB, our method captures realistic specular highlights and geometric details, outperforming NVDiffRec-MC(Hasselgren et al., ) and InvRender (Zhang et al., ). (b) On Objects-with-Lighting, our results closely match the ground truth, while baselines show over-specularity or artifacts. (c) For TensoIR-Synthetic, our method achieves comparable relighting with significantly fewer views. Notably, our method requires only 6 to 8 views and completes relighting in 2-3 seconds, while baselines need over 50 views and hours of processing.
Refer to caption: x3.png

### 5.1 Comparison with optimization-based methods

Current optimization-based inverse rendering algorithms achieve state-of-the-art 3D relighting results, but typically require dense view captures (over 40 views) and long optimization times, often taking several hours, using traditional rendering techniques like rasterization, path tracing, or neural rendering.

We evaluate our method against these approaches on three publicly available datasets: Stanford-ORB (Kuang et al., ), Objects-with-Lighting (Ummenhofer et al., ), and TensoIR-Synthetic (Jin et al., ). The Stanford-ORB dataset comprises 14 objects captured under three lighting conditions, with 60 training views and 10 test views per lighting setup. The Objects-with-Lighting dataset contains 7 objects with dense views captured under one training lighting condition and 3 views for two additional lighting conditions for testing. The TensoIR-Synthetic dataset consists of 4 objects with 100 training views under one lighting condition and 200 test views for each of five lighting conditions.

Our method requires only six input views for Stanford-ORB and Objects-with-Lighting and eight input views for TensoIR-Synthetic, completing the entire reconstruction and relighting process in just 2 to 3 seconds. In contrast, state-of-the-art optimization-based methods typically rely on all available training views (60 views per object for Stanford-ORB, 42-67 views per object for Objects-with-Lighting, and 100 views per object for TensoIR-Synthetic) and take several hours to optimize. Despite using significantly fewer input views and processing time, our method achieves comparable reconstruction and relighting results, as demonstrated in Table 1 and Table 2, showcasing its remarkable efficiency. Qualitative comparisons in Figure 3 further highlight our method’s superior performance across various scenarios. Optimization-based methods struggle with high uncertainties when decomposing object appearances under input lighting and relighting, leading to baked highlights (Column-6 in Figure 3) and poor specular reflections (Columns 1 and 3). In contrast, our generative approach handles these challenges effectively.

**Table 1: Real world experiments. We compare our method with state-of-the-art optimization-based approaches on the Stanford-ORB (Kuang et al., ) and Objects-With-Lighting (Ummenhofer et al., ) datasets. Our method offers competitive relighting results, but uses only six input views and runs within seconds on a single A100 GPU, whereas other methods require 60 views for Stanford-ORB and 42 to 67 views for Objects-With-Lighting, taking hours to complete.**
| Method | Stanford-ORB | Objects-with-Lighting |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| PSNR-H $\uparrow$ | PSNR-L $\uparrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$ | PSNR $\uparrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$ | # Input views | Runtime |  |
| Ours | 24.67 | 31.52 | 0.969 | 0.032 | 23.08 | 0.79 | 0.284 | 6 | $\sim$2 seconds |
| Neural-PBIR | 26.01 | 33.26 | 0.979 | 0.023 | N/A | N/A | N/A | 42-67 | hours |
| Mitsuba+Neus | N/A | N/A | N/A | N/A | 26.24 | 0.84 | 0.227 | 42-67 | hours |
| NVDIFFREC-MC | 24.43 | 31.60 | 0.972 | 0.036 | 20.24 | 0.73 | 0.393 | 42-67 | hours |
| InvRender | 23.76 | 30.83 | 0.970 | 0.046 | 23.45 | 0.77 | 0.374 | 42-67 | hours |
| TensoIR | N/A | N/A | N/A | N/A | 24.15 | 0.77 | 0.378 | 42-67 | hours |
| NeRFactor | 23.54 | 30.38 | 0.969 | 0.048 | 20.62 | 0.72 | 0.486 | 42-67 | hours |
| NVDIFFREC | 22.91 | 29.72 | 0.963 | 0.039 | 22.60 | 0.72 | 0.406 | 42-67 | hours |
| PhySG | 21.81 | 28.11 | 0.960 | 0.055 | 22.77 | 0.82 | 0.375 | 42-67 | hours |

**Table 2: Comparison with dense-view optimization-based methods on TensoIR-Synthetic. This dataset from (Jin et al., ) includes 4 scenes, each having 100 training views and 5 lighting conditions with 200 test views. We report per-scene and average metrics. Baselines use 100 input views; our method uses only eight. ^∗ indicates rescaling in pixel space instead of albedo.**
| Method | Armadillo | Ficus | Hotdog | Lego | Average |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| PSNR | SSIM | PSNR | SSIM | PSNR | SSIM | PSNR | SSIM | PSNR | SSIM | LPIPS | # Input views | Runtime |  |
| Ours^∗ | 33.25 | 0.966 | 26.18 | 0.952 | 29.32 | 0.938 | 27.44 | 0.886 | 29.05 | 0.936 | 0.082 | 8 | $\sim$3 second |
| FlashCache | 34.81 | 0.959 | 26.29 | 0.960 | 29.24 | 0.941 | 28.56 | 0.917 | 29.72 | 0.944 | 0.080 | 100 | hours |
| TensoIR^∗ | 34.75 | 0.974 | 25.66 | 0.950 | 28.78 | 0.932 | 29.37 | 0.932 | 29.64 | 0.947 | 0.068 | 100 | hours |
| TensoIR | 34.50 | 0.975 | 24.30 | 0.947 | 27.93 | 0.933 | 27.60 | 0.922 | 28.58 | 0.944 | 0.081 | 100 | hours |
| InvRender | 27.81 | 0.949 | 20.33 | 0.895 | 27.63 | 0.928 | 20.12 | 0.832 | 23.97 | 0.901 | 0.101 | 100 | hours |
| NeRFactor | 26.89 | 0.944 | 20.68 | 0.907 | 22.71 | 0.914 | 23.25 | 0.865 | 23.38 | 0.908 | 0.131 | 100 | hours |

### 5.2 Comparison with data-driven based relighting methods

We also compare our method with state-of-the-art image relighting approaches (Jin et al., ; Zeng et al., ), which fine-tune text-to-image diffusion models for single-image relighting. These methods, however, only support relighting from the input views and do not handle novel view relighting. Real-world benchmarks like Stanford-ORB and Objects-with-Lighting focus on novel view relighting, making these methods unsuitable for direct evaluation on those datasets.

To ensure a fair comparison, we construct a held-out set from filtered Objaverse (unseen during training), comprising 7 highly specular objects and 6 concave objects with shadows. Each object is rendered under five environment maps with four viewpoints, resulting in a total of 260 images for evaluation. We compute PSNR, SSIM, and LPIPS metrics, after applying per-channel RGB scaling to align predictions with the ground truth, following (Jin et al., ). Our method outperforms the baselines, showing better image quality metrics as in Table 4.

Figure: Figure 4: Comparison with image-based relighting baseline on our held-out evaluation set shows that our model produces better visual quality, with improved shadow removal and highlight. Our model processes four input images jointly, while the baseline relights each image independently.
Refer to caption: x4.png

### 5.3 Object scene insertion

RelitLRM’s capability to accurately reconstruct and relight 3D objects makes it particularly effective for object scene insertion tasks, where the goal is to seamlessly integrate objects into existing environments, as demonstrated in Fig. 1(c) and Fig. 1(d). In Fig. 1(c), we demonstrate the insertion of two objects into an indoor living-room scene, where they harmonize naturally with the surrounding lighting and shadows. In Fig. 1(d), we present that eight objects are relit under strong directional sunlight with our method, showcasing their ability to cast realistic shadows and exhibit accurate specular highlights, demonstrating RelitLRM’s effectiveness in handling diverse lighting conditions.

### 5.4 Ablation study

#### Deterministic vs. probabilistic relighting.

Decomposing object’s appearance from input lighting and relighting is inherently ambiguous. Moreover, the radiance function of object under high-frequency lighting is extremely complex and highly multi-modal, e.g. containing multiple small sharp highlights. To address this, our model reconstructs geometry deterministically while relighting the object’s radiance probabilistically using a diffusion approach. To evaluate this design, we introduce a deterministic baseline that relights objects directly, without the diffusion step, using the same architecture, amount of parameters, and training iterations. We assess both models on our hold-out validation set, with metrics presented in Table 4 and visual comparisons in Fig. 5. Although the quantitative results are similar, the probabilistic approach significantly improves visual quality for specular objects. This supports the idea that our probabilistic design captures the complex multi-modal radiance function more effectively than the deterministic counterpart, which tends to produce over-smooth results.

Figure: Figure 5: Our probabilistic design yields significantly better results on specular highlights compared to the deterministic counterpart. The radiance function of specular objects under challenging lighting is highly multi-modal with long tails. Our denoising diffusion approach models this distribution more effectively, while the deterministic design fails to mode such complex distribution and produce overly smooth specular highlights.
Refer to caption: x5.png

**Table 3: Comparison with image-based relighting models on held-out validation set. The prediction is scaled per-channel to align with the target following (Jin et al., ). The PSNR is only computed in masked foreground regions. Our method takes four input views, and baseline methods take each view independently.**
|  | PSNR $\uparrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$ | NFE | Parameters |
| --- | --- | --- | --- | --- | --- |
| Ours | 28.50 | 0.93 | 0.058 | 5 | 0.4B |
| NeuralGaffer | 24.51 | 0.90 | 0.072 | 50 | 0.9B |
| DiLightNet | 20.02 | 0.85 | 0.132 | 20 | 0.9B |

**Table 5: Impact of more input views: We evaluate the Res-512 model by incrementally adding more input views and compare its performance on our hold-out evaluation set and TensoIR-Synthetic. The Res-512 model is fine-tuned from the Res-256 model using six input views and six denoising views, and fixed through this evaluation.**
| Holdout Validation Set | TensoIR-Synthetic |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Input Views | PSNR $\uparrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$ | Input Views | PSNR $\uparrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$ |
| 2 | 23.58 | 0.896 | 0.113 | 4 | 27.83 | 0.926 | 0.091 |
| 4 | 26.14 | 0.915 | 0.086 | 8 | 29.04 | 0.936 | 0.082 |
| 6 | 27.71 | 0.926 | 0.072 | 12 | 29.27 | 0.938 | 0.081 |
| 8 | 28.08 | 0.929 | 0.070 | 16 | 29.31 | 0.938 | 0.080 |

#### Effects of more input views.

We evaluate our model with gradually increased number of input images up to 16 (70K input tokens) on our held-out set and TensoIR-Synthetic as shown in Table 5. We observe a positive correlation between our relighting quality and the number of input images.

**Table 6: Ablation on the parameter space of diffusion with four input views at $256\times 256$ resolution in our holdout evaluation set. Default setting is marked as gray.**
| views | PSNR $\uparrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$ |
| --- | --- | --- | --- |
| 1 | 27.05 | 0.913 | 0.075 |
| 2 | 26.87 | 0.913 | 0.0745 |
| 4 | 27.63 | 0.922 | 0.064 |

#### Hyper-parameters in diffusion.

In our model, the number of denoising views is independent of the number of input views. After all denoising steps, the virtual denoising views are discard, leaving only the 3D Gaussians for relighting at arbitrary views. A key question is how the denoising views affect the results. Additionally, two other parameters significantly impact relighting performance: the classifier free guidance weight and number of denoising steps. We present qualitative ablation results in Table 6. For classifier-free guidance, we visualize its effect in Fig. 6, where the unconditional predictions looks like relighting under average lighting in our dataset, and higher CFG weights making the object appear more specular.

Figure: Figure 6: Effect of classifier-free-guidance (CFG). We show two novel-view relighting results with different CFG weight. Zero weight denotes unconditional relighting, which resembles relighting with average dataset lighting. Higher CFG weight makes the object more specular.
Refer to caption: x6.png

### 5.5 Limitations

Despite our model reconstructs and relights objects from sparse-view images, it requires camera parameters for the input views, which can be impractical in some cases. Although the model improves output quality with more input images, performance saturates around 16 views (as shown in Table 5). Improving scalability with additional input views and higher resolutions remains an area for future exploration. Additionally, our model uses environment maps as the target lighting representation, which cannot accurately represent near-field lighting.

## 6 Conclusion

In this paper, we introduced RelitLRM, a generative Large Reconstruction Model that reconstructs high-quality, relightable 3D objects from sparse input images using a transformer-based architecture. Our model disentangles geometry and appearance modeling, and uses a deterministic geometry reconstructor followed by a probabilistic appearance (in the form of radiance) generator based on relit-view diffusion. Trained on massive relighting datasets,
RelitLRM effectively captures complex lighting effects like shadows, specularity, etc; it not only matches but also often exceeds state-of-the-art baselines while requiring significantly fewer input images and achieves this in just 2 to 3 seconds, compared to the hours needed by per-scene optimization methods. This highlights its remarkable efficiency and potential for real-world applications in relightable 3D reconstruction.

#### Ethics Statement.

This work proposes a large reconstruction model with a generative relightable radiance component, which can be used to reconstruct relightable 3D assets from sparse images. The proposed model tries best to preserve the identity of objects in the captured images in the reconstruction process; hence it might be used for reconstructing 3D from images with humans or commerical IPs.

#### Reproducibility Statement.

We have included enough details in this main paper (Sec. 4) and appendix (Sec. A.1) to ensure reproducibility, which includes our training and testing data, model architecture, training setup, etc. We are also happy to address any questions regarding the implementation details of our paper, if any.

#### Acknowledgements.

This work was started when Tianyuan Zhang, Zhengfei Kuang and Haian Jin were intern at Adobe Research. This work is in part supported by the NSF PHY-2019786 (The NSF AI Institute for Artificial Intelligence and Fundamental Interactions, [http://iaifi.org/](http://iaifi.org/)).

## Appendix A Appendix

### A.1 Pseudo Code

Figure: Algorithm 1 RelitLRM pseudo code.

### A.2 Model details

#### Tokenizing HDR environment maps.

To make it easier for networks to ingest HDR environment maps $E\in\mathcal{R}^{H_{e}\times W_{e}\times 3}$, we convert each $E$ into two feature maps through two different tone-mappers: one $E_{1}$ emphasizing dark regions and one $E_{2}$ for bright regions.

$$ $E_{1}=\frac{\log_{10}(E)}{\max(\log_{10}(E))},\quad E_{2}=\text{HLG}(E),$ (9) $$

where $\text{HLG}(\cdot)$ represents the hybrid log-gamma transfer function.
We then concatenate $E_{1},E_{2}$ with the ray direction $D$, creating a 9-channel feature for each environment map pixel. Then we patchify it and apply a two layer MLP with to turn it into light tokens, $\{\bm{T}_{ij}^{\text{light}}\}_{j=1,2,\ldots,H_{e}W_{e}/p_{e}^{2}}$.

### A.3 Image and Lighting preprocessing

To address scale ambiguity in relighting, we normalize both the rendered images and the environment map lighting in the dataset. For a rectangular target environment map, we compute its total energy, weighted by the sine of each pixel’s elevation angle. We then normalize the environment map to ensure the weighted total energy matches a predefined constant. The same scaling factor is applied to the HDR rendered images. The detailed process is outlined in pseudo code 2.

Figure: Algorithm 2 Normalize environment map and rendered images .

### A.4 Training hyperparameters

We begin by training the model from scratch with four input views, four denoising views, and two additional supervision views under target lighting, at a resolution of $256\times 256$. The model is trained for 80K iterations with a batch size of 512, using the AdamW optimizer with a peak learning rate of $4e-4$ and a weight decay of 0.05. The $\beta_{1},\beta_{2}$ are set to 0.9 and 0.95 respectively. We use 2000 iterations of warmup and start to introduce perceptual loss after 5000 iterations for training stability. We then finetune the model at a resolution of $512\times 512$ with six input views, six denoising views, and two supervision views, increasing the number of input tokens to 31K. This finetuning runs for 20K iterations with a batch size of 128, using the AdamW optimizer with a reduced peak learning rate of $4e-5$ and 1000 warmup steps. Throughout training, we apply gradient clipping at 1.0 and skip steps where the gradient norm exceeds 20.0.

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: * In the tables with the numbers for metrics please highlight the best numbers (bold)
* What hardware was used for training the model? Training time, memory requirements. Add to A.4
* In theory the method should work for objects that traditionally have challenging reflectance properties such as hair or fur. I am not sure if hair and fur were part of the training dataset, but it still might be interesting to see if it works.
soundness: 3
strengths: * Pseudocode in A.1clarifies all misunderstandings from the text descriptions. I can not stress enough how much I like this way of communicating the pipeline.
* The approach uses a diffusion process instead of a multi-view optimization, which is exceptionally fast in direct comparison.
* Trained on a rich dataset with a huge variety of lighting conditions.
* Light representation (input) is a simple environment map. Changing lighting is therefore very simple (no complicated neural light representation)
summary: The paper presents a method to generate a relighable 3D representation using 3DGS for the geometry and a diffusion based model to get the illumination dependent appearance parameters for those Gaussians.
The geometry is predicted in form of per pixel Gaussians from the sparse input views (4 - 16) in a single Transformer forward step. The tokens of the geometric representation is concatenated with HDR features extracted from the illumination given as environment map and the noise target views. After denoising the tokens for everything except the input gaussians are discarded. The remaining tokens are decoded into the appearance (SH) of the Gaussians. The Gaussians are then rasterized into any novel view. The diffusion model is trained such that the lighting of the rendered Gaussians should match the lighting of the input environment map. During inference this environment map and the target view camera pose can be arbitrarily be chosen, thus a scene can be reconstructed from sparse views and then relit.
weaknesses: * Knowing only even a fraction of the research and work that has gone into disentangling the ambiguity between lighting and shading it rubs me the wrong way to read something that suggests it solved it without actually addressing the core problem. The method does not really decompose a scene into geometry, lighting and shading and is not usable if the use case would require extracting or editing reflectance properties. The way I see it this paper does relighting by decomposing a scene into geometry and appearance, however this is very different to what methods which explicitly extract reflectance and illumination do. The problem statement is profoundly different if you have to produce a explicit, reflectance representation which represents some underlying physical property of the surface, compared to just estimating the product of light and reflectance. I don't think much has to be changed to show respect for this difference: In 2.1 it should be mentioned that the method models the appearance under arbitrary illumination without an explicit reflectance representation. In the introduction the claim that the ambiguity between shading and lighting, is overcome should be phrased more carefully or be clarified. As far as I understand this paper estimates appearance as the integrated product between the unknown shading and a given lighting in form of a view dependent SH. This is really great work, but should not be confused with reflectance decomposition.

---

confidence: 5
contribution: 2
initial_rating: 3
presentation: 3
questions: See "weakness".
soundness: 2
strengths: - originality-wise: I appreciate the proposed end-to-end transformer-based architecture for relightable 3D object reconstruction. 
- quality-wise: when taking into the efficiency of the proposed approach, I think quantitatively the performance is good.
- clarity-wise: the paper is well-written and easy to follow.
- significance-wise: the task of relightable 3D object reconstruction is vital for many downstream applications, e.g., AR/VR and robotics.
summary: This paper tackles the task of reconstructing relightable 3D objects from sparse images. For this, the authors extend the idea of GS-LRM to incorporate the generation of a relightable appearance. Specifically, they feed the geometry feature from GS-LRM through a diffusion process based on the transformer architecture. The transformer attends to the target illumination and outputs appearance tokens. The combination of the appearance tokens from the newly added transformer and the geometry tokens from the original GS-LRM transformer concludes the generation of Gaussian Splats. Experiments on the Stanford-ORB and Objects-with-Lighting real-world benchmark as well as the TensoIR synthetic benchmark demonstrate the effectiveness of the proposed approach.
weaknesses: ## Concerns about the architecture

I think the authors missed an important question to answer: why do we choose the current architecture? Essentially, the mechanism of discarding many tokens (L231) is wasting the model's capability. 

The root question is why do we need to use self-attention instead of cross-attention? Why do the environment maps and the denoised images need to be treated the same as the appearance tokens, especially since only the appearance tokens will be used for the Gaussian Splats rendering?

## Not enough understanding of the current model

Even though with the current architecture, I do not think the authors provide enough analysis. Specifically, have the authors visualized the attention maps of the transformer? What does the transformer learn? Does it attend to some specific areas in the environment map that cause the specular effects in the rendering? How does the transformer attend to those denoised images?

## Concerns about the qualitative results

In Fig. 3.(a), the produced Pepsi can's color is quite different from the ground truth. A similar thing happens to the gnome's colors. Further, the characters on the Pepsi can are quite blurry compared to NVDiffRec-MC / ground-truth. Additionally, in Fig. 3.(c), the RelitLRM produces quite different shows from the ground truth. However, the shadows are correctly predicted by both InvRender and TensoIR. 

Why is this the case? Have the authors carefully studied the causes? Whether increasing the number of views will help? Will 16 views mitigate these issues as the authors state that "performance saturates around 16 views" (L525)? If even 16 views cannot resolve the issue, what are the intrinsic shortcomings of the proposed model?

I hope the authors can provide a systematic analysis for a good understanding of the model.

## Missed important baselines

I think the authors missed several quite related as well as important baselines, e.g., [a, b]. They both use the idea of diffusion-based relighting model to tackle the task of relightable reconstruction.

Especially IllumiNeRF [b], which directly tackles the task of relightable object reconstruction and competes on the benchmark of Stanford-ORB and TensoIR. Frankly speaking, [b] outperforms the proposed approach on both benchmarks quantitatively:
- PSNR-H / PSNR-L / SSIM / LPIPS: 24.67 / 31.52 / 0.969 / 0.032 (RelitLRM) vs 25.42 / 32.62 / 0.976 / 0.027 ([b] on Stanford-ORB)
- PSNR / SSIM / LPIPS: 29.05 / 0.936 / 0.082 (RelitLRM)) vs 29.709 / 0.947 / 0.072 ([b] on TensoIR)

[a] A Diffusion Approach to Radiance Field Relighting using Multi-Illumination Synthesis. EGSR 2024.

[b] IllumiNeRF: 3D Relighting without Inverse Rendering. ArXiv 2024.

## Concerns about the novelty claim

On L088, the authors claim the first contribution as "a regression-based geometry reconstructor followed by a diffusion-based appearance synthesizer". Though I appreciate the end-to-end transformer architecture, I may not be convinced that the idea is entirely novel since the above-mentioned IllumiNeRF has already proposed an almost exact idea. I would recommend modifying it to correctly position the work.

## Missed training details

What is the hardware setup required to train the model? How long does it take to complete the training, hours or days?

## Incorrect description about the benchmark

In L356, the authors state that the Stanford-ORB benchmark has "60 training views and 10 test views per lighting setup". This is not true. Please correct it.

## Typos

L290: "is involves" -> "involves"

---

confidence: 3
contribution: 2
initial_rating: 6
presentation: 3
questions: - In the paper, the authors provide the rendering performance. However, I cannot find the training time. Please provide more specifics on the training setup, such as the number of GPUs used and total training hours. 
- A more thorough analyais and discussion of why performance plateaus between 8 to 16 views would enhance the paper's quality.
- Please provide quantitative evaluations of the extracted geometries.
- In the object insertion image (Figure 1(c)), how is illumination in the scene accounted for? Did you sample on the position of the object to capture the surrounding environment and incorporate the environment map into your model? Additionally, how do you account for the indirect light from another object produced by your RelitLRM in the scene?
- The method still takes 2 to 3 seconds to render. In contrast, the geometry and textures obtained from other methods can be rendered using many real-time rendering techniques. Moreover, in current industrial applications, it is challenging to abandon triangle meshes and textures. Therefore, this method cannot be considered a viable solution for 3D digital assets. However, if this approach could be extended to scene relighting, its applicability could be significantly broadened.
- Is the number of Gaussians predicted by the model sufficient for capturing different topologies and structures across diverse objects?
soundness: 3
strengths: - The method models geometry and rendering independently, akin to NeRF’s modeling architecture. It first constructs geometry from multi-view images, then generates the rendering by combining geometry features with environmental maps.
- The method is practical as it only requires 4-8 views as input. It is able to effectively reconstruct relightable objects without per-scene optimization.
- On both synthetic and real-world datasets, the method offers competitive relighting results, requiring much fewer input views than other methods.
summary: This paper proposes a method trained end-to-end on synthetic multi-view renderings of objects under varied, known illuminations. The approach is able to generate high-quality Gaussian splatting representations of 3D objects under novel illuminations from sparse input images (4-8 views) captured in unknown, static lighting conditions.
weaknesses: - In the paper, the authors claim that performance plateaus at around 16 views. However, as shown in Table 5, there is only a marginal improvement in image quality on the TensoIR-Synthetic dataset when increasing from 8 views to 16 views. 
- The novelty of the approach needs clearer articulation. The authors state that their method differs from Neural Gaffer (Jin et al., 2024) by not being limited to single-view inputs. However, this advantage seems to stem from the use of GS-LRM (Zhang et al., 2024a). It is important to clarify how their application of diffusion for relighting distinguishes their method from existing techniques.
- This method separates geometry from rendering, but the paper does not show results of the decomposed geometry. It is unclear how good or bad the quality of the reconstructed geometries is.

Meta-review: The paper introduces a method to generate 3D gaussian representations from sparse views under novel lighting conditions. The paper was well-received by all reviewers and converged to all-positive scores, recommending acceptance. The reviewers highlighted the strong qualitative results and the effectiveness of the feed-forward paradigm.
I agree with the reviewers and follow their suggestion.

Decision: Accept (Spotlight)