UniRelight: Learning Joint Decomposition and Synthesis for Video Relighting
Authors: Kai He
Ruofan Liang
Jacob Munkberg
Jon Hasselgren
Nandita Vijaykumar
Alexander Keller
Sanja Fidler
Igor Gilitschenski
Zan Gojcic
Zian Wang
Venue: NeurIPS 2025 spotlight ()
Keywords: Relighting
Inverse Rendering
Diffusion Models
Primary area: applications

Abstract: We address the challenge of relighting a single image or video, a task that demands precise scene intrinsic understanding and high-quality light transport synthesis. Existing end-to-end relighting models are often limited by the scarcity of paired multi-illumination data, restricting their ability to generalize across diverse scenes. Conversely, two-stage pipelines that combine inverse and forward rendering can mitigate data requirements but are susceptible to error accumulation and often fail to produce realistic outputs under complex lighting conditions or with sophisticated materials. In this work, we introduce a general-purpose approach that jointly estimates albedo and synthesizes relit outputs in a single pass, harnessing the generative capabilities of video diffusion models. This joint formulation enhances implicit scene comprehension and facilitates the creation of realistic lighting effects and intricate material interactions, such as shadows, reflections, and transparency. Trained on synthetic multi-illumination data and extensive automatically labeled real-world videos, our model demonstrates strong generalization across diverse domains and surpasses previous methods in both visual fidelity and temporal consistency. Our
project page is https://research.nvidia.com/labs/toronto-ai/UniRelight/.

UniRelight: Learning Joint Decomposition and Synthesis for Video Relighting
Kai 1.2.3
Ruofan Liang ,2,3
Jacob Munkberg'
Jon Hasselgren'
Nandita Vijaykumar
Alexander Keller'
Sanja F.d
Igor Gilitschenskf..3t
Zan Gojeic
Zian Wang 1.2.37
NVIDIA
University of Toronto
Vector Institute
Abstract
We address the challenge of relighting a single image or video, task that demands precise scene intrinsic understanding and high-quality light transport synthesis. Existing end-to-end relighting models are often limited by the scarcity of paired multi-lllumnnttion data, restricting their ability to generalize across diverse scenes. Conversely, two-stage pipelines that combine inverse and forward rendering can mitigate data requirements but are susceptible to error accumulation and often fail to produce realistic outputs under complex lighting conditions or with sophisticated materials. In this work, introduce general purposse approach that jointly esti- mates albedo and synthesizes relit outputs in a single pass, harnessing the generative capabilities of video diffusion models. This joint formulation enhances implicit scene comprehension and facilitates the creation of realistic lighting effects and in- tricate material interactions, such shadows, reflections, and transparency. Trained on synthetic multi-llummintion data and extensive automatically labeled real-wort videos, our model demonstrates strong generalization across diverse domains and surpasses previous methods in both visual fidelity and temporal consistency. Our project page is https://research.nt www.ccc...iiiiiioo.o wttm://a//trroottoannniiiieeeeee
Introduction
Lighting is crucial in defining a scene's visual appearance, influencing both aesthetics and the perception of objects and materials. Modifying illumination while preserving scene content enables wide range of applications, including creative editing, simulation, and robust vision systems. Realistic relighting necessitates an accurate understanding of scene geometry and materials, as well as faithful light transport modeling. Recent methods aim to learn these complex relationships from data, but significant challenge is the scarcity of illliiiillmiitiionn datasets capturing the same scene under varied lighting conditions. Acquiring such data in the real world requires controlled environments, calibrated equipment, and repeated captures, making it costly and impractical. As result, most existing approaches rely on synthetic data or focus on narrow domains like portraiture, limiting their ability to generalize to real-world scenes.
To address the lack of multi-lllumination data, recent works [69, 39] have decomposed the relighting task into two stages: an inverse rendering step that estimates scene attributes such as albedo, normal, and depth (i.e., G-buffers), followed by forward rendering step that synthesizes relit images conditioned on these estimates. This pipeline avoids the need for multi-illuminationn supervision, allowing each stage be trained separately using ilgummiillinnnnoon data. However, inverse and forward rendering are inherently coupled, and modeling them separately with discrete intermediate representations introduces significant limitations. The forward renderer is highly sensitive to errors in
Joint Advising
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Input Albedo Pred.
Relighting
Relighting
E1
Figure I: Given an input image (top left) or video. our method jointly estimates albedo (bottom left) and synthesizes relit videos with novel lighting conditions using provided HDR probes. Notably, our estimated albedo maps effectively demodulate shadows and specular highlights, while the relit images exhibit plausible shadows and specular highlights.
the estimated G-buffers and struggles to capture complex material properties not encoded by them, such as transparency and subsurface scattering.
Learning an end-to-end relighting model in low data regime is challenging. While prior methods for intrinsic estimation (e.g., geometry [28, 19] and albedo [32, 39]) trained synthetic data often generalize well to real-world scenes, we observe that directly training relighting models as video- to-video translation leads to poor generalization to unseen domains. We argue that robust relighting requires an explicit understanding of illumination and scene properties.
Hence, we propose relighting framework that jointly models the distribution of scene intrinsics and illumination. Inspired by VideoJAM [12], we train a video generative model that jointly denoises latent space for relighting and albedo demodulation in single pass. Practically, latent representations and treat them as a single video clip. This design is motivated by the hypothesis that demodulation provides a strong prior for the relighting task, such as removing shadows. This joint formulation encourages the model to learn an internal representation of scene structure, leading to improved generalization across diverse and unseen domains. Our approach contrasts with two-stage pipelines that rely on explicit G-buffer estimation. Instead, we implicitly reason about intrinsic scene representation, enabling better representation learning. reducing error accumulation, and modeling of complex visual effects such as specular highlights, transparency, and subsurface scattering.
concatenate both
Our formulation also offers flexibility to learn from diverse sources of supervision. We leverage high-quality synthetic data for full supervision and complement with real-world single illumination videos that be auto-labeled at scale. This hybrid training strategy enables the model handle complex lighting effects while significantly improving generalization to unseen domains, see Figure 1. UNIRELIGHT enables high-quality relighting and intrinsic decomposition from single input image or video, producing temporally consistent shadows, reflections, and transparency, and outperforms state-of-the-ar methods.
2 Related Work
Inverse rendering estimates intrinsic scene properties such as geometry, materials, and lighting from input images. Traditional approaches use hand-crafted priors within an optimization framework [34, 3, 21, 11, 77, 2] to handle low-order effects like diffuse shading and reflectance. Recently, the field has been revitalized by applying supervised and self-supervised learning [4, 33, 35, 57, 66, 36, 37, 10, 61, 60, 63, 6] to inverse rendering tasks. These methods typically require large, domain-specific datasets, and struggle to generalize outside the training domain.

---

Relit
Fnvironment lighting
Noisy Intents
Inpat video
video model
DIT
Albedo
2
x
g g
Concatmate latents slong fione dits"
Figure 2: Method overview. Given an input video and larget lighting configuration (Exz, Eing. Ear). our method jointly predicts relit video Le and its corresponding albedo à. We pretrained VAE encoder- decoder pair (E, D) map input and output videos a latent space. The latents for target relit video and albedo are concatenated along the temporal (frame) dimension with the encoded input video. Lighting features he, derived from the environment maps, are concatenated along the channel dimension with the relit vidco latent. A finetuned DiT video model denoises the joint latent according to Equation I, enabling consistent generation of both relit appearance and intrinsic decomposition.
Most related to our approach are methods that leverage large generative image and video models for inverse rendering [18, 32, 54, 40, 30, 14, 47, 41]. Notably, RGB++X [69] demonstrates both intrinsic decomposition and neural rendering from intrinsics using finetuned image diffusion models. The DiffusionRenderer [39] extends RGB+>X video and also supports relighting.
Relighting is the task of modifying the lighting conditions in images or videos. Many methods first reconstruct 3D scenes from multi-view images [46, 22, 13, 9, 71, 76, 70, 56, 62, 38, 64, 42, 58, 25]. Previously, surface reflectance has been captured using calibrated light stages [15]. Now, relighting is supported by material properties recovered from inverse rendering. These methods typically require per-scene optimization and are limited to static, object-centric scenes with single illumination. Training across multiple scenes has led to the exploration of latent feature learning [78, 43, 72, 71, often incorporating neural rendering modules with intrinsic buffers as priors [53, 20, 50, 31, 65].
Recent approaches [55, 31, 68, 26, 39, 5, 75, 45] leverage diffusion models for relighting tasks. These methods are often domain- specifi,,, such as portraits, single objects, and outdoor scenes. Despite promising results, significant challenge remains the need for malti-lllumination datasets [48], which are difficult to capture at scale.
Joint generative modeling approaches enable diffusion models to predict multiple modalities. Matrix3I> [44] predicts pose estimation, depth, and novel view synthesis using single DiT [52] model. VideoJAM [12] extends this by predicting both generated pixels and their corresponding motion from single DIT. We leverage this approach jointly predict relit image and albedo.
3 Preliminaries: Video Diffusion Models
Diffusion models approximate data distribution Pana (I) by learning iteratively denoise samples corrupted by Gaussian noise [59, 23, 17J. For efficiency, most video diffusion models (VDMs) operate in lower-dimensiona. latent space [8, 1]. Given an RGB video E IR 4xxxWXx consisting of L. frames at resolution H x W, a pre-trained VAE encoder & encodes the video into latent tensor
z E(T) E R'xhxwxC Then, the final video is reconstructed by decoding z with pre-trained VAE decoder D. Both the training and inference stages of the VDM conducted in this latent space.
In this work, fine-tune recent Diffusion Transformer (DiT) video model, Cosmos-Predich [1]. Encoding and decoding to and from latent space are performed by the pre-trained VAE Cosmos-1.0-Tokenee O-Tokenizer--Vereeee
which compresses the video by factor of eight along the W spatial and temporal dimensions: I = L C 16, = & and w = 8 The base model supports 8
text- and image-guides video generation resolution of x04x 1280 pixels.
To train the VDM, noisy versions z, = a, + o+e are constructed by adding Gaussian noise e, with the noise schedule provided by a, and a, following the EDM [27]. The diffusion model parameters 0 of the denoising function fo are optimized using denoising score matching objective [27]. Once trained, iteratively applying fo to sample of Gaussian noise will produce sample of (1)

---

4 Method
We propose a generative relighting framework by fine-tuning DiT-based video diffusion model [52, II, which jointly predicts the albedo and relit appearance from input image or video under target lighting condition.
At the core of our method is joint denoising architecture: the latent representations for albedo and relit video are concatenated at the token level and denoised a single pass using the DiT model. This formulation enables cross-modal interaction via self-attention, allowing the model to capture shared scene structure and improve generalization and temporal consistency.
We leverage combination of synthetic datasets, multi- illliiilluminnnionn data, and auto- real-world data to train our model, ensuring robust generalization and high-quality output. the following sections, we detail the model architecture, data strategy, and training objectives.
4.1 Model Design
Given an input video I and a temporally-vane varying target lighting condition E E RL-XHX x3 of the same dimensions, our goal to train a model fo that jointly denoises the albedo a of the input video and relit video E under the target illumination E.
As illustrated in Figure 2, the model comprises a VAE encoder-decoder pair, denoted (E, D), and cansformer-basee denoising function, fg. We use the VAE encoder & to separately encode the input video I, the albedo a, and the relit video IE, producing the corresponding latent tensors (z 2dh z²).
Unlike VideoJAM [12]. which compresses a pair of latents (x", z²) using a linear layer, we find this approach does not yield acceptable quality for our applications. We adopt simple yet effective strategy: concatenating the latents 2. zz, and R along the temporal (frame) dimension. This formulation enables the DiT model apply full self-attention across input, albedo, and relit frames, facilitating cross-modal information exchange.
Token embeddings. To distinguish between the three modalities in the concatenated sequence, we combine standard RoPE positional embeddings with dedicated type embeddings. We first apply the same positional embedding to each of the three video clips, encoding the spatial-temporal position of tokens within each clip. To indicate modality, we introduce a learnable type embedding Comb E j Kane x CCar where Kemb denotes the number of video types. Each type embedding is broadcast across apatial-tempoon dimensions and concatenated along the channel dimension with its corresponding latent representation: c is appended to zE, c to na, and ca to z!.
Condition mask. We attach binary mask to each frame indicating whether it should be treated as a condition (e.g., input source video) or denoising target (e.g.. relit video). During training, we randomly vary which modalities are used inputs, allowing the model generalize under partial supervision. For instance, we may randomly treat a as either a conditioning input or prediction target (see Section 4.3 for details).
Encoding HDR lighting. Environment maps contain high dynamic range (HDR) values that often exceed the intensity range of standard images, while the VAEs used in latent diffusion models are trained on low dynamic range (LDR) inputs and cannot directly encode these
signals.
To address this, we follow prior work [26, 39] and represent the lighting condition using three complementary buffers: (1) an LDR panorama Ejdr obtained via standard Reinhard tonemapping: (2) normalized log-intensity map Ego log( E + 1)/ Fmaa. where Fmax is the maximum log intensity; and (3) a directional encoding Eur E IR x x3 where each pixel stores unit vector indicating its direction in the camera coordinate system. These representations are passed through the VAE encoder, yielding = (E(Eiar), (E.og), ((Eng)) ((air) which is concatenated along the channel dimension with F We also append a binary condition mask to indicate whether lighting features are present. For input video and albedo tokens, we use zero- sardep placeholders to maintain shape consistency.
4.2 Data Strategy
Similar to prior work [39], we combine a large-scale synthetic dataset with smaller real-world datasets. The synthetic dataset offers a large volume of high-quality data, which, when combined with

---

powerful diffusion models, shows impressive generalization to unseen domains [28, 19]. However, using synthetic target images biases the model output. resulting rendered look. Therefore, we also utilize smaller and lower-quality datasets of automatically labeled real-world data.
Synthetic data curation. Our synthetic dataset consists of rendered video clips that tuples (I, Ip, a, E) of input video, relit result, albedo, and environment map. To create a large dataset with complex and diverse scenes and lighting conditions, we follow the methodology of the Diffusion- Renderer [39] and procedurally generate simple scenes with randomized environment map lighting. We curate a collection of 36,500 3D objects from the Objaverse LVIS subset, 4,260 PBR materials, and 766 HDRI environment maps, gathered from publicly available resources. Each scene contains a ground plane with a randomly selected material, and up to three randomly placed 3D objects. We perform collision detection to avoid intersecting objects. Additionally, we place up to three primitive objects (cube, sphere, and cylinder) with randomized shapes and materials, increase variety. randomly selected HDR environment map illuminates each scene. We generate videos with random motions including camera orbits, camera oscillation, lighting rotation, object rotation and translation. Each scene is rendered with two different illuminations under the same motion.
We rendered the dataset using custom path tracer based OptiX [51] at high sample counts with path length three to capture global illumination effects, producing a total of 108k videos with 57 frames per video at a resolution of 704 1280 pixels.
MIT mnniiiilllminntionn labeling. The multi-lllumnnation dataset of Murmann et al. [48] consists of 985 real-world indoor scenes for training and 30 scenes for testing, each lit under 25 different conditions. Each image contains a reflective chrome sphere that has been isolated and exported as an HDR image, which can be easily remapped to the latitude-longitue representation expected by model. We create a dataset by randomly selecting pairs from the 25 lighting conditions and extracting (I, EI Ig).
To obtain graudd-goou albedo labels a, we re-implement the inverse renderer from Diffusion- Renderer [39] based on the Cosmos-1 1. O-Diffusion: (afffuiion-BBiidoo/// 7BVide22orrld model [49], fine-tuned on our synthetic dataset. The resulting model produces high-quality albedo estimates across diverse images and videos. For each scene, we estimate albedo under all 25 lighting conditions and average the results to obtain a stable albedo map.
Real-world auto-labeling. While multi-illumination datasets provide effective supervision for relighting, they costly acquire and limited in scale. In contrast, the Internet offers abundance of high-quality videos captured under single illumination. To leverage this resource, we curate 150k real-world video clips, each with 57 frames at a resolution of 704x1280, and automatically annotate them with albedo maps to create paired data (I, a) using the reproduced inverse rendering model. These auto-labeled RGB-albedo pairs significantly increase the diversity and realism of our training set, enabling improved generalization to real-worke scenes.
4.3 Training
Our model is trained on combination of the synthetic video dataset, the MIT multi-illumination dataset, and real-world auto-labeled data. For image datasets, we treat images as single-frame videos. For both the synthetic video dataset and the MIT multi-lllumintiin dataset, each data sample consists of an input video I, its corresponding albedo a, a new environment map E, and the target relit video Ig under this new illumination. The target latent variable for these datasets is constructed by concatenating the latent of the relit video, zf and the corresponding albedo, zî. along the temporal dimension. Noise is introduced independently to both z and z to produce z, and z.. The model parameters are optimized by minimizing the objective function:
5. (0),, (0) = fo (zz + h z., 1 Gemb; T)
(()) = E P~
2 2
z0 1
F. + Na 12_
z
(2)
where [-] denotes concatenation in the temporal dimension, and Aa = 0.1 is a scalar weight for albedo.
Training strategy. Our training process incorporates specific conditioning strategies tailored to the data source. For synthetic data and the MIT multi-illmminttion dataset, apply three different
5

---

Input
NeuralGaffer
DiLightNet
Refderr
Ours
GT
Env.
Figure 3: Qualitative comparison on the synthetic dataset and MIT Our method produces high-quality inter-reflections and shadows in synthetic scenes (top rows). Crucially, the MIT multi-lllumination dataset (bottom rows), it delivers relighting results with higher accuracy than baselines, which fail when faced with complex materials.
dataset.
conditioning strategies. In 12% of training steps, the input video is dropped (i.e., z = 0), and the ground truth albedo serves as the sole conditioning signal. The denoising function then fz( + h zß. 0]; Gemb.)), In another 18% of the training steps, the model is conditioned on both the input video z! and the ground truth albedo, resulting in the function (( + h z z!|: Cemb. T)- When using these two specific conditions, the conditioning latents of the input video and albedo have their corresponding condition masks set to 1. For the remaining 70% of the training steps, we use the default denoising function where we take input video tokens conditions and predict the denoised relit video and albedo.
Our real-world auto- labele data consists of RGB videos and corresponding albedos but lacks pairwise RGB videos and environment maps. The provided albedo a is used as a condition, and the RGB video treated the target relit video Ig. Since the the original input video and the environment map information are unavailable for this dataset, their respective representations are set to O: The environment map embedding h within the primary input array becomes 0, and the conditional input video representation z! is dropped. The denoising function is thus fg ( + 0, z 0j: Cemb. 7). We adopt a 10% probability of dropping conditions enable classifier-free guidance [24].
5 Results
Before evaluating our method, we introduce baselines, metrics, and datasets. Then we conduct ablation studies on the principle of joint modeling and real-world auto-labeling of data. We refer to model implementation details in the Supplement.
Baselines. For the relighting task, we compare with the 2D methods DiLightNet [68], Neural- Gaffer [26], and Diffusionnenndrer [39]. As the original DiffussinnRendeeer is built Stable Video Diffusion [8], to isolate the effect of the base model and enable a fair algorithmic comparison, we re-implement DiffusionRenderer using the same Cosmos [49] backbone our method, and adopt a data curation strategy closely following the original paper. We refer to this enhanced variant as Rendurio (Cosmos). For albedo estimation, compare with IntrinsiclmageDDiffusiion [32] and DiffuusoonRendeer [39].
Metrics. We apply the standard image metrics PSNR, SSIM, and LPIPS [74] to evaluate relighting and albedo estimation results. For relighting, we also conduct user study to evaluate perceptual quality of the produced results. Following prior works [32, 26, 39], we counteract scale ambiguity in both relighting and albedo estimation by a three-channel scaling factor using least-squares error minimization before computing the metrics.

---

Input
NeuralGaffer
DiLightNet
DiffusionnRederrer
Ours
Env.
Figure 4: Qualitative comparison on in-the-wild data. Our method generates more plausible results than the baselines, with higher quality and more realistic appearance.
Table 1: Quantitative evaluation of relighting, including study, where "Ours preferred" indicates preference over the baselines. A preference over the 50% indicates Ours outperforming baselines.
SyntheticSeenes
MIT mulii-lllmiiation
PSNR 1 SSIM 1 LPIPS
PSNR
SSIM 1 LPIPS J Ours preferred
DiLightNer [68]
19.13
0.584
0.319
15.79
0.539
0.368
92% 8%
Neural Gaffer [26]
19.17
0.638
0.263
17.87
0.683
0.241
84%
2%
DenfesionRenderer [39]
24.17
0.768
0.217
16.99
96%
4%
DiffusionRenderer (Cosmos)
26.61
0.841
0.222
17.29
0.622
0.355
88% 8%
Ours
26.97
0.847
0.190
20.76
0.749
0.251
n/a
Datasets. We have curated a high-quality synthetic dataset named SyntheticScenes for quantitative and qualitative evaluation of both relighting and albedo estimation tasks. This dataset includes 3D assets from PolyHaven [67] and Objaverse [16], ensuring that no assets were used in the training of our method or any of the baseline methods. SyntheticScenes features 40 scenes, each constructed with plane textured with random ahysically-based materials. Each scene is rendered into 57- video sequence under four different lighting conditions, incorporating orbiting camera motions and rotating environment lighting. We cycle through the lighting conditions, selecting one video as the input, and the following as the relighting target, resulting four different relighting tasks per scene. Additionally, we utilize the test set of MIT multi- illimmitiliinniionn benchmark [48], which includes 30 scenes under 25 different illuminations. In this dataset, images are captured sequentially under adjacent flashlight illuminations. For our relighting tasks, the image captured under the i-th lighting condition serves as input, while the image captured under the (i + 12)-th lighting condition is utilized as the target illumination.
5.1 Evaluation
Qualitative comparison. In Figure 3 and Figure 4, we present comparative analysis of our method against recent state-of-the-art relighting techniques: DiLightNet [68], NeuralGaffer [26], and DiffusionRenderrr [39]. Our approach demonstrates superior performance, particularly in handling intricate shadows and inter-reflections. As illustrated in Figure 3, while Diffusionnenderee performs well on the synthetic dataset, it struggles to accurately represent complex materials as anisotropic surfaces, glass, and transparent objects-noe when utilizing G-buffers as an intermediate state, leading to suboptimal results. NeuralGaffer and DiLightNet do not produce accurate shadows and reflections, leading to poor relighting effects. Specifically, NeuralGaffer often makes very subtle, changes to the input video.
Figure compares relighting results on in-the-wild data. For the first two object-centric scenes, we provided DiL.igtNet and NeuralGaffer with masks (we used Dil igghtNtt''s auto-masking, which is based on Segment Anything [29]) to produce more reasonable results. Notably, DiffusionRenderer struggles with fur, human skin, and car windows in the first three cases, as these hard to represent
7

---

Input
Ours (w/o jo modeling)
Shadow baking artifact
Ours
Env.
Ablation on joint modeling, Relighting results on urban street scenes. The orange and green crops highlight regions where the pure relighting model (w/o joint modeling) clearly bakes shadows from the input image into the relit result. Our joint model correctly demodulates the shadows.
Figure 5:
with simple G-buffers. also misestimates the material of the plastic basin, leading to metal-like appearances. Overall, our relighting model more general-purpose and can be used more effectively in diverse scenes, yielding comparatively more accurate and high-fidelity relighting results.
Quantitative comparison. In Table 1, we present a comparative analysis of our method against all baselines on both SyntheticScenes and the MIT multi-illuminntion benchmark [48]. DiffusionRenderer works well on SyntheticScenes, and the Cosmos version-a stronger base model-leads to improved fidelity. However, on the MIT multt-illumination dataset, Rendusionnenderer fails to model the more challenging materials using only the G-buffers, and even the Cosmos version cannot bridge this gap.
An outlier is observed in the LPIPS score for NeuralGaffer on the MIT mentioned before, NeuralGaffer makes minimal changes to the input and performs poorly, but still achieves the lowest LPIPS score. We speculate that I.PIPS considers the images sufficiently similar due to the relatively small deviation in light direction for this dataset.
dataset. As
User study. We conducted user study evaluate the perceptual quality of our relighting method compared to the baseline methods on the MIT multi-illumination dataset. Participants were shown a ground-truth relit image alongside by baseline model, with the order randomly shuffled. They were asked to select the result that more closely resembled the ground truth, considering aspects such as transparency, shadows, and reflections. For each sample pair, 11 users made a binary selection, and majority voting determined the preferred method for each comparison. We repeated the study three times, involving total of 33 users, and report the average percentage of samples where our method is preferred over baselines in Table 1, along with the standard deviation for the three experiments. Our results are consistently preferred, strongly outperforming the other baselines.
relighting results-one generated by method and the other
Albedo estimation. While primary focus is on relighting, we also evaluate the quality of our albedo estimates compared to the baselines in Ta- ble 2. While the Cosmos version of DiffusionRen- derer has slight edge, our method performs on par with previous work in all metrics.
Table 2: Quantitative evaluation of albedo estima- tion on SyntheticScenes.
SyntheticScenes
PSNR
SSIM 1
LPIPS 1
IntrinsicImage--ifftuiie [32]
16.41
0.543
0.395
DiffusionRenderer [39]
26.22
0.837
0.166
Diflusionnenderer (Cosmas)
28.56
0.911
0.131
Ours
28.07
0.877
0.167
5.2 Ablation Study
Ablation on joint modeling. We ablate our joint modeling approach with quantitative and qualitative results in Table and Figure We compare our model to a pure relighting model (without joint modeling), whose denoising function is defined as [ + h 2 r). Both models are trained exclusively on synthetic data. The quantitative results in Table reveal that our joint model slightly improves relighting quality compared to the ablated model. The table also shows that providing ground truth albedo as input our joint model further enhances quality, highlighting the model's 's capacity to leverage albedo information effectively. For qualitative assessment, Figure 5 compares the results on urban street scenes outside the training distribution, where urban street scenes typically exhibit strong shadows under sunlight. The joint modeling approach demonstrates superior generalization, characterized by reduced false shadowing, as clearly shown the figures, whereas the model without joint modeling largely bakes the shadow from the input.
8

---

Input
we auto-labeled data
Ours
Env.
Input
Random Samples
Figure 6: Ablation on real-world auto-labeled data. Although the dataset is sparsely labeled, it helps the model generalize to natural images.
Figure 7: Application for data augmentation. The top left image with green outline is the input image.
Table 3: Ablations on joint model- Table 4: User study of relighting on StreetScenes. Each row compares ing designs evaluated on the Synthet- ablated variant against the base version (Ours w/o auto- labellabeled data), icScene' dataset.
reporting percentage of samples where users prefer the base version.
Ablated Versions
PSNR
SSIM
LPIPS
Ablated Versions
Ours (wlo auto-labeled data) preferved
Ours (wlo joint modeling)
26.12
0,842
0.191
Ours (wlo auto-labeled data, wio joint modeling)
68%
14%
Ours
26.97
0.847
0.190
Ours (w) auto- laheled data)
45% 8%
Ours (w/ GT albedo)
27.45
0.857
0.181
Ours auto-labeled
inferonce-time albedo)
1336
Ablation on real-world automatically labeled data. We ablate the usefulness of our automatically labeled real-wor waaldworld data in Figure 6. The model trained on solely the synthetic and MIT multi- illumination datasets (center column) exhibits noticeable artifacts for out-of-focus regions and outdoor scenes. This is expected, both datasets have very limited depth of field and primarily feature indoor scenes or lack sky-like backgrounds. As shown in the figure, even though the real-world auto-labeled dataset contains only a subset (I. a) of the video labels, significantly helps the model generalize to out-of-domain data, thereby improving image quality.
User study. We conducted user study to evaluate the perceptual impact of our design choices on relighting quality, using urban street scenes. For each scene, participants were shown a reference image and two relit videos produced by ablated versions of our method (randomized order). They were asked to select the video with more realistic lighting, focusing on aspects such as shadows and reflections. As before, each comparison and the preferred result was determined by majority vote. We repeated the study three times with a total of 33 participants and report the average preference rate and the standard deviation in Table 4.
evaluated by 11 participants through binary selection,
When trained only on multi-illumination data (w/o auto-labeled data), the joint modeling variant is strongly preferred over direct relighting. Adding real-world auto-labeled data further improves perceptual quality, consistent with our qualitative and quantitative findings. When comparing models with and without estimated albedo at inference time, the larger standard deviation in preference suggests the perceptual quality is comparable. This indicates our method does not strictly rely on estimated albedo at test time.
5.3 Application: Illumination Augmentation
Our model's strong generalization capability enables effective data augmentation for scenarios such as driving and robotics scenes. We test our model under conditions without environment maps and with different random seeds. Figure 7 shows five random illuminations of input scene. Our model generates diverse data, including nighttime and dusk scenes, demonstrating that it accurately models the illumination distribution and can sample realistic relighting results under varying lighting conditions.
Discussion
To overcome data scarcity and the limitations of decoupled two-stage methods in realistic relighting. we present a novel framework UNIRELIGHT, that jointly models scene intrinsics and illumination. This approach enhances generalization, reduces error accumulation, and more effectively captures complex visual effects by implicitly reasoning about scene properties rather than relying on explicit G-buffers. Furthermore, when trained on real-world auto-labeled data, our model achieves state-of- the-art, high-quality, and realistic relighting results.

---

Limitations and future work. While our method performs well on diverse data, our model cannot handle emitting objects, such as toggling lights within scenes. Our design, which is focused on environmental lighting, does not extend to emittance effects, which remains an area of future research. Moreover, incorporating text-based relighting could significantly enhance usability. Future work could investigate the conditioning of lighting with natural language by incorporating text cross- attention in our DiT model, a feature currently not implemented. While our method can generate reasonable shadows, integrating our generative approach with explicit 3D information- as traced shadow buffers-represemt a promising direction for improving physical accuracy. Besides, in UNIRELIGHT, incorporate the albedo to improve the model's 's generalization, and predicting additional properties could further enhance the model's 's understanding of scene structure and materials. However, memory requirements scale with the number of properties, making this approach expensive for high-resolution videos; is worth exploring this direction further.
Acknowledgments and Disclosure of Funding
The authors thank Tianshi Cao and Huan Ling for their insightful discussions that contributed to this project. This work was conducted at and supported by NVIDIA.
References
Niket Agarwal, Arslan Ali, Maciej Bala, Yogesh Balaji, Erik Barker, Tiffany Cai, Prithvijit Chattopadhyay, Yongxin Chen, Yin Cui, Yifan Ding, et al. Cosmos world foundation model platform for physical AL. arXiv preprint arXiv:2501 003575, 2025. 3,4
[2] Jonathan T. Barron and Jitendra Malik. Shape, illumination, and reflectance from shading. TPAMI, 2014.
[3] Harry Barrow, J. M. Tenenbaum, A. Hanson, and E. Riseman. Recovering intrinsic scene characteristics. Comput. Vis. Syst, 1978. 2
[4] Sean Bell, Kavita Bala, and Noah Snavely. Intrinsic images in the wild. ACM ToG, 2014. 2
[5] Shrisha Bharadwaj. Haiwen Feng. Victoria Abrevaya, and Michael J. Black. GenLit: Reformulating Singee-Image Relighting as Video Generation. arXiv preprint 1rX1v:2412.11224, 2024.
[6] Anand Bhattad, Daniel McKee, Derek Hoiem, and A. Forsyth. StyleGAN knows normal, depth, albedo. and more. arxiv preprint arXiv:2306.00987, 2023.
[7] Anand Bhattad, James Soole, and D.A. Forsyth. Stylitgan: Image-based relighting via latent control. In CVPR, 2024. 3
[8] Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, al. Stable video diffusion: Scaling latent video diffusion models to large datasets, arXiv preprint arXiv:2311.15122 2023. 3,
Mark Boss, Raphael Braun, Varun Jampani, Jonathan T. Barron, Ce Liu, and Hendrik P.A. Lensch, NeRD: neural reflectance decomposition from image collections. ICCV, 2021. 3
[10] Mark Boss, Varun Jampani, Kihwan Kim, Hendrík P.A. Lensch, and Jan Kautz. Two-shot spatially-varying BRDF and shape estimation. In CVPR, 2020. 2
[11] Adrien Bousscau, Sylvain Paris, and Frédo Durand. User-assisted intrinsic images. ACM ToG, 2009. 2
[12] Hila Chefer; Uriel Singer, Amit Zohar, Yuval Kirstain, Adam Polyak, Yaniv Taigman, Lior Wolf, and Shelly Sheynin. VideoJAM: Joint appearance-motion representations for enhanced motion generation in video models. arXiv preprint arXiv: 2502.02492 2025. 2, 3,4
[13] Wenzheng Chen, Joey Litalien. Jun Gao. Zian Wang. Clement Fuji Tsang, Sameh Khalis, Or Litany, and Sanja Fidler. DIB-R++: Learning to predict lighting and material with a hybrid differentiable renderer. In NeurlPS, 2021.
[14] Zhifei Chen, Tianshuo Xu, Wenhang Ge, Leyi Wu, Dongyu Yan, Jing He, Luozhou Wang. Lu Zeng. Shunsi Zhang, and Yingcong Chen. Uni Renderer: Unifying Rendering and Inverse Rendering Via Dual Stream Diffusion. arXiv preprint 15500 2025.
10

---

[15] Paul Debevec, Tim Hawkins, Chris Tchou, Haarm-Pieter Duiker, Westley Sarokin, and Mark Sagar. Acquiring the reflectance field of human face. In SIGGRAPII, 2000.
[16] Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: universe of annotated 3D objects. In CVPR, 2023. 7
[17] Prafulla Dhariwal and Alexander Quinn Nichol. Diffusion models beat GANs on image synthesis. In NeurIPS, 2021. 3
[18] Xiaodan Du, Nicholas Kolkin, Greg Shakhnarovich, and Anand Bhattad. Generative models: What do they know? Do they know things? Let's find out! arXiv preprint arXiv:2311.17137, 2024. 3
[19] Xiao Fu, Wei Yin, Mu Hu, Kaixuan Wang, Yuexin Ma. Ping Tan, Shaojie Shen, Dahua Lin, and Xiaoxiao Long. GeoWizard: unlcashing the diffusion priors for 3D geometry estimation from a single image. In ECCV, 2024. 2,5
[20] David Griffiths, Tobias Ritschel, and Julien Philip. Outcast: Outdoor single-image relighting with cast shadows, Computer Graphics Forum, 2022.
[21] Roger Grosse, Micah Johnson, Edward Adelson, William Freeman. Ground truth dataset and baseline evaluations for intrinsic image algorithms. In ICCV, pages 2335-2342. IEEE, 2009.
[22] Jon Hasselgren, Nikolai Hofmann, and Jacob Munkberg. Shape, light, and material decomposition from images using Monte Carlo rendering and denoising. arXiv preprint arXiv:2206.03388) 2022. 3
[23] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In NeurIPS, 2020.
[24] Jonathan Ho and Tim Salimans. Classifier-free diffiusion guidance. arXiv preprint arXiv:2207. 2022.
[25] Yingwenqi Jiang, Jiadong Tu, Yuan Liu, Xifeng Gao, Xiaoxiao Long, Wenping Wang. and Yuexin Ma. GaussianShader: 3D Gaussian splatting with shading functions for reflective surfaces. In CVPR, 2024. 3
[26] Haian Jin, Yuan Li. Fujun Luan, Yuanbo Xiangli, Sai Bi, Kai Zhang, Zexiang Xu, Jin Sun, and Noah Snavely. Neural Gaffer: Relighting any object via diffusion. In NeurIPS, 2024. 3, 6,
[27] Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion--assed generative models. In NeurIPS, 2022.
[28] Bingxin Ke, Anton Obukhov, Shengyu Huang, Nando Metzger, Rodrigo Caye Daudt, and Konrad Schindler. Repurposing diffusion-based image generators for monocular depth estimation. In CVPR, 2024. 2,
[29] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander Berg, Wan- Yen Lo, Piotr Dollár, and Ross Girshick. Segment Anything. arXiv preprint arXiv:2304.02643, 2023.7 7
[30] Peter Kocsis, Lukas Höllein, and Matthis Nießner. IntrinsiX: High-Quality PBR Generation using Image Priors. arXiv preprint ar arXiv:2504.0100 2025.
[31] Peter Kocsis, Julien Philip. Kalyan Sunkavalli, Matthias Nießnner, and Yannick Hold Hold-Geoffroy. LightIt: illumination modeling and control for diffusion models. In CVPR, 2024. 3
[32] Peter Kocsis, Vincent Sitzmann. and Matthias Nießner. Intrinsic image diffusion for indoor single-view material estimation. In CVPR, 2024. 2, 3, 6, 8
[33] Balazs Kovacs, Sean Bell, Noah Snavely, and Kavita Bala. Shading annotations in the wild. In CVPR, 2017. 2
[34] Edwin H. Land and John J. McCann. Lightness and retinex theory. J. Opt. Soe. Am., 1971.
[35] Zhengqi Li and Noah Snavely, CGintrinsics: Better intrinsic image decomposition through physically-based rendering. In ECCV, 2018.
[36] Zhengqin Li. Mohammad Shafiei, Ravi Ramamoorthi, Kalyan Sunkavalli, and Manmohan Chandraker. Inverse rendering for complex indoor scenes: Shape, spatially lighting and svbrdf from a single image. In CVPR, 2020. 2

---

[37] Zhengqin Li, Ting-Wei Wei Yu, Shen Sang, Sarah Wang, Sai Bi, Zexiang Xu, Hong-Xing Yu, Kalyan Sunkavalli, Miloš Hašan, Ravi Ramamoorthi, al. OpenRooms: end-to-end open framework for photorealistic indoor scene datasets. arXiv preprint arXiv:2007. 2020. 2
[38]
Ruofan Liang. Huiting Chen, Chunlin Li, Fan Chen, Selvakumar Panneer, and Nandita Vijaykumar. ENVIDR: Implicit differentiahle renderer with neural environment lighting. In ICCV, 2023.
[39] Ruofan Liang. Zan Gojeic, Huan Ling, Jacob Munkberg, Jon Hasselgren, Zai-Hao Lin, Jun Gao, Alexander Keller, Nandita Vijaykumar, Sanja Fidler, and Zian Wang. Diflusionrenderer: Neural inverse and forward rendering with video diffusion models. In CVPR, 2025. 1, 3, 4, 5, 6, 7, 8, 22, 23
[40] Ruofan Liang. Zan Gojcie, Merlin Nimier-Dar David. David Acuna, Nandita Vijaykumar, Sanja Fidler, and Zian Wang. Photorealistic object insertion with diffusion-guiden inverse rendering. In ECCV, 2024. 3
[41] Ruofan Liang. Kai He, Zan Gajeie, Igor Gilitschenski, Sanja Fidler, Nandita Vijaykumar, and Zian Wang.
Luxdit: Lighting estimation with video diffusion transformer, 2025. 3
[42] Zhihao Liang, Qi Zhang. Ying Feng. Ying Shan, and Kui Jia. GS-IR: 3D Gaussian splatting for inverse rendering. arXiv preprint arXiv:2311.1647, 2023.
[43] Andrew Liu, Shiry Ginosar, Tinghui Zhou, Alexei A. Efros, and Noah Snavely. Learning factorize and relight a city. In ECCV, 2020.
[44] Yuanxun Lu, Jingyang Zhang, Tian Fang, Jean-Daniel Nahmias, Yanghai Tsin, Long Quan, Xun Cao, Yao Yao, and Shiwei Li. Matrix3D: Large Photogrammetry Model All-in-One. arXiv preprint arXiv:250.00688, 2025. 3
[45] Yiqun Mei, Mingming He, Li Ma. Julien Philip, Wenqi Xian, David M George, Xueming Yu, Gabriel Dedic, Ahmet Levent Taşel, Ning Yu, et al. Lux post facto: Learning portrait performance relighting with conditional video diffusion and hybrid dataset. In CVPR, 2025.
[46] Jacob Munkberg, Jon Hasselgren, Tianchang Shen, Jun Gao. Wenzheng Chen, Alex Evans, Thomas Müller, and Sanja Fidler. Extracting triangular 3D models, materials, and lighting from images. arXiv preprint arXiv:2111 11..21111.1200, 2021. 3
[47] Jacob Munkberg, Zian Wang, Ruofan Liang. Tianchang Shen, and Jon Hasselgren. VideoMat: Extracting PBR Materials from Video Diffusion Models. In EGSR CGF Track, 2025. 3
[48] Lukas Murmann, Michael Gharbi, Miik Aittala, and Fredo Durand. A multi-illmmintion dataset of indoor object appearance. In ICCV, 2019, 3, 5, 7,8
[49]
NVIDIA. Cosmos world foundation model platform for physical Al. arXiv preprint arXiv:2501.053 03575. 2025. 5, 2.
[50] Rohit Pandey, Sergio Orts- Orts-Escolano, Chloc LeGendre, Christian Hacne, Sofien Bouaziz, Christoph Rhemann, Paul Debevee, and Scan Fanello. Total relighting: Learning to relight portraits for background replacement. ACM ToG, 2021. 3
[51] Steven Parker. James Bigler. Andreas Dietrich, Heiko Friedrich, Jared Hoberock, David Luebke, David McAllister, Morgan McGuire, Keith Morley, Austin Robison, and Martin Stich. OptiX: a general purpose ray tracing engine. ACM ToG. 2010. 5
[52] William Peebles and Saining Xie. arXiv:2212.09748, 2022. 3,4
Scalable diffusion models with transformers. arXiv preprint
[53] Julien Philip, Michael Gharbi, Tinghui Zhou, Alexei A Efros, and George Drettakis. Multi-view relighting using geometry network. ACM ToG, 2019.
[54] Pakkapon Phongthawee, Worameth Chinchuthakun, Nontaphat Sinsunthithet, Amit Raj. Varun Jampani, Pramook Khungurn, and Supasorn Suwajanakom. DiffusionLight: light probes for free by painting a chrome ball. arXiv preprint arxiv:2312.09166 2023.
[55] Yohan Poirier-Ginter Alban Gauthier, Julien Philip. lean-François Lalonde, and George Drettakis. Diffusion Approach to Radiance Field Relighting using Multi-lllmination Synthesis. Computer Graphics Forum, 2024. 3
[56] Viktor Rudnev, Mohamed Elgharib. William Smith, Lingjee Liu, Vladislav Golyanik, and Christian Theobalt. NeRF for outdoor scene relighting. In ECCV, 2022. 3
12

---

[57] Soumyadip Sengupta. Jinwei Gu, Kihwan Kim. Guilin Liu. David W. Jacobs, and Jan Kautz. Neural inverse rendering of an indoor scene from single image. In ICCV, 2019. 2
[58] Yahao Shi. Yanmin Wu, Chenming Wu, Xing Liu, Chen Zhao, Haocheng Feng, Jingtuo Liu, Liangjun Zhang, Jian Zhang. Bin Zhou, et al. GIR: 3D Gaussian inverse rendering for relightable scene factorization. arXiv preprint arXiv:2312.05133, 2023.
[59] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In ICML, 2015. 3
[60] Zian Wang, Wenzheng Chen, David Acuna, Jan Kautz, and Sanja Fidler. Neural light field estimation for street scenes with differentiable virtual object insertion. In ECCV, 2022. 2
[61] Zian Wang, Jonah Philion, Sanja Fidler, and Jan Kautz. Learning indoor inverse rendering with 3D waatinl lighting. In ICCV, 2021. 2
[62] Zian Wang, Tianchang Shen, Jun Gao, Shengyu Huang, Jacob Munkberg, Jon Hasselgren, Zan Gojeic, Wenzheng Chen, and Sanja Fidler. Neural fields meet explicit geometric representations for inverse rendering of urban scenes. In CVPR, June 2023. 3
[63] Felix Wimbauer, Shangzhe Wu, and Christian Rupprecht. De-rendering 3D objects in the wild. In CVPR, 2022. 2
[64] Chen Xi, Peng Sida, Yang Dongchen, Liu Yuan, Pan Bowen, Lv Chengfei, and Zhou. Xiaowei. Intrinsi- cAnything: learning diffusion priors for inverse rendering under unknown illumination. arXiv preprint arXiv: 2404.11595 2024.
[65] Xianyan Xing, Konrad Groh, Sezer Karaoglu, Theo Gevers, and Anand Bhattad. Luminet: Latent intrinsics meets diffusion models for indoor scene relighting. arXiv preprint arXiv:2412.0011 2024. 3
[66] Ye Yu and William A. P. Smith. InverseRenderNet: learning single image inverse rendering. In CVPR, 2019. 2
[67] Greg Zaal and et al. Poly Haven The Public 3D Asset Library, 2024.7 7
[68] Chong Zeng, Yue Dong, Picter Peers, Youkang Kong, Hongzhi Wu, and Xin Tong. DiLightNet: fine-grained lighting control for diffusion- image generation. In SIGGRAPH, 2024. 3, 6, 7, 21
[69] Zheng Zeng, Valentin Deschaintre, Iliyan Georgiev, Yannick Hold Geoffroy, Yiwei Hu, Fujun Luan, Ling- Qi Yan, and Miloš Hašan. RGB++X image decomposition and synthesis using material-and lighting- aware diffusion models. In SIGGRAPH, 2024. 1,3
[70] Kai Zhang, Fujun Luan, Zhengqi Li, and Noah Snavely. IRON: inverse rendering by optimizing neural SDFs and materials from photometric images. CVPR. 2022.
[71] Kai Zhang, Fujun Luan. Qianqian Wang. Kavita Bala. and Noah Snavely. PhySG: Inverse rendering with spherical Gaussians for physics-based material editing and relighting, CVPR, 2021. 3
[72] Longwen Zhang, Qixuan Zhang, Minye Wu, Jingyi Yu, and Lan Xu. Neural video portrait relighting in real-time via consistency modeling. In ICCV, 2021. 3
[73] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Scaling in-the-wild training for diffusion-basee -ased illumination harmonization and editing by imposing consistent light transport. In ICLR, 2025. 22, 23
[74] Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as perceptual metric. In CVPR, 2018. 6
[75] Tianyuan Zhang, Zhengfei Kuang, Haian Jin, Zexiang Xu, Sai Bi, Hao Tan, He Zhang, Yiwei Hu, Milos Hasan, William T. Freeman, Kai Zhang, and Fujun Luan. RelitL.MM: Generative relightable radiance for large reconstruction models. In ICLR, 2025. 3
[76] Xiuming Zhang, Pratul P. Srinivasan, Boyang Deng, Paul Debevec, William T. Freeman, and Jonathan T. Barron. NeRFactor: neural factorization of shape and reflectance under unknown illumination. ACM ToG, 2021.3
[77] Qi Zhao, Ping Tan. Qiang Dai, Li Shen, Enhua Wu, and Stephen Lin. A closed-form solution to retinex with nonlocal texture constraints. TPAMI, 34(7)1437711444, 2012.
[78] Hao Zhou, Sunil Hadap, Kalyan Sunkavalli, and David W. Jacobs. Deep single- portrait relighting. In ICCV, October 2019. 3
13

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justificcton right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No]", ", it is perfectly acceptable to answer "[No] provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: The abstract and introduction clearly state our contributions, which are fully supported by the experiments and analysis presented.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
14

---

Justification:
We discuss limitations in Section 6.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approxinations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proof'
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer:
[NA]
Justification:
The paper does not contain theoretical results or formal proofs.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: We include all necessary details for reproduction the main paper and supplementary materials.
Guidelines:
15

---

The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: We plan to release the code and data upon acceptance. The internal guidelines of our institution prevent us from releasing code at this stage.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm cc/ public/guien Seaeeees/ooSSuumisseeeeeee
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
/
http:///gggiiee//eeeummmiiii (Codeeumiissiiiii ssionPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
16

---

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
All relevant details provided in the paper and supplementary materials.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Due to the high computational cost of training large-scale video diffusion models, we did not perform multiple runs to report error bars.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula,
call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
Question:
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification:
We include sufficient information supplementary materials.
Guidelines:
The answer NA means that the paper does not include experiments.
17

---

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https:://nnn /ttppp:///euriss.... httpppsccccpplcccccceeeeccceeeee http:/tttiiicciieeeeeeeeeeeeeeee ines?
Answer: [Yes]
Justification: Our work adheres the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discussed societal impacts of our work in the paper.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our work does not involve models or data with high risk of misuse.
18

---

Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Third- -arttpartyy assets are properly credited, and their licenses are respected.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of in the wwwrsssithhhheeeeeee package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re- re-packaged, both the original license and the license of the derived asset (if has changed) should provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: Our paper does not release new assets.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
19

---

Answer: [Yes]
We include the details of all crowdsourcing experiments in supplementary materials.
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [Yes]
Justification: An approval is obtained for all of our experiments.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: LLMs
used in the core methods of this research.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non non-standart components.
Please refer to our LLM policy (https /ttp//eeeenp/eee/ee ww/Conferences///00 /LLM) for what should or should not be described.
20

---

In this Appendix, we first discuss the broader impact of our project (Appendix A). We then provide additional implementation details of our model and experiments (Appendix B), followed by further results and analysis (Appendix C). Please refer to the accompanying video for additional qualitative results and comparisons.
A Broader Impact
We present UniRelight, generative framework that jointly estimates albedo and synthesizes relit videos from a single input, enabling diverse lighting manipulation across both synthetic and real- world scenes. This capability can support range of applications, including creative content generation, visual effects, virtual production, and potentially data augmentation for training more robust computer vision models in domains such as robotics and autonomous driving.
As with all generative video models, UniRelight may reflect biases present in its training data. Such biases could lead to relighting results that fail to generalize to underrepresented scene types or lighting conditions. Furthermore, tools for lighting manipulation carry the risk of misuse, such as altering or misrepresenting visual content in sensitive contexts like surveillance or media.
We discourage the use of UniRelight in applications where relighting may contribute to misinforma- tion, misattribution, or privacy violations. In human-centric use cases, we recommend careful dataset curation ensure fair representation across skin tones, races, and gender identities. Practitioners are encouraged to critically assess and de-bias training data to mitigate unintended harms where appropriate.
B Experimental Details
B.1 Implementation Details
We fine-tune models based on (omme--Pradictt11--Bdded20 [49], pre-trained DiT video diffusion model.
The encoded latents zz, zz, _ E(Ear)) E (Eiog), and {(Eir) are all in gxxhxuxC where C' = 16. We use Cemb as the dimension of the type embedding. Thus, the concatenated tokens have a channel dimension of 16 condition mask (added as an extra channel), and the lighting features (E Ejog. Edit) each include a condition mask to indicate whether they are provided.
+ (16 + 1) x 3
71, where each latent has associated binary
We adopt an image-video co-training strategy and train the model two stages. Firstly, we train with only synthetic data by mixing the image data (sampling one frame from the video data) and the video data in ratio of 1 1 for 15,000 iterations. Then we train with all data with random sampling, including synthetic video data, synthetic image data, real-world auto-labeled data, and MIT 3 2 for 12,000 iterations. We augment the real-world
multi-lllmmintionn data, in a ratio auto-labeled data with random flipping.
All training is done with a batch size of 64, using the AdamW optimizer with a learning rate of mixed-precision
2 x 10 with mixed precision (BF16) training at resolution of 480 x 848 pixels. The AdamW optimizer was employed with a weight decay of 0. 1. The exponential decay rates for the moment estimates 3 are set to 0.9 for the first moment and 0.99 for the second moment, with at x 10-10. The total training of two stages takes around days on 32 A100 GPUs.
During inference, we use 35 denoising steps. We do not apply classifier-free guidance (CFG), as we empirically found that inference without CFG yields more accurate and visually consistent results.
Baseline configurations. Since Dil iighNet [68] requires a text prompt per example, we use meta/llama-3 -wt00-vissioninssttreeeeeseeeeee
generate short prompt for each example in the datasets based on the first image each clip with the instruction "What is in this image? Describe the materials. Be concise and produce an answer with a few sentences, more than 50 words." "
As each the baselines generates videos in different resolutions, for UNet-based baselines, we run inference the model with the video first resized to 486 x 864 and then center-cropped to
htttps:/ :/tww 11 aaaaa.c. com/
21

---

Figure 8: Synthetic data visualization. Randomly sampled example images are shown from our synthetic rendering data. The top-most images show albedo maps, while the bottom two rows display rendered scenes under two different illuminations with the corresponding environment maps.
Table 5: Quantitative comparison with IC-
Table 6: Evaluation of inference runtime cost.
Light.
Runtime cost (seconds)
PSNR t SSIM LPIPS
DiffusiionRendeee [39]
566.6
IC-Light [73]
18.08 23.19
0.834
0.096
DiffusionRRnderer (Cosmos)
780.0
Ours
0.901
0.079
Ours
445.5
resolution of 448 832; for our DiT-based model, we run inference the model with the resized video with resolution of 486 x 864 and then center-cropped to 448 x 832 to align the results.
Quantitative comparison configurations. For quantitative evaluation, we apply background masks to the synthetic dataset to focus on foreground appearance. For the MIT multi-illumination dataset, we follow the dataset protocol and mask out light probes in all outputs before computing metrics.
B.2 User Study Details
We conducted two user studies on Amazon Mechanical Turk evaluate the perceptual quality of relighting results.
dataset public benchmark with ground truth relighting. Participants were shown three images: ground truth relit image and two relighting results one generated by our method and one by a baseline model. Their task was to choose the result that more closely resembled the ground truth, considering attributes such transparency, shadows, and reflections.
MIT
The exact instructions shown to participants were follows:
Carefully compare Image A, the Reference Image, and Image B. Your task is to determine which image (A B) is more similar to the Reference Image. To make an informed decision, you may zoom in examine the details. Pay close attention to aspects such as lighting, reflections, and shadows, as these can affect how natural the image appears.
Once you have compared the images, select the one that best matches the Reference Image.
Image
Image B
We evaluated 30 scenes from the test set, comparing our method against four basclines. The study was repeated three times with 11 unique participants in each run. In total, this resulted in 30 3 3960 individual comparisons.
StreetScenes Dataset. This dataset contains 19 urban street scenes without ground-truth relighting. Participants were shown a reference image along with two relit videos generated by different ablated versions of our method.
The instructions presented to participants
follows:
In this study, you will be shown Reference Image and two videos Video and Vidco B that changes the lighting of the scene. Your task is to watch both
22

---

Env.
Input
IC-Light
Ours
GT
Figure 9: Qualitative comparison with IC-Light [73]. We provide the environmental background used for IC-Light conditioning, with the reference environment ball on the left. Our method produces higher-quality and more accurate relighting results.
videos and choose which one (A or B) you think has more realistic shadows and reflections. To make an informed decision, you may zoom in to examine the details. Pay close attention to aspects such as lighting, reflections, and shadows, as these can affect how natural the image appears.
Once you have compared the videos, select the one that has more realistic lighting effects.
Video A
Video B
We evaluated 19 scenes, comparing a base version against two ablated versions of our method. The study was repeated three times with 11 unique participants in each run. total, this resulted in 19 x 2 x 11 x 3 1254 individual comparisons.
B.3 Synthetic data visualization
We show synthetic data visualization in Figure 8. Each contains albedo videos, two environ- ment maps, and pairwise videos rendered under the environment maps.
C Additional Results
Runtime cost. We evaluate the inference runtime of our model 57-frame video resolution of 480 x 848. The overall inference time for performing 35 denoising steps, including VAE encoding and decoding, 445.5 seconds, measured
single A100 GPU.
To contextualize this cost, we compare our method against two baselines: DiffusiionRenderer [39] and its Cosmos-based variant. All methods use 35 denoising steps for consistency. DiffusionnRnderer is run at 448 x 832 resolution (slightly smaller than ours but divisible by 32 fit its architecture) while the Cosmos variant is run at native resolution.
For baseline methods, the total runtime is the sum the inverse rendering and forward rendering durations. Notably, DiffusionnRendeeer requires five separate inverse rendering passes and one forward rendering pass per video, resulting in significantly higher computational cost. contrast, approach performs joint relighting and albedo estimation in a single pass and is correspondingly faster. Full timing results are shown Table 6.
Comparison with IC-Light [73]. We compare our method with the single-image relighting ap- proach IC-Light [73] on object-centric synthetic data, as shown in Table 5 and Figure 9. Note that the
23

---

DiflusionRenderer (Cosmos)
Input
NeuralGuffer
Dittg
Ours
GT
Env
Figure 10: Additional qualitative comparison on MIT multi-illumination dataset. Our method consistently achieves more accurate relighting results than all baselines on the MIT multi-lllumination dataset, demonstrating strong capability in relighting complex materials.
two methods follow different relighting formulations: IC-Light is designed for object relighting using background context as the primary cue-without access to an explicit environment map, while our method is conditioned on full HDR illumination, but is not specifically tuned for object-centric data.
Our method shows improved quantitative and qualitative performance. Since IC-Light relies on background appearance as its primary cue and has less information about the surrounding lighting, it may retain input-specific effects in its outputs-such as specular highlights and shadows, which can limit accuracy under novel lighting conditions. In contrast, our method produces more faithful relighting results, with sharper specular highlights. more realistic shadows, and improved visual fidelity.
Additional qualitative comparison on the MIT multi-illumination dataset. provide addi- tional qualitative comparisons on the MIT Multi-lllumination dataset in Figure 10. To ensure fair comparison, we include results from our re-implemented version of DiffusionnRnderer using the Cos- mos backbone, which achieves higher visual fidelity than the original implementation. Our method

---

Input
Relighting
Relighting II
Relighting III
Figure 11: Additional qualitative results under point-light illumination. The bottom right of each column indicates the target lighting conditions. Our results show strong robustness of method under point-light illumination.
Input
Estimated Albedo
Relighting
Env.
Figure 12: Additional qualitative results real scenes. Our method provides high-quality albedo estimation and realistic relighting results.
consistently produces more accurate transparency, specular highlights, and shadows across scenes, demonstrating strong capability in handling complex materials and outperforming baselines in visual quality.
Additional qualitative results under point-light illumination. We further evaluate the robustness of our method in extreme cases, such point-light illuminations, which do not exist in training data. As shown in Figure 11, our method produces high-quality relighting results, demonstrating the strong robustness and generalization capability of our method.
Additional qualitative results on real scenes. We present additional results on real scenes in Figure 12. Our method produces nigh-quality albedo and relighting results with realistic specular highlights and shadows under target lighting conditions.
25


Reviews:
[{"review_id": "tHvfZTyfXZ", "reviewer": "Reviewer_QAPg", "summary": "The paper proposes UniRelight, a video-diffusion framework that jointly estimates an albedo map and synthesises a relit video. By concatenating latent codes for albedo and lighting and denoising them together, the model avoids the error accumulation typical of two-stage pipelines, while allowing faster processing. \nTo overcome the scarcity of multi-illumination data, the authors want to release a large synthetic dataset for training and a validation dataset SyntheticScenes, and a auto-label 150 k Internet video clips with albedo maps generated by their own inverse-rendering approach. Experiments show higher PSNR / LPIPS and a strong user-study preference over baselines such as DiffusionRenderer and Neural Gaffer and DiLightNet. Ablations confirm the benefits of the joint formulation and the auto-labelled real-video pre-training.", "questions": "-  Probe robustness – How much do PSNR / LPIPS drop if the HDR probe is rotated, clamped, or replaced by a point-light?\n-  Auto-label quality – What is the albedo noise level, and how does training without labels change results? Did you filter the datasets somehow for quality? Please add a few dataset visuals.\n-  Geometry handling – Have you tested on shots with strong parallax or moving occluders? \n-  Efficiency – What FPS and GPU memory does UniRelight need at 720p, 1080p, and 4 K, and how does that compare to DiffusionRenderer?\n-  Synthetic-set coverage – Which lighting types (point, HDR dome, emissive) are in SyntheticScenes, and how were they sampled? Include representative examples.", "limitations": "Limitations are discussed in the manuscript.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Strengths\n\n•\tSingle-pass joint decomposition + relighting elegantly unifies intrinsic estimation and light transport, giving visibly cleaner results than two-stage pipelines and simplifying the model architecture, which also leads to faster inference.\n\n•\tTemporal consistency is highlighted through a video-diffusion backbone, avoiding flicker (l. 109). However, the benefit over previous work could be shown in more detail with direct comparisons to DiffusionRenderer or DiLightNet.\n\n•\tThe hybrid training strategy (l. 54) improves generalisation, though is only roughly explained in (l. 198-214). Further details are only added in the supplementary material and should be perhaps added in the main document.\n\n•\tThorough evaluation shows strong baselines, standard metrics, user studies, and ablations.\n\n•\tA clear discussion of limitations (l. 332) aids the exploration of future work.\n\n\nWeaknesses\n\n•\tDependence on an HDR environment probe at inference; robustness to probe errors or indoor spotlights is untested.\n\n•\tNo explicit geometry reasoning means large view changes or strong occlusions may degrade results.\n\n•\tQuality of auto-labels is not quantified; the effect of noisy albedo on final quality is unclear. It is also not fully clear how the overall auto-labelled dataset looks on a larger scale, as almost no qualitative figures are provided.\n\n•\tThe run-time footprint (FPS, memory) is only briefly mentioned; practical deployment is uncertain and still far from real-time capabilities needed for adaptive resimulation frameworks or similar approaches.\n\n•\tSimulation datasets are neither qualitatively nor quantitatively shown. They may lack complex lighting, such as moving point sources or emissive objects, and the lighting paths—including how they are sampled—are not described.", "was_revised": true, "final_justification": "I thank the authors for their response, which addresses my questions. I will leave my rating as set to accept.", "rebuttal": "We thank the reviewer for their valuable feedback, and appreciate that they recognize our contributions. We respond to the comments and questions below.\n\n**Probe robustness.** Conditioning our model on light-probes assumes distant lighting, which is indeed a limitation. We note that our model often produces plausible results for indoor scenes, but do not expect it to scale to extreme cases such as local spotlights. Our training data includes HDR environment maps with randomized rotations and intensity, enabling the model to follow the control of environment map condition, and generalize well diverse probes. In practice, we observe stable performance under rotated or clamped probes. We can add results with environment maps with a single lit pixel/OLAT config in the revision.\n\n**Auto-label quality visualization.** We re-implemented DiffusionRenderer [39] using the Cosmos backbone [46] to label the G-buffers, and achieved high-quality estimation on diverse scenes. We appreciate the suggestion and will include additional visualizations of the auto-labeled dataset in the revised supplementary material. While the labels are “pseudo-groundtruth”, Fig. 6 shows that training on this data improves relighting quality, demonstrating the value of large-scale real-world data. \n\n**Synthetic dataset visualization.** Our synthetic dataset was rendered using around 2,000 HDR environment maps. We use a path-tracer to render video clips under randomized lighting rotations, intensities, and color temperatures. We will show some representative examples and include more details of our synthetic data generation in the revision. \n\n**Geometry handling.** Our model implicitly handles occlusions and synthesizes cast shadows as demonstrated in several scenes (e.g., tree shadows) shown in Supplementary video 0:44~0:59, Fig. 5, and Supp. Fig. 3. However, performance may degrade with thin structures or fast-moving occluders. Incorporating 3D geometry-aware priors (e.g., traced shadows from estimated depth or meshes) is a promising direction for future work. \n\n**Efficiency.** During inference, UniRelight uses more memory (37.9 GB @ 480×848 pixels x 25 frames) than Diffusionrenderer (32.0 GB @ 448×832 pixels x 25 frames). This is due to the joint modeling of albedo and relighting in a single denoising pass. Our inference is faster than DiffusionRenderer, as shown in Supp. Tab. 2, thanks to the single-pass architecture. We acknowledge that runtime and memory remain deployment challenges. As future work, we plan to adopt memory-efficient architectures such as Framepack [1] and explore model distillation. While 1080p and 4K are currently infeasible memory-wise using the Cosmos backbone on current GPUs, we leave acceleration and support for higher resolutions as future work.\n\n[1] Zhang, Lvmin and Agrawala, Maneesh. “Packing Input Frame Contexts in Next-Frame Prediction Models for Video Generation” Arxiv 2025."}, {"review_id": "CF4zaTNnQS", "reviewer": "Reviewer_tfJ3", "summary": "The paper introduces a video relighting model that can be conditioned on a target environment map. Prior work in video modeling showed that video models can learn to produce better motion if they are trained to output an auxiliary output connected to the motion (like optical flow), and this paper builds on top of that idea and outputs an albedo estimate of the video, in hopes that will improve the model's performance in video relighting. The authors use a mixture of synthetic data, and real-world data with and without GT.", "questions": "If predicting albedo improves performance, then why stop there? Wouldn't outputting material property also be beneficial? It seems that the ideal architecture would be outputting all the G-buffers like DiffusionRenderer does, and also output an RGB relighting video along it. Since the authors already reproduced DiffusionRenderer, this can be a much stronger method. I don't expect the authors to do this for the rebuttal, but it could potentially make the paper much stronger.\n\nLine 145, encoding the ray maps with the video encoder seems unnatural. Why not only downsample it bilinearly? do we know if those ray maps are within distribution for the video encoder?\n\nIt appears in the results that the method is strongest relative to the baselines in scenes with objects that are semi-transparent or reflective, would it be possible to construct a dataset that makes the method completely stand out relative to the baselines and show off its strengths?\n\nA quantitative dataset ablation, that shows the benefits of having each dataset can be interesting. Like for the synthetic dataset, MIT multi-illumination dataset, and the auto-labeling. And how the performance is affected on the test split of each dataset. I ask for this because it is not clear to me how much each dataset is helping the performance on the other datasets.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- The paper's idea is sound. The proposed hypothesis is: would outputting auxiliary information related to lighting improve the performance? and the paper goes on to validate the hypothesis and show that it is true\n\n- The relighting performance is consistent for single images with rotating environment maps, and also addresses shadows and reflections nicely\n\n- The work manages to incorporate real world datasets into the training, which is nice since most prior relighting work relies exclusively on synthetic data\n\nWeaknesses:\n- I'm surprised that in Table 3, the gain of modeling the albedo is actually marginal. Isn't that the entire thesis of the paper? Also, I'd like to see similar ablation (w/o the GT albedo row) on the other used datasets like MIT multi illumination dataset. Restricting the results to the SyntheticScenes is odd\n\n- DiffusionRenderer (Cosmos) has a better performance in outputting albedo (indicating that outputting the other buffers are beneficial), so this is begging the question, why not build on top of that and output the relit RGB as another \"buffer\"\n\n\nSide comments:\n- NeuralGaffer is trained on objects only, so it makes sense that it would behave oddly on scenes\n- There are some multiview relighting papers that could be relevant to the related work (since prior work that relights multiple frames at the same time are limited). For example, SimVS (Trevithick et. al 2025) and generative multiview relighting (Alzayer et. al 2025)\n- line 99: following the EDM —> following EDM\n\n\nTrevithick, Alex, et al. \"Simvs: Simulating world inconsistencies for robust view synthesis.\" Proceedings of the Computer Vision and Pattern Recognition Conference. 2025.\n\nAlzayer, Hadi, et al. \"Generative multiview relighting for 3d reconstruction under extreme illumination variation.\" Proceedings of the Computer Vision and Pattern Recognition Conference. 2025.", "was_revised": true, "final_justification": "The paper introduces an interesting idea of outputting intrinsics as a pseudo-task to improve relighting, and shows positive results, so I recommend accepting the paper", "rebuttal": "We thank the reviewer for their valuable feedback, and appreciate that they recognize our contributions. We respond to the comments and questions below.\n\n**Joint modeling ablation.** Our joint modeling formulation encourages the model to learn internal representations of scene structure, which improves generalization to unseen domains and mitigates shadow baking artifacts. When evaluated on the in-domain synthetic test set (Table 3) or the MIT benchmark (PSNR 18.33 vs. 18.09), the gain from joint modeling may appear modest. However, its benefits are more evident in real-world out-of-domain scenarios. As shown in Fig. 5 and Supplementary Video (0:44–0:59), our joint modeling version effectively decomposes shadows and generalizes to challenging in-the-wild scenes. We will clarify this in the paper and add additional qualitative examples in the supplement. \n\n**Treating relit RGB as another “buffer”.** While relighting may appear structurally similar to intrinsic prediction, we empirically observed that simply adding relit RGB as another output buffer often leads to “shadow baking”, where shadows in the input are directly copied into the output. This motivates our design choice: we increase model capacity via token-wise concatenation and explicitly model the joint distribution of scene intrinsics and relit appearance. This encourages the model to disentangle shading effects and produce controllable, high-quality relighting.\n\n**Predicting more material properties.** We agree that this is a promising direction, and predicting additional properties could further improve the model’s understanding of scene structure and materials. Currently, memory requirements scale with the number of properties, making this expensive for high-resolution videos. We will further discuss it in the revision and leave it as a direction for future work. \n\n**Ray maps encoding.** Compared to downsampling, encoding the ray maps can preserve more details and align with other conditions in latent space during the denoising process.\n\n**Dataset usage.** We agree that constructing a more specific dataset, including semi-transparent or reflective objects, can help our model be more robust in these scenarios. Meanwhile, our auto-labeled data also helps our model generalize to more data domains, such as skin and fur (Fig. 4), which are all very challenging domains.\n\n**Dataset ablation.** We report an ablation on the auto-labeled dataset in Fig. 6 and Tab. 4. While we agree that a full dataset-level ablation would provide additional insight, it is currently prohibitive due to the high computational cost of training large-scale video diffusion models."}, {"review_id": "Q1PdEyc7lo", "reviewer": "Reviewer_a48D", "summary": "The paper introduces a generative relighting method that takes an input image or video alongside with a target environment lighting and produces relit images or videos. The key idea is that instead of doing intrinsic image decomposition first and then shynthesis, as usually done in the literature, the authors propose to model the the decomposition and relighting task jointly by predicted albedo and relighting at the same time. The model is fine-tuned on a set of datasets, such as synthetic relightings, following DiffusionRenderer, real multi-illumination dataset, and auto-labeled real-world dataset. \n\nComprehensive experiments validate the idea and show impressive, extremely realistic results.", "questions": "## Method\n* The key insight of the paper is that jointly training for albedo estimation improves the relighting quality. Would it further improve if also training for further material parameters, such as roughness, metallic, or for lighting maps, as shading map, maybe even for geometric maps, as normal/depth? Also, is this improvement bi-directional, i.e. the albedo estimation also gets improved?\n* As Table 3 also shows, the model can utilize albedo information as conditioning. Would it be possible to use this conditioning for material editing applications?\n\n## Related works\n* Please double-check the references and cite the conferences instead of arxiv, where possible (e.g. 32 was published at CVPR2024). \n\n## Writing\n* Typo: L.284: performs *better* or ...", "limitations": "yes", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths\n* **S1** Very impressive results. The proposed method seems to achieve emerging understanding of more complex reflectance and refraction properties beyond the albedo (as also seen in Figure 3 and the bottle in row 3). \n* **S2** Well-written, easy to follow. \n* **S3** Clear and logical morivation, provides valuable insight for the community. \n\n\n## Weaknesses\n* **W1 - Key claim not ablated** One of the key claims of the paper is that joint modelling is beneficial for relighting; better than two-stage inverse+forward rendering. However, there is no ablation in the paper, which underlines this claim. The authors compare against DiffusionRenderer, which indeed follows the usual inverse and forward rendering pipeline; however, DiffusionRenderer was trained on a different set of datasets (did not use the MIT multi-illumination dataset and smaller auto-labeled real dataset). Therefore, it is not clear, whether the improvement indeed comes from the joint modelling, or rather from the better training data. The paper provides comparisons against the single-stage, direct relighting, but not against a variant, where first and image-conditional model estimates the albedo, then another image+albedo+lighting conditional model generates the relit images.", "was_revised": true, "final_justification": "The paper introduces a novel idea of jointly modelling intrinsic image decomposition and relighting, which can provide valuable insight to the community.", "rebuttal": "We thank the reviewer for their valuable feedback, and appreciate that they recognize our contributions. We respond to the comments and questions below.\n\n**Joint modeling ablation.** Our joint modeling formulation encourages the model to learn internal representations of scene structure, which improves generalization to unseen domains and mitigates shadow baking artifacts. We include a quantitative ablation in Table 3, and qualitatively demonstrate in Fig. 5 and Supplementary Video (0:44–0:59) that joint modeling effectively decomposes shadows and generalizes to challenging in-the-wild scenes. \n\n**Comparison with DiffusionRenderer.** We re-implemented DiffusionRenderer using the same Cosmos backbone and trained it on the same synthetic and auto-labeled real-world datasets, denoted as DiffusionRenderer (Cosmos). DiffusionRenderer decomposes relighting into two stages: an inverse rendering step that estimates scene G-buffers, followed by a forward rendering step that synthesizes relit images conditioned on these estimates. This design introduces fundamental limitations: it is sensitive to errors in the predicted G-buffers (e.g., roughness) and struggles to capture complex material properties not encoded by G-buffers, such as transparency. We present quantitative comparisons in Tab. 1 and qualitative results in Supp. Fig. 2, showing that our joint modeling yields higher quality and greater flexibility when learning from diverse supervision sources. \n\n**Comparison with two-stages model.** We experimented with an “image+albedo+lighting-conditional relighting model”. In practice, low-capacity versions of this design often either bias toward “flat” outputs resembling the input albedo, or suffer from “shadow baking” where shadows in the input image are preserved in the output. These limitations motivated our design: we increase model capacity via token-wise concatenation and explicitly model the joint distribution of scene intrinsics and relit appearance. Our single-stage formulation also enables faster inference compared to two-stage pipelines.\n\n**Predicting more material properties.** We agree that this is a promising direction, and predicting additional properties could further improve the model’s understanding of scene structure and materials. Currently, memory requirements scale with the number of properties, making this expensive for high-resolution videos. We will further discuss it in the revision and leave it as a direction for future work."}, {"review_id": "TiIFixptys", "reviewer": "Reviewer_PiUf", "summary": "The paper performs relighting by leveraging the implicit lighting knowledge captured in a video diffusion model. The key insight in this paper is borrowed from similar ideas in the past which performed joint estimation of different modalities (i.e. curvature, depth, semantics) to constrain the optimization of the diffusion model weights. The method trains using the known diffusion model cost functions with the extra albedo regularization term. Results exhibit convincing results on both qualitative (user survey) and quantitive.", "questions": "See weaknesses", "limitations": "Authors have provided a limitations section.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strength:\n- Method is intuitive, simple and effective. Borrowing from joint-estimation approaches that are well studied and shown to work. \n- Paper well organized with good set of experiments.\n\nWeaknesses:\n- One of the main concerns is while results are qualitatively good (seen in the supplemented video), no comparisons were made with classical inverse rendering approaches. Hallucinations of shadows which might appear convincing to the naked eye will theoretically score lower against classical accurate methods.\n- Not a weakness per-se, and I'm also not asking for an experiment. But I am curious how much effect the video diffusion model prior had on obtaining these results vs. a image based diffusion model.", "was_revised": true, "final_justification": "The paper shows how video diffusion priors could be leveraged for re-lighting. The community will benefit from this contribution.", "rebuttal": "We thank the reviewer for their valuable feedback, and appreciate that they recognize our contributions. We respond to the comments and questions below.\n\n**Comparison with classical approaches.** We agree that classical physically based rendering (PBR) methods can achieve high accuracy when high-quality 3D geometry and materials are available. However, in the absence of carefully authored scene content, methods like screen-space ray tracing (SSRT) often struggle to produce artifact-free shadows and reflections, as also noted in DiffusionRenderer [39].\n\nIn this work, we focus on enabling high-quality relighting directly from video without requiring full scene reconstruction. We agree that integrating our generative approach with explicit 3D information—such as traced shadow buffers—is a promising direction for improving physical accuracy, and we will discuss this in the revision as future work. \n\n**Video diffusion model vs. Image diffusion model.** Video diffusion models have better capacities to model dynamic scenes and ensure temporally consistent results. This is especially important for effects like reflections and shadows, which may otherwise flicker, pop in/out or change shape abruptly when occluders enter or leave the frame. Please refer to our qualitative comparisons in the supplementary video, where we also include results from image-based baselines for comparison."}]

Author rebuttal:

Meta-review: The reviewers all appreciate the contributions of this work and recommend acceptance. The AC and SAC also found the joint denoising of scene albedos and relit outputs to be a significant contribution that overcomes the limitations of previous approaches and produces compelling results. A decision was therefore made to accept this paper for spotlight presentation.
Decision: Accept (spotlight)