LuxDiT: Lighting Estimation with Video Diffusion Transformer
Authors: Ruofan Liang
Kai He
Zan Gojcic
Igor Gilitschenski
Sanja Fidler
Nandita Vijaykumar
Zian Wang
Venue: NeurIPS 2025 poster ()
Keywords: Lighting estimation
Inverse rendering
Video diffusion models
Primary area: applications

Abstract: Estimating scene lighting from a single image or video remains a longstanding challenge in computer vision and graphics. Learning-based approaches are constrained by the scarcity of ground-truth HDR environment maps, which are expensive to capture and limited in diversity. While recent generative models offer strong priors for image synthesis, lighting estimation remains difficult due to its reliance on indirect visual cues, the need to infer global (non-local) context, and the recovery of high-dynamic-range outputs. We propose LuxDiT, a novel data-driven approach that fine-tunes a video diffusion transformer to generate HDR environment maps conditioned on visual input. Trained on a large synthetic dataset with diverse lighting conditions, our model learns to infer illumination from indirect visual cues and generalizes effectively to real-world scenes. To improve semantic alignment between the input and the predicted environment map, we introduce a low-rank adaptation finetuning strategy using a collected dataset of HDR panoramas. Our method produces accurate lighting predictions with realistic angular high-frequency details, outperforming existing state-of-the-art techniques in both quantitative and qualitative evaluations.

LuxDiT: Lighting Estimation with Video Diffusion Transformer
Ruofan Liang" 1,,3+
Kai Hel.2,3
Zan Gojeic'
Igor Gilitschenski
Sanja Fidler 12,3
Nandita Vijaykumar2.0
Zian Wang
1.2.3 1,2.3
NVIDIA
University of Toronto
Vectoo Institute
Abstract
Estimating scene lighting from a single image or video remains longstand- ing challenge in computer vision and graphics. Learning-bass based approaches constrained by the scarcity of ground-truth HDR environment maps, which are expensive capture and limited in diversity. While recent generative models offer strong priors for image synthesis, lighting estimation remains difficult due its reliance indirect visual cues, the need to infer global (non-local) con- text, and the recovery of high-dynamic-rang outputs. We propose LuxDiT, a novel data-driven approach that fine-tunes video diffusion transformer gen- erate HDR environment maps conditioned on visual input. Trained on large synthetic dataset with diverse lighting conditions, our model learns to infer il- lumination from indirect visual cues and generalizes effectively to real-world scenes. To improve semantic aligmment between the input and the predicted en- vironment map, we introduce low-rank adaptation finetuning strategy using a collected dataset of HDR panoramas. Our method produces accurate lighting predictions with realistic angular high-frequency details, outperforming existing state-of-the-art techniques both quantitative and qualitative evaluations. Project page: attps://reseds nvidia. com/labs//////eoo tataasttoooooooooo paronto-ai/Luxiii ai/LuxDiT/
Introduction
In physically-based rendering, lighting plays central role in shaping the appearance-hov objects cast shadows, reflect, and appear integrated within a scene. From virtual object insertion and augmented reality synthetic data generation, many downstream tasks rely estimating scene illumination. Yet inferring lighting from casually captured images or video remains an open challenge.
A common representation of the scene illumination is the high-dynamic-range (HDR) environment map, which describes incoming light intensity from all directions. HDR maps can acquired by using light probes multi-exposure panoramas, requiring specialized setups that are impractical for everyday use [9]. To overcome this, several learning-based methods that estimate environment maps directly from casually captured LDR images videos have been proposed [15, 16, 33, 74]. However, these methods typically depend on paired datasets of input images or videos and HDR environment maps, leading to chicken laakeen--n---eee and-egg problem: large collection of HDR environment maps is needed to train a model that aims to alleviate the need for acquiring such expensive data in the first place.
Recently, generative diffusion models have demonstrated strong capabilities in modeling complex image distributions. Diffusionl light [45] demonstrated that pretrained text-to-image models encode implicit knowledge of illumination, which can be cleverly extracted by inpainting virtual chrome ball into an image, generating plausible appearances under varying exposure settings. However,
Work done during internship at Nvidia
Joint Advising
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Input Image
Lighting Estimation
Virtual Object Insertion
Figure 1: LuxDiT is generative lighting estimation model that predicts high-quality HDR envi- romment maps from visual input. produces accurate lighting while preserving scene semanties, enabling realistic virtual object insertion under diverse conditions.
without task-specific fine-tuning, the inpainting priors of pre-trained diffusion models are insufficient for producing reliable lighting estimates in single inference and cannot directly generate HDR outputs. As a result, DiffusionL.ight relies on an expensive test-time ensemble strategy to improve robustness. Moreover, sampling multiple exposures through separate inference passes introduces inconsistencies and limits the dynamic range of the reconstructed illumination.
conditional generative task and propose LuxDiT, a neural lighting predictor trained on synthetic data and adapted to real-world scenes. Conditioned on visual input, approach fine-tunes a diffusion transformer (DiT) to synthesize HDR panoramas from noise. Unlike pixel-aligned tasks, lighting estimation requires global reasoning over scene context. DiTs are particularly suited to this task: their attention- architecture supports global context aggregation, and their generative priors facilitate reasoning from indirect cues such as shading and reflections.
In this work, we formulate lighting estimation
Training such a model requires diverse lighting data. overcome the lack of real-world HDR lighting supervision, we construct a large-scale synthetic dataset with randomized geometry, materials, and lighting conditions. Training on this dataset allows the model to learn physically grounded cues for light direction and intensity. While this imparts general lighting priors, models trained purely on synthetic data often hallucinate lighting based on dataset priors, producing environment maps that plausible but semantically mismatched with the input scene. For example, image of an urban street may yield an environment map depicting a rural landscape. To address this, we further apply low-rank adaptation (LoRA) [24] on curated set of real HDR panoramas, improving alignment between predicted lighting and scene semantics.
Given a single image or video, LuxDiT produces HDR environment maps with accurate direction, intensity, and scene-consistent content. reduces lighting estimation error by 45% on Laval Outdoor sunlight direction and improves temporal consistency for video input, enabling reliable use in downstream applications such as virtual object insertion. Our main contributions are:
A DiT-based generative architecture that synthesizes HDR environment maps from visual input.
A LoRA-based fine-tuning strategy using curated HDR panoramas to improve semantic alignment between the input scene and predicted illumination.
A large-scale synthetic dataset with randomized geometry, materials, and lighting.
Related Work
Lighting estimation aims to infer environment illumination from input imagery, and is critical for photorealistic rendering and virtual object insertion. Early learning-basee methods treat lighting estimation as supervised regression problem, predicting spherical lobes [16, 33, 72, 69], parametric sources [66, 14], or low-resolution environment maps [15, 50, 74, 52] directly from a single image.
These models are trained on paired data obtained from real-worle captures [15, 50, 58] synthetic

---

rendering [33, 74, 52]. However, their performance often degrades in complex, in-the-wild scenes due to limited diversity in the training data.
Recent methods incorporate generative priors to address the ambiguity of scene illumination. Style- Light [55] fine-tunes StyleGAN to generate L.RR and HDR panoramas from latent codes, using GAN inversion at test time. However, its performance hinges on inversion quality and often breaks semantic alignment on out-of-domain inputs. EverLight [8] regresses a parametric lighting estimate and refines it with a GAN to add high-frequency detail, but relies on pseudo-labeled HDR data and struggles with complex or bright lighting. DiffusionLightt [45] uses a diffusion model to inpaint a virtual chrome ball under multiple exposures, merging them into an HDR map. While visually plausible, this multi multi-stage process yields distorted panoramas and limited dynamic range.
Inverse rendering recovers scene properties such as geometry, material reflectance, and illumination from image observations. Lighting estimation is often treated as a subcomponent of this broader task, with prior work jointly estimating lighting alongside depth, normals, and albedo. Learning-baa based approaches [49, 33, 59] typically leverage physies-based constraints and use re-rendering losses to supervise predictions. However, these methods often assume simplified reflectance models such as Lambertian shading, which limits their ability handle complex lighting effects.
Optimization based methods leverage differentiable rendering [4, 71, 70, 6, 60, 42, 18, 34] to jointly optimize lighting parameters and other scene attributes through photometric losses and regularization terms. Some approaches [31] follow a deccompsitiion cetpiionntheenoooiiiittttt strategy: estimating geometry and albedo first, then solving for lighting via optimization. Other works also explore priors from proxy geometry [65] or pretrained general models [38, 36, 43]. The optimization-basee pipelines often require dense multi view captures known proxy geometry, and involve expensive test-time optimization procedures. In contrast, our method directly predicts HDR illumination in a feed-forward manner without requiring scene geometry or iterative inference.
Diffusion model priors. Diffusion models (DMs) have emerged as a powerful class of generative models in high-fidelity image [46, 2, 47, 7] and video synthesis [22, 73, 3, 63, 1]. Beyond generation, pretrained DMs have been adapted to perception tasks through task task-specific finetuning on carefully curated datasets [62, 39, 19]. showing strong results on spatially aligned predictions such as depth [30, 25, 29], surface normals [13, 64, 35), albedo [11, 31, 68, 35], and material properties [31, 68, 35, 43]. Adapting DMs non-local tasks like lighting introduces new modeling challenges, as outputs such as HDR panoramas are not apatially-alignel with the input.
3 Preliminaries: Diffusion Models
Diffusion models learn to approximate data distribution (x) through iterative denoising. Fol- lowing DDPM [21], forward process progressively adds Gaussian noise to data sample XØ ~ Pdata- producing noisy version at timestep t E [1, T) as: x, = Vāzxo + VI- ājc, where ~ (O, I) and 0g defines the noise schedule. During training, a neural network 14g learns to reverse this process by minimizing:
E Xxo- (x). ((. [x]xo(x,, v1
(1)
where e represents optional conditioning inputs. The denoising target y varies by formulation, and can be the noise [21], the v-prediction Vaze - vI VI- ax [48], or the clean signal Xa itself [28]. At inference time, samples are generated by denoising an initial Gaussian sample through a fixed number of reverse steps. In this paper, build on Cog VideoX [63], latent video diffusion model trained on compressed video representations. A pretrained auto-encoder pair {E, D) maps RGB videos to and from a latent space, such that (x)) = z and D(z) s x. All diffusion training and generation is performed in this lower oower-dimensional latent space to reduce memory and computational.
4 Method
We propose LuxDiT, a diffusion-based generative framework for estimating high-dynamic-range (HDR) environment maps from a single image or video. We tailor a recent video diffusion transformer architecture [63] for lighting estimation, by jointly processing denoising targets (environment lighting) and condition tokens (LDR input images) through self-attention layers. Since a single image can be treated as a one- me-frame video, refer to both inputs uniformly as input video in the remainder of this section. An overview of the architecture is shown in Figure 2. In the following sections, we describe the model design, data sources, and training procedure.
3

---

Input Env
Pred Eav
Imput Video
Fdtr
Diftision
Tokes concal
HDR
Batr
Stage
MLP
HDDR Dav
Chirne
Stage
LoRA
Frozen
Exs
Figure 2: Method Overview. Given an input image or video LuxDiT predicts an environment map two tone- aaee-apeed representations, Ejar and Elgg, guided by a directional map Eúir. Environment maps are encoded with a VAF, and the resulting latents are concatenated and jointly processed with visual input by DiT. The outputs Eaa: and Elug are decoded and fused by lightweight ML.P reconstruct the final HDR panorama.
4.1 Model Design
We formulate HDR enviromment map estimation as conditional denoising task. Given an input video I R_xHxWs x3 with frames, the model generates a corresponding sequence 360 HDR panoramas E IR2XH,W,X33,
Two core challenges arise: (1) standard VAEs used latent diffusion models are trained on LDR images and cannot faithfully encode HDR content, and (2) the output panoramas are not spatially aligned with the input, requiring flexible conditioning mechanisms. We address these challenges using a mual-tonemapping HDR representation, token-based conditioning, and unified transformer architecture that jointly denoises two latent representations lighting.
HDR lighting representation. Realistic lighting involves high-in -igh-intensity components such as the sun or artificial sources, with radiance values often exceeding 100 or 1,000. Representing this range in latent space is non trivirivial: standard VAEs are trained on [0, 1] -o-rmaallee LDR images and cannot reconstruct such dynamic content, and retraining on HDR data is impractical due to data scarcity
Inspired by prior works [27, 35), we represent each HDR panorama E using two complementary tonemapped representations:
log(1 + E) Eing
Eidr
+
M2
log(1 1 + Mogg)
where Ejur is standard Reinhard tonemapping and Ejog captures normalized log- intensity. We set Mar 16 and Mog = 10,000. Both outputs are clipped to j0, 1] before VAE encoding.
At inference time, the HDR environment map is reconstructed using lightweight MLP sb:
E
EEary Eing)
Diffusion latents. Our model builds on transformer- diffusion model μ4s, adapted predict HDR environment maps from visual input. The model operates in latent space and jointly denoises two tonemapped representations of the HDR lighting.
The tonemapped inputs Eddr and Etog are encoded by the pretrained VAE into latent tensors [z _ with shape as IR x xC' These are concatenated along the channel dimension to form the diffusion Rlxhex x x2C The input and output projection layers of the diffusion network μ4 are extended to accommodate the increased channel dimension.
target z ldr z lo8] R
Conditioning visual input in DIT. Accurate lighting estimation requires the model extract fine- grained shading cues from the input image, such as shadow orientation, surface reflections, and specular highlights. Unlike pixel-aligned image-to-image translation tasks, empirically observe that concatenating conditions to the noisy latents leads to poor performance (see Table 7), indicating the need for more flexible conditioning mechanism.
To this end, we adopt a fully attention-based architecture for the input video conditions. Specifically, encode the input video I IR Hxwx3 into a latent tensor E(I) [ixxxxxX using the pre- trained VAE encoder, and flatten it into token sequence
R To help the model distinguish between condition tokens and denoising targets, we apply separate adaptive layer normalization (AdaLN) modules [44, 63] to each token type at every transformer block.

---

Directional embedding. To improve angular continuity in the predicted panoramas, we inject directional information into the model. Specifically, we construct direction map of unit vectors Eair that encodes per-pixel lighting directions in the camera coordinate system. This map is passed through the same VAE encoder &, then projected and fused into the noise tokens using channel-wise concatenation before the transformer blocks. During training, we apply random horizontal rotations to Edir to encourage rotational equi-variance and robust directional encoding.
Conditioned denoising process. To put it together, at each denoising timestep t, the model receives noisy latent
|z,يا los] and predicts the corresponding clean latents conditioned on visual input as pa (2:; c, l). This transformer-based design allows the model to propagate indirect lighting cues-such as shadows and reflections- eefeccions-throuugh global self-attention, enabling lighting prediction that is both scene-consistent and directionally accurate.
4.2 Data Strategy
Supervised training of our model requires paired data in the form (I, Endr Elog), where I is LDR input and Eut: Eing are tonemapped versions of the target HDR environment map. To overcome the scareity of real-world HDR annotations, we leverage three complementary data sources: synthetic renderings, HDR panorama images, and LDR panoramic videos.
Synthetic rendering data. To supervise lighting prediction using physically accurate visual cues, we generate synthetic data by rendering randomized 3D scenes lit by HDR environment maps. Each scene consists of (i) ground plane with randomly assigned PBR materials, (ii) 3D objects sampled from Objaverse [10], and (iii) simple geometric primitives such as spheres, cubes, and cylinders with varied materials. We render multiple frames per scene with randomized camera trajectories and environment map rotations. Despite their simplicity. these scenes exhibit diverse lighting effects, including cast shadows, specular highlights, and reflectofss, all paired with ground truth HDR illumination. Empirically, we find this data is critical for enabling the model to learn accurate shading cues and light-source location (see Table 7).
HDR panorama images. We generate training pairs by sampling perspective crops from HDR environment maps with data augmentation. Specifically, given panorama, we randomly sample camera parameters including azimuth, elevation, field of view, and exposure scale. These parameters define virtual pinhole camera, which we use to project the panorama into an LDR perspective view The corresponding HDR environment map serves as the ground truth lighting target E. To support temporal training, we extend this procedure generate multi-frama sequences by smoothly varying the camera pose over time.
LDR panorama videos. To enable the generation of dynamic panorama environment maps, we also incorporate training data from LDRR panoramic videos. Although ground-truth HDR environment maps are not available for this source, we use it in the form (I. Ejár- Ø). where Ejár is derived using tonemapping and Ø indicates the absence of log-space intensity. The panoramic vidco is projected into a perspective-view view video using randomized camera parameters, following the same procedure as above. Despite the lack of HDR intensity, this data improves robustness and temporal consistency by exposing the model to natural image statistics, motion patterns, and diverse real-world lighting conditions. We use 2,000 panoramic videos from the WEB360 dataset [57] for training, and hold out 114 videos for evaluation.
4.3 Training Scheme
We adopt a two-stage training strategy to progressively build the model's capacity and improve generalization. The first stage focuses learning physically grounded lighting cues from synthetic data. The second stage adapts the model to real-world distributions through LoRA-based fine-tuning.
Stage I: Synthetic supervised training. We begin by training the model on the synthetic rendering dataset described in Section 4.2. This stage enables the model to learn the fundamental relationship between image- mage-based shading cues and HDR environment lighting.
We follow the standard DDPM training objective [21] adopted by the Cog base model [63]:
(0) E=,, [||e ( c, t)]
(4)
where denotes the clean latent pair (x. _00)) and e is the conditioning latent from the input video. During training, we randomly drop either 2id or 2/18 with probability p = 0.1 to encourage robustness to missing tonemapped representations.
5

---

Table 1: Comparison of our method with baselines three benchmark datasets. The results are
reported in terms of scale-invariant RMSE, angular error, and normalized RMSE.
Scalc-invariant RMSE +
Angular Error
Normalized RMSE 4
Dataset
Method
Diffuse
Matte
Mirror
Diffuse
Matte
Mirror
Diffuse
Matte
Mirror
StyleLight
0.135
0.315
0,552
4.238
4.742
6.781
0.234
0.404
0.511
Laval Indoor
DiffusionLightt
0.124
0.325
0.597
2.500
3,421
5,936
0.216
0.361
0.431
Ours
0.112
0.297
0.586
2.555
3.526
5.641
0.196
0.341
0.457
H-G et al. [23]
0.300
0.437
0.587
7.851
8.755
26.052
0.551
0.627
0.740
NLFE DiffusionLight
0.112
0.234
0,431
4.804
5.279
7.278
0.217
0.331
0.496
Laval Outdoor
0.083
0.224
0.414
1.936
2.955
5.491
0.167
0.330
0,472
Ours
0.068
0.190
0.396
2.018
2.939
5.286
0.137
0.271
0.454
StyleLight
0.138
0.336
0.620
3.034
4.272
6.602
0.198
0.344
0.474
Poly Haven
NLFE
0.159
0.326
0.571
3.305
4.240
5.180
0.224
0.365
0.458
DiffusionLight
0.113
0.270
0.519
2.199
3.121
4.104
0.191
0.282
0.391
Ours
0.077
0.196
0.442
1,235
1.977
2.783
0.111
0.199
0.323
Table 2: Angular error on estimated peak luminance light direction on Laval Outdoor sunny scenes.
Table 3: Quantitative comparison with video input. Peak an- gular error (PAE) is used to evaluate PolyHaven- Peak videos. Angular error (AE) used to evaluate WEB360 LDR videos.
Method
Mean
Peak Angular Error
Median
Method
PAE Mcan
PolyHaven-Pead Peak
PAE Std
AE
WEB360
AE Std
H-G ct al. [23]
52.8
47.8
Diffusionl light
19.09
10.31
6.504
0.269
NLFE
52.9
Diffusionl Light
44.4
32.1
Ours (image)
5.74
3.68
5,679
0,382
Ours
23.7
17.5
Ours (video)
5.21
1.95
5,218
0.072
Stage II: Semantic adaptation. After base training, we fine-tune the model to improve semantic alignment between the input appearance and the predicted HDR environment map.
This stage uses real-world data sources, including perspective projections from HDR panoramas and LDR panoramic videos. Since HDR ground truth is not available in the latter, supervise only the tDDetonne component. To avoid overfitting and preserve the pretrained model capacity, apply parameter-efficiene LoRA fine-tuning [24], optimizing small set of injected low-rank parameters A# in the transformer layers:
(40) = E (.N0 [[|e [
(5)
5 Experiments
5.1 Experiment Settings
Implementation details. We use the pre-trained Cog [63] model as our backbone, All training is conducted on 16 NVIDIA A100 GPUs. Input resolutions are randomly sampled be- tween 512 5122 and 480 720, and output environment map resolutions are between 128 256 and 256x512. The image-based model trained with a batch size of 192 for 12,000 iterations. For video training, we use the same spatial resolutions and uniformly sample frame lengths from 9. 17, 25. The video model trained with an average batch size of 48 for an additional 12,000 iterations. LoRA modules are applied to all attention layers with rank of 64. We fine-tune the LoRA parameters for 5,000 iterations during the adaptation stage. Please refer to supplement for implementation details.
Datasets. We evaluate our method on the following three benchmark datasets, covering various indoor and outdoor scenes. 1) Laval Indoor [15]: We use the same set of 289 test HDRIs used by prior works [45, 55]; 2) Laval Outdoor [23]: We evaluate on 1 116 sunny HDR panoramas with concentrated sunlight selected from the original dataset; 3) Poly Haven [67]: We select 181 Poly Haven HDRIs not used during model training to evaluate performance across both indoor and outdoor scenes.
Metries. Following prior works [55, 45), we use three standard metrics for evaluating HDR lighting: scale-invariant root mean square eиror (si- (si-RMSE) [17], angular error in degrees [32], and normalized RMSE (n-RMSE) [45]. For scenes with concentrated sunlight, we additionally report peak angular error (PAE) [23, 58], which measures the angular deviation of the predicted peak light direction.
Baselines. For indoor scenes, we compare against DiffusionLight [45], StyleLight [55], Weber et al. [61], and EMLight [69], using metrics reported by [45] when applicable. For outdoor scenes, we compare against DiffusionLight [45], Hold- Soll-Geoffro al. [23], and NLFE [58].

---

Input
StyleLight
DiffusiooLight
Ours
GT
StyleLight
DiflusioLLght
Input Image
DittusioLLght
Input Image
DifluskoLLght
Input Image
GI
NLHE
DiftusioaLight
Ours
Input Image
GT
NLPE
Dirtuskol.ight
Curs
1012
Mi
dlED
Figure 3: Qualitative comparison with baseline methods on three benchmark datasets.
Reference
StyleLight
DiffusionLight
DiPIR
Ours
Figure Qualitative comparison of virtual object insertion.
Table 4: Ablation study impact LoRA scale at inference time.
Table 5: Ablation study on im- pact of camera selddof-vieww
Table 6: Ablation study on impact of camera elevation.
LoRA Scale Diffuse
Matte
Mirror
FOV Diffuse..
Matte
Mirror
Elevation
Diffuse
Mirror
0.00
2.98
5.02
1.29
2.14
2.95
-30
1.70
3.04
3,95
0.25
2.09
3.69
50'
1.26
2,06
2,86
-15*
1.22
2.05
2.87
0.50
1.52
2.66
3.56
60'
1.17
1.92
2.72
0"
1.17
1.92
2.72
0.75
2.05
2.88
70"
1.15
1.85
2.63
15a
1.28
2.09
2.94
1.00
1.17
1.92
2.72
75"
1.13
1,80
2,59
3.51
Table 7: Ablation study on model design choices and training data. spheres protocol.
We report the angular error with three-
Laval Indoor
Poly Haven
Settings
Diffuse | Matte Mirror
Diffuse
Matte J Mirrrr
Ours (channel concat.)
7.09
10.04
11.07
7.09
10.04
11.07
Ours (wfo synthetic data) Ours
4.50
5.14
6.96
1.48
2.08
2.86
2.56
3,53
5.64
1.23
1.98
2.78
5.2 Evaluation of Image I Lighting Estimation
We follow the evaluation protocol from prior work to render spheres with three representative materials (gray-diffuse, silver-matte, and mirror), using the estimated HDR environment map from the LDR input image [15, 55, 45].
7

---

Frame
Frame 12
Frame
Frame 0
Frame 12
Frame 24
Reference
Difusionk.ihh
Rederence
DiftusionLight
Our image
Ours video
Ours image
Ours video
Figure 5: Qualitative comparison of video lighting estimation.
Table reports quantitative comparisons three benchmarks spanning both indoor and outdoor scenes, On the Laval Indoor dataset, our method performs comparably or better than DiffusionLighh across most metries, despite not using Laval Indoor dataset during training. This dataset exhibits a noticeable shift in color and intensity distribution compared to our training set, and our strong performance demonstrates robust generalization.
From qualitative comparison shown in Figure 3, DiffusionLight can lose angular high-freue eighffreuency details from the input image due to its distorted representation. In contrast, our estimated environment maps can recover more high-frequency details while preserving accurate lighting.
On the Laval Outdoor and Poly Haven datasets with broader dynamic range, method consistently outperforms prior state-of-the-an aht methods. Hold-Geoffroy er al. [23] can estimate concentrated peak light source such as sunlight; however, its results do not adapt well to the details of the input image. NLFE [58] can estimate in-context environment maps, but it often fails to estimate accurate highlights. DiffusionLight performs better than other baselines, but due its limited dynamic range, struggles with outdoor high-intensity light sources.
To further assess directional accuracy, we evaluate the angular error of the peak luminance direction on subset the Laval Outdoor dataset containing direct sunlight. Table reports the mean and median peak angular errors. Our method reduces peak angular error by nearly 50% compared to Diffusionl igght confirming its advantage capturing accurate light direction-a critical factor for casting realistic shadows in downstream applications such object insertion.
5.3 Evaluation of Video Lighting Estimation
To evaluate lighting estimation accuracy and consistency video input, we construct two types of test sequences:
PolyHaven-Peaks We project 12 unseen Poly Haven panoramas (each with direct sunlight) into videos using smooth panning camera. This setting is used to evaluate peak angular error.
WEB360: We randomly select 12 LDR panoramic videos featuring dynamic content from WEB360 and render them into perspective views with fixed horizontal camera motion. This setting evaluates temporal consistency using chromatic angular error on rendered mirror spheres.
Each contains 12 videos resolution of 480 x 720 and length of 25 frames. To quantify temporal consistency, we compute the standard deviation (std) of per-frame emror metrics for each video clip, and average the results across the 12-video set.
We compare our video inference to two baselines: our own image-based inference (applied frame-by- frame) and DiffusionLight [45]. Table 3 reports the results. Our method outperforms DiffusionLight Comparing to Ours (image), video inference achieves higher accuracy and significantly lower temporal variance, indicating more stable predictions across time.
Figure shows qualitative examples of video inference. Both DiffusionLight and our image-based variant exhibit visible temporal flickering. In contrast, our method produces smooth lighting transi- tions, successfully aligning content across frames and preserving consistent lighting behavior over time.

---

5,4 Evaluation of Virtual Object Insertion
Virtual object insertion is a key downstream applica- tion of lighting estimation. We evaluate our method this task using the benchmark from [36], using 11 HDR panoramas from the Poly Haven dataset [67]. For each scene, a virtual object and a known ground plane are manually placed into the environment. Each test case includes an LDR background image rendered from the HDR panorama, along with posed object and ground plane. gpeudo-ground-rrgg truth object insertion is gen- erated by rendering the object using the original HDR environment map. This allows for controlled comparison against renderings produced using predicted lighting.
Table 8: Quantitative evaluation of virtual object insertion. report the percentage of images where users preferred Ours over base- lines. A preference > 50% indicates Ours outperforming basclines.
Method
RMSE
SSIM
Ours Preferred
StyleLight
0.056
0.986
60.6%
Diff
0.057
0.987
60.6%
DiPIR
0.048
0.989
54.5%
Ours
0.047
0.990
We report quantitative metrics in Table 8. In addition, we conduct a user study to assess perceptual quality (details provided in the supplement), and report the percentage of samples where users preferred our results over baseline methods.
Our method achieves visual quality comparable to DiPIR and significantly outperforms other baselines. Notably, DiPIR specialized for object insertion and incorporates additional modules for tone mapping and appearance harmonization. In contrast, our model estimates lighting alone, yet still produces realistic composite renderings. We include qualitative results in Figure 4.
5.5 Ablation Study
Model Design and Training Data We evaluate two model variants to ablate the contributions of our architectural and training design: (1) Channel concatenation: This variant fuses input and environment map (resized match input image) latents along the channel dimension [27], and no token-wiss concatenation is used. Our two-stage training is also applied. (2) Training withour synthetic data: This variant skips Stage training and uses only panorama crops for fine-tuning.
Table 7 reports angular errors on Laval Indoor and Poly Haven. Channel concatenation significantly underperforms, confirming the importance of token-level conditioning. Without synthetic pretraining, the model performs well in-domain (Poly Haven) but degrades out-of-domain (Laval Indoor), showing synthetic data pre- tre-trainin is crucial for learning generalized lighting priors.
LoRA scale. We vary the LoRA interpolation weight from 0.0 1.0 ablate how fine-tuned LoRA affects the predicted lighting content. Table 4 shows that higher LoRA weights yield lower angular error on Poly Haven, validating the effectiveness of LoRA for improving semantic alignment.
Camera sensitivity. We test robustness to camera variation by rendering crops from Poly Haven under varying field of view (459 to 75%) and camera elevation (-30" 30%). Results in Tables 5 and 6 show that while extreme viewpoints introduce mild error increases, performance remains stable, demonstrating robustness to moderate viewpoint shifts.
Discussion
We introduce LuxDiT, a conditional generative model for estimating HDR scene illumination from casually captured images and videos. Our approach fine-tunes a video diffusion transformer (DiT) to synthesize HDR environment maps, combining large-scale synthetic data for learning physically grounded priors with L-Rseb adaptation on real HDR panoramas to improve semantic alignment. Extensive experiments demonstrate that LuxDiT produces accurate, high-frequency, and scene- consistent lighting predictions from limited visual input.
Limitations and future work. While LuxDiT produces high-quality lighting predictions, inference remains computationally intensive due to the iterative nature of diffusion models, limiting its use in real-time applications. Future work could explore model distillation or more efficient architectures to accelerate inference. Additionally, the resolution of predicted panoramas is limited by data and training scale; generating high- righ-resolution outputs for immersive applications will require richer, more diverse HDR supervision. Looking ahead, with recent progress in joint generative modeling [5, 37], we see LuxDiT as a step toward unified inverse and forward rendering frameworks, complementing recent progress in neural forward rendering and G-buffer estimation [68, 35, 20]. Future directions include joint modeling or co-training of lighting, geometry, and material for general-purposs scene reconstruction and appearance synthesis.

---

Acknowledgments and Disclosure of Funding
The authors thank Jacob Munkberg, Jon Hasselgren, Jun Gao, and Huan Ling for their insightful discussions. This work was conducted at and supported by NVIDIA.
References
[1] Niket Agarwal, Arslan Ali, Maciej Bala, Yogesh Balaji, Erik Barker, Tiffany Cai, Prithvijit Chattopadhyay, Yongxin Chen, Yin Cui, Yifan Ding, et al. Cosmos world foundation model platform for physical ai. arXiv preprint arXiv:2501.0357 2025.
[2] Yogesh Balaji, Seungjun Nah. Xun Huang, Arash Vahdat, Jiaming Song, Qinsheng Zhang, Karsten Kreis, Miika Aittala, Timo Aila, Samuli Laine, Bryan Catanzaro, Tero Karras, and Ming- Yu Liu. eDiff-I: text-to-image diffusion models with ensemble of expert denoisers. arXiv preprint arXiv:2211 1..1111,1444 2022.
[3] Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Scung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High- reghureeoo video synthesis with latent diffusion models. In JEEE Conference Computer Vision and Pattern Recognition (CVPR), 2023.
[4] Mark Boss, Raphael Braun, Varun Jampani, Jonathan T. Barron, Ce Liu, and Hendrik P.A. Lensch. NeRD: neural reflectance decomposition from image collections. In ICCV, 2021.
[5] Hila Chefer, Uriel Singer, Amit Zohar, Yuval Kirstain, Adam Polyak, Yaniv Taigman, Lior Wolf, and Shelly Sheynin. VideoJAM: Joint appearance motion representations for enhanced motion generation in video models. arXiv: 2502.02492, 2025.
[6] Wenzheng Chen, Joey Litalien, Jun Gao, Zian Wang, Clement Fuji Tsang, Sameh Khalis, Or Litany, and Sanja Fidler. DIB-R++: Learning to predict lighting and material with a hybrid differentiable renderer. In NeurlPS, 2021.
[7] Xiaoliang Dai, Ji Hou, Chih- Yao Ma, Sam Tsai, Jialiang Wang, Rui Wang, Peizhao Zhang, Simon Vandenhende, Xiaofang Wang, Abhimanyu Dubey, et al. Emu: Enhancing image generation models using photogenic needles in haystack. arXiv preprint arXiv:2309. 15807, 2023.
[8] Mohammad Reza Karimi Dastjerdi, Jonathan Eisenmann. Yannick Hold Sold-Geeffroy, and Jean- Saan-François Lalonde. Everlight: Indoor-outdoor editable hdr lighting estimation. In Proceedings of the IEFE/CVF International Conference on Computer Vision (ICCV), pages 7420-7429, October 2023.
[9] Paul E. Debevec and Jitendra Malik. Recovering high dynamic range radiance maps from photographs. In Proceedings the 24th Annual Conference on Computer Graphics and Interactive Techniques, SIG- GRAPH '97, page 369 378, USA, 1997. ACM Press/Addison Wesley Publishing Co.
[10] Matt Deitke, Dustin Schwenk. Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: A universe of annotated 3D objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13142- 13153, 2023.
[11] Xiaodan Du, Nicholas Kolkin, Greg Shakhnarovich, and Anand Bhattad. Generative models: What do they know? do they know things? let's find out!, 2024.
[12] Egor Ershov, Alexey Savchik, Illya Semenkov, Nikola Banić, Alexander Belokopytov, Daria Senshina, Karlo Koščević, Marko Subašić, and Sven Lončarié. The cube++ illumination estimation dataset. IEEE access, 1:227511-227522 2020.
[13] Xiao Fu, Wei Yin, Mu Hu, Kaixuan Wang. Yuexin Ma, Ping Tan, Shaojie Shen, Dahua Lin, and Xiaoxiao Long. GeoWizard: unleashing the diffusion priors for 3D geometry estimation from a single image. In ECCV, 2024.
[14] Mare-André Gardner, Yannick Hold-Geoffroy Kalyan Sunkavalli, Christian Gagné, and lean-François Lalonde. Deep parametric indoor lighting estimation. In ICCV, pages 7175-7183, 2019.
[15] Mare-André Gardner, Kalyan Sunkavalli, Ersin Yumor, Xiaohui Shen, Emiliano Gambarctto, Christian Gagné, and Jean- Françoiss Lalonde. Learning to predict indoor illumination from a single image. arXiv preprint arXiv: 1704.00440000 00090, 2017.
[16] Mathieu Garon, Kalyan Sunkavalli, Sunil Hadap, Nathan Carr, and Jean- -rançoraççii Lalonde. Fast spatially- varying indoor lighting estimation. In CVPR, pages 6908-6911 2019.
10

---

[17] Roger Grosse, Micah K. Johnson, Edward H. Adelson, and William T. Freeman. Ground truth dataset and baseline evaluations for intrinsic image algorithms. In ICCV, pages 2335-2342. IEEE, 2009.
[18] Jon Hasselgren, Nikolai Hofmann, and Jacob Munkherg. Shape, light, and material decomposition from images using Monte Carlo rendering and denoising. arXiv:2206.03380, 2022.
[19] Jing Hc, Haodong Li, Wei Yin, Yixun Liang, Lcheng Li, Kuiqiang Zhou, Hongbo Zhang, Binghing Liu, and Ying-Cong Chen. Lotus: Diffusion-based visual foundation model for high-quality dense prediction. arXiv preprint arXivv2409.1 18 8824, 2024.
[20] Kai He, Ruofan Liang, Jacob Munkberg. Jon Hasselgren, Nandita Vijaykumar, Alexander Keller, Sanja Fidler, Igor Gilitschenski, Zan Gojeie, and Zian Wang. Unirelight: Learning joint decomposition and synthesis for video relighting. arXiv preprint arXiv:2506. 2025.
[21] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6851, 2020.
[22] Jonathan Ho, Tim Salimans, Alexey Gritsenko. William Chan, Mohammad Norouzi, and David J Fleet. Video diffusion models. arXiv:2204.03458, 2022.
[23] Yannick Hold-Geoffroy, Akshaya Athawale, and Jean-François Lalonde. Deep sky modeling for single image outdoor lighting estimation. In CVPR, pages 1927-6935, 2019.
[24] Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022.
[25] Wenho Hu, Xiangjun Gao, Xiaoyu Li, Sijie 7han, Xiandong Cun, Yong Zhang, Long Quan, and Ying Shan. Deptherafter: Generating consistent long depth sequences for open-world videns. In CVPR, 2025.
[26] Wenzel Jakob, Sébastien Speierer, Nicolas Roussel, Merlin Nimier-David, Delio Vicini, Tizian Zeltner, Baptiste Nicolet, Miguel Crespo, Vincent Leroy, and Ziyi Zhang. Mitsuba 3 renderer, 2022. https://mitsuba- renderer.org.
[27] Haian Jin, Yuan Li. Fujun Luan, Yuanbo Xiangli, Sai Bi, Kai Zhang, Zexiang Xu. Jin Sun, and Noah Snavely. Neural gaffer: Relighting any object via diffusion. In Advances in Neural Information Processing Systems, 2024.
[28] Tero Karras, Miik Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models. In Proc. NeurIPS, 2022.
[29] Bingxin Ke, Dominik Narnhofer, Shengyu Huang. Lei Ke, Torben Peters, Katerina Fragkiadaki, Anton Obukhov, and Konrad Schindler. Video depth without video models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2025.
[30] Bingxin Ke, Anton Obukhov, Shengyu Huang, Nando Metzger, Rodrigo Caye Daudt, and Konrad Schindler. Repurposing diffusion- image generators for monocular depth estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
[31] Peter Kocsis, Vincent Sitzmann, and Matthias Nießner. Intrinsic image diffusion for single-view material estimation. In arxiv, 2023.
[32] Chloc LeGendre, Wan-Chun Ma, Graham Fyffe, John Flynn, Laurent Charbonnel, Jay Busch, and Paul Debevee. Deeplight: Learning illumination for unconstrained mobile mixed reality. In CVPR, pages 5918-5928, 2019.
[33] Zhengqin Li, Mohammad Shafiei, Ravi Ramamoorthi, Kalyan Sunkavalli, and Manmohan Chandraker. Inverse rendering for complex indoor scenes: Shape, spatially-varying lighting and svbrdf from a single image. In CVPR, pages 2475-2484, 2020.
[34] Ruofan Liang, Huiting Chen, Chunlin Li, Fan Chen, Selvakumar Panneer, and Nandita Vijaykumar. Envidr: Implicit differentiable renderer with neural environment lighting. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 79-89, 2023.
[35] Ruofan Liang. Zan Gojeic, Huan Ling, Jacob Munkberg. Jon Hasselgren, Hhi-Hao Lin, Jun Gao, Alexander Keller, Nandita Vijaykumar, Sanja Fidler, and Zian Wang. Diffusionrendere: Neural inverse and forward rendering with video diffusion models. arXiv preprint arviv: 2501.18504 2025.
11

---

[36] Ruofan Liang, Zan Gojcie, Merlin Nimier-David. David Acuna, Nandita Vijaykumar, Sanja Fidler, and Zian Wang. Photorealistic object insertion with diffusion-guided inverse rendering. In ECCV, 2024.
[37]
Yuanxun Lu, Jingyang Zhang. Tian Fang. Jean-Daniel Nahmias, Yanghai Tsin, Long Quan, Xun Cao, Yao Yao, and Shiwei Li. Matrix3D: Large Photogrammetry Model All-in-One, 2025.
[38] Linjie Lyu, Ayush Tewari, Mare Habermann, Shumsuke Saito, Michael Zollhöfer. Thomas Leimkühler, and Christian Theobalt. Diffusion posterior illumination for ambiguity inverse rendering. ACM Transactions on Graphics, 42(6), 2023.
[39] Gonzalo Martin Garcia, Karim Abou Zeid, Christian Schmidtt Daan de Geus, Alexander Hermans, and Bastian Leibe. Fine-tuning image-conditional diffusion models is easier than you think. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 2025.
[40] Oscar Michel, Anand Bhattad, Eli VanderBilt, Ranjay Krishna, Aniruddha Kembhavi, and Tanmay Gupta. Object 3dit: Language-guided 3d-aware image editing. Advances in Neural Information Processing Systems, 36:3497-3516, 2023.
[41] Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. NeRF: representing scenes as neural radiance fields for view synthesis. arXiv preprint arXiv:2003.082 2020.
[42] Jacob Munkberg, Jon Hasselgren, Tianchang Shen, Jun Gao, Wenzheng Chen, Alex Evans, Thomas Müller. and Sanja Fidler. Extracting triangular 3D models, materials, and lighting from images. arXiv:2111.25031 2021.
[43] Jacob Munkberg, Zian Wang, Ruofan Liang. Tianchang Shen, and Jon Hasselgren. VideoMat: Extracting PBR Materials from Video Diffusion Models. In Eurographics Symposium on Rendering CGF Track, 2025.
[44] William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF international conference computer vision, pages 4195-4205, 2023.
[45] Pakkapon Phongthawee, Worameth Chinchuthakun, Nontaphat Sinsunthithet, Amit Raj. Varun Jampani, Pramook Khungurn, and Supasorn Suwajanakom. DiffusionLight: light probes for free by painting a chrome ball. In ArXiv, 2023.
[46] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[47] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily I. Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to- image diffusion models with deep language understanding. Advances in Neural Information Processing Systems. 35:36479-36494, 2022.
[48] Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29. 2022. OpenReview.net, 2022.
[49] Soumyadip Sengupta, Jinwei Gu, Kihwan Kim, Guilin Liu, David W. Jacobs, and Jan Kautz. Neural inverse rendering of an indoor seene from a single image. In ICCV, 2019.
[50] Shuran Song and Thomas Funkhouser. Neural illumination: Lighting prediction for indoor environments.
In CVPR, pages 6918-6926, 2019.
[51] Roman Suvorov, Elizaveta Logacheva, Anton Mashikhin, Anastasia Remizova, Arsenii Ashukha, Alek- sei Silvestrov, Naejin Kong, Harshith Goka, Kiwoong Park, and Victor Lempitsky. Resolution-robust large mask inpainting with fourier convolutions. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 2149-2159, 2022.
[52] Jiajun Tang, Yongjie Zhu, Haoyu Wang, Jun- Jun-Hoong Chan, Si Li, and Boxin Shi. Estimating spatially- varying lighting in urban scenes with disentangled representation. In ECCV, 2022.
[53] Benjamin Ummenhofer. Sanskar Agrawal, Rene Sepulveda, Yixing Lao, Kai Zhang, Tianhang Cheng. Stephan Richter. Shenlong Wang, and German Ros. Objects with lighting: A real-world dataset for evaluating reconstruction and rendering for object relighting. In 2024 International Conference on 3D Vision (3DV), pages 137-147. IEEE, 2024.
12

---

[54] Giuseppe Vecchio and Valentin Deschaintre, Matsymth: A modern pbr materials dataset. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024.
[55] Guangeong Wang, Yinuo Yang, Chen Change Lay, and Ziwei I Liu. Stylelight: Hdr panorama generation for lighting estimation and editing. In European Conference on Computer Vision (ECCV), 2022.
[56] Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang. Neus: Learning neural implicit surfaces by volume rendering for nulti-view reconstruction. NeurIPS, 2021.
[57] Qian Wang, Weiqi Li, Chong Mou, Xinhua Cheng, and Jian Zhang. 360dvd: Controllable panorama video generation with 360-degree video diffusion model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6913-6923, 2024.
[58] Zian Wang, Wenzheng Chen, David Acuna, Jan Kautz, and Sanja Fidler. Neural light field estimation for street scenes with differentiable virtual object insertion. In ECCV, 2022.
[59] Zian Wang, Jonah Philion, Sanja Fidler, and Jan Kautz, Learning indoor inverse rendering with 3D spatially-varying lighting. In ICCV, 2021.
[60] Zian Wang. Tianchang Shen, Jun Gao, Shengyu Huang, Jacob Munkberg, Jon Hasselgren, Zan Gojeic, Wenzheng Chen, and Sanja Fidler. Neural fields meet explicit geometric representations for inverse rendering of urban scenes. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2023.
[61] Henrique Weber, Mathicu Garon, and cean-François Lalonde. Editable indoor lighting estimation. In European Conference on Computer Vision, pages 677-692. Springer, 2022.
[62] Guangkai Xu, Yongtao Ge, Mingyu I Liu, Chengxiang Fan, Kangyang Xie, Zhiyue Zhao, Hao Chen, and Chunhua Shen. What matters when repurposing diffusion models for general dense perception tasks? arXiv preprint 4xXiv:2403006090 2024.
[63] Zhuayi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuamming Yang, Wenyi Hong, Xiaohan Zhang, Guanyu Feng, et al. Cogvideox: Text-to-video diffusion models with an expert transformer.
arXiv preprint arXiv:2408.06077, 2024.
[64] Chongjie Ye, Lingteng Qiu, Xiaodong Gu, Qi Zuo, Yushuang Wu, Zilong Dong, Liefeng Bo, Yuliang Xiu, and Xiaoguang Han. Stablenormal: Reducing diffusion variance for stable and sharp normal. ACM Transactions on Graphies. 2024.
[65] Hong-Xing Yu, Samir Agarwala, Charles Herrmann, Richard Szeliski, Noah Snavely, Jiajun Wu, and Deqing Sun. Accidental light probes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12521-12530, 2023.
[66] Ye Yu and William A. P. Smith. InverseRenderNet: learning single image inverse rendering. In CVPR, 2019.
[67] Greg Zaal and et al. Poly Haven - The Public 3D Asset Library, 2025.
[68] Zheng Zeng, Valentin Deschaintre, Iliyan Georgiev, Yannick Hold- nold-Geoffroy, Yiwei Hu, Fujun Luan, Ling- Qi Yan, and Miloš Hašan. + image decomposition and synthesis using material-and lighting-aware diffuusion models. In ACM SIGGRAPH 2024 Conference Papers, pages 1-11, 2024.
[69] Fangneng Zhan, Changgong Zhang, Yingchen Yu, Yuan Chang, Shijian Lu, Feiying Ma, and Xuansong Xic. Emlight: Lighting estimation via spherical distribution approximation. In Proceedings of the AAA! Conference on Artificial Intelligence, 2021.
[70] Kai Zhang, Fujun Luan, Qianqian Wang, Kavita Bala. and Noah Snavely. PhySG: Inverse rendering with spherical Gaussians for physics-based material editing and relighting. In CVPR, 2021.
[71] Yuanging Zhang, Jiaming Sun, Xingyi He, Huan Fu, Rongfei Jia, and Xiaowei Zhou. Modeling indirect illumination for inverse rendering. In CVPR, 2022.
[72] Yiqin Zhao and Tian Guo. Pointar: Efficient lighting estimation for mobile augmented reality. arXiv preprint arXiv:2004, 00006, 2020.
[73] Daquan Zhou, Weimin Wang, Hanshu Yan, Weiwei Ly, Yizhe Zhu, and Jiashi Feng. Magicvideo: Efficient video generation with latent diffusion models. arXiv preprint arXiv:2211.11015 2022.
[74] Yongjie Zhu, Yinda Zhang, Si Li, and Boxin Shi. Spatially-varying outdoor lighting estimation from intrinsics. In CVPR, 2021.
13

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justifiicton right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No]", ", it is perfectly acceptable to answer "[No] provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: The abstract and introduction clearly state our contributions, which are fully supported by the experiments and analysis presented.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
14

---

Justification:
We discuss limitations in Section 6.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approxinations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g. if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proof'
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer:
[NA]
Justification:
The paper does not contain theoretical results or formal proofs.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: We include all necessary details for reproduction the main paper and supplementary materials.
Guidelines:
15

---

The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: We plan to release the code and data upon acceptance. The internal guidelines of our institution prevent us from releasing code at this stage.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
cablic/guidess Sawiiddee/eoeSSumissiisii sts.bmmssssoooiccii for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: / mnips.cc/publitt gww.e// (CodeSubmiiii aasionoPoccy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
16

---

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
All relevant details provided in the paper and supplementary materials.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Due to the high computational cost of training large-scale video diffusion models, we did not perform multiple runs to report error bars.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula,
call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
Question:
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification:
We include sufficient information supplementary materials.
Guidelines:
The answer NA means that the paper does not include experiments.
17

---

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics attpss///ne /ttppsss///uuisss... cwcppblic//thiicooooooccccccccee hewth//eeeeheeedeeleeeeeeeeeeeee ines?
Answer: [Yes]
Justification: Our work adheres the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discussed societal impacts of our work in the supplementary material.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our work does not involve models or data with high risk of misuse.
18

---

Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Third- -arttpartyy assets are properly credited, and their licenses are respected.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of in the package should be provided. For popular datasets, paperswi weworsswihcodee thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re- re-packaged, both the original license and the license of the derived asset (if has changed) should provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: Our paper does not release new assets.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
19

---

Answer: [Yes]
We include the details of all crowdsourcing experiments in supplementary materials.
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [Yes]
Justification: An approval is obtained for all of our experiments.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: LLMs
used in the core methods of this research.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non non-standart components.
Please refer to our LLM policy (https : httpp::///nuurtte //euriiis.....// cc/Cont wwwnffeennee///n (aa.cceee///////// /LLM) for what should or should not be described.
20

---

Appendix
In the supplementary material, discuss the broader impact of project in Sec. A, and provide additional details for implementation and experiments in Sec. B. Sec. C provides additional quantita- tive and qualitative results. We refer the accompanied video for extended comparisons video lighting estimation.
A Broader Impact
We introduce LuxDiT, generative model for estimating high-dynamic-range (HDR) environment lighting from casually captured images and videos. Lighting estimation is core challenge in photorealistic rendering due its non-local and indirect nature. LuxDiT produces scene-consistent HDR panoramas, enabling applications in virtual object insertion, relighting, AR/VR, and visual effects. can also support synthetic data generation for downstream tasks in roboties and perception, where realistic illumination is critical.
Similar other generative methods, LuxDiT could be misused to produce visually convincing but deceptive content. While it does not directly generate synthetic scenes, it enables realistic virtual object insertion and may facilitate the creation of manipulated imagery that difficult to distinguish from real footage. We encourage responsible use of LuxDiT and caution against its deployment in contexts where synthetic content could mislead viewers or undermine public trust, such as misinformation or falsified media.
B
Additional Details
B.1 HDR Reconstruction
Section 4.1 describes our method for reconstructing HDR environment maps from two tone-mapped LDR images using a lightweight MLP 2/ (Enr,, Elog). This MLP consists of 5 layers with 64 hidden units per layer and LeakyReLU activation. softplus activation applied to the final output layer to ensure non-negative outputs.
The MI.P 2a operates on a per-pixel basis: it takes pair of LDR RGB values as input and predicts single HDR RGB value. It is trained using the same HDR environment maps as the diffusion model, with augmentations including random intensity rescaling and exposure adjustments for diversity. To simulate limited input precision, LDR inputs are randomly quantized to 8-bit RGB values. We train MLP using a Huber loss with å = 1.0, which provides robustness against large HDR outliers while preserving smooth gradients.
Additionally, we show the tone-mapping curves used generate the LDR images in Fig. 6. Our dual-tone mapping strategy ensures sufficient sampling across the full dynamic range [0, 10,000). supporting accurate HDR reconstruction.
1.0
140
0.8
120-
0.6
1108
10
0.44
40
Fas
0.0
36
100
1000
10000
20. 9-th Purcentiee Intensity Scale)
Figure 6: The two tone- mapping curves used to gen- erate the LDR images. The 128 dot points along the curve are evenly spaced along (0, 1] LDR value range.
Figure 7: The histogram 99,9-th percentile intensity of all HDR environment maps in our training set.
21

---

B.2 Datasets
We provide more details about the datasets used in our experiments.
The data sources of HDR environment maps. We collected 2386 HDR environment maps from the following 4 data sources either publicly available commercially available.
Poly Haven": : 626 HDR environment maps with a wide range of indoor and outdoor lighting.
HDR Maps': 403 HDR environment maps with diverse lighting conditions, including 294 panorama maps and 109 hemi-sphere sky maps.
HDRI Skies 457 HDR environment maps with outdoor lighting conditions.
DOSCH DESIGN 900 HDR environment maps mainly for outdoor lighting conditions.
Figure 7 shows the histogram of the 99.9-th percentile intensity of all HDR environment maps in training set. With over 50% of the HDR environment maps having 99.9-th percentile intensity greater than 2.93. Note that the for outdoor lighting, the highest intensity can be orders of magnitude higher than the 99.9-th percentile. Among these, Poly Haven and HDR Maps offer greater diversity across scene types. To balance the training distribution across data sources, we apply sampling weights in the ratio 3:2:2:1 in the order listed above.
For quantitative and qualitative evaluation, we use the Laval Indoor and Laval Outdoor datasets, which contain calibrated HDR panoramas of real-world indoor and outdoor scenes.
Synthetic rendering data. Similar OBJect [40] and DiffusionnRendere [35], we create synthetic 3D scenes by compositing multiple 3D objects from OB- javerse [10] and randomly placing them a plane with varying plane textures. We use filtered sub- set of Objaverse, containing ~269,000 3D objects with decent geometries and material textures, to cre- ate synthetic 3D scenes. The varying plane textures are sampled from ~4000 PBR textures from Mat- Synth [54]. Each composited scene contains up to sampled Objaverse objects. We additionally add up to 3 random geometry primitives (sphere, cube, and cylinder) with varying material textures to provide rich shading cues for model to learn. For each scene, we randomly render 1~4 video clips with varying camera motions (e.g., orbiting camera and oscilating camera) and environment lightings. We use path-tracing renderer with 128 samples per pixel (spp) and the default OptiX denoiser render the video clips with resolution of 480 720 512 512. The HDR rendering results are tone-mapped to LDR images using Blender's AgX tonemapping" In total, we created ~ 19,,000 random synthetic scenes, resulting in 260,,00 video clips with least 16 frames per video clip.
Figure 8: Randomly sampled example images from our synthetic rendering data.
Perspective crops of HDR panorama images. We use subset of 1251 HDR panoramas with meaningful contents from Poly Haven, HDR Maps, and HDRI Skies for the training with perspective crops. Instead of pre-processing the perspective crops from the HDR panoramas, crops -nythe-fll during the training. The projection camera's 's azimuth angle is randomly sampled from [0, 360 and the elevation angle is randomly sampled from -00" to 10?, The camera's 's field of view (FOV) is randomly sampled from 45 to 80 The perspective crops are rendered with a resolution of 480 720. A random tone-mapping function is applied perspective projection crops
do the perspective
"nttps://polyhawee com/ /nttpp://hdrmaps. com/
hnttpe://hdri-skies com/
"https://doschhess/ com/
'nttp://hdrdb. com.irdoo.commindor//
'http://hdzzs htw./atddb..cmmuuuuoooro..
unttps://uugingtter co/
'https://ww/ httt::::/www.bledde. blender org/
22

---

to generate LDR images. The one-mappings include ACES, Filmic. AgX, and Gamma-2.. 2.4 sRGB mappings. Auto-exposure (i.e., remapping the 99-th percentile intensity to 0.9) also randomly applied to the LDR crops. For video input, we create trajectories of projection cameras by smoothly rotating the camera angle within an angular cone of 15".
Perspective crops of LDR panorama videos. Similar to the perspective crops of HDR panorama images. We on-the-fly sample perspective crops from the LDR panorama videos. Due to the lack of HDR content, we only apply a random auto-exposure tone-mapping to the perspective crops.
B.3 Model Details and Initialization
LuxDiT' is fine-tuned from the pre-trained Cog VideoX-5b--22v' To adapt this model for our task, replace the original text token with an image input token. This image token generated in the same manner as the environment map noise token, but without adding noise. We reuse the model's existing text-po processnnn layers (e.g.. AdaLN) to process these new image input tokens. Furthermore, we extend the input projection layer to incorporate additional conditioning channels derived from the concatenated noise token; these extended channels are initialized to zero. Similarly, the output projection layer is extended to predict dual tone-mapped environment tokens, with its newly added channels initialized from the original model's weights.
B.4 User Study Details for Virtual Object Insertion
Following prior works [14, 16. 15, 58, 36], we conduct user study on Amazon Mechanical Turk to compare our method against baseline approaches in terms of perceptual realism for virtual object insertion. Each participant is shown pair of rendered results one from our method and one from baseline-aa and asked to assess lighting realism, focusing on shadows, reflections, and overall visual integration.
The specific instructions shown to participants are:
Instruction: Find the inserted virtual object, look the difference, and select the more realistic image.
An AI system trying insert a virtual object into an image natural way. It aims to make the virtual object look as if it is part of the scene. There are two results: Trial and Trial B, and the virtual object located the center of each image. Please zoom in to compare the differences between the two images, and pay attention to the lighting effects such as the reflections and shadows.
Which one looks more realistic?
D
B
Participants are required to use a monitor 24 inches or larger. Image pairs are randomly shuffled to prevent bias. Following [36], repeat the user study three times, and recruited 11 unique participants for each experiment. We compute the percentage of images for which users preferred our method the haseline, and report the average user preferences for three repeated experiments. In total, the study includes 11 x 3 11 x 3 = 1089 individual comparisons.
B.5 Three-sphere Evaluation Protocol
We adopt the three-sphere rendering setting described in StyleL.ight [55], with evaluation scripts provided by DiffusionL.ghtt..
For the Laval Indoor dataset, we use the same set HDR environment maps and corresponding perspective crops DiffusionLight. We resize and crop the input image to 480 x 720 for our model. For Laval Outdoor and Poly Haven environment maps, we generate perspective crops using fixed horizontal camera with 60' field of view and resolution of 480 720. For Laval Outdoor, apply auto-exp by scaling the 50th percentile intensity to 0.5.
23

---

C Additional Experiments
C.1 Array-of-Spheres Evaluation
Following prior work [61, 8]. we evaluate our method using the urray-of-spheres protocol, which renders grid of diffuse spheres on ground plane using the predicted environment map.
Table 9: Scores on indoor array-of-spheres.
Method
si-RMSE 4 AE |
We 2,240 perspective crops from 224 Laval Indoor panoramas, provided by DiffusionL.ight ages are resized 512 512 match our model input. Quantitative results are shown in Table 9 and qualitative results in Fig. 9.
EverLight [8]
0.091
6.36
All input im-
StyleLight [55]
0.123
7.09
Weber et al. [61]
0.081
4.13
EMLight [69]
0,099
3,99
DiffusionLight [45]
0.090
5.25
Ours
0.089
4.90
While our method performs slightly below specialized sys- tems like Weber al. [61] and EMLight [69], it remains competitive-desyee despite not being trained on Laval Indoor.
Notably, it outperforms StyleLight [55] and Diffusionl.ight [45], demonstrating strong generalization across lighting domains.
Input Image
GT
DiffusionLLight
Ours
Input Image
GT
DiffusionLightt
Ours
Figure Visual results on oray-off-spheres protocol.
C.2 Lighting Estimation with the Cube++ Dataset
We also evaluated our method on the Cube++ dataset [12], specifically designed for illumination esti- mation and color constancy. This dataset includes illu- mination information annotated by the SpyderCube II For our experiment, we selected 100 processed JPEG images from Cube++. We then applied both Diffusion- Light and our method to estimate the illumination from each image. Subsequently,
Table 10: Scores SpyderCube white face rendering on Cube++ datasct.
RMSE 1.
AE J. Left Right
Method
Left
Right
D.Light [45] 0.044 0.035 7.221 5.741 Ours
rendered the left and right white faces of the SpyderCube under the estimated illumination, assuming purely Lambertian diffuse
0.024 0.025 3.985 4.003
faces. To prevent information leakage from the SpyderCube in the input images, we masked out the SpyderCube from the tested images and inpainted the masked region using LaMa [51]. We then compared the rendered face colors to the colors sampled directly from the SpyderCube JPEG images. Table 10 presents the RMSE and angular errors, demonstrating that our method clearly outperforms DiffusionLight, achieving angular errors of less than 5" on both faces. Visual comparison results are further illustrated in Fig. 10.
10 http:://ggg.taaaccc/////faaaaaaa array _f__herree_pe_ee_____e http::/:/ww..aaaaoo../o/////////
datacolor wwtpspp/der//rooouctttataataseee wttp:e/tttttssd/edeepeeeeeeeeeee wpwsssyy-eeee cube/
24

---

PARKING FOR CROATIANS ONLY CEHERS SILLI
GT
Ours
D.Light GT
Ours D.Light GT
Ours D.Light GT
Ours D.Light GT
Ours D.Light
Figure 10:
Visual results on Cubc++ dataset. We show the rendered two white cube faces, mirror ball, and mate silver ball from our method and DiffusionLight for visual comparison.
C.3 Lighting Estimation from Foreground Objects
Since our model is trained on object-centric synthetic rendering data, we can also apply it to estimate lighting from foreground objects. We selected 4 NeRF synthetic objects [41] and 4 real-world objects [53], aiming to estimate lighting from videos containing nine consecutive rendering views.
We qualitatively compare LuxDiT with optimization- based inverse rendering methods [42, 18] that reconstruct 3D geometry and lighting from full NeRF scenes. Using the ground truth camera poses, we rotate each frame's 's estimated lighting into the global coordinate system and average across frames to produce the final environment map.
Qualitative results are shown in Fig. 11. On mostly diffuse objects like lego and hotdog, our method recovers highlight directions accurately, enabling shadow rendering consistent with the input. For glossy objects like mic and ficus, our model estimates lighting nearly identical to the ground truth. While these HDR environment maps included in our training set, the NeRF scenes differ signifi- cantly from our synthetic renderings (see Fig. 8), indicating that our model leverages shading cues and learned priors rather than direct memorization. In contrast, optimization- based baselines struggle to capture high-frequency lighting detail and often introduce noise and artifacts in lighting.
We further tested our method real-world foreground objects from the Objects sirtettwwith--iie wigh-nigh dataset [53], which provides ground truth distant environment lighting. Similar to the NeRF synthetic scene setup, the estimated lighting was then aligned into the global coordinate system using ground truth camera poses. We compared our approach to MeuS+Mitsuba [56, 26), the top- nop-performing method this dataset [53]. The metrics, using the three-sphere protocol, are presented in Table 11, with visual results in Fig. 12.
While model performs well overall. minor errors remain, e.g. color shifts in the NeRF Lego scene (Fig. 11) and a slightly higher si-RMSE compared to NeuS+Mitsuba (Table 11). We believe combining generative model with oopimization-basee methods could further enhance lighting estimation, which we leave for future work.
Table 11: Comparison of our method with NeuS+ on Objects with Lighting datasets.
Scale-invariant RMSE L
Angular Error |
Normalized RMSE 1
Method
Diffuse
Matte
Mirror
Diffuse
Matte
Mirror
Diffuse
Matte
Mirror
NeuS+Mitsuba
0.082
0.232
0.424
3.145
3.383
3.526
0.180
0.545
0.717
Ours
0.086
0.253
0.482
1.262
1.594
2.000
0.153
0.339
0.479
25

---

GT & Input
NVDIFFREC [42]
NVDIFFRECMC [18]
Ours
Figure 11: Lighting estimation from the NeRF synthetic objects. We use the estimated lighting from different
methods to re-render the original NeRF Blender scenes.
Input
GT
Input
GT
Input
GT
Mitsuba
Ours
NeuS Mitsuba
Ours
NeuS Mitsuba
Ours
Figure 12: Lighting estimation from the masked real objects from Objects with Lighting.
C.4 Additional Ablations
C.4.1 The Choice of the HDR Fusion Model
As detailed in Sec. 4.1, a lightweight MLP z is employed to merge the dual-tonenn dual-tonemapped environment maps, Eur and Ejog. thereby reconstructing the HDR environment map E. There are also alternative fusion methods, such as using a more complex CNN model to incorporate adjacent pixel information for HDR fusion, applying rule-based approach with explicit inverse equations. To justify our choice of a simple MIP, we evaluate various HDR fusion techniques, including MI., CNN, and rule-based method. The CNN model has an identical number of layers to our MI.P model, using x 3 convolution kernels across layers. The rule-based method involves applying the inverse Reinhard map for lights with intensity below linear interpolation between Reinhard and log maps for intensities ranging from 8 to 16, and exclusively the log map for intensities exceeding 16.
Table 12 presents the RMSE results on testing Polyhaven HDRIs. All three methods demonstrate comparable accuracy, with the MLP approach exhibiting slight advantage. Compared to the rule-based approach, we believe the neural approach can better handle numer- ical inconsistency after image uint8 quantization, and the potential data range overflow (e-g., lights beyond the pre-defined maximum intensity 10000).
Table 12: Comparison differ- ent HDR fusion approaches.
MLP CNN Rule
RMSE 11.55
26

---

C.4.2 The Impact of LoRA on Synthetic Scenes
Section 5.5 demonstrates the impact of varying LoRA scales (0.0 to 1.0) on the predicted lighting content of real-world images. This ablation study, conversely, investigates how our LoRA model, trained with real images, affects the lighting estimation of synthetic foreground objects. Table 13 presents the angular errors using a shree-sphere evaluation, and Fig. 16 provides the visual results.
Table 13: Ablation study on impact of LoRA scale on synthetic foreground objects.
LoRA Scale
0.00
0.25
0.50
0.75
1.00
Diffuse 1.594
1.737
2.170
3.832
3.937
Matte 1
2.068
2.311
2.914
5.322
5.891
Mirror 3.405
3.690
4.342
6.783
7.400
In contrast to the ablation performed on scene images, a larger LoRA scale leads to lower lighting estimation accuracy. As Fig. 16 illustrates, increasing the LoRA scale causes foreground content to gradually appear on the estimated environment map, which is consistent with our LoRA model's behavior. Nevertheless, the estimated highlights remain consistent across different LoRA scales.
D Additional Results
We provide additional visual results in this section to further support the claims made in the main paper.
Model Ablation and LoRA Scale: Figure 13 details the ablation study model's design and the exploration of different LoRA scales.
Camera Parameter Variations: Figures 14 and 15 show lighting estimation performance
when varying camera field of view (FOV) and elevation angles, respectively.
Three-Sphere Rendering Evaluations: Figures 17, 18, and 19 display further lighting estima- tion outcomes using the three-sphere rendering protocol on the Laval Indoor, Laval Outdoor, and Poly Haven datasets.
Virtual Object Insertion: Figures 20 and 21 illustrate additional virtual object insertion results on Poly Haven panorama crops and Waymo driving scenes.
27

---

Input Image
Model Ablation
LoRA Scale Exploration
channel concat.
LoRA
LoRA 0.1
LoRA 0.2
LoRA 0.4
w/o syn. data
LoRA 0.6
LoRA 0.7
LoRA (1.
LoRA 1.0
Figure 13: Model design ablation and LoRA scale exploration. The "Model Ablation" column shows the results of our two model design variants: 1) channel concatenation and 2) training without synthetic rendering data. The "LoRA Scale Exploration" columns show the visual results of model with different LoRA scales.
FOV
45
50
60
70%
75
GT
Figure 14: Lighting estimation from input images with varying camera FOV.
Elevation
30%
15"
0
15a
30
GT
Figure 15: Lighting estimation from input images with varying camera elevation.
Input Image
LoRA Exploration
LoRA 0.00
LoRA 0.25
LoRA 0.50
LoRA 0.75
LoRA 1.00
Figure 16: LoRA scale exploration on synthetic foreground scenes.
28

---

Input Image
GT
SiyleLight
DiffusionLight
Ours
Figure 17: Additional qualitative results on Laval Indoor dataset.
29

---

Input Image
GT
H-G et
NLFE
DiffusionLight
Ours
Figure 18: Additional qualitative results Laval Outdoor dataset.
30

---

Input Image
GT
StyleLight
NLFE
DiffusionLight
Ours
NEEO
Figure 19: Additional qualitative results on Poly Haven dataset.
31

---

Reference
BhyleLight
Diffusionl Light
DiPIR
Ours
Figure 20: Additional virtual object insertion on Poly Haven perspective crops.
H-G et al. [23]
NLFE
DiffusionLight
DiPIR
Ours
Figure 21: Additional virtual object insertion Waymo driving scenes.
32


Reviews:
[{"review_id": "u2EoHqEf7w", "reviewer": "Reviewer_b4Ep", "summary": "The paper proposes LuxDiT, a novel method for estimating high-dynamic-range (HDR) lighting from single images or videos using a video diffusion transformer (DiT). The model is trained on a large-scale synthetic dataset and then adapted to real-world scenes using LoRA-based fine-tuning on curated HDR panoramas. The approach demonstrates state-of-the-art results in both image- and video-based lighting estimation tasks and downstream applications such as virtual object insertion.", "questions": "1. Can you provide more details on new provided dataset? \n2. Can you demonstrate your approach using TCC or Cube++ dataset?\n3. Can you show variances of errors in your experimental table? \n4. Does your improvement positively affects on final renedring quality?", "limitations": "1. Very high computational time\n2. Not clearly described issues which have been significantly improved with new approach", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\nPaper written with clear motivation has good structure and reasoning.\n\n1, Authors demonstrate the power of modern diffusion model and showed possibility to proivide good prediction of HDR maps.\n2. Exhaustive work with existing dataset helps to verify that authors achieves a quality improvement. Existing SotA methods are also mentioned.\n3. Authors provides a good dataset for highlighted problem, which has a good potential to attract more attention to the problem\n\nWeaknesses:\n1. My main concern is that this paper has only incremental improvement. It is hard to understand from the paper answer on the question: does the authors achieved solution of some existing issue? May be in scope of the photorealistic rendering some user study would help to highlight significance of the result.\n2. Experimental study a bit indirectly indicate improvements, moreover authors didn't provide variance of errors, so it is impossible to evalute the significance of improvement\n3. Not all the propper dataset have been used for the problem. I recommend to use TCC and Cube++, since they also have mirror ball in the scene", "was_revised": true, "final_justification": "The main point appers to be very convincing about paper valuability is a demo of the results. It is really nice to see significant improvement in this very complicated problem. I still feel that this paper have issues with physical side of the problem, yet neural side looks pretty good", "rebuttal": "We thank the reviewer for their valuable feedback, and appreciate that they recognize our contributions. We respond to the comments and questions below.\n\n**W1: Contribution.** We thank the reviewer for prompting a deeper discussion of our contributions. \n\nLuxDiT tackles two persistent challenges in HDR lighting estimation: (1) predicting accurate intensity and direction from spatial inputs, which requires inferring global, non-spatial outputs (360° panoramas) from local, spatial visual cues; and (2) recovering angular high-frequency details consistent with the scene content.\n\nOur solution combines: (1) a scalable generative architecture that leverages a powerful pre-trained video diffusion model and a carefully curated large-scale synthetic dataset; (2) dual-tonemapped latent representations for HDR structure; and (3) LoRA fine-tuning on real HDR panoramas to improve semantic alignment. While each component builds on existing techniques, their integration is purposefully designed for the unique demands of lighting estimation. \n\nTo our knowledge, LuxDiT is the first end-to-end training method to formulate lighting prediction as a conditional generative diffusion task. We conduct extensive evaluations and ablations. LuxDiT reduces lighting direction error by 45% on Laval Outdoor (Table 2) and improves temporal consistency on video input (Table 3), enabling reliable use in downstream tasks such as virtual object insertion. We will release our models, dataset, and code to support reproducibility and future work. \n\n**W1&Q4: Rendering quality evaluation with estimated lighting.** We use the lighting estimated by LuxDiT  to render virtual objects for insertion into real scenes, as described in the main paper Sec. 5.4. Qualitative results are shown in Fig. 1, Fig. 4, and Supplementary Figs. S11 and S12. In response to reviewer W1, a user study was also conducted to evaluate perceptual quality (further details in Supp. Sec. B.4). As shown in Table 8, renderings generated with LuxDiT's lighting were favored over those produced by baseline methods.\n\n**W2&Q3: Variance of metrics.** We thank the reviewer for pointing out the lack of variance reporting. Due to the stochastic nature of diffusion models, we evaluated LuxDiT three times using different random seeds and reported the average metrics. For brevity, we omitted the standard deviations in the main paper. Below, we provide the standard deviations for the metrics reported in Table 1 of the main paper.\n\n|               | si-RMSE      |              |              | Angular | Error       |              | norm-RMSE  |           |              |\n| ------------- | ------------ | ------------ | ------------ | ------------- | ------------ | ------------ | ------------ | ------------ | ------------ |\n| Dataset       | Diffuse      | Matte        | Mirror       | Diffuse       | Matte        | Mirror       | Diffuse      | Matte        | Mirror       |\n| Laval Indoor  | 0.112±0.001 | 0.297±0.007 | 0.586±0.011 | 2.555±0.084  | 3.526±0.092 | 5.641±0.114 | 0.196±0.011 | 0.341±0.010 | 0.457±0.019 |\n| Laval Outdoor | 0.068±0.002 | 0.190±0.005 | 0.396±0.014 | 2.018±0.081  | 2.939±0.018 | 5.286±0.021 | 0.137±0.006 | 0.271±0.009 | 0.454±0.020 |\n| Poly Haven    | 0.077±0.003 | 0.196±0.008 | 0.442±0.009 | 1.235±0.061  | 1.977±0.058 | 2.783±0.058 | 0.111±0.005 | 0.199±0.004 | 0.323±0.003 |\n\nAs shown in the table above, the variance of metrics is small enough, not affecting the comparison results shown in the paper.\n\n**W3&Q2: Additional evaluation on TCC or CUBE++.** We thank the reviewer for suggesting additional datasets for evaluation. We performed an additional experiment using 100 images from Cube++. We applied both DiffusionLight and our method to estimate lighting from each image, then rendered the left and right white faces of the SpyderCube under the estimated illumination, assuming purely diffuse reflectance. We compared the rendered face colors to the provided ground truth and reported RMSE and angular error. Our method outperforms DiffusionLight across all reported metrics. \n\nWe note that TCC and Cube++ are primarily designed for color constancy rather than 360° HDR lighting estimation, and the reflective spheres in these datasets lack accurate HDR ground truth or consistent annotations. As such, this experiment is inherently approximate and should be considered a supplementary evaluation. \n\n|                | RMSE       |            | Angular    | Error      |\n| -------------- | ---------- | ---------- | ----------- | ----------- |\n|                | Left       | Right      | Left        | Right       |\n| DiffusionLight | 0.115      | 0.112      | 18.648      | 18.379      |\n| Ours           | **0.110** | **0.110** | **18.066** | **18.050** |\n\n**Q1: More details on our dataset.** We thank the reviewer for their interest in our provided dataset. Details of the dataset curation process, including selection criteria, rendering setup, and preprocessing steps, are provided in Supplementary Sec. B.2. This dataset will also be released."}, {"review_id": "ZPj5nqWhBO", "reviewer": "Reviewer_GYdA", "summary": "This paper presents LuxDiT, a transformer-based diffusion framework for estimating HDR environment maps from a single image or video. The model is built on top of a video diffusion transformer and operates on a dual-tonemapped representation of HDR lighting (Reinhard and log-tonemapped variants), which is jointly denoised in latent space. A novel conditioning mechanism, directional embedding, and two-stage training (synthetic pretraining followed by LoRA-based adaptation) are introduced. Evaluation is conducted on three benchmarks, showing improved performance over prior methods like StyleLight and DiffusionLight.", "questions": "-There’s no discussion of why these baselines were excluded, and as a result, the evaluation appears narrow and potentially cherry-picked.\n-How camera parameters are selected for the LDR crops.\n-How robustness to real-world scenes is assessed.\n-Whether the model generalizes to truly novel lighting conditions (e.g., high-contrast outdoor settings).", "limitations": "The major limitation is that improvements are not significant.", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n-HDR environment map estimation from a single image or video is a challenging and important task for applications in graphics, relighting, and AR/VR.\n-The use of dual tonemapping (Reinhard + log) for representing HDR panoramas and the joint transformer denoising approach is technically interesting.\n\nWeaknesses:\n-While the architecture is inspired by recent diffusion transformers and introduces some adaptations (e.g., token-based conditioning, direction maps), these are incremental and follow standard practices (e.g., AdaLN, LoRA fine-tuning).\nThe novelty of combining two tonemapped HDR representations is not sufficiently justified. Prior works (e.g., in inverse rendering or HDR synthesis) have used log-space representations extensively. The added complexity of dual-tonemapping is not ablated clearly.\n-The reported improvements are small and inconsistent. For instance, on the Laval Indoor dataset, the model slightly improves on RMSE but does not outperform prior work on angular error across the board. Some results are within statistical noise and lack significance testing or confidence intervals.\n-Missing Evaluation on Key Baselines: The baselines compared against (StyleLight, DiffusionLight) are relevant, but the paper does not consider broader categories of lighting estimation models—e.g., inverse rendering models or methods like NeRFactor, InverseRenderNet, or relighting-specific diffusion models.", "was_revised": true, "final_justification": "Although the concerns of experiments and some clarification have been well addressed, I still feel the novelty is an issue. But I won't argue for rejecting this paper.", "rebuttal": "We thank the reviewer for their valuable feedback, and appreciate that they recognize our contributions. We respond to the comments and questions below.\n\n**W1: Technical contribution.** We thank the reviewer for prompting a deeper discussion of our contributions. \n\nLuxDiT tackles two persistent challenges in HDR lighting estimation: (1) predicting accurate intensity and direction from spatial inputs, which requires inferring global, non-spatial outputs (360° panoramas) from local, spatial visual cues; and (2) recovering angular high-frequency details consistent with the scene content.\n\nOur solution combines: (1) a scalable generative architecture that leverages a powerful pre-trained video diffusion model and a carefully curated large-scale synthetic dataset; (2) dual-tonemapped latent representations for HDR structure; and (3) LoRA fine-tuning on real HDR panoramas to improve semantic alignment. While each component builds on existing techniques, their integration is purposefully designed for the unique demands of lighting estimation. \n\nTo our knowledge, LuxDiT is the first end-to-end training method to formulate lighting prediction as a conditional generative diffusion task. We conduct extensive evaluations and ablations. LuxDiT reduces lighting direction error by 45% on Laval Outdoor (Table 2) and improves temporal consistency on video input (Table 3), enabling reliable use in downstream tasks such as virtual object insertion. We will release our models, dataset, and code to support reproducibility and future work.\n\n**W1: Dual-Tonemapping Light Representation.** The use of multiple tonemapped light maps for HDR representation is common in rendering tasks. Similar approaches have been applied in prior methods for both lighting estimation (e.g., DiffusionLight) and relighting tasks (e.g., Neural Gaffer).\n\nAn alternative choice for our HDR light representation is to follow DiffusionLight’s multi-pass, multi-EV tonemapping. However, this design choice also falls short of our requirement for higher dynamic range environment maps with spatially coherent content.\nDiffusionLight's reliance on separate diffusion passes for estimating environment maps with varied exposure values introduces spatial inconsistencies. Its multi-EV HDR fusion results in a limited dynamic range, with a maximum intensity of 32. \nIn contrast, our proposed method, LuxDiT, addresses these limitations by jointly denoising dual-tonemapped environment maps within the same latent tokens. This approach significantly enhances spatial consistency during HDR fusion. Furthermore, LuxDiT utilizes Reinhard and logarithm tonemapping functions, which provide sufficient sampling to cover a broad range of lighting scenarios, from rich LDR content to high-contrast spotlights (as shown in Supplementary Fig. S1). This enables HDR reconstruction with a peak intensity of up to 10,000.\n\n\n**W2: Significance of quantitative improvements.** We evaluated our method on various benchmark datasets with different evaluation metrics. Specifically, on Laval Outdoor scenes, our model substantially enhances peak light direction estimation quality, reducing the error by 45% (Table 2). Furthermore, it improves video lighting consistency by decreasing temporal jittering by up to 81% (Table 3, \"std\" terms). \n\nThe reviewer's concern regarding metric inconsistency primarily stems from the Laval Indoor experiment. As stated in the paper (L242-L245), our model was not trained on the Laval Indoor dataset. This dataset exhibits a notable shift in color and intensity distribution compared to our training set. In contrast, the baseline method, StyleLight, is exclusively trained on Laval Indoor. While DiffusionLight does not directly use Laval Indoor, the Text2Light model, which generates its training data, is trained with Laval Indoor data. Despite these factors, our method demonstrates strong robustness by still achieving competitive metrics on the Laval Indoor dataset.\n\nRegarding statistical noise and confidence interval, as also mentioned by R1, we evaluated LuxDiT three times using different random seeds and reported the average metrics. For brevity, we omitted the standard deviations in the main paper.\nBelow, we provide the standard deviations for the metrics reported in Table 1 of the main paper.\nAs shown in the table, the variance of metrics is small enough, not affecting the comparison results shown in the paper. \n\n|               | si-RMSE      |              |              | Angular Error |              |              | norm-RMSE    |              |              |\n| ------------- | ------------ | ------------ | ------------ | ------------- | ------------ | ------------ | ------------ | ------------ | ------------ |\n| Dataset       | Diffuse      | Matte        | Mirror       | Diffuse       | Matte        | Mirror       | Diffuse      | Matte        | Mirror       |\n| Laval Indoor  | 0.112±0.001 | 0.297±0.007 | 0.586±0.011 | 2.555±0.084  | 3.526±0.092 | 5.641±0.114 | 0.196±0.011 | 0.341±0.010 | 0.457±0.019 |\n| Laval Outdoor | 0.068±0.002 | 0.190±0.005 | 0.396±0.014 | 2.018±0.081  | 2.939±0.018 | 5.286±0.021 | 0.137±0.006 | 0.271±0.009 | 0.454±0.020 |\n| Poly Haven    | 0.077±0.003 | 0.196±0.008 | 0.442±0.009 | 1.235±0.061  | 1.977±0.058 | 2.783±0.058 | 0.111±0.005 | 0.199±0.004 | 0.323±0.003 |\n\n\n**W3&Q1: Evaluations on other key baselines.** We thank the reviewer for highlighting broader categories of lighting-related methods. Our work focuses specifically on single-view, geometry-free estimation of scene-scale 360° HDR environment maps. \n\nMethods like NeRFactor operate under different assumptions. They typically require multi-view posed images and geometry to optimize lighting, shape, and material, and are designed for object-centric, per-scene optimization. Our method, conversely, focuses on generalizable, image-conditioned lighting prediction, making direct comparisons on standard lighting estimation benchmarks (e.g., Laval, PolyHaven) infeasible due to the distinct input conditions and scale of light estimation (object vs. scene).\n\nWe thank the reviewer for mentioning inverse rendering models for the lighting estimation. We did not deliberately exclude the inverse rendering models in our evaluation. The input conditions for inverse rendering models and image-based lighting estimation are very different. Inverse rendering requires additional multi-view posed images and 3D geometry for estimating light probes, and such light estimation is usually on object scale instead of scene scale. It is very difficult for us to report inverse rendering models’ metrics in Table 1. \n\nThat said, we include preliminary comparisons with inverse rendering methods (e.g., NVDIFFREC and NVDIFFRECMC) in Supplementary Sec. C.2 (Fig. S5). We also evaluated our method on the Objects-with-Lighting dataset [1] against NeuS+Mitsuba, as reported in the table below. \n\n|              | si-RMSE |       |        | Angular | Error     |        | norm |   RMSE    |        |\n| ------------ | ------- | ----- | ------ | ------------- | ----- | ------ | --------- | ----- | ------ |\n| Method       | Diffuse | Matte | Mirror | Diffuse       | Matte | Mirror | Diffuse   | Matte | Mirror |\n| NeuS+Mitsuba | 0.082   | 0.232 | 0.424  | 3.145         | 3.383 | 3.526  | 0.180     | 0.545 | 0.717  |\n| Ours         | 0.086   | 0.253 | 0.482  | 1.262         | 1.594 | 2.000  | 0.153     | 0.339 | 0.479  |\n\nOur method is not expected to completely surpass inverse rendering approaches. Instead, we believe that integrating our generative model with inverse rendering methods could further improve lighting estimation.\n\nThe reviewer mentioned InverseRenderNet, an inverse rendering model that estimates lighting using low-frequency spherical harmonics (SH) from a single image. Our paper compares against models that also predict low-frequency lighting from RGB input (e.g., H-G et al. in Tables 1 & 2), and LuxDiT surpasses their performance. EMLight [2] is another method that relies on low-frequency light as guidance; the StyleLight paper included EMLight in its comparison, and EMLight did not outperform StyleLight. A comprehensive evaluation of earlier methods like InverseRenderNet is beyond the scope and time constraints of this rebuttal, but we will incorporate a discussion of its relevance to our work in the revised version.\n\n**Q2: Camera parameters for LDR crops.** We uniformly sample the FOV of the perspective camera within the range between 45 and 80 degrees. The camera elevation angle is sampled between -10 and +10 degrees.\n\n**Q3&Q4: Robustness and generalization to real-world scenes.** In the paper, we used the virtual object insertion rendering to show our model’s robustness and generalization to real-world scenes. We show the metrics in Table 8, visual rendering results in Fig. 1, Fig. 4, and Supplementary Fig. S11, S12. Note that all the Waymo driving scenes can be treated as in-the-wild real-world outdoor captures (not cropped from any panorama). Our model effectively estimates high-contrast sunlight, allowing for the accurate casting of sharp shadows from inserted objects.\n\n\n> [1] Ummenhofer, Benjamin, et al. \"Objects with lighting: A real-world dataset for evaluating reconstruction and rendering for object relighting.\" 3DV, 2024.\n\n> [2] Zhan, Fangneng, et al. \"Emlight: Lighting estimation via spherical distribution approximation.\" AAAI, 2021."}, {"review_id": "XmW5WVCF8P", "reviewer": "Reviewer_DYHn", "summary": "This paper presents LuxDiT, a DiT for HDR environmental lighting estimation from an LDR image or video. The model predicts LDR and Log-encoded environmental maps, and uses a separate MLP to fuse them back into the full HDR environment map. A small-scale LoRA finetuning stage with real-world HDR panoramas is added to improve real-world semantic consistency. The approach achieves state-of-the-art results on multiple benchmarks.", "questions": "1. Can you provide a direct comparison between your learned MLP HDR fusion and a simple rule-based baseline? As well as a CNN variant.\n2. Clarify the concatenation method of z_log and z_ldr\n3. Provide a quantitative evaluation of video temporal consistency\n4. Redo the lora scale ablation on an object-centric dataset", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 4, "strengths_and_weaknesses": "**Strengths**\n1. The paper presents a clean and effective latent diffusion-based framework (DiT) for HDR panorama prediction, which is an important task, especially under today's trend that redoes everything with video generation models.\n2. LoRA finetuning on real-world panorama crops improves semantic consistency; tone-mapping and exposure augmentations enhance robustness to in-the-wild image distributions, showing strong domain bridging ability.\n3. The method achieves state-of-the-art performance on multiple public benchmarks.\n4. The supplementary video shows qualitatively good temporal consistency on video inputs.\n\n**Weaknesses**\n1. **HDR Fusion**\n    - The HDR fusion MLP is purely per-pixel and does not leverage spatial consistency, which may lead to local inconsistency.\n    - No ablation is provided against rule-based HDR fusion methods like those used in DiffusionLight.\n2. **Latent Concatenation Description Inconsistency**\n    - Figure 2 may be misleading: Eldr, Elog, and Edir are encoded separately by the VAE; z_ldr and z_log are either channel- or token-wise concatenated, while z_dir is added.\n    - Sec. 4.1 claims channel-wise concat, but Sec. 5.5 shows token-wise concat performs better. It's unclear which is actually used.\n3. **Lack of Quantitative Temporal Analysis**\n    - While qualitative results on video are shown, no quantitative metrics (e.g., angular jitter, HDR SSIM) are reported.\n4. **LoRA Scale Ablation Unclear**\n    - The LoRA scale ablation is inconclusive, as the finetuning data (PolyHaven) matches the test domain. It would be more informative to test degradation on other settings (e.g., synthetic or object-centric inputs).", "was_revised": true, "final_justification": "Thank the authors for their rebuttal. The author's rebuttal clarified and resolved my concerns well, and I decided to keep my current positive rating.", "rebuttal": "We thank the reviewer for their valuable feedback, and appreciate that they recognize our contributions. We respond to the comments and questions below.\n\n**W1&Q1: HDR fusion.** We found that per-pixel HDR fusion is sufficient for generating quality fused HDR maps. Our method exhibits fewer inconsistency issues compared to prior work, DiffusionLight. DiffusionLight estimates environment maps with varied exposure values through separate diffusion passes. In contrast, our method utilizes a single pass to obtain environment maps with two distinct tone-mappings. These two tone-mapped environment maps are decoded from different latent channels of the same denoised DiT tokens, which also enhances their spatial alignment.\n\nTo justify our design choice, we compared various HDR fusion approaches, including MLP, CNN, and a rule-based method, to justify the use of a simple MLP. The rule-based method applies the inverted Reinhard map for lights with intensity below 8, a linear interpolation between Reinhard and log maps for intensities between 8 and 16, and exclusively the log map for intensities greater than 16. The table below shows the RMSE\n\n|      | MLP   | CNN   | Rule|\n| ---- | ----- | ----- | ---------- |\n| RMSE | 11.55 | 11.74 | 11.71      |\n\n\nAll three approaches achieve a similar level of accuracy, with the MLP approach performing slightly better than the other two approaches. Compared to the rule-based approach, we believe the neural approach can better handle numerical inconsistency after image uint8 quantization, and the potential data range overflow (e.g., lights beyond the pre-defined maximum intensity 10000).\n\n**W2&Q2: Clarification of the concatenation method.** Our final model employs a dual concatenation approach. Specifically, the latents for $E_{ldr}$ , $E_{log}$ , and $E_{dir}$ are concatenated channel-wise. On the other hand, the latent for the input image condition $\\mathbf{I}$ is token-wise concatenated with envmap tokens, as illustrated in Fig. 2. Regarding the ablation results in Table 7, Sec. 5.5, the \"channel concat.\" variant concatenates all noisy and conditional latents along the channel dimension. For this variant, the envmap $\\mathbf{E}$ is resized to match the input image $\\mathbf{I}$, and no token-wise concatenation is used. To prevent confusion and misunderstanding, we will further clarify this design and enhance the consistency of our descriptions in the revised paper. \n\n**W3&Q3: Quantitative temporal analysis.** We provided the metrics for measuring temporal consistency in Table 3. We computed the standard deviation (std) of per-frame error metrics for each video clip (L268-L270). For example, “PAE Std” in Table 3 measures the jittering scale of the peak light position in a video sequence.\n\n**W4&Q4: LoRA Scale Ablation.** As suggested by the reviewer, we made an ablation on the LoRA Scale with input synthetic objects. The angular errors with three-sphere evaluation are shown in the table below\n\n| LoRA Scale   | Diffuse | Matte | Mirror |\n| ----- | ------- | ----- | ------ |\n| 0.00 | 1.594   | 2.068 | 3.405  |\n| 0.25 | 1.737   | 2.311 | 3.690  |\n| 0.50 | 2.170   | 2.914 | 4.342  |\n| 0.75 | 3.832   | 5.322 | 6.783  |\n| 1.00 | 3.937   | 5.891 | 7.400  |\n\nContrary to the ablation made on scene images (Polyhaven crops), as the LoRA scale becomes larger, the corresponding lighting estimation accuracy becomes lower. We will also include visual results in the revised paper."}, {"review_id": "3fdwTPxJgi", "reviewer": "Reviewer_YCWS", "summary": "This paper proposes LuxDiT, a method for estimating HDR environment maps from images or videos. It fine-tunes CogVideoX by replacing its text tokens with visual condition tokens derived from the environment map. To address the VAE's LDR limitation, the method uses dual tone-mapped representations (LDR and LogHDR) instead of raw HDR. A small MLP is then used to reconstruct the HDR map from these. Finally, LoRA fine-tuning with real-world LDR panoramas enhances semantic alignment. The method achieves state-of-the-art results in both qualitative and quantitative evaluations.", "questions": "1. **Stage II Input Representation:**\n In Equation 5 (Page 6, Line 214), what does $z_t$ represent during Stage II (semantic adaptation)? Since $z^{log}$ is unavailable for LDR-only inputs, how do you maintain the expected latent shape for the diffusion model? Equation 4 suggests the model expects a concatenated pair [$z^{ldr}$,$z^{log}$], which is shape [l,h,w,2c] but LDR-only inputs would be shape [l,h,w,c]\n\n2. **VAE Usage with Single Images:**\n How do you apply the CogVideoX VAE, which compresses 4-frame video segments for single-frame inputs (ie, estimate lighting for image input)? Was the VAE modified, or is there a workaround to ensure compatibility?\n\n3. **Directional Embedding Effectiveness:**\nDirectional embedding (Page 4, Line 156) is introduced to improve angular continuity, but no supporting ablation is provided. However, no evidence or ablation study is provided. Could you show results comparing models with and without this component?\n\n4. **Inference Procedure:**\nAt inference time, when the $z^{ldr}$ and $z^{log}$ are initialized from Gaussian noise, is directional embedding still applied?\n\n5. **Clarification of Figure 2:**\nFigure 2 implies the latent shape becomes [l, h, w, 3C] through channel concatenation, but Lines 159–160 describe a channel-wise addition of E_dir to the token, suggesting the shape should be [l,h,w,2C]. Could you clarify this discrepancy?", "limitations": "Yes", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 4, "strengths_and_weaknesses": "## Strengths \n- **Quality:**  Medium-quality work with solid experiments compared against state-of-the-art methods. The method of LuxDiT is technically sound. However, it lacks some key ablations (e.g., the effect of directional embedding).\n\n - **Significance:** The method shows significant improvements in lighting estimation, supported by strong quantitative and qualitative results.\n\n - **Originality:** The paper introduces 2 novel ideas\n> 1. Use of a video diffusion transformer to estimate lighting from images or video.\n> 2. Directional embedding to enhance angular continuity in the predicted environment map.\n\n## Weakness\n- **Clarity:** While the core idea is clear, the paper lacks sufficient implementation details. For example, the architecture of the MLP and the process of fine-tuning CogVideoX—especially how the model handles the removal of text tokens—are not adequately explained in the main paper. Some of this is covered in the appendix, but a brief summary or a clear reference in the main text would help. Additionally, Figure 2 seems inconsistent with the description in Lines 159–160: it suggests direct channel concatenation of $E_{ldr}$, $E_{log}$ and $E_{dir}$, whereas the text explain that $E_{ldr}$ and $E_{log}$ are concatenated and $E_{dir}$ is added via channel-wise addition.", "was_revised": true, "final_justification": "I find the technical contribution of this paper—conditioning a video diffusion model to generate HDR environment maps—both novel and effective, with clear improvements over previous works.\n\nThe authors have satisfactorily addressed my concerns, which primarily focused on implementation details to ensure reproducibility. Based on the strength of the results and the clarified responses, I recommend acceptance.", "rebuttal": "We thank the reviewer for their valuable feedback, and appreciate that they recognize our contributions. We respond to the comments and questions below.\n\n**W: Clarity of Implementation Details.** We appreciate the reviewer's feedback regarding implementation details. Supplements B.1 and B.3 provide detailed designs of our model, including the HDR MLP and modifications to CogVideoX. We will incorporate additional implementation details in the revised main paper.\n\n**W&Q5: Clarification of Figure 2.** We apologize for the inconsistent description in the draft regarding Figure 2. The concatenation shown in Figure 2 and the addition described on Lines 159-160 for directional latent conditioning are essentially equivalent. Specifically, in CogVideoX, before being processed by transformer blocks, the concatenated latents $[\\mathbf{z}^{ldr}, \\mathbf{z}^{ldr}, \\mathbf{z}^{dir}]$ are processed by a projection layer: $\\mathbf{h} = W \\cdot [\\mathbf{z}^{ldr}, \\mathbf{z}^{ldr}, \\mathbf{z}^{dir}]$. This equation can be re-formulated as $\\mathbf{h} = W_{0:2C} \\cdot [\\mathbf{z}^{ldr}, \\mathbf{z}^{ldr}] + W_{2C:3C} \\cdot [\\mathbf{z}^{dir}]$, which aligns with the description on Lines 159-160. To eliminate this confusion, we will correct this description in the revised paper.\n\n**Q1: Stage II input representation.** In Stage II, the noisy latent $\\mathbf{z}_t$ is identical to that in Stage I. Initially, $\\mathbf{z}_T$ is set to Gaussian noise with dimensions $[l, h, w, 2c]$. The DiT model continues to denoise this concatenated noisy latent $\\mathbf{z}_t$ throughout Stage II training. The main distinction is that the denoised log part $\\mathbf{z}^{log}_T$ is excluded from the diffusion loss computation if the input videos lack log maps.\n\n**Q2: VAE usage with single images.** When the single image is used as the input, CogVideoX’s 3D causal VAE will encode the single-frame image as special independent latent tokens (8x8x1 compression for the first frame). The original CogVideoX model is also jointly trained with single-frame image generation to support this.\n\n**Q3: Directional embedding effectiveness.** To enhance our model's ability to reason about the 3D orientation of environment maps, we utilize an additional directional embedding. This embedding allows us to augment our model training by randomly rolling or warping the input directional map, enabling the model to generate a corresponding environment map. Without this directional embedding, the model would only be able to predict environment maps with a fixed orientation, which could negatively impact its angular continuity. We additionally fine-tuned the model without using directional embedding and evaluated it on a subset of Polyhaven envmaps; the angular errors with three-sphere evaluation are reported in the table below, which justifies the effectiveness of directional embedding.\n\n|             | Diffuse | Matte | Mirror |\n| ----------- | ------- | ----- | ------ |\n| W/   Dir. | 1.273   | 1.979 | 2.967  |\n| W/o Dir.    | 1.481   | 2.420 | 3.501  |\n\n**Q4: Inference procedure.** At inference time, when $\\mathbf{z}^{ldr}_T$ and $\\mathbf{z}^{log}_T$ are initialized from Gaussian noise, we still concatenate the directional embedding along with denoising latents. However, the noise is not added to the directional embedding."}]

Author rebuttal:

Meta-review: This paper presents a framework for HDR panorama prediction based on a latent diffusion transformer. The approach utilizes LoRA finetuning on real-world panorama crops to ensure semantic consistency and applies tone-mapping and exposure augmentations to enhance robustness, resulting in strong domain bridging capabilities. The method achieves SOTA results across multiple benchmarks, with supplementary materials demonstrating impressive qualitative performance and temporal consistency.

While some concerns were raised regarding physical realism and limited novelty, these issues were adequately addressed in the rebuttal and during discussion. Reviewers generally agree that the technical contribution of conditioning a video diffusion model for HDR environment maps is both novel and effective.

Given the consistent support from reviewers, the strong empirical performance, and the demonstrated practical impact, the consensus is to accept the paper, and AC agrees with the majority.
Decision: Accept (poster)