LuxDiT: Lighting Estimation with Video Diffusion Transformer
Authors: name: Ruofan Liang; name: Kai He; name: Zan Gojcic; name: Igor Gilitschenski; name: Sanja Fidler; name: Nandita Vijaykumar; name: Zian Wang
Published: 2025-09-03 19:59:20

Abstract: Estimating scene lighting from a single image or video remains a longstanding
challenge in computer vision and graphics. Learning-based approaches are
constrained by the scarcity of ground-truth HDR environment maps, which are
expensive to capture and limited in diversity. While recent generative models
offer strong priors for image synthesis, lighting estimation remains difficult
due to its reliance on indirect visual cues, the need to infer global
(non-local) context, and the recovery of high-dynamic-range outputs. We propose
LuxDiT, a novel data-driven approach that fine-tunes a video diffusion
transformer to generate HDR environment maps conditioned on visual input.
Trained on a large synthetic dataset with diverse lighting conditions, our
model learns to infer illumination from indirect visual cues and generalizes
effectively to real-world scenes. To improve semantic alignment between the
input and the predicted environment map, we introduce a low-rank adaptation
finetuning strategy using a collected dataset of HDR panoramas. Our method
produces accurate lighting predictions with realistic angular high-frequency
details, outperforming existing state-of-the-art techniques in both
quantitative and qualitative evaluations.