Title: LuxDiT: Lighting Estimation with Video Diffusion Transformer
Venue: NeurIPS 2025
Accepted: yes
Keywords: Lighting estimation

Inverse rendering

Video diffusion models
Abstract: Estimating scene lighting from a single image or video remains a longstanding challenge in computer vision and graphics. Learning-based approaches are constrained by the scarcity of ground-truth HDR environment maps, which are expensive to capture and limited in diversity. While recent generative models offer strong priors for image synthesis, lighting estimation remains difficult due to its reliance on indirect visual cues, the need to infer global (non-local) context, and the recovery of high-dynamic-range outputs. We propose LuxDiT, a novel data-driven approach that fine-tunes a video diffusion transformer to generate HDR environment maps conditioned on visual input. Trained on a large synthetic dataset with diverse lighting conditions, our model learns to infer illumination from indirect visual cues and generalizes effectively to real-world scenes. To improve semantic alignment between the input and the predicted environment map, we introduce a low-rank adaptation finetuning strategy using a collected dataset of HDR panoramas. Our method produces accurate lighting predictions with realistic angular high-frequency details, outperforming existing state-of-the-art techniques in both quantitative and qualitative evaluations.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes LuxDiT, a method for estimating HDR environment maps from images or videos. It fine-tunes CogVideoX by replacing its text tokens with visual condition tokens derived from the environment map. To address the VAE's LDR limitation, the method uses dual tone-mapped representations (LDR and LogHDR) instead of raw HDR. A small MLP is then used to reconstruct the HDR map from these. Finally, LoRA fine-tuning with real-world LDR panoramas enhances semantic alignment. The method achieves state-of-the-art results in both qualitative and quantitative evaluations.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths \n- **Quality:**  Medium-quality work with solid experiments compared against state-of-the-art methods. The method of LuxDiT is technically sound. However, it lacks some key ablations (e.g., the effect of directional embedding).\n\n - **Significance:** The method shows significant improvements in lighting estimation, supported by strong quantitative and qualitative results.\n\n - **Originality:** The paper introduces 2 novel ideas\n> 1. Use of a video diffusion transformer to estimate lighting from images or video.\n> 2. Directional embedding to enhance angular continuity in the predicted environment map.\n\n## Weakness\n- **Clarity:** While the core idea is clear, the paper lacks sufficient implementation details. For example, the architecture of the MLP and the process of fine-tuning CogVideoX—especially how the model handles the removal of text tokens—are not adequately explained in the main paper. Some of this is covered in the appendix, but a brief summary or a clear reference in the main text would help. Additionally, Figure 2 seems inconsistent with the description in Lines 159–160: it suggests direct channel concatenation of $E_{ldr}$, $E_{log}$ and $E_{dir}$, whereas the text explain that $E_{ldr}$ and $E_{log}$ are concatenated and $E_{dir}$ is added via channel-wise addition.\n\nQUESTIONS:\n1. **Stage II Input Representation:**\n In Equation 5 (Page 6, Line 214), what does $z_t$ represent during Stage II (semantic adaptation)? Since $z^{log}$ is unavailable for LDR-only inputs, how do you maintain the expected latent shape for the diffusion model? Equation 4 suggests the model expects a concatenated pair [$z^{ldr}$,$z^{log}$], which is shape [l,h,w,2c] but LDR-only inputs would be shape [l,h,w,c]\n\n2. **VAE Usage with Single Images:**\n How do you apply the CogVideoX VAE, which compresses 4-frame video segments for single-frame inputs (ie, estimate lighting for image input)? Was the VAE modified, or is there a workaround to ensure compatibility?\n\n3. **Directional Embedding Effectiveness:**\nDirectional embedding (Page 4, Line 156) is introduced to improve angular continuity, but no supporting ablation is provided. However, no evidence or ablation study is provided. Could you show results comparing models with and without this component?\n\n4. **Inference Procedure:**\nAt inference time, when the $z^{ldr}$ and $z^{log}$ are initialized from Gaussian noise, is directional embedding still applied?\n\n5. **Clarification of Figure 2:**\nFigure 2 implies the latent shape becomes [l, h, w, 3C] through channel concatenation, but Lines 159–160 describe a channel-wise addition of E_dir to the token, suggesting the shape should be [l,h,w,2C]. Could you clarify this discrepancy?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-06-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents LuxDiT, a DiT for HDR environmental lighting estimation from an LDR image or video. The model predicts LDR and Log-encoded environmental maps, and uses a separate MLP to fuse them back into the full HDR environment map. A small-scale LoRA finetuning stage with real-world HDR panoramas is added to improve real-world semantic consistency. The approach achieves state-of-the-art results on multiple benchmarks.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n1. The paper presents a clean and effective latent diffusion-based framework (DiT) for HDR panorama prediction, which is an important task, especially under today's trend that redoes everything with video generation models.\n2. LoRA finetuning on real-world panorama crops improves semantic consistency; tone-mapping and exposure augmentations enhance robustness to in-the-wild image distributions, showing strong domain bridging ability.\n3. The method achieves state-of-the-art performance on multiple public benchmarks.\n4. The supplementary video shows qualitatively good temporal consistency on video inputs.\n\n**Weaknesses**\n1. **HDR Fusion**\n    - The HDR fusion MLP is purely per-pixel and does not leverage spatial consistency, which may lead to local inconsistency.\n    - No ablation is provided against rule-based HDR fusion methods like those used in DiffusionLight.\n2. **Latent Concatenation Description Inconsistency**\n    - Figure 2 may be misleading: Eldr, Elog, and Edir are encoded separately by the VAE; z_ldr and z_log are either channel- or token-wise concatenated, while z_dir is added.\n    - Sec. 4.1 claims channel-wise concat, but Sec. 5.5 shows token-wise concat performs better. It's unclear which is actually used.\n3. **Lack of Quantitative Temporal Analysis**\n    - While qualitative results on video are shown, no quantitative metrics (e.g., angular jitter, HDR SSIM) are reported.\n4. **LoRA Scale Ablation Unclear**\n    - The LoRA scale ablation is inconclusive, as the finetuning data (PolyHaven) matches the test domain. It would be more informative to test degradation on other settings (e.g., synthetic or object-centric inputs).\n\nQUESTIONS:\n1. Can you provide a direct comparison between your learned MLP HDR fusion and a simple rule-based baseline? As well as a CNN variant.\n2. Clarify the concatenation method of z_log and z_ldr\n3. Provide a quantitative evaluation of video temporal consistency\n4. Redo the lora scale ablation on an object-centric dataset\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-06-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents LuxDiT, a transformer-based diffusion framework for estimating HDR environment maps from a single image or video. The model is built on top of a video diffusion transformer and operates on a dual-tonemapped representation of HDR lighting (Reinhard and log-tonemapped variants), which is jointly denoised in latent space. A novel conditioning mechanism, directional embedding, and two-stage training (synthetic pretraining followed by LoRA-based adaptation) are introduced. Evaluation is conducted on three benchmarks, showing improved performance over prior methods like StyleLight and DiffusionLight.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n-HDR environment map estimation from a single image or video is a challenging and important task for applications in graphics, relighting, and AR/VR.\n-The use of dual tonemapping (Reinhard + log) for representing HDR panoramas and the joint transformer denoising approach is technically interesting.\n\nWeaknesses:\n-While the architecture is inspired by recent diffusion transformers and introduces some adaptations (e.g., token-based conditioning, direction maps), these are incremental and follow standard practices (e.g., AdaLN, LoRA fine-tuning).\nThe novelty of combining two tonemapped HDR representations is not sufficiently justified. Prior works (e.g., in inverse rendering or HDR synthesis) have used log-space representations extensively. The added complexity of dual-tonemapping is not ablated clearly.\n-The reported improvements are small and inconsistent. For instance, on the Laval Indoor dataset, the model slightly improves on RMSE but does not outperform prior work on angular error across the board. Some results are within statistical noise and lack significance testing or confidence intervals.\n-Missing Evaluation on Key Baselines: The baselines compared against (StyleLight, DiffusionLight) are relevant, but the paper does not consider broader categories of lighting estimation models—e.g., inverse rendering models or methods like NeRFactor, InverseRenderNet, or relighting-specific diffusion models.\n\nQUESTIONS:\n-There’s no discussion of why these baselines were excluded, and as a result, the evaluation appears narrow and potentially cherry-picked.\n-How camera parameters are selected for the LDR crops.\n-How robustness to real-world scenes is assessed.\n-Whether the model generalizes to truly novel lighting conditions (e.g., high-contrast outdoor settings).\n\nLIMITATIONS:\nThe major limitation is that improvements are not significant.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes LuxDiT, a novel method for estimating high-dynamic-range (HDR) lighting from single images or videos using a video diffusion transformer (DiT). The model is trained on a large-scale synthetic dataset and then adapted to real-world scenes using LoRA-based fine-tuning on curated HDR panoramas. The approach demonstrates state-of-the-art results in both image- and video-based lighting estimation tasks and downstream applications such as virtual object insertion.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\nPaper written with clear motivation has good structure and reasoning.\n\n1, Authors demonstrate the power of modern diffusion model and showed possibility to proivide good prediction of HDR maps.\n2. Exhaustive work with existing dataset helps to verify that authors achieves a quality improvement. Existing SotA methods are also mentioned.\n3. Authors provides a good dataset for highlighted problem, which has a good potential to attract more attention to the problem\n\nWeaknesses:\n1. My main concern is that this paper has only incremental improvement. It is hard to understand from the paper answer on the question: does the authors achieved solution of some existing issue? May be in scope of the photorealistic rendering some user study would help to highlight significance of the result.\n2. Experimental study a bit indirectly indicate improvements, moreover authors didn't provide variance of errors, so it is impossible to evalute the significance of improvement\n3. Not all the propper dataset have been used for the problem. I recommend to use TCC and Cube++, since they also have mirror ball in the scene\n\nQUESTIONS:\n1. Can you provide more details on new provided dataset? \n2. Can you demonstrate your approach using TCC or Cube++ dataset?\n3. Can you show variances of errors in your experimental table? \n4. Does your improvement positively affects on final renedring quality?\n\nLIMITATIONS:\n1. Very high computational time\n2. Not clearly described issues which have been significantly improved with new approach", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-05"}