Title: IllumiCraft: Unified Geometry and Illumination Diffusion for Controllable Video Generation
Venue: NeurIPS 2025
Accepted: yes
Keywords: Diffusion Model; Video Generation
Abstract: Although diffusion-based models can generate high-quality and high-resolution video sequences from textual or image inputs, they lack explicit integration of geometric cues when controlling scene lighting and visual appearance across frames. To address this limitation, we propose IllumiCraft, an end-to-end diffusion framework accepting three complementary inputs: (1) high-dynamic-range (HDR) video maps for detailed lighting control; (2) synthetically relit frames with randomized illumination changes (optionally paired with a static background reference image) to provide appearance cues; and (3) 3D point tracks that capture precise 3D geometry information. By integrating the lighting, appearance, and geometry cues within a unified diffusion architecture, IllumiCraft generates temporally coherent videos aligned with user-defined prompts. It supports the background-conditioned and text-conditioned video relighting and provides better fidelity than existing controllable video generation methods.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces IllumiCraft, a diffusion-based framework for controllable video relighting that integrates lighting, appearance, and geometry cues. It takes HDR lighting maps, synthetically relit frames, and 3D point tracks as input to generate temporally coherent videos aligned with user prompts. Additionally, the authors present a large-scale dataset of paired videos with corresponding HDR maps and 3D tracks, enabling broader research in controllable video generation. Extensive evaluations validate the model's effectiveness on the video relighting task.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The paper is well motivated and focus on an important limitation of current video diffusion models, which is lack of explicit control over lighting with changing geometry.\n2. IllumiCraft effectively fuses appearance, lighting, and geometric cues into a unified framework and leads to high-quality and temporally consistent video outputs.\n3. The introduction of a large-scale, well-annotated dataset (20,170 video pairs with synchronized HDR, 3D tracks, etc.) is a valuable contribution for the community.\n\nWeakness:\n1. The generated videos appear visually coherent, but the paper does not evaluate more complex temporal dynamics, such as moving light sources, dynamic shadows, or object interactions.\n2. While the paper includes quantitative ablations, it lacks qualitative comparisons showing how each of the three input cues (HDR maps, relit frames, 3D point tracks) contributes to visual improvements.\n\nQUESTIONS:\n1. Can you provide qualitative ablation examples illustrating the impact of each guidance cue?\n2. The current results suggest strong performance under relatively constrained conditions. Could the authors comment on how IllumiCraft would perform with moving light sources or larger dynamics?\n3. Will the dataset be released?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-19"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents IllumiCraft, a video relighting framework based on DiT. The model incorporates multiple control signals—geometry (3D point tracks), lighting (HDR environment maps), text prompts, and background image into a unified generation pipeline. It adopts a DiT backbone with ControlNet-style conditioning and is trained on a large-scale dataset created via the proposed IllumiPipe pipeline. Experiments demonstrate superior performance including perceptual quality, temporal consistency compared to existing baselines.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n- Addresses a relevant and practical problem in video editing and content creation.\n- Integration of geometry, lighting, and text guidance into a single architecture.\n- Demonstrated improvements over multiple baselines across quantitative and qualitative metrics.\n- A data processing pipeline for video relighting.\n\n**Weaknesses:**\n- Novelty is limited; mainly integrates known components (IC-Light, ControlNet, DiffusionLight, DaS, etc.).\n- Omits comparison with latest diffusion renderer baselines, especially [CVPR2025] DiffusionRenderer, which is highly similar to the paper’s pipeline.\n- HDR warping from the first frame is unvalidated and may be unreliable.\n- Sparse geometry inputs (3D point tracks) may be suboptimal for relighting tasks.\n- No mechanism to resolve conflicts among background, lighting, and text conditions.\n- Incomplete ablations (e.g., geometry-only setting not shown).\n- Missing key evaluation metrics (e.g., PSNR) in ablation result tables. (table 4, 5 and supp table 3, 4)\n- Fig. 3 is confusing and could mislead readers about the model’s core task. (I believe the input should be “original video” instead of “relit video”, and the output should be “relit video”. The relit video from Light-A-Video should only be serving as the input video under another lighting.)\n\nQUESTIONS:\n1. Clarify Fig. 3. It looks like the Light-A-Video relit video is used as input to IllumiCraft. If so, what is the model generating? I assume this is only a training-time input to simulate the input foreground video under another lighting condition. Please confirm.\n2. Compare warped HDR vs. per-frame DiffusionLight HDRs. The paper assumes warping is more stable, but no evidence is given. A proper ablation is needed to validate this design choice.\n3. What happens when the lighting changes mid-video? The warping strategy assumes stationary illumination. Please discuss how the model behaves in dynamic lighting scenarios, and provide results with varying lighting within a video.\n4. Are lighting prompts/HDR maps at test time unseen during training?\n5. Include geometry-only ablations in table 3. This would clarify how much 3D point guidance alone contributes.\n6. How does the model handle conflicting inputs? For example, bright background but dark HDR map—what dominates? Any fallback mechanism?\n7. 3D tracking points might not be the best way of providing geometry guidance. An alternative solution can be using feature maps in VGGT, or using depth maps like in DiffusionRenderer. Please add experiments with these other design choices.\n8. Since you are already using ControlNet, why is the main backbone DiT also fine-tuned? Alternative ways could be inputting all condition signals into the main backbone, or putting all condition signals into the copied ControlNet and freezing the main backbone. The current architecture looks pretty weird.\n\nLIMITATIONS:\nAs a relighting paper, the paper does not provide analysis on the quality of different object materials or lighting complexity. The results shown in the paper are mostly diffuse objects, with no evaluation on specular objects, anisotropic material, sub-surface scattering, and global illumination effects like indirect lighting are not very obvious in the shown results, nor analyzed. The shadow is also not properly handled in the method, with a lot of baking artifacts.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-22"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis manuscript presents IllumiCraft, a framework achieving end-to-end video relighting through diffusion-based controllable video generation. Compared to existing approaches, this work explicitly incorporates geometric cues for more accurate illumination-geometry interactions, particularly helpful when scene’s geometry changes, which is achieved by injecting 3D tracking video via a ControlNet. In conjunction with this model, they also propose a data collection pipeline, IllumiPipe, to create training data containing both illumination and geometric cues. Comparison shown in the paper exhibits clear quantitative and qualitative improvements than previous arts, while results in the supplementary material demonstrate that the framework is capable of synthesizing visually appealing videos.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\n1.\tThe idea of integrating geometric cues is reasonable and well-motivated;\n\n2.\tThe paper is overall well-written and easy to follow;\n\n3.\tThe proposed framework outperforms state-of-the-art methods under the same conditions.\n\n**Weaknesses:**\n\n1.\tThe main concern of the reviewer is about the ablation of geometry guidance, which constitutes the core contribution of this work. Since the introduction of geometric cues has logically sound motivation as mentioned in the paper, it is suggested to include qualitative ablation results to provide more insights about how it works and for what scenarios it is especially helpful. Actually, only showing improvements in FVD, LPIPS, PSNR, and CLIP similarity are insufficient to draw a reliable conclusion that the geometric guidance is effective. For example, if the geometry guidance does not work as expected but instead just introduces some noise, it might still bring some quantitative gains as a kind of data augmentation.\n\n2.\tLacking of video comparisons with existing works. There are only image form qualitative comparisons in the current submission, which cannot convincingly compare the temporal coherence. Additionally, L36–38 mentions that existing methods struggle with scene’s geometry changes and this work aims to \"address these shortcomings.\" It is therefore essential to provide video comparisons to support the claimed advantages under such conditions.\n\nQUESTIONS:\n1.\tI’m curious why to emphasize \"during training\" whenever “leverage geometric cues” is mentioned in paper (even in Section 3.4 Inference). Does this imply geometric cues are leveraged exclusively during training and not in inference? If geometric cues are also utilized during inference, how to handle potential motion in background?\n\n2.\tThe current data curation pipeline relies entirely on augmenting in-the-wild videos. Why not incorporate synthetic data from graphics engines, as IC-light and RelightVid?\n\n3.\tDo you think the integration of geometric cues would also benefit diffusion-based image relighting methods? IC-light has suggested that properly scaling training of diffusion-based image relighting models can already enables accurate geometric inference.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces IllumiCraft, a controllable video relighting method that jointly uses HDR maps, synthetically relit foregrounds, and 3D point tracks to guide a diffusion-based video generator. To support training, the authors also build IllumiPipe, an automated pipeline that creates a dataset (20K+ videos) with various annotations like lighting, masks, background, and geometry. The model builds on a DiT backbone and injects lightweight modules for illumination and geometry conditioning.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n1. This paper’s writing is clear and well-structured, walking readers through data collection, architecture, training, and evaluation.\n2. Method is flexible—supports text-only, HDR-guided, or full multimodal inference.\n3. It builds a large-scale automated dataset (20 k+ videos with HDR, relit clips, masks, backgrounds, and 3D tracks).\n\n**Weaknesses**\n1. It does not evaluate robustness when HDR warping or 3D tracking fails under rapid motion.\n2. It lacks ablations that isolate the impact of each modality (relit vs. HDR vs. geometry).\n3. The authors do not address potential conflicts among their multiple conditioning signals— for example, when HDR illumination cues, 3D geometry guidance, and background references are misaligned or contradictory. They should discuss how the model resolves or is robust to such inconsistent inputs.\n4. The paper’s primary contributions lie in constructing a automated  dataset and introducing geometric conditioning, but there is little methodological or architectural novelty.\n\nQUESTIONS:\nWhat happens to relighting quality when one or more auxiliary inputs (HDR maps or 3D tracks) are missing or noisy? The authors could include visual comparisons of single‐condition relighting—e.g. illumination‐only versus geometry‐only guidance—to illustrate each modality’s standalone impact.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}