IllumiCraft: Unified Geometry and Illumination Diffusion for
  Controllable Video Generation
Authors: name: Yuanze Lin, name: Yi-Wen Chen, name: Yi-Hsuan Tsai, name: Ronald Clark, name: Ming-Hsuan Yang

Although diffusion-based models can generate high-quality and high-resolution
video sequences from textual or image inputs, they lack explicit integration of
geometric cues when controlling scene lighting and visual appearance across
frames. To address this limitation, we propose IllumiCraft, an end-to-end
diffusion framework accepting three complementary inputs: (1)
high-dynamic-range (HDR) video maps for detailed lighting control; (2)
synthetically relit frames with randomized illumination changes (optionally
paired with a static background reference image) to provide appearance cues;
and (3) 3D point tracks that capture precise 3D geometry information. By
integrating the lighting, appearance, and geometry cues within a unified
diffusion architecture, IllumiCraft generates temporally coherent videos
aligned with user-defined prompts. It supports background-conditioned and
text-conditioned video relighting and provides better fidelity than existing
controllable video generation methods. Project Page:
https://yuanze-lin.me/IllumiCraft_page