IllumiCraft: Unified Geometry and Illumination Diffusion for Controllable Video Generation
Authors: Yuanze Lin
Yi-Wen Chen
Yi-Hsuan Tsai
Ronald Clark
Ming-Hsuan Yang
Venue: NeurIPS 2025 poster ()
Keywords: Diffusion Model; Video Generation
Primary area: deep_learning

Abstract: Although diffusion-based models can generate high-quality and high-resolution video sequences from textual or image inputs, they lack explicit integration of geometric cues when controlling scene lighting and visual appearance across frames. To address this limitation, we propose IllumiCraft, an end-to-end diffusion framework accepting three complementary inputs: (1) high-dynamic-range (HDR) video maps for detailed lighting control; (2) synthetically relit frames with randomized illumination changes (optionally paired with a static background reference image) to provide appearance cues; and (3) 3D point tracks that capture precise 3D geometry information. By integrating the lighting, appearance, and geometry cues within a unified diffusion architecture, IllumiCraft generates temporally coherent videos aligned with user-defined prompts. It supports the background-conditioned and text-conditioned video relighting and provides better fidelity than existing controllable video generation methods.

IllumiCraft: Unified Geometry and Illumination Diffusion for Controllable Video Generation
Yuanze Lin
YinWen Chen
Yi-Hsuan Tsai
Ronald Clark
Ming-Hsuan Yang
University of Oxford
UC Merced
NNEC Labs America
AAmanitt Inc. Google DeepMind
Project Page: https: : https://yuanzer http//yaaazzeiinne
me/ cellmmmmmillmmmaa www.lllrlaattppape.
Input Vidco
Background
Relit Vidco Frames
..y stroing lighting"
.y natural light"
cool-blue spotlights pierce mist,
moody depth"
bright white spotlights cut through grey haze, stark contrast"
Figure 1:
Given prompt and input video, IllumiCraft edits scene illumination conditioned on the static background image. It handles a variety of illumination scenarios, including spotlight effects.
Abstract
Although diffusion-based models can generate high-quality and high-resolution video sequences from textual image inputs, they lack explicit integration of geo- metric cues when controlling scene lighting and visual appearance across frames. To address this limitation, we propose IllumiCratt, end-to-end diffusion frame- work accepting three complementary inputs: (1) high-dynamic-range (HDR) video
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

maps for detailed lighting control; (2) synthetically relit frames with randomized illumination changes (optionally paired with static background reference image) to provide appearance cues; and (3) 3D point tracks that capture precise 3D ge- ometry information. By integrating the lighting, appearance, and geometry cues within unified diffusion architecture, IllumiCraft generates temporally coherent videos aligned with user-defined prompts. supports background-conditione and (ext-conditioned video relighting and provides better fidelity than existing controllable video generation methods.
1 Introduction
Illumination plays an important role in creating appealing visuals as it helps accentuate the three- dimensional structure of scenes in otherwise flat 2D images. For example, an apple in the sun typically has a bright specular highlight on the side where the surface normal most directly faces the light source, while the far side has dark soft shadow. This accentuates both the roundness and the glossy texture of the apple. However, despite the importance of this light-geometry interaction, most video generation models [1, 2, 3] treat illumination an uncontrollable implicit factor.
Achieving coherent relighting in videos involves two main challenges: maintaining consistent illumination over time and ensuring physically plausible light-scene interactions. Light sources should not change abruptly between frames, as this introduces distracting flicker. In addition, shadows, highlights, and reflections need to move consistently with the camera and object motion. Traditional inverse- rendering techniques [4, 5, 6, 7] decompose scenes into albedo, normals, and lighting, but rely on specialized inputs (e.g.. HDR captures or spherical harmonics) and typically assume static scenes. This inhibits their practicality for motion-rich, real-world videos.
Recent diffusion models such RelightVid [8] and Light-A-Video [1] extend the single-image relighting model (i.c.. Light [9]) to the video domain. RelightVid [10] modifies the 2D U- Net in IC-Light to 3D backbone and adds temporal attention layers to enforce frame-to-frame consistency, while Light-A- Video [I] integrates cross-fras light-attention module into the self- attention blocks of IC-Light and applies a progressive fusion step suppress flicker and smooth illumination transitions. However, both methods rely on implicit temporal correlations and omit explicit geometric guidance. As such, they suffer from overall loss of lighting fidelity and coherence whenever the scene's 's geometry changes.
To address these shortcomings, we propose to jointly model lighting and geometry within a unified diffusion framework for controllable video generation. By fusing precise 3D geometry trajectories with detailed illumination cues, our model learns the joint evolution of appearance and motion under dynamic illumination. This approach enables high-fidelity editing of scene illumination, preserving accurate illminationn-goomeii interactions at every frame.
Specifically, we present IllumiCraft, an end-to-end diffusion architecture that enables controllable video generation, tailored for video relighting, by specifying either: (1) HDR environment maps encoding detailed lighting information; (2) synthetically perturbed video frames with randomized illumination shifts (optionally paired with static background references) to capture appearance variations; and (3) 3D trajectory videos that trace object geometry through space and time. During training, these streams are fused within a DiT-b diffusion model [11, 2]. enabling the generation of temporally consistent geometry-awart relit videos. The main contributions of this work are:
We propose a unified diffusion architecture that jointly incorporates illumination and geometry guidance, enabling high-quality video relighting. It supports text-conditioned and background- conditioned relighting for videos.
We introduce high-quality video dataset comprising 20. video pairs, featuring paired original videos and synchronized relit videos, HDR maps, and 3D tracking videos. This dataset supports vidco relighting and serves valuable resource for broader controllable video generation tasks.
We conduct extensive evaluations demonstrating our model's effectiveness against state-of-ther methods the video relighting task.
2

---

2 Related Work
Diffusson Models. Building on the success of rext-to-image diffusion techniques [12. 13, 14, 15],
researchers have fine-tuned diffusion models to tackle wide range of vision tasks. Interactive image manipulation frameworks like InstructPix2Pix [16], ControlNet [17] and LearnablRRegion [18] enable semantic editing conditioned on textual prompts, while geometry geometry-aware aware extensions such as DreamFusion [19] and DreamPolisher [20] repurpose diffusion priors for text-to-3I) generation. In parallel, single-view 3D reconstruction methods [21, 22, 23, 24] leverage score-based priors for 3D reconstruction from a single image. Specialized relighting models [10, 25, 26, 27] have achieved precise per-image illumination adjustments. Recently, IC-Light [9] enforces light-transport indepen- dence for image relighting; instead of extending to video like Rel [8] or Light-A- [1], we build on the DiT architecture [11] to achieve temporally coherent video relighting.
Video Diffusion Models and Video Editing. In recent years, diffusion- based video genera- tion [28, 29, 3, 30, 31, 32] has seen rapid advancements. Building on these foundations, training- free approaches like V2V [33], MotionClone [30], and BroadWay [34] enable prompt-driven inpaint- ing, style transfer, and motion retargeting without additional tuning. For frame-precist coherence, fine-tuning approaches such as Videoo [35] and Tune-A- AVideo [36] adapt pretrained video diffusion models to user references, delivering seamless object insertion and consistent color grading. Although current video editing approaches [36, 37, 33] achieve impressive performance, they do not explicitly model lighting cues. In contrast, our method embeds both illumination and geometry cues directly into the diffusion model, unlocking the capacity of high-fidelity video relighting.
Video Relighting. Recent video relighting techniques have primarily focused on facial videos. SunStage [38] reconstructs facial geometry and reflectance from rotating outdoor selfie video, using the sun as a natural "light stage" to enable on-device portrait relighting under novel lighting conditions. DifFRelight [39] uses dlfffusion-bssen image-to-image translation for free-vid viewponnt facial performance relighting, conditioning on flat-lit captures and unified lighting controls to produce high-fidelity relit sequences. SwitchLight [40] predicts per-frame normals and shading maps guided by HDR environment maps. More recently, Light-A- Video [1] enhances IC-Light [9] with consistent light attention module for cross-frame lighting coherence, while Relight [8] introduces temporal attention layers into IC-Light to improve temporal consistency. However, they depend exclusively on illumination cues and omit geometric guidance, which hinders their video relighting quality.
Method
We introduce IllumiCraft, unified diffusion framework that jointly leverages geometry and illu- mination cues for controllable video generation. First, in Section 3.1 we describe IllumiPipe, our data collection pipeline that constructs high-quality dataset from real videos, complete with HDR environment maps, 3D tracking video sequences, and synthetically relit foreground clips. Next, Sec- tion 3.2 details the core architecture of IllumiCraft, which fuses appearance, geometric, and lighting guidance in a single architecture. Finally, in Section 3.3 and 3.4 we present our training strategy and inference workflow, respectively, highlighting how each component contributes to high- high-fidelity, controllable video generation.
3.1 lllumiPipe
Collecting a paired video dataset with comprehensive annotations is essential for training a robust video generation model capable of supporting high-fidelity video relighting. However, public video datasets rarely include both HDR environment maps and 3D tracking sequences, limiting progress in video relighting and geometry-guide NlumiPipe, an efficient data collection pipeline designed to extract HDR environment maps data, relit video clips, and precise 3D tracking video sequences from real-world videos. Figure 2 illustrates the detailed workflow of IllumiPipe.
video editing performance. To address this gap, we introduce
Specificallyy each appearance video Vapor E RTxHXWXx3 augmentation data, which together enable joint modeling of geometry and illumination:
x3 is accompanied by 6 types of distinct
Vappr ++ (it. Vog. Vidro Vg00, Vmask C},
3

---

video frame
Foreground
Object Segmenter
Mask
Mask Video
Video Inpainter
Frieee's Madd
Video
Captiomer
Inpointtd
Dictusion Light
Lightin
Vidzo
porrpt
Vidio 2IDR Eateradon
Video
Input Vidoo
Video Matting
Augmenter
(4) Bookgrawol Video
the frame
Geumeery Tracker
HDR Video
Input Video
Foreground Videa
Augmented Video
Inpul Videu
Geumetry
(1) HDR Maps
Foragroued (odd: Video
Rabit. Video
Tracking Video
Video
Figure 2: Data collection mechanism of our proposed IllumiPipe. For each input video, our proposed IllemiPipe extracts various types of data: (1) HDR maps, (2) foreground video and mask video, (3) relit video, (4) background video, and (5) 3D tracking video. The details and collection process for each data type described in Section 3.1.
where VIT E 18TxHxW3 represents the relit foreground video, Vog E IRT'XXXW x3 is the background video, Vndr E R2 denotes the HDR environment maps, Vgeo E RRT'Xxx x3 represents the 3D tracking video sequences, Vmakk E RT x Xww.xxxxxx x3 denotes the masks of the foreground video, and C is the caption describing the appearance video Vappe- Here, T, W and H denote the number of frames, frame width and frame height, respectively.
HDR Environment Maps. HDR environment maps encode per-direction radiance values that represent both the angular distribution and absolute intensity of scene illumination, enabling physically accurate image-based lighting. We leverage DiffusionLight [41] extract these HDR maps. However, since DiffusionLight is designed for single-image inputs, applying it independently to each video frame introduces severe temporal inconsistency, where the synthesized chrome ball often varies significantly from frame to frame. To enforce temporal stability, we extract the chrome ball image only from the first frame of each video, and then warp this initial chrome ball onto all subsequent frames, yielding temporally coherent HDR environment maps across the entire sequence.
Specifically, given the chrome ball image from the first frame I. (extracted using DiffusionLLight [41] we use Video Depth Anything [42] to obtain depth maps for the video. We then (1) track a sparse set of reliable image points across frames, (2) use their depth values infer the camera's 's 3D motion via a constrained affine fit, and (3) apply that motion to warp the reference chrome ball by lifting its pixels representative depth, projecting them through the estimated transform and camera intrinsics, and resampling to estimate current frame's chrome ball image. We explain more details about the whole process the Appendix.
Relit Videos and Background Videos. Given a real-world video Vapor. we first apply Grounded SAM-2 [43] to obtain the foreground mask from the first frame. We then feed the appearance video Vuppe together with the first frame's 's mask into the video object matting model MatAnyone [44] to extract the foreground appearance video Vig and the corresponding mask video Vmask-
Next, we generate relit video Viein by applying the video relighting method Light-A- Video [1] to Vig. conditioned on diverse set of user-defined lighting prompts. In total, we curated 100 distinct lighting prompts: some are drawn from the official Light-A-Vider lighting prompts examples (e.g.. "red and blue neon light", "sunset over sea". "in the forest, magic golden lit") while others are generated via GPT-4o mini to cover more challenging or unusual settings (c.g., "urban jungle glow". "subtle office g)ow"). The resulting relit video Vielit retains the original motion and viewpoint but exhibits the new target illumination. The full list of all 100 lighting prompts is shown in the Appendix. To obtain the background video Vog. we feed the foreground mask video Vinas together with the appearance video Vuppe into the video inpainting model DiffEraser [45].
3D Tracking Videos. For real-world appearance videos Vapor- where ground-truth geometry is unavailable, we employ Spatial/Tracker [46] to detect and localize salient 3D interest points directly in 3D space. We initialize uniform grid of 4, 4,900 points per video to ensure even spatial coverage across the scene. For each consecutive frame pair, SpatialTracker estimates the 3D positions of these points and computes their correspondences using learned spatial matching. The output is dense and

---

A long broun-haired woman white top intently applies eyelid makeup with large brush, under natural light against a neutral backdrop.
Lighting
Text Enc
Enc
HDR Maps
Illumination Tokens
Prompt Embedding
VAE Enc
Relit Video
DiT Block
VAE Dec
Latent Noise
Padding to video
Generated Video
Reference Image
Zero Linear
Add
VAE
DiT Block
Concut
Enc
Trainuble Weights
3D Tracking Video
Condition DiT (Trainable Copy)
Frozen Weights
Figure 3: Famework of IllumiCraft. It uses HDR maps, relit foreground video, 3D tracking, and an optional background image to jointly model illumination, appearance, and geometry, then generates videos from an illumination-wware text prompt. The figure shows an example with illumination tokens; HDR maps, background images, and 3D tracking videos are all optional during training.
temporally coherent set of 3D point tracks that closely approximates the true motion of the scene, even in unconstrained and dynamic environments.
Video Captions. To generate detailed descriptions of each video, we employ CogVL.M2-Vdden LLLMA3-Chal [47] with the following prompt: "Carefully watch the video and describe its content, the object's motion, lighting and atmosphere in vivid detail, highlighting the object's movement (e.g.. drifting left slowly, darting forward quickly, bouncing up and down, floating upward gently), lighting conditions (e.g., red-and-bles neon, natural sunlight, studio spotlights, sci-fi RGB glow, cyberpunk neon, a sunset over the sea, magical forest illumination) and the overall atmosphere (e.g,, warm, moody, ethereal, cozy, gritty urban, or futuristic haze)." This prompt guides the model to produce rich captions emphasizing visual content, object motion. lighting, and overall atmosphere.
3.2 Model Architecture
We build our framework on top of the pre-trained video generation model Wan2.1 [2], which utilizes a transformer-basee video diffusion architecture [11]. By initializing our network with Wan's learned weights, we both leverage its strong video priors and significantly accelerate training. The overall architecture of IllumiCraft is illustrated in Figure 3.
Latent Feature Extraction. We first vero-pad the reference image Inef E x33x3 (the first frame of the background video Vag) along the temporal axis to form a reference video Vret E x x3 randomly zero out the reference image with 10% probability to
Note that during training, enable flexible video editing even when no reference image is provided. Next, we apply the VAE encoder EVAr to the appearance video Vappe. the relit foreground video Vtr, and the reference video Vref to obtain their corresponding latent representations: z = EvAe(Vappe' E F # x x16 Zyf Fv E IR fx
# _ x16 and žref (V E R x x16 We concatenate the relit foreground latent and reference latent along the channel dimension to form the control latent: =e Concat(22,, 2ret) E I f x $ W. x32 We then corrupt the appearance latent over t diffusion steps to produce 24, concatenate 2t with the control latent 2. along the channel dimension, and feed the result into the diffusion transformer [11, 2] for iterative denoising. Finally, the denoised latent is passed through the VAE decoder DYAE reconstruct the output video.
Inject Illumination Control. extract illumination cues from the HDR maps, we first encode the tensor Vnde& E F x33x3xx33x via the lighting encoder, compact MLP- TLL-TTransfomer (see Appendix),
5

---

producing a feature matrix Kndr E IR NxD. We then introduce the learnable illumination embedding X' E RNXD, D updated as X' X + Xhdr- To encourage X to internalize a robust illumination prior, we randomly zero out Mhdr with 50% probability during training, simulating the absence of HDR inputs inference time. To condition the diffusion model, we concatenate X with the text prompt embedding P RLXD obtain the final prompt embedding P' E R(L+N)x (with N 3 in our experiments). At inference, users can steer illumination solely via the text prompt, as X' has learned a standalone representation of lighting priors, eliminating the need for explicit HDR map input.
Integrate 3D Geometry Guidance. We extend ControlNet [17] in IllumiCraft by using the 3D tracking video Vgro E R x3 as additional conditioning signal. We encode Vgoo with the
VAE encoder EVAE to produce geometry latents 5y = EVAE (Vgeo) E R x * x16 To inject these signals into DiT. we clone the first 4 blocks of the pretrained 32-block denoising transformer, forming a lightweight "condition DiT": at each cloned block, we pass its output through a zero-initialized linear layer (matching the DiT hidden dimension) and add the result to the corresponding feature map in the main DiT stream, thereby hierarchically fusing geometry information. During fine-tuning, optimize all DìT blocks under the diffusion denoising loss [31], and we have 30% possibility of replacing 3D tracking videos with zero tensors to simulate inference without any 3D tracking input.
3.3 Training
Building on the DiT backbone [11, 2]. we freeze the VAE and CLIP text encoder to retain pretrained priors, optimizing only the DiT blocks and gero-initialized linear layers for stable integration of appearance, geometry, and illumination. Let 3 = {2g. Ze, P'} be the conditional latents and 2, the noisy latent at diffusion step I; we then minimize the diffusion loss by predicting the added noise 6 as:
/| - c0 t, E)]12 E { Ze. P'}.
min 11
(1)
Here eg is the 3D UNet and I denotes the appearance video Vapgr- Conditioning on composite latents enables joint reasoning over geometry, appearance, and lighting. Embedding the pero-p reference frame with text and lighting ensures edits align with the scene and user intent.
3.4 Inference
At inference time, users provide text prompt and an input video, without HDR geometry to relight the scene, optionally including reference images for video customization. IllumiCraft leverages geometric cues (i.e., 3D tracking videos) during training, and it can generate coherent, illuiintioon-wwrre videos. We will release the model and curated dataset to support future research.
4 Experiments
4.1 Baselines and Evaluation Metrics
Our work addresses the task of video relighting. benchmarking our approach against 4 state-of-the-art approaches: IC-Light [9], adapted video by processing each frame independently; IC-Light + V2V [33]. where IC-Light relights only the first frame, and V2V then propagates those changes to later frames while preserving the original content; Relight [8], which natively supports the first 16 frames (we therefore report results on both the full 19-frame video sequence and on its first 16 frames); and Light-A-Viee Video [1], using Wan2. 1.3B [2] as its backbone (the same base model used in our method). All evaluation results are based on each baseline's publicly released code.
We report 5 metrics for evaluation. The visual quality of the generated videos is evaluated by computing the FVD [48]. LPIPS [49] and PSNR [50] scores against the results of the existing methods. Text alignment is quantified as the mean CLIP [51] cosine similarity score between each individual frame and the text prompt. Temporal consistency, on the other hand, is measured as the average CLIP cosine similarity score across every pair of consecutive frames.
4.2 Implementation Details
We collect 170 high-quality, free-to-use videos from Pexels (https: ://www. pexels 1s. com/) for training. Wan2. 1.3B [2] serves as the DiT backbone. Models are trained for 3,000 iterations

---

"A brown rabbit nestled in greenery, "y, soft lighting"
A hooded in dark, shadowy lighf"
IC Ligiig
Any-ay
LAV
Ours
Original Video
Figure 4: Visual results under the text-conditioned setting. We compare IC-LLight [9], V2V
[33], Light-A- Video [8] (abbreviated LAV in the figure), and our proposed method, IllumiCraft.
Table I: Quantitative comparison of text-conditioned video relighting with existing methods.
Method
FVD (L)
LPIPS (t)
PSNR (1)
Text Alignment (t) Temporal Consistency (1)
IC-Light [9]
4914.83
0.7330
8.55
0.3091
0.9508
Any [33] + Light
3857.09
0.6979
11.12
0.2781
0.9808
Light-A-Video [I]
3946.71
0.6754
11.71
0.3020
01.9100
IllumiCraft
2186.40
0.5623
12.03
0.3342
0..9948
4x A6000 GPUs with batch size of 2 and gradient accumulation over 4 steps, taking approximately 90 hours. Videos center-cropped and resized to 720x480 (widthxheight resolution with 49 frames. We use the AdamW optimizer [52] with a learning rate of 4e-5, weight decay of 0.001, and 100 warmup steps. The learning rate follows cosine schedule with restarts. For evaluation, collect an additional 50 high-quality Pexels videos and show the results on this set.
4.3 Comparison with State-of-the-art Methods
Coxt-Conditooned Video Relighting. In this setting, only foreground video and text prompt are provided for relighting. As shown Table 1, our method significantly outperforms prior work, achieving the lowest FVD and improved perceptual quality across all metrics. Compared to baselines like IC-Light [9], Any [33], and Light-A- Video [1], our approach yields sharper visuals, better alignment with text descriptions, and higher temporal stability. For instance, we observe a 43% strongest baseline. Qualitative comparisons
Figure further illustrate the differences: under prompts like "soft lighting" for rabbit or "dark, shadowy light" for a man, IC-Light produces overly smoothed fur, V2V introduces color distortions, and Light- A- A-Video blurs fine details and dampens contrast. In contrast, IllumiCraft preserves fine textures, captures lighting nuances, ensures prompt relevance, and produces ficker-freee, coherent videos.
reduction in FVD compared
Baekground-Conditione Video Relighting. We also evaluate on background-conditione video relighting. where both the input foreground and background are given. Table reports results across multiple baselines. including LCghi [9], V2y [33] + IC-Light, Light-A- AVideo [1], and Relight [8]. Our method consistently achieves superior performance in both short (16-frame) and long (49-frame) sequences. For example, on 49-frame inputs, our method cuts FVD by 37% compared Light-A-Video, while improving perceptual similarity, alignment with prompts, and temporal consistency, On 16- 6-frame sequences, we further improve fidelity and detail preservation.

---

"sunlit blue sea, hazy green coast" A Woman applies pink lipstick under backlighting,
IC-Light
V2V
LAV
RelightVid
Ours
Input Video Background
Figure 5: Visual results under the bookground-conditionne setting. We compare IC-Light [9],
V2V [33], Light-A-Vidhe Video [8], Relight Vid [8] and our proposed method, IllumiCraft.
Table 2: Quantitative comparison of background-conditioned video relighting with existing methods.
denotes results evaluated with the first 16 frames.
Method
FVD (L)
LPIPS (1)
PSNR (t)
Text Alignment (1)
Temporal Consistency (1)
IC-Light [9]
2175.97
0.3049
17.20
0.3037
0.9795
Any [33] + IC-Light
1901.41
0.3447
17.98
0,3021
0.9854
Light-A-Video [L]
1704.63
0.3834
15.64
0,3266
0,9912
RelightVid" [8]
1492.18
0.2989
17.19
0.3055
0.9858
IllumiCraft
1011.08
0.2232
19.78
0.3283
0,9932
IllumiCraft
1072.38
0.2592
19.44
0.3292
0.9945
outperforming Relight [8] in every metric. As shown in Figure 5, when relighting scenes like sunlit blue sea and backlit woman, competing methods struggle with color shifts, exposure mismatches, or excessive smoothing. Instead, IllumiCraft delivers prompt-faithie lighting, clear bobject-bbckgrool separation and highly realistic textures, yielding the most photorealistic and coherent outputs.
4.4 Ablation Studies
Impact of Illumination and Geometry Guidance. In Table 3, we compare using illumination-oni (I) versus illumination combined with geometry guidance (I+G) during training. Incorporating geometry leads to consistent improvements across metrics, including ~18% reduction in FVD and

---

Table 3: Ablation study of adopting different guidance. "T" and "G" denote the illumination and geometry guidance respectively, used during training.
Guidance
FVD (4)
LPIPS ())
PSNR (t)
Text Alignment (1)
Temporal Consistency (1)
G 1157.23
0.2654
18.89
0.2781
0.9932
I
1305.45
0.2816
18.28
0.3211
0.9864
I+G
1072.38
0.2592
19.44
0.3292
0.9945
Table 4: Effect of dropping Xndr (text-only).
Table 5: Effect of dropping Mhdr (background).
Possibility
FVD (4)
TA (t)
TC (1)
Possibility
FVD (4)
TA (1)
TC (1)
40%
2172.35
0.3325
0.9942
40%
1048.24
0.3265
0.9926
50%
2186.40
0.3342
0.9948
50%
1072.38
0.3292
0.9945
60%
2123.23
0.3312
0.9932
60%
1065.21
0.3277
0.9937
70%
2138.35
0.3301
0.9923
70%
1051.14
0.3228
0.9910
better perceptual quality, aligmment, and temporal consistency. These results suggest that geometry offers critical spatial context that complements illumination cues, helping the model better understand surface structure and light interaction. As a result, the relit videos are not only more photorealistic, but also maintain better alignment with the input prompt and show fewer temporal artifacts.
Effect of Dropping Xhar- We study the effect of partially dropping Xhar by varying the drop rate from 40% to 70% and analyzing its influence on FVD, text alignment (TA), and temporal consistency relighting (Tables 4 and 5). A 50% drop rate
(TC) for both text- and background-condie consistently yields the best trade-off between visual fidelity and alignment with text prompts. This shows that a moderate omission rate preserves essential high-dynamic-rane range cues while improving generalization and temporal consistency.
Inference Cost. All experiments were conducted on an NVIDIA A6000 GPU. For 49-frame video at 480, LCght [9] takes 228.31 seconds via frame-wise relighting (about 4.66 seconds/frame),
V2V [33] requires 1033.21 seconds in total (649.10 seconds for DDIM inversion [53] + 384. 11 seconds for frame editing), and Light-A-Video [1] finishes in 645.53 seconds using its progressive light-injection pipeline. In contrast, IllumiCraft completes inference just 105.21 seconds per sample owing to its efficient spatio-temporal latent encoding. When relighting 16-frame vidco at 480, RelightVid takes 23.24 seconds, whereas our model requires only 22.12 seconds, faster while achiving higher visual fidelity. Additional experiments are included in the Appendix.
4.5 Discussion
While geometry guidance offers powerful control over 3D scenes, and even enables broader control- lable video generation tasks, our work tackles the more demanding challenge of video relighting. In video relighting, each frame's 's illumination must remain temporally consistent while adapting to the scene's evolving geometry. To achieve this, IllumiCraft is trained with both 3D-geometry cues and illumination cues, enforcing stable lighting across all frames. We build on DiffusionLight [41] to obtain HDR environment maps as our illumination guidance, thus the performance of DiffusionLight can influece the lighting quality of relit videos. To further reduce artifacts like misplaced shadows, we can extend the current collected video dataset under wider variety of lighting conditions.
5 Conclusion
In this work, present IllumiCraft, a unified diffusion framework that integrates geometry and illumination guidance for controllable video generation. To support high-quality and diverse relighting control, curate scale dataset of 20.170 video pairs, enriched with HDR maps and 3D tracking sequences. Extensive experiments demonstrate the effectiveness of the proposed method.
Limitations and Broader Impact. While our approach enables effective video relighting in diverse scenes, its fidelity depends on the generative prior of the base model. In cases where this prior lacks accurate shading cues or high-frequency details, the output may exhibit artifacts such as texture blurring. In addition, by enhancing illumination realism and temporal coherence, our method may inadvertently increase the credibility of manipulated videos, raising ethical concerns around deepfakes. We encourage future work on safeguards and detection techniques to mitigate potential misuse.

---

References
[1] Yujie Zhou, Jiazi Bu, Pengyang Ling, Pan Zhang, Tong Wu, Qidong Huang, Jinsong Li, Xiaoyi Dong, Yuhang Zang, Yuhang Cao, et al. Light-a-video: Training- free video relighting via progressive light fusion. arXiv:2502.08590, 2025. 2, 3, 4, 6, 7, 8, 9, 21, 22
[2] Ang Wang, Baole Ai, Bin Wen, Chaojie Mao, Chen- Wei Xie, Di Chen, Feiwu Yu, Haiming Zhao, Jianxiao Yang, Jianyuan Zeng, et al. Wan: Open and advanced large-scale video generative models. arXiv:2503.203/44 14, 2025. 5,
[3] Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiaohan Zhang, Guanyu Feng. et al. CogVideoX: Text-to-videe diffusion models with an expert transformer. arXiv:2408.0607 2024. 2,
[4] Rui Xia, Yue Dong, Pieter Peers, and Xin Tong. Recovering shape and spatially-varying surface reflectance under unknown illumination. ACM Transactions Graphics, 35(6):1-12, 2016.
[5] Giljoo Nam, Joo Ho Lee, Diego Gutierrez, and Min H Kim. Practical svbrdf acquisition of 3d objects with unstructured flash photography. ACM Transactions on Graphics, 37(6):1-12, 2018.
2
[6] Kai Zhang, Fujun Luan, Qianqian Wang, Kavita Bala, and Noah Snavely. PhySG: Inverse rendering with spherical gaussians for physics-based material editing and relighting. In CVPR, 2021. 2
[7] Ziqi Cai, Kaiwen Jiang, Shu-Ya Yu Chen, Yu-Kun Lai, Hongbo Fu, Boxin Shi, and Lin Gao. Real-time 3d-aware portrait video relighting. CVPR, 2024.7
[8] Ye Fang, Zeyi Sun, Shangzhan Zhang, Tong Wu, Yinghao Xu, Pan Zhang, Jiaqi Wang, Gordon Wetzstein, and Dahua Lin. RelightVid: Temporal-consisten diffusion model for video relighting. arXiv:2501. 16330, 2025. 2, 3, 6, 7, 8, 21,22
[9] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. IC-Light GitHub Page, 2024. 2, 6, 7, 21,27 22
[10] Mengwei Ren, Wei Xiong, Jae Shin Yoon, Zhixin Shu, Jiamming Zhang, HyunJoon Jung, Guido Gerig, and He Zhang. Relightful harmonization: Lighting-awee awarre portrait background replacement. In CVPR, 2024. 2,3
[11] William Peebles and Saining Xie. Scalable diffusion models with transformers. In 2023.
6.6
[12] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models. In CVPR, 2022. 3
[13] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat GANs on image synthesis. In NeurIPS, 2021. 3
[14] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In NeurlPS, 2020. 3
[15] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv:2010.02502, 2020.
[16] Tim Brooks, Aleksander Holynski, and Alexei A Efros. InstructPix2Pix: Learning to follow image editing instructions. In CVPR, 2023. 3
[17] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to ext-to-image diffusion models. In ICCV, 2023. 3,
[18] Yuanze Lin, Yi-Wen Chen, Yi-Hsuan Tsai, Lu Jiang, and Ming-Hsuan Yang. Text-driven image editing via learnable regions. In CVPR, 2024. 3
[19] Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall. DreamFusion: Text-to-3d using
2d diffusion. 2209:::229.1444400 2022. 3
10

---

[20] Yuanze Lin, Ronald Clark, and Philip Torr. DreamPolisher: Towards high high-quality text-to-3d generation via geometric diffusion. arXiv:2403.17237 2024. 3
[21] Titas Anciukevičius, Zexiang Xu, Matthew Fisher, Paul Henderson, Hakan Bilen, Niloy Mitra, and Paul Guerrero. RenderDiffusion: Image diffusion for 3d reconstruction, inpainting and generation. In CVPR, 2023. 3
[22] Zeyi Sun, Tong Wu, Pan Zhang, Yuhang Zang, Xiaoyi Dong, Yuanjun Xiong, Dahua Lin, and Jiaqi Wang. Bootstrap3D: Improving 3d content creation with synthetic data. 2xXx:2406.00092 2024.
[23] Hansheng Chen, Jiatao Gu, Anpei Chen, Wei Tian, Zhuowen Tu, Lingjie Liu, and Hao Su. Single-stage diffusion nerf: A unified approach to 3d generation and reconstruction. ICCV, 2023. 3
[24] Junshu Tang, Tengfei Wang, Bo Zhang, Ting Zhang, Ran Yi, Lizhuang Ma, and Dong Chen. Make-it-3d: High-fidelity 3d creation from a single image with diffusion prior. In ICCV, 2023. 3
[25] Kangle Deng, Timothy Omernick, Alexander Weiss, Deva Ramanan, Jun- Yan Zhu, Tinghui Zhou, and Maneesh Agrawala. FlashTex: Fast relightable mesh texturing with lightcontrolnet. In ECCV, 2025. 3
[26] Chong Zeng, Yue Dong, Pieter Peers, Youkang Kong, Hongzhi Wu, and Xin Tong. DiLightNet: Fine-grained lighting control for diffusion-basse image generation. In SIGGRAPH, 2024. 3
[27] Peter Koesis, Julien Philip, Kalyan Sunkavalli, Matthias Nießner, and Yannick Hold -eoffroy Lightlt: Illumination modeling and control for diffusion models. In CVPR, 2024. 3
[28] Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with latent diffusion models. In CVPR, 2023.
[29] Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, et al. Make-a-video: Text-to-video generation without text-video data. arXiv:2209. 14792, 2022.
[30] Pengyang Ling, Jiazi Bu, Pan Zhang, Xiaoyi Dong, Yuhang Zang, Tong Wu, Huaian Chen, Jiaqi Wang, and Yi Jin. MotionClone: Training-free motion cloning for controllable video generation. M4Xiv:2406.05336 2024." 3
[31] Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Do- minik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models large datasets. a1111:231..15127, 2023.
[32] Yuanze Lin, Yunsheng Li, Dongdong Chen, Weijian Xu, Ronald Clark, and Philip HS Torr. Olympus: A universal task router for computer vision tasks. arXiv preprint arXiv:2412.09612. 2024.7 3
[33] Max Ku, Cong Wei, Weiming Ren, Harry Yang, and Wenhu Chen. Any V2V: A tuning-free framework for any video-to- video editing tasks. arXiv:2403. 2024. 3, 6, 7, 8, 9, 2,2,,,12 22
[34] Jiazi Bu, Pengyang Ling, Pan Zhang, Tong Wu, Xiaoyi Dong, Yuhang Zang, Yuhang Cao, Dahua Lin, and Jiaqi Wang. BroadWay: Boost your text-to-video generation model in fraening-free way. 10X0v:2410.06221,, 2024.
[35] Jiaxin Cheng, Tianjun Xiao, and Tong He. Consistent video-to-video transfer using synthetic dataset. arXiv:2311.00
2023.
[36] Jay Zhangjie Wu, Yixiao Ge, Xintao Wang. Stan Weixian Lei, Yuchao Gu, Yufei Shi, Wynne Hsu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. Tune-a-video: One-shot tuning image diffusion models for text-to-video generation. In ICCV, 2023.
11

---

[37] Zekai Gu, Rui Yan, Jiahao Lu, Peng Li, Zhiyang Dou, Chenyang Si, Zhen Dong, Qifeng Liu, Cheng Lin, Ziwei Liu, al. Diffusion as shader: 3d-aware video diffusion for versatile video generation control. wrXiv:2501.08844 2025. 3
[38] Yifan Wang, Aleksander Holynski, Xiuming Zhang, and Xuaner Zhang. SunStage: Portrait reconstruction and relighting using the sun as light stage. CVPR, 2023.
[39] Mingming He, Pascal Clausen, Ahmet Levent Taşel, Li Ma, Oliver Pilarski, Wenqi Xian, Laszlo Rikker, Xueming Yu, Ryan Burgert, Ning Yu, et al. DifFRelight: Diffusion-based facial performance relighting. In SIGGRAPH Asia, 2024.
[40] Hoon Kim, Minje Jang, Wonjun Yoon, Jisoo Lee, Donghyun Na, and Sanghyun Woo. Switch- Light: Co-design of physics-driven architecture and pre- pre-training framework for human portrait relighting. In CVPR, 2024. 3
[41] Pakkapon Phongthawee, Worameth Chinchuthakun, Nontaphat Sinsunthithet, Varun Jampani, Amit Raj, Pramook Khungurn, and Supasorn Suwajanakorn. DiffusionLight: Light probes for free by painting a chrome ball. In CVPR, 2024. 4, 9, 20, 24
[42] Sili Chen, Hengkai Guo, Shengnan Zhu, Feihu Zhang, Zilong Huang, Jiashi Feng, and Bingyi Kang. Video depth anything: Consistent depth estimation for super-long videos. arXiv:2501.1237. 12375, 2025. 4, 20
[43] Tianhe Ren, Shilong Liu, Ailing Zeng, Jing Lin, Kunchang Li, He Cao, Jiayu Chen, Xinyu Huang, Yukang Chen, Feng Yan, et al. Grounded sam: Assembling open open-world models for diverse visual tasks. arXiv preprint arXiv:2401.14111 2024. 4
[44] Peiging Yang, Shangchen Zhou, Jixin Zhao, Qingyi Tao, and Chen Change Loy. MatAnyone: Stable video matting with consistent memory propagation. arXiv:2501.1467) 2025.
[45] Xiaowen Li, Haolan Xue, Peiran Ren, and Liefeng Bo. DiffuEraser: diffusion model for video inpainting. arXiv:2501.10011 2025.
[46] Yuxi Xiao, Qianqian Wang, Shangzhan Zhang, Nan Xue, Sida Peng, Yujun Shen, and Xiaowei Zhou. SpatialTracker: Tracking any 2D pixels in 3D space. In CVPR, 2024.
[47] Wenyi Hong, Weihan Wang, Ming Ding, Wenmeng Yu, Qingsong Lv, Yan Wang. Yean Cheng, Shiyu Huang, Junhui Ji, Zhao Xue, al. CogVLM2: Visual language models for image and video understanding. arXiv:2408. 16500, 2024. 5
[48] Thomas Unterthiner, Sjoerd Van Steenkiste, Karol Kurach, Raphael Marinier, Marcin Michalski, and Sylvain Gelly. Towards accurate generative models of video: A metric & challenges. arXiv: 18/2.0177 2018. 6
[49] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreason- able effectiveness of deep features perceptual metric. In CVPR, 2018. 6
[50] Yule Sun, Ang Lu, and Lu Yu. -o-ptteddtt---peiiallllly rectional video. IEEE Signal Processing Letters,
quality evaluation for omnidi- 2017. 6
[51] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et Learning transferable visual models from natural language supervision. In ICML, 2021. 6
[52] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv: 2017.7
[53] Ron Mokady, Amir Hertz, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Null-text inversion for editing real images using guided diffusion models. In CVPR, 2023. 9
12

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction clearly state the claims made, including the contributions made in the paper and important assumptions and limitations.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
Limitations the work are discussed in Section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model sell-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
13

---

Answer:
[NA]
Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: The model architecture is clearly described in Section 3, and the implementa- tion details are included in Section 4.2.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
14

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: The data and code are not publicly available at the time of submission but will be released upon acceptance of the paper to ensure reproducibility of the main experimental results.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (uttps://nips.. cc/
Mablic/guides/Cne SawiiddssCooeSuunisssoooooee
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: http///oooubmssiooooooo
/nips. wwtppblic/guiieeeeeccoocooopcppe
weiaiiisssooooicyyy...ooo for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question:
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification:
The experimental details are specified in Section 4.2.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Prior work in this area typically does not include error bars, and we follow the same convention to ensure fair comparison.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
15

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The information on the computer resources is provided in Section 4.2.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that NeurlPS Code of Ethics https: : /ttp///eeipsp///pe/ee cc/ ctwteeeeeeeiiiieeeeeeeeeeeeeeeee
didn't make into the paper).
9. Code ethics
Question: Does the research conducted the paper conform, in every respect, with the
Answer: [Yes]
Justification: Ethics.
The research conducted in the paper conforms with the NeurIPS Code of
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g.. there special consid- eration due to laws or regulations in their jurisdiction..
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Broader impacts discussed in Section 5,
Guidelines:
The answer NA means that there is no societal impact of the work performed.
16

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Existing assets used in the paper are properly cited,
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
17

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithhaeee thcode. wwwccoo/////o//o datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
No new assets are released at the time of submission.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
18

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [Yes]
The usage of LLMs in prompt generation and video caption generation is described in Section 3.1.
Justification:
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy (https: battps://nette http://inecncc////o/ http:/ppp....ooo.eeeee////////// for what should or should not be described.
19

---

Appendix
A Overview
In the Appendix, we provide the following content:
(a) Details of HDR environment map transformation in Section B.
(b) Details of the presented lighting encoder in Section C.
(c) Additional experimental results in Section D.
(d) Additional visualization results in Section E.
B HDR Map Transformation
Let Io denote the first (base) video frame, from which we extract reference chrome ball image using DiffusiinLLihht [41]. Then we relabel Zo Zo and denote each subsequent frame by I₃ for t [1,T], where T is the total number frames (49 our experiments), We use Video-Depth-Anyiee to provide per-frame depth maps (u, v). Our goal is synthesize the appearance of the chrome ball in each Ze by estimating the camera's 3D motion and warping Ty accordingly.
[42]
B.1 Sparse Feature Tracking
From the base frame Zo, we detect up N 200 reliable 2D corners p (u. & [I using the OpenCV Shi-Tomasi detector and track them into the current frame I, using the OpenCV Lucas-Kanade optical flow, which produces p (ut, of): points with failed tracks missing depth are discarded, leaving a robust set of correspondences for motion estimation.
B.2
3D Motion Estimation via Constrained Affine Fit
, uf)) and 2 Di uf) For each correspondence (p.. p(), we first read the depths 2 (o
from the base and current depth maps, then lift them to the 3D points of normalized camera by X 2! K K [u,. vi, 1] and X 2 K K=
[uf. of. 1]T, where K is the intrinsic matrix. We estimate the rigid affine transform (R, t) by minimizing E, || + - X subject to R R / and det R 1 using the OpenCV estimateAff Seattefffineee ine To enforce temporal smoothness, we dampen the raw 3x4 transform Mss [R| toward the 3x4 identity affine matrix Î via M3d t I+ a (Mza -M) with a = 0.05, and finally re- re-orthogonalize R via SVD while clamping both rotation angle and translation magnitude.
B.3 Warping the Reference Chrome Ball
For each pixel (Io, Ho) in the reference chrome ball image To size (w, h), we compute the 1 W- -H-1 (", v), map (æo, yo) into video-frame coordinates by T. W W. and ye 4 Yo. lift it to 3D via X, davgk- Vo. TJ apply the affine transform
mean depth davg = WH
X = RX, + f, project back via [ul, v', 1] x K KX,, recover warped coordinates z6 = w/- and v'
36 иТЛ and finally use the OpenCV remap to sample Zo at (it 16), producing the warped chrome ball in the current frame.
B.4 Discussion
Our method tracks few key points the chrome ball in 3D using depth maps to directly recover camera motion, avoiding the pitfalls of 2D model fitting. We reduce jitter and ensure smooth results by gently smoothing each new motion estimate and capping its maximum change. Since the chrome ball is nearly spherical, using its average depth introduces only negligible error. This fully automatic pipeline produces accurate, stable warps with minimal manual intervention.
Robustness to HDR/3D Inputs. During training, we utilize high-quality, aligned HDR maps and 3D geometry data to help the model learn robust priors for spatial structure and lighting. However,
20

---

"Turquoise waver crash besalf rocks, dull stormy light""A womaan gazer at lit candle, low-key candlelight"
IC-Light
Any
LAV
Ours
Original Video
Figure 6: Visual results under the text-conditioned setting. We compare IC-Light [9], Any [33], Light-A-Video [8] (abbreviated LAV in the figure), and our proposed method, IllumiCraft.
these auxiliary signals takes the input video, a text prompt, and an optional background image as inputs. It generates relit results solely based priors learned during training, without relying HDR maps 3D tracking test time. This design ensures that the model is inherently robust to any potential HDR or 3D tracking failure inference, including under fast motion.
required at inference. For video relighting evaluation, our model only
Collected Lighting Prompts. To generate truly diverse relit videos, we curated 100 unique lighting prompts (see Table 11 and Table 12). These prompts span both everyday and fantastical illumination scenarios, ranging from simple indoor scenes to dramatic, otherworldly effects. This variety ensures that our model delivers outputs that are both quantitatively robust and qualitatively rich.
C Lighting Encoder
The lighting encoder first reshapes the input HDR video tensor Vhar R x3x32x32X3 (with T 49) into X1 E 4973 then applies a four-layer MLP, each Linear layer followed by LeakyReLU, with dimensions 3072 4096 - 4096 4096 - 4608 (where 4608 3 1536, corresponding to 3 illumination tokens) to produce Yı E 2249x4005 4608 This is reshaped and permuted into Y, e 33x49x 1536 processed by single-layer TransformerEncodee (dmoded 1536, Rhead 8, dimg 2048) yielding Ya E 133 and then passed through a depth-wise Convld followed by LeakyReLUU and squeeze, collapse the temporal axis into the final output Z 3x315366
D Additional Experimental Results
D.1 Comparison with Existing Methods
Rext-Conditioned Video Relighting. Figure qualitatively benchmarks four video relighting methods on two different illumination conditions, dull stormy light and low-key candle light, showing side-by-side comparisons of IC-Light [9], Any [33], Light-A-Vice A-Vieo [1] and IllmmCrrft.. In the coastal scene, LAV yields overly smooth, desaturated outputs, Any introduces temporal jitters and erratic color shifts. IC-Light also causes color shifts and cannot preserve the fine details in the original video frames. In contrast, IllmmCrrft preserves the original structure, faithfully renders
21

---

"A majestic waterfall, "", natural lighting"
A young man with beard, diffuse daylight"
IC-Light
AnyV2V
LAV
RelightVid
Ours
Input Video Background
Figure 7: Visual results under the background-conditionee setting. We compare IC-Light [9],
V2V [33], Light-A-Video [8], Vid [8] and our proposed method, IllumiCraft.
Table 6: Impact of dropping 3D tracking videos (text-only). Possibility
Table Effect of dropping 3D tracking videos (hackground). Possibility
FVD (L)
TA (t)
TC (t)
FVD (4)
TA (f)
TC (1)
10%
2285.32
0.3303
0.9893
10%
1154.32
0.3231
0.9902
20%
2251.21
0.3332
0.9939
20%
1102.21
0.3273
0.9935
30%
2186.40
0.3342
0.9948
30%
1072.38
0.3292
0.9945
40%
2234.35
0.3325
0.9915
40%
1098.35
0.3278
0.9937
prompt-specific cues (e.g., turquoise waves, intimate candlelight glow), and maintains temporal stability without artifacts. This demonstrates superior fidelity and consistency over all baselines.
Biikgrounnd-Cnnittoonn Video Relighting, As shown in Figure 7, we compare two relighting scenarios, majestic waterfall under natural lighting (left) and bearded man under diffuse daylight (right), over four baselines (IC Light [9], V2V [33]. Light-A- Videeo [1] and Relight [8]) and our method. RelightVid introduces banding and creates unnatural illumination
the waterfall. IC Light and Vny preserve the overall brightness, but blur fine details such as droplets, hair, and clothing. Light-A-Video desaturates tones, oversmooths the water spray, and alters the portrait background, causing artifacts. In contrast, our method follows each prompt precisely, achieving high-fidelity waterfall and sharp rock edges with rock-solid frame-to-frame consistency, enhaneing detail preservation and temporal coherence in both scenarios.
22

---

" "An ancient stone castle island, ",, bright chalky spotlight in misty blue haze"
Input Video
Background
Ours
"branch of an apple tree in full bloom, "h, fowering LED floodlight, stark snowy illumination
Imput Video
Background
Ours
Figure 8: Failure cases of IllumiCraft. We show the failure cases generated by our method.
Table 8: Effect of dropping the reference image (lext-only). Possibility
Table 9: Effect of dropping the reference image (background). Possibility
FVD (4)
TA (t)
TC (f)
FVD (L)
TA (1)
TC (t)
5%
2232.46
0.3331
0.9939
5%
1065.83
0.3284
0.9941
10%
2186.40
0.3342
0.9948
10%
1072.38
0.3292
0.9945
20%
2175.23
0.3341
0.9943
20%
1105.28
0.3275
0.9928
30%
2158.32
0.3338
0.9941
30%
1127.32
0.3269
0.9925
D.2 Ablation Study
Impact of Dropping 3D Tracking Videas. We evaluate the impact of randomly dropping 3D tracking videos during training under both text-conditioned (Table 6) and nackground-condittonn (Table 7) settings. A 30% drop rate offers the best balance across visual quality, text alignment, and frame consistency.
the text-conditioned scenario, reduces FVD, increases the text alignment score to 0.3342, and improves the temporal coherence score to 0.9948. In the hackground-conditiones setting, the same drop rate also lowers FVD and boosts text alignment to 0.3292. These results suggest that 30% drop rate consistently yields optimal performance across all key metrics.
23

---

Table 10: Ablation study on per-frame HDRs and our HDR warping method.
HDR
FVD (})
I.PISS (L)
PSNR (t)
Text Alignment (f)
Temporal Consistency (1)
Per-frame
1267.23
0.2689
17.95
0.3102
0,9901
HDR Warping (Ours)
1072.38
0.2592
19.44
0.3292
0.9945
Effect of Dropping Reference Image. As shown in Tables and 9, 10% possibility of dropping reference images during training achieves the best overall trade-off in both text-only and background- only settings. In the text-only condition, while the 10% drop rate does not result the lowest FVD, it delivers the highest text alignment (0.334) and the highest temporal coherence (0.995), making the most balanced choice. In the background-only scenario, the same 10% drop rate lowers FVD to 1072.38, raises text alignment to 0.3292, and maintains coherence at 0.9945. Overall, a 10% drop rate maximizes visual realism, alignment, and frame consistency across both settings.
Effect of HDR Warping. To evaluate the effectiveness of HDR warping in training, we compare our HDR warping strategy with per-frame HDRs estimated by DiffusionLightt [41] under the background- conditioned setting. As shown in Table 10, our HDR warping consistently achieves better results across all metrics, including lower FVD and LPIPS, higher PSNR and text alignment, and improved temporal consistency. These findings confirm that incorporating temporally consistent HDR warping during training provides a significant advantage for video relighting.
D.3 Failure Cases
As shown in Figure 8, in the top example (an ancient stone castle illuminated by bright chalky spotlight in misty blue haze), our relighting sometimes shifts and misaligns the lower foliage, resulting in oversaturated greens. In the bottom example (a flowering apple branch moving in front of towering LED floodlight), when the branch crosses the illuminated region, parts of the floodlight are occluded and mistakenly treated as foreground, causing unwanted changes in the floodlight's appearance. To address these issues, we plan to expand our curated dataset to include more scenes with dynamic occlusions and strong directional lighting.
E
Additional Visualization Results
Figure 9, 10, 11, 12, 13 and 14 show additional relit videos generated by IllumiCraft. Across wide range of custom backgrounds, including varied spotlight arrangements, colored backdrops, and complex scene contents, IllumiCraft adapts seamlessly to diverse illumination scenarios, producing smooth shading transitions, consistent specular highlights, and accurate shadows. These examples demonstrate its versatility in handling challenging lighting conditions while faithfully preserving original background details, resulting in high-quality relit videos without visible artifacts.
24

---

"A hooded man in hard-edged luminous spotlight over a dark floor, suspenseful stage mood"
Input Video
Background
Ours
"An elephant with textured skin, "", razor-sharp blue shafts, futuristic,
Imput Video
Background
Ours
Figure 9: Visual results of IllumiCraft. Our method produces high- fidellty prompt-aligned videos that adapt to diverse lighting conditions, including dramatic spotlight effects.
25

---

"A man in tan coat, ">, praceful, cozy morning glow"
Input Video
Background
Ours
A young Woman in light green sweater, ", soft frosted beams into a dark room
Input Video
Background
Ours
Figure 10: Visual results of IllumiCraft. Our method produces figh-fiielitt, prompt-aligned videos that adapt to diverse lighting conditions, including dramatic spotlight effects.
26

---

"A glowing, translucent sphere, "", crisp radiant overhead, a solemmn, cinematic stage vibe.
Input Video
Background
Ours
"A traditional Russian church, "", warm-white wall sconces, cozy, minimallist gallery feel"
Input Video
Background
Ours
Figure 11: Visual results of IllumiCraft. Our method produces figh-fidelity, prompt-aligned videos that adapt to diverse lighting conditions, including dramatic spotlight effects.
27

---

"Two cheerful young girls, ",, broad cool-teal wash, a tranquil, moody spotlight effect"
Input Video
Background
Ours
"A two-tiered round wooden table, "", slender white spotlights, an eerie, suspenseful scene"
Input Video
Background
Ours
Figure 12: Visual results of IllumiCraft. Our method produces high- fideliiyyytt, prompt-aligned videos that adapt to diverse lighting conditions, including dramatic spotlight effects.
28

---

"A dark wooden desk, ",, soft-edged white beams overlapping on cyan backdrop"
Input Video
Background
Ours
"A stationery airplane wing, '", crisp bright spoflights, minimal, heutral-gallery atmosphere"
Input Video
Background
Ours
Figure 13: Visual results of IllumiCraft. Our method produces high-fidelity, prompt-aligned videos
that adapt to diverse lighting conditions, including dramatic spotlight effects.
29

---

"A man in red jacket and blue beanie, "", clean radiant pendants, minimalist gallery"
Input Video
Background
Ours
"Show-covered pine trees, "", towering LED floodlight, stark snowy illumination'
Input Video
Background
Ours
Figure 14: Visual results of IllumiCraft. Our method produces prompt-aligned videos that adapt to diverse lighting conditions, including dramatic spotlight effects.
30

---

Table Our collected lighting prompts (1-50) relit videos.
#
Lighting Prompt
red and blue neon light sunset over sea
2
3
sunlight through the blinds in the forest, magic golden lit backlighting
4
6
sunset
sunshine, hard light dappled light magic lit, sci-fi RGB glowing, key lighting neon light
8
9
10
magic golden shadow from window, sunshine sunlight through the blinds neon light
12
13
14
15
cozy bedroom illumination natural lighting
16
17
soft lighting
18
candle light
19
pink neon light
20
sunlit
21
warm sunshine
22
warm yellow and purple neon lights neon, Wong Kar-wai, warm
23
24
cyberpunk style and light yellow and purple neon lights sunshine from window moon light soft sunshine dark shadowy light neo punk, city night cyberpunk
25
26
27
29
31
golden hour light
33
blue hour lighting
tungsten light fluorescent office lighting
35
36
street light at night studio spotlight rim light on subject
38
39
bokeh city light at night TV screen glow in dark room modern minimalistic LED glow ambient underlit glow soft dusk lighting
40
41
42
43
44
harsh industrial light
45
warm ambient room light
46
icy blue fluorescent glow mystical twilight shimmer
47
48
low-key candlelight
49
rainy city neon
50
glowing backlight
31

---

Table 12: Our collected lighting prompts (51-100) for relit videos.
#
Lighting Prompt
51
strong lighting
52
rustic lantern light
overcast day glow
54
golden twilight shimmer
55
dull stormy light
56
subtle overhead illumination steely warehouse lighting
58
vintage street lamp
cool-blue spotlights
60
glowing river reflections sunburst window light morning haze glow
61
62
63
afterglow silhouette
64
dawn light shadows
65
broken neon flicker
66
sleek futuristic luminescence soft pastel glow
67
68
urban dusk illumination
69
underwater blue light
70
interior design spotlight backlit street sign glowing neon arches morning sunbeam rustling leaves with sun rays subdued candlelit ambiance serene twilight light
71
72
73
75
77
prismatic light effects diffuse daylight geometric LED array rain-soaked neon reflections subterranean light glow bright white spotlights dazzling sun flare
79
80
81
82
83
84
vibrant festival lights enchanting aurora borealis
85
86
subtle office glow twinkling fairy lights
87
88
chrome and neon reflections fiery red spotlight icy neon glow
89
90
91
sun-dappled forest light
92
electric dreamscape glow irradiated room ambiance scattered light beams colored spectrum radiance mystic foggy illumination urban jungle glow
93
94
96
97
98
warm campfire light bioluminescent glow caustic rippling light
99
100
32


Reviews:
[{"review_id": "1MLe8p5Q14", "reviewer": "Reviewer_CuT5", "summary": "This paper introduces IllumiCraft, a controllable video relighting method that jointly uses HDR maps, synthetically relit foregrounds, and 3D point tracks to guide a diffusion-based video generator. To support training, the authors also build IllumiPipe, an automated pipeline that creates a dataset (20K+ videos) with various annotations like lighting, masks, background, and geometry. The model builds on a DiT backbone and injects lightweight modules for illumination and geometry conditioning.", "questions": "What happens to relighting quality when one or more auxiliary inputs (HDR maps or 3D tracks) are missing or noisy? The authors could include visual comparisons of single‐condition relighting—e.g. illumination‐only versus geometry‐only guidance—to illustrate each modality’s standalone impact.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n1. This paper’s writing is clear and well-structured, walking readers through data collection, architecture, training, and evaluation.\n2. Method is flexible—supports text-only, HDR-guided, or full multimodal inference.\n3. It builds a large-scale automated dataset (20 k+ videos with HDR, relit clips, masks, backgrounds, and 3D tracks).\n\n**Weaknesses**\n1. It does not evaluate robustness when HDR warping or 3D tracking fails under rapid motion.\n2. It lacks ablations that isolate the impact of each modality (relit vs. HDR vs. geometry).\n3. The authors do not address potential conflicts among their multiple conditioning signals— for example, when HDR illumination cues, 3D geometry guidance, and background references are misaligned or contradictory. They should discuss how the model resolves or is robust to such inconsistent inputs.\n4. The paper’s primary contributions lie in constructing a automated  dataset and introducing geometric conditioning, but there is little methodological or architectural novelty.", "was_revised": true, "final_justification": "The authors have addressed many of my concerns. Therefore, I raise the rating to 4.", "rebuttal": "We sincerely appreciate your constructive review, and will carefully address all the concerns you have raised.\n\n---\n\n**W1. Robustness to HDR/3D failure under rapid motion.**\n\n**Response:**  During training, we utilize high-quality, aligned HDR maps and 3D geometry data to help the model learn robust priors for spatial structure and lighting. However, these auxiliary signals are *not required at inference*. For video relighting evaluation, our model only takes the input video, a text prompt, and an optional background image as inputs. It generates relit results solely based on priors learned during training, without relying on HDR maps or 3D tracking at test time. This design ensures that the model is inherently robust to any potential HDR or 3D tracking failure at inference, including under fast motion. We will calrify this point in the revised manuscript.\n\n**W2. Ablations for modality impact.**\n\n**Response:** Extending the ablation study from Table 3 of the main paper, we show a comprehensive ablation study across three settings below: (i) geometry-only guidance (G), (ii) HDR-only guidance for illumination (I), and (iii) joint geometry + HDR guidance (I + G). Results are shown in the following table, where \"I\" and \"G\" denote illumination and geometry cues used during training (not required during inference). Using illumination guidance alone (I) improves text alignment but offers lower perceptual quality and temporal consistency. Geometry guidance alone (G) yields better perceptual quality and temporal stability but less effective prompt alignment. When both cues are combined (I + G), the model achieves the best results across all metrics, demonstrating their complementary strengths and the benefit of joint guidance for robust video relighting. We will add the result of \"G\" to the revised manuscript.\n\n| Guidance | FVD (↓) | LPIPS (↓) | PSNR (↑) | Text Alignment (↑) | Temporal Consistency (↑) |\n|----------|----------|------------|-----------|---------------------|---------------------------|\n| G        |    1157.23      |   0.2654         |  18.89         |     0.2781                |                  0.9932         |\n| I        | 1305.45  | 0.2816     | 18.28     | 0.3211              | 0.9864                    |\n| **I + G**| **1072.38** | **0.2592**  | **19.44**  | **0.3292**            | **0.9945**                  |\n\n**W3. Petential conflicts among conditioning signals.**\n\n**Response:** During training, we ensure that HDR maps, geometric cues and background references are well aligned. To improve robustness, we also introduce stochastic guidance dropout, by randomly omitting HDR or geometry signals during training. This encourages the model to learn flexible priors that are not overly reliant on any single input. At inference, only the input video, a text prompt and an optional background image are needed. The model leverages learned priors to generate consistent output even when HDR or geometric cues are absent. This training strategy ensures robustness in real-world usage.\n\n**W4. Limited methodological/architectural novelty.**\n\n**Response:** While our dataset and conditioning strategies are key contributions, our method also introduces novel components. In summary, we introduce: (1) A large-scale, high-quality dataset comprising paired original and relit videos, HDR maps, and 3D tracking videos. Its scale and quality offer enduring value to the research community. (2) A multimodal training setup with illumination tokens and geometry guidance that allows the model to learn spatially coherent and lighting-aware priors. (3) A training–inference decoupling strategy that learns from multiple modalities during training but can perform inference using only the input video and text prompts. This framework represents a significant advancement in text-driven video relighting, both methodologically and in practical applicability.\n\n**Q1. Impact of missing/noisy HDR or 3D inputs.**\n\n**Response:** Our model is designed to not require HDR maps or 3D tracks at inference. It relies entirely on priors learned during training. While we cannot include visual examples in the rebuttal due to policy constraints, we have conducted single-guidance evaluations: \"illumination-only\" yields smooth lighting but may lead to inferior temporal coherence; \"geometry-only\" delivers strong coherence but yields lighting that does not align well with text prompts. The combined guidance consistently produces both temporally coherent and prompt-aligned relit outputs, even when HDR or geometry signals are absent. We will include these visual comparisons in the revised manuscript."}, {"review_id": "aZARNUsVgo", "reviewer": "Reviewer_icTF", "summary": "This manuscript presents IllumiCraft, a framework achieving end-to-end video relighting through diffusion-based controllable video generation. Compared to existing approaches, this work explicitly incorporates geometric cues for more accurate illumination-geometry interactions, particularly helpful when scene’s geometry changes, which is achieved by injecting 3D tracking video via a ControlNet. In conjunction with this model, they also propose a data collection pipeline, IllumiPipe, to create training data containing both illumination and geometric cues. Comparison shown in the paper exhibits clear quantitative and qualitative improvements than previous arts, while results in the supplementary material demonstrate that the framework is capable of synthesizing visually appealing videos.", "questions": "1.\tI’m curious why to emphasize \"during training\" whenever “leverage geometric cues” is mentioned in paper (even in Section 3.4 Inference). Does this imply geometric cues are leveraged exclusively during training and not in inference? If geometric cues are also utilized during inference, how to handle potential motion in background?\n\n2.\tThe current data curation pipeline relies entirely on augmenting in-the-wild videos. Why not incorporate synthetic data from graphics engines, as IC-light and RelightVid?\n\n3.\tDo you think the integration of geometric cues would also benefit diffusion-based image relighting methods? IC-light has suggested that properly scaling training of diffusion-based image relighting models can already enables accurate geometric inference.", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths:**\n\n1.\tThe idea of integrating geometric cues is reasonable and well-motivated;\n\n2.\tThe paper is overall well-written and easy to follow;\n\n3.\tThe proposed framework outperforms state-of-the-art methods under the same conditions.\n\n**Weaknesses:**\n\n1.\tThe main concern of the reviewer is about the ablation of geometry guidance, which constitutes the core contribution of this work. Since the introduction of geometric cues has logically sound motivation as mentioned in the paper, it is suggested to include qualitative ablation results to provide more insights about how it works and for what scenarios it is especially helpful. Actually, only showing improvements in FVD, LPIPS, PSNR, and CLIP similarity are insufficient to draw a reliable conclusion that the geometric guidance is effective. For example, if the geometry guidance does not work as expected but instead just introduces some noise, it might still bring some quantitative gains as a kind of data augmentation.\n\n2.\tLacking of video comparisons with existing works. There are only image form qualitative comparisons in the current submission, which cannot convincingly compare the temporal coherence. Additionally, L36–38 mentions that existing methods struggle with scene’s geometry changes and this work aims to \"address these shortcomings.\" It is therefore essential to provide video comparisons to support the claimed advantages under such conditions.", "was_revised": true, "final_justification": "The further explanation about missing details and the design choices included in the response addresses most of my concerns. Given that the paper is built on a reasonable motivation and the proposed components are thoroughly validated, I’m willing to raise my final recommendation to accept.", "rebuttal": "We are thankful for your insightful review and the valuable recommendations you shared, and we will carefully address your concerns.\n\n----\n**W1. Ablation of geometry guidance.**\n\n**Response:** We evaluate our model using standard metrics: FVD, LPIPS, PSNR, and CLIP similarity, which assess spatiotemporal coherence, perceptual realism, pixel fidelity, and semantic alignment, respectively. Consistent gains across these metrics indicate that geometry guidance provides meaningful benefits beyond mere data augmentation or noise injection. While visual results cannot be shown due to rebuttal policy, offline ablations reveal that geometry guidance improves spatialtemporal consistency, especially with object motion. In contrast, models trained without geometry guidance may produce relit videos with inconsistent shading across frames. We will include these examples and a more detailed analysis in the revised manuscript to clarify the specific scenarios where geometry cues are especially helpful.\n\n**W2. Video comparisons with existing works.**\n\n**Response:** We acknowledge the importance of evaluating temporal coherence. Due to the rebuttal policy, we are unable to provide additional visual examples at this stage. While the manuscript includes only frame-based comparisons, in Figure 4 (right) and Figure 1 (right) of the supplementary material, we show that our method achieves better temporal stability than other models, particularly in scenes with dynamic geometry such as object motion and interaction, as discussed in L36–38. In addition, we provide video examples (such as 2.gif, 4.gif, 5.gif, and 14.gif) in the supplementary material that demonstrate our model’s ability to handle complex temporal dynamics (e.g., moving light and dynamic shadows). We will include more examples for comparisons in the revised paper. \n\n**Q1. Unclear if geometry is used at inference; background motion handling?**\n\n**Response:** Yes, geometric cues are used only during training, not inference. This allows the model to learn strong spatialtemporal and structural priors from HDR and 3D tracks while keeping inference efficient. At test time, the model requires only the input video, a text prompt and an optional background image. Since geometry estimation can be unreliable during inference, especially with moving backgrounds, the model instead applies learned priors to generate coherent results without extra input.\n\n**Q2. Why not use synthetic data like IC-light or RelightVid?**\n\n**Response:** While synthetic data offers an alternative option, it remains challenging to capture the full diversity of motions and lighting conditions observed in in-the-wild videos. In contrast, real-world footage naturally encompasses a wide range of scene dynamics, object interactions, and illumination variations. Scaling up our pipeline with in-the-wild videos enables broader coverage, improved generalization, and richer real-world dynamics. We will include more discussions in the paper.\n\n**Q3. Could geometry guidance also help diffusion-based image relighting?**\n\n**Response:** Yes, we believe explicit geometric cues can benefit diffusion-based image relighting models as well. For Lambertian surfaces, appearance is primarily governed by the illumination, surface normals (geometry), and albedo. While IC-Light shows that scaling diffusion training with light-transport constraints enables the model to implicitly learn geometric structure, providing explicit geometry can offer more direct and reliable spatial guidance. This can improve lighting accuracy, especially in cases where implicit reasoning may be ambiguous or unstable (e.g., occlusions, fine surface detail). We see this as a promising direction for future diffusion-based image relighting work."}, {"review_id": "TDMSNz3CzV", "reviewer": "Reviewer_mt3V", "summary": "This paper presents IllumiCraft, a video relighting framework based on DiT. The model incorporates multiple control signals—geometry (3D point tracks), lighting (HDR environment maps), text prompts, and background image into a unified generation pipeline. It adopts a DiT backbone with ControlNet-style conditioning and is trained on a large-scale dataset created via the proposed IllumiPipe pipeline. Experiments demonstrate superior performance including perceptual quality, temporal consistency compared to existing baselines.", "questions": "1. Clarify Fig. 3. It looks like the Light-A-Video relit video is used as input to IllumiCraft. If so, what is the model generating? I assume this is only a training-time input to simulate the input foreground video under another lighting condition. Please confirm.\n2. Compare warped HDR vs. per-frame DiffusionLight HDRs. The paper assumes warping is more stable, but no evidence is given. A proper ablation is needed to validate this design choice.\n3. What happens when the lighting changes mid-video? The warping strategy assumes stationary illumination. Please discuss how the model behaves in dynamic lighting scenarios, and provide results with varying lighting within a video.\n4. Are lighting prompts/HDR maps at test time unseen during training?\n5. Include geometry-only ablations in table 3. This would clarify how much 3D point guidance alone contributes.\n6. How does the model handle conflicting inputs? For example, bright background but dark HDR map—what dominates? Any fallback mechanism?\n7. 3D tracking points might not be the best way of providing geometry guidance. An alternative solution can be using feature maps in VGGT, or using depth maps like in DiffusionRenderer. Please add experiments with these other design choices.\n8. Since you are already using ControlNet, why is the main backbone DiT also fine-tuned? Alternative ways could be inputting all condition signals into the main backbone, or putting all condition signals into the copied ControlNet and freezing the main backbone. The current architecture looks pretty weird.", "limitations": "As a relighting paper, the paper does not provide analysis on the quality of different object materials or lighting complexity. The results shown in the paper are mostly diffuse objects, with no evaluation on specular objects, anisotropic material, sub-surface scattering, and global illumination effects like indirect lighting are not very obvious in the shown results, nor analyzed. The shadow is also not properly handled in the method, with a lot of baking artifacts.", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "**Strengths:**\n- Addresses a relevant and practical problem in video editing and content creation.\n- Integration of geometry, lighting, and text guidance into a single architecture.\n- Demonstrated improvements over multiple baselines across quantitative and qualitative metrics.\n- A data processing pipeline for video relighting.\n\n**Weaknesses:**\n- Novelty is limited; mainly integrates known components (IC-Light, ControlNet, DiffusionLight, DaS, etc.).\n- Omits comparison with latest diffusion renderer baselines, especially [CVPR2025] DiffusionRenderer, which is highly similar to the paper’s pipeline.\n- HDR warping from the first frame is unvalidated and may be unreliable.\n- Sparse geometry inputs (3D point tracks) may be suboptimal for relighting tasks.\n- No mechanism to resolve conflicts among background, lighting, and text conditions.\n- Incomplete ablations (e.g., geometry-only setting not shown).\n- Missing key evaluation metrics (e.g., PSNR) in ablation result tables. (table 4, 5 and supp table 3, 4)\n- Fig. 3 is confusing and could mislead readers about the model’s core task. (I believe the input should be “original video” instead of “relit video”, and the output should be “relit video”. The relit video from Light-A-Video should only be serving as the input video under another lighting.)", "was_revised": true, "final_justification": "Thank the authors for their hard work in rebuttal. While some points are resolved, I still believe the novelty is limited. Using DiT over U-Net is not a contribution, especially given that IC-Light has a DiT version based on FLUX. Other parts of the work are basically engineering/combination, and the overall quality is not good in the overall generation community. Limitations like being unable to handle abrupt lighting changes and global illumination effects are also several. So I decided to keep my original rating.", "rebuttal": "We greatly appreciate your time and constructive comments in the review, and aim to address each of your concerns.\n\n---\n\n**W1. Limited novelty: integration of known components.**\n\n**Response:** Our method goes beyond combining existing techniques. First, we adopt a DiT-based transformer backbone rather than the U-Net-style architectures used in IC-Light, enabling better temporal coherence in video generation. Second, we introduce geometry and illumination guidance *during training* to imbue the model with spatial and photometric priors, while keeping inference lightweight. DiffusionLight is only used for HDR extraction in our IllumiPipe, not in generation. Finally, IllumiPipe itself is a scalable automatic pipeline for generating aligned HDR, geometry, and relit video pairs. Together, these contributions represent a novel and practical advancement in video relighting.\n\n**W2. Missing comparison with DiffusionRenderer.**\n\n**Response:** We appreciate this suggestion. While DiffusionRenderer and our work both aim at physically grounded relighting, our approach differs fundamentally: DiffusionRenderer performs inverse rendering by estimating geometry and material buffers (G-buffers), while our method avoids explicit reconstruction and relies instead on learned priors via DiT with conditioning. At inference, our model requires only a video and text prompt, without the need of scene geometry or lighting input, offering greater simplicity and robustness. We will include a detailed comparison with DiffusionRenderer in the revised manuscript.\n\n**W3 & Q2. HDR warping from the first frame is unvalidated and may be unreliable.**\n\n**Response:** We only utilize HDR maps during training on filtered, high-quality sequences; no HDR warping is performed at inference, as the model relies on learned geometry priors for robust results. To evaluate the effectiveness of HDR warping in training, we compare our HDR warping strategy with per-frame HDRs estimated by DiffusionLight on **5,000** randomly sampled videos. As shown in the table below, HDR warping achieves consistently better results across all metrics, including lower FVD/LPIPS, higher PSNR/text alignment, and improved temporal consistency. These findings confirm that using temporally consistent HDR warping during training provides a significant advantage for video relighting, and we will include this experiment in our revised manuscript.\n\n| HDR | FVD (↓) | LPIPS (↓) | PSNR (↑) | Text Alignment (↑) | Temporal Consistency (↑) |\n|----------|----------|------------|-----------|---------------------|---------------------------|\n| Per-frame        |    1252.35      |    0.2678        |    18.12       |    0.3119                 |         0.9905                  |\n|   **Warping**     |  **1054.32** |  **0.2537**    |  **19.65**    |  **0.3301**             |      **0.9948**               |\n\n**W4 & Q7. Sparse geometry (3D point tracks) may be suboptimal. Use of alternative geometry signals like VGGT or depth maps.**\n\n**Response:** We chose sparse 3D tracks for their low overhead and effectiveness in capturing motion cues. As demonstrated in Table 3, this guidance meaningfully improves performance. We also show that our method outperforms existing baselines on video relighting tasks in Tables 1 and 2. While VGGT features or depth maps may offer finer geometry, our setup favors scalability and simplicity. We plan to explore these alternative forms in future work, and will mention this in the revision.\n\n**W5 & Q6. No mechanism to resolve conflicting inputs.**\n\n**Response:** During training, all modalities (text prompt, HDR, geometry, background) are well-aligned, and we apply random dropout of HDR and geometry cues to improve robustness. At inference, users provide only a video, a text prompt, and an optional background image. The model leverages learned priors to resolve minor inconsistencies and produce temporally coherent results. While we currently do not include an explicit fallback mechanism, we find that our model handles mild conflicts effectively in practice.\n\n**W6 & Q5. Missing geometry-only ablation in Table 3.**\n\n**Response:**  As suggested, we conduct the ablation under the geometry-only setting, and extend the ablation study from Table 3 of the main paper below. Geometry-only guidance (G) improves spatial and temporal consistency but offers lower lighting realism and prompt alignment. Illumination-only guidance (I) enhances lighting quality and text alignment but provides weaker spatial structure and temporal stability. Combining both geometry and illumination guidance (I + G) leads to the best results across all metrics, demonstrating their complementary strengths. We will update Table 3 in the manuscript to include the geometry-only ablation.\n\n| Guidance | FVD (↓) | LPIPS (↓) | PSNR (↑) | Text Alignment (↑) | Temporal Consistency (↑) |\n|----------|----------|------------|-----------|---------------------|---------------------------|\n| G        |    1157.23      |   0.2654         |  18.89         |     0.2781                |                  0.9932         |\n| I        | 1305.45  | 0.2816     | 18.28     | 0.3211              | 0.9864                    |\n| **I + G**| **1072.38** | **0.2592**  | **19.44**  | **0.3292**            | **0.9945**   \n\n**W7. Missing PSNR in ablation tables (Tables 4, 5 and Supp Tables 3, 4).**\n\n**Response:** Due to layout constraints, PSNR was initially omitted from these tables to preserve readability. We have now included PSNR in the tables. The updated Tables 4 and 5 are shown below (including the original FVD, TA, TC metrics), *bold font indicates the chosen possibility*:\n\n| Possibility | FVD    | PSNR | TA    | TC    |\n|-------------|---------|------|--------|--------|\n| 40%         | 2151.32 | 11.89   | 0.3321 | 0.9943 |\n| **50%**     | 2186.40 | 12.03    | 0.3342 | 0.9948 |\n| 60%         | 2123.23 | 11.82    | 0.3312 | 0.9932 |\n| 70%         | 2113.23 | 11.75    | 0.3325 | 0.9937 |\n\n| Possibility | FVD    | PSNR | TA    | TC    |\n|-------------|---------|------|--------|--------|\n| 40%         | 1042.22 | 19.31    | 0.3261 | 0.9923 |\n| **50%**     | 1072.38 | 19.44    | 0.3292 | 0.9945 |\n| 60%         | 1065.21 | 19.22    | 0.3277 | 0.9937 |\n| 70%         | 1032.12 | 19.03    | 0.3232 | 0.9912 |\n\nThe updated Tables 3 and 4 of the supplementary material are shown below:\n\n| Possibility | FVD   | PSNR | TA    | TC   |\n|------------|---------|------|---------|---------|\n| 5%         | 2232.46 | 11.79   | 0.3331  | 0.9939  |\n| **10%**    | 2186.40 | 12.03   | 0.3342  | 0.9948  |\n| 20%        | 2175.23 | 11.98   | 0.3341  | 0.9943  |\n| 30%        | 2158.32 | 11.87   | 0.3338  | 0.9941  |\n\n| Possibility | FVD   | PSNR | TA  | TC  |\n|------------|---------|------|---------|---------|\n| 5%         | 1065.83 | 19.40   | 0.3284  | 0.9941  |\n| **10%**    | 1072.38 | 19.44   | 0.3292  | 0.9945  |\n| 20%        | 1105.28 | 19.28   | 0.3275  | 0.9928  |\n| 30%        | 1127.32 | 19.17   | 0.3269  | 0.9925  |\n\nWe will add the PSNR metric to all these tables in the revised manuscript.\n\n**W8 & Q1. Fig. 3 is confusing.**\n\n**Response:** We appreciate the opportunity to clarify this. Figure 3 depicts the training process: the input is a relit video from Light-A-Video (under synthetic lighting), and the output is the original video. HDR maps and geometry are used during training to guide the model in reconstructing natural lighting. During inference, the model takes a user-provided video, a text prompt, and an optional background image, without HDR or geometry, and generates the relit result. We will revise the figure and caption to make this clearer.\n\n**Q3. Warping assumes static lighting.**\n\n**Response:**  Our HDR warping strategy uses the first frame’s HDR map and warps it to all frames in the sequence, so it assumes lighting remains relatively static. To ensure reliability, we filter out sequences with significant lighting changes during data curation. Importantly, HDR maps are only used in training; at inference, the model relies on learned priors and does not require HDR input. While our current design handles moderate variation well, supporting abrupt lighting changes is an important future direction.\n\n**Q4. Are lighting prompts/HDR maps at test time unseen during training?**\n\n**Response:** Yes. While training videos are paired with HDR and prompt variations, the test-time HDRs and prompts are held out. Furthermore, our model does not require HDR maps at inference. The lighting instructions (prompts) used at test time are typically unseen during training, demonstrating the model’s ability to generalize to novel conditions through learned priors.\n\n**Q8. Why fine-tune DiT if using ControlNet?**\n\n**Response:** Although ControlNet provides a flexible mechanism for conditioning, we find that fine-tuning the DiT backbone significantly improves performance. Our input domain (relit foregrounds on gray backgrounds) differs from the original Wan2.1 training distribution. Fine-tuning the backbone helps adapt it to this new data distribution, while ControlNet handles additional guidance. Freezing the backbone led to underfitting in our early experiments. We will clarify this design decision in the final paper."}, {"review_id": "83mCgTL4dP", "reviewer": "Reviewer_znNG", "summary": "This paper introduces IllumiCraft, a diffusion-based framework for controllable video relighting that integrates lighting, appearance, and geometry cues. It takes HDR lighting maps, synthetically relit frames, and 3D point tracks as input to generate temporally coherent videos aligned with user prompts. Additionally, the authors present a large-scale dataset of paired videos with corresponding HDR maps and 3D tracks, enabling broader research in controllable video generation. Extensive evaluations validate the model's effectiveness on the video relighting task.", "questions": "1. Can you provide qualitative ablation examples illustrating the impact of each guidance cue?\n2. The current results suggest strong performance under relatively constrained conditions. Could the authors comment on how IllumiCraft would perform with moving light sources or larger dynamics?\n3. Will the dataset be released?", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The paper is well motivated and focus on an important limitation of current video diffusion models, which is lack of explicit control over lighting with changing geometry.\n2. IllumiCraft effectively fuses appearance, lighting, and geometric cues into a unified framework and leads to high-quality and temporally consistent video outputs.\n3. The introduction of a large-scale, well-annotated dataset (20,170 video pairs with synchronized HDR, 3D tracks, etc.) is a valuable contribution for the community.\n\nWeakness:\n1. The generated videos appear visually coherent, but the paper does not evaluate more complex temporal dynamics, such as moving light sources, dynamic shadows, or object interactions.\n2. While the paper includes quantitative ablations, it lacks qualitative comparisons showing how each of the three input cues (HDR maps, relit frames, 3D point tracks) contributes to visual improvements.", "was_revised": true, "final_justification": "Due to policy constraint, the author cannot include more visual result. I would keep my origin score and hope the author can include the visual result as promised.", "rebuttal": "We greatly appreciate your time and constructive comments. We will make sure to address the points you have raised.\n\n---\n\n**W1 & Q2. Evaluation on complex temporal dynamics (e.g., moving lights, dynamic shadows and object interactions).**\n\n**Response:** We included several videos in the supplementary material (e.g., 2.gif, 4.gif, 5.gif, and 14.gif) that showcase IllumiCraft’s ability to handle complex temporal dynamics such as moving light sources, dynamic shadows, and object interactions. These examples demonstrate that our model maintains coherent lighting effects and object consistency even in challenging scenarios with significant scene and illumination changes. Due to rebuttal policy, we cannot add more examples here, but we will include additional qualitative results covering complex dynamics in the revised manuscript.\n\n**W2 & Q1. Qualitative ablations showing the contribution of each input cue (HDR maps, relit frames, 3D point tracks).**\n\n**Response:** While we are unable to include new visual results in this rebuttal, we will add qualitative ablation comparisons in the revised manuscript. Our analysis indicates that:\n* 3D point tracks are most beneficial in scenes with substantial motion, improving temporal coherence and geometric consistency.\n* HDR maps contribute to lighting accuracy and better adherence to prompt-driven illumination.\n* Relit frames provide strong appearance priors that guide the generation toward realism.\n\nTogether, these cues provide complementary benefits, and combining them yields the best visual results, especially in dynamic scenes involving both motion and lighting variation.\n\n**Q3. Dataset release.**\n\n**Response:** Yes, we will release the collected dataset along with the source code and trained models upon publication."}]

Author rebuttal:

Meta-review: The paper received 3 positive reviews and 1 negative reviews. After rebuttal, most reviewers appreciate the technical contributions of the work, while one reviewer present some concerns about some details in the rebuttal, which has been addressed by the authors in new response.  Based on authors' response and all reviewers' final rating and reviews, I think the novelty and contribution of the work is above the NeruIPS's quality bar and I thus recommend to accept the paper.
Decision: Accept (poster)