TC-Light: Temporally Coherent Generative Rendering for Realistic World Transfer
Authors: Yang Liu
Chuanchen Luo
Zimo Tang
Yingyan Li
yuran Yang
Yuanyong Ning
Lue Fan
Junran Peng
Zhaoxiang Zhang
Venue: NeurIPS 2025 poster ()
Keywords: video relighting
video editing
diffusion
highly dynamic
world-to-world transfer
Primary area: deep_learning

Abstract: Illumination and texture rerendering are critical dimensions for world-to-world transfer, which is valuable for applications including sim2real and real2real visual data scaling up for embodied AI. Existing techniques generatively re-render the input video to realize the transfer, such as video relighting models and conditioned world generation models. Nevertheless, these models are predominantly limited to the domain of training data (e.g., portrait)  or fall into the bottleneck of temporal consistency and computation efficiency, especially when the input video involves complex dynamics and long durations. In this paper, we propose **TC-Light**, a novel paradigm characterized
by the proposed two-stage post optimization mechanism. Starting from the video preliminarily relighted by an inflated video relighting model, it optimizes appearance embedding in the first stage to align global illumination. Then it optimizes the proposed canonical video representation, i.e., **Unique Video Tensor (UVT)**, to align fine-grained texture and lighting in the second stage. To comprehensively evaluate performance, we also establish a long and highly dynamic video benchmark. Extensive experiments show that our method enables physically plausible re-rendering results with superior temporal coherence and low computation cost. The code and video demos are available at our [Project Page](https://dekuliutesla.github.io/tclight/).

Ti-Ltght: Temporally Coherent Generative Rendering for Realistic World Transfer
Yang Liu
Chuanchen Luo Zimo Tang
Yingyan Li
Yuran Yang
Yuanyong Ning Lue Fan Junran Peng*.7 Zhaoxiang Zhang 1.2-
NLPR, MAIS, Institute of Automation, Chinese Academy of Sciences,
Univeesitty of Chinese Academy of Sciences SShnddong University
Shunde Innovation School, University of Science and Technology Beijing Teencett 'Huazhong University of Science and Technology LLinketic
(1iuyang2022, liyingyan2021, lue. fam, zhaoxiang bhaooiaaggzhaan/// zhang) ac u202315173@hust.o edu. yangyuran@bupt. edu. yymingotencent. chuanchen, luodsdu. edu. jrpeng4ever0126.cmmm com
Abstract
Illumination and texture rerendering are critical dimensions for world-to- toodd- transfer, which is valuable for applications including sim2real and real2real visual data scaling up for embodied AI. Existing techniques generatively re-render the input video to realize the transfer, such video relighting models and conditioned world generation models. Nevertheless, these models are predominantly limited the domain of training data (e.g., portrait) or fall into the bottleneck of temporal consistency and computation efficiency, especially when the input video involves complex dynamics and long durations. In this paper, propose TC-Light, novel paradigm characterized by the proposed two-stage post optimization mechanism. Starting from the video preliminarily relighted by inflated video relighting model, it optimizes appearance embedding in the first stage to align global illumination. Then it optimizes the proposed canonical video representation, i.e., Unique Video Tensor (UVT), to align fine-ge ane-grained texture and lighting in the second stage. To comprehensively evaluate performance, we also establish long and highly dynamic video benchmark. Extensive experiments show that our method enables physically plausible re-rendering results with superior temporal coherence and low computation cost. The code and video demos are available at our Project Page
Introduction
Lighting and its interaction with both real and synthetic environments fundamentally shapes how humans-and embodied agents-perreive the world. The ability re-render the illumination and texture (or so-called relighting) of captured image sequences, especially in complex, highly dynamic scenes, critically valuable for various world-to-world transfer use cases like filmmaking (49) and augmented reality [35. Crucially, by re-rendering the CG-simulated or realistic video data used to train embodied agents, can bridge the sim-to-read gap and enable real-to-real transfer, thus unlocking access to massive high-quality data that is essential for stepping towards embodied intelligence.
Despite its importance, the video illumination and texture rerendering remains highly challenging problem, particularly when camera motion is highly dynamic and foreground objects frequently enter and exit scenes, shown in Fig. 1 Most existing generative relighting techniques [60]
35
23
48
29
57
are tailored for static images. As shown in Sec. 4.2] naively inflating them to
Correponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Source
Re lighted
A busy urban street scene, warm, sunny day
Magic lit, sci-fi RGB glowing, studio lighting
Source
Relighted
The room is well lit by sunshine from window
A busy urban intersection, snow, winter. The sky is clear and blue
Source
Relighted
(c) ... The sky clear, suggesting bright,
At home, living room, clean and organized workspace, realistic
sunny day
Figure 1: Relighting results on long videos under various dynamic scenes, averaging 256 frames per clip. Though the video involves frequent changes of foreground objects (row (a)), highly dynamic camera motions (row (b)), the TC-Light realizes consistent and physically plausible relighting results. Row (c) also shows its potential to mitigate the sim2real gap for synthetic renderings.
video model with existing zero-shot strategies struggles to balance the consistency and quality. Moreover, the considerable training cost and scarcity of video lighting datasets hinder fine-tuning pretrained model for this task. Besides, though generative video relighting and world generation
models are emerging, they are either restricted on domain training data [59 10
7 or burdened
by considerable computation overhead 64 [7]
long video, validated in Sec. 4.2
To address the limitations outlined above, we propose TC-Light. We utilize the SOTA image relighting model IC-Light (60] as the baseline, and inflate it to a video model in zero-shot manner with our decayed multi-axis denoising model, which distinguished by the proposed Decayed Noise Weighting and Noise Statistic Alignment. It provides preliminary video relighting result. The core innovation TC-Light lies in two-stage post-optimization framework that substantially improves temporal consistency. The first stage introduces per- per-frame appearance embedding to compensate for exposure discrepancy. It is optimized with photometric loss against the preliminarily relighted video and flow-based loss between adjacent frames. This enforces global illumination consistency and facilitates consequent optimization. The second stage compresses the output to a canonical representation, i.c., Unique Video Tensor (UVT), according priors including optical flow and depth of the source video. UVT is then optimized by minimizing the warping error across decompressed frames while aligning the content with the first stage result. As shown in Tab. 2 our optimization procedure is extremely efficient and introduces minimal VRAM overhead.
To comprehensively assess the effectiveness of our model, we introduce challenging benchmark tailored for complex and highly dynamic scenes. It comprises 58 videos of averagely 256 frames per clip, spanning both indoor and outdoor environments, realistic and synthetic settings, and

---

wide range of lighting and weather conditions. Extensive experiments demonstrate that our method achieves high-quality, temporally consistent video relighting while maintaining low computational overhead, highlighting its great potential for downstream applications such as embodied AI. Our main contributions are as follows:
A novel optimization- based video relighting paradigm for long videos with high and compli- cated dynamics, significantly improving the temporal consistency of the relighting result.
We establish a new long-video relighting benchmark characterized by high motion dynamics and broad scene diversity, covering various environments and data domains.
Extensive experiments validate that our method achieves SOTA performance in producing temporally consistent, naturally relighted videos with minimal computational cost.
2 Related Work
2.1 Learning-based Illumination Editing
Over the past few years, deep neural networks have become one of the main forces behind re-
search in the field of illumination control. Pioneering works [55] 42 [3] train convolutional encoder-decoder networks on light-stage data. The learned prior knowledge enables models to relight a portrait according to the specified light conditions. More recently, large diffusion- generators have gained popularity for illumination editing. Lightlt [3) explicitly conditions the diffusion process on estimated shading and normal maps, giving fin-grained lighting control ability, while SwitchLight 299 incorporates physics-guided architecture to simulate light-surface interactions better. [63 leverage video foundation models to generate realistic lighting variations over a static image. Light [60]. the current state of the art, learns illumination mixture and decomposition from large quantity of data. Building on these advances in image relighting, video relighting has started to gain traction. [0] learns to disentangle light and intrinsic appearance portrait videos. (7] represents talking faces as relightable NeRFs guided by predicted albedo and shading features. Extending ICgLight, Light-A-Video [64] introduces zero-shot tross-frame attention modification, while RelightVid [17] trains a temporally inflated IC-Light with a carefully designed video relighting dataset. However, these methods either restricted portrait scenarios or struggle with computational efficiency on long videos. In contrast, our model delivers high-quality relighting with strong temporal consistency and low computation cost, even in complex and highly dynamic scenes.
2.2 Diffusion-based Video Editing
The diffusion model (20 has become the go-to model for visual domain transfer and content editing.
Based on training paradigms, recent advancements can be grouped into three categories: (i) training- based models extend pretrained image diffusion models with temporal layers and are trained on
large-scale video datasets, such as B7 integrate depth and flow cues for improved consistency and control. (ii training-free models mainly rely on cross-frame attention to enforce temporal coherence. TokenFlow [9] and FLATTEN [12
38 46
58). CCEdit [18] and FlowVic [36] further
guide attention using estimated optical flow. RAVE (24) enhances latent interactions by denoising over a reorganized latent grid, while Slicedit [1 uses spatiotemporal slices to inject motion priors. VidToMe [34]. on the other hand, exploits temporal redundancy through token merging and unmerging. (iii) one-shot-tuned models typically learn a canonical video representation in a few iterations and propagate its edits across frames. Video [8] learns to represent video as foreground and background atlas. CoDeF 43) learns a hash table and decoding MLP to map frames to a single canonical image. Video-3DGs (53) adapts deformable 3DGS [66 to model input video. Our method combines (ii) and (iii) and proposes an explicit, compact, and efficient canonical representation. i.c., Unique Video Tensor. It enables optimization to be finished within several minutes, which is much faster than 10-30 minutes cost [53] of CoDF and Video-3DGS. Our method also inherits the diffusion model design from training- free algorithms to reduce overall memory and time cost, enabling the processing of long videos.
3

---

3
Denoised
Source Video (
V--E
Intial Noise
stop
"In aftermoo,, Sunset,
(a) Decayed Multi-Axis Denoising X steps
Relighting Results (T,)
Aesrur
photto
L
Te
I
Afgned Results (7.3
Eg
Preliminary Relighting Results (
1,41
Gother
Warp
Scotter
Initialize
Warp
Appearance Embeddings (6,)
E141
Stage II
Stage
Unique rgb
(c) Detailed Illumination Adjustment
Video Tansor
(b) Exposure Alignment
Figure 2: TC-Light overview. Given the source video and text prompt p. the model tokenizes input latents in zy plane and yt plane separately. The predicted noises adaptively combined together for denoising (ef. Sec. (.2). Its output then undergoes two-stage optimization to enhance temporal consistency of illumination and texture, which are respectively detailed in Sec. 3.3.] and Sec. 3.3.2
and
3 Method
In this section, we first introduce the task setting and preliminary knowledge about latent diffusion models S... See. further illustrates how our proposed Decayed Noise Weighting and Noise Statistics Alignment helps effectively lift the image relighting model to video space. Sec. 3.3 details how our key innovation, i.e., the two-stage temporal consistency optimization strategy, helps align overall illumination and texture appearance.
3.1 Preliminaries
Task Setting. As shown in Fig. 2 we take RGB video as input. The axes of the video space-time volume are denoted by (x, y. f), where zy planes correspond to video frames and yt planes are defined as spatiotemporal slices [IT] Since the camera motion highly dynamic, the target illumination can no longer be simply appointed by static image or an HDR environment map. Due to superior flexibility and operability, we use textual prompts as the control signal and relight the entire frame.
Latent Diffusion Models (LDMs)). Denoising Diffusion Probabilistic Models (DDPMs) (20 are class of generative models that aim recover target data distribution through iterative denoising process. Due to the high computational cost of operating directly in pixel space, LDMs 47 50 52 perform diffusion in a lower-dimensional latent space. Given clean image zo, and pretrained autoencoder {-). D(-),, LDMs first encode the image into latent space 2g E(zp). The forward diffusion process then gradually corrupts 2y with Gaussian noise over time steps 1....,
ar - O,6,
where } is monotonically decreasing noise schedule. The reverse process begins from pure noise 2T (), With guidance from control signal (image, text, depth, etc) c, the trained UNet [51] eæ estimates the noise direction and progressively removes the noise from æT. After the final denoising step, the estimated clean latent & is decoded by D(.) to obtain the generated image fn D( which approximates the training distribution.
3.2 Lifting Image Diffusion Model to Video Space
Considering outstand ability in physical plausibility and intrinsic property preservation, we adapt IC-Light [60] into zero-shot video diffusion model. Concretely, we (i) enhance its diffusion blocks

---

to capture spatiotemporal dependencies and (ii) introduce consistency prior from original frames. For (i), we apply the token merging and unmerging technique of VidToMe B4 to self-attention blocks. divides the video frames into chunks and applies intra-chunk local token merging and inter-chunk global token merging, enabling short- and long-term consistency. The derived model ep serves as the basis of (ii). Since it reduces the token count fed to the self-attention module, the computation cost is significantly decreased. For full details, please refer to the original VidToMe paper (34].
For (ii,, we apply multi-axis denoising and adapt it with our Decayed Noise Weighting and Noise Statisties Aligmment strategy. This modified version is named decayed multi-axis denoising. Similar to Slicedit [Ii], the denoiser has two components with shared weights 68 (+.p) that tokenizes each frame and merges tokens from local temporal slots, while c "*) tokenizes the yt planes (cf. ay conditions on target prompt while Eg takes empty prompt as input (making the denoiser unconditional). The noises separately predicted by two parts according the same input latents are combined together [[]
Sec. 3.1) b and merges tokens from local image width slot. Note that Eg
< (..p) - (T000" + V1 -E-0"( ""),
where hyperparameter Y E [0. 1] balances effect from eg. However, the unconditional ex" would overly biases texture and lighting toward the source video, and therefore lead to unnatural relighting results, as validated in Fig. 3 and Fig. 4 To alleviate this problem, we introduce Decayed Noise Weighting, which replaces y with timestep-dependent y that exponentially decays during denoising. To further align predicted noise from eg (AIN) [21] to align noise statistics
use Adaptive Instance Normalization nt that of ayy Eg"
&g (..p)= VTT 27 (1P)) +
11- - 11-7700 "");
(3)
Ml (. un)
Heg
21 (,, un)
+ pe
(4)
where (4s. and a, are the channel-wise mean and standard deviation of each frame. This design preserves motion guidance from the source video while reducing unwanted texture and lighting bias, as validated by ablation studies in Sec. Sec. The output denoised video is denoted as {
3.3 Post Optimization for Temporal Consistency
Although the video diffusion extension in Sec. 3.2 has introduced motion prior from the source video, noticeable illumination and texture flicker persist. To efficiently remove these artifacts, we introduce a two-stage post-optimization framework, as illustrated in parts
awareness and
(b) and (c) of Fig. 2
3.3.1 Stage I: Exposure Alignment
As shown in part (b) of Fig. 2 the first stage introduces a per-frame appearance embedding E, to compensate for exposure misalignment between adjacent frames. Inspired by (27). we model E, 3 x affine transformation matrix, initialized to the identity and optimized via Adam [30]. Its supervision combines a photometric term with a fow-warp alignment term using hyperparameter X
Lerposure (1 Xe) Ephoto (Ir. + dehi froM, Warp (Tes1) @ M.).
(5)
where the homogeneously transformed pixel color I, (a. y) = E, (x, y) 11]" The photometric loss Ephoto is the weighted sum of L1 loss and D-SSIM loss (27), ensuring the transformed frame retains its original content and structure. The second term warps the next frame back to the current timestamp 1, according to forward and backward flows Find,t and Food,r estimated through MemFlow [[5] or provided by the dataset. Then applies an I.1 penalty L1 to align their exposures. To mask out regions with unreliable flow occlusion, apply a soft mask M;
5

---

M, sigmoid (5 (Eflow Ellow) @ sigmoid (8 (Ergb Ergbb))
(6)
Estow = Norm (Fowd,t + Warp (F ffdddd-1)) Engb = 1e - Warp++1-14 (4+11) I.
(7)
Here, 33r is constant scaling factor, Enow and Ergb are thresholds set from the statistics of error map E flow and Ergb. This soft mask is also applied in the second stage of optimization. As shown in Tab. soft masking outperforms the hard one in both temporal consistency and prompt alignment.
3.3.2 Stage II: Optimization over Unique Video Tensor
In the second stage, we refine illumination and texture details. Compared with vanilla video, its canonical representation can incorporate apatial-tempore priors and facilitate consistency 43 53 But popular NeRF or 3DGS are too complex and costly for learning (cf. Sec. (.2 Instead, we compress the video to a me-diieeniional RGB vector of shape (N, Specifically, we define a dddimensional index x( y. t) for each pixel based on priors extracted from the source video. An example index could be [22, 127, O, 255), where the first element is the flow ID (pixels connected by the optical flow predicted by MemFlow share the same flow ID), and the rest are 8-bit quantized RGB values. It is also allowed extend this 4-element index to more elements with voxel coordinate (from depth projection) or any other cues that indicate ppatial-temprrl similarity and locality. All pixels with identical x are gathered via averaging to form one element of the one-dimensional vector, where N is the number of unique K. Take the source video { as an example, the gathering and scattering operations are formulated
shown in part (c) of Fig. 2
U (N.)= Avg (fT (x, ( v, kn)), I (xr, v) U (M(Z,, 1))
(8)
where U is referred to as the Unique Video Tensor (UVT). With an appropriate definition of k, the scattered II:" (xr, y) reconstructs the original I (z, y) with minimal information loss, validated in Tab. 5 For relighting, the ideal edited video frames must preserve consistent motion and intrinsic image details with the source; thus, they share the same index tensor k. for UVT representation.
Accordingly, we compress the first-stage output (zz, y) into U via Eq. (8). which then serves as the primary optimization target. This formulation not only facilitates optimization but also naturally embeds spatial-temporal similarity priors (ef. Sec. 4.3) With CUDA parallelism, the gathering and scattering process can be performed instantly. The optimization of Ü is supervised by
Lunigue = AevLee (i+)+ +(1 Au) LSSIM (ie, Is +
+duli (I, MM, Warp (Te+1) @ M,).
Mc.
where ie (x, = U (M(I, y. ()), and New and A, E [0, 1] balance the loss terms. The total variation loss Liv suppresses noise. Notably, Eq. (9) applies SSIM loss instead of photometric loss. This leaves space to fine-grained appearance and illumination adjustment without altering image structure. Finally, the optimized Ú is used to reconstruct [e(x,, y) according Eq. (8) the final output.
4 Experiments
4.1 Experiment Setting
Implementation Details. Following IC-Light (60), we apply T = 25 sampling steps and a classifier- free guidance scale of 2.0. When inflated to video model with VidToMe [34], the local and global token merging ratios are 0.6 and 0.5, respectively, to accommodate high video dynamics. In our decayed multiaxis denoising strategy, the initial 7 is set to 0.2 and decays exponentially to 0.002 until the final sampling step. For the cost-optimization stages, we use Adam [1] as optimizer and run 35 epochs in the first stage and 70 in the second with batch size of 16, ensuring fast yet sufficient convergence. x(x,, y, () mainly contains quantized RGB and estimated masked flow, and optionally depth if provided. Emperically, the weighting coefficients X.. is set to 0.01, > and X, are set to 0.8. Following (26), the learning rate in the first stage decays from 0.01 0.001, while the second stage uses fixed learning rate of 0.05. Additional details are included in the Appendix.

---

Table 1: Datasets 39
16 54
28
33
contained in established benchmark. N Neg. and
N fromes denote number of sequence and average frames. C, F, D, S respectively denote RGB image. Optical Flow, Depth, Instance Segmentation. Notably, AgiBot here denotes AgiBot Digital World. Due to lacking of extrinsics, its depth is indeed not applicable. Only DRONE is self-collected data.
Datasets
SceneFlow
CARLA
Waymo
NavSim
AgiBot
DROID
InteriorNet
SCAND
DRONE
Agent
Vehicle
Vehicle
Vehicle
Vehicle
Robot
Rohot
Rohot
Rohot
Drone
Synthetic Modality
v
C.F.D.S
C,D,S
C
C,D.S
C
Nacy-
12
5
Frunnes
300
250
305
243
300
289
213
Width
960
960
960
960
960
640
960
1280
Height
512
536
640
536
480
536
480
536
720
SANO
(o) At dusk, bathed in the soft glow of streetlight and building lights
(b) The atmosphare is overzast, contributing to the overall gloomy weather
(c) home, living madern television, robotic realistic
Figure 3: Qualitative comparison of results. The proposed TC-Light avoids unnatural relighting like Slicedit [T] and COSMOS-Transter [3] in (a) and blurring like in (b), or inconsistent illumination like per-frame IC- IC-Light [60] and VidToMe [34] as highlighted by the red squares.
Dataset. To comprehensively evaluate the generation capability, we collect video clips with high motion dynamics and broad scene diversity. This subjective evaluation benchmark, as detailed in Tab. covers scenarios like autonomous driving, robot manipulation, and navigation, well as drone flight. It includes data from synthetic and realistic environments under various weather conditions. Each clip long video with on average 256 frames, making it extremely challenging. To provide more accurate and robust probe on the performance, we also conduct objective ground-trun hound-truth-- based evaluation on the Virtual KITTI 2 dataset (6]. We selected five scenes and relit them match the illumination of morning, sunset, rain, overcast, and fog settings. Each sequence averages 281 frames resolution of 1248x384. To obtain edit prompts, we use some prompts from (60) and generate
others using COSMOS 11.
Metrics. Following prior works 45
24, 58 we assess the relighting performance along following
four dimensions: (i) Temporal consistency is quantified via motion smoothness (Motion-S) [22) and structural warping error (Warp-SSIM), Motion-S evaluates the continuity and physical plausibility of motion in the edited sequence, whereas Warp-SSIM computes the SSIM between a frame and its warped neighbors using flow from RAFT [6). (ii) Textual aligument measured by average CLIP embedding similarity between the text prompt and all edited frames (CLIP-T). (iii) User preference is evaluated by a study on 19 randomly selected videos and valid submissions collected. Participants choose their preferred relighting results among our method and established baselines, from which we derive the Bradley-Terry preference rate (User-PF) ( Additional details are included in the Appendix. (iv) Alignment with Groundtruth is measured using SSIM and LPIPS (26) with ground

---

Table 2: Comparison with existing methods. "OOM" here means the method is unable to finish the task due to an out-of-memory error. For a fair comparison, the base models VidToMe and Slicedit are replaced with IC-Light here. Ours-light applies pnst-optimization to VidToMM,, while Ours-full further introduces decayed multi-axis denoising. Experiments are conducted on 40G A100. The best and the second best of each metric are separately highlighted in red and blue.
Method
Motion-S1
WarpSSIM1
CLIP-TT
User-Po PP
FPSt
Time(s)]
VRAM(G))
IC-Iiight" 60
94.52%
71.22
0.2743
10.97%
0.123
2075
16,49
VidToMe 34
95.38%
73.69
0.2731
6.97%
0.409
626
11.65
Slicedit [+]
96.48%
85.37
0.2653
18.39%
0.122
2101
17.87
VideoDirector
58
OOM
OOM
OOM
OOM
OOM
OOM
OOM
Light-A-Video
64
OOM
OOM
OOM
OOM
OOM
OOM
OOM
RelightVid 17
0OM
OOM
QOM
OOM
OOM
OOM
OOM
Cosmos-I1 3
96.83%
83.47
0.2529
16.06%
0,101
2543
34.87
Ours-light
97.39%
88.53
0.2700
23.66%
0.359
771
14.36
Ours-full
97.80%
91.75
0.2679
23.96%
0.204
1255
14.37
Table 3: Comparison with existing methods on the Virtual KITTI 2 dataset [6]. The symbol definition
aligns with Tab. 2 Experiments are conducted on 40G A100. The best and the second best of each metric are separately highlighted in red and blue.
Method
SSIM
LPIPS
Motion-S
Warp-SSIMt
Time(s)
VRAM(G)
IC-Light* 60
0,5102
0.4470
95.23
68.13
1770
10.25
VidToMe
34
0.5359
0.4262
95.95
71.33
444
6.96
Slicedit |Ti
0,5080
0.4237
96.91
80.74
2346
17.68
VideoDirector
0OM
OOM
OOM
OOM
OOM
OOM
Light-A-Video
64
0OM
0OM
OOM
OOM
OOM
OOM
RelightVid 17
OOM
OOM
OOM
OOM
OOM
OOM
Cosmos-TI
3
0.4833
0.4841
97.81
84.35
3314
34.83
Ours-light
0,5855
0,4026
98.51
90.94
580
15.21
Ours-full 0.5910
0.3971
98.62
92.38
1002
15.21
truth relighted results. These two metrics replace (ii) and (iii) on Virtual KITTI 2 [6] for a more accurate performance evaluation. (v) Computation efficiency is reported in terms of runtime speed (FPS) and peak GPU memory consumption (VRAM) during editing. All experiments on a 40GB A100 GPU. Additionally, to appraise the reconstruction quality of UVT, we report average PSNR. SSIM, and LPIPS between the original and reconstructed frames.
conducted
Baselines. We benchmark our approach against several recent state-of-the-an techniques, whose code is publicly available at the time of writing. These include per- per-frame IC-Light (denoted as IC-Light*)) and its video extensions, Light-A-Vidhs Video [64] and RelightVid [17]. We also implement two IC-Light variants by incorporating leading zero-shot video editing methods: VidToMe [34] and Slicedit ([]. For fairness, we disable the image downsampling to 512 512 resolution before the diffusion step in Slicedit. In addition, we compare two advanced maining-based methods-Videe Video (58) and
COSMOS Transferl B For the latter, due to out-of tit-of--mmory (OOM) issues when applying full multimodal control on long videos, employ only its edge branch, which offers favorable balance between preserving image details and adhering to relighting prompts.
4.2 Comparison with SOTA
Quantitative and qualitative comparisons with state-of-the-art methods are reported in Tab. 2 and Fig. 3 The result indicates that framframe relighting (IC-Light")) follows prompts well and produces physically plausible illumination, but the adapted illumination suffers from severe flicker, as shown in columns (a) and (b) of Fig. IC-Light would even randomly hallucinate non-existent objects in textureless regions (cf. column (c) of Fig. 3. further degrading consistency. Extending IC-Light* with VidToMe R4 yields modest gains in temporal coherence but dramatically lowers computation cost for long videos, so we adopt it as our primary baseline. Slicedit (IT) significantly suppresses flicker and hallucinations, yet its computation overhead exceeds that of IC-Light* Besides,
Tab.
8

---

Table 4: Ablation over module component. The experiments here are conducted on CARLA [T6 and the Interiornet [33 subset, which both provide depth and instance mask as priors. There are 13 sequences in total and 254 frames on average, covering scenes of indoor and outdoor scenarios. The gray row denotes modification that is aborted and not included in the following experiments.
Method
Motion-St
WarpSSIMr
CLIP-TI
FPS1
Time(s)4
VRAM(G)
Baseline
94.51%
77.60
0.2871
0.693
364
10.63
+Ist Stage
95.71%
81.29
0.2868
0.651
388
11.33
+2nd Stage(video)
96.40%
90.58
0.2876
0.552
460
13.53
+2nd Stage(UVT)
96.44%
91.04
0.2866
0.563
449
11.81
+soft mask
96.44%
91.05
0.2868
0.559
452
11.81
from (UVT)
96,30%
90.65
0,2866
0.552
458
12.40
+depth
96.56%
91.12
0.2863
0.569
444
11.57
+instance
96.50%
91.01
0.2851
0.545
462
11.67
+multi-axis
98.41%
95.52
0.2813
0.310
805
11.57
+AIN
98.38%
95.44
0.2832
0.310
805
11.57
+weight decay
97.75%
93.74
0.2865
0.310
805
11.57
Table 5: Ablation over Unique Video Tensor (UVT). Here, %Cmpr is the compression rate after applying UVT on the source video. The subscripts "f" and "f+d" indicate that, besides color cues, the UVT representation incorporates optical flow cues and both flow and depth cues, respectively.
Scene
&Cmpry4 |
SSIMyt
PSNRyt
LPIPS,4
#Cmpry4ad
SSIM,+#f
PSNRf+47
LPIPS
CARLA $339.
0,9940
50.71
0.025
%:29.2
0.9925
48,98 40.86
0.028
InteriorNet
$649.0
0,9908
46.17
0.021
%12.8
0.9755
0.047
its output remains overly biased by the original appearance of the source video. As a result, produces unnatural relighting in many cases, as shown in column (a) of Fig.
We also evaluated the T2V-model-based video editing approach 58 and concurrent video relighting
techniques [64] [7]. Unfortunately, they all failed on long clips due to OOM errors caused by high computation resource demands. For the same reason, Cosmos-Transferr 3 can only operate in single-modality mode under GPU constraints, yet still requires over 30 GB GPU memory and more than 30 minutes per clip. Moreover, on video with high dynamics, it suffers from more severe blur and loss of details, as shown in columns (a) and (b) of Fig. These failures are likely because Cosmos- Transferr is limited to the data domain of its training data, which contains less varied, moderately dynamic videos.
In contrast, our TC-Light first enables physically plausible relighting on long videos with high dynamics, while outperforming all baselines in temporal consistency and preference rate by large margin, as shown in Tab. 2 Tab. alsso demonstrates that our model outperforms all baselines in perceptual and structural similarity with ground truth relighting results. Considering computation cost, the light version adds only 2.4 minutes and 2.7 GB of VRAM overhead compared to the VidToMe baseline (cf. Tab. 2). while faithfully preserving object identity, albedo, and adherence to text prompts, as shown Fig. Incorporating our decayed multi-axis denoising further enhances temporal coherence, with modest trade-off in efficiency and quality. Limited by page, we provide additional visualization and performance different scenarios types the Appendix. And the video demos and comparison can be found on our project page
4.3 Ablation
This section analyzes the contribution of each component in our model. The first stage optimization, as shown in Tab. and Fig. 4 markedly boosts consistency by aligning cross-framd exposure. The 6-7th rows of Tab. also illustrate that, initializing UVT optimization from the first-stage results converges more efficienlly than directly optimizing UVT for the same overall epochs. The second stage optimization, as shown in Tab. 4 further reinforces temporal coherence. Tab. 5 confirms that UVT can compress the source video with near-zero loss, which underpins our design in Sec. 3.3.2 Using UVT as the second-stage target not only boosts consistency but also cuts computational overhead. Additionally, replacing hard mask with soft mask consistently improves both Warp-

---

Source
Basellne
+3.t Stage
-2nnd Stage
-softdddepth
-AIN&decca
e
heefDe ii
eeclDe
coffce day.
coffee
Gollce day.
all day.
DRY
BBD
OBI
BOB
Kiven 1110
Koreo tho0 on knnsee
Sevo
day.
day.
BDS N
eitaoo
Ursioo
coSTee
J.
a alrx.
all
Figure 4: Ablation on main module components. The experiment is conducted on one sequence of the InteriorNet (33) subset, where the text prompt is "This video showcases modern interior space, which is dimly lit". The baseline here denotes VidToMe (3 in Tab.
SSIM and CLIP-T metrics, demonstrating its importance. Incorporating the depth cues alongside UVT yields more compact representation (also in Tab (5). which aids illumination alignment and release computation burden. contrast, instance segmentation masks provide no clear benefit and thus omitted from the final implementation.
For the diffusion module, multi-axis denoising notably enhances temporal consistency. However, tends to inherit appearance distribution from the source video, causing drift from the target prompt and sometimes unnatural lighting, as shown in Fig. 4 The introduced AIN and weight decay mitigate these issues, achieving promising balance between consistency and faithful prompt alignment.
4.4 Limitation and Discussion
Despite achieving impressive results, our method is still limited by its base models. For instance, the current version of IC-Light [60] still struggles relight hard shadows or make large modifications to low-light images. Similarly, since IC-Light is pretrained on 512 resolution and fine-tuned on 1024 resolution, our model struggles to preserve image details if the resolution is lower than 512. For instance, downscaling the NavSim subset from 960x536 to 480x264 causes Warp-SSIM to drop from the average value 90.46 88.36, although CLIP-T remains at 0.304. Besides, since the optimization process relies on the optical flow estimation model. artifacts sometimes occur in textureless areas or under very fast motion, where the flow becomes unreliable. For instance, when downsampling videos from the NavSim subset by 4 times to simulate very fast motion, the CLIP-T and Motion Smoothness fluctuate less than 2%, while Warp-SSIM declines by 5%. Furthermore, the temporal consistency loss has the tendency to smooth the texture of flickering areas, and therefore might sacrifice some details. It can be observed by comparing our model with IC-Light* in part (c) of Fig. 3 Though the proposed decayed multiaxis denoising alleviates the problem, developing temporally more consistent and computationally more efficient denoising strategy is desired future work.
Conclusion
In summary, we present TC-Light, a one-shot-tuned framework that delivers temporally consistent and physically plausible relighting on long, highly dynamic videos. The optimization-baz based illumination alignment provides new paradigm for video relighting. Central our approach the Unique Video Tensor-an an explicit, canonical, and differentiable video representation that enables highly efficient optimization. Over the established long video relighting benchmark, TC-Light achieves state-of-the-an performance in both consistency and efficiency, endowing it with value and potential for broader application areas such as sim2real and real-world video scaling in embodied Al training and validation pipelines.
10

---

Acknowledgments
This work was supported in part by the National Natural Science Foundation China (No. U21B2042, No. 62320106010) and the Guangdong Provincial Foshan Joint Funds (No. 2024A1515 10065).
References
[1] Niket Agarwal, Arslan Ali. Maciej Bala, Yogesh Balaji, Erik Barker, Tiffany Cai, Prithvijit Chattopadhyay. Yongxin Chen, Yin Cui, Yifan Ding. et al. Cosmos world foundation model platform for physical ai. arXiv preprint arX1::2501.33575, 2025.
[2]
Qingwen Bu, Jisong Cai, Li Chen, Xiuqi Cui, Yan Ding, Siyuan Feng. Shenyuan Gao, Xindong He, Xuan Hu, Xu Huang, Shu Jiang, Yuxin Jiang, Cheng Jing. Hongyang Li, Jialu Chiming Liu, Yi Liu, Yuxiang Lu, Jianlan Luo. Ping Luo, Yao Mu, Yuehan Niu, Yixuan Pan, Jiangmiao Pang. Yu Qiao, Guanghui Ren, Cheng Ruan, Jiaqi Shan, Yongjian Shen. Chengshi Shi, Mingkang Shi, Modi Shi, Chonghao Sima, Jianheng Song, Huijie Wang, Wenhao Wang, Dafeng Wei, Chengen Xie, Guo Xu, Junchi Yan, Cunbiao Yang. Lei Yang, Shukai Yang. Maoqing Yao, Jia Zeng. Chi Zhang, Qinglin Zhang, Bin Zhao, Chengyue Zhao, Jiaqi Zhao, and Jianchao Zhu. Agibot world colosseo: A large-scale manipulation platform for scalable and intelligent embodied systems. arXiv preprint X1v:2550300666 2025.
AgiBol-Wortd-onttbbuue
[3] Hassan Abu Alhaija. Jose Alvarez. Maciej Bala, Tiffany Cai, Tianshi Cao, Liz Cha, Joshua Chen, Mike Chen, Francesco Ferroni, Sanja Fidler, et al. Cosmos Cosmos-transferl: Conditional world generation with adaptive multimodal control. arXiv preprint 2r5vv /4492, 2025.
[4] Shrisha Bharadwaj. Haiwen Feng, Victoria Abrevaya, and Michael J. Black. Genlit: single-image relighting as video generation, 2024.
Reformulating
[5] Ralph Allan Bradicy and Milton E Terry. Rank analysis incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4)3324-33 345, 1952.
[6] Yohann Cabon, Naila Murray, and Martin Humenherger. Virtual kitti 2020.
[7] Ziqi Cai, Kaiwen Jiang, Shu-Yo Chen, Yu-Kun Lai, Hongbo Fu, Boxin Shi, and Lin Gao. Real-time 3d aware portrait video relighting. In Proceedings of the IEEE/CVF Conference Computer Vision and 2024.
Pattern Recognition, pages
[8] Wenhao Chai, Xun Guo, Gaoang Wang, and Yan Lu. Stablevideo: Text-driven consistency-aware diffuusin video editing. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 23040-23050, 2023.
[9] Weifeng Chen, Jie Wu, Pan Xie, Hefeng Wu, Jiashi Li, Xin Xia, Xuefeng Xiao, and Liang Lin. Control-a- vidco: Controllable text-to-video generation with diffusion models, 2023.
[10] Jun Myeong Choi, Max Christman, and Roni Sengupta. Personalized video relighting with at-home light stage. In European Conference on Computer Vision, pages 394 410. Springer, 2024.
[11] Nathaniel Cohen, Vladimir Kulikoy, Matan Kleiner, Inbar Huberman Huberman-Spiceelglee Spiegelglas, and Tomer Michaeli. Slicedit: Zero-shot video editing with eextto-imaae diffusion models using ppatio-tempont slices. In Proceedings the 41st International Conference Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 9109-9137. PMLR, 21-27 Jul 2024.
[12] Yuren Cong, Mengmeng Xu, Christian Simon, Shoufa Chen, Jiawei Ren, Yanping Xie, Juan-Manuce Rua, Bodo Rosenhahn, Tao Xiang, and Sen He Flatten: optical flow-guided attention for consistent text-to-vidco editing. arXiv preprint arXiv:2310.05922, 2023.
[13] Sourya Dipta Das, Nisarg A Shah. Saikat Dutta, and Himanshu Kumar. Dsrn: efficient deep network for image relighting. In 202/ IEEE International Conference on Image Processing (ICIP), pages 2788-2792. IEEE, 2021.
[14] Daniel Dauner, Marcel Hallgarten, Tianyu Li, Xinshuo Weng, Zhiyu Huang, Zetong Yang, Hongyang Li, Igor Gilitschenski, Boris Ivanovic, Marco Pavone, Andreas Geiger, and Kashyap Chitta. Navsim: Data-driven non-reactive autonomous vehicle simulation and benchmarking. In Advances in Neural Information Processing Systems (NeurIPS), 2024.
[15] Qiaole Dong and Yanwei Fu. Memflow: Optical flow estimation and prediction with memory. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19068 19078, 2024.
11

---

[16] Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. Carla: An open urban driving simulator. In Conference on robot learning, pages 1-16. PMLR, 2017.
[17] Ye Fang. Zeyi Sun, Shangzhan Zhang. Tong Wu, Yinghao Xu, Pan Zhang. Jiaqi Wang. Gordon Wetzstein, and Dahua Lin. Relightvid: Semporal-consister diffusion model for video relighting. arXiv preprint arXiv:2501. 16330. 2025.
[18] Ruoyu Feng. Wenming Weng, Yanhui Wang. Yuhui Yuan, Jianmin Bao, Chong Luo, Zhibo Chen, and Baining Guo. Coedit: Creative and controllable video editing via diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 6712-67777770 6722, 2024.
[19] Michal Geyer, Omer Bar-Tal, Shai Bagon, and Tali Dekel. Tokenflow: Consistent diffusion features for consistent video editing. arXiv preprint arxiv:2307. 10373, 2023.
[20] Jonathan Ho, Ajay Jain, and Picter Ahbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851,, 2020.
[21] Xun Huang and Serge Belongie. Arbitrary style transfer in real-time with adaptive instance normalization. In Proceedings of the IEEE international conference on computer vision, pages 1501-1510, 2017.
[22] Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianxing Wu, Qingyang Jin, Nattapol Chanpaisit, et al. Vbench: Comprehensive benchmark suite for video generative models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 21807-21818, 2024.
[23] Haian Jin, Yuan Li, Fujun Luan, Yuanbo Xiangli, Sai Bi, Kai Zhang, Zexiang Xu, Jin Sun, and Noah Snavely. Neural gaffer: Relighting any object via diffusion. In Advances in Neural Information Processing Systems, 2024.
[24] Ozgur Kara, Bariscan Kurtkaya, Hidir Yesiltepe, James M Rehg, and Pinar Yanardag. Rave: Randomized noise shuffling for fast and consistent video editing with diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6507-6516, 2024.
[25] Haresh Karnan, Anirudh Nair, Xuesu Xiao, Garrett Warnell, Sören Pirk, Alexander Toshev, Justin Hart, Joydeep Biswas, and Peter Stone. Socially compliant navigation dataset (scand): A large-scale dataset of demonstrations for social navigation. IEEE Robotics and Automation Letters, 2022.
[26] Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphicx, 42(4). July 2023.
[27] Bernhard Kerbl, Andreas Meuleman, Georgios Kopanas, Michael Wimmer, Alexandre Lanvin, and George Drettakis. A hierarchical 3d gaussian representation for real-time rendering of very large datasets. ACM Transactions on Graphics (TOG), 43(4):1-15 15, 2024.
[28] Alexander Khazatsky, Karl Pertsch Suraj Nair, Ashwin Balakrishna, Sudeep Dasari, Siddharth Karamcheti, Soroush Nasiriany, Mohan Kumar Srirama, Lawrence Yunliang Chen, Kirsty Ellis, Peter David Fagan, Joey Hcjna. Masha Itkina, Marion Lepert, Yecheng Jason Ma, Patrick Tree Miller, Jimmy Wu, Suncel Belkhale, Shivin Dass, Huy Ha, Arhan Jain, Abraham Lec, Youngwoon Lee, Marius Memmel, Sungjac Park, Ilija Radosavovic, Kaiyuan Wang. Albert Zhan, Kevin Black, Cheng Chi, Kyle Beltran Hatch, Shan Lin, Jingpei Lu, Jean Mercat, Abdul Rehman, Pannag R Sanketi, Archit Sharma, Cody Simpson, Quan Vuong. Homer Rich Walke, Blake Wulfe, Tod Xiao, Jonathan Heewon Yang, Arefeh Yavary, Tony Z. Zhao, Christopher Agia. Rohan Baijal, Mateo Guaman Castro, Daphne Chen, Qiuyu Chen. Trinity Chung. Jaimyn Drake, Ethan Paul Foster. Jensen Gao. David Antonio Herrera, Minho Heo. Kyle Hsu, Jiaheng Hu, Donovon Jackson, Charlotte Le, Yunshuang Li, Kevin Lin, Roy Lin, Zehan Ma. Abhiram Maddukuri. Suvir Mirchandani, Daniel Morton, Tony Nguyen, A Abigail O'Neill, Rosario Scalise, Derick Scale, Victor Son, Stephen Tian, Emi Tran, Andrew E. Wang, Yilin Wu, Annic Xie, Jingyun Yang. Patrick Yin, Yunchu Zhang, Osbert Bastani, Glen Berseth, Jeannette Bohg, Ken Goldberg, Abhinav Gupta, Abhishek Gupta, Dinesh Jayaraman, Joseph Lim, Jitendra Malik, Roberto Martin-Martin, Subramanian Ramamoorthy, Dorsa Sadigh, Shuran Song, Jiajun Wu, Michael C. Yip. Yuke Zhu, Thomas Kollar, Sergey Levine, and Chelsea Finn. Droid: A large-scale wn-the-wild robot manipulation dataset. 2024.
[29] Hoon Kim, Minje Jang, Wonjun Yoon, Jisoo Lee. Donghyun Na, and Sanghyun Woo. Switchlight: Co-desiga of physics-driven architecture and pre-training framework for human portrait relighting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 25096- 25106, 2024.
[30] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun, editors, ICLR (Poster), 2015. 2015.
12

---

[31]
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2017.
[32] Peter Kocsis, Julien Philip, Kalyan Sunkavalli, Matthias Nießner, and Yannick Hold- Geof-Geofrry. Lightit: Illumination modeling and control for diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9359-9369, 2024.
[33] Wenbin Li, Sajad Saeedi, John McCormac, Ronald Clark, Dimos Tzoumanikas, Qing Ye, Yuzhong Huang. Rui Tang, and Stefan Leutenegger. Interiomet: Mega-scale multi-sensor photo-realistic indoor scenes dataset. In British Machine Vision Conference (BMVC), 2018.
[34] Xirui Li, Chao Ma. Xiaokang Yang, and Ming-Hsuan Yang. Vidtome: Video token merging for zero-shot video editing. In Proceedings of the IEEE/CVF Conference Computer Vision and Pottern Recognition, pages 7486 7495, 2024.
[35] Zhengqin Li, Jia Shi, Sai Bi, Rui Zhu, Kalyan Sunkavalli, Miloš Hašan, Zexiang Xu, Ravi Ramamoorthi, and Manmohan Chandraker. Physically-based editing of indoor scene lighting from single image. In European Conference on Computer Vision, pages 555-572 Springer, 2022.
[36] Feng Liang. Bichen Wu, Jialiang Wang, Licheng Yu, Kunpeng Li, Yinan Zhao, Ishan Misra, Jia-Bin Huang. Peizhao Zhang, Peter Vajda, et al. Flowvid: Taming imperfect optical flows for consistent video-to- ti-vo synthesis. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognation, pages 8207-8216, 2024.
[37] Jun Hao Liew, Hanshu Yan, Jianfeng Zhang, Zhongcong Xu, and Jiashi Feng. Magicedit: High-fidelity and temporally coherent video editing, 2023.
[38] Haoyu Ma, Shahin Mahdizadehaghuam, Bichen Wu, Zhipeng Fan, Yuchao Gu, Wenliang Zhao, Lior Shapira, and Xiaohui Xie. Maskint: Video editing via interpolative non- con-autoregressive masked transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7403- 7412, 2024.
[39] N. Mayer, F. 1lg, P. Häusser, P. Fischer, D. Cremers, A. Dasovitskiy, and T. Brox. A large dataset to train convolutional networks for disparity, optical flow, and scene flow estimation. In IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), 2016. arXiv:1512.02134
[40] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes neural radiance fields for view synthesis. Communications of the ACM, 65(1):99-106, 2021.
[41] Eyal Molad, Eliahu Horwitz, Dani Valevski, Alex Rav Acha, Yossi Matias, Yael Pritch, Yaniv Leviathan, and Yedid Hoshen. Dreamix: Video diffusion models general video editors, 2023.
[42] Thomas Nestmeyer. lean-François Lalonde, lain Matthews, and Andreas Lehrmann. Learning physics- guided face relighting under directional light. In Proceedings of the IEEE/CVF Conference Computer Vision and Pottern Recognition, pages 5124-5133 5133,, 2020.
[43] Hao Ouyang, Qiuyu Wang. Yuxi Xiao, Qingyan Bai, Juntao Zhang, Kecheng Zheng, Xiaowei Zhou, Qifeng Chen, and Yujun Shen. Codef: Content deformation fields for temporally consistent video processing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8089-8099, 2024.
[44] Rohit Pandey, Sergio Dits-Escolano, Chloe Legendre, Christian Haene, Sofien Bouaziz, Christoph Rhe- mann, Paul F. Dehevec, and Sean Ryan Fanello. Total relighting: learning to relight portraits for background replacement. ACM Trans. Graph., 40(4):43-1, 2021.
[45] Chenyang Qi, Xiaodong Cun, Yong Zhang, Chenyang Lei, Xintao Wang, Ying Shan, and Qifeng Chen. Fatezero: Fusing attentions for zero-shot text-based video editing. In Proceedings of the IEÉE/CVF International Conference on Computer Vision, pages 15932 15942, 2023.
[46] Bosheng Qin, Juncheng Li, Siliang Tang, Tat-Seng Chua, and Yueting Zhuang. Instructvid2vid: Con- trollable video editing with natural language instructions. In 2024 IEEE International Conference on Multimedia and Expo (ICME), pages 1 6. IEEE, 2024.
[47] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.0612 1(2):3, 2022.
[48] Mengwei Ren, Wei Xiong. Jae Shin Yoon, Zhixin Shu, Jianming Zhang, HyunJoon Jung, Guido Gerig, and He Zhang. Relightful harmonization: Lighting aware portrait background replacement. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6452 6462, 2024.
13

---

[49] Christian Richardt, Carsten Stoll, Neil A. Dodgson, Hans-Peter Seidel, and Christian Theobalt. Co- herent spatiotemporal filtering, upsampling and rendering of rgbz videos. Computer Graphics Forum, 31(2pt1):247-256 May 2012.
[50] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684-10695, 2022.
[51] Olaf Ronneberger. Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical image computing and computer-assise assisted intervention .MICAA 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part III 18, pages Springer, 2015.
[52] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, al. Photorealistic text-to- image diffusion models with deep language understanding. Advances in neural information processing systems, 35:36479-36494, 2022.
[53] Inkyu Shin, Qihang Yu, Xiaohui Shen, In So Kweon, Kuk-Jin Yoon, and I Liang-Chich Chen. Enhancing temporal consistency in video editing by reconstructing vidcos with 3d gaussian splatting, 2024.
[54] Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo. Yin Zhou, Yuning Chai, Benjamin Caine, et al. Scalability in perception for autonomous driving: Waymo open dataset. In Proceedings of the IEFE/CVF conference on computer vision and pattern recognition, pages 2446-2454, 2020.
[55] Tiancheng Sun, Jonathan T Barron, Yun-Ta Tsai, Zexiang Xu, Xueming Yu, Graham Fyffe, Christoph Rhemann, Jay Busch, Paul E Debevec, and Ravi Ramamoorthi. Single image portrait relighting. ACM Trans. Graph., 38(4):79-1, 2019.
[56] Zachary Teed and Jia Deng. Raft: Recurrent all-pairs field transforms for optical flow. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow. UK. August 23-28, 2020, Proceedings, Part II 16. pages 402-419. Springer, 2020.
[57] Yifan Wang, Aleksander Holynski, Xiuming Zhang, and Xuaner Zhang. Sunstage: Portrait reconstruction and relighting using the sun a light stage. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20792-20802, 2023.
[58] Yukun Wang, Longguang Wang, Zhiyuan Ma, Qibin Hu, Kai Xu, and Yulan Guo. Videodirector: Precise video editing via text-to-video models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2025.
[59] Longwen Zhang, Qixuan Zhang, Minye Wu, Jingyi Yu, and Lan Xu. Neural video portrait relighting in real-time via consistency modeling. In Proceedings of the IEEE/CVF international conference computer vision, pages 012-812, 2021.
[60] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Scaling in-the- wild training for diffusion based illumi- nation harmonization and editing by imposing consistent light transport. In The Thirteenth International Conference on Learning Representations. 2025.
[61] Xiuming Zhang, Pratul P Srinivasan, Boyang Deng, Paul Debevec, William T Freeman, and Jonathan T Barron. Nerfactor: Neural factorization of shape and reflectance under an unknown illumination. ACM Transactions on Graphics (ToG), 40(6):1-18 2021.
[62] Yuanqing Zhang, Jiaming Sun. Xingyi He. Huan Fu, Rongfei Jia, and Xiaowei Zhou. Modeling indirect illumination for inverse rendering. In CVPR, 2022.
[63] Yuxin Zhang, Dandan Zheng, Biao Gong, Jingdong Chen. Ming Yang. Weiming Dong, and Changsheng Xu. Lumisculpt: A consistency lighting control network for video generation. 2024.
[64] Yujie Zhou, Jiazi Bu, Pengyang Ling, Pan Zhang, Tong Wu, Qidong Huang, Jinsong Li. Xiaoyi Dong, Yuhang Zang, Yuhang Cao, et al. Light-a-video: Training-fre vidco relighting via progressive light fusion. arXiv preprint arXiv:2502.00 08590, 2025.
14

---

Relighe
Cyberpunk city street at midnight, sci-fi RGB glowing, illuminated the soft glow of neon lights and building lights.
Tokyo, distant sky.
Soveri
Sunset, warm atmosphere.
the environment is dimly lit and bit dark
Saerre
Fallyyhes
Outdoor, in the morning soft light, sunny day.
Palighe
the morning, snowy, with hint sunlight breaking through clouds
Figure 5: Qualitative results on additional long highly dynamic videos.
Appendix
A
Additional Experimental Results
Fig. presents additional visualizations of our relighting results across diverse range of scenarios. Whether under nighttime daytime conditions, in outdoor indoor environments, or from aerial or ground-level viewpoints, the proposed TC-Light method consistently produces temporally coherent and physically plausible illumination edits, demonstrating strong generalization capabilities. It is also worth noticing that the top row demonstrates our model's ability to handle spatially varying lighting. In the middle three images, white car drives from left to right. Initially, it is illuminated by orange street lamps, reflecting an orange hue. As moves right, rear remains orange-lit, while the front becomes blue due to nearby advertising screen. Eventually, the car fully bathed in blue
15

---

(o) the room well lit sunshine from window.
(b) In Tokyo, distant sky.
(c) The atmosphere is overcast, contri- buting to the overall gloomy weather.
Figure 6: Additional qualitative comparison of results. The proposed TC-Light avoids unnatural relighting like Slicedit [IT] and 1 [3] (a) and (b), or temporal inconsistency like per-frame IC-Light [60] and VidToMe 34 as highlighted by the red squares.
Table Comparison on synthetic [39
The average resolutions are respectively 794 x 503 and 960 555, while the frame numbers are 272 and 246. "OOM" here means the method is unable to finish the task due to an out-of-memory error. Ours- light applies post-optimization to VidToMe, while Ours-full further introduces decayed multi-axis denoising. The best and the second best of each metric are separately highlighted in red and blue.
and realistic scenarios [44
28 25
Synthetic WarpSSIM1
Realistic WarpSSIM1
Method
Motion-St
CLIP-TY
Motion-ST
CLIP-T
IC-Light
93.43%
66.02
0.2779
05.14%
77.13
0.2837
VidToMe
94.61 %
69.45
0.2776
95.82%
79.33
0,2815
Slicedit
96.28%
84.90
0,2717
96.38%
88.89
0,2715
VideoDirector
58
OOM
OOM
OOM
OOM
0OM
OOM
Light-A-Video
64
OOM
OOM
OOM
OOM
OOM
OOM
RelightVid 17
OOM
0OM
0OM
0OM
OOM
OOM
Cosmos-TI
96.31 %
80.87
0.2537
96.78%
83.57
0.2659
Ours-light
97. 97.02% 97.36%
88.63
0.2707
97.46%
89.42
0.2816
Ours-full
91.07
0.2695
97.90%
92.67
0.2792
light. This dynamic lighting response indicates our model can correctly handle spatially varying light. Fig. provides qualitative comparisons against state-of-the-art methods across additional scenarios. As shown, our model effectively adheres to textual instructions while generating relighting results that are both natural and temporally consistent.
We also provide corresponding quantitative evaluations on synthetic and real-world scenarios. As reported in Tab, (6) performance on real-world scenes consistently exceeds that on synthetic ones. This diserepancy likely arises from the training data of the video model the foundational image model IC-Light [60], which are biased towards realistic scenes. Furthermore, the higher resolution and richer textures of real-world data mitigate hallucinations in textureless regions and help better preserve the intrinsic details of source frames for IC-Light. Such attributes are particularly critical for consistency of methods with comparatively limited temporal modeling, namely, IC-Light* and VidToMe, which exhibit substantially higher Motion-S and WarpSSIM metrics on real-world videos than on synthetic ones. In contrast, our approach attains state-of-the-an temporal consistency across both scenario types while maintaining favorable balance with prompt adherence.
(3] and
16

---

Table 7: Licenses and video resolution of datasets 39 established benchmark. Notably, AgiBot here denotes AgiBot Digital World. DRONE is our self-collected subset. Sceneflow has no license, but is only allowed for research purposes.
16
54
28
33
251
contained in
Datasets
SceneFlow
CARLA
Waymo
NavSim
AgiBot
DROID
InteriorNet
SCAND
DRONE
Width
960
960
960
960
640
960
640
960
1280
Height
512
536
640
536
480
536
480
536
720
License
N/A
CC-BY
Custon
NC-SA 4.0
CC BY-
NC-SA 4.0
CC BY-
Apache -2.0
Custon +
CCO
N/A
B Details of Assets
In Tab. 7 we summarize the license and resolution for each subset. All source videos are resized and center-cropped to their designated resolutions. Considering the computation source limitation, we keep all frames if the sequence length is shorter than 300, and randomly sample 300 consecutive frames otherwise. Statistics are provided in Table of the main paper. The DRONE subset includes three clips captured using our DJI Mini4 Pro and two additional clips obtained from DroneStock" which released under the CCO 1.0 License. For AgiBot Digital World 2 where the robot's head moves in coordination with its body while performing tasks, relighting is performed from the head-mounted camera view. For each scene of DROID (28). we apply relighting to both the static side camera and the dynamic left wrist camera views. For Waymo (54) and NavSim [14, relighting conducted using the front-facing camera view. Domain balance is maintained between synthetic and real environments (25 vs. 28 videos), also balanced within sub-domains: autonomous driving (12 synthetic, 10 real), robotic manipulation (8 synthetic, 12 real), indoor navigation (5 synthetic, real). The aerial subset is excluded from balance due to limited long dynamic drone videos in the simulation environment and serves mainly for generalization validation
This paper also benefits from the code of IC-Light (60] (Apache-2.0) License), VidToMe 34 (MIT
License), Slicedit [II] (MIT License), VideoDirector (58) (MIT License), Light-A-Vider
(64
(Apache-2.0 License), RelightVid (17) (CC BY-NC-SA 4.0 License), and Cosmos-Transfert Transferl 3 (Apache-2.0 License).
C Additional Implementation Details
For competing methods, we adopt the hyperparameters from their official implementations for VideoDirector (58), Light-A-Vides [64]. RelightVid [17), and Cosmos-TI (3). We replace base models of VidToMe 34] and Slicedit [ with IC-Light [60], and therefore we align their classifier- free guidance scale and diffusion sampling steps with those in 60]. Additionally, we VidToMe's local and global voken-merging ratios to 0.6 and 0.5, respectively, mirroring the setting of our approach. For Slicedit, we adjust the weighting factor y in Eq. (2) from the default 0.2 to 0.05 to better balance temporal coherence and instruction adherence. All other hyperparameters remain at their default values. The modified VidToMe serves as the baseline of our model design.
During implementation, each k(x, v) comprises three components: (1) per-pixel flow ID, (2) quantized RGB color, and, optionally, (3) a world-frand framme voxel coordinate. For (1), flow IDs are derived from the optical flow estimated by the state-of-the-ar MemFlow method (15) and the binary mask obtained by thresholding the soft mask in Eq. (6) of the main paper (values 0.5 are set to I; otherwise 0). In the initial frame, pixels receive unique flow IDs from 0 to HW - 1, where H and W denote image height and width. In subsequent frames, a pixel inherits the flow ID of its predecessor if connected by an unmasked flow; otherwise, it is assigned new ID. This injects motion priors into the UVT representation. For (2), we quantize RGB values to 7 bits, ensuring that all pixels sharing the same UVT element differ by less than 2/255 in any channel. This constraint mitigates erroneous flows that escape the mask and reinforces representation in regions exhibiting view-dependent effects. For (3), when fer-frame depth maps are available, they are reprojected into point cloud using the camera intrinsics and extrinsics to determine world-frame coordinates. This point cloud is then voxelized at a specified voxel size, and each pixel's voxel coordinate is appended
https: htttps:/wwayo..oo httpa//wwwo.comm//pe/tt/tmsa/a//
httpe://dronestook.cco
17

---

to k(z, t), yielding a more compact representation of static regions. For the CARLA [16] and InteriorNet [33] datasets, voxel sizes are set 0.05 and 0.02 m, respectively. Notably, dynamic objects at different timestamps may spatially overlap in 3D, but they remain distinguishable by their flow IDs and quantized RGB colors. Consequently, each object at each timestep represented by a distinct set of UVT elements, while the L temporal consistency loss preserves object identity across frames.
D User Study
conducted an online user study with 78 anonymous participants, evaluating 19 randomly selected video-text pairs from our datasets. The compared methods were IC-Light [60], VidToMe [34], Slicedit [Л. Cosmos-Transferi [3]. Ours-light, and Ours-full. A screenshot of the questionnaire interface is shown in Fig. 7 For each question, methods were anonymized and relighted videos were presented in random order; participants selected the two most preferred results. In compliance with the NeurIPS Code of Ethics, each participant received compensation of $0.70. Besides, we ensured that all collected data remained confidential and was not disclosed to any institutions or individuals.
Since each video spanned 10-20 seconds, completing the questionnaire took on average 13.5 minutes. Submissions requiring less than four minutes were deemed unreliable and excluded, yielding 65 valid responses. Fig. 8 reports the frequency with which each method was chosen among the top two. Our full model achieved the highest preference rate, while the light variant ranked second. Although IC-Light and VidToMe follow instructions well (cf. Tab. of the main paper). their inferior temporal consistency make them much less preferred by users. Finally, we computed Bradley-Terr preference as comprchensive metric of user preference, presented in the Tab. 2 of the main paper.
scores [5]
E Physical Plausibility
This section illustrates how our model maintains physical plausibility. The physical plausibility of our method inherited from LC-hight [60]. which pre-trained on high-quality Light Stage data and has learned physically grounded relighting process. Our main contribution lies in improving temporal consistency without altering the illumination priors embedded in the base model.
As detailed in Sec. 3.2 we introduce video model inflation mechanism based on token merg- ing/unmerging and decayed multi-axis denoising to enable temporal feature-Ievel information ex- change. Since these components do not alter the prior knowledge encoded in the base model, the distribution of edited illumination aligns with that of IC-Light while enhancing temporal consistency.
In Section Sec. 3.3 we propose a two-stage post-optimization strategy. The first stage smooths global exposure transitions using an appearance embedding, following practices in physically- -based rendering methods like NeRF-W W [40] and 3DGS (26). The second stage refines local fluctuations without altering the overall lighting. Thus, our final results maintain the physically plausible qualities of Light, while significantly improving temporal coherence. As shown in Fig, of the main paper and the video results, our illumination remains qualitatively aligned with LC-Light and VidToMe, but with fewer artifacts and greater temporal stability.
Thanks to the strong priors of IC-hight, our method focuses on temporal coherence and computation efficiency. Compared to optimization heavy approaches such as Nerfactor [61] and InvRender [62]. our post-optimization stage takes only around 2 minutes, with the entire pipeline completing in 10 minutes- subutas--substantially faster than training NeRF or 3DGS models, as discussed in Sec. 2.2 of the main paper
F
Social Impact
Positive Impacts. The proposed TC-Light framework for long video relighting stands to benefit a wide range of applications in both industry and research. First, by enabling consistent and physically plausible illumination editing at low computational cost, can substantially lower the barrier to high-quality visual content creation, empowering independent filmmakers, educators, and artists to produce compelling video narratives without access to specialized hardware. Second, the capability to scale Ilumination-divers training data through sim2real and real2real transfer can accelerate
18

---

Questionnaire Survey on Video Relighting Quality
Hello! Thank for taking your precious time to participate in this video relighting quality questionnaire. This questionnaire aims collect your preferances for the relighting effects of different algorithms
understand the advantages and disadvantages among different methods, We guarantee that the collected personal information no: be disclosed.
If your computer available, it is recommended to browse and answer questions on to have better experience.
you feel same videos take too much time to load, you first go back to the previous page and then returm. We recommend logging into WeChat automatically record form filling progress.
Introduction
questionnaire contains 19 questions Each question equipped with one source video, the relighting textual prompt, and six relighted videos from different algorithms, According to the following two dionesions, please select the top two relighting results that best match your preferences (from high low):
Relighting Effect
Whether the relighting effect the requirements the textual promp:
Whether the relighting effect physically plausible
Video Quality
Visual Quality (no blurring, no distortion)
Content Alignment (the essence of the original video is not changed)
Temporal Consistency (no flickering or unnatural transitiors)
Sorting Instructions
Select video clips order of preferred and the secone preferred.
questionnaire only involves video sorting; no additional text filling is required.
The following are respectively original video and results obtained by editing the lighting according to this prompts "Modem em kitchen, matte sink and faucet, shadows cast from the window". ". Please select the two clips that you think have the highest quality in sequence.
Original Video:
Videos Relighted by Different Algorithms:
Figure 7: A screenshot of the user study.
19

---

322
313
Rank
300
First
274
263
Second
250
250
212
Frequency
200
192
154
150
121
103
100
82
50
0
Cosmos-TI
IC-Light*
Ours-full
Ours-light
Slicedit
VidToMe
Figure Results from user study with 65 valid submissions. The methods are arranged in alphabetical order. This figure reports the frequency that each method is chosen the first- and second-most preferred video.
progress in embodied Al-robots and autonomous agents exposed to rich, temporally coherent visual environments may learn more robust perception and planning behaviors, thereby advancing safety and reliability in numan-robor interaction. Finally, by fostering TC-Light may encourage energy-aware design practices in large-scale media processing systems, contributing reduced resource consumption and attendant carbon emissions.
efficient video synthesis pipelines,
Negative Impacts. Despite these benefits, improved video relighting carries potential risks if misused. Enhanced realism in dynamic relighting could facilitate the creation of deceptive multimedia, includ- ing deepfake videos that manipulate shadows and highlights to conceal tampering or impersonate individuals, thereby eroding trust in digital media. Moreover, large-scale deployment of relighting tools raises privacy concerns: adversarial actors might relight surveillance footage to obscure identi- ties or fabricate altered event sequences. To mitigate these harms, advocate for gated access to pretrained models, integration of provenance metadata to flag relit content, and collaboration with platform providers monitor and throttle suspicious bulk relighting requests.
20

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The claims are supported by experiments in Sec. 4 both quantitatively and qualitatively. Since the dataset also covers the main application scenarios of embodied agents, we trust the potential of applying our work in embodied AI.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Please refer Sec.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-specfiiaatinn asymptotic approximations only holding locally). ). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
21

---

Answer: [NA]
Justification:
No theoretical result is involved.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: The method is detailedly illustrated in Sec. 3, and implementation details provided Sec. 4.1 and Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
22

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification:
The full code and dataset is likely to be open-sourced upon acceptance. But
the anonymous link to the partial dataset would be provided in the supplementary.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://mips.cc.
cublic/guuddes/Cdeettbii.. wtt.eeesuuuiiiiioooooooooooo for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
wewpde/e/oooSSSmmissssssie. ssissssliccyy for more details. / //nips. wttps/cccppcccccuui.eeeeeeccc
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Please refer Sec.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Since both the quantitative experiments in comparison and ablation reports mean metrics over 10 video sequences, we trust that the fluctuation caused by noise is sufficiently suppressed.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
23

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative eπτor rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: All the quantitative results are accompanied by the execution time and memory
cost. Since GPU is the main compute workers, we provide its details in Sec. 4.1
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
As far know, there is break with NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Due to the page limitation of the main paper, the discussion about positive and negative societal impacts of the work is included in the Appendix.
Guidelines:
The answer NA means that there is no societal impact of the work performed.

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This technique poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best. faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Due to the page limitation of the main paper, we list the license of used assets in the Appendix.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
25

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
batasets.papersstt
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New Assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We include self-collected drone data and introduce it in Sec. Due the page limitation of the main paper, we put details and the anonymized URL of the asset in the Appendix.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [Yes]
Justification: This paper involves user study. Due to the page limitation of the main paper, the related details are provided in the Appendix.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [Yes]
Justification: Yes, the potential risks are discussed in the Appendix and are disclosed the subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
26

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
27


Reviews:
[{"review_id": "fHZD0w9Ovo", "reviewer": "Reviewer_phEG", "summary": "This paper proposes a method, TC-Light, that tackles the long-standing challenge of editing illumination in long, highly dynamic videos by inflating the single-image relighting model (IC-Light) into a zero-shot video diffusion framework and introducing decayed multi-axis denoising that blends per-frame ( xy ) and time-slice ( yt ) predictions for improved temporal coherence. A subsequent two-stage post-optimization first aligns global exposure through a learned per-frame affine “appearance embedding,” then refines local flicker using a Unique Video Tensor (UVT)—a compact 1-D RGB lookup indexed by flow, depth, and color cues. Qualitative results and some hand-crafted metrics are evaluated to showcase the effectiveness of the proposed method with extensive ablation studies. However, metrics computed against ground truth relight videos are absent due to the scarcity of such data. Plus, there is no qualitative video presented in the supplementary which is somehow important for papers working on video generation.", "questions": "- How sensitive is performance to the composition of κ (e.g., color + flow vs flow + depth)? Did you explore learned index hashing or adaptive bin sizes?\n- The method currently edits entire frames uniformly via a text prompt. Could UVT be extended to support spatially varying lighting (e.g., spotlight on a moving actor / near-field point lights)?\n- Have you tried TC-Light with recent transformer-flow models (e.g., Flowformer)? How does flow noise degrade Warp-SSIM, and could self-supervised flow refinement within stage II help?\n- As mentioned in your Limitations section, given IC-Light’s limitations, how does it perform on those failed corner cases?", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n- The paper is well-written and easy to follow.\n- The evaluations especially ablation studies are extensive, including strong zero-shot and training-based baselines, ablations, user study, and efficiency measurements. \n- By using decayed multi-axis denoising, it is an elegant way to inject motion priors without training, which improves consistency with negligible extra VRAM.\n- The proposed UVT representation is simple yet effective, which improves the visual quality while lower the runtime VRAM.\n\nWeaknesses:\n- Flow dependence and error analysis: Optical-flow errors propagate into UVT, occasionally causing blur or ghosting in low-texture areas; robustness analysis is limited. \n- Groundtruth Benchmark: Though capturing video with different lightings is hard, it is still encouraged to have evaluation against ground truth which is typically lacking in this field of generative video relighting. Moreover, it will provide more accurate and robust probe on the performance.\n- No Video results: As a paper working on video-related generative tasks, I would weigh more on the qualitative results presented in the demo video, which could clearly show the temporal smoothness and overall quality of the relit videos. However, there is no video presented in the supplementary material, which makes it hard to determine the effectiveness of the proposed method.\n- Reproducibility: Will the authors release the code? Exact UVT implementation (index hashing, GPU kernels) may be non-trivial to replicate without a reference.", "was_revised": true, "final_justification": "Most of my concerns have been properly addressed. My acceptance recommendation is conditioned on that the authors will include all the additional discussions, evaluations, and limitations in the revision.", "rebuttal": "We sincerely thank you for the very professional comments. We are inspired and hope our discussion brings more insights.  Below, we address the key concerns, with all citations consistent with the original paper.\n\n## Q1: Results on Groundtruth Benchmark\n| Virtual KITTI 2 | SSIM↑ | LPIPS↓ | Motion-S↑ | Warp-SSIM↑ |  Time(s) | Mem.(G) ↓ |\n| ----------- | :-----------: | :-----------: | :-----------: | :-----------: | :-----------: | :-----------: |\n| IC-Light* [58] | 0.5102 | 0.4470 | 95.23 | 68.13 | 1770 | 10.25 |\n| VidToMe [31] | 0.5359 | 0.4262 | 95.95 | 71.33 | **444** | **6.96** |\n| Slicedit [10] | 0.5080 | 0.4237 | 96.91 | 80.74 | 2346 | 17.68 |\n| VideoDirector [56] | OOM | OOM  | OOM  | OOM  | OOM  | OOM  |\n| Light-A-Video [60] | OOM | OOM  | OOM  | OOM  | OOM  | OOM  |\n| RelightVid [16] | OOM | OOM  | OOM  | OOM  | OOM  | OOM |\n| COSMOS-T1 [3] | 0.4833| 0.4841 | 97.81 | 84.35 | 3314  | 34.83 |\n| **Ours-light** | 0.5855 | 0.4026 | 98.51 | 90.94 | 580 | 15.21 |\n| **Ours-full** | **0.5910** | **0.3971** | **98.62** | **92.38** | 1002 | 15.21 |\n\nWe appreciate the suggestion and agree that ground-truth evaluation remains scarce but essential in generative video relighting.  To this end, we evaluate on Virtual KITTI 2, which renders consistent geometry across varied lighting conditions under autonomous driving scenarios. We selected five scenes and relit them to match illumination of morning, sunset, rain, overcast, and fog settings, using COSMOS prompt upsampler to extract lighting and weather descriptors from the ground truth video as target prompts. Each sequence averages 281 frames at a resolution of 1248×384.\n\nWith access to ground truth, we replace CLIP-T and user preference with SSIM and LPIPS [26]. Results are shown above, following the same symbol definitions as Table 2 in the main paper. Our method outperforms all baselines in perceptual and structural similarity, as well as motion consistency (Motion-S, Warp-SSIM), while remaining efficient. Related video comparisons are provided in Q2.\n\n## Q2: Video Results\nThank you for the suggestion—we fully agree on the importance of video results for assessing temporal quality. Videos with corresponding prompts are provided in the _Video_Comparison_ folder linked anonymously in the supplementary (Line 9). This folder includes seven representative scenarios across both synthetic and real-world settings, including the ground truth benchmark in Q1.  As shown, our method significantly outperforms IC-Light and VidToMe in temporal consistency, avoids the blurriness and unnatural lighting seen in COSMOS-Transfer1, and surpasses Slicedit in instruction adherence. TC-Light consistently delivers high-quality, stable results across dynamic scenes. All videos will be made publicly available on our paper's project page.\n\n## Q3: Reproducibility of TC-Light\nYes, we are planning to release the code upon acceptance. In fact, our code is already in a state where it can be open-sourced at any time. We have also used ChatGPT to improve the quality and readability of the whole repository. We hope the code release can benefit the research community.\n\n## Q4: Alternative Design Choices\n| Components | Motion-S↑ | Warp-SSIM↑ | CLIP-T↑ |  FPS↑ | Mem.(G) ↓ |\n| ----------- | :-----------: | :-----------: | :-----------: | :-----------: | :-----------: |\n| flow+color | 96.44 | 91.05 | 0.2866 | 0.559 | 11.81 |\n| flow+depth | 95.20 | **91.38** | 0.2778 | 0.553 | 11.62 |\n| flow+color+depth | **96.56** | 91.12 | **0.2862** | 0.569 | **11.57** |\n| flow+color+depth+inst. | 96.50 | 91.01 | 0.2851 | **0.545** | 11.67 |\n| flow+color+depth, +vq | 97.25 | 87.98 | 0.2830 | 0.420 | 18.12 |\n\nWe evaluate different compositions of **κ** and the use of **learned index hashing** (via _vector-quantize-pytorch_ library). \"inst.\" and \"vq\" denote instance masks and vector quantization, respectively. Rows 1, 3, and 4 match rows 6, 8, and 9 of Table 3 in the main paper.\n\nColor proves essential in **κ**: flow+depth leads to a notable drop in Motion-S and CLIP-T compared with flow+color+depth. This is because even when pixels are close in 3D or flow-connected, they may belong to different objects or textures—color helps disambiguate such cases and avoid incorrect grouping. And the performance is relative stable to depth and instance mask clues.\n\n**Learned index hashing** breaks spatiotemporal correspondence: we found this process introduced significant overhead while degrading Warp-SSIM and CLIP-T. The learned codebook often failed to capture true spatiotemporal correspondence, grouping unrelated pixels and receiving conflicting supervision. Hence, we excluded it in our model development. \n\n## Q5: Robustness Analysis\n|Flow | Noise Level | Denoise | Motion-S↑ | Warp-SSIM↑ | CLIP-T↑ |\n| ----------- | :-----------: | :-----------: | :-----------: | :-----------: | :-----------: |\n| MemFlow | 0.0 | | 96.56 | 91.12 | 0.2873 |\n| MemFlow | 0.5 | | 96.46 | 89.64 | 0.2863 |\n| MemFlow | 1.0 | | 96.41 | 88.22 | 0.2866 |\n| MemFlow | 2.0 | | 96.32 | 86.17 | 0.2855 |\n| MemFlow | 2.0 | ✔ | 96.37 | 86.13 | 0.2851 |\n| FlowFormer++| 0.0 | | 96.56 | 88.16 | 0.2854 |\n\nHere, we provide quantitative results under different flow conditions. The used data subsets align with that of the ablation study in the paper, and row 1 here comes from row 8 of Table 3 in the main paper. The noise level indicates the standard deviation of Gaussian noise added to the flow. As can be observed, the Motion-S and CLIP-T are relatively stable to the flow noise (drop less than 1%).  From low to high noise, the drop of Warp-SSIM ranges from 2%-5%. Qualitatively, the frames become noisy, and flickering becomes more salient. We have also tried flow denoising supervised by warping error, but it brings 6-10 GB additional VRAM cost, and the results show no clear benefits. We have also tried FlowFormer++ during model development. As validated in [14], FlowFormer++ underperforms MemFlow. And as shown in the Table, a weaker flow estimator would also deteriorate the Warp-SSIM. Therefore, it is excluded from our model design.\n\n## Q6: Extension to Spatially Varying Lighting\nYes, the top row of Figure S1 (supplementary) demonstrates our model’s ability to handle spatially varying lighting. In the middle three images, a white car drives from left to right. Initially, it is illuminated by orange street lamps, reflecting an orange hue. As it moves right, its rear remains orange-lit, while the front becomes blue due to a nearby advertising screen. Eventually, the car is fully bathed in blue light. This dynamic lighting response indicates our model can correctly handle spatially varying light.\n\n## Q7: Limitation Analysis\nAs discussed in Section 4.4 of the main paper, the limitations are fourfold. **First**, TC-Light inherits IC-Light's inability to relight hard shadows. As shown in the _Virtual KITTI_ video example in Q2, the tree shadows in ours_full.gif align with 0_input.gif rather than 1_groundtruth.gif. This results in an SSIM deviate from the average value 0.59 to 0.47. **Second**, the model struggles when the input resolution is significantly lower than that used to train IC-Light, leading to more severe flickering. For instance, downscaling the NavSim subset from 960×536 to 480×264 causes Warp-SSIM to drop from the average value 90.46 to 88.36, although CLIP-T remains at 0.304. **Third**, flow estimation becomes unreliable in textureless regions, introducing artifacts. Under _Video_Comparison/bad_case_ folder of Q2, the example from the AgiBot subset shows how unreliable flow leads to instability along the robotic arm's edge, with CLIP-T falling from the average value 0.275 to 0.259.  **Fourth**, the temporal consistency loss tends to smooth the texture of flickering areas. In the Carla subset of Q2, ours_full.gif shows smoother road and sky textures than iclight.gif, trading off realism for consistency. We will expand these discussions in the revised paper to provide a clearer assessment of robustness, including visualizations of failure cases."}, {"review_id": "KoIMMKXVQ0", "reviewer": "Reviewer_xtUc", "summary": "This paper introduces TC-Light, a video relighting focused on long videos. It proposes a two-stage post-optimization framework that refines an initial relighting result produced by inflating an image-based model to achieve both global lighting consistency and temporal alignment while maintaining low computational cost. A new benchmark dataset is also proposed, containing 58 long video clips of about 256 frames across diverse environment and conditions to evaluate video relighting performance under complex dynamic scenes.", "questions": "1. Although the paper mentions this in passing, please state explicitly which parts of TC-Light are novel and which are adapted from VidToMe, Slicedit, or other prior work. A concise side-by-side comparison table would help show that the method is more than a straightforward combination of existing techniques.\n\n2. Discuss TC-Light’s behaviour when optical-flow estimates are poor (e.g., very fast motion, heavy occlusions, etc.) and provide qualitative or quantitative evidence of any failure and mitigation strategies if any.\n\n3. For the proposed dataset, detail the clip-selection process, domain balance, and user-study protocol, and explain how test clips were kept fully separate from development. Clarify whether any ground-truth relighting pairs exist or if the benchmark is solely for subjective/relative evaluation, and include an expanded section or appendix summarizing the dataset’s content and statistics.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "TC-Light achieved state-of-the-art temporal consistency while staying prompt-controllable with its two-stage framework. Its UVT representation is lightweight yet enforces cross-frame coherence, and the authors back their claims with a large, diverse long-video benchmark. However, the method assumes reasonably accurate optical-flow grouping and benefits from depth when available, so very noisy flow or missing depth may leave residual artifacts. The heavy consistency constraints seems to slightly lower prompt fidelity and image detail. Even with the proposed new benchmark dataset, the quantitative evaluation is limited because no ground-truth relighted videos exist, making results rely on proxy metrics and user studies.", "was_revised": true, "final_justification": "The authors addressed my questions diligently in the rebuttal. Originality attribution (Q1) and dataset details (Q3) are now clearly documented. They added a robustness experiment that relights videos after sub-sampling every two to four frames to mimic faster motion, yet in my view this does not fully resolve Q2. Severe scenarios such as large occlusions, blur, transparency, or complete flow failure are still underexamined. I have read and agree with the other reviewers’ concerns as well and have reflected them in my final rating.", "rebuttal": "We really appreciate your affirmation of our work and constructive advice! Below, we address the key concerns, with all citations consistent with the original paper.\n## Q1: Side-by-side comparison of TC-Light Modules\n| Modules | VidToMe [33] | Slicedit [10] | 3DGS [25] | TC-Light |\n| ----------- | :-----------: | :-----------: | :-----------: | :-----------: |\n| Token Merging     | ✔       |         |         | ✔       | \n| Multi-Axis Denoising  |         |✔       |         |✔       | \n| Decayed Noise Weight |         |         |         |✔       | \n| Noise Statistics Alignment (AIN)  |         |         |         |✔       | \n| Appearance Embedding Optimization  |         |         |✔       |✔       | \n| Unique Video Tensor Optimization  |         |         |       |✔       | \n\nAs suggested, we provide a side-by-side comparison table for clarity. Note that 3DGS is used for reconstruction instead of video editing. We will include this table and summarize our module distinctions at the start of Sections 3.2 and 3.3 to highlight our contributions.\n\n## Q2: Results on Groundtruth Benchmark\n| Virtual KITTI 2 | SSIM↑ | LPIPS↓ | Motion-S↑ | Warp-SSIM↑ |  Time(s) | Mem.(G) ↓ |\n| ----------- | :-----------: | :-----------: | :-----------: | :-----------: | :-----------: | :-----------: |\n| IC-Light* [58] | 0.5102 | 0.4470 | 95.23 | 68.13 | 1770 | 10.25 |\n| VidToMe [31] | 0.5359 | 0.4262 | 95.95 | 71.33 | **444** | **6.96** |\n| Slicedit [10] | 0.5080 | 0.4237 | 96.91 | 80.74 | 2346 | 17.68 |\n| VideoDirector [56] | OOM | OOM  | OOM  | OOM  | OOM  | OOM  |\n| Light-A-Video [60] | OOM | OOM  | OOM  | OOM  | OOM  | OOM  |\n| RelightVid [16] | OOM | OOM  | OOM  | OOM  | OOM  | OOM |\n| COSMOS-T1 [3] | 0.4833| 0.4841 | 97.81 | 84.35 | 3314  | 34.83 |\n| **Ours-light** | 0.5855 | 0.4026 | 98.51 | 90.94 | 580 | 15.21 |\n| **Ours-full** | **0.5910** | **0.3971** | **98.62** | **92.38** | 1002 | 15.21 |\n\nWe appreciate the suggestion and agree that ground-truth evaluation remains scarce but essential in generative video relighting. To this end, we evaluate on Virtual KITTI 2, which renders consistent geometry across varied lighting conditions under autonomous driving scenarios. We selected five scenes and relit them to match illumination of morning, sunset, rain, overcast, and fog settings, using COSMOS prompt upsampler to extract lighting and weather descriptors from the ground truth video as target prompts. Each sequence averages 281 frames at a resolution of 1248×384.\n\nWith access to ground truth, we replace CLIP-T and user preference with SSIM and LPIPS [26]. Results are shown above, following the same symbol definitions as Table 2 in the main paper. Our method outperforms all baselines in perceptual and structural similarity, as well as motion consistency (Motion-S, Warp-SSIM), while remaining efficient.\n\n## Q3: Robustness Analysis\n| Sample Interval | Motion-S↑ | Warp-SSIM↑ | CLIP-T↑ |\n| ----------- | :-----------: | :-----------: | :-----------: |\n| 1 (Normal) | 97.48 | 89.94 | 0.2971 |\n| 2 (Fast) | 96.28 | 87.40 | 0.2986 |\n| 4 (Very Fast) | 95.75 | 85.44 | 0.3018 |\n\nFirst, we evaluate TC-Light’s robustness under **fast motion** using the NavSim subset (5 sequences and ≥ 584 frames), controlling the total frames to 145 while varying sampling intervals.  As shown in the table above, when motion speed increases, CLIP-T and Motion Smoothness remain stable (<2% fluctuation), while Warp-SSIM declines by ~5% under very fast motion due to less reliable optical flow.  Second, in the _scand_ subfolder of Q1's video results, we demonstrate TC-Light’s handling of **heavy occlusion**. As shown in _ours_full.gif_, our model correctly preserves the appearance of the same identity, while there is a minimal drop (<1%) in Warp-SSIM and Motion Smoothness compared to the SCAND subset average. These discussions would be supplemented to our revised manuscripts.\n\n## Q4: Details about the benchmark\nThank you for the valuable suggestion. Key details are as follows:\n- The **user study protocol** is described in Section D of the supplementary, while **dataset statistics** appear in Table 1 (main paper) and Table S2 (supplementary). **Dataset content**, including scenarios and camera setups, is outlined in Lines 195–196 (main paper) and Lines 23–30 (supplementary). We plan to add thumbnails of each subset for clearer visualization.\n- For **clip selection**, sequences shorter than 300 frames are used fully; otherwise, 300 consecutive frames are randomly sampled. Statistics are provided in Table 1 of the main paper. \n- **Domain balance** is maintained between synthetic and real environments (25 vs. 28 videos), also balanced within sub-domains: autonomous driving (12 synthetic, 10 real), robotic manipulation (8 synthetic, 12 real), indoor navigation (5 synthetic, 6 real). The aerial subset is excluded from balance due to limited long dynamic drone videos in the simulation environment and serves mainly for generalization validation.\n- **Test clip separation** is strictly ensured. IC-Light, trained solely on images, has never seen these video datasets. Our temporal inflation is zero-shot, and the post-optimization process is input-agnostic. Text prompts are randomly selected from IC-Light prompts or COSMOS-generated ones, with unreasonable prompts resampled (e.g., \"a modern interior room\" for a driving scenario). This selection process is fully independent of model development.\n- This benchmark is solely for **subjective/relative evaluation**. But we also supplemented results on the Virtual KITTI dataset, which contains ground-truth video (see Q2). The table in Q2 sufficiently validates the superiority of our method's performance.\n\nAs recommended, we will add a dedicated section in the paper to comprehensively introduce the benchmark and include these details."}, {"review_id": "iy66qRfXJt", "reviewer": "Reviewer_4dnF", "summary": "This paper introduces a novel method for relighting long and highly dynamic videos. It extends the image-based relighting model IC-Light to the video domain using training-free techniques, followed by a two-stage post-optimization process to enhance temporal consistency and visual fidelity. Experimental results demonstrate that the proposed method achieves superior performance with low computational cost.", "questions": "- In line 142, the paper mentions *the first stage introduces a per-frame appearance embedding $E_t$*, but $E_t$ is not previously defined or explained.\n\n- In Table 2, the proposed method underperforms on the CLIP-T metric. Please explain the reason behind this discrepancy.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "## Strengths\n\n- The paper presents a novel and efficient approach to video relighting, achieving strong performance with reduced computational cost. This makes it promising for various downstream applications.\n\n- The authors propose a new benchmark that specifically targets long and highly dynamic videos. The release of this benchmark could benefit the research community.\n\n## Weaknesses\n\n- My primary concern is the overall stability of the pipeline. The method comprises multiple submodules, each relying on different base models or hand-crafted optimization losses. A discussion on the success rate, along with visualizations of failure cases, would provide a clearer picture of the method's robustness.\n\n- The paper lacks video result demonstrations. Since this work focuses on video editing, quantitative comparisons based on only a few frames are insufficient to evaluate temporal consistency and overall quality. Including video comparisons would significantly strengthen the evaluation.", "was_revised": true, "final_justification": "The authors have addressed most of my concerns, and I am inclined to recommend acceptance.", "rebuttal": "We do appreciate your willingness to recommend our paper for acceptance and insightful advice. Below, we address the key concerns, with all citations consistent with the original paper.\n\n## Q1: Video Results and Enhanced Evaluation\nThank you for the suggestion—we fully agree on the importance of video results for assessing temporal quality. Videos with corresponding prompts are provided in the _Video_Comparison_ folder linked anonymously in the supplementary (Line 9). This folder includes seven representative scenarios across both synthetic and real-world settings, including a ground truth benchmark (described in the following paragraph). As shown, our method significantly outperforms IC-Light and VidToMe in temporal consistency, avoids the blurriness and unnatural lighting seen in COSMOS-Transfer1, and surpasses Slicedit in instruction adherence. TC-Light consistently delivers high-quality, stable results across dynamic scenes. All videos will be made publicly available on our paper's project page.\n\nAdditionally, following suggestions from reviewers _phEG_ and _xtUc_, we include results on the Virtual KITTI 2 benchmark—a rare ground-truth-based relighting dataset with consistent geometry under varying lighting.  We selected five scenes and relit them to match illumination of morning, sunset, rain, overcast, and fog settings, using COSMOS prompt upsampler to extract lighting and weather descriptors from the ground truth video as target prompts. Each sequence averages 281 frames at a resolution of 1248×384.\n\nWith access to ground truth, we replace CLIP-T and user preference with SSIM and LPIPS [26]. Results are shown below, following the same symbol definitions as Table 2 in the main paper. Our method outperforms all baselines in perceptual and structural similarity, as well as motion consistency (Motion-S, Warp-SSIM), while remaining efficient.\n\n| Virtual KITTI 2 | SSIM↑ | LPIPS↓ | Motion-S↑ | Warp-SSIM↑ |  Time(s) | Mem.(G) ↓ |\n| ----------- | :-----------: | :-----------: | :-----------: | :-----------: | :-----------: | :-----------: |\n| IC-Light* [58] | 0.5102 | 0.4470 | 95.23 | 68.13 | 1770 | 10.25 |\n| VidToMe [31] | 0.5359 | 0.4262 | 95.95 | 71.33 | **444** | **6.96** |\n| Slicedit [10] | 0.5080 | 0.4237 | 96.91 | 80.74 | 2346 | 17.68 |\n| VideoDirector [56] | OOM | OOM  | OOM  | OOM  | OOM  | OOM  |\n| Light-A-Video [60] | OOM | OOM  | OOM  | OOM  | OOM  | OOM  |\n| RelightVid [16] | OOM | OOM  | OOM  | OOM  | OOM  | OOM |\n| COSMOS-T1 [3] | 0.4833| 0.4841 | 97.81 | 84.35 | 3314  | 34.83 |\n| **Ours-light** | 0.5855 | 0.4026 | 98.51 | 90.94 | 580 | 15.21 |\n| **Ours-full** | **0.5910** | **0.3971** | **98.62** | **92.38** | 1002 | 15.21 |\n\n## Q2: Robustness Analysis\nThe influence of different submodules mainly reflects on the fluctuation of metrics. **First**, TC-Light inherits IC-Light's inability to relight hard shadows, as discussed in Section 4.4 of the paper. As shown in the Virtual KITTI video example in Q1, the tree shadows in ours_full.gif align with 0_input.gif rather than 1_groundtruth.gif. This results in SSIM deviating from the average value 0.59 to 0.47. **Second**, the performance is relatively stable to the design of the UVT index **κ**.  Comparing rows 6, 8, and 9 of Table 3 of the main paper, the additional depth and instance mask clues bring a slight performance gain and drop. **Third**, unreliable flow estimation in some textureless regions may introduce artifacts, as discussed in Section 4.4 of the paper. Under the _Video_Comparison/bad_case_ folder of Q1, an example from the AgiBot subset illustrates how unreliable flow leads to instability along the robotic arm's edge, with CLIP-T falling from the average value 0.275 to 0.259. We will expand these discussions in the revised paper to provide a clearer assessment of robustness, including visualizations of failure cases.\n\n## Q3: $E_t$ Illustration\nSorry for the confusion. We explain $E_t$ in Lines 143-144 that \"Inspired by [26], we model $E_t$ as a 3 × 4 affine transformation matrix, initialized to the identity and optimized via Adam [29].\" We would highlight this text and give an example of $E_t$ for an easier understanding in the modification.\n\n## Q4: CLIP-T Performance\nThe drop in CLIP-T reflects a trade-off between perceptual alignment and temporal consistency. Our post-optimization smooths flickering textures and illumination, slightly simplifying details and causing minor shifts in CLIP features. While CLIP-T decreases by ~1%, Warp-SSIM improves by over 20%. For example, in the video results of the CARLA subset in Q1, VidToMe and TC-Light achieve CLIP-T scores of 0.2556 and 0.2540, respectively, while Warp-SSIM improves from 83.90 to 93.66. As evidence, ours_full.gif is significantly smoother and more consistent than vidtome.gif."}, {"review_id": "yttq6CjDKr", "reviewer": "Reviewer_rT24", "summary": "This paper proposes a novel method to adapt an image-based relighting network, IC-Light, to relight dynamic long videos. The main challenge is to maintain temporal consistency in the video relighting results. To this end, the paper proposes a 2-stage pipeline to post-process the delighted video. In the first stage, it aligns the exposure across the frames. In the second stage, it further refines illumination and texture details based on the Unique Video Tensor representation. Quantitative experiments demonstrate that it produces better results than prior art.", "questions": "In L165, the paper says, \"All pixels with identical κ are gathered via averaging to form one element of the one-dimensional vector.\" In my understanding, κ is a vector of [flow_id, RGB value]. What is the meaning of identical κ? Let κ1=[1,1,1,1], κ2=[1,1,1,1], κ3=[1,2,3,4]. Is κ1=κ2=κ3, or just κ1=κ2? I think the former is more reasonable. If so, \"All pixels with identical κ\" should be changed to \"All pixels with identical flow id\"?", "limitations": "Previous methods for relighting use a physically-based inverse rendering technique (e.g., Nerfactor [a] and InvRender [b]) or a data-driven approach (train the network on a Light Stage dataset to learn the rendering process, e.g., IC-Light) to ensure the results are physically plausible, but the paper only uses heuristics to post-process the lighting effects. I suggest that the author discuss this point in the paper.\n\n[a] NeRFactor: Neural Factorization of Shape and Reflectance Under an Unknown Illumination, SIGGRAPH Asia 2021\n[b] Modeling Indirect Illumination for Inverse Rendering, CVPR 2022", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths\n* The paper works on a promising direction: relighting any video. \n* The proposed technique seems simple but effective.\n* The Unique Video Tensor representation is novel and efficient.\n\n## Weaknesses\n* My main concern is about the quality of the proposed method. I think a paper working on this task should provide video relighting results as supplementary material. Otherwise, it is really hard for me to judge the performance of this work. Although the paper reports some quantitative metrics and the method seems to produce the best results, they are not enough to evaluate the paper. For example, without a side-by-side qualitative comparison, I can not translate the quantitative performance gain into actual visual improvement.\n* I suggest that the author add the MemFlow method to the Preliminaries section. It is quite confusing when the flow ID appears in L162. I understand its meaning after checking the supplementary material.\n* The paper mentions multiple times that their method can boost the embodied AI field, but there is no experiment to support this claim. For example, in the conclusion, it says: \"making it particularly well suited for sim2real and real2real data scaling in embodied AI,\" and the paper's keywords also contain embodied AI.\n\nAs I cannot evaluate the performance gain in the current version, I lean negatively on this paper.", "was_revised": true, "final_justification": "Most of my concerns are addressed. Thus, I raise the score to borderline accept. I suggest that the author include all the discussions and experiments in the final version.", "rebuttal": "Thanks sincerely for your attentive review and profound comments. Below, we address the key concerns, with all citations consistent with the original paper.\n\n## Q1: Video Results and Enhanced Evaluation\nThank you for the feedback—we fully understand the importance of video results for evaluating visual quality. Videos with corresponding prompts are provided in the _Video_Comparison_ folder linked anonymously in the supplementary (Line 9). This folder includes seven representative scenarios across both synthetic and real-world settings, including a ground truth benchmark in the following description. As shown, our method significantly outperforms IC-Light and VidToMe in temporal consistency, avoids the blurriness and unnatural lighting seen in COSMOS-Transfer1, and surpasses Slicedit in instruction adherence. TC-Light consistently delivers high-quality, stable results across dynamic scenes. All videos will be made publicly available on our paper's project page.\n\nAdditionally, following suggestions from reviewers _phEG_ and _xtUc_, we include results on the Virtual KITTI 2 benchmark—a rare ground-truth-based relighting dataset with consistent geometry under varying lighting.   We selected five scenes and relit them to match illumination of morning, sunset, rain, overcast, and fog settings, using COSMOS prompt upsampler to extract lighting and weather descriptors from the ground truth video as target prompts. Each sequence averages 281 frames at a resolution of 1248×384.\n\nWith access to ground truth, we replace CLIP-T and user preference with SSIM and LPIPS [26]. Results are shown below, following the same symbol definitions as Table 2 in the main paper. Our method outperforms all baselines in perceptual and structural similarity, as well as motion consistency (Motion-S, Warp-SSIM), while remaining efficient.\n\n| Virtual KITTI 2 | SSIM↑ | LPIPS↓ | Motion-S↑ | Warp-SSIM↑ |  Time(s) | Mem.(G) ↓ |\n| ----------- | :-----------: | :-----------: | :-----------: | :-----------: | :-----------: | :-----------: |\n| IC-Light* [58] | 0.5102 | 0.4470 | 95.23 | 68.13 | 1770 | 10.25 |\n| VidToMe [31] | 0.5359 | 0.4262 | 95.95 | 71.33 | **444** | **6.96** |\n| Slicedit [10] | 0.5080 | 0.4237 | 96.91 | 80.74 | 2346 | 17.68 |\n| VideoDirector [56] | OOM | OOM  | OOM  | OOM  | OOM  | OOM  |\n| Light-A-Video [60] | OOM | OOM  | OOM  | OOM  | OOM  | OOM  |\n| RelightVid [16] | OOM | OOM  | OOM  | OOM  | OOM  | OOM |\n| COSMOS-T1 [3] | 0.4833| 0.4841 | 97.81 | 84.35 | 3314  | 34.83 |\n| **Ours-light** | 0.5855 | 0.4026 | 98.51 | 90.94 | 580 | 15.21 |\n| **Ours-full** | **0.5910** | **0.3971** | **98.62** | **92.38** | 1002 | 15.21 |\n\n## Q2: Illustration for MemFlow\nSorry for the confusion. We will revise the original phrasing from \"where the first element is the flow ID (grouping pixels with shared optical flow)\" to \"where the first element is the flow ID (pixels connected by the optical flow predicted by MemFlow share the same flow ID)\" for improved clarity. As suggested, we will also add a brief explanation of MemFlow in the Preliminaries section to provide proper context.\n\n## Q3: Claim Regarding Embodied AI\nThank you for highlighting this concern. To clarify, our mention of embodied AI was not intended to imply direct improvements to agent training, but to highlight a broader application context where dynamic video relighting serves as a foundational tool. Our method addresses critical limitations of prior work—namely, temporal inconsistency and computational inefficiency—making it possible to be applied to use cases beyond traditional domains like filmmaking or AR. To reduce confusion and improve precision, we will revise the keyword from \"embodied AI\" to \"world-to-world transfer\", and adjust the conclusion to: \"endowing it with value and potential for broader application areas such as sim2real and real-world video scaling in embodied AI training and validation pipelines.\"\n\n## Q4: Meaning of Identical κ\nSorry for the confusion. Lines 161–165 present the general definition of κ. A crucial detail in interpreting \"identical κ\" lies in that the RGB component can be **quantized**. Without quantization, “identical κ” would require both identical flow connections and exact color matches—too strict for cases like view-dependent effects. By applying 7-bit quantization, we allow pixels within a slight color variation (~2/255) to be considered in gathering, enabling more robust grouping and filtering out unreliable flow connections. In our final implementation, κ uses flow ID and 7-bit quantized RGB. To enhance clarity, we will explicitly add this explanation regarding the physical meaning of \"identical κ\" to Line 165.\n\n## Q5: Physical Plausibility\nThe physical plausibility of our method is inherited from IC-Light, which is pre-trained on high-quality Light Stage data and has learned a physically grounded relighting process. Our main contribution lies in improving temporal consistency without altering the illumination priors embedded in the base model.\n\nAs detailed in Section 3.2, we introduce a video model inflation mechanism based on token merging/unmerging and decayed multi-axis denoising to enable temporal feature-level information exchange. Since these components do not alter the prior knowledge encoded in the base model, the distribution of edited illumination aligns with that of IC-Light while enhancing consistency across frames.\n\nIn Section 3.3, we propose a two-stage post-optimization strategy. The first stage smooths global exposure transitions using an appearance embedding, following practices in physically-based rendering methods like NeRF-W [CVPR’21] and 3DGS [SIGGRAPH’23]. The second stage refines local fluctuations without altering the overall lighting. Thus, our final results maintain the physically plausible qualities of IC-Light, while significantly improving temporal coherence. As shown in Figure 3 of the main paper and the video results in Q1, our illumination remains qualitatively aligned with IC-Light and VidToMe, but with fewer artifacts and greater temporal stability.\n\nThanks to the strong priors of IC-Light, our method focuses on temporal coherence and computation efficiency. Compared to optimization-heavy approaches such as Nerfactor [a] and InvRender [b], our post-optimization stage takes only ~2 minutes, with the entire pipeline completing in ~10 minutes—substantially faster than training NeRF or 3DGS models, as discussed in Lines 88–93 of the main paper. As suggested, we will incorporate the above discussion into the revised manuscript and add citations for both Nerfactor and InvRender."}]

Author rebuttal:

Meta-review: This paper proposes a method for relighting long videos. The main idea is to give an image-based relighting model a new temporal axis, aligning the exposure across time within the video and then refining the illumination and texture details. The paper also includes a new benchmark dataset, with 58 diverse "long" clips (256 frames). This paper received borderline/positive reviews: 5, 4, 4, 4.
The reviewers find the paper well-written and easy to follow, and are impressed with the results, particularly given the low computational cost. The reviewers are strongly encouraged to revise the paper according to the facts clarified in the rebuttal, especially relating to the originality attribution. The AC recommendation is to accept.
Decision: Accept (poster)