Title: TC-Light: Temporally Coherent Generative Rendering for Realistic World Transfer
Venue: NeurIPS 2025
Accepted: yes
Keywords: video relighting

video editing

diffusion

highly dynamic

world-to-world transfer
Abstract: Illumination and texture rerendering are critical dimensions for world-to-world transfer, which is valuable for applications including sim2real and real2real visual data scaling up for embodied AI. Existing techniques generatively re-render the input video to realize the transfer, such as video relighting models and conditioned world generation models. Nevertheless, these models are predominantly limited to the domain of training data (e.g., portrait)  or fall into the bottleneck of temporal consistency and computation efficiency, especially when the input video involves complex dynamics and long durations. In this paper, we propose **TC-Light**, a novel paradigm characterized
by the proposed two-stage post optimization mechanism. Starting from the video preliminarily relighted by an inflated video relighting model, it optimizes appearance embedding in the first stage to align global illumination. Then it optimizes the proposed canonical video representation, i.e., **Unique Video Tensor (UVT)**, to align fine-grained texture and lighting in the second stage. To comprehensively evaluate performance, we also establish a long and highly dynamic video benchmark. Extensive experiments show that our method enables physically plausible re-rendering results with superior temporal coherence and low computation cost. The code and video demos are available at our [Project Page](https://dekuliutesla.github.io/tclight/).


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel method to adapt an image-based relighting network, IC-Light, to relight dynamic long videos. The main challenge is to maintain temporal consistency in the video relighting results. To this end, the paper proposes a 2-stage pipeline to post-process the delighted video. In the first stage, it aligns the exposure across the frames. In the second stage, it further refines illumination and texture details based on the Unique Video Tensor representation. Quantitative experiments demonstrate that it produces better results than prior art.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n* The paper works on a promising direction: relighting any video. \n* The proposed technique seems simple but effective.\n* The Unique Video Tensor representation is novel and efficient.\n\n## Weaknesses\n* My main concern is about the quality of the proposed method. I think a paper working on this task should provide video relighting results as supplementary material. Otherwise, it is really hard for me to judge the performance of this work. Although the paper reports some quantitative metrics and the method seems to produce the best results, they are not enough to evaluate the paper. For example, without a side-by-side qualitative comparison, I can not translate the quantitative performance gain into actual visual improvement.\n* I suggest that the author add the MemFlow method to the Preliminaries section. It is quite confusing when the flow ID appears in L162. I understand its meaning after checking the supplementary material.\n* The paper mentions multiple times that their method can boost the embodied AI field, but there is no experiment to support this claim. For example, in the conclusion, it says: \"making it particularly well suited for sim2real and real2real data scaling in embodied AI,\" and the paper's keywords also contain embodied AI.\n\nAs I cannot evaluate the performance gain in the current version, I lean negatively on this paper.\n\nQUESTIONS:\nIn L165, the paper says, \"All pixels with identical κ are gathered via averaging to form one element of the one-dimensional vector.\" In my understanding, κ is a vector of [flow_id, RGB value]. What is the meaning of identical κ? Let κ1=[1,1,1,1], κ2=[1,1,1,1], κ3=[1,2,3,4]. Is κ1=κ2=κ3, or just κ1=κ2? I think the former is more reasonable. If so, \"All pixels with identical κ\" should be changed to \"All pixels with identical flow id\"?\n\nLIMITATIONS:\nPrevious methods for relighting use a physically-based inverse rendering technique (e.g., Nerfactor [a] and InvRender [b]) or a data-driven approach (train the network on a Light Stage dataset to learn the rendering process, e.g., IC-Light) to ensure the results are physically plausible, but the paper only uses heuristics to post-process the lighting effects. I suggest that the author discuss this point in the paper.\n\n[a] NeRFactor: Neural Factorization of Shape and Reflectance Under an Unknown Illumination, SIGGRAPH Asia 2021\n[b] Modeling Indirect Illumination for Inverse Rendering, CVPR 2022", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-20"}

comments: SUMMARY:
This paper introduces a novel method for relighting long and highly dynamic videos. It extends the image-based relighting model IC-Light to the video domain using training-free techniques, followed by a two-stage post-optimization process to enhance temporal consistency and visual fidelity. Experimental results demonstrate that the proposed method achieves superior performance with low computational cost.

STRENGTHS AND WEAKNESSES:
## Strengths

- The paper presents a novel and efficient approach to video relighting, achieving strong performance with reduced computational cost. This makes it promising for various downstream applications.

- The authors propose a new benchmark that specifically targets long and highly dynamic videos. The release of this benchmark could benefit the research community.

## Weaknesses

- My primary concern is the overall stability of the pipeline. The method comprises multiple submodules, each relying on different base models or hand-crafted optimization losses. A discussion on the success rate, along with visualizations of failure cases, would provide a clearer picture of the method's robustness.

- The paper lacks video result demonstrations. Since this work focuses on video editing, quantitative comparisons based on only a few frames are insufficient to evaluate temporal consistency and overall quality. Including video comparisons would significantly strengthen the evaluation.

QUESTIONS:
- In line 142, the paper mentions *the first stage introduces a per-frame appearance embedding $E_t$*, but $E_t$ is not previously defined or explained.

- In Table 2, the proposed method underperforms on the CLIP-T metric. Please explain the reason behind this discrepancy.

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 2
IMPACT: 2
ORIGINALITY: 3
DATE: 2025-07-01

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces TC-Light, a video relighting focused on long videos. It proposes a two-stage post-optimization framework that refines an initial relighting result produced by inflating an image-based model to achieve both global lighting consistency and temporal alignment while maintaining low computational cost. A new benchmark dataset is also proposed, containing 58 long video clips of about 256 frames across diverse environment and conditions to evaluate video relighting performance under complex dynamic scenes.\n\nSTRENGTHS AND WEAKNESSES:\nTC-Light achieved state-of-the-art temporal consistency while staying prompt-controllable with its two-stage framework. Its UVT representation is lightweight yet enforces cross-frame coherence, and the authors back their claims with a large, diverse long-video benchmark. However, the method assumes reasonably accurate optical-flow grouping and benefits from depth when available, so very noisy flow or missing depth may leave residual artifacts. The heavy consistency constraints seems to slightly lower prompt fidelity and image detail. Even with the proposed new benchmark dataset, the quantitative evaluation is limited because no ground-truth relighted videos exist, making results rely on proxy metrics and user studies.\n\nQUESTIONS:\n1. Although the paper mentions this in passing, please state explicitly which parts of TC-Light are novel and which are adapted from VidToMe, Slicedit, or other prior work. A concise side-by-side comparison table would help show that the method is more than a straightforward combination of existing techniques.\n\n2. Discuss TC-Light’s behaviour when optical-flow estimates are poor (e.g., very fast motion, heavy occlusions, etc.) and provide qualitative or quantitative evidence of any failure and mitigation strategies if any.\n\n3. For the proposed dataset, detail the clip-selection process, domain balance, and user-study protocol, and explain how test clips were kept fully separate from development. Clarify whether any ground-truth relighting pairs exist or if the benchmark is solely for subjective/relative evaluation, and include an expanded section or appendix summarizing the dataset’s content and statistics.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a method, TC-Light, that tackles the long-standing challenge of editing illumination in long, highly dynamic videos by inflating the single-image relighting model (IC-Light) into a zero-shot video diffusion framework and introducing decayed multi-axis denoising that blends per-frame ( xy ) and time-slice ( yt ) predictions for improved temporal coherence. A subsequent two-stage post-optimization first aligns global exposure through a learned per-frame affine “appearance embedding,” then refines local flicker using a Unique Video Tensor (UVT)—a compact 1-D RGB lookup indexed by flow, depth, and color cues. Qualitative results and some hand-crafted metrics are evaluated to showcase the effectiveness of the proposed method with extensive ablation studies. However, metrics computed against ground truth relight videos are absent due to the scarcity of such data. Plus, there is no qualitative video presented in the supplementary which is somehow important for papers working on video generation.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The paper is well-written and easy to follow.\n- The evaluations especially ablation studies are extensive, including strong zero-shot and training-based baselines, ablations, user study, and efficiency measurements. \n- By using decayed multi-axis denoising, it is an elegant way to inject motion priors without training, which improves consistency with negligible extra VRAM.\n- The proposed UVT representation is simple yet effective, which improves the visual quality while lower the runtime VRAM.\n\nWeaknesses:\n- Flow dependence and error analysis: Optical-flow errors propagate into UVT, occasionally causing blur or ghosting in low-texture areas; robustness analysis is limited. \n- Groundtruth Benchmark: Though capturing video with different lightings is hard, it is still encouraged to have evaluation against ground truth which is typically lacking in this field of generative video relighting. Moreover, it will provide more accurate and robust probe on the performance.\n- No Video results: As a paper working on video-related generative tasks, I would weigh more on the qualitative results presented in the demo video, which could clearly show the temporal smoothness and overall quality of the relit videos. However, there is no video presented in the supplementary material, which makes it hard to determine the effectiveness of the proposed method.\n- Reproducibility: Will the authors release the code? Exact UVT implementation (index hashing, GPU kernels) may be non-trivial to replicate without a reference.\n\nQUESTIONS:\n- How sensitive is performance to the composition of κ (e.g., color + flow vs flow + depth)? Did you explore learned index hashing or adaptive bin sizes?\n- The method currently edits entire frames uniformly via a text prompt. Could UVT be extended to support spatially varying lighting (e.g., spotlight on a moving actor / near-field point lights)?\n- Have you tried TC-Light with recent transformer-flow models (e.g., Flowformer)? How does flow noise degrade Warp-SSIM, and could self-supervised flow refinement within stage II help?\n- As mentioned in your Limitations section, given IC-Light’s limitations, how does it perform on those failed corner cases?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}