Title: Temporal In‑Context Fine‑Tuning with Temporal Reasoning for Versatile Control of Video Diffusion Models
Venue: NeurIPS 2025
Accepted: yes
Keywords: Diffusion

Video Diffusion

Flow Matching

In context learning
Abstract: Recent advances in text-to-video diffusion models have enabled high-quality video synthesis, but controllable generation remains challenging—particularly under limited data and compute. Existing fine-tuning methods for conditional generation often rely on external encoders or architectural modifications, which demand large datasets and are typically restricted to spatially aligned conditioning, limiting flexibility and scalability. In this work, we introduce Temporal In-Context Fine-Tuning (TIC-FT), an efficient and versatile approach with temporal reasoning for adapting pretrained video diffusion models to diverse conditional generation tasks. Our key idea is to concatenate condition and target frames along the temporal axis and insert intermediate buffer frames with progressively increasing noise levels. These buffer frames enable smooth transitions, aligning the fine-tuning process with the pretrained model’s temporal dynamics. TIC-FT is architecture-agnostic and achieves strong performance with as few as 10–30 training samples. We validate our method across a range of tasks—including image-to-video and video-to-video generation—using large-scale base models such as CogVideoX-5B and Wan-14B. Extensive experiments show that TIC-FT outperforms existing baselines in both condition fidelity and visual quality, while remaining highly efficient in both training and inference.


Peer reviews:
comments: SUMMARY:
This paper presents a novel paradigm for controllable video generation. Similar to in-context learning, this method concatenates conditionaframes to target frames and encourages the model to learn the in-context pattern in conditional frames. This method also introduces a buffer frame between target and condition frames to preserve the pretrained model's capability.

STRENGTHS AND WEAKNESSES:
Stength:

1) The proposed method is novel and highly extensible. Compared to previous approaches such as ControlNet, it enables multiple applications through simple concatenation without introducing additional parameters.

2) The paper's writing is clear. The method is straightforward and easy to follow.

3) The experiments are sufficient and the performance is good.

Weakness:

(1) Positional Encoding. After concatenation of conditional , buffer and target frames, the total number of frames likely exceeds the original video length supported by the pretrained model. If the video length during fine-tuning differs from that of the pretrained model, it is unclear whether this would cause issues. It would be helpful to clarify how the positional encoding is handled for the longer frames.

(2) Scaling. In this paper, each application requires a set of training data for fine-tuning. Is it possible to jointly fine-tuning on multiple tasks and  achieve zero-shot performance at inference time? 

(3) Precise Control. If fine-grained control over each frame (e.g., conditioning each frame on a depth map) is needed, is it feasible under the proposed in-context learning framework?

QUESTIONS:
See weakness.

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Temporal In-Context Fine-Tuning (TIC-FT), a parameter-efficient method for adapting pretrained video diffusion models to diverse conditional generation tasks. Key innovations include:\n\n1. Temporal Context Construction:\n   - Concatenates condition frames (e.g., input images/videos) and target frames along the temporal axis\n   - Inserts noisy buffer frames with progressive noise levels between condition-target pairs to align with pretrained diffusion dynamics\n2. Training-Free Adaptation:\n   - Requires no architectural changes or external encoders\n   - Achieves strong performance with 10–30 training samples\n3. Versatile Control:\n   - Validated on image-to-video and video-to-video tasks using billion-scale models (CogVideoX-5B, Wan-14B)\n   - Maintains high condition fidelity and visual quality while reducing training/inference costs\n\nTIC-FT establishes temporal concatenation as an effective paradigm for lightweight control of video diffusion models.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n- Quality:\n  Rigorous validation on billion-scale models across 6+ tasks (style transfer/motion transfer/etc.). Few-shot efficiency is empirically verified.\n- Originality:\n  Novel temporal concatenation + noisy buffer frames mechanism elegantly aligns pretrained/fine-tuning distributions. Unifies diverse conditioning tasks without architectural changes.\n- Significance:\n  Democratizes control of large video models for low-resource users. Memory-efficient design.\n- Clarity:\n  Exceptional methodological description; clean task taxonomy.\n\nWeaknesses:\n\n- Quality:\n  - Baseline mismatch: ControlNet (image-centric) and Fun-pose (pre-finetuned) are suboptimal comparators. Missing video-specific adapters (e.g., TADA, VidControl).\n  - Limited temporal metrics: Relies on FVD/CLIP – neglects *motion consistency* (e.g., optical flow variance).\n- Significance:\n  - 10-second input limit restricts long-video applications.\n\nQUESTIONS:\n1. Baseline Relevance\n   - Q: Why omit video-specific control methods (e.g., ControlVideo, VidControl)? Does TIC-FT outperform them under identical compute?\n2. Temporal Artifact Analysis\n   - Q: Buffer frames reduce distribution shift but may cause motion flicker. Did you quantify temporal instability (e.g., FLD score)?\n3. Long-Sequence Scalability\n   - Q: The 10-second limit stems from memory constraints. Could sequence chunking or gradient checkpointing mitigate this?\n4. Real-World Generalization\n   - Q: Synthetic datasets (GPT-4o/Sora) may bias results. Does performance hold on complex real-world motions?\n\nLIMITATIONS:\nYES.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\n- Many existing methods for conditional video generation rely on large training datasets and introduce additional architectural modifications. To address these limitations, this paper proposes Temporal In-Context Fine-Tuning (TIC-FT), an efficient and versatile approach for adapting pretrained video diffusion models to diverse conditional generation tasks.\n- Extensive experiments show that TIC-FT outperforms existing baselines in both condition fidelity and visual quality.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n- The paper introduces a simple yet effective method for adapting pretrained video diffusion models to diverse conditional generation tasks.\n- TIC-FT shows strong empirical performance with small training data, offering a highly efficient fine-tuning strategy.\n- The proposed method avoids modifying the model architecture, making it easy for adoption.\n\n**Weaknesses**\n- L168–169 describe the inference procedure as “iteratively identifying the frames currently at the maximal noise level t and applying the video diffusion sampler exclusively to those frames”. It seems like the buffer frames need to be denoised iteratively, possibly one by one, because they have different noise levels? This seems to increase the inference time complexity? What's the inference time complexity of TIC-FT, and how does it compare to baseline methods?\n- It is unclear to me how the number of buffer frames is determined during training and inference.\n- It is unclear whether, in cases with multiple conditional frames, does the model only use the first conditional frame to construct the buffer frames? if so, how does the model use multiple conditional frames when constructing the buffer frames?\n\nQUESTIONS:\nHave the pretrained video diffusion backbones been trained on any conditional generation tasks (i.e., sth like the downstream tasks of this paper) except for general video generation? If the former, how does this influence the effectiveness of TIC-FT?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a method, called temporal in-context learning (TIC), for conditional video generation, including image-to-video and video-to-video tasks. Essentially, the visual conditions are concatenated with the noisy latents along the time dimension, allowing the generated frames to be viewed as a natural extension of the conditional frame(s). Several buffer frames are introduced between them to smooth out the transition from input conditions to generated frames. This design spares the need to modify the model architecture and can be trained efficiently with just a few samples. Experimental results compared to a few baselines address the effectiveness of the method in consistency and transferability.\n\nSTRENGTHS AND WEAKNESSES:\n(+) TIC-FT is a clever approach to enabling image/video conditions for video generation without requiring the introduction of a large number of new parameters. Viewing the generation as a continuation of the visual condition in the temporal dimension provides a new perspective and more flexible design for future work in this domain. \n\n(+) The design of buffer frames also makes sense in providing space for a transition process between conditions and generated frames. The noise schedule is carefully considered to make the transition smoother. \n\n(+) TIC-FT seems like an efficient fine-tuning strategy that works with just tens of training samples in limited GPU hours. \n\n(+) Experimental results demonstrate superiority in consistency and guidance following compared to ControlNet and Fun-pose. \n\n\nSome concerns about experiments:\n\n(-) Small-scale evaluation set. Each task is evaluated on only 20 condition-target pairs, which may not be very representative of showing the whole picture of the performance superiority. Are there larger scale benchmarks or more standard ones that can be used for evaluation?\n\n(-) Metrics: One drawback I could imagine from the temporal concatenation is the strong reliance of every generation frame on the input conditions (or condition shortcuts), which may lead to less dynamics in the output videos. VBench provides metrics beyond just consistency (including dynamic degree), but they are not reported in the tables. \n\n(-) Ablation study on the number of buffer frames. I like the design of buffer frames and that the noise schedule of them is compared carefully. But it seems to me that the paper should also come with certain takeaways or conclusions based on quantitative evaluations (beyond qualitative figures in the appendix) on how many buffer frames to choose. There are some interesting questions to ask: Is there a connection between the number of buffer frames and the number of frames to be generated? How about the connection with the number of condition images/frames? \n\nThe writing could be improved by making some of the claims more rigorous. See below points\n\n(-) I feel that the claim of \"requires no architectural changes\" is a bit off, as TIC-FT indeed requires additional LoRA parameters. The essential difference between TIC-FT and baselines, such as ControlNet, is probably the fact that the former introduces a much smaller number of new parameters, and these new modules are more general and not specifically designed for a certain type of visual condition. \n\n(-) I am unsure if the setting falls within the definition of in-context learning. Usually, the in-context demonstrations consist of several examples belonging to the same tasks. The V2V setting seems fine, but the image-to-video setting doesn't really sound like in-context learning because the input image is merely a condition used for generating the video instead of being a \"demonstration\" that falls into the same modality or task type as the output video frames.\n\nQUESTIONS:\nSee weaknesses\n\nLIMITATIONS:\nSee weaknesses", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-04"}