Title: UniTransfer: Video Concept Transfer via Progressive Spatio-Temporal Decomposition
Venue: NeurIPS 2025
Accepted: yes
Keywords: video concept transfer

video editing

controllable video generation
Abstract: Recent advancements in video generation models have enabled the creation of diverse and realistic videos, with promising applications in advertising and film production. However, as one of the essential tasks of video generation models, video concept transfer remains significantly challenging.
Existing methods generally model video as an entirety, leading to limited flexibility and precision when solely editing specific regions or concepts. To mitigate this dilemma, we propose a novel architecture UniTransfer, which introduces both spatial and diffusion timestep decomposition in a progressive paradigm, achieving precise and controllable video concept transfer. Specifically, in terms of spatial decomposition, we decouple videos into three key components: the foreground subject, the background, and the motion flow. Building upon this decomposed formulation, we further introduce a dual-to-single-stream DiT-based architecture for supporting fine-grained control over different components in the videos. We also introduce a self-supervised pretraining strategy based on random masking to enhance the decomposed representation learning from large-scale unlabeled video data. Inspired by the Chain-of-Thought reasoning paradigm, we further revisit the denoising diffusion process and propose a Chain-of-Prompt (CoP) mechanism to achieve the timestep decomposition. We decompose the denoising process into three stages of different granularity and leverage large language models (LLMs) for stage-specific instructions to guide the generation progressively. We also curate an animal-centric video dataset called OpenAnimal to facilitate the advancement and benchmarking of research in video concept transfer. 
Extensive experiments demonstrate that our method achieves high-quality and controllable video concept transfer across diverse reference images and scenes, surpassing existing baselines in both visual fidelity and editability.


Peer reviews:
comments: SUMMARY:
This paper introduces UniTransfer, a video generation framework that supports controllable concept transfer (e.g., character, motion, garment, and background) using image-based references. Extensive experiments across TikTok, UBC, and a newly released OpenAnimal dataset show performance gains over baseline methods.

STRENGTHS AND WEAKNESSES:
Strengths
1. Clear motivation and modular design: Tackles a real problem in video generation—lack of flexible and fine-grained control.

2. Methodologically comprehensive: Combines spatial disentanglement, motion conditioning, and progressive textual control into a unified pipeline.

3. Solid empirical performance: Competitive results across several tasks (character transfer, background replacement, garment/motion transfer).

4. Dataset contribution: OpenAnimal provides non-human evaluation data and enhances the paper’s experimental diversity.


Weaknesses:

1. The CoP mechanism is conceptually interesting but remains weakly justified. No baseline comparison (e.g., single prompt or learned embeddings), nor any analysis of prompt quality, semantic drift, or hallucination is provided.
2. The use of LLMs for generating stage-specific prompts introduces non-determinism, inference overhead, and variability, which are not quantified or discussed.

QUESTIONS:
1. What evidence supports that CoP improves generation beyond standard prompting?

Please compare with static prompts, or learnable embeddings. How robust are the LLM prompts under rephrasing or minor perturbations?

2. How deterministic is CoP?

Given the stochastic nature of LLMs, how do different CoP generations affect the final result? Do prompt variations lead to style shifts or artifacts?

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-29

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces UniTransfer, an image-guided video generation framework that achieves controllable video concept transfer by progressively decomposing both the spatial content of a video by separating each clip into foreground, background and motion flow; and diffusion denoising timeline by applying a chain-of-prompt strategy. To reduce annotation effort, the model first trained on self-supervised random mask scheme , then fine-tuned with limited masks and optical flow. They also release a 10k-clip dataset of single-animal videos.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The paper tackles with the long-standing challenge of flexible, high-fidelity video editing with a dual-decomposition strategy that is both technically sound and motivated.\n- By splitting generation along spatial (foreground, background, motion) and temporal (coarse-to-fine denoising) axes, the authors deliver a unified framework that handles character, garment, background and motion transfer in a single model.\n- The proposed dual-to-single-stream DiT architecture and Chain-of-Prompt guidance are rigorously described, backed by equations, ablations and intuitive figures, reflecting high quality and clarity.\n- The self-supervised random-mask pre-training further reduces annotation cost and is validated to be essential, adding practical value beyond raw performance.\n\nWeaknesses:\n- The framework depends on a three-branch diffusion transformer, a large language model for prompt decomposition, and an external optical-flow estimator, which raises the bar for reproducibility and deployment, diminishing practical quality despite strong results.\n- Some important scenarios remain unexplored: the method assumes a single foreground subject and largely static backgrounds, so its behaviour with multiple moving objects, heavy camera motion, or depth-layered scenes is unclear, limiting its claimed significance.\n- Quantitative evaluation is thorough for character replacement but only qualitative for garment and background transfer, leaving a gap in empirical evidence for those tasks and slightly weakening overall clarity of validation.\n- The paper also omits discussion of inference cost and LLM latency, and gives limited insight into failure modes of the Chain-of-Prompt mechanism.\n\nQUESTIONS:\n- Please specify the timestep boundaries (T1, T2) and the prompt lengths produced by the LLM for each stage. How sensitive are results to (i) the choice of those boundaries and (ii) the quality or length of the LLM-generated coarse/mid prompts.\n- Report end-to-end inference time, peak GPU memory, and FLOPs for generating a 16-frame 512×512 video.\n- The current pipeline assumes one foreground mask. Can UniTransfer handle scenes with two interacting subjects (e.g. two people, or one animal-one person, or multiple subjects ) or must it be run sequentially per subject.\n- Only qualitative results are shown for these tasks. Could you report objective metrics versus any baselines where applicable?\n- Include a brief section (main or appendix) with representative failure modes (e.g. extreme perspective mismatch, inaccurate optical flow, or misleading LLM summaries) and describe potential remedies.\n\nLIMITATIONS:\nThe paper lists a basic limitation (artifact‐prone foreground/background compositing) and notes potential misuse of synthetic videos, but some important points remain unaddressed:\n- Computational demand and reproducibility:\n- Single-foreground assumption.\n- Performance depends on accurate optical flow and LLM prompt quality.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes UniTransfer, a novel architecture for precise and controllable video concept transfer. It spatially decomposes videos into foreground, background, and motion flow, using a dual-to-single-stream DiT-based architecture for fine-grained control. It also employs a \"Chain-of-Prompt\" mechanism for timestep decomposition, leveraging LLMs to guide the denoising process progressively. The paper also introduces the OpenAnimal dataset.\n\nSTRENGTHS AND WEAKNESSES:\n[Strength]\n- The paper proposes a single integrated model to perform various concept transfer tasks.\n- The authors collect OpenAnimal dataset o facilitate the training and benchmarking of research community.\n\n[Weakness]\n- For character transfer or video motion transfer, the reference input images are quite spatially aligned with the target video (e.g., fig1 top left, fig 4, fig 5). Is it necessary to have spatially-aligned reference input image? How does UniTransfer perform given spatially-unaligned reference input image?\n- Regarding timestep decomposition, can the authors discuss why just providing a long, detailed prompt that contains all information of high-level, structural guidance, detailed, appearance-level description for all timesteps does not work? In other works, can the authors provide comparison experiments against this approach, which is similar to LLM-based prompt enhancing practice of recent vdms?\n\nQUESTIONS:\n- Is it reasonable to understand 'Sec. 3.1.2 Self-supervised Pre-training via Random Masking' as finetuning the vdm to learn inpainting prior?\n- Is the model trained from scratch or finetuned from a pretrained video diffusion model? If the model is trained from scratch, why couldn't the authors finetune from a pretrained video model?\n\nLIMITATIONS:\n- The paper omits discussing a highly relevant video concept personalization paper \"Dynamic Concepts Personalization from Single Videos\". What are the strength/limitation compared to this paper?\n- The paper omits discussing pioneering video motion transfer works, such as \"MotionDirector: Motion Customization of Text-to-Video Diffusion Models\", \"Space-Time Diffusion Features for Zero-Shot Text-Driven Motion Transfer\", and \"VMC: Video Motion Customization using Temporal Attention Adaption for Text-to-Video Diffusion Models\". Specifically, how does the method performs compared to MotionDirector?", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces UniTransfer, a novel architecture for precise and controllable video concept transfer. The core of UniTransfer lies in a progressive decomposition paradigm applied to both spatial and diffusion timestep dimensions. Spatially, it decouples a video into three key components: the foreground subject, the background, and the motion flow. This separation is managed by a dual-to-single-stream DiT-based architecture, enabling fine-grained control over each element.\n\nSTRENGTHS AND WEAKNESSES:\nStrength\n\nMulti-stage Training Strategy: The model is first pre-trained on a large-scale dataset using a random masking strategy to acquire extensive prior knowledge. During this process, the model's learning is progressively guided by gradually increasing the size of the masked regions.\n\nGranularity-aware Denoising: This work addresses the phenomenon in diffusion models where the denoising process first recovers structural information before progressively generating fine-grained textural details. To leverage this, a large language model is used to transform a single text prompt into three distinct, granularity-specific prompts. These are applied at different stages of the denoising process to focus the model's attention on the relevant aspects at each step.\n\nDecomposition and Fusion for Versatility: The training strategy involves decomposing the video into foreground, background, and motion components, which are then fused. This framework allows the model to adapt to a wide variety of transfer and combination tasks, demonstrating strong versatility. In contrast, other methods are often limited to editing or transferring only single, more isolated regions or objects\n\n\nWeakness\n\nIn the ablation study, was the comparison between the dual-stream and single-stream architectures conducted under the same computational cost? Similarly, was the comparison with early motion injection also performed under an equivalent computational budget?\n\nDoes the spatial branch use multi-frame noise? Why not use single-frame noise, given that this branch might only need to handle the spatial foreground-background relationship, while temporal dynamics are modeled by the motion branch? Modeling temporal features early on could potentially be misled by the text features.\n\nConsidering the results shown in Figure 4, Animate Anyone uses the pose sequence from the target video to drive a reference image. If we disregard the background and focus only on the foreground for comparison, the results in the second and third rows of Figure 4 suggest that the method proposed in this work is inferior to Animate Anyone in terms of preserving the person's appearance and identity.\n\nThis work uses optical flow as the motion condition. Does it account for the motion of the background? Generating consistent background motion would make the video more realistic. The framework could be further improved by considering the motion of multiple objects.\n\nQUESTIONS:\nNone\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-03"}