Multi-Shot Character Consistency for Text-to-Video Generation

Authors: Yuval Atzmon, Rinon Gal, Yoad Tewel, Yoni Kasten, Gal Chechik
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: text to video, subject consistency, video personalization, motion alignment, feature injection, extended attention

Abstract:
Text-to-video models have made significant strides in generating short video clips from textual descriptions. Yet, a significant challenge remains: generating several video shots of the same characters, preserving their identity without hurting video quality, dynamics, and responsiveness to text prompts. We present Video Storyboarding, a training-free method to enable pretrained text-to-video models to generate multiple shots with consistent characters, by sharing features between them. Our key insight is that self-attention query features (Q) encode both motion and identity. This creates a hard-to-avoid trade-off between preserving character identity and making videos dynamic, when features are shared. To address this issue, we introduce a novel query injection strategy that balances identity preservation and natural motion retention. This approach improves upon naive consistency techniques applied to videos, which often struggle to maintain this delicate equilibrium. Our experiments demonstrate significant improvements in character consistency across scenes while maintaining high-quality motion and text alignment. These results offer insights into critical stages of video generation and the interplay of structure and motion in video diffusion models.

Decision: reject
Reviewer scores: 5, 5, 8, 5

Reviews and discussion:
<thread>
creation date:
1730291146803

thread_type:
official_review

<thread_object>
creator: reviewer_ntrv

<content>
summary:
This paper aims to generate multi-shot videos with consistent characters in a zero-shot manner. They claim that there is a trade-off between preserving character identity and video dynamics, thereby designing a two-phase approach, Q-preservation and Q-Flow, to balance the two respects.

strengths:
1. **Training-Free Approach for Subject Consistency Across Shots**: This work offers a training-free method for generating subjects with consistent identity across varying scenarios and shot transitions, which is valuable for practical applications where maintaining coherence in subject appearance is essential.

2. **Novel Insights on Self-Attention Query Features**: The authors provide fresh insights into the role of self-attention query features, demonstrating that these features effectively capture both motion and identity.

3. **Query-Preservation and Q-flow Techniques**: By preserving query features during early denoising and applying a tokenflow-inspired approach to select keyframes, the method achieves partial injection of query features to adjacent frames. Although it draws heavily from ConsisStory and TokenFlow, this approach has demonstrated effectiveness in enhancing subject consistency and motion dynamics to a certain extent.

weaknesses:
1. **Limited Novelty in Video Storyboarding**: The innovation of the proposed video storyboarding approach is limited. The primary method relies on frame-wise SDSA, which largely mirrors the approach used in ConsiStory. The only notable difference lies in the mask source, utilizing CLIPseg and OTSU segmentation rather than cross-attention. 

2. **Poor Writing and Project Organization**: The paper's writing and the project page's layout hinder comprehension, making it difficult for readers to follow the key contributions the authors intend to convey.

3. **Minimal Improvement over Baseline Models**: The generated video storyboarding results appear similar to those produced by existing video generation baselines like Videocrafter2 or TokenFlow encoder, with little noticeable difference in output quality.

4. **Lack of Motion Dynamics**: The method demonstrates limited motion dynamics. In most video segments, the objects remain static, and in every case, the object consistently occupies the center of the frame, resulting in rigid, uninspired visuals.

5. **Overclaiming the Benchmark**: The authors’ claim of establishing a benchmark based on a dataset of only 30 videos, each containing 5 video shots, is unsubstantiated. This dataset is insufficiently sized and lacks diversity, with evaluations limited to character consistency and dynamic degree, providing a narrow view that does not comprehensively assess the model's capabilities.

questions:
1. **Inconsistencies in Object Appearance and Action**: In the ablation study on query preservation (`ablation_q`), inconsistencies persist. For example, in the first video, the right hand of the Muppet character appears red, while in the third shot, it is not. Additionally, although the Muppet is intended to perform aerobics in a Sesame Street setting, it merely flicks its hand briefly, failing to convey the intended action sequence.

2. **Static Object Issues in ConsiStory Component Ablation Study**: In the ablation study on ConsiStory components for video generation, the rabbit character intended to surf, train, and ride a bike appears mostly static in the first and third shots. This raises the question of whether these issues stem from limitations in the base model’s dynamic capabilities. If so, would using models with stronger dynamic performance, such as Dynamic Crafter or CogVideo, potentially improve motion consistency and address these static object limitations?

If the video dynamic problem is addressed, I am willing to increase my score.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730293000528

thread_type:
official_review

<thread_object>
creator: reviewer_yj9c

<content>
summary:
This paper proposes a method called "Video Storyboarding" to generate multi-shot videos with consistent characters across different scenes while preserving motion quality.

strengths:
1. Framewise Subject-Driven Self-Attention to maintain consistency without compromising motion
2. Novel two-phase query injection strategy to balance identity preservation and motion quality
3. Adaptation of refinement feature injection to both conditional and unconditional denoising steps

weaknesses:
1. Approach limited to short video clips, unsure how it would scale to longer videos
2. Balancing identity preservation and motion quality is still challenging, with potential tradeoffs

questions:
Overall, this is a well-designed and rigorously evaluated method that represents a significant advancement in generating coherent multi-shot video sequences. The authors have done a commendable job in addressing the complex challenge of maintaining character consistency while preserving natural motion.

Some questions for the authors:

1. Have you explored any strategies to further improve the balance between identity preservation and motion quality? Are there other techniques beyond query injection that could be investigated?
2. How do you envision this approach scaling to longer video sequences? What additional challenges might arise, and how could the method be adapted to handle them?
3. The user study results showed that the ConsiS Im2Vid baseline achieved the highest set consistency among the baselines. Can you comment on the strengths of this approach and how it might be combined or compared with your Video Storyboarding method?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730626291734

thread_type:
official_review

<thread_object>
creator: reviewer_klfg

<content>
summary:
This paper presents Video Storyboarding, a training-free method that enhances pre-trained text-to-video models to generate multiple shots with consistent characters while maintaining high video quality and responsiveness to text prompts. By leveraging self-attention query features that capture motion and identity, the method addresses the trade-off between character consistency and video dynamics through a novel query injection strategy. Experimental results show significant improvements in character consistency and motion quality, offering insights into the video generation process and the interplay of structure and motion in diffusion models.

strengths:
1. This paper introduces a training-free method to ensure charactoer consistency and motion adherence in producing multi-shot video sequences.
2. This paper presents a two-phase query injection strategy to balance encoding motion and identity.
3. A benchmark and evalution protocol are proposed to evaluate consistency of video generation.

weaknesses:
1. The conducted experiments are not comprehensive, including two aspects: (a) The paper only provides several comparison samples. (b) The paper misses some important baseline methods, e.g., VSTAR[1]
2. Although the purpose of the paper is to maintain consistency of characters across different video clips, the results are not particularly good. For example, in Fig.3, the color and style of clothes change across different video shots.

[1] Li, Yumeng, William Beluch, Margret Keuper, Dan Zhang, and Anna Khoreva. "VSTAR: Generative Temporal Nursing for Longer Dynamic Video Synthesis." arXiv preprint arXiv:2403.13501 (2024).

questions:
1. The paper only demonstrates the performance of a single character across different videos, and the reviewer is curious about how the proposed  method performs with multiple characters.
2. The prompt in the paper provides overly detailed descriptions of the character. Would a more concise description impact character consistency? For example, replace the "Cinematic, middle-aged female athlete" in Fig.8 with "A woman".
</content>
</thread_object>

</thread>
<thread>
creation date:
1730657786204

thread_type:
official_review

<thread_object>
creator: reviewer_e8ut

<content>
summary:
This paper targets the problem of character consistency in text-to-video generation. The authors propose a training-free method to solve this problem. They find the query in the attention encodes the information of both motion and identify, which leads to the trade-off between motion dynamics and identity consistency. The experimental model used is VideoCrafter2. To solve the trade-off problem, they propose a new query injection method. Specifically, they share features between different video clips. Then, they replace the Q (query) with those from the original generation (to maintain motion from the original generation). After that, they leverage the flow map from vanilla keyframes to guide the Q injection. Their results achieve the character consistency while keeping the original motion dynamics and text alignment. The text alignment is evaluated via user study. The overall metrics for evaluation are three aspects: motion degree, id consistency, and motion text alignment.

strengths:
1. The method is tuning-free and does not require any further training.
2. The results outperform baseline methods. Some of the provided visual results look good.

weaknesses:
1. The paper is relatively hard to follow regarding the details of the method part.
2. Lack of novelty: 1. The id presevering mechanism is built upon the SDSA. the SDSA is adopted from ConsiStory [Tewel et al. 2024] with two minor modifications: (1) The attention not attend to each other with all frames from different clips, but one single frame from each clip. (2) The mask estimation use ClipSeg, rather than estimated from the cross attention. 2. The motion preserving is leveraging TokenFlow [Geyer et al. 2023] to inject the motion based on the flow from original keyframes. Thus, the method is like a A+B combination with some minor modifications.
3. The key insight "self-attention query features (O) encode both motion and identity" lack experimental results to demonstrate.
4. The results are not perfect, e.g., inconsistent hairstyles in the 3rd row of Figure 1.
5. The evaluation does not contain the overall video generation quality and the qualitative semantic alignment scores. 
6. Minor formate issues like inconsistent figure reference:  Figure 1 and Fig. 4; And strange line break at line 244 and line 291.

questions:
1. Does the overall generation quality decrease after the proposed method?
2. How does the motion quality changed after the proposed method?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732487575288

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General response to reviewers

comment:
We thank all the reviewers for their useful and insightful feedback. We are encouraged that the reviewers found our work to offer “**novel insights**” into structure and motion (ntRv, kLfg) and commended our “**novel two-phase query injection strategy**” (yj9C, kLfg, ntRv). The reviewers appreciated our “**valuable**,” tuning-free problem setup (**all**) and found our work to be “**sound**” (e8uT, kLfg, yj9C), while acknowledging our method as “**effective**” (ntRv), with “**significant improvements**” (kLfg), “**outperform baseline**” (e8uT) and being “**rigorously evaluated**” (yj9C).


Our work has benefited tremendously from your feedback. Below are the main modifications to the manuscript (colored blue in the updated PDF):

1. **Stronger pretrained model, T2V-Turbo-V2**

    Following Reviewer ntRv’s suggestion, we tested our method with T2V-Turbo-V2 [Li et al. NeurIPS 2024], a state-of-the-art video model with enhanced motion capabilities. Results (Appendix: Table 1, Figure 8) show a threefold improvement in dynamic degree (from 20 to 62) while maintaining text alignment and subject consistency.

2. **Expanded Experimental Results and Qualitative Demonstrations**

    - **VSTAR Baseline Comparisons**: We added qualitative and quantitative results comparing our method with VSTAR (Appendix: Table 1, Figure 11), as suggested by Reviewer kLfg.
    - **Semantic Alignment Metrics**: New results include text-similarity and subject-consistency scores (Appendix: Table 1), as suggested by Reviewers (e8uT, ntRv).
    - **Additional Qualitative Results**: We demonstrated our method’s ability to handle general subject categories like “woman” or “rabbit” (Appendix: Figure 9) and multiple subjects (Appendix: Figure 10), as suggested by Reviewer kLfg.

3. **Improved Clarity**: 

    We refined the paper’s writing, as suggested by Reviewers (e8uT, ntRv). Specifically, we added a dedicated *Notations Section* to define key terms, and revised the methods section to align with ConsiStory's three-step structure. Technical derivations and complex details were moved to the appendix to ensure the main text focuses on core challenges and solutions.


We look forward to addressing any remaining questions from the reviewers and continue engaging in further discussion. If you find our response satisfactory, we kindly ask you to consider raising your rating in recognition of our core contributions.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733310556009

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
A final note

comment:
Dear Reviewers and AC,

We sincerely thank you for your valuable time and insightful feedback, which has greatly benefited our work.

In a final note, we wish to emphasize that 3 of 4 reviewers found our approach to be “**sound**”, offering “**novel insights**” into the representation of motion, structure and identity (ntRv, kLfg) and commended our “**novel two-phase query injection strategy**” (yj9C, kLfg, ntRv). 

In a broader context, for the first time, our work enables consistent multi-shot video generation, combined with motion-preservation. Notably, the baseline alternatives exhibit very limited performance, both qualitatively and quantitatively, which renders them largely ineffective in this task. Empirically, users had 2-4x preference over the alternatives (66-79%). Additionally, our new experiments with T2V-Turbo-V2 demonstrated that limitations such as video quality and motion issues diminish as our approach scales to stronger models.

We are confident that our work contributes to the dialogue in the field, advancing the understanding of motion, structure and identity in diffusion models,.

Thank you for considering our work.

Best regards,

The Authors
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper presents a training-free approach to ensure character consistency and motion adherence in generating multi-shot video sequences.

The paper received mixed ratings (5, 5, 5, 8) after rebuttal, with key concerns revolving around clarity of writing (e8uT, ntRv), novelty of the method (e8uT, ntRv), and quality of results (e8uT, kLfg, ntRv).

In the post-rebuttal discussion, reviewers acknowledged that the revised version improved clarity. However, concerns about the novelty and quality of results remained. Specifically:
- The motivation of “query features encode both motion and identity” was interesting, but was not convincingly demonstrated through the results. 
- The proposed method offered only incremental innovations compared to ConsiStory and TokenFlow.
- Despite including T2V-Turbo, the results failed to address issues with motion consistency, video fidelity, and multi-shot consistency.

The AC agrees with the majority of the reviewers and recommends rejecting the paper.