Title: MoCha: Towards Movie-Grade Talking Character Generation
Venue: NeurIPS 2025
Accepted: yes
Keywords: video generation

diffusion

lipsync

talking head

dialogue-driven video generation
Abstract: Recent advancements in video generation have achieved impressive motion realism, yet they often overlook character-driven storytelling, a crucial task for automated film, animation generation. 
We introduce Talking Characters, a more realistic task to generate talking character animations directly from speech and text. Unlike talking head tasks, Talking Characters aims at generating the full portrait of one or more characters beyond the facial region. 
In this paper, we propose MoCha, the first of its kind to generate talking characters. To ensure precise synchronization between video and speech, we propose a localized audio attention mechanism that effectively aligns speech and video tokens.
To address the scarcity of large-scale speech-labelled video datasets, we introduce a joint training strategy that leverages both speech-labelled and text-labelled video data, significantly improving generalization across diverse character actions. We also design structured prompt templates with character tags, enabling, for the first time, multi-character conversation with turn-based dialogue—allowing AI-generated characters to engage in context-aware conversations with cinematic coherence.
Extensive qualitative and quantitative evaluations, including human evaluation studies and benchmark comparisons, demonstrate that MoCha sets a new standard for AI-generated cinematic storytelling, achieving superior realism, controllability and generalization.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a text-to-video model that supports speech audio conditioning. It details the implementation of progressive training and audio conditioning techniques. Built on a 30B base model, the approach outperforms baseline methods across multiple evaluation metrics.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The paper is clearly written, providing detailed explanations of the implementation.\n2. The visual results are strong, showing noticeable improvements over baseline methods.\n\nWeaknesses:\n1. The experimental setup is flawed: the baselines compared are image-plus-audio to video models, but the test images come from the proposed text-to-video model. This likely makes the test images out-of-domain for the baselines but in-domain for the proposed method, resulting in an unfair comparison.\n2. The method shares many similarities with existing work, such as the audio conditioning approach and multi-stage training, but the paper lacks clear discussion or analysis of these overlaps.\n3. The method relies on a large, strong base video generation model which differs from other methods and this difference is not addressed or analyzed in the paper.\n\nQUESTIONS:\n- Using audio conditioning with features extracted from adjacent frames and multi-stage training is very common in audio-conditioned video generation methods such as EMO, Hallo, Hallo3, and OmniHuman. A clearer explanation of the paper’s unique contributions in this context would be helpful.\n- The current experimental comparison setup is inconsistent for other methods, making fair evaluation difficult, as noted in the weaknesses. It is recommended that the authors compare their method directly with other text-to-video approaches (e.g., using the metrics in Figure 7) while excluding audio-related metrics, which are already clearly reported in Table 2.\n- The authors should provide some evaluation or discussion of the base model’s performance itself, as this would improve the completeness and clarity of the paper.\n\nLIMITATIONS:\n- The paper has some questionable experimental settings, such as comparing image-to-video methods using images generated by the proposed text-to-video model.\n- It lacks analysis and comparison with other text-to-video models, including the base video generation model.\n- The method’s contributions are unclear, as audio conditioning and multi-stage training are common practices in this field.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-29"}

comments: SUMMARY:
Mocha proposed a DiT-based text- and speech-driven Talking Character generation method, which uses cross attention and localized audio attention to inject text and speech conditions, achieving state-of-the-art performance in Talking Character generation.

STRENGTHS AND WEAKNESSES:
**Strengths:**

1. Compared to other methods, this approach achieves video generation that is better aligned with both text and speech, marking a significant step toward the practical application of video generation.
2. The use of localized audio attention enables better alignment between lip movements and speech in the generated videos, making it a sensible and effective design.

**Weaknesses:**

1. The method has not been open-sourced, nor are there plans for open-sourcing it. Additionally, the experiments were conducted using Meta’s internal MovieGen framework, making it difficult for the community to reproduce the results.
2. The five steps mentioned in Curriculum Training lack ablation studies for each stage, making it seem unnecessary to design such a complex process.
3. The method has high resource consumption. Despite being based on a powerful video generation model, it still requires 64 computing nodes for post-training. It does not consider freezing most of the network modules during training or using other efficient fine-tuning methods.

QUESTIONS:
See weakness.

LIMITATIONS:
As mentioned in weakness, this method is based on a closed-source model and the experimental overhead is too high to be reproduced.
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 2
DATE: 2025-06-29

comments: SUMMARY:
The paper presents a new complex task of talking character generation, i.e. generating lip-synced video of full portrait of one or more characters beyond just the facial region. Given it is a new task, the paper also presents a new benchmark and method for the same. The method is novel for four reasons: 1) The method does not use any additional signals apart from the coarsely aligned speech and video description text 2) Uses windowed attention between video and audio instead of full cross-attention 3) Uses curriculum-based training across two axes - the modality and task complexity 4) able to generate coherent multi-character clips in parallel. Subjective (human rating on scale 1-4) and Objective results (lip-sync metrics) clearly state the effectiveness of the method. The benchmark for talking character generation contains 200 instances of human verified text, speech and video.

STRENGTHS AND WEAKNESSES:
Strengths:
1. Introduces new task, new benchmark and new method for talking character generation. 
2. Limitation section is well thought out.
3. The appendix is detailed and contains good ablation studies and subjective evaluation details
4. Writing is clear and good.
5. The proposed method clearly outperforms existing methods both in subjective and objective evaluations.

Weaknesses:
1. Please include some sections from appendix to main work in the final version
2. No mention of intention to open source the code, data and checkpoints. Though this is not compulsory but recommended.

QUESTIONS:
1. The authors have presented three good limitations in the appendix. What are some possible ways to resolve these problems and further improve the method?
2. What is the total number of training hours and what kind of GPUs were used for training?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 4
IMPACT: 4
ORIGINALITY: 4
DATE: 2025-06-30

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents an end-to-end video generation model that synthesizes talking characters from only text and audio inputs. It goes beyond facial regions to include full-body motions, multi-character interactions and environmental dynamics. To this end, the authors modify the text-to-video DiT model and inject audio conditions through temporally aligned audio cross-attention. A curriculum-based multimodal training strategy is introduced to combine speech-labeled and text-only data to improve generalization. Experiments show that the proposed method support coherent, turn-based dialogue among multiple characters, outperforming baselines in lip-sync quality, expression naturalness, and text alignment.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n* Unlike traditional talking-head generation (limited to facial regions and single speakers) , this paper explicitly targets full-body generation with multi-character interactions. This framing fills a clear gap in automating cinematic storytelling and content creation, \n\n* Human evaluations across five axes (lip-sync, expression naturalness, action alignment, etc.) show that the proposed method outperforms existing baselines (SadTalker, Hallo3) by a large margin.\n\n* The paper is overall well-written, and the technical details is clearly provided in the appendix. \n\n### Weaknesses\n* The so-called \"Localized Audio Attention\" seems straight-forward and is not novel to me. Injecting audio condition through temporally-aligned cross-attention has already been explored in Hallo3 [27]. It would be better if the authors could provide more discussion on the difference between this method and Hallo3 in terms of the design of audio injection modules. \n\n* The curriculum-based multimodal training strategy is also not novel enough; a similar idea has already been proposed in OmniHuman [20], which also introduces a multi-stage training strategy and follows a curriculum progressing from text to audio. Unfortunately,  this paper does not explicitly address the connections and distinctions between its strategy and that of OmniHuman.\n\n* The baselines in the comparison experiments rely on reference image inputs. However, it is unclear how the reference images are selected for these baselines in the comparison. In addition, it is unclear how the proposed method could be adapted to reference-conditioned settings.\n\nQUESTIONS:\nNone\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-02"}