MoCha: Towards Movie-Grade Talking Character Generation
Authors: Cong Wei
Bo Sun
Haoyu Ma
Ji Hou
Felix Juefei-Xu
Zecheng He
Xiaoliang Dai
Luxin Zhang
Kunpeng Li
Tingbo Hou
Animesh Sinha
Peter Vajda
Wenhu Chen
Venue: NeurIPS 2025 spotlight ()
Keywords: video generation
diffusion
lipsync
talking head
dialogue-driven video generation
Primary area: applications
TL;DR: We introduce MoCha, the first model for dialogue-driven movie shot generation.

Abstract: Recent advancements in video generation have achieved impressive motion realism, yet they often overlook character-driven storytelling, a crucial task for automated film, animation generation. 
We introduce Talking Characters, a more realistic task to generate talking character animations directly from speech and text. Unlike talking head tasks, Talking Characters aims at generating the full portrait of one or more characters beyond the facial region. 
In this paper, we propose MoCha, the first of its kind to generate talking characters. To ensure precise synchronization between video and speech, we propose a localized audio attention mechanism that effectively aligns speech and video tokens.
To address the scarcity of large-scale speech-labelled video datasets, we introduce a joint training strategy that leverages both speech-labelled and text-labelled video data, significantly improving generalization across diverse character actions. We also design structured prompt templates with character tags, enabling, for the first time, multi-character conversation with turn-based dialogue—allowing AI-generated characters to engage in context-aware conversations with cinematic coherence.
Extensive qualitative and quantitative evaluations, including human evaluation studies and benchmark comparisons, demonstrate that MoCha sets a new standard for AI-generated cinematic storytelling, achieving superior realism, controllability and generalization.

MoCha: Towards Movie-Grade Talking Character Generation
Cong Wei! Bo Sun", Haoyu Ma', Ji Hou', Felix Juefei-Xu', Zecheng He², Xiaoliang Dai?, Luxin Zhang?. Kunpeng Li,, Tingbo Hou? Animesh Sinha'. Peter Vajda', Wenhu Chen is "University of Waterloo GenAI, Meta {cong.wer wei, wenhu. chenc @uwaterloo.ce
Talking Character
"Close-up shot of doctor in white lab coat over blue scrubs, speaking.."
Camera Movement Control
+*A tilt up shot nan standing in dinly Lit roun, speaking to the tamera.."
Emotion Control
+
"Two distinct streans of tears trail down her cheeks as she speaks with an angry expression.""
Action Control
interacting warnly
talks
+ "Two video clips. Personl: Waran with short brown hair.. Person2: Mon with curly hair- First cLip: Personi inside space station. Second clip: Person2 in the some cabin.""
Calti-Character Turn-based Talk
Figure 1: MoCha is an end-to-end dialogue-centric video generation model that takes only speech and text as input, without requiring any auxiliary conditions. More videos are available at http://coogwee1222..ghhunnttc
Abstract
Recent advancements in video generation have achieved impressive motion realism, yet they often overlook character-driven storytelling, crucial task for automated film, animation generation. We introduce Talking Characters, a more realistic task to generate talking character animations directly from speech and text. Unlike talking head, Talking Characters aims at generating the full portrait of one or
WWor done during the first author's internship GenAI, Project Lead Cooresponniig Author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

more characters beyond the facial region. In this paper, we propose MoCha, the first of its kind to generate talking characters. To ensure precise synchronization between video and speech, we propose a localized audio attention mechanism that effectively aligns speech and video tokens. To address the scarcity of large- scale speech-labelles video datasets, we introduce joint training strategy that leverages both speech-labeled and text-labeled video data, significantly improving generalization across diverse character actions. We also design structured prompt templates with character tags, enabling. for the first time, multi-character conver- sation with turn-based dialogue aallowiigalliigggggg/. AI- Al-generated characters engage in context-aware conversations with cinematic coherence. Extensive qualitative and quantitative evaluations, including human evaluation studies and benchmark com- parisons, demonstrate that MoCha sets a new standard for Al-generated cinematic storytelling, achieving superior realism, controllability and generalization.
Introduction
Automating film production holds immense commercial potential, promising to democratize cinematic-level storytelling by enabling content creators effortlessly generate films through natural language [[-]] In film, dialogue plays a central role in conveying narratives. Ideally, creators should be able to specify rich storylines involving multiple tharacters-whetiee cartoons-that engage in meaningful dialogue, express emotions, and perform full-body actions. Such talking characters serve as powerful mediums for delivering impactful messages, communi- cating idcas, and engaging audiences. Beyond film, they also enable a wide range of downstream applications, including digital assistants, virtual avatars, advertising, and educational content.
realistic humans stylized
Despite impressive progress in video generation, current video foundation models are primarily designed for narration-style, non-dialogue scenes. Models such as SoRA, Pika, Luma, Hailuo, and Kling [H produce characters with arbitrary lip movements and disconnected facial expressions, lacking control over actual speech content. As a result, these characters appear lifeless and fail to deliver messeage to the audience, limiting the models' applicability in real-life cinematic production.
Meanwhile, speech- crrech-coniiiooeed video generation is still in its infancy, primarily focused on simplified talking-head scenarios. Models such as Loopy. Hallo3, and EMO [3]11 are limited to cropped face regions with static cameras, ignoring essential elements such as full-body actions, camera motion, and multi-charann interactions. These limitations hinder their characters' expressiveness and make them unsuitable for realistic, engaging storytelling.
To bridge the gap between non-dialogue video generation and constrained talking- head synthesis, we introduce the novel task of Talking Characters, which directly targets the goal of automating dialogue-centric film production. The task is defined as generating lifelike digital characters from natural language and speech inputs that express synchronized speech, realistic emotions, and full-body actions under dynamic camera movements (see seectsstionn 2). To tackle this task, we propose MoCha, the first and-to-end Diffusion Transformer (DiT) model designed to produce high-quality. movic- -rade talking character videos. MoCha demonstrates compelling storytelling capabilities. As shown on the project website ahttps///ongweill 230. 230. iothub.io/MoCha/ we lightly edited MoCha- generated clips MoCha-generated into a 1-minute, emotionally engaging narrative, illustrating its potential for real-world filmmaking.
MoCha introduces several key technical innovations tailored for this task:
First End-to-End DiT Without Auxiliary Control Signals: MoCha is the first DiT-based model to demonstrate that high-quality lip synchronization and natural character motion can be achieved using only text and speech-withond reference images, pose skeletons, or facial keypoints [14] 15] 20 [3 16 17). Contrary the dominant belief that audio alone is insufficient, we show that strong audio-visual alignment can emerge purely from end-to-end training.
relying external control signals such as
We propose a attention mechanism tailored for DiT-based dialogue-diven video generation, which addresses the temporal mismatch between com- pressed video tokens and high-resolution audio inputs (see See. (.2). This design signifi- cantly improves lip-sync accuracy and speech-video alignment.
Localized Audio Attention:
2

---

Curriculum-Barn Multimodal Training: We introduce training strategy that combines limited speech-labeled and large-scale text-only video data through a modality-award cur- riculum. By mixing multimodal and unimodal supervision and progressively increasing visual complexity, our approach improves convergence, enhances generalization, and enables MoCha's universal controllability via natural language prompts cranpts-suuportinn fine-grained control over character expressions, actions, interactions, camera movement, and scene composition without relying on auxiliary signals.
Hulti-Character Conversational Video Generation: MoCha is the first model to support coherent, turn-based dialogue among multiple ovvercominoo the single-speaker limitation of prior methods. This capability enables cinematic, story-driven video synthesis with dynamic character interactions.
To evaluate MoCha's performance, we further curated MoCha-Bench, benchmark tailored for Talking Characters generation tasks. Both human evaluations and automatic metrics demonstrate that MoCha set a new standard for talking character video generation and represents a significant step toward achieving controllable, narrative-driven video synthesis, with broad applications in film production, animation, virtual assistants, and beyond.
2 Problem Definition: Talking Characters
We introduce novel task, Talking Characters, which focuses on generating digital characters that exhibit realistic, human-like behaviors from natural language and speech input. The task is motivated by the goal of automating dialogue-centrie film production, going beyond traditional video generation that typically focuses on non- non-dialogue, narration-baa scenes.
Talking Characters differs from conventional talking-head ceneration-whicl is restricted to single face, fixed camera, and square crop- by enabling full-body character synthesis across range of shot sizes (e.g., close-up, medium, wide) with dynamic camera motion. It supports the generation of one or more characters situated within a contextually appropriate scene.
The task is formally defined by the input-output specification and evaluation protocol, in Table
Aspect
Description
Input
Tex Prompt y: Natural language description including (1 (L)envirommen,, (2)character appearance and (3)attions, (4)facing direction and emotion. (5)fre position, (6)camera movement, and (7)shot size. Speech Audio a E R": Raw waveform signal that drives characters' lip movements, facial expressions, and body motions.
Output
Video v E T x W33, A rendered video of one or more talking characters (human, 3D cartoon, or animal), where T is the number of frames and (H, W) is the spatial resolution.
Eval
The generated characters are expected to perform well across the following five axes:
(1) Lip-Syne Quality: Accurate and temporally aligned lip movements with respect
(2) Expression Naturalness: Expressive and coherent facial emotions that align with both and
(3) Action Naturalness: Realistic body gestures reflect deseribed actions in y, synchronized with a.
(4) Text Alignment: Coherence between visual content and y, including Visual layout, character appearance, and camera motion.
(5) Visual Quality: High fidelity and temporal consistency, free from visual artifacts.
Table 1: Task definition of Talking Characters: formalized inputs av, output v, and evaluation criteria.
3 Model: MoCha
this section, we introduce the MoCha model, the first model to generate talking characters. We begin by outlining its architecture in musubecctiinn 3.1, followed by the localized audio attention mechanism in ssubsection 3.2] Next, we deseribe the method of generating multiple clips in subsection 3.3) Finally provide explanation of the training strategy in subsection 3.4

---

"A close-up shot of a woman standing in modern house, speaking to the camera while facing straight ahead
3D VAB
She has shoulder-Length dark hair
Window Cruss Attenionn
Way 2eec2
The background features
Cross Attention
sleek, modern furniture
Projector
Projector
Self Attention
She holds single rose in her hands, gently running her fingers along its petals
Specch Tokens
Test
DiT block
XN
Tekers
The static camera captures her face upper body"
Speech
Noisy Video Tokens
Figure 2:
MoCha Architecture. MoCha is an end-to-end Diffusion Transtormer(Dit) that generates video frames from the joint conditioning of speech and text, without relying any auxiliary signals. Both speech and text inputs are projected into token representations and aligned with video tokens through cross-attention.
3.1 Audio + Text to Video Diffusion Transformers
Figure presents the overall architecture of MoCha. Unlike prior works that employ rext-to-image for talking head generation, MoCha is a diffusion transformer (DiT) [2]
(T2I) U-Net [1 15
Model Architecture. MoCha adopts fully tokenized design, where both text and speech inputs are projected into token sequences and integrated with video tokens through cross-attention, Given an video y E H x3 with T frames, we encode into latent representation 2o E IR R using a 3D VAE, which down-samples the video spatially and temporally. We define the temporal down-sampling ratio and passed to the DiT model fe(-). Within each DiT block, the model first applies self ceff-attention the video tokens ro, followed by sequential cross-attention with the text condition tokens y and audio condition tokens a. The audio tokens E IR is derived from raw waveforms using Wav2Vec2 24 and processed through MIPs align its feature dimension with the video tokens.
I. Next, xnp flattened into a sequence of tokens of size (r x w)
Training Objective. We adopt Flow Matching [25], which enables efficient simulation of continuous-tinn time dynamies, to train model. Given a latent video representation 2 E R xxxxxxxxx (encoded from the input video), random noise e ~ (o,, 7), and a continuous time step / E [0, 1J. we construct intermediate latent Tı by interpolating between and æ1:
(1
Lx1.
The model is trained to predict the velocity, defined as the difference between the data and noise:
dx.
11 dt
x1
(2)
The training loss is then:
(x1 =0
I. = -EENN a a, tE(0.1] So (at,
where si is the latent video, y and text and audio conditions, and So (-) is the DiT model. Unlike prior works [10 14 [5]
16 21 17, MoCha does not rely on auxiliary objectives such as face body masking. Instead, it learns speech-video correlations purely from data, using fully tokenized and end-to-end training approach.
20 13
Localized Audio Attention
Most existing talking head generation models rely on 2D diffusion architectures (e.g.. U-Net) that generate frames autoregressively, with each frame vs conditioned directly on its corresponding audio token E R This one-to-one mapping between audio and video timesteps naturally ensures tight synchronization. In contrast, diffusion transformer (DiT) architectures depart from this design in two critical ways that complicate temporal alignment:

---

Noisy Video
frome
fromes
Jromey
frown
Figure 3: MoCha's Localized Audio Attention. To address the mismatch between com- pressed video tokens and high- resolution audio, MoCha em- plays a localized attention strat- egy where each video token at- window of nearby audio tokens. This promotes precise lip-sync and improves temporal coherence.
e E p2lskooo
Wav2Voe2 Projector
Audio
Taken
(Key'
Value)
3V% GE
tends only
frome
Video
x(Qurry)
g
1. Temporal Resolution Mismatch: DiT-based models operate on latent representations produced by a 3D VAE, which temporally compresses videos by a factor of r (commonly r = 4 or 8 in recent T2V models (8] [o]) As result, video tokens span only T/r steps, while audio remains at the original resolution T,, eliminating direct alignment.
2. Fully Parallel Decoding: Unlike autoregressive designs, DiT generates all latent frames in parallel. Without constraints, naïve cross-attention permits each video token to attend globally to the full audio sequence, which can lead to incorrect associations.
To address these challenges, we propose a Localized Audio Attention mechanism that introduces a temporal inductive bias. Inspired by the observation that lip movements depend on short-term phonetic patterss while body gestures and expressions reflect longer-term textual semantics, we constrain each video token's 's attention to a limited audio span.
Specifically, for a latent video frame (1) E hxxxx at timestep i E {L....,7]. we restrict its cross-attention to audio tokens (i) in the window:
j E [max(11 (i-1)r-1), iinn(Tir+1))
This r+ 2-toet window corresponds to the uncompressed temporal segment that (0() summarizes, with one token of padding on each side for context smoothing. This simple yet effective constraint encourages alignment between speech and video while preserving local continuity across frames.
3.3 Multi-character Turn-based Conversation
Thanks to the clean and fully tokenized design of MoCha (see subseecion 3.1). the model supports multi-clip video generation in exactly the same way as single-clip generation without any addi- tional architectural modifications. As illustrated in Figure unlike video extension methods that rely on autoregressive generation conditioned on previously generated content, MoCha generates all clips in parallel. It leverages self-attention across video tokens to maintain character consistency across clips and preserve coherence in the surrounding environment.
Assuming only character speaks at time, observe that speaker changes in the audio implicitly guide MoCha to transition between clips- withhut requiring any explicit indicators such as clip tokens 26 Simultaneously, the text condition y specifies the content of each clip.
Formally, given an audio sequence a aa .e......000 aT), where the tokens correspond to two speakers segmented
a²), with r being the VAE temporal down- sampling ratio and & being an integer index. The model generates the latent video sequence in parallel as: x fo a). where (a x² ...... ..,,,, ,²), = T/r. The output sequence can be segmented into two clips aligned with the respective speaker turns:
(a' ..... rk) | (k+1
While MoCha supports seamless multi-clip generation, challenge lies in prompting ensuring that character attributes are consistently grounded across clips. This becomes especially difficult when characters interact or reappear in different clips. Naive captioning models typically rely on visual descriptions to refer to characters. As a result, they must repeat detailed appearance descriptions each time a character is mentioned, leading to long, redundant, and confusing prompts. For example in Figure 5] This verbosity not only increases the risk of exceeding token limits (e.g., 256 tokens) also confuses the model during generation geserration-sppcciilll in multi-clip scenarios.
5

---

Clip Pensenl Soene
Person2 Soone
Prompt
Video
Two video clips.
Characters:
NAB
Persont: Woman short grey hair,
medium skin tonc, wearing maroon
Viloo Token
sweater and apron.
Person2: Man with beard, darker skin
tone, wearing a black t-shirt and jeuns.
Projector
Window Cross Ateerionn
Encoder
First clip: Personl stands cozy
Speech
Cross Attention
Projector
kitchen, facing loft, speaking warmly
Encoder
while preparing
on the counter, Pots
Self Attention
and pans hang in the background The
cumeru grudually zooms on face.
Notty Viceo
Tiken
Second clip: Person2 stands on the other
Speck
-Blll
Iun
side of the counter, facing left, smiling and replying to Personi. The camera remains static."
1imatoe
Speech
Person2 Sprech
5s
Figure 4: Multi-character Conversation and Character Tagging, MoCha supports generating multi-characte conversion with scene cuts. We design specialized prompt template: first specifies the number of clips, then introduces the characters along with their descriptions and tags. Each clip is subsequently descrihed using only the character tags, simplifying the prompt while preserving clarity. MoCha leverages self-attention across video tokens to ensure character and environment consistency. The audio signal implicitly guides MoCha when the transition between clips happens.
"The first video clip captures girt aged 10-15 in green dress waving another girl dressed in green shirt with braided hair holding book... another girl dressed
"The Video Clip- Characters: [Person])A girl aged 18-15 in green dress [Person2)A girl in a green shirt with braided hair (Person3)A girl aged 18-15 in blue hoodie
green shirt in the niddle of the frane respords with a saile... Nearby, another girl aged 18-15 in a blue hoodle points touard the whiteboord while looking the girl dresses in the girl in shirt with braided heir.
Clip1
waves to Person2 holding Person2 responds with snile... Nearty, Personl,
Clip:
The second video clip captures the girl in green dress who stands on the left side of the frome is gone. The girl in a green shirt with braided hair
whirtebard
looking
hands a book to another girl in a blue hoodia. The girl. in the blue hoodie gently takes the book with
Second CLip: Personl is gone. Person2 hands book to Person3. Person3 gently takes the book with soft seile."
soft smile."
Skmmns
Clip2
captioning is long, redundunt, confusing
(Oues) character tagping
compact
Figure 5: Character tagging provides more compact and structured prompts compared to naive captioning.
We address this by introducing structured prompt template with fixed keywords and character tagging mechanism that promotes clarity, compactness, and consistency:
"Two Video Clip. Characters:" Defines list of characters, each described by visual attributes and assigned a unique tag (27 (e.g., Person1, Person2).
"First clip", "Second clip" Each video segment is described using only the defined character tags.
This design significantly reduces redundancy and helps the model reliably associate visual attributes with character actions, even across multiple clips.
3.4 MoCha Training Strategy
We identify two key challenges in training high-quality talking character models: (i) Data Scarcity and Limited Diversity: Speech-annotated video datasets are relatively scarce and often lack sufficient visual and semantic diversity, making it difficult directly train a MoCha model on them. (ii) Varying Speech Influence Across Shot Types: The impact of speech on video generation differs between spatial scales: it strongly governs lip and facial movements in close-up shots but has diminishing influence in medium or wide shots involving full-body motion. Training across all shot types simultaneously may lead to slow convergence. To address both issues, we propose a Curriculum-based Multi-modal Training strategy that integrates modality awwrr supervision with progressive visual complexity.
Mixed-Modal Sampling. Our data consists of a balanced mix of multimodal and unimodal data:
80% Multimodal (speech+text): The majority enables fine-grained audiovisual grounding.

---

Sagee
Figure 6: Progressive Currieulum for Multimodal Training in MoCha. MoCha first pretrained on text-only video data to acquire general visual gen- eration capability. The model is then pro- gressively exposed
Stage
ClossUUp Up Starl Clumaser)
80%
20%
Stage 2.
Close-Up (hot(1 Chanster)
Medium Shot Churacteri
40%
Text
20%
Complexity
Chumater)
Stage 3.
(1
40%
20%
crech-conditioned data across different task difficulties.
Madiun
Stage
Hard
1P%
40%
20%
Strong
Vidra-Spesch Aligoment
Weak
20% Unimodal (text-only): Text-only video samples offer broader visual diversity and varied camera movements, helping the model retain strong generalization capabilities. In this setting, the speech embedding is replaced with a zero vector before the audio projector.
Shot- Type-Based Curriculum. We organize training into multiple stages based on shot complexity:
Stage O: We pretrain on large-scale text-only datasets to establish strong visual priors.
Stages 1-N: We begin with close-up shots, which have high speech-visual correlation, and pro- gressively incorporate more challenging scenarios such as medium/wide shots and multi-character scenes. At each stage, we halve the share of easier examples from the previous stage while maintaining the 80%/20% multimodal-unimede ratio.
This combined strategy allows MoCha to benefit from both abundant text-only data and limited multimodal supervision while progressively mastering harder generation tasks.
4 Experiment
In this section, we first describe the details of our model in subsection 4.1 We then introduce MoCha- Bench for Talking Characters task and benchmark MoCha against baseline methods in subsection 4.2) and finally, we present an ablation study to analyze the impact of key design choices in Table 4.2
McCha(Curs)
Hallod
SadTalker
AniPortrait
3.85
3:82
8.88
3.85
2.95
Score
2:45
2:28
2.18
2.35
2:36
1.48
1.14 1.12
1,00
N/A N/A
0
Lip By-S Quality
Fac Expression Naturalnsss
Action Naturainess
Text Allgnment
Visuai Quality
Figure 7: Human evaluation scores with SoTA MoCha-Bench. Scores range from to 4 across five evaluation axes, where score 4 reflects performance that is nearly indistinguishable from video or cinematic production. Participants rated each method on five aspects: lip-syne quality, facial expression naturalness, action naturalness, text-prompt aligmment, and visual qualit. MoCha significantly outperforms all baselines across all axes. SadTalker and AniPortrait consistently received score of 1 for action naturalness, these methods only perform head movements. Text alignment marked as not applicable (N/A) for these baselines since they not accept text input. (see appendix table version)
4.1 Implementation Details
MoCha builds upon a pretrained 30B-parame aoB-paraeeter MovieGen backbone 10 which we extend for ppeech-conditioned video generation in fully tokenized, end-to-end setting. The model is configured to produce 128 frames at 24 frames per second, resulting in 5.3-second video clips. Training data is standardized to spatial resolution of approximately 720 x 720, with flexible aspect ratios to support various shot types. To enable multimodal learning, we curate two complementary datasets: (1) large-scale collection of O(100) million text-captioned videos and (2) a smaller set of O(800) thousand speech-annotated samples (28.. Training is distributed across 64 compute nodes. More details of the implementation and the data processing pipeline are provided in the Appendix.
7

---

t=1
AniP Prttrai
Hallos
SadTalker
(Ours)
/sAm/
/fi:l/
Aatk/
/tai/
/tfes/
Prompt: "A close-up shot of young enbracing her cat outdoors. She is speaking while facing with frustrating expression. background speaking, her expression remaining tense with anger while still facing s siihht the right.."
slightly right
continues
Figure 8: Qualitative comparison with SoTA on MiCCha-Bench. MoCha not only produces lip movements that align closely with the input speech-enhaneing enhancing the clarity and naturalness of articulation but also generates expressive facial animations SadTalker and AniPortrait exhibit minimal head motion and limited lip synchronization. Hallo3 mostly follows the lip-syncing but suffers from inaccurate articulation, erratic head movements, and noticeable visual artifacts. Since the baselines operate in an image-to-vider (I2V) setting, provide them with the first frame generated by MoCha input for comparison. The first frame is cropped and resized needed to meet the requirements of each baseline.
realistic, complex actions that faithfully follow the textual prompt. In contrast,
Method
Sync-C'
Syne-D
Ablation
Sync-C1
Syne-D
SadTalker
4.727
9.239
Ours
6.333
8.185
AniPortrait
1.740
11.383
Hallo3
4.866
8.963
w/o Curriculum Training
5.659
8,435
Ours
6.333 (+1.47)
8.185 (-0.78)
w/o Localized Audio Attn.
5.103
8.851
Table 2: Comparison with So on MoCha-Benchh We report lip-sync metrics, and MoCha outperforms baselines with superior lip-syne quality.
Table Ablation MoCha-Benchs Removing Local- ized Audio Attn. degrades lip-sync, Curriculum Training improves generalization.
4.2 Evaluation
Baselines. We compare our method with state-of-the-ue audio-driven talking face generation models, including SadTalker [29), AniPortrait (30), and Hallo3 (28]. These models generate talking faces conditioned on audio and auxiliary signals, such as the first frame, facial keypoints, or pose skeletons. In contrast, MoCha generates talking characters directly from raw speech and text.
Benchmark. We introduce MoCha-bench, benchmark tailored for the Talking Character generation task. It contains 200 diverse samples, each comprising text prompt and corresponding audio clip. The dataset spans various camera shot angles and camera movement- example, close- up shots emphasize facial expressions and lip-sync, while medium shots highlight hand gestures and body movement. Scenes cover a wide range of human activities and object interactions (c.g.. woman holding coffee cup, professor talking to student), with characters speaking with various emotions and facing directions. All prompts were manually curated and further enriched using the publicly released LLaMA-3 31 model to enhance expressiveness and diversity.
demonstrating its ability to generate diverse and realistic human motion while maintaining precise speech synchro- nization, even during complex actions. Contrary to the traditional view that treats audio as a weak conditioning signal requiring auxiliary supervision (e.g., pose annotations), our results show that strong audio-visual alignment can emerge purely from end-to-end training. More examples can be found in Appendix and the httpss//cooggwei12300..o hwt.://220.iithbbbooooooCooaooa
Qualitative Experiments We present qualitative results of MoCha Figure

---

Figure 8 fillustrates a side-by sid evaluation of MoCha against baseline methods on MoCha-Benche Since MoCha generates video directly from speech and text, while baselines operate in an image- to-video (12V) setup, we ensure fair comparison by providing each baseline with the first frame generated by MoCha (cropped/resized to focus on the head region as needed ). Additional results are provided in the Appendix. MoCha not only produces lip movements that closely align with the speech-enhancing both articulation and naturalness- also generates expressive facial emotion that accurately follow the textual prompt. In contrast, SadTalker and AniPortrait exhibit minimal head motion and limited lip synchronization. While Hallo3 achieves mostly consistent lip-syncing, suffers from inaccurate articulation and erratic head movements.
Quantitative Experiments We evaluate video quality using the automatic metries measure the lip-sync quality. Table 2 presents a comparison on the MoCha-Bende Our model achieves the best scores across all lip-syne metrics, demonstrating the effectiveness of MoCha 's end-to-end DiT design. These results further confirm that strong audio-visual alignment can emerge purely from audio conditioning without any auxiliary signal.
Human Evaluations. We conduct a comprehensive human evaluation to compare MoCha against baseline methods on the MoCha-Bench dataset. The evaluation based on five axes tailored for the Talking Characters task (see Section 2). with scores ranging from I to 4 (see Appendix). Each model output received 5 independent ratings per example, resulting in over 1000 responses per model. As illustrated in Figure 7, MoCha significantly outperforms all baselines across all five axes, with average scores approaching 4 indicating performance that nearly indistinguishable from real video or cinematic production.
Ablation Studies We conduct ablation experiments to assess the individual contributions of MoCha 's core components. Table 3 presents the impact of each component, (i)We disable our localized audio attention mechanism during training, which results in noticeable drop in Sync-C and increased Sync-D. We also have design variant comparison with RoPE + global audio attention in the Appendix. We train MoCha exclusively on speech-annotated data. This results in noticeable drop in lip-sync quality, indicating degraded generalization due to the reduced diversity of the dataset.
5 Related Work
5.1 Talking Head Generation
Given an audio sequence and a reference face, pioneer talking-head generation works typically utilize biometric signals such as facial keypoints (52 or 3D priors [36 29 41
intermediate motion representation to animate the reference face while ensuring lip synchronization. For example, SadTalker [29] first extracts 3DMM coefficients from audio and then renders the face in 3D-aware manner. AniPortrait [30] predicts 2D facial landmarks from audio and then uses diffusion models to generate a portrait video from the 2D landmark maps. VLOGGER (42) predicts both 3D expression coefficient and 3D body pose from speech and enables the simultaneous generation of talking- face animations and upper-body gestures. Although effective, videos generated by these methods often lack expressiveness and naturalness due to the limited representation of 2D/3D priors. Recently works, such as EMO [8 and Hallo [77, generate audio-driven portrait videos end-to-end using diffusion models, which eliminate intermediate facial representations and learn natural motion from data [8 17
16 43 Hallo3 (28) builds upon pretrained transformer-ban based video diffusion models to animate faces with dynamic head poses and background elements. Although these methods can generate natural expressions, they rely on complex auxiliary signals-such as reference images or scypoints-wiae not only limit the naturalness and flexibility of facial expressions and body movements but also limit the generalization ablity of those methods.
22
5.2 Video Diffusion Model
Recent diffusion based video models have focused on improving visual quality and temporal co- herence, particularly in text-guided synthesis. Early works such as Make-A-Vidae Videe (44) Tune-A-
Video 45, Video LDM [46.. MagicVideo (7). and AnimateDiff [48) adapt text-to-image (T2I) backbones to model motion dynamics. More recent methods based on diffusion transformers-suu
as Hunyuan Video 110), VideoX [7]. MovieGen allong with open-source frameworks like

---

VideoCrafter [6], ModelScopeT22 [49]. and Pyramidal Flow [50], further enhance apatio-tempory consistency and generation fidelity.
However, these models lack mechanisms for aligning speech with character behavior, often producing disjointed lip motions or gestures that fail to reflect spoken content. In contrast, MoCha pioneers new direction by jointly conditioning on both speech and text to drive character crinattionbbidgiing the gap between realistic motion synthesis and dialogue-driven storytelling.
6 Conclusion
In summary, our work pioneers the task of Talking Characters Generation, pushing beyond traditional talking head synthesis to enable full-body, multi-character animations directly driven by speech and text. We present MoCha, the first framework to address this challenging task, introducing key innovations such as the localized audio attention mechanism for precise audio-visual alignment and curriculum- based multi-modal training strategy that leverages both speech- and text-labelled data for enhanced generalization. Additionally, our structured prompt design unlocks multi-character, turn-based dialogues with contextual awareness. Comprehensive experiments and human evaluations demonstrate that MoCha delivers state-of-the-ar performance in dialogue-driven video generation, marking a significant step toward scalable, cinematic AI storytelling.
7
Acknowledgment
We thank Xinyi Ji, Tianquan Di, Anqi Xu, Matthew Yu, Emily Luo for providing speech samples used the MoCha demo.
References
[1] Yupeng Zhou, Daquan Zhou, Ming-Ming Cheng, Jiashi Feng, and Qibin Hou. Storydiffusion: Consistent self-attention for long-range image and video generation. Advances in Neural Information Processing Systems, 37:110315 110340, 2024.
[2] Sergey Tulyakov, Ming-Yu Yu Liu, Xiaodong Yang, and Jan Kautz. Mocogan: motion and content for video generation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1526-1535, 2018
Decomposing
[3] Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David Fleet. Video diffusion models. Advances in Neural Information Processing Systems, 35:8633-8646, 2022.
[4] Wilson Yan, Yunzhi Zhang, Pieter Abbeel, and Aravind Srinivas. Videogpt: Video generation using vq-vae and transformers. arXiv preprint arXiv:2104.10157, 2021.
[5] Shoufa Chen, Chongjian Ge, Yuqi Zhang, Yida Zhang, Fengda Zhu, Hao Yang, Hongxiang Hao, Hui Wu, Zhichao Lai, Yifei Hu, Ting-Che Lin, Shilong Zhang. Fu Li, Chuan Li, Xing Wang, Yanghua Peng. Peize Sun, Ping Luo, Yi Jiang, Zehuan Yuan, Bingyue Peng. and Xiaobing Liu. Flow based video generative foundation models. arXiv preprint arXiv:2502.0488 2025.
Goku:
[6] Haoxin Chen, Menghan Xia, Yingqing He, Yong Zhang, Xiaodong Cun, Shaoshu Yang, Jinbo Xing, Yaofang Liu, Qifeng Chen, Xintao Wang, et al. Videocrafterl: Open diffusion models for high-quality video generation. arXiv preprint 10. 2023.
[7] Zhuoyi Yang. Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang. Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiaohan Zhang, Guanyu Feng, et al. Cogvideox: Text-to-video diffusion models with expert transformer. arXiv preprint arXiv:2408.06072 2024.
[8] Adam Polyak, Amit Zohar, Andrew Brown, Andros Tjandra, Animesh Sinha, Ann Lee, Apoorv Vyas, Bowen Shi, Chih-Yar Yao Ma, Ching- Yao Chuang, et al. Movie gen: A cast of media foundation models. arXiv preprint 241.v:2410.13720, 2024.
10

---

[9] Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Do- minik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint (rxiv:2331.15122 2023.
[10] Weijie Kong, Qi Tian, Zijian Zhang, Rox Min, Zuozhuo Dai, Jin Zhou, Jiangfeng Xiong, Xin Li, Bo Wu, Jianwei Zhang, et al. Hunyuanvideo: A systematic framework for large video generative models. arXiv preprint arXiv:2412.036// 2024.
[11] Hila Chefer, Uriel Singer, Amit Zohar, Yuval Kirstain, Adam Polyak, Yaniv Taigman, Lior Wolf, and Shelly Sheynin. Videojam: Joint appearance-motion representations for enhanced motion generation in video models. arXiv preprint arXiv:2502.02499. 2025.
[12] Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, et al. Video generation models world simulators. OpenAl Blog, 1:8, 2024.
[13] Rang Meng, Xingyu Zhang, Yuming Li, and Chenguang Ma. Echomimiev2: Towards striking, simplified, and semi-body human animation. arXiv preprint arXiv:2411.1000 2024.
[14] Linrui Tian, Qi Wang, Bang Zhang, and Liefeng Bo. Emo: Emote portrait alive generating expressive portrait videos with audio2video diffusion model under weak conditions. In European Conference Computer Vision, pages 244-260, 2024.
[15] Linrui Tian, Siqi Hu, Qi Wang, Bang Zhang, and Liefeng Bo. Emo2: End-effector guided audio-driven avatar video generation. arXiv preprint wxXiv2251.10687 2025.
[16] Jianwen Jiang, Chao Liang, Jiaqi Yang, Gaojie Lin, Tianyun Zhong, and Yanbo Zheng. Loopy: Taming audio-driven portrait avatar with long-term motion dependency. In The Thirteenth International Conference on Learning Representations, 2025.
[17] Mingwang Xu, Hui Li, Qingkun Su, Hanlin Shang, Liwei Zhang, Ce Liu, Jingdong Wang, Yao Yao, and Siyu Zhu, Hallo: Hierarchical audio-driven visual synthesis for portrait image animation. arXiv preprint arXiv:2406.. 240.00801, 2024.
[18] Linrui Tian, Qi Wang, Bang Zhang, and Liefeng Bo. Emo: Emote portrait alive generating expressive portrait videos with audio2video diffusion model under weak conditions. In European Conference on Computer Vision, pages 244-260. Springer, 2024.
[19] Hongwei Yi, Tian Ye, Shitong Shao, Xuancheng Yang, Jiantong Zhao, Hanzhong Guo, Terrance Wang, Qingyu Yin, Zeke Xie, Lei Zhu, et al. Magicinfinite: Generating infinite talking videos with your words and voice. arXiv preprint arXiv:2503.05978. 2025.
[20] Xiaozhong Ji, Xiaobin Hu, Zhihong Xu, Junwei Zhu, Chuming Lin, Qingdong He, Jiangning Zhang, Donghao Luo, Yi Chen, Qin Lin, et al. Sonie: Shifting focus to global audio perception in portrait animation. arXiv preprint arXiv:2411.1633
2024.
[21] Gaojie Lin, Jianwen Jiang, Jiaqi Yang, Zerong Zheng, and Chao Liang, Omnihuman-1 1: Rethinking the scaling-up up of one-stage conditioned human animation models. arXiv preprint arXiv:2502.0106/ 2025.
[22] Jiahao Cui, Hui Li, Yao Yao, Hao Zhu, Hanlin Shang, Kaihui Cheng, Hang Zhou, Siyu Zhu, and Jingdong Wang. Hallo2: Long-duration and bigh-resolution audio-driven portrait image animation. ICLR, 2025.
[23] William Peebles and Saining Xie. Scalable diffusion models with transformers. ICCV, 2023.
[24] Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vec 2.0: framework for self-supervised learning of speech representations. Advances neural information processing systems,
12460, 2020.
[25] Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling, arXiv preprint arXiv:2210.02747, 2022.
11

---

[26] Ziyi Wu, Aliaksandr Siarohin, Willi Menapace, Ivan Skorokhodov, Yuwei Fang, Vammihh Chordia, Igor Gilitschenski, and Sergey Tulyakov. Mind the time: Temporally-ccontrolles multi-event video generation. In CVPR, 2025.
[27] Feng Liang, Haoyu Ma, Zecheng He, Tingbo Hou, Ji Hou, Kunpeng Li, Xiaoliang Dai, Felix Juefei-Xu, Samanch Azadi, Animesh Sinha, Peizhao Zhang, Peter Vajda, and Diana Marculescu. Movie weaver: Tuning-free multi-concept video personalization with anchored prompts. CVPR, 2025.
[28] Jiahao Cui, Hui Li, Yun Zhan, Hanlin Shang, Kaihui Cheng, Yuqi Ma, Shan Mu, Hang Zhou, Jingdong Wang, and Siyu Zhu, Hallo3: Highly dynamic and realistic portrait image animation with diffusion transformer networks. arXiv preprint arXiv:24/2.0000 000333,, 2024.
[29] Wenxuan Zhang, Xiaodong Cun, Xuan Wang, Yong Zhang, Xi Shen, Yu Guo. Ying Shan, and Fei Wang. Sadtalker: Learning realistic 3d motion coefficients for stylized audio-driven single image talking face animation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8652-8661, 2023.
[30] Huawei Wei, Zejun Yang, and Zhisheng Wang. Aniportrait: Audio-driven synthesis of photore- alistic portrait animation. arXiv preprint arXiv:2403. 2024.
[31] Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 2024.
[32] KR Prajwal, Rudrabha Mukhopadhyay, Vinay P Namboodiri, and CV Jawahar. A lip sync expert is all you need for speech to lip generation in the wild. In Proceedings of the 28th ACM international conference on multimedia, pages 484-492, 2020.
[33] Shuai Shen, Wenliang Zhao, Zibin Meng, Wanhua Li, Zheng Zhu, Jie Zhou, and Jiwen Lu. Difftalk: Crafting diffusion models for generalized audio-driven portraits animation. In Pro- ceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1982-1991, 2023.
[34] Yunfei Liu, Lijian Lin, Fei Yu, Changyin Zhou, and Yu Li. Moda: Mapping-onee audio- driven portrait animation with dual attentions. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 23020-23029, 2023.
[35] Weizhi Zhong, Chaowei Fang, Yinqi Cai, Pengxu Wei, Gangming Zhao, Liang Lin, and Guanbin Li.
talking face generation with landmark and appearance priors. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9729-9738, 2023.
[36] Michail Christos Doukas, Stefanos Zafeiriou, and Viktoriia Sharmanska. Headgan: One-shot neural head synthesis and editing. In Proceedings of the IEEE/CVF International conference on Computer Vision, pages 13398-14409 2021.
[37] Yifeng Ma, Shiwei Zhang, Jiayu Wang, Xiang Wang, Yingya Zhang, and Zhidong Deng. Dreamtalk: When expressive talking head generation meets diffusion probabilistic models. arXiv preprint arXiv:2312.09767, 2023.
[38] Xinya Ji, Hang Zhou, Kaisiyuan Wang, Wayne Wu, Chen Change Loy, Xun Cao, and Feng Xu. audio-driven emotional video portraits. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 14080-14089 2021.
[39] Xusen Sun, Longhao Zhang, Hao Zhu, Peng Zhang, Bang Zhang, Xinya Ji, Kangneng Zhou, Daiheng Gao, Liefeng Bo, and Xun Cao. Vividtalk: One-shot audio-driven talking head generation based on 3d hybrid prior. arXiv preprint arXiv:2312.00 2111111....11111111 2023.
[40] Yuan Gan, Zongxin Yang, Xihang Yue, Lingyun Sun, and Yi Yang. Efficient emotional adapta- tion for audio-driven talking-head generation. In Proceedings of the IEEF/CVE International Conference Computer Vision, pages 22634-22645, 2023.
12

---

[41] Zhenhui Ye, Tianyun Zhong, Yi Ren, Jiaqi Yang, Weichuang Li, Jiawei Huang, Ziyue Jiang, Jinzheng He, Rongjie Huang, Jinglin Liu, al. poartatportratt: One-shot realistic 3d talking portrait synthesis. ICLR, 2024.
[42] Enric Corona, Andrei Zanfir, Eduard Gabriel Bazavan, Nikos Kolotouros, Thiemo Alldieck, and Cristian Sminchisescu. Vlogger: Multimodal diffusion for embodied avatar synthesis. arXiv preprint wtXXvv:2403008766 2024.
[43] Cong Wang, Kuan Tian, Jun Zhang, Yonghang Guan, Feng Luo, Fei Shen, Zhiwei Jiang, Qing Gu, Xiao Han, and Wei Yang. V-express: Conditional dropout for progressive training of portrait video generation. arXiv preprint urXiv:2406.0251/ 2024.
[44] Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang. Oron Ashual, Oran Gafni, et al. Make-a- v-ieoo: Text-to-video generation without tixtovideo data. arXiv preprint arXiv:2209. 14792, 2022.
[45] Jay Zhangjie Wu, Yixiao Ge, Xintao Wang. Stan Weixian Lei, Yuchao Gu, Wynne Hsu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation. arxiv 2022. arXiv preprint arXiv:2212. 2022.
[46] Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Scung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 22563-22575, 2023.
[47] Daquan Zhou, Weimin Wang, Hanshu Yan, Weiwei Lv, Yizhe Zhu, and Jiashi Feng. Magicvideo: Efficient video generation with latent diffusion models. arXiv preprint arXiv:2211.11012 2022.
[48] Yuwei Guo, Ceyuan Yang, Anyi Rao, Zhengyang Liang, Yaohui Wang, Yu Qiao, Maneesh Agrawala, Dahua Lin, and Bo Dai. Animatediff: Animate your personalized text-to-image diffusion models without specific tuning. arXiv preprint arXiv:2307.04425, 2023. 2023.
[49] Jiuniu Wang, Hangjie Yuan, Dayou Chen, Yingya Zhang. Xiang Wang, and Shiwei Zhang. Modelscope text-to-video technical report. arXiv preprint 2rXiv:2308.06571 2023.
[50] Yang Jin, Zhicheng Sun, Ningyuan Li, Kun Xu, Hao Jiang, Nan Zhuang, Quzhe Huang, Yang Song, Yadong Mu, and Zhouchen Lin. Pyramidal flow matching for efficient video generative modeling. arXiv preprint arXiv:2410.05954 2024.
[51] Brandon Castellano. PySceneDetect. PySceneDetect
URL
[52] Steffen Schneider, Alexei Baevski, Ronan Collobert, and Michael Auli. wav2vee: Unsupervised pre-training for speech recognition. arXiv preprint arXiv: 1904.05862, 2019.
[53] AI Meta. Introducing meta llama 3: The most capable openly available Ilm date. Meta A1, 2
(5)66, 2024.
13

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer:
[Yes]
Justification: We provide extensive qualitative, quantitative, and human evaluation to support our claims.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We provide Limitations and Broader Impact in the Appendix.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
14

---

Justification:
Our paper focuses on empirical results supported by extensive experiments.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We have detailed discussion about the new model architecture.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
15

---

Answer: [No]
Justification:
We publicly release the evaluation benchmark, but the code is not yet available.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https guwdes/ddeeeeee Cooeeeoodeesssseeeeeee //nips.com htte://ulllllliiiee/e//eeeuaaass ssiniiiicyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification:
We provide a detailed discussion of the experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: ratings.
We report error bars in quantitative evaluations based repeated human
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
16

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide a detailed discussion of the compute resources.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
Our research conform in every respect, with the NeurlPS Code of Ethics
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We have border impacts paragraph.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
17

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification:
We have a paragraph about this.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We cited the original paper.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
18

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We provide a document for the public available benchmark.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [Yes]
Justification:
We have a paragraph for the instructions.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (TRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [No]
Justification: Our study includes human evaluations of model outputs. Participants were volunteers from our institution, informed of the task, and no sensitive personal data was collected. Given the low-risk nature the study, IRB approval was not required under our institution's policy.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
19

---

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [Yes]
Justification: We use LLM enrich video descriptions and prompts for our evaluation benchmark and training data. We describe our data processing pipeline in appendix.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non non-standard components.
Please refer to LLM policy http:///eeerrpppcc//oeeneeeeeeee cc/Coonfeeeencce/2225//.. for what should or should not be described.
20

---

Technical Appendices and Supplementary Material
In this appendix, provide details of the model architecture in Appendix BJ training setting in Appendix C data processing pipeline in Appendixx D and we provide comprehensive ablation studies and experiment results in Appendix A] We discuss limitations and broader impact in Ampenddix and Appendix F
A
Experiments
A.1 Audio Cross-Attention Design Ablation
To better understand how to inject audio conditioning into DiT-based video diffusion models, we conduct systematic design study comparing multiple audio cross-attention mechanisms. Due to the high computational cost of training the full 30B MoCha model, we perform our ablation using a smaller 4B DiT backbone pretrained on MovieGen [8], and fine-tune it on 400K close-up. single-character clips with audio annotations for 200K training steps.
All models share identical training hyperparameters and differ only in the design of the audio cross- attention module. We evaluate lip-syne quality on MoCha-Bess -encch using the SyncNet-based metrics Sync-C (1) and Sync-D (4). Results are summarized in Table
Variants. Below, define the attention query R the video tokens, and key/value k a E 7xd as the audio tokens. We denote A(q, k v) as the standard attention function.
. Naive Audio Cross-Attention: Each video token attends to all audio tokens without positional encoding:
2 A k, v).
Learnable Positional Embedding: Audio tokens are augmented with learnable positional embeddings Py initialized to zero:
A(q, p,v).
Sinusoidal Positional Embedding: Instead of learned Py, we add fixed sinusoidal embed-
dings:
A(q, k Sinusoidal(j), v).
Rotary Positional Embedding (RoPE): RoPE is applied by rotating the queries and keys:
z = A(RoPE(q), RoPE(K)) v).
Localized Audio Attention (proposed): Each video token attends only to a window of audio tokens k(jo51)) centered on its corresponding temporal segment (see Section 3.2
2; A(gi (joiil P)och)))
where jo max(1, (i-1)r-1) and ji iin(T,ir+1).
Localized Audio Attention + RoPE: We additionally apply RoPE to q and within the window.
Findings. As shown in Table 4 the naive attention baselines perform poorly in lip-syne accu- racy. Adding positional information sinusoidal or RoPE embeddings Soadddiigs--iiniiiaanni. improves performance. suggesting that positional priors are critical for learning speech-video align- ment. However, our proposed Localized Audio Attention consistently outperforms all other variants, demonstrating the effectiveness of constrained temporal windows for resolving the resolution mis- match between video and audio tokens.
Interestingly, adding RoPE to the Localized Attention variant slightly degrades performance, indicat- ing potential interference between the inductive bias introduced by RoPE and the explicit temporal alignment imposed by windowing
21

---

Method
Syne-C f
Sync-D
DiT-4B + Localized Audio Attention (MoCha-4B)
5.692
8.403
DiT-4B + Localized Audio Attention + RoPE
5,027
9.038
DiT-4B + Naive Audio Attention + RoPE
4.872
8.893
DmT-4B + Naive Audio Attention Sinusoidal Embedding
4.747
8.986
DmT-4B + Naive Audio Attention + Learnable Embedding
2.540
10.363
DiT-4B + Naive Audio Attention
2.364
10.385
Table Comparison of Audio Cross-Attention Variants on MoCha-Benee We report lip-syne metrics. The proposed Localized Audio Attention achieves the best performance.
A.2 Ablation of Curriculum-Based Multimodal Training Strategy
We conduct ablation study to assess the effectiveness of our proposed Curriculum-based Multi- modal Training strategy (described in Section 3.4). which is designed to address two core challenges: (i) data scarcity and limited diversity in speech-annotate datasets, and (ii) varying speech relevance across spatial scales in different shot types.
To evaluate this strategy, compare the full MoCha-30B model with two ablated variants:
w/o Curriculum Training: Trained on mixed modalities (speech and text-only) but without curriculum progression (i.e., trained directly mixed data with full shot-type complexity from the beginning).
w/o Mixed Multimodal Training: Trained solely on seeech-annotated videos, without any text-only data or curriculum scheduling.
All models use the same architecture and are trained for an equal number of steps.
We report both automatic lip-syne metrics (Sync-C 1. Sync-D J) and three human evaluation metrics from MoCha- Bench: Text Aligument, Visual Quality, and Action Naturalness (detailed in Section A.4). Qualitative examples can be found on the https://congweii 1 220. github.io/MoCha/
As shown in Table 5 removing the curriculum phase (w/o Curriculum Training) causes a moderate performance drop across all metrics, confirming the benefit of staged training that gradually increases visual complexity. Although trained for the same number of steps, the non-curriculum baseline converges more slowly and underperforms.
The w/o Mixed Multimodal Training variant-traiin trained only on limited speech-annotated data data-perfore peih-perfomms significantly worse, especially on Text Alignment and Action Naturalness. This confirms that unimodal speech-driven training causes overfitting to front-facing talking- facceng-facee data, impairing the model's ability to generalize to diverse prompts for diverse scenes and full-body activities.
These results validate the necessity of both modality mixing and curriculum progression for robust and generalizable talking character generation.
Method
Sync-C f
Syne-D
Text Alignment
Visual Quality
Action Naturalness
MoCha-30B (with Curriculum)
6.333
8.185
3.85
3.72
3.82
w/o Curriculum Training
5.659
8.435
3.17
w/o Mixed Multimodal Training
5,798
8.231
2.71
2.91
2.97
Table 5: Ablation of Currieulum-Bant Multimodal Training Strategy MoCha-Bench We report lip-syne metrics (Sync-C 1. Sync-D 4) and human evaluation scores across three axes.
A.3 Ablation of Character Tagging Strategy
We ablate the effectiveness of the Character Tagging strategy introduced in Section for handling multi-character conversations with turn-based dialogue. This strategy is particularly important in multi-clip scenes, where multiple characters appear across different segments and speaker transitions inferred from the audio input alone.
22

---

Our tagging mechanism assigns unique identifier (e.g.. Personl, Person2) to each character intro- duced at the beginning of the prompt. These tags are then reused across individual clip descriptions. allowing for clear, consistent references without repeating verbose appearance descriptions. This structured prompting significantly reduces prompt length and improves character consistency (see Figure5).
To evaluate the impact of character tagging. we compare the full MoCha-30B model (with tagging) to a baseline trained with naïve captioning-where detailed character descriptions are repeated verbosely in each clip. Both models are evaluated on the turn-based dialogue subset of MoCha-Benchh
As shown in T removing character tagging results in a drastic drop in Text Alignment. indicating the model often confuses which character appears in which scene. Qualitative examples show that, without tagging, the model may generate scene-swap artifacts, fail to transition characters correctly between clips, or maintain similar lip-sync for mismatched dialogue.
We also observe degradation in Visual Quality, as the model occasionally blends inconsistent character features across clips. Overall, these results highlight that character tagging is essential for multi- character consistency and semantic alignment dialogue-driven video generation.
Method
Sync-C
Syne-D
Text Aligament
Visual Quality
Action Naturalness
MoCha-30B (with Character Taggingg
4,951
8.601
3.81
3.64
3.69
w/o Character Tagging
4,465
8.792
2.01
2.15
3.03
Table 6: Ablation of Character Tagging on MoCha- Bench (Turn-Basse Dialogue Category). We report lip-sync metrics (Sync-C t. Sync-D J) and human evaluation scores. Character tagging improves semantic consistency and reduces scene confusion in multi-clip dialogue scenarios.
A.4 MoCha-Bench Human Evaluation
We conduct a comprehensive human evaluation to compare MoCha against baseline methods on the MoCha- Bench dataset. The evaluation is based on five axes tailored for the Talking Characters task with scores ranging from to 4. Each model output received independent ratings per example, resulting in over 1000 responses per model. We provide the evaluation guidance below. Besides the text guideline, we also include some visual examples to better help the annotators to judge.
This document provides evaluation guidelines, including axis definitions, scoring rubrics, and instruc- tions for annotators. Visual examples are provided separately to support consistent judgments.
Task Overview
Each evaluation sample consists of:
A generated video with audio,
A text prompt describing the scene and character behavior.
Your task is to evaluate how well the generated video across five dimensions.
Evaluation Axes
Lip-Syne Quality: Measures how accurately the character's lip movements align with the spoken audio.
Scale: Not aligned all, 2 Weak alignment, 3 Mostly aligned, Perfectly aligned.
Facial Expression Naturalness: Evaluates whether facial expressions and lip-sync appear natural and contextually appropriate, without seeming robotic or exaggerated.
Scale: 1 Completely unnatural, 2 Noticeably synthetic or stiff, 3 Mostly natural and believable, Indistinguishable from real or cinematic performance.
Action Naturalness: Assesses how naturally the character's body movements and gestures
align with the audio.
Scale: - Completely unnatural, 2 Noticeably unnatural, 3 Mostly natural, 4 Indistin- guishable from real movie or TV characters.
23

---

Text Alignment:
Measures how well the generated actions, expressions, and presence of characters follow the behaviours described in the prompt.
Scale: 1 - No alignment (e.g.. missing character or major mishehavior),
2 Partial alignment, 3 Mostly aligned, 4 Perfect alignment with the prompt.
Visual Quality: Evaluates the overall visual fidelity, including image sharpness, coherence, and absence of rendering issues such artifacts, glitches, or anatomical distortions (e.g.. broken limbs or unnatural body proportions).
Scale: 1 Severe artifacts, 2 Noticeable artifacts, 3 Mostly artifact-free, 4 Flawless visuals.
Annotation Instructions
1. Review the prompt and watch the full video at least once.
2. Evaluate each axis independently. Use the provided 1-4 scale. Do not assign the same score across all axes unless truly justified.
3. Use the full scale range. Assign low high scores needed. Avoid defaulting to the midpoint.
4. If no character present the video, assign a score of for Text Alignment.
FAQ
Q: What if the same scene includes multiple characters?
A: Focus your evaluation on the central or speaking character as described in the prompt. Other characters may be present, but your ratings should reflect the behavior and alignment of the primary subject.
Q: What if the video does not include a character?
A: If the generated video fails to include the main character described the prompt (e.g,, g, an empty scene or wrong subject), you should assign score of 1 for Text Alignment.
Lip-Sync Quality
Facial Expression Naturalness
Action Naturalness
Text Alignment
Visual Quality
Method
Hallo3 17
2.45
2.25
2.13
2.35 N/A
2.36
SadTalker 20
1.21
1.14
1.00
2.95
AniPortrait 30
1.16
1.12
1.00
N/A
1.45
MoCha (Ours)
3,85 (+1.40)
3.82
(+1.57)
3.82 (+1.69)
3,85
(+1.50)
3.72 (+1.36)
Table 7:
Human evaluation scores on MoCha-Bench. Scores range from to 4 across five evaluation axes, where a score of reflects performance that is nearly indistinguishable from real video or cinematic production. Participants rated each method on five aspects: lip-syne quality, facial expression naturalness, action naturalness, text-prompt alignment, and visual quality. MoCha significantly outperforms prior methods across all categories. Green numbers indicate absolute improvements (A) over the second-best method (underlined). SadTalker and AniPortrait consistently received score of for action naturalness, these methods only perform head movements.
A.5 MoCha-Bench Qualitative Comparison
Figure 8 presents direct comparison between MoCha and baseline methods on MoCha. All baselines require reference image as an auxiliary input. To ensure fairness, we first generate a video using MoCha and then use its first frame as the reference image for all baseline models. For models that do not support arbitrary aspect ratios, we crop the first frame to focus on the head region before feeding it into their networks. We provide two groups of qualitative comparisons: one featuring close-up shots and the other medium shots. The close-up up group emphasizes lip-sync quality, head movement, and facial expressions, while the medium shot group focuses on hand movements during speech. MoCha not only produces lip movements that closely align with the input speech- enhhancin both articulation and naturalness-ae but also generates expressive facial animations and realistic, coordinated actions that accurately follow the textual prompt. In contrast, SadTalker and AniPortrait exhibit minimal

---

head motion and limited lip synchronization. While Hallo3 achieves mostly consistent lip-syncing, it suffers from inaccurate articulation and erratic head movements. In the medium shot comparisons. Hallo3 also introduces noticeable visual artifacts, particularly during complex actions.
B MoCha Model Architecture
B.1
3D VAE
Our 3D-VAE is based on variational autoencoder and compresses the input pixel space video of shape E x2 xW into a lower-dimensional, continuous latent representation xp of shape compress the input 8x across the spatial dimension
h x c, In our implementation, while 4x across the temporal dimension, i.c., H/h = W/w = 8 and T/T = The latent channel dimensionality is fixed at 16 for all experiments reported herein.
B.2 Text Encoder
We employ triad of text encoding architectures UL2, ByT5, and MetaCLIP- equip the backbone with both high-level semantic and fine-grained character-level textual comprehension. Each encoder generates a sequence of 256 token embeddings. To unify these heterogeneous representations, we apply dedicated linear projection layers and LayerNorm to each encoder's output, transforming them into the model dimension 3144-dimensional feature space. The resulting normalized embeddings from all three streams are then concatenated to produce the final comprehensive text representation fed into the backbone. Among these, the MetaCLIP encoder specializes in generating text features inherently aligned with visual modalities, optimizing performance for cross-modal generation tasks. UL2, conversely, excels at encoding deep linguistic reasoning and semantics, While ByT5 captures character-level details, making it effective for encoding visual text.
variant of
B.3 Audio Encoder
Our audio pipeline is powered by Wev2Vec2, but instead of relying solely on its final output, we extract and stitch together the embeddings from all 12 internal layers. This approach gives us deeper, layered view of the audio content, with each layer contributing 768-dimensional slice to the overall representation. After running the audio through Wav2Ve2's tokenizer, we stretch or compress the resulting sequence using linear interpolation before Before the audio hits Wav2Vec2. So that end up with the same number audio features as there are video sauncs-effectively assigning a unique audio token each frame. To provide each frame's 's audio token with extra context, expand its feature vector by gluing on the tokens from the five frames before and after it. So for any given frame f. the final embedding is built A() [A(] - 5),.... A(f)...,, A(f + 5)]. This chunky, context-aware audio feature then passes through a straightforward two-layer neural net (an MLP with a hidden size of 512) which reshapes it into the 6 144-dimenioonal token (xf) needed by our model's backbone.
B.4 DiT Architecture
The core architectural hyperparameters of our MoCha-30B DiT backbone are provided in Table 8
Localized Audio Cross-Attention. To incorporate temporal locality in the audio stream, we adopt a windowed cross-attention mechanism with a window size of r r 6, where r = 4 is the temporal downsampling factor of the video encoder. For latent video frame 2i(() hxxxxe at timestep i E {l,..., J. attention is restricted to audio tokens (i) falling within the interval:
j E [max (1, (i - 1)r - 1), min (T, ir + 1)],
(5)
which corresponds to the original (pre-downsampled) temporal span of (6) with one token of padding on each side to enable smoother context transitions at boundaries.
Modality Integration Pathway. Each attention block integrates modalities in sequential manner. It begins with video self-attention, followed by localized audio cross-attention and then text cross-
25

---

Layers
Model Dimension
FFN Dimension
Attention Heads
Activation Function
Normalization
6144
16384
SwiGLU
RMSNorm
Table 8: Core architecture hyperparameters for the MoCha-30B Transformer. 'The model has 30 billion parameters in the Transformer stack alone, excluding auxiliary components such as text embedding models, speech embedding models, and the 3D-VAE.
attention. Each stage incorporates a residual connection, modulated by a scalar weight. our implementation, these residual weights are fixed 1. The full process is given by:
(1)
Z SelfAttn Zin)
(2)
out text
CrossAttthext Videoo y) + Atext out video
CrossAtth (u Qx) + Aauioo Fxext
audio
where:
is the input latent sequence.
SelfAttnvido denotes self-attention over video tokens.
CrossAAtumnoo a) performs localized cross-attention with audio tokens a. CrossAttmea y) performs cross-attention with global text tokens y- audio and Avx residual weights, both set to in our implementation.
This staged fusion enables progressive enrichment of the video representation by incorporating auditory and textual information, while preserving intermediate features through residual addition.
C MoCha Training
We provide the training details for our MoCha-30B model in Table 9 rate scheduler with 2000 warm-up steps. We use a Progressive Curriculum for Multimodal Training describe in subsection 3.4
We used constant learning
Mixed-Modal Sampling. Our data consists of a balanced mix of multimodal and unimodal data:
80% Multimodal (speech+text): The majority enables fine-grained audiovisual grounding.
20% Unimodal (text-only): Text-only video samples offer broader visual diversity and varied camera movements, helping the model retain strong generalization capabilities. In this setting, the speech embedding is replaced with zero vector before the audio projector.
Shot-Typer Tape--ey Curriculum. We organize training into multiple stages based on shot complexity:
Stage We pretrain on large-scale text-only datasets to establish strong visual priors.
Stages I-N: We begin with close-up shots, which have high speech-visual correlation, and pro- gressively incorporate more challenging scenarios such as medium/wide shots and multi-character scenes. At each stage, we halve the share of casier examples from the previous stage while maintaining the 80%/20% multimodaluuimodall ratio.
This combined strategy allows MoCha to benefit from both abundant text-only data and limited multimodal supervision while progressively mastering harder generation tasks.
We build MoCha based on Movie Gen Backbone. The Stage 0 training included in the Movie Gen backbone pertaining. Then we add the speech cross attention and speech projector to the Movie Gen backbone to build MoCha. During Stages 1-N training. We full-finetuning the entire 30B MoCha model while freezing the text encoder and speech encoder and text projector. Throughout training, all
26

---

input examples are resized to resolution of approximately 720 px, preserving their original aspect ratios.
Stage
#GPUs
Global
LR
Shot Types
#iters
#Audio Samples
#Text Samples
1024
Images Videos
100M
512
Close-Up Up Chur)
100M
Close-Up Close (1 Char)
512
1o-5
Close-Up Medium/ Medium Close (1 Char)
200K
100M
512
512
le.5
Close Medium Multi Char Multi-Cipp
100K
1OUK
100M
Table Progressive Curriculum for Multimodal Training in MoCha. Stages gradually increase in temporal and compositional complexity, progressing from static images and short clips to multi-shot, multi-character, dialogue-driven video. Residual text supervision is maintained throughout.
D Data Processing Pipeline
Step 1. Speech Scene Filter
Step 2. Prominent Characters Filter by LLM.
① Sepurator
Llama3
Pronpt: Any Prominent Characters?
Reason:
||||adi11000
men bar. scene occupy significant portion of the frane.
The video shows But of then appear be the nain
Music Remover
2 Noise Music Remover
Decision: False
Speech Detector
Spooch Detector
Wav2Nee2
Motion Filter
Sync Filter
Step 3. Captioning
X Llama3
The maan sits to the RRIGHT SIDE of the frame.
speaking. looking at his phone, occasionally raising his head
P Pronpt:
Begin with general description of the
2. Description of Character's APPEARANCE and POSITION fron Left to right.
The wonan appears ANGERY and WORRIED.
3. For each character
GESTURING
Description whether they are SPEAKING or NOT. Description of their EMOTIONS B0DY Language. Description of their ACTIONS.
front of her as she speaks, and she is LOOKING directly toward the nan
4. Description of Background
5. Conclusion CAMERA Movenents
Figure 9: Data Processing Pipeline. Our four-stage pipeline includes: (1) Speech Scene Filtering, (2) Prominent Character Filtering, (3) Motion and Lip-Sync Verification, and (4) Scene Cap- tioning. Together, these steps produce high-quality, speech-aligned training samples.
To ensure high-quality supervision for speech-conditioned video generation, we develop a four-stage data processing pipeline, illustrated in Figure 9 Each stage is designed to filter noisy data and produce rich, structured annotations for training.
(1) Speech Scene Filtering: Raw videos are first segmented into scenes using PySceneDe- tect [51]. We then detect and retain segments containing clean, spoken audio by removing clips with excessive background noise or dominant music. For valid segments, we apply noise suppression and extract speech embeddings using Wav2Vec2 [52 14).
27

---

examples in (2) Prominent Character Filtering: To focus scenes with clearly visible speakers, we use an LLM-based filter that analyzes visual cues and removes clips lacking a central character figure. This step emphasizes narrative relevance and visual clarity.
(3) Motion and Lip-Syne Verification: We further refine the data by checking for facial and body motion aligned with speech. Segments without meaningful movement with weak audio-visual correspondence are excluded.
(4) Scene Captioning: Finally, a large language model [53 used to generate structured captions for each clip. These include detailed descriptions of character appearance, spatial layout, speaking behavior, emotional expression, and physical gostures-enabling rich conditioning during training.
This pipeline results in a curated dataset of approximately 300 hours of high-quality, speech-aligne video content, totaling around 800K annotated samples.
E
Limitations
Despite the strong performance of our model across various talking character scenarios, we identify several limitations that affect generation quality under certain conditions. We provide corresponding examples on the https://congweil2000 1230 wtt.:/ewii.......bb......
Failure Lip Syne in Wide Exteme Wide Shots: When the input caption is too vague- particularly lacking details about facial attributes or shot type-the model may default to generating a wide or extreme wide shot. In such cases, the character often appears the lip region contains only a few pixels. As a result, the model may fail to perform accurate lip synchronization. Example prompt: "A man playing skateboard at skatepark."
far from the camera, and
Multiple Characters in Scene: While the model is generally capable of making the intended character speak in scenes with multiple characters, we observed occasional confusion about which character should be speaking once both characters' face are visible in a single shot. This can lead to degraded lip-syne quality. The limitation likely stems from scarcity of similar the training data, where two or more characters appear in the same shot and speak to the camera. Example prompt: "A medium shot set in a dimly lit tavern. The central figure, rugged man with long, wet-looking hair and a thick beard, sits on a rustic wooden bench. He wears a weathered wool cloak over leather armor, evoking the image as he speaks, holding a short sword in his right hand. To his left, another man with tied-back hair and f fur-lined garments watches him closely."
batte-hardened warrior. His expression intense
Over-Expression with High Speech CFG: Increasing the speech classifier-fres guidance (CFG) value beyond the default (e.g., from 7.5 to 12) can cause the model to generate characters with overly expressive facial and body motions. While this can improve speech emphasis, it may also reduce realism or break immersion in otherwise grounded scenes. Example prompt: "A tracking shot circling around the man as he ties a tie over his blue suit. He speaks to the camera while adjusting the knot, maintaining eye contact throughout."
F Broader Impact
The goal of MoCha is to advance the field of dialogue-centric video generation and enable creative pro- fessionals-such as filmmakers, animators, educators, and content creators-to produce emotionally engaging character-driven videos using natural language and speech inputs. By lowering the barrier cinematic-quality storytelling, this technology democratizes digital media production and unlocks new possibilities for interactive entertainment, educational content, and virtual communication.
However, with any powerful generative technology, there are important societal and ethical considerations to address.
Misuse and Synthetic Media Risks. The use of speech-driven character generation raises concerns around the potential misuse of synthetic media. While MoCha synthesizes characters and scenes entirely from noise, without cloning real human identities voices, there is still risk that the gener- ated videos could be misrepresented as real partiaa-particularry if aligned with real-world audio. This raises issues of misinformation, media authenticity, and potential psychological manipulation.
28

---

Anrortta
HalleS
SadTalker
MoCha (Ours)
/SAm/
/ft://
/lank/
/tar/
/tfes/
Prompt: "A close-up shot of wonan embracing her cat outdoors. She is speaking while facing slightly to the right of the frame with Trustrating expression. The background is.. she cont inues speaking, expression ion renaining tense with anger while still facing slightly the right.."
AniPortriit
SadTTalles
Prompt: nedium of speaking while adjusting glasses.. talks, first glasses, puts
Figure 10: Qualitative comparison between MoCha and baselines on MoCha--Bench, MoCha not only produces lip movements that align closely with the input speech-enhancing the clarity and naturalness articulation but also generates expressive facial animations and realistic, complex actions that faithfully follow the textual prompt. In contrast, SadTalker and AniPortrait exhibit minimal head motion and limited lip synchronization. Hallo3 mostly follows the lip-syncing but suffers from inaccurate articulation, erratic head movements, and noticeable visual artifacts. Since the baselines operate in an image-to-video (12V) setting, we provide them with the first frame generated by MoCha input for comparison. The first frame cropped and resized needed to meet the requirements each baseline.
Unlike deepfake systems that typically manipulate real people's faces, MoCha does not operate on real identity inputs. All visual content is generated from scratch using text and speech prompts. Nevertheless, the audio guidance-if if paired with sensitive or impersonated speech- could still be used to create misleading portrayals. To mitigate this risk, we recommend responsible disclosure practices and support for watermarking or synthetic media detection tools.
Privacy and Consent. Although MoCha does not require real human images voices for generation, broader deployment of similar technologies in the future may prompt privacy concerns, especially adapted to personalized avatars or voice-based likenesses. To ensure responsible use, it is critical to uphold striet data privacy standards, including transparent data usage policies, informed consent when training human likenesses, and tools for individuals opt out of potential misuse.
29

---

Pronpt: "A close-up shot of sitting dark couch. Behind the three white cyldrricat Light fixtures with yellow Lights inside then. the nan continues to speak to the carera while he holds lit the snoke curling gently into the air."
Pronpt: "A nediun shot of man aged 25 to 35 is sitting in the Living roon in leisurely environment. Hif is Live-streaning sitting in frunt of desk with laptap front of hin. His deneanar relaxed and friendly, gesturing with his hands while speaking_"
Prompt: "A close-up talks with worrled expression. Look the window."
young
facing stight right progresses, she speaking eventually
Figure 11: Qualitative results of MoCha on MoCha-Bench, MoCha not only generates lip move- ments that well-synchre selchrynchro with the input speech, but also produces natural facial expressions that reflect the prompt along with realistic hand gestures and action movements
Bias and Representation. As with many generative models, outputs from MoCha may reflect biases present in the underlying training data part--aalllllrlyy in terms of character appearance, behavior, or cultural representation. Careful dataset curation and evaluation are needed to ensure diversity and inclusiveness, and to avoid reinforcing harmful stereotypes or excluding underrepresented groups in generated media.
Ethical Deployment. We encourage deployment of MoCha only in contexts that respect consent, truthfulness, and creative integrity. As the technology matures, we advocate for the establishment of community-drive ethical guidelines, collaboration with media regulators, and public education on the capabilities and limits of Al-generated video.
By proactively identifying and addressing these risks, we aim support the safe and beneficial advancement of talking character technologies and ensure they are developed and applied in ways that are ethical, inclusive, and socially responsible.
30


Reviews:
[{"review_id": "mcUOVpVgrU", "reviewer": "Reviewer_Mvje", "summary": "This paper presents an end-to-end video generation model that synthesizes talking characters from only text and audio inputs. It goes beyond facial regions to include full-body motions, multi-character interactions and environmental dynamics. To this end, the authors modify the text-to-video DiT model and inject audio conditions through temporally aligned audio cross-attention. A curriculum-based multimodal training strategy is introduced to combine speech-labeled and text-only data to improve generalization. Experiments show that the proposed method support coherent, turn-based dialogue among multiple characters, outperforming baselines in lip-sync quality, expression naturalness, and text alignment.", "questions": "None", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "### Strengths\n* Unlike traditional talking-head generation (limited to facial regions and single speakers) , this paper explicitly targets full-body generation with multi-character interactions. This framing fills a clear gap in automating cinematic storytelling and content creation, \n\n* Human evaluations across five axes (lip-sync, expression naturalness, action alignment, etc.) show that the proposed method outperforms existing baselines (SadTalker, Hallo3) by a large margin.\n\n* The paper is overall well-written, and the technical details is clearly provided in the appendix. \n\n### Weaknesses\n* The so-called \"Localized Audio Attention\" seems straight-forward and is not novel to me. Injecting audio condition through temporally-aligned cross-attention has already been explored in Hallo3 [27]. It would be better if the authors could provide more discussion on the difference between this method and Hallo3 in terms of the design of audio injection modules. \n\n* The curriculum-based multimodal training strategy is also not novel enough; a similar idea has already been proposed in OmniHuman [20], which also introduces a multi-stage training strategy and follows a curriculum progressing from text to audio. Unfortunately,  this paper does not explicitly address the connections and distinctions between its strategy and that of OmniHuman.\n\n* The baselines in the comparison experiments rely on reference image inputs. However, it is unclear how the reference images are selected for these baselines in the comparison. In addition, it is unclear how the proposed method could be adapted to reference-conditioned settings.", "was_revised": true, "final_justification": "Thanks for the detailed response, which addresses most of my concerns. After reading the rebuttal and other reviews (as well as the discussion), I would like to maintain my initial positive scores. I am glad to see that the authors have implemented MoCha on top of the open-source backbone Wan2.1 and promise to release it in the revision.", "rebuttal": "Dear reviewer `Mvje`,\n\nWe sincerely appreciate the time and effort you have dedicated to reviewing our work and providing valuable feedback. Below, we are pleased to share our detailed responses to your comments. \n\n**W1 Localized Audio Attention seems has been explored in Hallo3**\n\nTo address the reviewer’s concern, we present a design comparison between Hallo3 and MoCha.\nHallo3 applies stacked convolutions twice to temporally compress audio features, aligning them with the 3D VAE’s compression ratio used for video frames. However, we find that such compression degrades fine-grained audio details. In real-life, dialogue-driven movie generation scenarios, mouth movements can be extremely rapid, and accurately reproducing phonemes such as /iː/, /ai/, /tʃ/, and /θ/ in quick succession requires high-resolution audio features.\nTo preserve these fine-grained details, MoCha avoids temporal compression of audio features. Instead, we propose to constrain each video token to attend to a local window of high-resolution audio tokens, enabling precise audio-visual alignment while retaining rich temporal fidelity in the audio signal.\n\nWe provide a direct comparison of both methods on a DiT-T2V-4B backbone:\n\n|Method |Sync-C↑| Sync-D ↓|\n|-|-|-|\n|DiT-4B + Localized Audio Attention (MoCha-4B)|  5.692 |8.403|\n|DiT-4B + Hallo3 Audio Attention|  4.885 |8.896|\n\nKey Observation: Our novel localized audio attention approach results in superior audio-visual synchronization. Additionally, as shown in visual demos, Hallo3 often exhibits articulation errors and lip-sync artifacts when generating challenging phonemes like /iː/, /ai/, /tʃ/, and /θ/.  In contrast, MoCha captures fine-grained articulation accurately and surpasses all baselines by a large margin.\n\nFurthermore, we acknowledge that Hallo3(Section 3.2) studies 3 audio-injection strategies: (1) self-attention, (2) adaptive-norm, and (3) cross-attention, with cross-attention proving to be the most effective. This is indeed a valuable insight! However no existing work thoroughly studies how to effectively align audio features with temporally compressed video tokens within the cross-attention framework.  Our work addresses this gap directly. As detailed in Appendix Section A.1, we conduct a comprehensive ablation study comparing:(1) Naive Audio Cross Attention (2) Naive Cross-Attn with various positional embeddings (RoPE, learnable, sinusoidal) (3) Our proposed localized audio attention. Our method consistently outperforms all variants. This study provides valuable useful for future research on audio-visual alignment.\n\n**W2 Address the distinctions between proposed curriculum-based multimodal training strategy and that of OmniHuman**\n\nWe provide an explanation below, followed by supporting results.\n\nOur training strategy contains two parts **Shot-Type-Based Curriculum** + **Mixed-Modal Sampling**.\n\nWe agree that existing works(omnihuman) do use multi-stage training. However, they use it to add auxiliary signals, gradually introducing them (face image → audio → pose) to the model in different stages (omnihuman Section 4.1). At the audio stage, they simply feed all audio-annotated data into the model. This works fine as these models' tasks are talking heads generation, and the data are mostly cropped human face videos with similar face sizes. But for MoCha, our task and data are much more complex, involving talking faces→half-body gesture→full-body movements→camera control→multi-character, multi-clip. As the visual scope increases, the mouth region becomes smaller and harder to supervise. Training on all data types simultaneously to learn audio control is clearly inefficient, especially since MoCha doesn't use strong control signals like reference frames. To tackle this, our **Shot-Type-Based Curriculum** is not used to add new signals like existing work but instead structures the learning process based on shot complexity.  **This method significantly improves convergence, as shown in Appendix Table 5.** \n\nSoTA methods(including hallo3, omnihuman) struggle with generating talking characters that go beyond static poses and fixed camera views. They fail to generalize to realistic cinematic scenarios, such as characters talking while expressing emotions, naturally performing actions, interacting with objects, or moving alongside the camera in response to text conditions. We identify two reasons (1) They overlook the richness of text conditions, often reducing prompts to oversimplified inputs like “a person is talking.” (2)They rely exclusively on human data, sampled primarily from audio/pose-driven datasets. To directly address this, we propose **Mixed-Modal Sampling**: We always provided high-quality and multi-grained captions to the model (as shown in Figure 2); During training we consistently sample 20/% of the data from a diverse text2video dataset.\n**This method enables MoCha to achieve superior generalization** in scenarios involving complex lighting, human-animal interactions, and unlock the universal controllability via prompts, eventually **unlock the multiple-character turned based converstation**-**a capability not supported by any existing method**.\n\nWe provide the following experiment against the strongest baseline Hallo3.\nThe base models of Hallo3 and MoCha-4b have similar size and on-pair performance initially.\nAfter learning the audio condition, MoCha-4b outperforms Hallo3 on all axes. Hallo3 suffers from worse generalization, as evidenced by its degraded performance in text alignment and action naturalness compared to its base model.\nThis highlights the effectiveness of our proposed training strategy and underscores the uniqueness of our contribution.\n| Method |Sync-C↑| Sync-D ↓| Facial Expression | Action Naturalness | Text Alignment | Visual Quality |\n|- | - | - |- |- |- | - |\n| CogVideoX-5b (Base for Hallo3) | N/A   | N/A     | 2.88  |  2.85 | 2.91 | **2.83**  |\n| Hallo3 | 4.866 | 8.963   | 2.25 (-0.63) |  2.13 (-0.72)  | 2.35 (-0.56)   | 2.36 (-0.47) |\n| T2V Base for MoCha-4b | N/A | N/A | 2.86   |  2.76  | 2.93 | 2.71  |\n| MoCha-4b | **5.692** | **8.403**  | **2.92** (+0.06)  | **2.87** (+0.11)   | **2.94**(+0.01)| 2.69 (−0.02) |\n\n**W3 It is unclear how the reference images are selected for these baselines in the comparison**\n\n**Answer:**  We would like to emphasize that our work is dedicated to the novel task: **talking characters**. \nThe experiments were designed to compare with existing methods that can only do talking head generation, which are all I2V models. To follow their inference settings, we used the first frame of the video generated from MoCha as input. We manually inspect each frame and avoid blurry or artifacts. The frames also contain clear facial details, making them suitable as reference images. We then crop and resize the frame to match each baseline's training distribution and ensure the head region is centred. \nDespite these efforts, we acknowledge that the first frame from MoCha may still fall outside the baselines’ training distributions. To address this, we conducted further experiments using three groups of inputs:\n\nGroup 1: 3–5 real images from the Internet matching the prompts (e.g., “woman driving a car”), with clear head poses and scene alignment.\n\nGroup 2: 5 synthetic images generated by CogView4, GPT-Image-1, and FLUX.1-dev.\n\nGroup 3: Original MoCha-generated frames.\n\nThese groups test the robustness of baselines across diverse inputs. \n| Method  |Sync-C↑| Sync-D ↓| Facial Expression | Action Naturalness | Text Alignment | Visual Quality |\n|-|-|-|-|-|-|-|\n| Hallo3 (Group1)  | 4.820 | 8.899 | 2.33   | 2.33 | 2.35  | 2.32  |\n| Hallo3 (Group2)  | 5.024 | 9.263 | 2.38  | 2.35 | 2.38 | 2.36  |\n| Hallo3 (Group3)  | 4.866 | 8.963 | 2.25  | 2.13 | 2.35  | 2.36 |\n| SadTalker (Group1) | 4.824 | 9.133   | 1.35 | 1.00 | N/A  | 2.98|\n| SadTalker (Group2) | 4.569 | 9.310   | 1.04  | 1.00 | N/A  | 2.93  |\n| SadTalker (Group3) | 4.727 | 9.239   | 1.14 | 1.00 | N/A   | 2.95  |\n| AniPortrait (Group1)| 1.749 | 11.005  | 1.17 | 1.00 | N/A   | 1.46 |\n| AniPortrait (Group2)| 1.688 | 12.084  | 1.13 | 1.00 | N/A  | 1.45 |\n| AniPortrait (Group3)| 1.740 | 11.383  | 1.12 | 1.00 | N/A | 1.45 |\n| MoCha |**6.037**|**8.103** | **3.82**  | 3.82 | **3.85** | 3.72 |\n\n**Key Observations: (1)Baseline performances are robust across groups, showing MoCha-generated frames are not out-of-domain for the baselines. (2)MoCha significantly outperforms all baselines, confirming its superiority is not due to biased inputs**.\n\n**W4 How the proposed method could be adapted to reference-conditioned settings.**\n\nReference-conditioned generation is a relatively well-studied area, with many existing methods achieving strong results. MoCha can be extended using approaches like FullDiT—encoding the reference image via the VAE and concatenating it with video tokens for 3D self-attention processing, or following the Movie Gen's Personalization method, concatenating the reference image embedding to the text embedding and further being processed through cross-attention.\nMeanwhile, dialogue-driven video generation area was limited to talking-head synthesis, the main research focus of this paper is to **bridge the gap between generic video generation and talking-head synthesis, making cinematic storytelling possible**. We introduced a new task called talking characters, along with a corresponding benchmark and a promising model. MoCha is among the first to demonstrate that movie-grade storytelling can be achieved without relying on auxiliary prior signals, and the first to support multi-character, turn-based dialogue clip generation. Given the promising results of MoCha, we believe it paves the way for future research on unified models that generate talking characters guided by reference images."}, {"review_id": "WmkgwH1lvF", "reviewer": "Reviewer_oyFP", "summary": "The paper presents a new complex task of talking character generation, i.e. generating lip-synced video of full portrait of one or more characters beyond just the facial region. Given it is a new task, the paper also presents a new benchmark and method for the same. The method is novel for four reasons: 1) The method does not use any additional signals apart from the coarsely aligned speech and video description text 2) Uses windowed attention between video and audio instead of full cross-attention 3) Uses curriculum-based training across two axes - the modality and task complexity 4) able to generate coherent multi-character clips in parallel. Subjective (human rating on scale 1-4) and Objective results (lip-sync metrics) clearly state the effectiveness of the method. The benchmark for talking character generation contains 200 instances of human verified text, speech and video.", "questions": "1. The authors have presented three good limitations in the appendix. What are some possible ways to resolve these problems and further improve the method?\n2. What is the total number of training hours and what kind of GPUs were used for training?", "limitations": "yes", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 4, "strengths_and_weaknesses": "Strengths:\n1. Introduces new task, new benchmark and new method for talking character generation. \n2. Limitation section is well thought out.\n3. The appendix is detailed and contains good ablation studies and subjective evaluation details\n4. Writing is clear and good.\n5. The proposed method clearly outperforms existing methods both in subjective and objective evaluations.\n\nWeaknesses:\n1. Please include some sections from appendix to main work in the final version\n2. No mention of intention to open source the code, data and checkpoints. Though this is not compulsory but recommended.", "was_revised": true, "final_justification": "The authors have resolved my questions. I would like to maintain my high score. The work has strong evaluation and high impact on the area of video generation. Some reviewers have questioned the novelty and results of the work, but I found the claims in paper to be well supported and the results are solid.", "rebuttal": "Dear reviewer `oyFP`,\n\nWe sincerely appreciate the time and effort you have dedicated to reviewing our work and providing valuable feedback. \n\nWe are encouraged that you find our paper to be \"well written\" and our proposed methods to be **significant and novel**.\"\n\nBelow, we are pleased to share our detailed responses to your comments.\n\n**W1 Please include some sections from the appendix to the main work in the final version**\n\nThank you for the suggestion. In the final version, we will move the Section A ablations from the appendix to the main paper to ensure clarity and completeness\n\n**W2 No mention of intention to open source the code, data and checkpoints. Though this is not compulsory but recommended.**\n\nAs authors, we are fully committed to open-source research.\nWe have implemented MoCha on top of the open-source backbone Wan2.1 and will include a link to the code in the final revision.\nWe have already included technical details such as model parameters, the full data processing pipeline, and example captions in the submission. We will further enrich the final version with as many details as possible to benefit the community.\n\n**Q1 The authors have presented three good limitations in the appendix. What are some possible ways to resolve these problems and further improve the method?**\n\nFor the limitation: \"If the caption is too vague and doesn't describe the facial attribute or shot type, the model may generate a wide shot where the character is far from the camera, making lip sync difficult to observe.\"\nWe can fine-tune the model via SFT on a curated dataset. We could collect a dataset of (caption, preferred shot framing) pairs where vague captions are matched with preferred video compositions (e.g., close-up shots for talking scenes). Use this to fine-tune the model to better infer appropriate shot types even when explicit cues are missing. For instance, given \"A man playing skateboard at a skatepark\" with accompanying speech, the model could learn to prioritize closer shots for improved lip sync alignment.\nWe could also deploy annotators to rate generated videos on lip sync clarity and framing quality. Use these ratings to train a reward model and apply reinforcement learning or preference-based learning to optimize generation toward human-preferred outputs. This would teach the model to implicitly prioritize facial visibility when speech is present, even from vague captions.\n\nFor the limitation : \"When multiple characters appear in the same scene, sometimes the lip-sync quality is degraded when the character is far from the camera, potentially due to limited training data.\"\nWhen multiple characters are present, the model may struggle to identify which character the speech corresponds to, based solely on the text. To mitigate this, we could incorporate character embeddings alongside speech embeddings to provide clearer grounding.  For example, a set of learnable embeddings could be added to the speech embeddings to indicate the target character. This may help the model better associate the audio with the correct character and improve lip sync quality, particularly in multi-character scenes.\n\nFor the limitation :\"When increasing the speech CFG from the default value of 7.5 to 12, the model tends to generate overly expressive characters. For example: 'A tracking shot circling around the man as he ties a tie over his blue suit. He speaks to the camera while adjusting the knot, maintaining eye contact throughout.'\" To mitigate this, we could rescale the norm of the guided prediction at each denoising step to match the original conditional signal. This helps prevent exaggerated behaviors caused by excessively high CFG values.\n\n**Q2 What is the total number of training hours and what kind of GPUs were used for training?**\n\nOur experiments were conducted on H100 GPUs, totaling 49,152 GPU hours."}, {"review_id": "iJXRCiJKa9", "reviewer": "Reviewer_gs1v", "summary": "Mocha proposed a DiT-based text- and speech-driven Talking Character generation method, which uses cross attention and localized audio attention to inject text and speech conditions, achieving state-of-the-art performance in Talking Character generation.", "questions": "See weakness.", "limitations": "As mentioned in weakness, this method is based on a closed-source model and the experimental overhead is too high to be reproduced.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "**Strengths:**\n\n1. Compared to other methods, this approach achieves video generation that is better aligned with both text and speech, marking a significant step toward the practical application of video generation.\n2. The use of localized audio attention enables better alignment between lip movements and speech in the generated videos, making it a sensible and effective design.\n\n**Weaknesses:**\n\n1. The method has not been open-sourced, nor are there plans for open-sourcing it. Additionally, the experiments were conducted using Meta’s internal MovieGen framework, making it difficult for the community to reproduce the results.\n2. The five steps mentioned in Curriculum Training lack ablation studies for each stage, making it seem unnecessary to design such a complex process.\n3. The method has high resource consumption. Despite being based on a powerful video generation model, it still requires 64 computing nodes for post-training. It does not consider freezing most of the network modules during training or using other efficient fine-tuning methods.", "was_revised": true, "final_justification": "I recommend accepting this article.", "rebuttal": "Dear reviewer `gs1v`,\n\nWe sincerely appreciate the time and effort you have dedicated to reviewing our work and providing valuable feedback. Below, we are pleased to share our detailed responses to your comments.\n\n**W1 The method has not been open-sourced, nor are there plans for open-sourcing it.**\n\nAs authors, we are fully committed to open-source research. \nWe have implemented MoCha on top of the open-source backbone Wan2.1 and will include a link to the code in the final revision. \nWe have already included technical details such as model parameters, the full data processing pipeline, and example captions in the submission. We will further enrich the final version with as many details as possible to benefit the community.\n\n**W2 The five steps mentioned in Curriculum Training lack ablation studies for each stage, making it seem unnecessary to design such a complex process.**\n\n**Answer:** Thank you for the suggestions. Below, we provide both explanation and supporting results to justify the necessity of our curriculum design. \n\nTraditional talking-head generation tasks typically use cropped face videos with limited variability. In contrast, MoCha tackles a significantly more complex task, involving: talking faces→half-body gesture→full-body movements→camera control→multi-character, multi-clip. As the visual scope increases, the mouth region becomes smaller and harder to supervise. Training on all data types simultaneously is highly inefficient for learning audio-driven control—especially since MoCha does not rely on strong control signals like reference frames. To tackle this, our Shot-Type-Based Curriculum is used to structures the learning process based on shot complexity. This method significantly improves convergence.\n\nWe provide an ablation study of different stages using the 4B version, trained for the same number of steps (20K) and evaluated on MoChaBench.  Stage 0 corresponds to T2V pretraining, and Stage 1 introduces the audio signals, which are indispensable for the task. Therefore, our ablation focuses on the contributions of Stage 2 and Stage 3\n\n| Method                                                       |Sync-C↑| Sync-D ↓| Facial Expression | Action Naturalness | Text Alignment | Visual Quality |\n|-                                                             | -     | -       |-                  |-                   |-               | -              |\n| DiT-4b + (close-up and medium close up curriculum training)  | 5.718 | 8.374   | 2.96              | 1.37               | 1.18           | 2.67           |\n| DiT-4b + (close-up and medium close up joint training)       | 3.846 | 10.378  | 1.25              | 1.16               | 1.06           | 2.55           |\n\nAnalysis: Medium close-up data introduces information such as co-speech gesture and the character's overall motion. Joint training on close-up and medium close-up data shows significantly degraded results in lip-sync metrics compared with curriculm training, as the facial region in medium close-up data is relatively smaller, providing limited supervision for the audio-conditioning module and leading to slower learning.\n\n| Method                                                       |Sync-C↑| Sync-D ↓| Facial Expression | Action Naturalness | Text Alignment | Visual Quality |\n|-                                                             | -     | -       |-                  |-                   |-               | -              |\n| DiT-4b + (close-up, medium close-up and medium shot curriculum training)  | 5.692 | 8.403   | 2.92      | 2.87       | 2.94           | 2.69           |\n| DiT-4b + (close-up, medium close-up and medium shot joint training)       | 2.394 | 11.291  | 1.22      | 2.66       | 1.58           | 2.39           |\n\nAnalysis: Medium shot data focuses on full-body actions and camera control.\nJoint training with all shot types at once further degrades lip-sync performance and text alignment. In contrast, curriculum training gradually builds the model’s ability to handle increasing complexity, resulting in balanced performance across metrics. While lip-sync scores are slightly lower than the previous stage, gains in action naturalness and text alignment are substantial.\nThese ablation studies collectively demonstrate the necessity and effectiveness of each stage in our proposed curriculum. \n\n\n**W3: Consider freezing most of the network modules during training. L1: The experimental overhead is too high**\n\n**Answer:** We thank the reviewer for raising this important question.\n\nTo address concerns about high resource consumption,\nwe provide the following experiment to show that our proposed training strategy and localized audio attention methods work for small models.\nThe base models of Hallo3 and MoCha-4b have similar sizes and comparable initial performance.\nAfter learning the audio condition, MoCha-4b outperforms Hallo3 across all axes. Hallo3 suffers from worse generalization, as evidenced by its degraded performance in text alignment and action naturalness compared to its base model.\n| Method                         |Sync-C↑| Sync-D ↓| Facial Expression | Action Naturalness | Text Alignment | Visual Quality |\n|-                               | -     | -       |-                  |-                   |-               | -      |\n| CogVideoX-5b (Base for Hallo3) | N/A   | N/A     | 2.88              |  2.85              | 2.91           | **2.83**       |\n| Hallo3                         | 4.866 | 8.963   | 2.25 (-0.63)      |  2.13 (-0.72)      | 2.35 (-0.56)   | 2.36 (-0.47)   |\n| T2V Base for MoCha-4b          | N/A   | N/A     | 2.86              |  2.76              | 2.93           | 2.71           |\n| MoCha-4b                | **5.692** | **8.403**  | **2.92** (+0.06)  | **2.87** (+0.11)   | **2.94**(+0.01)| 2.69 (−0.02)   |\n\nThis demonstrates that the effectiveness of our approach is not limited to large models requiring high resource consumption.\n\nOur research goal—as stated in the claims—is to **bridge the gap between generic video generation and constrained talking-head synthesis**, which requires a promising talking character model that can sync well with speech and perform actions naturally with overall good visual quality.\n\nHowever, as shown in the table above, base models with 4B/5B parameters still fail to consistently generate videos with a score of 3, meaning they cannot reliably produce mostly artifact-free and prompt-following results.\nTherefore, to support our goal, we conduct the research on top of a large base model.\n\nWe have also conducted experiments during research where we only trained the newly introduced audio attention module while freezing the remaining backbone.\nWe provide the results as follows:\n| Method                          |Sync-C↑| Sync-D ↓ | Facial Expression | Action Naturalness | Text Alignment | Visual Quality |\n|-                               | -     | -       |-                  |-                   |-               | -      |\n|MoCha                                        | 6.037 | 8.103    | 3.82              | 3.82               | 3.85           | 3.72 |\n|DiT + Train Audio Cross Attention Only       | 1.859 | 12.784   | 1.04              | 1.00               | 1.17           | 1.21           |\n|DiT                                          | N/A   | N/A      | 3.49              | 3.82      | 3.16          | 3.73       |\n\nWe observed a dramatic degradation in model performance across all axes when training only the audio cross-attention module, compared with full training (MoCha) for the same number of steps.\nThis observation differs from talking-head tasks, where training only the newly added module is often more effective than full fine-tuning.\nWe hypothesize that since talking-head tasks always include a first-frame/pose condition—a strong conditioning signal—they converge more easily.\nIn contrast, talking character generation is a more complex task.\nThus, we consider full training of the model to be more effective in our setting.\n\nThe resulting model, MoCha, is **among the first to demonstrate that movie-grade storytelling can be achieved without relying on auxiliary prior signals**, and **the first to support multi-character, turn-based dialogue clip generation**, which **bridges the gap between generic video generation and constrained talking-head synthesis**.\nGiven the promising results of MoCha, we believe it paves the way for future research on smaller yet stronger models (possibly distilled from large models) to extend talking character generation to broader applications."}, {"review_id": "EZ5xFnYjjf", "reviewer": "Reviewer_qDos", "summary": "This paper proposes a text-to-video model that supports speech audio conditioning. It details the implementation of progressive training and audio conditioning techniques. Built on a 30B base model, the approach outperforms baseline methods across multiple evaluation metrics.", "questions": "- Using audio conditioning with features extracted from adjacent frames and multi-stage training is very common in audio-conditioned video generation methods such as EMO, Hallo, Hallo3, and OmniHuman. A clearer explanation of the paper’s unique contributions in this context would be helpful.\n- The current experimental comparison setup is inconsistent for other methods, making fair evaluation difficult, as noted in the weaknesses. It is recommended that the authors compare their method directly with other text-to-video approaches (e.g., using the metrics in Figure 7) while excluding audio-related metrics, which are already clearly reported in Table 2.\n- The authors should provide some evaluation or discussion of the base model’s performance itself, as this would improve the completeness and clarity of the paper.", "limitations": "- The paper has some questionable experimental settings, such as comparing image-to-video methods using images generated by the proposed text-to-video model.\n- It lacks analysis and comparison with other text-to-video models, including the base video generation model.\n- The method’s contributions are unclear, as audio conditioning and multi-stage training are common practices in this field.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1. The paper is clearly written, providing detailed explanations of the implementation.\n2. The visual results are strong, showing noticeable improvements over baseline methods.\n\nWeaknesses:\n1. The experimental setup is flawed: the baselines compared are image-plus-audio to video models, but the test images come from the proposed text-to-video model. This likely makes the test images out-of-domain for the baselines but in-domain for the proposed method, resulting in an unfair comparison.\n2. The method shares many similarities with existing work, such as the audio conditioning approach and multi-stage training, but the paper lacks clear discussion or analysis of these overlaps.\n3. The method relies on a large, strong base video generation model which differs from other methods and this difference is not addressed or analyzed in the paper.", "was_revised": true, "final_justification": "This paper demonstrates that multi-character, turn-based conversational video generation can be achieved through joint data training and specific architectural designs. The results are impressive. However, in the initial submission, the description of certain methods lacks clarity, particularly the audio conditioning method. The authors will revise these descriptions in the final version. I recommend this paper for acceptance.", "rebuttal": "Dear reviewer `qDos`,\n\nWe sincerely appreciate the time and effort you have dedicated to reviewing our work and providing valuable feedback. Below, we are pleased to share our detailed responses.\n\n**W1&L1 Use the test images from the MoCha are likely out-of-domain for the baselines. Q2 & Q3 &L2: Recommended to compare MoCha directly with other t2v approaches and the base model**\n\n**Answer:** We thank the reviewer for raising this important point!  Our work is dedicated to the novel task: **talking characters**. \nThe experiments were designed to compare with existing methods that can only do **talking head** generation.\nTo follow their inference settings, we used the first frame from MoCha as input.\nWhile we cropped the frames to match baselines' training distribution and avoid blurry or artifacts, we acknowledge that potential distribution mismatch may still exist.\nWe conducted further experiments using three diverse groups of inputs to test the robustness of baselines. \n\nGroup 1: 3–5 real images from the Internet matching the prompts (e.g., “woman driving a car”), with clear head poses and scene alignment.\n\nGroup 2: 5 synthetic images generated by CogView4, GPT-Image-1, and FLUX.1-dev.\n\nGroup 3: MoCha-generated first frame.\n\n**Key Observations: (1)Baseline performances are robust across groups, showing MoCha-generated frames are not out-of-domain for them. (2)MoCha significantly outperforms all baselines, confirming its superiority is not due to biased inputs**.\n\nIn response to Q2 & Q3 &L2.  We thank the reviewer for the suggestions! and compared MoCha with other t2v models as required.  We observed that since mocha is trained on talking human videos, it achieves the best facial expression realism. It can handle complex prompts like \"speaking while gently sipping water.\" while other models fail to produce talking behavior in this case. This comparison further supports our claim:\n**MoCha bridges the gap between Movie Generation and Talking-Head Synthesis, making cinematic storytelling possible.**\n\n| Method|Sync-C↑| Sync-D ↓| Facial Expression | Action Naturalness | Text Alignment | Visual Quality |\n|-|-|-|-|-|-|-|\n| Hallo3 (Group 1)| 4.820 | 8.899   | 2.33  | 2.33| 2.35  | 2.32|\n| Hallo3 (Group 2)| 5.024 | 9.263   | 2.38  | 2.35| 2.38 | 2.36|\n| Hallo3 (Group 3)| 4.866 | 8.963   | 2.25  | 2.13| 2.35 | 2.36 |\n| SadTalker (Group 1)   | 4.824 | 9.133   | 1.35   | 1.00 | N/A| 2.98       |\n| SadTalker (Group 2)   | 4.569 | 9.310   | 1.04    | 1.00 | N/A | 2.93       |\n| SadTalker (Group 3)   | 4.727 | 9.239   | 1.14 | 1.00      | N/A            | 2.95       |\n| AniPortrait (Group 1) | 1.749 | 11.005  | 1.17 | 1.00   | N/A| 1.46       |\n| AniPortrait (Group 2) | 1.688 | 12.084  | 1.13  | 1.00      | N/A | 1.45       |\n| AniPortrait (Group 3) | 1.740 | 11.383  | 1.12| 1.00      | N/A            | 1.45       |\n| Kling1.6 | N/A   | N/A     | 3.51    | 3.12      | 3.15          | 3.74       |\n| Wan2.1  | N/A   | N/A     | 3.48    | 3.79      | 3.48          | 3.67       |\n| OpenAI SoRA  | N/A   | N/A     | 3.25  | **3.83**  | 3.21   | **3.89**   |\n| MoCha(base T2V model)| N/A   | N/A     | 3.49              | 3.82      | 3.16          | 3.73       |\n| MoCha   |**6.037**|**8.103** | **3.82** | 3.82      | **3.85**      | 3.72       |\n\n\n**W2 & Q1 & L3: Lack of Discussion on the uniqueness of audio conditioning and multi-stage training strategy**\n\nWe provide discussions below, followed by supporting results.\n\n**1. Localized audio attention**\nMost talking head generation models (EMO, Hallo) employ 2D Unet model and align audio and video features on a per-frame basis. DiT utilizes 3D VAEs with temporal compression of video frames, creating a resolution mismatch between audio and video features. Notably, no existing work (Hallo3, omnihuman) thoroughly studies how to align audio and video within a DiT framework effectively. In Appendix Section A.1, we are among the first to conduct a comprehensive ablation on this, evaluating (1) Naive Audio Cross Attn (2) Naive Cross-Attn with positional embeddings (RoPE, learnable, sinusoidal) (3) Our proposed localized audio attention. Our method consistently outperforms all variants. This study offers valuable insight for future research. Furthermore, we compare it to Hallo3’s alignment strategy. Hallo3 temporally compresses audio features twice via stacked convolutions to match the video tokens. We find that this compression degrades fine-grained audio details, making it challenging to render characters accurately pronouncing phonemes like /iː/, /ai/, /tʃ/, and /θ/. In contrast, our method avoids temporal compression. We constrain each video token to only attend to a local window of audio tokens, which preserves the fine-grained details of the audio signal.\n\nWe provide a direct comparison of both methods on a DiT-T2V-4B backbone. Our approach results in superior audio-visual synchronization.\n|Method |Sync-C↑| Sync-D ↓|\n|- |-  | - |\n|DiT-4B + Localized Audio Attention (MoCha-4B)|  5.692 |8.403|\n|DiT-4B + Hallo3 Audio Attention|  4.885 |8.896|\n\n\n**2. Training Strategy**\nOur training strategy contains two parts **Shot-Type-Based Curriculum** + **Mixed-Modal Sampling**.\n\nExisting works(hallo, hallo3, omnihuman) do use multi-stage training. However, they use it to add auxiliary signals, gradually introducing them (face image → audio → pose) to the model. At the audio stage, they simply feed all audio-annotated data into the model. This works fine as these models' tasks are talking heads generation, and the data are mostly cropped human face videos with similar face sizes. But for MoCha, our task and data are much more complex, involving talking faces→half-body gesture→full-body movements→camera control→multi-character, multi-clip.  As the visual scope increases, the mouth region becomes smaller and harder to supervise. Training on all data types simultaneously to learn audio control is clearly inefficient, especially since MoCha doesn't use strong control signals like reference frames. To tackle this, our **Shot-Type-Based Curriculum** is not used to add new signals like existing work, but instead structures the learning process based on shot complexity. **This method significantly improves convergence, as shown in Appendix Table 5.**\n\n\nSoTA methods(including hallo3, omnihuman) struggle with generating characters that go beyond static poses and fixed camera views. They fail to generalize to realistic cinematic scenarios, such as characters talking while expressing emotions, naturally performing actions, interacting with objects, or moving alongside the camera in response to text conditions. We identify two reasons (1) They overlook the richness of text conditions, often reducing prompts to oversimplified inputs like “a person is talking.” (2)They rely exclusively on human data, sampled primarily from audio/pose-driven datasets. To directly address this, we propose **Mixed-Modal Sampling**: We always provided high-quality and multi-grained captions to the model (as shown in Figure 2); During training we consistently sample 20/% of the data from a diverse text2video dataset.\n\n**This method enables MoCha to achieve superior generalization** in scenarios involving complex lighting, human-animal interactions, and unlock the universal controllability via prompts eventually **unlock the multi-character turned based converstation ablity**.\n\nWe provide the following experiment against the strongest baseline Hallo3.\nThe base models of Hallo3 and MoCha-4b have similar size and on-pair performance initially.\nAfter learning the audio condition, MoCha-4b outperforms Hallo3 on all axes. Hallo3 suffers from worse generalization, as evidenced by its degraded performance in text alignment and action naturalness compared to its base model.\nThis underscores the uniqueness of our contribution.\n| Method |Sync-C↑| Sync-D ↓| Facial Expression | Action Naturalness | Text Alignment | Visual Quality |\n|-  | -  | - |-  |- |- | - |\n| CogVideoX-5b (Base for Hallo3) | N/A   | N/A     | 2.88  |  2.85    | 2.91| **2.83**       |\n| Hallo3  | 4.866 | 8.963   | 2.25 (-0.63)      |  2.13 (-0.72)      | 2.35 (-0.56)   | 2.36 (-0.47)   |\n| T2V Base for MoCha-4b    | N/A   | N/A     | 2.86    |  2.76   | 2.93   | 2.71|\n| MoCha-4b  | **5.692** | **8.403**  | **2.92** (+0.06)  | **2.87** (+0.11)   | **2.94**(+0.01)| 2.69 (−0.02)   |\n\n**W3: The method uses a large, strong base video generation model differs from the others**\n\nAs shown in the tables above, our proposed methods consistently outperform baselines, even when using base models of similar size and capability. This demonstrates the effectiveness of our approach, not limited to large models.\nOur research goal—as stated in the claims—is to **bridge the gap between generic video generation and constrained talking-head synthesis**. We also want to deliver the message that **movie-grade storytelling can be achieved without relying on auxiliary prior signals**.\nHowever, as shown in the table above, base models with 4B/5B parameters still fail to consistently generate videos with a score of 3, meaning they cannot reliably produce mostly artifact-free and prompt-following results. \nTherefore, to support our goal, we intentionally build on top of a large base model to ensure satisfactory visual fidelity in the results.\n\n**L3: The method’s contributions are unclear.**\n\nWe would like to clarify and highlight the key contributions of our work:\nMoCha is **among the first to bridge the gap between video generation and talking-head synthesis, making cinematic storytelling possible**. We introduced a new task —talking characters, along with a corresponding benchmark and a promising model for this task.\nMoCha is also  **among the first to demonstrate that movie-grade storytelling can be achieved without relying on auxiliary prior signals**. Finally, it is **the first to support multi-character, turn-based dialogue clips generation.** A capability not achieved by any existing method to date."}]

Author rebuttal:

Meta-review: This paper received overwhelmingly positive feedback from all reviewers. It presents a text-to-video generation model that supports full-body motions, multi-character interactions, and environmental dynamics. This represents a novel and meaningful task. The alignment of speech and video is particularly appreciated. The method demonstrates significantly better performance than baseline approaches in both subjective and objective evaluations. With the authors’ commitment to releasing the source code, this work has the potential to make a high impact on the video generation community.

The reviewers raised some concerns regarding the experimental setup and the degree of novelty. The authors provided thorough clarifications during the rebuttal period. Taking all of this into account, I recommend acceptance.
Decision: Accept (spotlight)