Instilling an Active Mind in Avatars via Cognitive Simulation
Authors: Jianwen Jiang
Weihong Zeng
Zerong Zheng
Jiaqi Yang
Chao Liang
Wang Liao
Han Liang
Weifeng Chen
XING WANG
Yuan Zhang
Mingyuan Gao
Venue: ICLR 2026 Oral (2026)
Keywords: Video Generatio
Human Animation
Avatar
Multimedia
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: This paper introduces a novel framework that uses a Large Language Model (LLM) for semantic guidance and a Multimodal Diffusion Transformer (DiT) for fusion to generate expressive, context-aware video avatars, demonstrating competitive performance

Abstract: Current video avatar models can generate fluid animations but struggle to capture a character's authentic essence, primarily synchronizing motion with low-level audio cues instead of understanding higher-level semantics like emotion or intent. To bridge this gap, we propose a novel framework for generating character animations that are not only physically plausible but also semantically rich and expressive. Our model is built on two technical innovations. First, we employ Multimodal Large Language Models to generate a structured textual representation from input conditions, providing high-level semantic guidance for creating contextually and emotionally resonant actions. Second, to ensure robust fusion of multimodal signals, we introduce a specialized Multimodal Diffusion Transformer architecture featuring a novel Pseudo Last Frame design. This allows our model to accurately interpret the joint semantics of audio, images and text, generating motions that are deeply coherent with the overall context. Comprehensive experiments validate the superiority of our method, which achieves compelling results in lip-sync accuracy, video quality, motion naturalness, and semantic consistency. The approach also shows strong generalization to challenging scenarios, including multi-person and non-human subjects. Our video results are linked in https://omnihuman-lab.github.io/v1_5/ .

Published as a conference paper at ICLR 2026
INSTILLING AN ACTIVE MIND IN AVATARS VIA COG- NITIVE SIMULATION
Jianwen Jiang", Weihong Zeng', Zerong Zheng', Jiaqi Yang,, Chao Liang', Wang Liao", Han Liang". Weifeng Chen, Xing Wang, Yuan Zhang, Mingyuan Gao ByteDance j anwen. a anê gma com
ABSTRACT
Current video avatar models can generate fluid animations but struggle to capture character's s authentic essence, primarily synchronizing motion with low-level audio cues instead of understanding higher-level semantics like emotion or in- tent. To bridge this gap, propose novel framework for generating character animations that are not only physically plausible but also semantically rich and technical innovations. First, employ
expressive. Our model is built Multimodal Large Language Models to generate a structured textual representa- tion from input conditions, providing high-level semantic guidance for creating contextually and emotionally resonant actions. Second, to ensure robust fusion of multimodal signals, we introduce a specialized Multimodal Diffusion Transformer architecture featuring novel Pseudo Last Frame design. This allows our model to accurately interpret the joint semantics of audio, images and text, generating motions that are deeply coherent with the overall context. Comprehensive exper- iments validate the superiority of our method, which achieves compelling results in lip-sync accuracy, video quality, motion naturalness, and semantic consistency. The approach also shows strong generalization to challenging scenarios, including multi- multi-person person and non-human subjects. Project page is linked here
INTRODUCTION
"System operates automatically and quickly, with little or no effort and no sense of voluntary control. System 2 allocates attention to the effortful mental activities that demand it, including complex computations."
Daniel Kahneman, Thinking, Fast and Slow
The field of video avatars (He al.
2023
Tian al. 2025c Xu et al. 2024a Wang al. 2024a
Chen et al. 2024c Xu et al.] 2024b
2025a et al. 2024 Jiang et al. 2025 Lin in et al.
Gan et al.] 2025| Kong et al. 2025
Wang et al.]|[222222 2022b Hu [2024b] Lin et al. 2025b] Qiu al. 2025) aims to synthesize realistic character videos from driving signals, with the goal of creating
lifelike digital humans capable of reasoned action and authentic emotion. Recent years have seen
rapid progress, evolving from early lip-sync (Jiang et al.) 2024 Zhang et al. 2023 Wang et al. 2021 Tian
Zhao & Zhang 2022, Siarohin et al.) 2019 2021) and portrait animation (Jiang et al. 2025
et al. [2025c] Zhong et al. 2024 Xu et al. 2024b Cui et al. 2024) to half-body (Lin et al. 2025a
Tian et al. (025a) and full body generation (Lin al. (0225). As the scope of generation expands,
so does the expectation for models to move beyond mere physical likeness and capture a character's authentic essence, their underlying personality, emotion, and intent, as shown in Figure [
A recent wave of audio-driven methods based on Diffusion Transformers (DiT) (Peebles & Xie 2023 Esser et al., 2024b] SSawead et al., 2025 Kong et al. 2024 Wang et al. 2025a) can gener-
human motion synchronized with audio (Lin al.) 2025b Kong al. 2025 Gan et al. 2025
Wang et al. 2025b, Qiu et al., 2025). However, these models typically learn only low-level corre- lations, resulting in accurate lip movements but simple, repetitive gestures. Their outputs lack the
"Equal contribution.
I Project Lead and Corresponding author.

---

Published as a conference paper at ICLR 2026
Cognitive Simulation
Existing Methods (System only) - Motion is repetitive and lack context
Seaech sice pariind diicei dease grreee You? lakk heriiiing lari and al the thing badee De Ohotecso Pevet oel and the, ges see held you pourpe 11 stoee free onyy unpeeiee Spreet Phi wool thet lock
System Reactive
Direcity Autio--Diiven
System 2
Ours (System System 2) Motion is logical and contextu ally dlyven
Deliserative
Logical
Conditioning via Audio Only
Capable of Joint Conditioning via Text and Audio
Herboad kulg swaay body dhamiigh. wigh
book al ompitasion behied lis rowaiigg teukkukk vieee for dud eu fuve bean Mown to strood
Figure I: Simulating Mind for Avatars. We model avatar behavior using dual-process theory, distinguishing between reactive (System I) and deliberative (System 2) cognition. Top Right: Con- ventional methods, analogous to System excel at reactive tasks like lip-syne but produce repetitive, non-contextual motions. Top Left & Bottom: In contrast, our framework simulates both systems, integrating System 2's high-level reasoning to generate diverse, context-aware behaviors that are semantically aligned with the audio and text.
contextual appropriateness and plausibility of authentic human behavior, revealing a significant gap between current capabilities and the goal of creating convincing avatars. We attribute this gap to failure to model higher-order order reasoning. Drawing inspiration from the dual duaa--rocesss theory of hu- cognition (Kahneman, 2011 Kahneman & Tversky 2013), which distinguishes between fast, intuitive "System 1s thinking and slow, deliberative "System 2" reasoning, we observe that existing models operate mainly at the level of System 1. They excel reactive mappings, like audio-to-lip- movement, but cannot perform the goal-oriented, contextual reasoning characteristic of System 2. We argue that the path forward lies in emulating both systems. To this end, we propose leveraging the powerful reasoning capabilities (Wei et al.) 2022 Wang et al.) Yao et al. 2023 Schick et al., 2023 Park al. 2023 Yuan al.) 2024 Wu et al.) 2d.55b) of Multimodal Large Language Models
Team et al. 2023 OpenAl 2024) to explicitly simulate the deliberative processes of System 2.
However, integrating MI.L. --geerraee textual guidance into an avatar framework is non-trivial. The text, which articulates high-level reasoning, introduces a new modality that can conflict with existing signals like audio (for rhythm) and a reference image (for identity). Naive fusion can lead to modal interference, where, for example, low-level audio cues might disrupt high-level, text-guided semantic actions, and reference image conditioning could alter their motion magnitude. Therefore, novel architecture is required to effectively manage these interdependencies and mitigate conflicts, enabling the simultaneous simulation of both System and System 2.
Motivated by this analysis, propose a Multimodal DiT framework with two key designs. First, an MIL.M-powere agent generates a high-level, logically coherent "System 2" signal by reason- ing over multimodal inputs Second, we introduce specialized MMDiT architecture and training
Wee employ the "System 1" and "System 2" dichotomy as a functional analogy. Our goal is not neuro- scientific replication, hut to adopt the interplay between reactive and deliberative processes to improve avatar coherence. For conciseness, also use these terms refer to the corresponding modules in framework.
2

---

Published as a conference paper at ICLR 2026
methodology to fuse these inputs and mitigate interference. This architecture incorporates novel pseudo-last-frame strategy that preserves identity without constraining dynamic, content-driven mo- tion. by leveraging the model's temporal extrapolation capabilities maintain identity, instead of directly conditioning on the reference image during training. Our main contributions are as follows:
A New Perspective on Avatar Modeling. We are the first frame the video avatar problem through the cognitive science lens of System and System 2, identifying the limitations of current models and proposing holistic approach that models both.
A Framework for Dual-System Simulation. We propose a novel framework featuring MLLM- based agents for deliherative "System 2n guidance and specialized MMDIT architecture with pseudo-last-frame strategy to synergistically fuse this guidance with reactive "System 1" signals, resolving critical modal conflicts.
Slate-of-the-Ar Performance and Generalization. Our method achieves highly competitive re- sults on multiple benchmarks and significantly preferred in user studies for its contextual natural- ness. Its versatility is further demonstrated by its successful extension to complex multi multi-person and non-human scenarios.
2 RELATED WORK
Video Generation. The field of video generation has rapidly advanced, largely building on the suc- cess of diffusion models in visual synthesis (Ho et al.] [020]Song al., 2020). Current approaches can be broadly categorized by their architecture. The first category adapts pre-trained text-to-image
U-Nets (Esser et al. 2024a Chen et al. 2024a) by inserting temporal modules and fine-tuning on video data (Guo et al. 2023 Wang et al. 2023b). While leveraging powerful image priors, these methods can be constrained by their original image-centric design. The second category employs
Diffusion Transformer (DiT) architectures (Brooks et al. 2024;) Yang et al., 2024] Zheng al. 2024
Kong et al. 2024 Ma et al. 2025 Polyak et al.) 2024| Chen et al.) 2024b Menapace et al.] 2024). which treat video as sequence of spatiotemporal patches. This unified approach has demonstrated
superior scalability and flexibility, enabling high-resolution video generation with variable dura- tions. A third, emerging direction involves integrating Large Language Models (LLMs) enhance logical coherence and narrative structure. While this approach is still nascent in video generation, it has shown great promise in the image domain (Yu et al.) 2023b) Koh et al., 2023] Pan et al., 2025 Shi et al. 2024) and holds significant potential for our field.
Wu et al. 2025a
2024
Video Avatars. Video avatar models aim to create realistic human videos from driving signals, with audio being a primary focus of recent research. These audio-driven animation methods must tackle the dual challenges of motion generation and rendering. common paradigm two-stage pipeline: first, model translates audio into an intermediate motion representation (e.g.. keypoints, meshes, or tokens) (Zhuang et al. 2024; Meng et al. 2025 Deng et al., 2025 Wei et al., 2024] Hogue et al. 2024 Tian et al.) 2025b); then, rendering model synthesizes the final video. This rendering stage, which animates a character from an explicit motion sequence, constitutes an inde-
pendent research area known pose-driven animation (Zhang al. 2024 Shao al.) 2024 Chang jet al. 2023] Tu et al.; 2024 Wang et al.) 2024b) Karras et al. 2023 Xu et al. 2024c), where the primary focus is on rendering fidelity. In contrast to these two-stage methods, recent end-to-end approaches directly generate video from audio for better synchronization (Lin et al. 2025a b Wang al. 2025c Liang et al. 2025). Despite their architectural diversity, all these audio-driven methods fundamentally treat motion generation as direct, reactive mapping. This process lacks an explicit cognitive phase of planning and reasoning, which our work introduces to generate more plausible and intelligent behaviors.
LLMss for Cognitive Simulation and Control. Large Language Models (L.LMs) have evolved from
foundational models (Brown et al., 2020 Ouyang et al.) 2022; Achiam et al.)
2023) to powerful
cognitive engines with advanced reasoning (Hurst et al. 2024 Jaech et al. 2024 Guo et al. 2025a).
unlocked by prompting techniques such as Cf -fhuughhtt (Wei et al. 2022 Wang et al. Yao
et al., 2023) and other applications (Liang et al., 2024 Kannan et al. 2024, Qu et al. 2023). This
has empowered autonomous agents to strategize and simulate believable human behaviors (Schick)
et al. 2023 Hong et al.) 2023, Wang et al., 2023a] Park et al.] (023). Beyond standalone agents, L.M reasoning is also used to steer generative models, acting universal planner for controllable
3

---

Published as a conference paper at ICLR 2026
The Dual-System Simulation Framework
Module Details
STT TEA 2. AGRNTI HEARONING
Mutti-stip Ressoning Pipeline
INPUTS
Reserig
I1fl--
Conditions
Conditioss
All inpura
MILM Andlysor
N
Lerping Contett Adter Expaion
Audi
MILMM Agents
Output
MILM Planner
SYSTEM REACTIVE
NiM- Branch Worm-up
Pesudoo Last Frame
Referonce
Audio
Text Encoder
Text Branch
Daloon Warm-up
Audio
Thit
Training
Fromee
T
Audio Encoder
Audio Branch
Vicee
Images
Generated Video
Re-inildtred
Teid
VAE Encoder
Network
Audo
Teet
Promee
00000000
The Dual-System Simulation Framework. Our framework models avatar behavior by integrating a deliberative System 2 for planning with reactive System I for synthesis. Left: Over- all Pipeline. An MLLM-based System reasons over multimodal inputs (audio, image, text) to generate high-level "schedule". This schedule guides the System MMDiT, which synthesizes the final video by fusing information through dedicated text, audio and video branches. Right: Key Components. (a) The System 2 reasoning pipeline, comprising MILM Analyser and Planner. (b, c) Our proposed MM-Branch Warm-up and Pseudo Last Frame strategies, designed to mitigate modal conflicts.
Figure 2:
image and video synthesis (Brooks al. 2023 Pan et al. 2025 Hu al. 2024a Yuan al. 2024
Li et al. 2024] Wu et al.) (2225)). However, applying the reasoning capabilities of these powerful models to generate fine-grained, intelligent avatar behavior remains largely unexplored area which our work directly addresses.
APPROACH
3.1 OVERVIEW
Our goal is generate character animations that both visually realistic and logically coherent with multimodal inputs. To this end, we introduce a framework that simulates both reactive (System 1) and deliberative (System 2) cognitive processes. as illustrated in Figure 2 The core of our model
is Diffusion Transformer (DiT) backbone (Seawead et al. 2025: Gao et al. 2025 Esser et al.) 2024b. Peebles Xie 2023), pre-trained on general video tasks to acquire foundational synthesis capabilities. We enhance this base model with two critical designs. To simulate System 2 (Sec. 3.2) we employ MLLM-based agents to reason over the input context and generate high-level semantic guidance for deliberative control. To simulate System 1 (Sec. 3.3), a specialized Multimodal DiT (MMDiT) architecture (Esser et al. (0024b) fuses this guidance with reactive signals like audio. To mitigate modal conflicts within the MMDiT, introduce pseudo-last woatddo--st-ffaane fast-frame strategy that prevents the static reference image from interfering with dynamic motion generation.
Following common practices for simplicity (Lin et al. 2025b Gan et al. 2025 Kong al. 2025
our framework operates in the latent space of a pre-trained 3D VAE (Yu et al. 2023a) and trained with flow matching objective (Liu et al.) 20022). To support long-form video synthesis, the model can operate autoregressively by conditioning new segments on the final frames of the previous one (Stypulkowski et al. 2024). We omit further discussion of these standard components fo- cus on our core contributions.
3.2 AGENTIC REASONING FOR DELIBERATIVE CONTROL
Agentic Reasoning for Deliberative Guidance. To simulate the deliberative nature of "System 2", our agentic reasoning module processes multimodal inputs to generate high-level, logically coherent guidance. The module takes the character's reference image, the audio clip, and optional text
4

---

Published as a conference paper at ICLR 2026
prompt describing the desired behavior. It outputs this guidance reasoning text, the explicit textual output generated by the MLLM agents in response to our designed step-by-step guided probing, which directly conditions the synthesis model.
Multi-Step Reasoning Pipeline. As shown in Figure 2 (top-right), this guidance is generated by two-stage MLLM pipeline. First, an Analyzer MLLM receives the reference image, its caption, the audio, and the user prompt. Guided by step-by-step probing prompts, the model infers the character's speech content, emotional state, and intent. These insights are then consolidated into structured JSON object. This output is then passed to a Planner MLLM, which uses this context to devise a detailed action plan. This plan is structured the character's 's expressions and actions for single generation pass. This collaborative process yields a comprehensive motion schedule that ensures a coherent character persona across the entire video. Details and examples are provided in Appendix D
sequence of shots, with each shot defining
Framework Extensibility. The flexibility of our agentic framework also enables various design explorations. For instance, to enhance long-form coherence, the Planner can incorporate a reflective re-planning step to correct for semantic drift during synthesis. We also explored an alternative conditioning method using reasoning-infused audio latents. While we leave a full investigation of these extensions to future work, our primary experiments focus on the straightforward reasoning text approach for its proven robustness and effectiieness. We provide preliminary examples of these explorations in Appendix E to illustrate the broader potential of our framework. This chosen design enables our agent to formulate global, coherent plan, integrating deliberative "System 2" reasoning to provide thoughtful, top-down guidance absent in purely reactive "System I" methods.
3.3 REACTIVE RENDERING VIA MULTIMODAL DIFFUSION
In this section, we detail how our diffusion model synthesizes the final video. It synergistically fuses high-level guidance from the System 2 agents (represented as text) with low-level, reactive signals from the audio input.
Rethinking Reference Image Conditioning. A critical input in video avatar models is the refer- ence image, which serves two distinct purposes: first, providing an initial frame as a conditioning prefix for the generated sequence, and second, maintaining identity consistency. While the former is a necessary function, the latter (using a static reference image to enforce identity) is problematic. Prior methods. whether using dedicated networks (Hu [024a]] Tian et al.] [025c]] Zhu et al.) (023) or parameter reuse (Lin et al.) 2025b Kong et al.) 2025). typically condition the model on refer- ence image sampled from the training video. This training strategy teaches the model a spurious correlation: as illustrated in Figure 3] it learns that the reference image must appear literally within the generated sequence. We argue this is a critical artifact that severely restricts motion dynamics and conflicts with other driving signals. The root cause is that the reference image is an artificial construct, not a condition native to the video data itself. This leads to a training dilemma regarding the semantic distance of the reference image. Sampling semantically "close" reference (e.g., from the same clip) creates the static artifact, while sampling a semantically "distant" one can teach the model exhibit excessive, identity-altering variation.
Our solution is to discard the reference image entirely during training and introduce a novel guidance mechanism. As shown in Figure 2 (bottom-right), we instead probabilistically condition the model on the GT first and last frames of the video clip, both native signals, each with a dropout probability of 0.1. During inference, we repurpose this mechanism by placing the user's reference image in the last frame's position, creating a "pseudo-last- RoPE (Su et al. 2024). by assigning it a positional index corresponding to a fixed temporal distance beyond the final generated frame. This pseudo frame functions as a "carrot on stick": it guides the model toward the target identity without ever forcing it to replicate the static image, which discarded after synthesis. As our experiments show, this approach climinates training artifacts and mitigates autoregressive error, achieving a superior trade-off between motion dynamics and identity stability.
Crucially, we shift its positional encoding,
Symmetric Fusion and Modality Warm-Up. Having established data-native conditions, we now address their joint modeling. We adopt an MMDiT backbone but introduce dedicated audio branch, architecturally symmetric to the video and text branches. Instead of using pross-attention, all three modalities fused within each transformer block by concatenating their tokens and applying sin-
5

---

Published as a conference paper at ICLR 2026
Ref Model
DiT Model
Audio and Others
Ref Image
Video Latents
Earlier Methods
Remove
DiT Model
Audio
Ref Image
Video Latents
and Others
Recent Methods
Remove
DiT Model
Audio
Video Latents
and Others
Our Method
Figure 3: The Reference Conditioning Dilemma. Left: The evolution of reference conditioning methods towards simpler designs. Right: The core sampling trade-off. Sampling within-clip sures high relevance but teaches the model to produce static results. Conversely, sampling outside- clip introduces low-relevance signals, causing the model to generate identity content inconsisten- cies.
gle shared multi-head self- attention mechanism. This symmetric design enables true joint modeling, as tokens from all modalities mutually attend to one another, allowing for iterative refinement and deep semantic alignment.
However, this architecture introduces a critical training challenge: naive joint training causes the model over-rely on the dense audio signal, which washes out text guidance and disrupts the pre- trained video branch's patterns, degrading overall synthesis capability. To resolve this, propose a two-stage warm-up strategy. stage 1, we train the full three-branch model jointly, forcing the model learn optimal division of labor where the audio branch specializes in its core competen- cies (e.g., lip sync, speech mannerisms). In stage 2, we initialize the text and video branches with their original weights and the audio branch with its specialized weights from stage then fine-tune the entire model. This strategy provides each branch with a strong prior, mitigating modality conflict and preserving each input's distinct conditioning power.
Ultimately, our redesigned reference conditioning and symmetric audio fusion enable the model to effectively execute the deliberative plan of System 2, translating high-level guidance while main- taining the reactive fidelity of System
4 EXPERIMENTS
4.1 EXPERIMENTAL SETUP
Implementation and Training. Our model, built upon the MMDiT architecture, generates 120- frame clips at 24 fps. For efficiency, most ablation studies are conducted at a 480p resolution. For comparison with prior work, these clips are then upscaled to 720p to match common evaluation setting. The model underwent three-stage training process: phase on 15,000 hours of video, and a final fine-tuning stage on a 100-hour high-quality subset. Further implementation and training details are provided in the Appendix
audio branch warm-up, main
Evaluation Datasets and Metrics. To rigorously test generalization, we constructed two challeng- ing custom benchmarks: diverse single-subject set (150 cases) and multi-subject set (57 cases),

---

Published as a conference paper at ICLR 2026
Table 1: Ablation studies on our proposed framework.
Method
IQA t
ASE f
Syne-C1 1
HKC HKV 1
Ablation on Agentic Reasoning
Ours w/o Multi-Step Reasoning
4.795
3,901
3,853
0.576
157.638
Ours w/o Analyzer
4.793
3,910
4.278
0.572
148.381
Ours w/o Reasoning (System Only)
4.784
3.885
3.507
0.544
122.376
Ablation on Conditioning Modules
Ours w/ Cross-Attention
4.745
3,856
3.263
0.558
116.317
Ours w/o MM- MMM-Warup
4.752
3.866
3,993
0.549
164.080
Ours w/ Ref. Image
4.772
3.896
3.982
0.559
160.889
Ours w/o Ref. Pseudo Frame
4.682
3.878
4.141
0.564
160.986
Ours (Full Model)
4.790
3.901
4.087
0.571
168.912
Table 2: Pairwise subjective ablation study and component comparison. We report Lip-
sync Inconsistency (LSI). Motion Unnaturalness (MU), Image Distortion (ID), and an overall Good/Same/Bad preference score (GSB). Lower is better for LSI, MU, and ID.
(a) Ablation on reasoning.
(b) Comparison conditioning.
(c) GSB Score Comparisons
Method
LS1, MUL ID4 GSB
Conditioning Method LS14 MUL IDJ GSBT
GSB Comparisons
TA f Mot VQ
Ours (wil Reasoning)
0.12 0.58 0.11 0.29
OmniHuman.1
0.21
0.39
0.17
-0.23
Ours (Pall Model)
0.12 0.37
+0.29
Ours (Proposed)
0.03
0.25
0.07
4-023
Model
+0.14
both featuring a wide range of characters and complex audio. For fair comparison with prior work, also evaluate on the CelebV-HQ (Zhu et al. 2022) and CyberHost (Lin et al. 2025a) test sets.
Our evaluation protocol includes both objective and subjective metrics. We measure image quality (FID, IQA), video quality (FVD), lip-sync (Sync-C), and gesture dynamics (HKC, HKV). Crucially, to assess perceptual quality, we conducted user studies with 40 participants, using pairwise compar- isons (GSB score) and flaw identification (LSI, MU, ID) capture nuances that objective metrics miss. A detailed description of our datasets and evaluation protocols can be found in the Appendix
4.2 ABLATION STUDIES
We conduct series of ablation studies to validate the contributions our proposed components, using custom single-subject test set of 150 video clips. Our analysis systematically isolates the impact of two key elements: (1) the agentic reasoning module and (2) our proposed conditioning architecture. For comprehensive assessment, employ both quantitative metrics and subjective user studies to evaluate performance, perceptual quality, and user preference.
Effectiveness of Agentic Reasoning. We analyze Agentic Reasoning module by progressively ablating its components: first removing multi-step reasoning, then the Analyzer, and finally the full module for a "System 1 only" baseline. As shown in Table standard metries like IQA and Sync-C show minimal variation. This is expected, as these metrics measure low-level fidelity, are largely saturated, and are insensitive higher-level semanties. A more telling trend emerges from Hand Keypoint Variance (HKV), which progressively decreases as reasoning is removed, indicating that the generated motion becomes more static and less expressive. This directly demonstrates the value of our reasoning module.
To assess the module's semantic impact, we conducted subjective evaluations. As shown in Ta- ble our full model shows a substantial advantage. Agentic reasoning yields 29% gain in GSB and a significant reduction in perceived motion unnaturalness (MU), while maintaining strong lip- sync (LSI) and image quality (ID). These findings confirm reasoning module enhances motion plausibility and naturalness, qualities not captured by standard objective metrics. This is further corroborated by MLLM-based analysis (Appendix D.2) confirming method's superior contextual coherence. We strongly encourage viewing the supplementary material and project page, which showcase compelling visual results unattainable with existing "System 1" models.

---

Published as a conference paper at ICLR 2026
Table 3: Comparison with existing methods on multi-person animation. We report quantitative metrics and pairwise subjective evaluation results, including Driving Accuracy (DA), Lip-syne In- consistency (LSI), Motion Unnaturalness (MU) and an overall user preference score derived from Good/Same/Bad (GSB) evaluation.
Subjective Evaluation
Quantitative Metries
Method
DA
LSL
MUJ
GSBt
IQAt
ASE
Sync-DI
HKC1
HKV1
InterActHuman
4.574
3.643
8.163
0.553
103.91
Ours w/o Reasoning
0.88
0.13
0.63
-0.26
4.576
3.631
7.541
0.611
138.43
Ours (Full Model)
0.94
0.04
0.12
+0.26
4.529
3.653
6,904
0.614
158.36
Table 4: Quantitative comparison with audio-conditioned animation baselines. (Left) Portrait animation the CelebV-HQ test set. (Right) Full-body animation on the CyberHost test set.
Method
IQA
ASE
Sync-C't
FID
FVD
Method
IQA
ASB
Sync-C
FID 1
FVD
HKC
HKV
SadTalker
3.843
Skyreel-AI
3,889
2.525
2,983
69.619
70.678
0.786
28.840
Hallo
5,505
2.262
4.130
35.961
53.992
VantasyTalkiing
3,892
2.738
3.548
52,332
47.052
0.838
18.845
Loopy
EcheMimic
3,307
2.128
3.136
35.373
54.715
Omni-Aattrr
3,871
43.998
0.795
56.574
Hallo-3
3.451
2.257
3.933
38.481
42.125
MultiTalk
3.822
2.681
6.868
37.308
32.783
0.817
62.753
OmniHumann-1
3,875
2,656
5,199
31.435
46.393
OmniHuman
4.142
3.024
7.443
31.641
27.031
0.898
47.561
2.663
5.083
Ours
4.144
3.030
7,243
31.160
27.642
0.875
72.113
Effectiveness of Proposed Conditioning Modules. We also ablate our core architectural designs in Tables and keeping the agentic reasoning module fixed. We ablate our core architectural de- signs by evaluating three key changes: replacing our MM-Atteention-basee audio conditioning with standard cross-attention, removing the MM- MMM-aarmup stage, and substituting our pseudo-last-fran strategy with conventional reference image attention. Our full model consistently leads in objective metries, with superior HKC/HKV scores highlighting enhanced motion dynamics. Subjectively, our method also significantly outperforms baseline using cross-attention for both audio and refer- ence image conditioning (akin to OmniHuman-1 (Lin et al.) 20256),, ), showing clear advantages in unique conditioning architecture as a robust foundation for executing agentic plans. Further visual analysis in the appendix demonstrates how PLF maintains identity consistency during dynamic motion.
lip-sync (LSI), motion naturalness (MU), and visual quality (ID). These results validate
CoxtiCondiiioning Fidelity. As a supplementary experiment, we verified that our multi-modal training does not degrade text-following capabilities. Under text-only conditioning, a user study (Ta- 2le confirms our model achieves on-par text alignment (TA) with the base model, while demon- strating significantly improved motion naturalness (MU) and visual quality (ID). This validates that our approach enhances generation quality without sacrificing core contioondiiinnin fidelity.
4.3 FURTHER EXPLORATION ON APPLICATIONS
Generalization to Diverse and Multi-Person Scenarios. Our model demonstrates robust gen- eralization, enabled by a dual-system framework whose agentic reasoning provides context-aware motion guidance for non-human subjects and interprets conversational turn-taking (Figure 4. To further showcase this extensibility, we conducted a preliminary study on multi-person animation by incorporating predicted speaker mask to guide the system (details in Appendix B). In focused comparison ()) our full model significantly outperforms baselines like InterActHuman (Wang al. 2025c) and ablation without reasoning. The observed improvements in gesture dynamics (HKC/HKV), lip-sync (Sync-D), and driving accuracy (DA) primarily validate the effectiveness of reasoning component in enhancing coordination in such complex settings. These results under- score our method's 's broad applicability and potential for creating interactive avatars (see supplemen- tary material for video results).
8

---

Published as a conference paper at ICLR 2026
Supe t
Lafen
T3
Lieen
TXX
Figure 4: Generalization and Multi-Person Results. Our model generalizes to diverse non- human subjects (top rows), correctly interprets conversational turn-taking (fourth row), and generates coor- dinated behavior in multi-person scenes (bottom rows).
Figure 5: Subjective User Preference Study. We present results from two evaluation settings: (Left) best-choice selection task comparing our method against academic baselines, and (Right) GSB pairwise comparison against leading proprietary models.
Method
Top-1 (%)
Methad Preferred
Method Net Preiened
User
Study: Ours Proprietary Models
Skyreel-Al
5%
PantasyTalking
8%
C5-A (EZE)
OmniAvata Avatar
14%
MultiTalk
18%
CS-C (25)
OmniHuman-1
22%
Ours
33%
(a) Best-Choice Selection.
(b) GSB against leading proprictary models.
4.4 COMPARISON AMONG RECENT METHODS
Comparisons with State-of-the-Art Methods. We conduct comprehensive evaluation against leading academic baselines across two scenarios: portrait and full-body generation. For por- traits, we compare on the CelebV-HO test set against specialized and DiT-based methods, includ- ing SadTalker (Zhang et al. 2023
FchoMimic (Chen et al. 2024c). Hallo (Xu et al. 2024a

---

Published as a conference paper at ICLR 2026
2024), Loopy (Jiag al. 2025 and OmniHuman-1 1 (Lin al. 2025b). For the more challenging bull-body synthesis task, we evaluate on the CyberHost test set against recent DiT
Hallo3 (Cui et al.
models like Skyreels-A1 (Fei et al., 2025).
(Wang et al. 2025b), OmniAvatar GGnn 2025 and OmniHuman-1.
FantasyTallke
et al. 2025). MultiTalk (Kong et al.
Quantitative results (Table show our method consistently ranking among the top two. While on par with OmniHuman-1 in portraits, a setting where limited motion challenges objective metrics, our advantages become pronounced in the full-body setting (see also Table 2b and Figure (2). In this setting, our model excels at generating dynamic, large-scale movements (high HKV) while preserv- ing local detail (competitive HKC). To assess perceptual quality, we conducted user studies against a top academic method (Figure Ba) and four leading, anonymized commercial systems (Figure [5b). In user studies, human evaluators consistently and significantly preferred our results overall, pref- erence we attribute to high-level qualities like contextual coherence that objective metrics fail to capture but are critical for perceptual realism.
5 CONCLUSION
Inspired by the dual-system theory of human cognition, we introduce a new paradigm for video avatars. We argue that existing methods, by simulating only reactive "System I" thinking, fail to align motion with high-level intent, resulting in behaviors that lack contextual appropriateness and logical coherence. To address this, we propose a novel framework that models deliberative "System 2" processes via two innovations: MLLM-based agents for semantic planning and a specialized MMDiT for high-fidelity synthesis. Experimental results validate that our approach generates more expressive and logically coherent motions in both single- and multi-person scenarios, achieving leading performance both subjective and objective metrics across multiple benchmarks. We hope this cognitive agency perspective offers the community a promising path toward creating video avatars that are not just visually realistic, but truly believable.
Ethics Statement. Our core contribution is a novel paradigm for video avatar generation. By simulating dual-system cognitive framework, our model achieves new level of expressive capa- bility and logical coherence, moving beyond the limitations of singlc-process process generation. While this advancement opens exciting possibilities for creative applications like Al-driven film production, we recognize the potential for misuse associated with highly realistic avatar technologies. To ad- dress these ethical concerns, we advocate for a robust framework of responsible deployment. Given its potentially powerful capabilities, this model requires strict access control to prevent potential abuse. Although current results may still bear subtle artifacts that can serve as an informal deterrent, proactive safeguards are essential. We strongly recommend the following measures: (1) applying prominent, visible watermarks to all generated content to clearly label as Al-generated; (2) im- plementing filtering algorithms to reject inappropriate malicious input prompts and to review output content; and (3) embedding traceable, invisible watermarks to ensure accountability and aid in source identification if misuse occurs. By integrating these safety protocols, that our technology fosters creativity while minimizing the risks of malicious applications such as fraud disinformation.
help ensure
Reproducibility Statement. To ensure the full reproducibility of our work, we provide compre- hensive details regarding our methodology, data, implementation, and evaluation procedures across the main paper and its appendices. Specifically, we detail: (I) the core architecture of our agen- tic system and the MMDiT model in Section 3 (2) complete implementation details, including all hyperparameters and the multi-stage training procedure, in Appendix B (3) our data curation pipeline, with specific filtering tools (e.g., PySceneDetect, Q-Align) and a statistical analysis of the dataset, also in Appendix B and (4) our agentic system and evaluation framework, which specifies the MLLM models used (miniCPM-o and Seed-1.5- instruction prompts, and the complete set of objective and subjective evaluation metrics in Appendices D and C Finally, while we provide these details to ensure technical reproducibility, we reiterate our ethical recommendation that for re- sponsible deployment, access to the trained models should be managed in a controlled environment to monitor usage and prevent potential misuse.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, llge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv preprint (eXXxx:2333087770 2023.
Tim Brooks, Aleksander Holynski, and Alexei Efros. Instructpix2pix: Learning to follow image editing instructions. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 18392-18402, 2023.
Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, al. Video generation models as world simulators. OpenAl Blog, 188);1 2024.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 13:1877-1901, 2020.
Di Chang, Yichun Shi, Quankai Gao, Jessica Fu, Hongyi Xu, Guoxian Song, Qing Yan, Yizhe Zhu, Xiao Yang. and Mohammad Soleymani. Magicpose: Realistic human poses and facial expressions retargeting with identity-awan diffusion. arXiv preprint arXiv:2311.
2023.
Junsong Chen, Yue Wu, Simian Luo, Enze Xie, Sayak Paul, Ping Luo, Hang Zhao, and Zhenguo Li. Pixart- |delta}: :dxltt-" Fast and controllable image generation with latent consistency models. arXiv preprint arXiv:2401.00222 2024a.
Shoufa Chen, Mengmeng Xu, Jiawei Ren, Yuren Cong, Sen He, Yanping Xie, Animesh Sinha, Ping Luo, Tao Xiang, and Juan-Manuel Perez-Rua. Gentron: Diffusion transformers for image and video generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1441-6451, 2024b.
Zhiyuan Chen, Jiajiong Cao, Zhiquan Chen, Yuming Li, and Chenguang Ma. Echomimic: Life- like audio-driven portrait animations through editable landmark conditions. arXiv preprint arXiv:2407.08116, 2024c.
Joon Son Chung and Andrew Zisserman. Out time: automated lip syne the wild. Computer Vision-ACCV 2016 Workshops: ACCV 2016 International Workshops, Taipei, Taiwan, November 20-24, 2016, Revised Selected Papers, Part 11 13, pp. 251-263. Springer, 2017.
Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capa- bilitics. arXiv preprint arXiv:2507.06661, 2025.
Jiahao Cui, Hui Li, Yun Zhan, Hanlin Shang, Kaihui Cheng, Yuqi Ma, Shan Mu, Hang Zhou, Jingdong Wang, and Siyu Zhu. Hallo3: Highly dynamic and realistic portrait image animation with diffusion transformer networks. arXiv preprint arXiv:2412.00733 2024.
Xiang Deng, Youxin Pang, Xiaochen Zhao, Chao Xu, Lizhen Wang, Hongjiang Xiao, Shi Yan, Hongwen Zhang, and Yebin Liu. Stereo-talker: Audio-driven 3d human synthesis with prior- guided mixture-of-experts. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025.
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. In Forty-first international conference on machine learning. 2024a.
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. In Forty-first International Conference on Machine Learning, 2024b.
11

---

Published as a conference paper at ICLR 2026
Zhengcong Fei, Debang Li, Di Qiu, Jiahua Wang, Yikun Dou, Rui Wang, Jingtao Xu, Mingyuan Fan, Guibin Chen, Yang Li, et al. Skyreels-a2: Compose anything in video diffusion transformers. arXiv preprint 0224: 2025.
Qijun Gan, Ruizi Yang, Jianke Zhu, Shaofei Xue, and Steven Hoi. Omniavatar: Efficient audio- driven avatar video generation with adaptive body animation. arXiv preprint arXiv:2506.1888 2025.
Yu Gao, Haoyuan Guo, Tuyen Hoang, Weilin Huang, Lu Jiang, Fangyuan Kong, Huixia Li, Jiashi Li, Liang Li, Xiaojie Li, et al. Seedance 1.0: Exploring the boundaries of video generation models. arXiv preprini 2rXiv:2506.091 13, 2025.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprini urXiv:2501.12944 2025a.
Dong Guo, Faming Wu, Feida Zhu, Fuxing Leng, Guang Shi, Haobin Chen, Haoqi Fan, Jian Wang. Jianyu Jiang, Jiawei Wang, et al. Seedl. 5-vl technical report. arXiv preprint 1rXiv:2505.07062 2025b.
Yuwei Guo, Ceyuan Yang, Anyi Rao, Zhengyang Liang. Yaohui Wang, Yu Qiao, Maneesh Agrawala, Dahua Lin, and Bo Dai. Animatediff: Animate your personalized text-to-image diffu- sion models without specific tuning. arXiv preprint urXiv:2307.04725. 2023.
Tianyu He, Junliang Guo, Runyi Yu, Yuchi Wang, Jialiang Zhu, Kaikai An, Leyi Li, Xu Tan, Chunyu Wang, Han Hu, et al. Gaia: Zero-shot talking avatar generation. arXiv preprint arXIv:2311.15230. 2023.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.
Steven Hogue, Chenxu Zhang, Hamza Daruger, Yapeng Tian, and Xiaohu Guo. Diffted: One-shot audio-driven ted talk video generation with diffusion- co-speech gestures. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 1922-1931, 2024.
Sirui Hong, Mingchen Zhuge, Jonathan Chen, Xiawu Zheng, Yuheng Cheng, Jinlin Wang, Ceyao Zhang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, et al. Metagpt: Meta programming for multi-agent collaborative framework. In The Twelfth International Conference on Learning Representations, 2023.
Li Hu. Animate anyone: Consistent and controllable image-to-vider synthesis for character anima- tion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8153-8163. 2024a.
Li Hu. Animate anyone: Consistent and controllable image-to-video synthesis for character anima- tion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8153-8163, 2024b.
Panwen Hu, Jin Jiang, Jianqi Chen, Mingfei Han, Shengcai Liao, Xiaojun Chang, and Xiaodan Liang. Storyagent: Customized storytelling video generation via multi-agent collaboration. arXiv preprint arXiv:2411 1.04925, 2024a.
Shengding Hu, Yuge Tu, Xu Han, Chaoqun He, Ganqu Cui, Xiang Long, Zhi Zheng. Yewei Fang, Yuxiang Huang, Weilin Zhao, al. Minicpm: Unveiling the potential of small language models with scalable training strategies. arXiv preprint arXiv:2404.06:3 2024b.
Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Os- trow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024.
12

---

Published as a conference paper at ICLR 2026
Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai ol system card. arXiv preprint arXiv:2412.16722 2024.
Jianwen Jiang, Gaojie Lin, Zhengkun Rong. Chao Liang. Yongming Zhu, Jiaqi Yang, and Tianyun Zhong. Mobileportrait: Real time one-shot neural head avatars on mobile devices, arXiv preprint arXiv:2407.057/2, 2024.
Jianwen Jiang. Chao Liang, Jiaqi Yang, Gaojic Lin, Tanyun Zhong, and Yanbo Zheng. Loopy: Taming audio-driven portrait avatar with long-term motion dependency. In The Thirteenth In- ternational Conference on Learning Representations, 2025. URL 1dweeMAYBeeeI
Daniel Kahneman. Thinking, fast and slow. macmillan, 2011.
Daniel Kahneman and Amos Tversky. Prospect theory: An analysis of decision under risk. In Handbook of the fundamentals of financial decision making: Part I, pp. 99-127. World Scientific, 2013.
Shyam Sundar Kannan, Vishnunandan LN Venkatesh, and Byung-Cheol Min. Smart-llm: Smart multi-agent robot task planning using large language models. In 2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 31400-2147, IEEE, 2024.
Johanna Karras, Aleksander Holynski, Ting-Chun Wang. and Ira Kemelmacher-SShizemma. Dream- pose: Fashion image-to-video synthesis via stable diffusion. 2023 IEEE/CVF International Conference Computer Vision (ICCV), pp. 22623-22626 IEEE, 2023.
Jing Yu Koh, Daniel Fried, and Russ Salakhutdinov. Generating images with multimodal language models. Advances in Neural Information Processing Systems, 36:21487-21506, 2023.
Weijie Kong, Qi Tian, Zijian Zhang, Rox Min, Zuozhuo Dai, Jin Zhou, Jiangfeng Xiong, Xin Li, Bo Wu, Jianwei Zhang, et Hunyuanvideo: A systematic framework for large video generative models. arXiv preprint arXiv:2412.03360 2024.
Zhe Kong, Feng Gao, Yong Zhang, Zhuoliang Kang. Xiaoming Wei, Xunliang Cai, Guanying Chen, and Wenhan Luo. Let them talk: Audio-driven multi-person conversational video generation. 2025.
arXiv preprint arXiv:2505.
Yunxin Li, Haoyuan Shi, Baotian Hu, Longyue Wang, Jiashun Zhu, Jinyi Xu, Zhen Zhao, and Min Zhang. Anim-director: A large multimodal model powered agent for controllable animation video generation. SIGGRAPH Asia 2024 Conference Pupers, pp. 1-11, 2024.
Chao Liang, Jianwen Jiang, Wang Liao, Jiaqi Yang, Weihong Zeng, Han Liang, et al. Alignhuman: Improving motion and fidelity via timestep-segment preference optimization for audio-driven hu- man animation. arXiv preprint arXiv:2506. 144, 2025.
Jinhua Liang, Huan Zhang, Haohe Liu, Yin Cao, Qiuqiang Kong, Xubo Liu, Wenwu Wang, Mark D Plumbley, Huy Phan, and Emmanouil Benetos. Waveraft: Audio editing and generation with large language models. arXiv preprint 240i::220330952.,, 2024.
Gaojie Lin, Jianwen Jiang, Chao Liang, Tianyun Zhong, Jiaqi Yang. Zerong Zheng, and Yanbo Zheng. Cyberhost: A one-stage diffusion framework for audio-driven talking body generation. In The Thirteenth International Conference on Learning Representations, 2025a. URL https:
Gaojie Lin, Jianwen Jiang, Jiaqi Yang, Zerong Zheng, and Chao Liang. Omnihuman-l: Re- thinking the scaling-up of one-stage conditioned human animation models. arXiv preprint arXiv:2502.01061, 2025b.
Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. arXiv preprint 2rXiv:2209.03003 2022.
13

---

Published as a conference paper at ICLR 2026
Guoqing Ma, Haoyang Huang, Kun Yan, Liangyu Chen, Nan Duan, Shengming Yin, Changyi Wan, Ranchen Ming, Xiaoniu Song, Xing Chen, et al. Step-video-12 12v technical report: The practice, challenges, and future of video foundation model. arXiv preprint arXh::2502.1022: 2025.
Willi Menapace, Aliaksandr Siarohin, Ivan Skorokhodov, Ekaterina Deyneka, Tsai-Shien Chen, Anil Kag. Yuwei Fang, Aleksei Stoliar, Elisa Ricci, Jian Ren, al. Snap video: Scaled spatiotem- poral transformers for text-to-video synthesis. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp- 7038-7048, 2024.
Rang Meng. Xingyu Zhang, Yuming Li, and Chenguang Ma. Echomimiev2: Towards striking, simplified, and semi-body human animation. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 5489-5499 2025.
OpenAl. Chatgpt (gpt-4o version). https:/ cwat. httphhaaa.ppa.ccc...... comenaa..cmm//// 2024.
Long Ouyang. Jeffrey Wu, Xu Jiang. Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to fol- low instructions with human feedback. Advances in neural information processing systems, 35:
17730-27744 2022.
Xichen Pan, Satya Narayan Shukla, Aashu Singh, Zhuokai Zhao, Shlok Kumar Mishra, Jialiang Wang, Zhiyang Xu, Jiuhai Chen, Kunpeng Li, Felix Juefei-Xu, et al. Transfer between modalities with metaqueries. arXiv preprint arXiv:2504.066 06256, 2025.
Joon Sung Park. Joseph O'Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein. Generative agents: Interactive simulacra of human behavior. Proceedings of the 36th annual acm symposium user interface software and technology, pp. 1-22, 2023.
William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4195 4205, 2023.
Adam Polyak, Amit Zohar, Andrew Brown, Andros Tjandra, Animesh Sinha, Ann Lee, Apoorv Vyas, Bowen Shi, Chih-Yas Ma, Ching- Yao Chuang, et al. Movie gen: A cast of media founda- tion models. arXiv preprint Xrxiv:2410.1372 2024.
Di Qiu, Zhengcong Fei, Rui Wang, Jialin Bai, Changqian Yu, Mingyuan Fan, Guibin Chen, and Xiang Wen. Skyreels-al: Expressive portrait animation in video diffusion transformers. arXiv preprint 2rX1:22502.10841 2025.
Leigang Qu, Shengqiong Wu, Hao Fei, Liqiang Nic, and Tat-Seng Chua. Layoutlmm-Ii: Eliciting layout guidance from 1lm for text-to-image generation. In Proceedings of the 31st ACM Interna- tional Conference on Multimedia, pp. 643-654, 2023.
Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. In International conference on ma- chine learning, pp. 28492-200 28518.8118..2 PMLR, 2023.
Timo Schick, Jane Dwivedi- Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. Advances in Neural Information Processing Systems, 36:68539- 68551, 2023.
Team Seawead, Ceyuan Yang, Zhijie Lin, Yang Zhao, Shanchuan Lin, Zhibei Ma, Haoyuan Guo, Hao Chen, Lu Qi, Sen Wang, et al. Seaweed-7b: Cost-effective training of video generation foundation model. arXiv preprint wrXiv:2504.08685 2025.
Ruizhi Shao, Youxin Pang, Zerong Zheng, Jingxiang Sun, and Yebin Liu. Human4dit: Free-view human video generation with 4d diffusion transformer. arXiv preprint arXiv:2405. 11.400577400 /7405, 2024.
Weijia Shi, Xiaochuang Han, Chunting Zhou, Weixin Liang, Xi Victoria Lin, Luke Zettlemoyer, and Lili Yu. Lmfusion: Adapting pretrained language models for multimodal generation. arXiv preprint 2xX 2024.
14

---

Published as a conference paper at ICLR 2026
Aliaksandr Siarohin, Stéphane Lathuilière, Sergey Tulyakov, Elisa Ricci, and Nicu Sebe. First order motion model for image animation. Advances in neural information processing systems, 32, 2019.
Aliaksandr Siarohin, Oliver J Woodford, Jian Ren, Menglei Chai, and Sergey Tulyakov. Motion rep- resentations for articulated animation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13653-13633 2021.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502: 2020.
Michal Stypulkowski, Konstantinos Vougioukas, Sen He, Maciej Zieba, Stavros Petridis, and Maja Pantic. Diffused heads: Diffusion models beat gans on talking-fate generation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 5091-5100, 2024.
Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: En- hanced transformer with rotary position embedding. Neurocomputingg 568: 127063, 2024.
Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.1 1805, 2023.
1805,
Zachary Teed and Jia Deng. Raft: Recurrent all-pairs field transforms for optical flow. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part 11 16. pp. 402-419. Springer, 2020.
Linrui Tian, Siqi Hu, Qi Wang, Bang Zhang, and Liefeng Bo. Emo2: End-effector guided audio- driven avatar video generation. arXiv preprint (tfi:2500..0877, 2025a.
Linrui Tian, Siqi Hu, Qi Wang. Bang Zhang, and Liefeng Bo. Emo2: End-effector guided audio- driven avatar video generation. arXiv preprint arXiv:2501. 10687, 2025b.
Linrui Tian, Qi Wang, Bang Zhang, and Liefeng Bo. Emo: Emote portrait alive generating ex- pressive portrait videos with audio2video diffusion model under weak conditions. In European Conference on Computer Vision, pp. 244-260. Springer, 2025c.
Shuyuan Tu, Qi Dai, Zihao Zhang, Sicheng Xie, Zhi-Qi Cheng, Chong Luo, Xintong Han, Zuxuan Wu, and Yu-Gang Jiang. Motionfollower: Editing video motion via lightweight score-guided diffusion. arXiv preprint arXiv:2405.20332: 2024.
Thomas Unterthiner, Sjoerd van Steenkiste, Karol Kurach, Raphaël Marinier, Marcin Michalski, and Sylvain Gelly. Fvd: A new metric for video generation. https://openrevieen Forum ? -eeiiiidddyllll ryggEULygUUtt CN, 2019.
Ang Wang, Baole Ai, Bin Wen, Chaojie Mao, Chen-Wei Xie, Di Chen, Feiwu Yu, Haiming Zhao, Jianxiao Yang, Jianyuan Zeng, Jiayu Wang. Jingfeng Zhang, Jingren Zhou, Jinkai Wang, Jixuan Chen, Kai Zhu, Kang Zhao, Keyu Yan, Lianghua Huang, Mengyang Feng, Ningyi Zhang, Pan- deng Li. Pingyu Wu, Ruihang Chu, Ruili Feng, Shiwei Zhang, Siyang Sun, Tao Fang. Tianxing Wang, Tianyi Gui, Tingyu Weng. Tong Shen, Wei Lin, Wei Wang, Wei Wang, Wenmeng Zhou, Wente Wang, Wenting Shen, Wenyuan Yu, Xianzhong Shi, Xiaoming Huang, Xin Xu, Yan Kou, Yangyu Lv, Yifei Li, Yijing Liu, Yiming Wang, Yingya Zhang, Yitong Huang, Yong Li. You Wu, Yu Liu, Yulin Pan, Yun Zheng, Yuntao Hong, Yupeng Shi, Yutong Feng, Zeyinzi Jiang, Zhen Han, Zhi-Fan Wu, and Ziyu Liu. Wan: Open and advanced acrge-scale video generative models. arXiv preprint ar 2503:253320:3
2025a.
Cong Wang, Kuan Tian, Jun Zhang, Yonghang Guan, Feng Luo, Fei Shen, Zhiwei Jiang, Qing Gu, Xiao Han, and Wei Yang. V-express: Conditional dropout for progressive training of portrait video generation. arXiv preprint arXiv:2406.0055/ 02511, 2024a.
Guanzhi Wang. Yuqi Xie. Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models. arXiv preprint arXiv:2305. 162.) 2023a.
Jiuniu Wang. Hangjic Yuan, Dayou Chen, Yingya Zhang, Xiang Wang, and Shiwei Zhang, Mod- elscope text-to-video technical report. arXiv preprint arXiv:2308.06:: 06571, 2023b.
15

---

Published as a conference paper at ICLR 2026
Mengchao Wang. Qiang Wang, Fan Jiang, Yaqi Fan, Yunpeng Zhang, Yonggang Qi. Kun Zhao, and Mu Xu. Fantasytalking: Realistic talking portrait generation via coherent motion synthesis. arXiv preprint arXiv:2504.0488 2025b.
Ting-Chun Chun Wang, Arun Mallya, and Ming- Yu Liu. One-shot free-view neural talking- heang synthesis for video conferencing. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10039-10049, 2021.
Xiang Wang. Shiwei Zhang. Changxin Gao, Jiayu Wang, Xiaoqiang Zhou, Yingya Zhang. Luxin Yan, and Nong Sang. Unianimate: Taming unified video diffusion models for consistent human image animation. arXiv preprint arXiv:2406.011 188, 2024b.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations.
Zhenzhi Wang, Jiaqi Yang, Jianwen Jiang, Chao Liang, Gaojie Lin, Zerong Zheng, Ceyuan Yang, and Dahua Lin. Interacthuman: Multi-concept human animation with ditions. arXiv preprint arXiv:2506.09995 2025c.
audio con-
Huawei Wei, Zejun Yang, and Zhisheng Wang. Aniportrait: Audio-driven synthesis of photorealistic portrait animation. arXiv preprint arXiv:2403.17694 2024.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 15:24824-24831 2022.
Chenfei Wu, Jiahao Li, Jingren Zhou, Junyang Lin, Kaiyuan Gao, Kun Yan, Sheng-ming Yin, Shuai Bai, Xiao Xu, Yilei Chen, et al. Qwen-image technical report. arXiv preprint N.Xiv:2508.02324 2025a.
Haoning Wu, Zicheng Zhang, Weixia Zhang, Chaofeng Chen, Liang Liao, Chunyi Li, Yixuan Gao, Annan Wang, Erli Zhang, Wenxiu Sun, al. Q-align: Teaching Imms for visual scoring via discrete text-defined levels. arXiv preprint arXiv:2312.11090 2023.
Shengqiong Wu, Hao Fei, Leigang Qu, Wei Ji, and Tat-Seng Chua. Next-gpt: Any-to-any multi- modal 11m. In Forty-first International Conference on Machine Learning, 2024.
Weijia Wu, Zeyu Zhu, and Mike Zheng Shou. Automated movie generation via multi-agent cot planning, arXiv preprint 2xXiv:2503300314 2025b.
Mingwang Xu, Hui Li, Qingkun Su, Hanlin Shang, Liwei Zhang, Ce Liu, Jingdong Wang, Luc Van Gool, Yao Yao, and Siyu Zhu. Hallo: Hierarchical audio-drived visual synthesis for portrait image animation. arXiv preprint 24xx:24066088/0 2024a.
Sicheng Xu, Guojun Chen, Yu-Xi Guo, Jiaolong Yang. Chong Li, Zhenyu Zang. Yizhong Zhang, Xin Tong, and Baining Guo. Vasa-1: Lifelike audio-driven talking faces generated in real time. arXiv preprint arXiv:2404. /0667, 2024b.
Zhongcong Xu, Jianfeng Zhang. Jun Hao Liew, Hanshu Yan, Jia-Wel Liu, Chenxu Zhang, Jiashi Feng, and Mike Zheng Shou. Magicanimate: Temporally consistent human image animation using diffusion model. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 1481-1490, 2024c.
Zhuoyi Yang. Jiayan Teng. Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiaohan Zhang, Guanyu Feng, et al. Cogvideox: Text-to-video diffusion models with an expert transformer. arXiv preprint Xiix:2408.06077 2024.
Shunyu Yao. Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimban. Tree of thoughts: Deliberate problem solving with large language models. Ad- vances in neural information processing systems, 36:1 1809-118822 2023.
16

---

Published as a conference paper at ICLR 2026
Lijun Yu, Jos Lezama, Nitesh B Gundavarapu, Luca Versari, Kihyuk Sohn, David Minnen, Yong Cheng, Vighnesh Birodkar, Agrim Gupta, Xiuye Gu, et al. Language model beats diffusion- tokenizer is key to visual generation. arXiv preprint arXiv:2310.05737. 2023a.
Lili Yu, Bowen Shi, Ramakanth Pasunuru, Benjamin Muller, Olga Golovneva, Tianlu Wang, Arun Babu, Binh Tang, Brian Karrer, Shelly Sheynin, et al. Scaling autoregressive multi-modal models: Pretraining and instruction tuning. arXiv preprint (rXiv:2309.0259/ 2023b.
Zhengqing Yuan, Yixin Liu, Yihan Cao, Weixiang Sun, Haolong Jia, Ruoxi Chen, Zhaoxu Li, Bin Lin, Li Yuan, Lifang He, al. Mora: Enabling generalist video generation via multi-agent framework. arXiv preprint arXiv:2403. 13248, 2024.
Wenxuan Zhang, Xiaodong Cun, Xuan Wang, Yong Zhang, Xi Shen, Yu Guo, Ying Shan, and Fei Wang- Sadtalker: Learning realistic 3d motion coefficients for stylized audio-driven single image talking face animation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patlern Recognition, pp. 8652-8661, 2023.
Yuang Zhang, Jiaxi Gu, Li-Wen Wang. Han Wang, Junqi Cheng, Yuefeng Zhu, and Fangyuan Zou. Mimicmotion: High-quality human motion video generation with confidence-awe pose guid- ance. arXiv preprint arXiv:2406. 19680, 2024.
Jian Zhao and Hui Zhang. Thin-plate spline motion model for image animation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 3657-3666, 2022.
Zangwei Zheng, Xiangyu Peng, Tianji Yang, Chenhui Shen, Shenggui Li, Hongxin Liu, Yukun Zhou, Tianyi Li, and Yang You. Open-sora: Democratizing efficient video production for all. arXiv preprint arXiv:2412.20400 2024.
Tianyun Zhong, Chao Liang, Jianwen Jiang, Gaojie Lin, Jiaqi Yang, and Zhou Zhao. Fada: Fast diffusion avatar synthesis with mixed-supervised multi-cfg distillation. arXiv preprint
2rXww:2412.16915, 2024.
Hao Zhu, Wayne Wu, Wentao Zhu, Liming Jiang. Siwei Tang. Li Zhang, Ziwei Liu, and Chen Change Loy. Celebv-hq: A large-scale video facial attributes dataset. In European con- ference on computer vision, pp. 650-667. Springer, 2022.
Luyang Zhu, Dawei Yang, Tyler Zhu, Fitsum Reda, William Chan, Chitwan Saharia, Mohammad Norouzi, and Ira Sereemmachhe-SSlzzemaaee Tryondiffusion: A tale of two unets. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 46064461111 2023.
Shaobin Zhuang, Kunchang Li, Xinyuan Chen, Yaohui Wang, Ziwei Liu, Yu Qiao, and Yali Wang. Vlogger: Make your dream vlog. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8806-8817, 2024.
17

---

Published as a conference paper at ICLR 2026
A APPENDIX
This appendix provides comprehensive supplementary material to the main paper, organized as fol- lows:
Section B (Implementation and Training): Details our model architecture, multi-person extension, training procedures, and data curation pipeline.
Section C (Evaluation Details): Describes the novel datasets and the multi-faceted evalu- ation protocol, covering both objective and subjective metrics.
Section D (Agentic System Analysis): Offers deeper analysis of the reasoning system, with process demonstrations, MLLM specifications, and the full prompt for our MLLM- based evaluation.
Section E (Qualitative Results): Presents visual comparisons against baselines and de- tailed ablation studies for key components (e.g.. PLF, MM- Warmup, reflection).
Section F (System Latency): Discusses the inference latency of the agentic system, fram- ing it as necessary trade-off for achieving higher-level coherence.
Section G (Statements): Concludes with statements on ethical considerations, Large Language Models (LLMs) in preparing the manuscript, and our efforts to ensure reproducibility.
of
H Section (Limitations Provides a discussion of the current limitations of our approach, including analysis of failure cases, and outlines promising avenues for future work.
B IMPLEMENTATION AND TRAINING DETAILS
Implementation Details. Our model is based on the Multi-modal Diffusion Transformer (MMDIT) architecture, sharing a similar design with recent works (Lin et al., 2025b] (Sawead et al. 2025 Gao et al.) 2025 Kong et al., 2024), and was pre-trained on a large-scale text-video dataset. The architecture processes separate branches for video, text, and audio, which are subsequently fused using multi-modal attention layers. We introduce the audio branch by duplicating the praitre text branch for parameter initialization, with its input derived from audio features extracted by Whisper (Radford et al. 2023) encoder. To maintain computational efficiency, each visual latent performs multi-modal attention with only its five temporally nearest audio and text tokens. Given that the audio and text token sequences are significantly shorter than the visual token sequence, this architectural extension introduces negligible computational overhead.
For most experiments, the model generates 120-frame clips at 24 fps and 480p resolution. For visualization and comparison purposes, these outputs are upscaled to 720p using a separate, identi- cally architected super-resolution model. Longer videos are produced autoregressively; specifically, for our pseudo-last-fras ditioning frames for the subsequent clip, with RoPE position shift of 30. A detailed analysis of this shift value is provided in the Appendix (see Figure 99). For optimization, we employed the Adam W optimizer with learning rate of 5e-5, global batch size of 256, and gradient clipping at norm of 1.0. The training process was conducted in three phases: an audio branch warm-up 1~~18 GPU-hours, measured on an A A00 equivalent), main training phase (~433 GPU-hours), and final fine-tuning phase on high-quality data (~66 GPU-hours). The MLLM Agent is designed to be plug- gable, thanks to a flexible framework architecture that supports both local and API-based models. For this work, we selected miniCPM-o (Hu et al., 2024b) as the Analyser and et al. 2025b) the Planner, based on their top-tier performance on relevant leaderboards.
strategy, the final five frames of a generated clip serve as the initial con-
(Guo)
Implementation of Multi-Person Scenarios. To enable multi-person animation, we extend our model with two key modifications. First, we condition the synthesis on a speaker-specific mask. This mask directs the injection of audio features exclusively the corresponding masked regions during the multimodal attention process. Following the approach of InterActHuman (Wang et al. 2025c), employ a lightweight, plug-and-play predictor to dynamically generate these masks. This ensures robust speaker tracking through movement and occlusion without altering the base- line single-person model. Second, leverage framework's inherent agent- based design by
18

---

Published as a conference paper at ICLR 2026
augmenting the Planner. Specifically, the Planner is modified to accept this speaker mask as an ad- ditional input alongside the reference image. The mask is binary image where pixels corresponding to the active speaker in the reference image are set to 255, while all other areas are O. We provide this mask to the Planner with a supplementary instruction clarifying its role as a speaker indicator. With the rest of the reasoning pipeline remaining identical, this simple yet effective extension enables the model to generate logically consistent and coordinated actions for all individuals in the scene.
Training Data Curation and Composition. Our training set comprises 15,000 hours of video data, curated through a multi-stage pipeline inspired by prior works (Lin et al. 2025b; Cui et al. 2024) to ensure data quality and relevance. The filtering process involved the following steps:
1. Temporal Segmentation and Filtering:
We first processed all videos for temporal con- sistency. We utilized PySceneDetect to segment videos at shot transitions, retaining only continuous clips with durations between and 20 seconds. We then filtered for videos containing human subjects based on their captions.
2. Visual and Motion Quality Enhancement: We employed PaddleOCR to identify and remove clips with intrusive or changing subtitles. The visual fidelity and aesthetic appeal of each video were then assessed using Q-align (Wu et al. 2023), discarding any clips that fell below a predefined quality threshold. To ensure the motion was suitable for learning, we calculated optical flow with Raft (Teed & Deng 2020) and filtered out videos exhibiting either excessively static or chaotic content.
3. Audio-Visual Synchronization: For our audio-driven generation task, we performed a crucial filtering step using SyncNet (Chung & Zisserman, 2017) to verify the synchroniza- tion between lip movements and the audio track, discarding out-of-sync examples. The resulting dataset was used for the main training phase. For samples with poor lip-audio correlation (constituting approx. 70% of the data), we discarded the audio and utilized them with audio-dropout during training.
4. Niihh-uaally Subset for Fine- Tuning: For the final fine-tuning stage, we required a higher-quality selection. We curated the top 100 hours of data by ranking all clips based on a suite of quality metrics, including aesthetics (Q-align) and lip-sync accuracy (SyneNet).
To better understand the composition of our final dataset, we analyzed the distribution of randomly sampled subset. In terms of character framing, the data primarily consists of closer shots: upper- body shots (from the chest up) constitute approximately 47% of the clips, followed by medium shots (from the hips up) at 30% Full-body and long shots make up the remaining 15% and 8%, respectively. Regarding scene type, the dataset shows a notable preference for indoor environments (approx. 45%). while outdoor scenes with static, high-quality backgrounds represent 41 %, and those with dynamic backgrounds constitute the final 14%.
C EVALUATION DETAILS
Evaluation Datasets. To rigorously assess our model, we recognized that existing DiT-based meth- ods already perform well in standard human speaking scenarios. We therefore constructed two novel and highly challenging test sets to probe the limits of generalization beyond standard benchmarks.
Single-Subject Generalization Set: A diverse set of 150 cases, featuring real-world hu- man portraits, AIGC figures, anime characters, and animals. Each image was expertly paired with corresponding audio track (c.g., speech, singing, theatrical performance) to create demanding generalization test. To assess text-conditioning, experts also authored descriptive prompts for all 150 cases.
Multi-Subject Interaction Set: A set of 57 cases with similar visual diversity to the single- subject set. Audio tracks were expert-paired to reflect multi characer interactions, enabling the evaluation of coordinated behaviors.
For fair comparison with prior work, we also adopted their experimental settings, using 100 videos from CelebV-HQ (Zhu et al. 2022) for the talking- head task and the CyberHost (Lin et al. 2025a) test set (269 videos, 119 identities) for full-body scenarios.
19

---

Published as a conference paper at ICLR 2026
Speech: Swallow the blue pill and this chapter closes for good. You'll wake to the morning fight and all the things you believe will stay real. Or choose pill and this ilussion hold you anymare. you how many unpatched cracks beneath this world that feels solid.
Speech: applied inner subject's For blush, a specific shade from the eyashadow palotto was utiizzd.
purple eyeliner applidd
Speech: Crystal ball glowe, wrinkled hands tell secrets. Futures twist and turn. Laugh aa they beg for mercy.
Figure 6: Reasoning Process Visualization. The generated motion matches the speech content and the character's style in the image. We also provide the corresponding reasoning process.
Evaluation Metrics, We employed multi-faceted protocol with objective, subjective, and MLLM- based metrics for a comprehensive assessment.
Objective Metrics: We measured generation quality via Fréchet Inception Distance
(FID)
(Heusel al., 2017) and Fréchet Video Distance (FVD) (Unterthiner al.)
alongside no-reference Image Quality (IQA) and Aesthetics (ASE) scores from Q- align Wu et al. 2023). We assessed audio-visual synchronization and hand quality using several metrics. For audio-visual synchronization, we primarily employed the widely used Syne-Confidence (Sync-C) metric from SyncNet (Chung Zisserman, 2017). However, recognizing its potential limitations in multi-person scenarios, where it can yield mislead- ingly high scores for non-spen son-spekinng faces, we additionally adopted the Sync-Distame D) metric specifically for evaluating these cases. For hand quality, used Hand Keypoint Confidence (HKC) and Hand Keypoint Variance (HKV) as proposed by [in al. (2025a).
2019
(Sync-
Subjective Metries (Human Evaluation): As objective metrics often fail to capture high- level semantics and perceptual realism, we conducted a comprehensive user study with 40 participants. The study consisted of two main protocols.
The first pairwise comparison, where participants viewed randomized pairs of videos from different methods. This comparison involved two distinct evaluation tasks:
Participants chose the video with the best overall quality. From these preferences, we computed Good/Same/Bad (GSB) score, defined as (Wins - Loses) (Wins Loses Ties).
Overall Quality ("Which is better?"):
Artifact Analysis ("Which less bad?"); Participants identified specific flaws in the less preferred video, including Lip-syne Inconsistency (LSI), Motion Unnaturainess (MU), and Image Distortion (ID), to calculate a defect rate for each category.
The second protocol was a best-choice selection, where participants selected the single best video from a group of all competing methods, yielding Top-1 selection rate as direct measure of overall appeal.
MLLM-basee Evaluation: To complement our human studies with a scalable and repro- ducible method for assessing semantic quality, we also employed MLLM-based eval- uation protocol. This automated approach assesses high-level aspects such as contextual coherence and logical consistency, which are difficult for traditional metrics to capture. The detailed instructional prompt used for this evaluation is provided in Appendix D.2
D.2
20

---

Published as a conference paper at ICLR 2026
D FURTHER ANALYSIS OF THE AGENTIC SYSTEM
D.I REASONING PROCESS DEMONSTRATION
In this section, we illustrate the workflow of our MLLM-based agentic system. The process begins with Analyzer module, which uses guided, step-by-step probing method to perform multi- dimensional analysis the multimodal input, including linguistic content, emotional state, and behavioral intent. Based the analysis results, Planner module then infers plausible and logically consistent sequence of actions for the character.
Figure 6 provides several visual examples generated by this process. Below, detail the corre- sponding reasoning steps the system took to generate the final video. The examples below illustrate how the system deconstructs complex inputs and formulates detailed action plan, "schedule", to guide the video synthesis. This process yields schedule that is not only rational but also guides the synthesis model to produce videos that are dynamically rich and logically coherent.
It is important to emphasize that our core contribution is the modular and framework-level design itself. By introducing dual-system design, we endow the avatar model with "an active mind" and context-wwareness, thereby achieving a level of vividness that is difficult for existing models to replicate. This provides a promising baseline for the community. We envision that this agentic system can be significantly enhanced in several ways. Future work could involve applying advanced training techniques
bolster reasoning, integrating more powerful MILLMs for planning, or even pursuing end-to-end training of the core LLM. We believe these directions are all worthy of further exploration to raise the performance ceiling of our framework.
Illustrating the Reasoning Process for the Sample in Row
Analysis Process
Identify the language used in the audio, such as English, Chinese, Korean, Japanese, Thai, etc. The language used the audio was identified as English.
Language Recognition:
Speech Transcription: Please transcribe the person's speech input completely and accurately. Do not omit or misspell any words while preserving the original expressions and sentence structure.
"Swallow the blue pill, and this chapter closes for good. You'll wake to the morning light, and all the things you believe will stay real. Or choose the red pill, and this illu- sion won't 't hold you anymore. I'll show you how many unpatched cracks lie beneath this world that feels solid."
Type Identification: Determine the type of audio just answer (speech) or (singing) only.
The audio is classified speech, not singing.
Emotion Analysis: Briefly analyze the emotional state the speaker or per- former.
The speaker's tone is serious and contemplative, reflecting a significant decision or revelation.
Intention and Purpose Analysis: Analyze the intention and purpose of the speaker or performer, and explain the information or motivation they aim to convey.
The speaker's intention to present a critical choice between blissful ignorance (blue pill) and the pursuit of truth (red pill).
Content Analysis: Divide the audio into multiple segments for separate analysis, and analyze and summarize the content expressed in the speech, with each seg- ment lasting seconds.
21

---

Published as a conference paper at ICLR 2026
The audio was segmented into four logical parts, each approximately 4.8 seconds in duration.
Multi-shot Analysis: Given the multi-shot Content Analysis for audio segments and above all information, analyze the intention and purpose of each segment. Segment introduces the blue pill, Segment 2 details its consequence, Segment introduces the red pill, and Segment details its consequence.
Analysis Summary
Language: English
Swallow the blue pill, and this chapter closes for good. You'll wake the morning light, and all the things you believe will stay real. Or choose the red pill, and this illusion won't hold you anymore. I'll show you how many unpatched cracks lie beneath this world that feels so solid.
Transcription:
Audio Type: speech
Emotion: The speaker's tone is serious and contemplative, reflecting a significant decision revelation.
Overall Intention:
The speaker's intention is to present a critical choice between blissful ignorance (blue pill) and the pursuit of truth (red pill).
Segmentation Method:
The audio was segmented into four logical parts, each ap- proximately 4.8 seconds in duration.
Segment Intentions:
Segment 1: Introduces the blue pill.
Segment 2: Details its consequence.
Segment 3: Introduces the red pill.
Segment 4: Details its consequence.
Generated Schedule
Segment 1: A leans forward, retracting his left hand holding red pill while ag- gressively pushing his right hand, which holds blue pill, directly toward the camera. His eyes are wide and intense, he speaks with serious expression, and his upper body tilts forward with the motion.
The man holds his pose, keeping his right hand with the blue pill promi- nently extended forward. He continues speak, his intense and persuasive gaze fixed on the camera. He makes slight, subtle shifts in his posture but the focus remains on his face and the offered pill.
Segment 2:
The man swiftly retracts his right hand holding the blue pill and thrusts his left hand, now holding the red pill, forward toward the camera. He leans in again, his expression shifting to dramatic, wide-eyed challenge as he presents the alternative choice, his mouth forming the words.
Segment 3:
Segment 4: While keeping the red pill extended in his left hand, the man brings his right hand with the blue pill forward again, presenting both choices simultaneously to the camera. He leans in even closer, his eyes widening in a manic, intense stare as he speaks, emphasizing the gravity of the decision.
22

---

Published as a conference paper at ICLR 2026
Illustrating the Reasoning Process for the Sample in Row
Analysis Process
Identify the language used in the audio, such as English, Chinese, Korean, Japanese, Thai, etc. The language used the audio was identified as Chinese.
Language Recognition:
Speech Transcription: Please transcribe the person's speech input completely and accurately. Do not omit or misspell any words while preserving the original expressions and sentence structure.
The content of the speech is "For the lower eyeliner, draw the front part with blue eyeliner, the back part with purple eyeliner, and for the red, use this one from the eyeshadow palette." (English translation)
Type Identification: Determine the type of audio just answer (speech) (singing) only.
The audio is classified as speech, not singing.
Emotion Analysis: Briefly analyze the emotional state of the speaker or per- former.
The speaker's tone is neutral and informative, consistent with providing instructions.
Intention and Purpose Analysis: Analyze the intention and purpose of the speaker or performer, and explain the information or motivation they aim to convey.
The speaker's intention is to provide instructions on how to apply eye makeup, specif- ically guiding the listener on using blue, purple, and red shades for a particular look.
Content Analysis: Divide the audio into multiple segments for separate analysis, and analyze and summarize the content expressed in the speech, with each seg- ment lasting seconds.
The audio segmented into two logical parts, with the first lasting 6 seconds.
Given the multi-shot Content Analysis for audio segments and above all information, analyze the intention and purpose of each segment. Segment instructs on applying blue and purple eyeliner to the lower lash line, while Segment 2 suggests using specific red eyeshadow to complete the look.
Multi-shot Analysis:
Analysis Summary
Language: Chinese
For the lower eyeliner, draw the front part with blue eyeliner, the back part with purple eyeliner, and for the red, use this one from the eyeshadow palette.
Transcription:
Audio Type: speech
Emotion: Neutral and informative.
Overall Intention: To provide instructions on how to apply eye makeup using blue, purple, and red shades.
Segmentation Method: The audio was segmented into two logical parts.
Segment Intentions:
Segment 1: Instructs on applying blue and purple eyeliner.
23

---

Published as a conference paper at ICLR 2026
Segment 2: Suggests using red eyeshadow complete the look.
Generated Schedule
A beautiful East Asian woman holds small compact mirror and metic- ulously applies eyeshadow to her upper eyelid using a fine-tipped makeup brush, her expression focused as she looks into the mirror.
Segment 1:
She then looks directly at the camera while speaking, and gestures with the brush towards the open eyeshadow palette in her other hand, if explaining the next step in makeup tutorial.
Segment 2:
Illustrating the Reasoning Process for the Sample in Row 3
Analysis Process
Identify the language used in the audio, such as English, Chinese, Korean, Japanese, Thai, etc.
Language Recognition:
The language used in the audio was identified as English.
Speech Transcription: Please transcribe the person's speech input completely and accurately. Do not omit or misspell any words while preserving the original expressions and sentence structure.
The complete transcription of the speech is: "Crystal ball glows, wrinkled hands tell secrets, futures twist and turn laugh as they beg for mercy."
Type Identification: Determine the type of audio just answer (speech) or (singing) only.
The audio is classified as speech, not singing.
Emotion Analysis: Briefly analyze the emotional state of the speaker or per- former.
The speaker's tone is serious and contemplative, suggesting a mystical or prophetic mood.
Intention and Purpose Analysis: Analyze the intention and purpose of the speaker or performer, and explain the information or motivation they aim to convey.
The speaker's intention is evoke a sense of mysticism and foresight, describing scene of fortune-telling or prophecy where futures are revealed.
Content Analysis: Divide the audio into multiple segments for separate analysis, and analyze and summarize the content expressed in the speech, with each seg- ment lasting seconds.
The audio was segmented into two logical parts, with the first lasting 4.8 seconds.
Multi-shot Analysis: Given the multi-shot Content Analysis for audio segments and above all information, analyze the intention and purpose of each segment. Segment sets mystical scene with a glowing "crystal ball", while Segment 2 adds human element of ancient wisdom with "wrinkled hands" revealing secrets.
Analysis Summary
24

---

Published as a conference paper at ICLR 2026
Language: English
Crystal ball glows, wrinkled hands tell secrets, futures twist and turn laugh they beg for mercy.
Transcription:
Audio Type: speech
Emotion: Serious and contemplative, with a mystical or prophetic mood.
Overall Intention: To evoke sense of mysticism and describe scene of fortune- telling.
Segmentation Method: The audio was segmented into two logical parts.
Segment Intentions:
Segment 1: Sets a mystical scene with glowing crystal ball.
Segment 2: Adds a human element of wisdom with wrinkled hands revealing secrets.
Generated Schedule
Segment 1:
The woman leans forward slightly, her gaze fixed and intense, her gnarled right hand gesturing mysteriously near her face. The crystal ball in her left hand begins emit a faint, ethereal glow, hinting at the power stirring within.
Segment 2: A swirling vortex of white light suddenly manifests and spins within the crystal ball. The witch's expression shifts dramatically: her eyes widen and her mouth opens in gasp of shock or revelation as she witnesses the unfolding future.
We also provide the full system prompt for the Planner MILM in the supplementary material, as too verbose to include even in the appendix.
D.2 FURTHER QUANTITATIVE EVALUATION
To further validate our design, we conducted supplementary MLLM-based evaluations using Gemini-2.5-Pro (Comanici et al.) 2025), leading MILM with strong audio-visual comprehen- sion. This evaluation focused on three key dimensions: Overall Quality, Prompt Following, and Motion-Context Coherence. As shown in Table 5 the results indicate that full model sub- stantially outperforms an ablation using OmniHuman-I1 I's conditioning method (reference attention and audio cross atteettntnn Furthermore, our approach demonstrates significant advantage when compared to variant of our model lacking the agentic reasoning module. These considerable im- provements underscore the effectiveness of our core designs. particular, the notable enhancement in Motion-Conten Coherence highlights our model's superior ability to generate action sequences with logical and self-coherence. The table also includes MLLM-based pairwise comparisons other architectural ablations, confirming that each proposed module contributes positively to the final performance.
Finally, provide additional subjective user study comparing method against the recent open-source model, Wan2.2-S2V. As shown in Table our model achieves significantly better per- formance, both in the overall GSB preference
and across all measured artifact dimensions.
MLLM Evaluation Prompt and Protocol. We provide the detailed instruction prompt used for the MLLM-based pairwise comparison. During the evaluation, videos from method and the competing method were stitched side-by-side into a single video file to be fed into the MILM. To mitigate positional bias, the placement of the videos (left right) randomized for each comparison, and each sample pair was evaluated five times to ensure robust results.
25

---

Published as a conference paper at ICLR 2026
Table 5: MLLM-based Evaluation. Pairwise comparison results (GSB Score) with individual scores for Overall Quality, Prompt Following, and Motion-Context Coherence.
Baseline
Proposed
Overall Quality
Prompt Following
Motion-Context Coherence
OmmiHunaann1 Conditioning
Ours (Full Model)
Ours w/o Agentic Reasoning
Ours (Full Model)
26
29%
28%
Ours w/ Ref. Attention
Ours (Full Model)
10%
7%
8%
wio Warm- up
Model)
Compurison between reazoning modiule vorionis
Singlo-Step Reasoning
Ours (Full, Multi-Step)
Ours w/o Ageatic Reasoning
Ours Single-Step Reasoning
22%
24%
17%
Table 6: Subjective Evaluation vs. Wan2.2-S2V. Subjective evaluation against Wan2.2-S2V, in- cluding GSB preference score (Which better?) and pairwise comparison of artifacts (Which less bad?).
Method
LSI
MUJ
ID
GSB
Wan2.2-S2V
0.24
0.60
0.21
-0.71
Ours (Full Model)
0.04
0.08
0.02
+0.71
Prompt for MLLM-based Pairwise Video Evaluation
Task:
Multi-dimensional Comprehensive Evaluation of AI-Generated Videos
1. Role You are comprehensive review expert specializing in filmmaking, visual story- telling, and multimedia arts. Your mission to conduct an objective, in-depth, and detailed comparative evaluation of two Al-generated videos based on rigorous and professional set of multi-dimennsionll criteria.
Core Requirements:
Objectivity: Disregard personal preferences and base your judgment solely on the provided criteria and input information.
Professionalism: Conduct your analysis from a professional perspective.
Focus Area: Your evaluation should primarily focus the actors and characters in the video, including their expressions and movements. You may moderately disre- gard other non-primary elements such backgrounds and props.
2. Core Evaluation Dimensions You will comparatively evaluate Video (left) and Video (right) across the following three core dimensions.
Dimension Global Quality A comprehensive assessment of the video's 's overall artistic appeal, technical execution, and viewer experience.
Evaluation Points:
Image Quality: Clarity, color representation, lighting, and the presence of any noticeable noise artifacts.
Motion Naturalness: Whether movements are fluid, conform to the laws of physies, and free from stiffness the uncanny valley effect.
Expressiveness: Whether facial expressions are vivid, nuanced, and capable of accurately conveying emotions.
Aesthetics & "Wow" Factor: Does the video possess unique visual style or artistic beauty? Does it offer surprises in creativity. composition, or visual ef- fects?
Viral Potential: Which video is more likely to resonate with an audience and inspire sharing?
Dimension Action Control & Prompt Adherence assessment of how accurately the video responds to specific "action" and "expression" instructions in the user's prompt.
26

---

Published as a conference paper at ICLR 2026
Evaluation Points:
Instruction Fidelity: Are the core actions and expressions exccuted strictly as described in the prompt? Are there any omissions or misinterppettations?
Action Completion: Are the instructed actions fully and clearly rendered? Is the sequence of movements coherent? Is the sense of power, rhythm, and amplitude appropriate?
Detail Representation: How well are complex subtle actions (e.g., hand ges- tures, eye contact) reproduced?
Dimension 3: Overall Content Coherence & Logicality An assessment of the logi- cal, contextual, and emotional consistency between the video content and the audio content.
Evaluation Points:
Contextual Appropriateness: Considering the audio, the character's actions and expressions logical and fitting for the scene and intent?
Audiovisual Synchronization: Does the rhythm of the on-screen actions match the audio? Are the movements human-like, natural, and reasonable?
Narrative Unity: Do the visual and audio elements work together unified theme or emotional tone? Do they enhance each other, or do they conflict?
3. Evaluation Task & Output Format
Input Information:
1) Video for Evaluation: A side-by-side video file with unified audio.
Video I: Located the left side of the screen. Video Located on the right side of the screen.
2) Original Prompt: The text instruction used to generate both videos.
Task Flow:
1) Carefully analyze the requirements.
2) Simultaneously watch and compare Video 1 and Video 2.
3) Conduct point-by-poin comparative analysis based the Core Evaluation Di- mensions.
4) Strictly adhere the JSON format below output your final evaluation.
Output Requirements:
Based on your comparative analysis, provide a final score for each dimension using the following rubric.
-1: The left video is significantly better.
The two videos are comparable, or each has its own merits.
1: The right video is significantly better.
JSON Output Format:
{ sscore:: "Enter -1, 0, or 1
"action_control"; {
27

---

Published as a conference paper at ICLR 2026
"score": "Enter -1, 0, or 1"
"overall "overall_contet__com content_cohereneeeeeeet "score"; "Enter -1, 0, or 1"
E ADDITIONAL QUALITATIVE RESULTS AND VISUAL ABLATIONS
In this section, we provide additional visual comparisons and ablation studies to further demonstrate the effectiveness of our method.
Qualitative Comparison with Baselines. Figure 7 showcases comparative analysis of our method against recent approaches. The first example highlights common limitation in existing methods: a tendency to produce static or minimally varied outputs. Even OmniHuman-1,, despite generating more varied gestures, fails to establish a meaningful correlation with the audio and struggles with temporal consistency, causing objects like the held capsule to flicker. Furthermore, its output often appears over-sharpened, which can degrade visual quality, particularly with high-saturation inputs. This issue of temporal instability is exacerbated in longer generation sequences, shown in the second example. There, competing methods either remain static or introduce significant visual ar- tifacts, such discoloration, when motion is attempted. Our method, in stark contrast, maintains both high-dynamic motion and excellent visual stability throughout the extended duration. This dual achievement is a direct result of our design: the logical, speech-matched motion stems from our agentic system, while the high-quality consistent object interaction is enabled by our condition- ing method's powerful fusion of multimodal cues.
In Figure 12 we further present visual comparison with the ofate-of-the-arr method, OmniHuman- 1. The corresponding speech content for each video provided. As can seen, our method demonstrates significantly stronger semantic relevance and logical correlation between the audio and the generated actions. For instance, in the first video, the character turns her head when calling out "Mary"; in the second, she performs the specific actions of applying eyeliner and gesturing towards the eyeshadow palette as described; and the third, the crystal ball glows and changes in response to the wizard's 's incantation. These high-level, context-aware results are difficult to capture with objective metrics and remain a challenge for existing methods. We encourage readers to view project page for generated motion, speech content, and character intent.
the examples
intuitive understanding of the correlation between
Visual Ablation of the Pseudo Last Frame (PLF). In Figure g demonstrate the critical role of our proposed Pseudo Last Frame (PLF). We compare generated results with and without the PLF module under conditions of significant camera and character motion. The difference is stark: without the PLF, the generated sequence undergoes drastic changes in character identity and scene content, rendering unusable. In contrast, the version equipped with the PLF successfully maintains high-quality visuals and character consistency while accommodating nigh-dynamic movements and large camera shifts.
Furthermore, Figure provides a deeper analysis of the PLF's mechanism by comparing it against two alternatives: standard reference attention mechanism and the PLF with varying RoPE distance offsets. While reference attention can respond to the text prompt, it gradually reverts to the reference image during autoregression, exhibiting a "reset" behavior. The PLF, however, effectively mitigates this issue. By increasing the RoPE distance offset, we can strike a better trade-off, enabling the model to preserve content from the reference image while still generating significant motion. For instance, with a 'rope-30* offset, the first example shows smooth transition from side profile to a full-frontal view the character. A similar effect is observed in the second example, where the "rope-30 offset locks the camera's viewpoint at the desired state, preventing reset.
28

---

Published as a conference paper at ICLR 2026
OmniAvatar
Skyreel-A
Multitakk
BAY PM
OmniAvatar
Skyreal-A
Multitalk
Ours
Figure 7: Qualitative Comparison with Recent Methods.
wio Pseudo Frame
Ours
10s
20s
50s
B0s
100s
Figure 8: Effectiveness of PLF in Preserving Content Consistency.
In summary, the visual results in Figures and 9 collectively demonstrate that the PLF module is essential for: (I) enabling large, continuous motions without the "reset" problem, and (2) simulta- neously preserving the stability and consistency of image quality, color, and character identity.
Visual Ablation of the MM-Warmup. Figure 10 presents an ablation study on the MM-Warmup stage. Without this stage, the model exhibits two primary deficiencies. First, the resulting motion appears less dynamic, contributing to subtle reduction in expressiveness. More critically, the qual- ity of hand gesture synthesis deteriorates significantly, with an increase in artifacts on local body parts, indicating compromised ability for holistic synthesis. This comparison underscores the effi cacy of approach: by integrating the proposed MM-W Warmup with MM-DiT architecture, successfully adapt a aet-and-image-ee
(T2V) model into a high-fidelity video avatar model, maintaining exceptional multimodal modeling capabilities and overall synthesis quality.
Visual Ablation of the Reflection Process. We visualize the impact of our optional reflection process in Figure II Without reflection (top row), the model generates action schedule in single pass, which can lead to logical inconsistencies. For instance, after the action "Takes out letter," the model generates "Rubs the surface," causing the letter vanish and breaking semantic continuity. In
29

---

Published as a conference paper at ICLR 2026
Ref. Image
rope-10
rope-20
rope-30
Ref. Image
rope-10
rope-20
rope-30
Figure 9: Effectiveness of PLF in Maintaining Motion Dynamies.
contrast, our model with reflection (bottom row) revises the plan after generating the first segment. It observes the outcome of "Takes out letter" and corrects subsequent actions to be relevant to the theme of letter-reading, ensuring logical progression and mitigating error accumulation. However, as this reflection process introduces additional inference overhead, it was disabled for the quantitative comparisons in this paper. We also investigated injecting reasoning latents directly into the synthesis model. While this encouraged more nuanced facial expressions, it also suppressed large, dynamic actions. As this appeared to be an aesthetic trade-off rather than a clear improvement, we excluded it from final model configuration.
30

---

Published as a conference paper at ICLR 2026
Figure 10: Qualitative results of the reflection process. Without reflection (first row), an ill- planned action ("Rubs the surface") causes object inconsistency. With reflection (second row), the model revises its plan to more logical action, ensuring consistency.
Simplified baseline schedule: Takes out letter
Rubs the surface
Holds to chest, looks away Lifts to leens in
Simplified schedule with reflection: Takes out letter -> Holds to chest ->i Lowers letter, looks Reveals face, confused
Qualitative results of the reflection process. Without reflection (first row), ill- planned action ("Rubs the surface") causes object inconsisteney. With reflection (second row), the model revises its plan more logical action, ensuring consistency.
Figure 11:
F LATENCY OF THE AGENTIC SYSTEM
The introduction of our agentic reasoning module adds a fixed, upfront latency of approximately 20-30 seconds for its "thinking" process, measurement taken without any engineering optimiza- tions. This cost is constant regardless of video length, and we argue is a justifiable trade-off for two primary reasons. First, this latency is minor when contextualized against the 5-6 minutes that mainstream Diffusion Transformer (DiT) based video generation models require for 10-second clip. We also anticipate this cost will diminish significantly with the rapid advancement of LLM inference technologies. Secondly, this reasoming phase is preciscly what enables the substantial improvements in expressive power and contextual coherence that characterize our method. This process mirrors the "slow thinking" of deliberate human cognition, making it necessary step for achieving higher-level intelligence, not merely a computational overhead. Complex reasoning is in- herently time-consuming. Therefore, consider this latency reasonable and even desirable price for the significant leap in generation quality.
G STATEMENTS
Ethics Consideration. Our core contribution is novel paradigm for video avatar generation. By simulating dual-system cognitive framework, our model achieves a new level of expressive capa- bility and logical coherence, moving beyond the limitations of single-process generation. While this advancement opens exciting possibilities for creative applications like AI-driven film production, recognize the potential for misuse associated with highly realistic avatar technologies.
To address these ethical concerns, we advocate for a robust framework of responsible deployment. Given its potentially powerful capabilities, this model requires strict access control to prevent po- tential abuse. Although current results may still bear subtle artifacts that can serve as an informal deterrent, proactive safeguards are essential. We strongly recommend the following measures: (1)
31

---

Published as a conference paper at ICLR 2026
Speech: Mary, pass me the ivory thread. This her,, needs be perfect
Speech: A blue eyeliner applied the inner section the subject's lower lash line, while eyeliner applied to the outer section. For blush, a specific shade from the eyeshadow palette was utilized.
Speech: Crystal ball glows, wrinkled hands tall secrets. Futures twist and tum. Laugh as they beg for mercy.
Figure 12: Qualitative comparison with OmniHuman-1 (Lin et al. 2025b), state-of-the-ard
model representing the predominant conditioning scheme of audio cross- tros-aattention and image reference attention. For each pair of examples, higher semantic consistency to the speech prompt than the baseline (top row). For example, our model correctly depicts character applying makeup and glowing crystal ball described in the speech, actions which are absent in the baseline's ' results.
model (bottom row) generates actions with
applying prominent, visible watermarks to all generated content to clearly label it as Al-generated; (2) implementing filtering algorithms to reject inappropriate or malicious input prompts and re- view output content; and (3) embedding traceable, invisible watermarks to ensure accountability and aid in source identification if misuse occurs. By integrating these safety protocols, we can help ensure that our technology fosters creativity while minimizing the risks of malicious applications such as fraud or disinformation.
LLM Usage Statement. We utilized Large Language Models (LLMs) as ceneral-purpose assistive tool in the preparation of this manuscript. The usage was confined to two main areas: (1) refining language and word choice for pre- pre-written sentences by seeking explanations determine the most effective phrasing, and (2) assisting with technical aspects of document preparation, such as gener- ating syntax for LaTeX tables and debugging compilation errors. The authors are fully responsible for all intellectual content of the paper.
Reproducibility Statement. To ensure the full reproducibility of our work, we provide compre- hensive details regarding our methodology, data, implementation, and evaluation procedures across the main paper and its appendices. Specifically, we detail: (1) the core architecture of our agen- tic system and the MMDiT model in Section 3 (2) complete implementation details, including all hyperparameters and the multi-stage training procedure, in Appendix B (3) our data curation pipeline, with specific filtering tools (e.g., PySceneDetect, Q-Align) and a statistical analysis of the dataset, also in Appendix B and (4) agentic system and evaluation framework, which specifies the MLLM models used (mniCPM-o and Seed- 1.5- VL), instruction prompts, and the complete set of objective and subjective evaluation metrics in Appendices and C Finally, while provide
32

---

Published as a conference paper at ICLR 2026
Figure 13: Limitations of our method. Despite its strong performance, our method exhibits limita- tions in both synthesis and reasoning. On the synthesis level, the model can struggle with fine details during rapid movements, leading to hand artifacts (row 1) and degraded facial identity during large head turns (row 2). Separately, on the reasoning level, the agent can generate motions that, while plausible, are occasionally over-articulated (e.g., excessive gesturing), lacking the subtlety required for cinematic performance.
these details ensure technical reproducibility, we reiterate our ethical recommendation that for sponsible deployment, access to the trained models should be managed in a controlled environment monitor usage and prevent potential misuse.
H LIMITATIONS
Limitations and Future Work. While our method achieves strong overall performance, is not without its limitations, as visualized in Figure [3] These challenges primarily fall into two cate- gories: minor artifacts the synthesis level and occasional imperfections in motion reasoning. To address these issues and further advance our model's capabilities, we have identified two key di- rections for future work. First, to enhance the model's fundamental generation quality and mitigate synthesis artifacts, we plan to continue scaling up by training larger, higher-quality datasets. Sec- ond, to improve the alignment between high-level reasoning and low-level motion synthesis, we will investigate more sophisticated integration strategies, with particular focus the end-to-end joint training of the DiT and the LLM.
33


Reviews:
[{"review_id": "Q0W0NUQvMQ", "reviewer": "Reviewer_MK7E", "summary": "This paper introduces a novel framework for generating dynamic and expressive video avatars by \"instilling an active mind,\" drawing an analogy from the dual-process theory of human cognition (System 1 and System 2). The authors argue that existing avatar models are largely \"System 1\" agents, capable of reactive tasks like lip-sync but failing to exhibit higher-level semantics like emotion, intent, or contextual awareness. To bridge this gap, they propose a dual-system framework. A \"System 2\" module, termed Agentic Reasoning and powered by a Multimodal Large Language Model (MLLM), acts as a deliberative planner. It processes all input modalities (audio, reference image, text prompt) to generate a high-level, structured plan (in JSON format) that outlines the avatar's expressions, actions, and emotional journey. This plan then guides a \"System 1\" module, a reactive rendering engine built upon a novel Multimodal Diffusion Transformer (MMDiT). This engine synthesizes the final video, conditioned on both the high-level plan and the immediate audio signal. Key technical innovations include a \"Pseudo Last Frame\" (PLF) conditioning mechanism to maintain identity without sacrificing motion dynamics, and a specialized training strategy (MM-Branch Warm-up) to effectively train the complex multi-modal architecture. The authors demonstrate state-of-the-art performance through extensive experiments, including quantitative metrics and user studies, against a wide range of strong baselines.", "questions": "See the Weakness.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The System 1/System 2 analogy is a standout feature, providing a clear and compelling motivation for the entire architecture. It reframes the problem in a way that is likely to inspire future work. Meanwhile, such method will improve the performance of avatars in the practice application.\n2. The introduction of the Pseudo Last Frame (PLF) to decouple identity from static posture is a significant practical contribution that directly addresses a common failure mode in existing models.\n3. The model achieves state-of-the-art performance across a comprehensive suite of metrics and, most tellingly, wins decisively in head-to-head user preference studies against top-tier competitors. And the paper provides textbook-quality ablation studies that convincingly demonstrate the necessity and effectiveness of its proposed components, from the high-level reasoning module down to specific architectural choices. Meanwhile, results shows that the model's success in diverse scenarios, including multi-person turn-taking and non-human avatars, highlights the robustness and power of the proposed framework.", "weaknesses": "1. About the Latentcy: The framework, involving a large MLLM for planning followed by a large diffusion transformer for rendering, is undoubtedly computationally intensive. The paper mentions training on a very large dataset (11,000 hours) and inference at 480p. A discussion of the computational requirements (e.g., VRAM, training time) and, more importantly, the ``inference latency'' would be valuable for understanding the practical deployability of the system.\n2. The experiments primarily showcase clips of up to 720 frames (around 30 seconds). While the Agentic Reasoning module is designed for long-range planning, the paper does not explicitly test the limits of this coherence over much longer durations (e.g., several minutes). It would be interesting to know if any failure modes, such as narrative drift or loss of consistency, emerge in longer-form content generation.\n3. A more complex settings with difficult reasoning will be beneficial to claim the improvement of this paper, and the setups of such benchmark will be useful for fairly comparison.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qZFE1Thm9s", "reviewer": "Reviewer_1USz", "summary": "This paper tackles controllable avatar generation that produces semantically meaningful, context-aware motion beyond basic lip-sync. The key challenge is aligning high-level cognitive intent with low-level reactive control, as naive multimodal fusion often causes semantic interference. To address this, it proposes a cognitive-simulation framework combining high-level reasoning (System 2) with low-level motion generation (System 1). Experiments demonstrate consistent improvements across multiple metrics.", "questions": "It would be helpful to better understand the robustness and efficiency of the proposed System 2 reasoning module. Specifically, are the results consistent when the reasoning outputs contain noise, particularly common hallucinations, or ambiguity in the input image or audio? Additionally, can the System 2 module be made more lightweight, and is there any analysis of its additional inference time or computational overhead?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper introduces a clear and creative dual-system framework (System 1 and System 2) for avatar generation, combining cognitive reasoning with generative modeling in a fresh way. The architecture includes well-designed and tested components such as the MMDiT backbone, semantic soft guidance, and the Pseudo Last Frame, which together improve the link between reasoning and motion generation. The experiments are thorough and the results show that the proposed method produces more expressive and context-aware avatars than strong baselines like VASA-1 and SadTalker, with ablation studies further supporting each design choice.", "weaknesses": "While MLLMs are powerful for reasoning, they come with high computational costs. For tasks such as analyzing a character's speech content, emotion, or planning shot composition, using a large MLLM may be unnecessary. A smaller, task-specific model jointly trained with the generative system could offer a more unified and efficient solution. Although incorporating an MLLM adds conceptual richness, the paper does not provide any quantitative analysis of the additional computational or latency overhead. Nonetheless, the work presents a creative and promising direction by integrating cognitive reasoning into avatar generation and demonstrates clear value. I would recommend a score above the acceptance bar.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MKpsa8cfu0", "reviewer": "Reviewer_Rgov", "summary": "This paper proposes a two-stage framework for avatar-centric video generation conditioned on audio, text, and reference images. In the first stage, a video plan is generated, while in the second stage, the actual video is synthesized based on this plan and the multimodal conditions. Notably, when incorporating the reference image, the paper introduces the Pseudo Last Frame (PLF) technique, which places the reference image at a distant, unreachable position in RoPE. This approach introduces reference information while avoiding forcing the model to regress to that exact image.\n\nAs far as I know, two-stage frameworks have been explored in various contexts, and the paper's emphasis on \"cognitive simulation\" appears to be more of a conceptual framing. Therefore, there are certain limitations in terms of novelty. However, considering the effort involved in adapting to this specific domain, the innovation of PLF, and the promising qualitative and quantitative results, I am tentatively inclined to accept this paper.", "questions": "1. Regarding PLF, during inference, is the reference image placed only at the \"pseudo last frame\" position, or is it placed at both the first frame and the \"pseudo last frame\" position simultaneously?\n\n2. Is there a comparison between PLF and the prior method that \"conditions the model on a reference image sampled from the training video\"?\n\n3. How does PLF perform on image-to-video benchmarks? In other words, within a broader scope, is PLF a superior method for introducing reference images in image-to-video generation?\n\n##### References\n\n[1] Qu L, Wu S, Fei H, et al. Layoutllm-t2i: Eliciting layout guidance from llm for text-to-image generation[C]//Proceedings of the 31st ACM International Conference on Multimedia. 2023: 643-654.\n\n[2] Kannan S S, Venkatesh V L N, Min B C. Smart-llm: Smart multi-agent robot task planning using large language models[C]//2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2024: 12140-12147.\n\n[3] Liang J, Zhang H, Liu H, et al. Wavcraft: Audio editing and generation with natural language prompts[C]. ICLR 2024 Workshop on LLM Agents, 2024.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is clearly written and easy to follow. The figures and tables are well-designed and visually appealing.\n\n2. The proposed PLF is quite interesting and demonstrates effectiveness.\n\n3. The quantitative metrics show notable improvements. The visualization and supplementary video results demonstrate significant quality enhancements.", "weaknesses": "1. The proposed two-system framework—using an LLM as a planner followed by another system/model as an executor—has been explored in various contexts including video generation, text-to-image generation [1], robotics [2], and audio generation [3]. While the application to avatar generation shows merit, the overall structural approach is relatively well-established in the literature.\n\n2. The concept of \"cognitive simulation\" proposed in the paper could benefit from more rigorous justification. From my understanding, what is referred to as cognitive simulation appears to largely correspond to the planner-executor framework mentioned above. Given the prevalence of this framework across different domains, it would be helpful to clarify the specific advantages or unique aspects that the cognitive simulation framing brings to this particular application.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MJi99eFXeV", "reviewer": "Reviewer_fLxy", "summary": "This study proposes an avatar video generation framework consisting of two synergistic systems: System 1 is responsible for audio-to-video generation, and System 2 undertakes tasks of context awareness and logical inference. Through the collaboration of the two systems, the framework can better understand contextual semantic information to generate avatar videos; verified by extensive experiments, this method can significantly improve the quality and vividness of the generated content.​", "questions": "The following points need to be further clarified:​\n- What are the core differences between the application of MLLM in this research and existing prompt refiner methods?​\n- What specific technical efforts has the authors made to enhance the synergy between MLLM and MMDiT?​", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- The proposed dual-system avatar generation framework makes full use of the capabilities of MLLMs and optimizes the performance of video generation models. It provides an accurate analysis of the current dilemmas in avatar generation tasks, demonstrating strong innovation and inspiration in this field. Moreover, the dual-system model is expected to be applied to more general video generation frameworks, reflecting its high potential.​\n- It puts forward optimization details such as \"pseudolast-frame\". Aiming at the problem of error accumulation in autoregressive generation in the field of video generation, this method effectively alleviates this issue and may significantly improve the performance in real-time streaming generation scenarios.​\n- The experimental design is comprehensive and extensive, covering a large number of comparative experiments, ablation experiments, and professional user studies, which provides strong support for the research conclusions.​", "weaknesses": "In the current research, the combination of MLLM and MMDiT lacks tightness, and MLLM tends to play the role of a \"prompt refiner\".", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)