AUHead: Realistic Emotional Talking Head Generation via Action Units Control
Authors: Jiayi Lyu
Leigang Qu
Wenjing Zhang
Hanyu Jiang
Kai Liu
Zhenglin Zhou
Xiaobo Xia
Jian Xue
Tat-Seng Chua
Venue: ICLR 2026 Poster (2026)
Keywords: Talking Head Generation
Diffusion model
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Realistic talking-head video generation is critical for virtual avatars, film production, and interactive systems. Current methods struggle with nuanced emotional expressions due to the lack of fine-grained emotion control. To address this issue, we introduce a novel two-stage method (AUHead) to disentangle fine-grained emotion control, i.e. , Action Units (AUs), from audio and achieve controllable generation. In the first stage, we explore the AU generation abilities of large audio-language models (ALMs), by spatial-temporal AU tokenization and an "emotion-then-AU" chain-of-thought mechanism. It aims to disentangle AUs from raw speech, effectively capturing subtle emotional cues. In the second stage, we propose an AU-driven controllable diffusion model that synthesizes realistic talking-head videos conditioned on AU sequences. Specifically, we first map the AU sequences into the structured 2D facial representation to enhance spatial fidelity, and then model the AU-vision interaction within cross-attention modules. To achieve flexible AU-quality trade-off control, we introduce an AU disentanglement guidance strategy during inference, further refining the emotional expressiveness and identity consistency of the generated videos. Results on benchmark datasets demonstrate that our approach achieves competitive performance in emotional realism, accurate lip synchronization, and visual coherence, significantly surpassing existing techniques.
Our implementation is available at https://github.com/laura990501/AUHead_ICLR

Published as a conference paper at ICLR 2026
AUHEAD: REALISTIC EMOTIONAL TALKING HEAD GENERATION VIA ACTION UNITS CONTROL
Jiayi Lyu', Leigang Qu", Wenjing Zhang', Hiang M Kai Liu', Zhenglin Zhou', Xiaobo Xia", Jian Xuel.", Tat-Seng Chua? Uniiversity of the Chinese Academy of Scienees National University of Singapore ZZhejiang University "Corresponding author (lyujiayi21, 2ha22 jianghanyu231 @mails Smails.ucas www. ac. en (leigangyy) xi xiaoboxia. /@ (kail, thenglinzhouu ind.eeeeeeu cn, xuejian@ucan aaliannucasaccco..
@egaill 1. com, descts edu. sg
ABSTRACT
Realistic talking-head video generation critical for virtual avatars, film pro- duction, and interactive systems. Current methods struggle with nuanced emo- tional expressions due to the lack fine-grained emotion control. To address this issue, we introduce a novel two-stage method (AUHcad) to disentangle fine- grained emotion control, i.e., Action Units (AUs), from audio and achieve con- trollable generation. In the first stage, explore the AU generation abilities of large audio-language models (ALMs), by spatial-temporal AU tokenization and an "emotion-then-ALL chain-of-thought mechanism. It aims disentangle AUs from raw speech, effectively capturing subtle emotional cues. In the second stage, we propose AU-driven controllable diffusion model that synthesizes re- alistic talking-head videos conditioned on AU sequences. Specifically, we first map the AU sequences into the structured 2D facial representation enhance spatial fidelity, and then model the AU-vision interaction within cross-attention modules. To achieve flexible AU-quality trade-off control, we introduce an AU disentanglement guidance strategy during inference, further refining the emo- tional expressiveness and identity consistency of the generated videos. Results on benchmark datasets demonstrate that approach achieves competitive perfor- mance in emotional realism, accurate lip synchronization, and visual coherence, significantly surpassing existing techniques. Our implementation is available at https: / thub. com womp/aaaar0000///0ieetee ICLR
INTRODUCTION
Audio-driven talking head generation focuses on creating realistic facial animations given the driv- ing audio (Zhou al., 2025; Zhang et al., 2024). It aims achieve accurate synchronization, pre- serve speaker identity, and express vivid and emotionally coherent facial movements (Wang et al., 2021; 2024). This task has gained widespread attention due its potential applications in virtual avatars (Ma et al., 2025; Zhou et al., 2025; Wu et al., 2022), film production (Gao et al., 2025; Wang al., 2022; Liu et al., 2025b), and interactive agents (Guan et al., 2023; Cui al., 2024b).
Existing methods typically feed input au- dio and target portrait into genera- tive model directly (Fig. I(a)), achiev- ing strong performance in lip synchroniza- tion (Chung & Zisserman, 2017; Xu et al.. 2024) and identity preservation (Wang et al,, 2025). However, they often fall short in producing natural and subtle expres- sions, as they overlook the deeper emo- tional cues embedded in speech. To gen- erate videos that not only temporally accurate but also rich in emotional expres- siveness and fine- grained facial dynamics,
Audio
Portrait
Audio
Portrait
AU Representations
Diffusion
ALM
Diffission
Video
Understamding
Video
(a) Generation Only
(b) Understanding then Generation (Ours)
Figure 1: Framework comparison between existing talking head generation and our AUHead. (a) Direct generation from audio and portrait. (b) Our method: audio understanding via ALM, and then generation.

---

Published as a conference paper at ICLR 2026
adopt the framework shown in Fig. 1(b): We leverage an Audio Language Model (ALM (Chu et al., 2023)) to reasonably incorporate world knowledge and to uncover the deep emotional and relational patterns in audio (Qu et al., 2025a:b;c). The high-level cues subsequently transformed into explicit representations that condition the generation stage.
To achieve this goal, seek structured intermediate representation that bridges the audio modal- ity and the visual modality derived from a reference image. Facial Action Units (AUs) (Ekman & Friesen, 1978), which describe localized muscle movements with semantic meaning. offer a com- pact and interpretable expression space. Unlike coarse-grainee emotion labels, AUs offer abundant spatial and intensity information, the combinations of which can represent broad spectrum of fa- cial expressions (Gan et al., 2022; Yan et al., 2019). Since speech often involves coordinated facial musele movements and carries rich emotional information, AUS are naturally suited to capture both articulatory and emotional cues. We therefore propose to disentangle AU representations from audio to guide the facial generation process a controllable manner.
Despite these advantages, leveraging AUs as intermediate control signals introduces two challenges: (i) Due to limited audio-AU AU paired data and the sparse and subtle nature of AU activations, accurately and efficiently disentangling AU sequences from audio signals is the first challenge. (ii) AU-guided generation is non-trivial due to the controllability versus quality dilemma. Thus, effectively inte- grating AU features into video synthesis models without compromising lip synchronization, speaker identity, and visual quality becomes the second challenge.
In this work, to address the above challenges, we present two-stage method, named AUHead, for generating realistic and emotional videos conditioned on a reference image and audio. First, we propose disentangle AU sequences from audio based on an ALM (Chu al., 2023). Specifically, we propose a spatial-temporal AU tokenization strategy to efficiently discretize and tokenize raw dense AU vectors into compact set. And then we present a coarse-to-fine AU generation method inspired by Chain-of- of-Thought (CoT) (Wei et al., 2022). Lightweight fine-tuning of the ALM can stimulate the potential of emotion understanding and expression during the pre-training phase, by capturing subtle emotional cues like pitch and rhythm, demonstrating its emotional intelligence be- yond simple speech recognition. In the second stage, we propose an AU-driven controllable gener- ation framework, including AU representation, context-aware AU embedding, and AU-vision inter- action. Specifically, a diffusion-based model takes AU representations, the original audio, and the portrait image as conditions to produce emotional and realistic talking-heaa -had videos. Furthermore, to achieve flexible and accurate AU control, we introduce a disentanglement guidance strategy specifically tailored for AU conditioning, enabling tions (e.g., audio cues and motion priors) and visual quality. Quantitative and qualitative experiments demonstrate that our method outperforms baseline approaches, achieving smoother, more detailed, and emotionally consistent animations. The primary contributions of this paper include:
balance AU control with auxiliary condi-
We propose a novel two-stage method for emotional talking-head generation that decouples Emotion-oriented speech understanding from video synthesis.
We propose to excavate the emotion understanding and AU generation abilities of ALMs by disentangling emotion-awart AU sequences from raw audio. To our best knowledge, we are the first to explore generating facial AU sequences through ALMs. This establishes AUs as a meaningful and generalizable control space for audio-driven facial animation.
We present flexible AU-driven controllable generation framework to generate emotional and realistic talking head videos by AU representation, context-aware embedding, and AU-vision interaction. To strive for better balance between AU and other conditions, we propose an AU-based disentanglement guidance strategy during inference.
2 RELATED WORK
2.1 TALKING HEAD GENERATION
The rapid evolution of multi modal priors(Qu et al., 2025a;b;c) has catalyzed significant break- throughs across diverse generative frontiers, (Shen et al., 2025a; 2024; Shen & Tang, 2024). Early talking head generation methods convert speech into facial motion with lip-syne discrimination, head-pose prediction, and compact pose/emotion codes. Wav2Lip adds lip-syne discriminator
2

---

Published as a conference paper at ICLR 2026
(Prajwal et al., 2020). Audio2Head predicts 6D head pose and dense motion from audio (Wang et al., 2021). PC-AVS learns a compact pose code (Zhou et al., 2021). EAMM injects emotion offsets in the same latent space (Ji et al., 2022). Diffusion models (Jin et al., 2025) have become the dominant paradigm for audio-conditioned portrait videos, scaling to long duration, high reso- lution, and richer controls. DiffTalk, Diffused Heads, EMO, and Loopy generate from audio with latent or autoregressive diffusion (Shen et al., 2023; Stypuikowski et al., 2024; Tian et al., 2024; Ferdowsifard et al., 2021). SadTalker (Zhang et al., 2023) generates 3D motion coefficients and implicitly modulates a 3D-aware face renderer. Somic (Ji et al., 2025) focuses on global audio per- ception, disentangling into intra-clip and inter-clip audio knowledge. DAWN and IF-MDM enable non-autoregressiive sampling or compressed motion latents (Cheng et al., 2024; Yang et al., 2024). Hallo, Hallo2, and Long-Tere Term TalkingFace factorize lip, expression, and pose or introduce motion priors for long-form and 4K synthesis (Xu et al., 2024; Cui et al., 2024a; Shen et al., 2025b). Ani- Portrait, HunyuanPortraic MODA, and Portrait adopt two-stage or controllable pipelines (Wei et al., 2024; Xu et al., 2025; Liu et al., 2023; Nazarich et al., 2024). EDTalk (Tan et al., 2024a) introduced an efficient disentanglement framework that separately models mouth shape, head pose, and emotional expression through lightweight, orthogonal latent spaces. EchoMimic and V-Express guide diffusion with sparse facial landmarks (Chen et al., 2024; Wang et al., 2024). LatentSync and EchoMimic''> V2 add alignment and conditioning modules (Li et al., 2024; Meng et al., 2025). EMO- Portraits reshapes the latent space to capture intense and asymmetric expressions (Drobyshev et al., 2024). Recent works have explored leveraging high-level semantic cues from text or multimodal in- puts. Wan-S2V (Gao et al., 2025) builds upon a base model to achieve enhanced expressiveness for film-level einematic contexts. MultiTalk (Kong et al., 2025) addresses multi-person conversational video generation using a Label Rotary Position Embedding (L--RoFE) to solve audio-person bind- ing. OmniHuman-Ia 1.5 (Jiang et al., 2025) utilizes Multimodal Large Language Models to synthesize structured textual guidance, enabling semantically coherent and expressive character animations be- yond simple rhythmic synchronization. Emotion adapters such as ETAU(Lyu et al., 2024; 2025), EAT(Gan et al., 2023) and SAAS(Tan et al., 2024b) attach compact affect code to frozen gen- erator, while MEMOOZheng et al., 2024) couples a memory-guided module with emotion-awan awwae diffusion, sustaining identity and lip-sync across clips. DICE-Talk (Tan et al., 2025) introduces disentangled emotion embedder and correlation-enhanced emotion conditioning for emotional por- traits. Takin-ADA (Lin et al., 2024) uses a two-stage approach with specialized loss and advanced audio processing to enhance subtle expression transfer and lip-sync precision. In contrast to prior methods that rely on emotion labels or latent codes, our method AUHead, guides diffusion model through temporally aligned AU features. This enables structured and interpretable control over fine- grained facial expression synthesis.
2.2 LLMS FOR AUDIO INFORMATION PERCEPTION
Instruction-tuner tune Audio-Language Models (ALMs) endow large language models with a "sense of hearing", enabling open-ended spoken interaction across speech, music, and ambient sounds (Tang et al., 2024; Liu et al., 2025a). (Qwen-Audi) scales multi-task pre-training to over 30 tasks and introduces hierarchical tags to reduce cross-task label interference, establishing a strong open-source baseline (Chu et al., 2023). WavLLM adopts dual encoders plus curriculum learning to improve robustness on universal speech benchmarks (Hu et al., 2024), while Speech Verse freezes foundation models and instruction-tunn
lightweight fusion layer, outperforming task-specific baselines on eleven benchmarks (Das et al., 2024). While cxisting ALMs focus on general audio understanding, method adapts them for a novel purpose: exploring the capacity of ALMs to understand and represent emotional cues through temporally aligned AU sequences.
3 METHOD
Preliminaries. Latent Diffusion Models (LDMs) (Rombach et al., 2022) operate by encoding im- ages into latent space, modeling the latent distribution, and then decoding back to the image space. The encoding-decodinn procedure is achieved by a variational auto-encoder (VAE) (Kingma et al., 2013), denoted as (EE(.)) The model eg learns to predict the injected noise in the latent space by minimizing:
L - EI, EE ca(2,, l, c) c)][2]
(1)
3

---

Published as a conference paper at ICLR 2026
Stage Diventanglemeut (understunding)
AU Representation
CoT-based Emution-than thetinnnthee-AUU Generation
AL Sequences
Inpet
Q: Based the umderstandingg of the given undio first, ouruu the conveyed emotion (s.g. happiness, sadness, anger, etc.), and reason output
sequence, each AU as (index, intensity) pair. Let's think step by step.
Qwen Andio Chat
fie
Model Output
Fiss, the likely emotioo happiness. The speaker's
Represcntation
Interpllation
|ll
bright and energetic, with slightly higher pitch and rhythmie... Second. AU: au, 110, 15), 12), (11,84), (12.90), (23_08)) (3.61), (8..5),, (13.62), (14,35), (19.74)).
LMK
ROM
Stnge Addriveen Controllable Genevation
AU-Attenion
-H1 dipe-
AU Encoder
Spucic--Atettion
Driving Audio
Audie-Attentio
Audio Encoder
Face Attention
Output Video
Face Encoder
Cross Attention
Decoder
MEMO only
Inference Image
Noisee
XF
Denoising UNet
Trainable
Fronen
Figure 2: Overview of the two-stage AU-guided talking head generation framework. Stage stimulates the AU generation abilities audio-language models to get 24-dimensional AU se- quences from input audio, capturing facial motion dynamics. Stage 2 models the interaction be- tween AU and visual facial representations in a diffusion model to synthesize emotionally expressive, and lip- spphhncched facial animations.
where I denotes image example, is conditioning input (e.g., text, images, other modal- ity prompts), is the sampled Gaussian noise, and represents the diffusion timestep (with t to (n..( TT)). The noised latent 2; is obtained from 2g E(T by the forward diffusion process under a predefined schedule (Ho et al., 2020). During inference, we sample 2y ~ (0 and iteratively apply the learned reverse process conditioned e to obtain clean latent zo, which is then decoded into the final image via D(z)).
3.1 FRAMEWORK OVERVIEW
We present a two-stage framework generate emotional and realistic facial animations, conditioned on speech audio and reference portrait image, illustrated in Fig. 2.
Stage We elicit the emotion understanding and expression abilities of an audio language model (ALM), i.e., Audio-Qwen-Che (Chu et al., 2023) to disentangle AU information from audio, by fine-tuning with AU sequences as supervision. This enables the ALM to generate aliine AU sequences during inference.
We introduce an AU-driven controllable generation framework that enhances emotion rendering of diffusion model via AU representation, AU embedding, and AU-vision cross- attention mechanism. Compared to emotion labels, explicit AU sequences serve as more fine- fine-graine grained controllable signals drive emotional facial animations.
Stage 2.
3.2
STAGE 1: FACIAL AU DISENTANGLEMENT FROM AUDIO
This stage focuses on disentangling AU sequences from speech signals, which involves (1) under- standing the relations between speech acoustics and facial dynamics, i.e.. imagining vivid facial expressions solely from audio, and (2) AU generative modeling. Toward this end, we propose to ex- cavate the potential of emotion understanding and AU generation capabilities of a pretrained ALM. In detail, this task requires the ALM to generate sequence of AU activation vectors at fixed rate over timesteps:
AU 1.7 jauj, aug, ..... Jur))
au; E 12
(2)
In this work, the 5 AU sequences per second.
4

---

Published as a conference paper at ICLR 2026
Each vector au, (with 24 in our implementation), conveys a semantically meaningful repre- sentation of facial muscle activity, with each dimension corresponding to the intensity of a specific facial muscle group (e.g., lips, jaw, cheeks, and eyebrows). As fine-grained signal for conveying emotion, the AU sequence A 1 offers strong interpretabilliyy completeness, and compactness.
Praatil-Temporr AU Tokenization. In this work, we propose to directly generate AU sequences by viewing them natural language. Inspired by prior work that converts continuous signals into tokens (Borsos et al., 2023; Wang et al., 2023), we tokenize AU vectors rather than predict them directly with regression. Note that this design not only (I) aligns with the native language modeling task, but also (2) makes use of the abundant knowledge acquired during pre-training.
However, directly generating AU sequences is non-trivial due to their high density. On average, a 4-second video at 25 FPS yields an AU sequence of approximately 13K tokens, posing significant challenges for the context window and AU modeling capacity of the ALM. To address this issue, we propose a spatial-tempordl temporal tokenization scheme for AU sequences.
For the spatial aspect, the observation of the inherent sparsity of AU activations (i.e., statistically only 7 7/24 AUs are activated) motivates us to tokenize the raw dense AU vector au, into a compact set of index-intee -intensity pairs:
aiu;-
au,,))auu.. > A}.
(3)
where À denotes a sparsity coefficient. Note that a larger À encourages a sparser transformation, potentially at the cost of losing subtle facial expression details. This strategy" substantially reduces the output sequence length', simplifies prediction, and facilitates alignment with the discrete token format expected by the ALM. Although the number of active AUS per frame varies, the fixed index order preserves consistency, and the ALM is naturally capable of processing variable-length token sequences. Furthermore, balance temporal resolution and modeling complexity, we downsample the AU supervision sequence uniformly by a factor Ty. A smaller y results in lower temporal resolu- tion. This temporal compression preserves essential facial expression dynamics while significantly reducing computational overhead. Importantly, the raw audio is always processed at its original sam- pling rate; only the AU targets are temporally downsampled. Since generating AUS at 25 fps would exceed the ALM's sss context length, we adopt 5 fps practical compromise to keep the sequence tractable. This preserves the main key dynamics, while short-term losses are partly compensated by the continuity of AU trajectories and the generative model.
CoT-based Aiottonnttenn-Ae Generation. Based on the AU tokenization scheme, we transform dense AU vectors into discrete tokens compatible with language models, enabling direct generation of AU sequences by modeling voice-to-face dependencies. To further exploit the potential of AU modeling with an ALM, inspired by the success of CoT for multistep reasoning (Wei et al., 2022), propose coarse-to-fine generation strategy. Specifically, leveraging the correlation between emotional states and AU activation patterns, the ALM first predicts an emotion category (e.g.. hap- piness, sadness, and anger) from the audio input, and subsequently generales the corresponding AU sequence. Similar to CoT, this coarse-to-fine AU approach enhances AU disentanglement from audio by explicitly incorporating the intermediate emotion prediction as high-level context into the autoregressive decoding process. After generation, we convert aix, back to its dense form au; for the subsequent AU representation and embedding.
3.3 STAGE 2: AU-DRIVEN CONTROLLABLE GENERATION
While audio-driven diffusion models have shown impressive performance in synthesizing lip- synchronized talking heads, they often generate videos with neutral facial expressions, struggling with conveying rich emotional nuance and fine-graine -ne--gaaned facial detail, as illustrated in Fig. 1. this stage, address this limitation by explicitly controlling facial expressions using AU sequences gen- erated in Stage 1. AU sequences enable fine-grained expression control for generating emotionally expressive facial animations, ensuring consistent identity, accurate audio-lip synchronization, and temporally coherent facial movements.
Forr instance, a dense AU vector [0.38, 0.45, 0.0, 0.0, 0.84, 0. 90, 0] is tok- enized into sparse set { [0, ,38,, (1,.45), (21, 84), (22,.90) ). 'The sequence length can be reduced by 80.95% on average.
5

---

Published as a conference paper at ICLR 2026
To this end, propose controllable generation framework for synthesizing talking- -head videos conditioned on an AU sequence, an audio waveform, and a reference image. To enable faithful AU-driven control in a pre-trained diffusion model, introduce three key components: AU repre- sentation, context-aware AU embedding, and AU-vision interaction.
AU Representation. To align with the target frame rate, we apply linear interpolation to upsam- ple the generated low-resolution AU sequence in Stage by the factor 1/7. To further enhance expression control during emotion rendering, we design an AU-to-2D mapping that projects each AU into a 2D facial representation. We consider two options for this representation: keypoint-based Landmark (LMK) (Liu et al., 2024; 2022) and Rendering-od Beaeerinngo----eeeeee of-Mesh (RoM) (Liu et al., 2022). This mapping transforms ID AU sequences into structured and spatially interpretable format, provid- ing explicit facial topology as informative guidance for generation.
enhance temporal coherence in facial expressions, we de- sign context-aware embedding for AU representations. For the t-th target frame, we construct a local temporal window of length n to capture local context for modeling continuous expression dynamics. Specifically, we concatenate the AU features within this window and encode them using a lightweight temporal convolutional network, yielding the AU embedding:
Context-Aware AU Embedding.
C₁ ConVAu aal .... au, au
where au, denotes the upsampled AU representation the t-th frame, and E R is the resulting AU embedding, with as the embedding dimension. We collect the AU embedding corresponding each frame and get ελU (c,-... cr-]- This approach captures short-term expression patterns and promotes temporal smoothness. By integrating both historical and future AU cues, it enables fine-grained control over subtle and continuous facial movements critical for realistic animation.
AU-Vision Interaction. To achieve AU-driven controllable talking-head generation, we facilitate cross-modal interaction between context-aware AU embeddings and visual latents. Specifically, we inserted an AU adapter comprising multiple cross-atlention layers into the backbone of the pre- trained diffusion model, with tro-initializztionn for stable training. At each diffusion timestep t and spatial resolution the intermediate visual latent (o) attends to the AU embeddings AU via cross-attention:
+- CrossAtin (z(²) c
This cross-modal interaction mechanism enables the model to adaptively refine visual facial latents with AU-guided expression cues at every denoising step.
3.4 TRAINING AND INFERENCE
In Stage 1, given speech input, fine-tune the VLM using LoRA (Hu et al., 2022), with ground- truth AU sequences as supervision. Following the language modeling paradigm (Brown et al., 2020), train the model via next-token prediction using cross-entropy loss. At inference time, the model generates AU sequences conditioned on the input speech.
In Stage 2, we train the AU adapter while keeping all other components frozen, using the diffusion loss defined in Eqn. (1), where e comprises three conditions: audio, inference image, and AU em- bedding. To support unconditional modeling, each condition is randomly zeroed during training. Inspired by prior guidance strategies (Rombach et al., 2022; Yi et al., 2025), during inference we in- troduce disentanglement guidance strategy specifically tailored for AU conditioning. Specifically, we can separately modulate the corresponding strengths through adjustable guidance scales. The overall guidance process is formulated as:
é Lu(zz, cb AU) + " [[o(z., .+) Co(zi, 6)] + Av , c" "e") Co(zi, c" .0), (6)
where & denotes the null condition, and s AU and " are guidance scales controlling the influence of AU and other conditions, respectively.

---

Published as a conference paper at ICLR 2026
4 EXPERIMENTS
4.1 EXPERIMENT SETUP
Datasets. We conducted experiments on two widely used emotional talking face datasets: MEAD (Wang et al., 2020) and CREMA (Cao al., 2014). The MEAD dataset consists of 10,000 high-quality talking face video clips across eight emotion categories. We followed the standard identity-bane -based train/test splits adopted in EAMM (Wang et al., 2020) and EAT (Gan et al., 2023)
to ensure subject- indeeendenn evaluation. The CREMA dataset contains 7,442 video clips from 91 actors performing six emotional expressions with varying intensity levels. We followed the official subject-based splits provided by Bigioi et al. (Bigioi et al., 2024) and DAWN (Cheng et al., 2025)
to guarantee consistent evaluation protocols. To standardize the data, all videos were uniformly resampled to 25 frames per second and resized to 512 pixels. The corresponding audio was resampled to 16 kHz. For audio-visual synchronization, we extracted mel-spectrograms using a window size and hop length of 640 samples.
Implementation Details. In Stage 1, we trained Qwen-Audio-Chat on 4xNVIDIA A100 GPUs for approximately 24 GPU-hours with a learning rate of x 10-4. The sparsity coefficient and the downsampling factor were set to O and 0.2, respectively. In Stage 2, we adopted Hallo VI (Xu et al., 2024) and MEMO (Zheng et al., 2024) as base models. Both models were trained on 4XNVIDIA A100 GPUs for 12 GPU-hours. The learning rates were 5 x 10-6 for Hallo VI and x 10-5 5 for MEMO. The window size for context-aware AU embedding is 5 (n = 2 in Eqn. (4)). During inference, both Stage-1 AU prediction and Stage-2 AU-driven generation were performed on a single NVIDIA A100 GPU.
We employed multiple quantitative metrics for comprehensive evaluation. Specifically, PSNR, SSIM, and FID (Heusel et al., 2017) were used to assess visual quality, struc- tural similarity, and perceptual realism, respectively. For audio-visual synchronization, we used SyncNet scores (Chung & Zisserman, 2017) and Mouth Landmark Distance (M-LMD) evaluate lip-sync accuracy. Facial Landmark Distance (F-LMD) was applied to verify the preservation of fa- cial structure and head pose. We further evaluated emotional expression accuracy ACCemo using the emotion classification model (Gan et al., 2023). Additionally, Precision, Recall, and Mean Absolute Error (MAE) were used to measure AU regression accuracy in Stage I and to verify whether the generated videos in Stage 2 correctly followed the AU guidance.
Evaluation Metries.
4.2 ABLATION STUDIES
Stage 1: Ablation Study on CoT-based Coarse-to-Fine AU Generation. We conducted an ablation study to evaluate the effectiveness of the CoT strategy in Stage 1, as shown in Table 1. Four input- output configurations were tested: (1) Audio and emotion as input to predict AU sequences. (2) Audio only input A
Table 1: Performance of different input-output combina- tions on AU and emotion prediction. A: Audio input; E: Emotion label; AU: Action Unit sequence.
Input A+E
Oulput
Recall
Precision
Accuracy
Fi
MAE
ACC,m. %
AU
0.74
0.72
0.61
0.71
0.1928
A
0.63
0.65
0.50
0.62
0.2447
A
AU-
0.66
0.68
0.53
0.65 0.2200
51.76
E
E
AU
0.71
0.71
0.58
0.69
0.2085 67.01
to predict AU sequences. (3) Audio as input to predict AUs first, followed by emotion. (4) Audio as input to sequentially predict emotion first, followed by AUs (CoT strategy). Configuration (4), which follows our proposed CoT strategy, significantly outperformed the others in AU regression accuracy. The model achieved 67% accuracy in emotion classification, and 71 % precision and recall for AU predictions. For AU intensity regression, the Mean Absolute Error (MAE) was 0.2, which comparable to the inter-annotator variability reported FEAFA (Yan et al., 2019; Gan et al., 2022). This indicates that our predictions are within the level of human annotation consistency. The results show that introducing emotion as an intermediate step improves AU estimation and helps the model capture semantic information. This highlights the benefit of coarse-to- coarse-to-fine hicrarchical AU reasoning. We also note that generating valid dense AU vectors remains challenging, largely due to the limited context window of Qwen-Audio Audio-Chat and the redundancy in dense AU representations, further motivating our sparse tokenization design.
Stage 2: Ablation Study in Video Generation Stage As shown in Fig. 3. the effect of AU classifier-free free guidance (AU CFG scale) on FID, emotion accuracy, and AU regression (MAE) is evaluated. As the scale increases, emotion accuracy improves and MAE decreases, indicating better expression control through AU features. Emotion accuracy and MAE show similar trends, reflecting consistent expression quality. FID first decreases then increases, consistent with previous findings
7

---

Published as a conference paper at ICLR 2026
10.92
FED
0.0812
10.79
0.0808
10.66
0.0804
Tresh Alguoraal
Biue
10.53
0,0800
10.40
10.27
0.0792
10.14
0.0788
2,0
2.5
3.0
3,5
4.0
5.0
Easition
Figure 3: Impact of different AU guidance scales on visual quality (FID) and emotion expression (Emotion ACC and MAE). * the best quality- emotion trade-off.
AniPortrait
Echomimic
HalloVi
MEMO
AUHend
Qualitative comparison with SOTA methods on MEAD and CREMA.
Figure 4:
Table 2: Ablation results on different AU representations for video generation. The top-2 results are marked in bold (best) and underlined (2nd-best). indicates results reproduced under the same training data and settings for fair comparison. AU Seq: ID AU sequence; LMK: 2D keypoint-basee landmark; RoM: 2D rendering of mesh.
Dataset
MEAD
CREMA
Metrics
Sync (1)
PSNR (1) SSIM (1) FID (4) M/F-LMD (4)
Syne (1) PSNR (1) SSIM (1) FID (L) M/F-LMD (L)
Method
Baseline (MEMO')
6.9885
23.1910
0.7345
11,1237
2.0684/2.2473
6.0922
24.2808
0.7410
8,3881
1,96782.4296
MEMO AU Seq
6.7445
23.1666
0.7322
11,1105
1 1,90602220097
6.2857
24.2713
0.7394
8,4199 1,9525/2.4257
MEMO + LMK
6.6311
23.3466
0.7395
10.9671
1 1.86882. 1604
6,2050
24.2912
0.7413
8,2361
1.9313/2.9991
MEMO + RoM
6,6095
23,3585
0.7399
10.8701
1.86022..1336
6.1833
24.3113
0.7417
8,3352
1.33392.4025
like Stable Diffusion (Rombach al., 2022). The best trade-off occurs at scale 3.5, which balances visual quality and expression accuracy.
To study the impact of different AU representations, we conducted ablation experiments on MEAD and CREMA with identical training settings. Results are shown in Table 2. Note that the baseline MEMO model is retrained on our data split to ensure a consistent comparison. "MEMO + AU Seq" uses 1D AU sequence as input, whereas "MEMO LMK"" and "MEMO + RoM" adopt 2D AU representations via keypoint landmarks (LMK) and mesh renderings (RoM). This design allows us to compare the effectiveness of spatial versus non-spatial AU encodings under the same framework. From the results, it can be observed that while the model can learn from AU features in all forms, the ID AU sequence provides weaker conditioning for the diffusion model, leading to suboptimal performance. In contrast, the 2D representations provide stronger spatial priors, resulting in sig- nificant improvements across PSNR, SSIM, FID, and LMD metrics on both datasets. Interestingly, the Sync score slightly drops when using 2D AU inputs. One possible reason is that the additional spatial information may increase the model's focus on expression accuracy, slightly compromising the temporal alignment between lip movements and audio. Despite this, overall generation quality and expression consistency are notably improved with 2D AU inputs.
4.3 COMPARISON WITH STATE-OF-THE OF-THE-ARTE
Quantitative Comparison We present a quantitative comparison against state-of-the-art methods on the MEAD and CREMA datasets, considering only those that use the same input setting (see Table 3). Overall. AUHead achieves strong performance across most metrics. In particular, we achieve higher PSNR and SSIM scores, indicating improved visual fidelity and structural coherence, as well as lower FID, demonstrating that AU-based expression modeling enhances the perceptual realism of generated videos. In addition, AUHead achieves lower M-LMD and F-LMD scores, indicating more accurate lip geometry and improved facial structure preservation. These results suggest that AU conditioning enhances the fidelity of mouth movements and expression details. The slight drop in Sync confidence may be caused by timing mismatches between the predicted AUs and the speech signal. However, human evaluation (cf., Section 4.3.1) indicates no noticeable audio-lip misalignment, suggesting that this limitation has minimal impact in real-world applications.
8

---

Published as a conference paper at ICLR 2026
Table 3: Comparison of state-of-the-aar audio-driven talking head generation methods on MEAD and CREMA benchmarks. The best results are marked in bold (best) and underlined (2nd-best). M/F-LMD denotes mouth/face landmark distance." indicates results reproduced under the same training data and settings for fair comparison.
Dataset
MEAD
CREMA
Metrics
Sync (t)
PSNR (f) SSIM (1) FID (L) M/F-LMD (U)
Syne (1)
PSNR (1)
SSIM (t) FID (.) M/F-LMD ())
Method
Wav2lip (2020)
8.7778
23.0296
0.7395
32.8043 2.6386/2
6.7109
24.2081
0.7533
25.6218 2.3794/2.4439
Audio2Head (2021)
6.7809
19.6335
0.6056
35.1387 3.3227/3.5371
5.7673
21,1881
0.6533
25,0426 2.4033/3 2905
Sadtalker (2023)
7.0015
20.9015
0,6660
28.7729
(.88402.8841
5,3062
22.0340
0.6814
23,6910
2.3035/2.8571
V Express (2024)
7.2952
15.9265
0.6023
32.2410
2.5586/2.5226
6,0578
21.2292
0.6952
18.1609
2.2274/2.4991
AniPortrait (2024)
3.0734
20.2589
21.5914 3,2615/3.3014
2.6863
22.4807
12.2362
2.2683/2.735
EDTalk (2024)
8.0570
22.4354
0,7251
21,9435 2.8209/2.4370
6,3703
22.6067
0.7400
19.6080
2.2614/2.4669
Echomimic (2025)
5,3461
21.6390
0.6978
13.9435
2.4156/2.7941
4.5033
22.3503
0.6952
11.9544
2.2285/2. 8633
HalloV2 (2025)
6,3832
21 4575
0.6779
15. 15.6245 2.3489/2.5880
5,0140
23,2052
0.7129
10,7165
2.2149/2.5266
Sonic (2025)
8,0988
28.1874
0.71188
14.2623
2.5822/2.4012
6.8620
23,0787
0.7341
9,9440
1.9454/2.663
DICE-Talk (2025)
7.3073
19.7293
0.6279
1..4995.. 3.1125/3.3559
5.7601
21.4570
0.6675
17.8824
2,8910/3 2486
HalloV1" (2024)
4,9512
22.0258
0.7101
13.0673
2.5016/2.5888
4,5161
23:2809
0.7074
10.0336
2.1814/2.6313
AUHead (HalloV1)
6.0201
22.0132
0.7113
12.842
2.3836/2.4599
4,7100
23.0818
0.7201
9.7086
2.2964/2.5337
MEMO" (2025)
6,9885
23.1910
0.7345
11.1237
2.0684/2.2473
6.0922
24.2808
0.7410
8.3881
1.9678/2.4296
AUHead (MEMO)
6.6311
23.3466
0.7395
10,9671
1.8608/2.1004
6.2050
24.2912
0. 0.7413
8.2361
1.9313/2.
Qualitative Comparison As demonstrated in Fig. 4, we conducted qualitative comparisons with state-of-the-an tht methods, including AniPortrait (Wei al., 2024), Echomimic (Chen et al., 2024), HalloV1 (Xu et al., 2024), and MEMO (Zheng et al., 2024), on both the MEAD (Wang et al., 2020) and CREMA (Cao al., 2014) datasets. While baseline models often suffer from blurred textures, distorted shapes, or emotionally flat expressions, AUHead produces clear, expressive, and visually coherent results. These empirical results show that integrating AU guidance into diffusion models is both effective and practical for expressive talking head synthesis.
To further demonstrate the effectiveness of AU-guided generation, we compare AUHead and the MEMO baseline under identical conditions: the same audio, identity image, and selected frames. The AUHead model uses AU features regressed from Qwen-Audio-Char as guidance. As shown in Fig. 5, AUHead produces more vivid and nuanced facial expressions, even without explicit emo- tion instructions. The results reveal finer expression details, including accurate lip movements and vivid eye dynamics, demonstrating that AU conditioning can drive realistic facial animation. More detailed examples can be found in Supplementary Section B.
To evaluate whether the generated expressions align with AU guidance, we present video frames alongside their corresponding 2D AU representations, including 2D keypoint-based landmark (LMK) and 2D rendering of mesh (RoM). As shown in Fig. 6, the generated expressions closely match the input AU features, indicating accurate emotional representation. The results show that 2D AU representations offer effective emotion encoding and reliable guidance for facial animation.
Neatral AUHaad
Surprise LMK
MEMO
Happy ROM
Happy
AUFlen
Figure 6: Visualization of generated frames and their corresponding AU-based 2D representations.
Figure 5: Facial animation compari- under neutral audio using AU from ALM.
To further validate the generalization ability and temporal stability of AUHead, Fig. 7 presents sev- eral 10-second video examples generated unseen data. Each row corresponds a different audio

---

Published as a conference paper at ICLR 2026
input, and each frame is randomly sampled from the corresponding second of the generated video. This setting allows us to examine both the consistency within long sequences and the model's ' ro- bustness across diverse conditions. As shown in the figure, AUHead successfully handles different visual styles, including sketch, oil-painting, and realistic faces, while producing fine-grained and temporally coherent facial expressions. These results demonstrate that AUHead maintains high syn- thesis quality, identity preservation, and stable motion dynamics over extended time spans.
Is
38 4s
5s
6s
7s
8s
9s
10s
Figure 7: Visualization of AUHead's generalization across 10-second sequences. Each row corre- sponds unique combination of unseen audio and new target identity. For each sequence, one frame is randomly sampled from each second of the generated video. The examples cover three visual styles, line-art sketches, oil oil-painting portraits, and realistic face images (Zhang et al., 2021) to illustrate the model's 's behavior under varied input domains.
4.3.1 USER STUDY
We conducted a user study evaluate the perceptual quality of the generated talking-face videos. Evaluation samples consisted of 32 clips randomly selected from the MEAD test set, covering iden- tities and 8 emotions. Each participant compared results from AUHead and the state-of-the-art model HalloV2 (Cui et al., 2024a) under the same conditions. The study followed a blind setting, where participants were not informed of the method names. For each video pair, participants were asked to choose which result was better or indicate if there was no significant difference, based on four aspects: emotional expression, video quality, audio-lip syn- chronization, and overall performance. A total of 25 participants rated all video pairs, so that every pair received the same number of ratings. As shown in Table 4, AUHead consistently received the highest preference scores across all metrics, confirming its advantage in generating realistic, expres- sive, and well- cell-synchromized talking-face videos. Participants also mentioned that AUHead delivered clear and coherent lip synchronization.
Table 4: User study evaluating the quality and emo- tional expressiveness of the generated talking heads. The better results are highlighted in bold.
User Preference
HalloV2 AUHcad Same
Emotional Expression
18.88%
64.63%
16.49%
Video Quality
21.28%
63.63%
15.09%
Audio-Lip Sync
13.75%
71.00%
15.25%
Overall Performance 16.13%
67.75%
16.12%
CONCLUSION
In this paper, we propose AUHead, two-stage audio-driven talking head generation framework that uses AU representations as an intermediate bridge. In the first stage, fine-grainee AU information disentangled from audio signals using an ALM, enabling structured and interpretable control. In the second stage, the AU features drive the generation of realistic and emotionally expressive facial animations with consistent identity. Results on the MEAD and CREMA datasets show that AUHead achieves superior performance compared to existing methods, highlighting the effectiveness of AUs as a controllable and emotion-aware representation. Beyond performance, our work suggests that introducing interpretable intermediate spaces, like AUs, can offer general strategy for more trollable and reliable cross- modal generation. In future work, we aim to extend AUHead to handle in-the-wil scenarios with diverse head poses and backgrounds.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
This work was supported by the National Natural Science Foundation of China (62320106007, 62236006, 62521007). The first author would like thank the China Scholarship Council (CSC) for the financial support during the doctoral research visit at the National University of Singapore.
ETHICS STATEMENT
Our work on talking head generation makes it easier to create expressive and accessible visual con- tent, with potential benefits for communication, education, and creative applications. At the same time, it poses risks such as possible misuse for deepfakes, biases generated faces, and the envi- ronmental impact of large-scale training. We also recognize potential concerns regarding privacy and security when handling facial data. Addressing these challenges requires responsible dataset curation, fairness evaluation, clear documentation of research practices, and careful deployment.
REPRODUCIBILITY STATEMENT
repository link a005990000111 /AOHead_TCLRR containing the implementa- tion used our experiments. In addition, both the main paper and the appendix detail the model architecture, training objectives, and evaluation settings. Together, these resources make it possible for others to replicate our work and verify the reported results.
To promote
(https: / / thub. reproducibility, we provide
REFERENCES
Dan Bigioi, Shubhajit Basak, Michat Stypuikowski, Maciej Zieba, Hugh Jordan, Rachel McDonnell, and Peter Corcoran. Speech driven video editing via an andio-conditione diffusion model. Image and Vision Computing, 142:104911, 2024.
Zalán Borsos, Raphaël Marinier, Damien Vincent, Eugene Kharitonov, Olivier Pietquin, Matt Shar- ifi, Dominik Roblek, Olivier Teboul, David Grangier, Marco Tagliasacchi, et al. Audiolm: language modeling approach to audio generation. ICASSP, 31:2523-2533, 2023.
Tom Brown, Benjamin Mann, Nick Ryder. Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. In NeurIPS, pp. 1877-1901, 2020.
Houwei Cao, David G. Cooper, Michael K. Keutmann, Ruben C. Gur, Ani Nenkova, and Ragini Verma. CREMA-D: crowd-sourced emotional multimodal actors dataset. IEEE Trans. Affect. Comput., 5(4):377-390, 2014.
Zhiyuan Chen, Jiajiong Cao, Zhiquan Chen, Yuming Li, and Chenguang Ma. Echomimic: Lifelike audio-driven portrait animations through editable landmark conditioning, 2024.
Hanbo Cheng, Limin Lin, Chenyu Liu, Pengcheng Xia, Pengfei Hu, Jiefeng Ma, Jun Du, and Jia Pan. Dawn: Dynamic frame avatar with non-autoregressive diffusion framework for talking head video generation. arXiv preprint 2rXiv:2410.13722 2024.
Hanbo Cheng. Limin Lin, Chenyu Liu, Pengcheng Xia, Pengfei Hu, Jiefeng Ma. Jun Du, and Jia Pan. Dawn: Dynamic frame avatar with non- autorrrrreeee diffusion framework for talking head video generation, 2025. URL https: / http///axvvvv/ee//////eeeeeaaeve
http::s:s//.a.l.ooo////tsss////s wwwp:/s/////aaaaaa...oo..o 2410 13726.
Yunfei Chu, Jin Xu, Xiaohuan Zhou, Qian Yang, Shiliang Zhang. Zhijie Yan, Chang Zhou, and Jingren Zhou Qwen-audio: Advancing universal audio understanding via unified large-scale audio-language models. arXiv preprint arXiv:2311.07911 2023.
Joon Son Chung and Andrew Zisserman. Out of time: automated lip syne in the wild. ACCV, pp. 251-263. Springer, 2017.
11

---

Published as a conference paper at ICLR 2026
Jiahao Cui, Hui Li, Yao Yao, Hao Zhu, Hanlin Shang, Kaihui Cheng, Hang Zhou, Siyu Zhu, and Jingdong Wang. Hallo2: Long-duratin and high-resolution audio-driven portrait image anima- tion, 2024a.
Jiahao Cui, Hui Li, Yun Zhan, Hanlin Shang, Kaihui Cheng, Yuqi Ma. Shan Mu, Hang Zhou, Jingdong Wang, and Siyu Zhu. Hallo3: Highly dynamic and realistic portrait image animation with diffusion transformer networks. arXiv e-prints, pp. arXiv-2412, 2024b.
Nilaksh Das. Saket Dingliwal, Srikanth Ronanki, Rohit Paturi, Zhaocheng Huang, Prashant Mathur, Jie Yuan, Dhanush Bekal, Xing Niu, Sai Muralidhar Jayanthi, Xilai Li, Karel Mundnich, Monica Sunkara, Sundararajan Srinivasan, J. J. Han, and Katrin Kirchhoff. Speechverse: A large-scale generalizable audio language model. CoRR, abs/2405.0822 08295, 2024, doi: 10.48550/ARXIV2400. 08295. URL https: / htt//oii.oggg..o org/10. 48550/ aar.vv.arrvv....o/ooo 2405. 08295.
Nikita Drobyshev, Antoni Bigata Casademunt, Konstantinos Vougioukas, Zoe Landgraf, Stavros Petridis, and Maja Pantic. Emoportraits: Emotion-enhanced multimodal one-shot head avatars. In CVPR, pp. 1498-8501 2024.
Paul Ekman and Wallace V Friesen. Facial action coding system. Environmental Psychology & Nonverbal Behavior, 1978.
Kasra Ferdowsifard, Shraddha Barke, Hila Peleg, Sorin Lerner, and Nadia Polikarpova. Loopy: interactive program synthesis with control structures. Proceedings of the ACM on Programming Languages, ((0OPSLA)) 2021.
Wei Gan, Jian Xue, Ke Lu, Yanfu Yan, Pengcheng Gao, and Jiayi Lyu. Feafa+: an extended well- annotated dataset for facial expression analysis and 3d facial animation. In ICDIP, volume 12342, pp. 307-316, 2022.
Yuan Gan, Zongxin Yang. Xihang Yue, Lingyun Sun, and Yi Yang. Efficient emotional adaptation for audio-driven talking-head generation. In ICCV, pp. 22634-22645, October 2023.
Xin Gao, Li Hu, Siqi Hu, Mingyang Huang, Chaonan Ji, Dechao Meng, Jinwei Qi, Penchong Qiao, Zhen Shen, Yafei Song, et al. Wan-s2v: Audio-driven cinematic video generation. arXiv preprint arXiv:2508. 18621, 2025.
Jiazhi Guan, Zhanwang Zhang, Hang Zhou, Tianshu Hu, Kaisiyuan Wang. Dongliang He, Haocheng Feng, Jingtuo Liu, Errui Ding, Ziwei Liu, et al. Stylesyne: High-fidelity generalized and person- alized lip syne in style-based generator. In CVPR, pp. 1505-1515, 2023.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. In NeurIPS, 2017.
Jonathan Ho, Ajay Jain, and Picter Abbeel. Denoising diffusion probabilistic models. In NeurIPS, pp. 6840-6851, 2020.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. In ICLR, 2022.
Shujie Hu, Long Zhou, Shujie Liu, Sanyuan Chen, Lingwei Meng, Hongkun Hao, Jing Pan, Xunying Liu, Jinyu Li, Sunit Sivasankaran, Linquan Liu, and Furu Wei. Wavllm: Towards robust and adaptive speech large language model. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Findings of EMNLP. pp. 4552-4572 2024.
Xiaozhong Ji, Xiaobin Hu, Zhihong Xu, Junwei Zhu, Chuming Lin, Qingdong He, Jiangning Zhang, Donghao Luo, Yi Chen, Qin Lin, et al. Sonic: Shifting focus to global audio perception in portrait animation. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 193- 203, 2025.
Xinya Ji, Hang Zhou, Kaisiyuan Wang, Qianyi Wu, Wayne Wu, Feng Xu, and Xun Cao. Eamm: One-shot emotional talking face via based emotion- aware motion model, 2022. URL https://arxe org/ atapaabs//22//22 15278.....
12

---

Published as a conference paper at ICLR 2026
Jianwen Jiang, Weihong Zeng, Zerong Zheng, Jiaqi Yang, Chao Liang, Wang Liao, Han Liang, Yuan Zhang, and Mingyuan Gao. Omnihuman- 1.3: Instilling an active mind in avatars via cognitive simulation. arXiv preprint arXiv: 19209, 2025.
Xin Jin, Yichuan Zhong, and Yapeng Tian. TP-blend: Textual-prompt attention pairing for precise object-style blending diffusion models. Transactions on Machine Learning Research, 2025. ISSN 2835-8856. URL https: : /pppneeeeeeie.eeoeee wwwerrvvieewee... new. htt//forune forum? d== BUOBZE.
Diederik Kingma, Max Welling, et al. Auto-cncoding variational bayes, 2013.
Zhe Kong, Feng Gao, Yong Zhang, Zhuoliang Kang, Xiaoming Wei, Xunliang Cai, Guanying Chen, and Wenhan Luo. Let them talk: Audio-driven multi-person conversational video generation. arXiv preprint arXiv:2505.2260 22647, 2025.
Chunyu Li, Chao Zhang. Weikai Xu, Jingyu Lin, Jinghui Xie, Weiguo Feng, Bingyue Peng, Cunjian Chen, and Weiwei Xing. Latentsync: Taming audio-conditioned latent diffusion models for lip sync with syncnet supervision. arXiv preprint arXiv:2412.09262. 2024.
Bin Lin, Yanzhen Yu, Jianhao Ye, Ruitao Lv, Yuguang Yang. Ruoye Xie, Pan Yu, and Hongbin Zhou. Takin-ada: Emotion controllable audio-driven animation with canonical and landmark loss optimization. arXiv preprini arXiv:2410. 14283, 2024.
Haiyang Liu, Zihao Zhu, Naoya Iwamoto, Yichen Peng. Zhengqing Li, You Zhou, Elif Bozkurt, and Bo Zheng. Beat: A large-scale semantic and emotional multi-modal dataset for conversational gestures synthesis. arXiv preprint wrXiv:2203.05299 2022.
Haiyang Liu, Zihao Zhu, Giorgio Becherini, Yichen Peng, Mingyang Su, You Zhou, Xuefei Zhe, Naoya Iwamoto, Bo Zheng, and Michael Black. Emage: Towards unified holistic co-speech gesture generation via expressive masked audio gesture modeling. In CVPR, pp. 1144-1154, 2024.
Kai Liu, Jungang Li, Yuchong Sun, Shengqiong Wu, Jianzhang Gao, Daoan Zhang, Wei Zhang, Sheng Jin, Sicheng Yu, Geng Zhan, et al. Javisgpt: A unified multi-modal Ilm for sounding-video comprehension and generation. arXiv preprint arXiv:2512.22905: 2025a.
Kai Liu, Wei Li, Lai Chen, Shengqiong Wu, Yanhao Zheng, Jiayi Ji, Fan Zhou, Rongxin Jiang, Jiebo Luo, Hao Fei, and Tat-Seng Chua. Javisdit: Joint audio-video diffusion transformer with hierarchical spatio-temporal prior synchronization, 2025b. URL htttt:s:///axxv..rrreee / 2503. 23377.
Yunfei Liu, Lijian Lin, Fei Yu, Changyin Zhou, and Yu Li. Moda: Mapping-once audio-driven portrait animation with dual attentions. In ICCV, pp. 23020-23029, 2023.
Jiayi Lyu, Xing Lan, Guohong Hu, Hanyu Jiang, Wei Gan, and Jian Xue. Ftau: Towards emo- tional talking head generation via facial action unit. In 2024 IEEE International Conference Multimedia and Expo (ICME), pp. 1-6, 2024, doi: 10.1 10011 106675555
Jiayi Lyu, Xing Lan, Guohong Hu, Hanyu Jiang, Wei Gan, Jinbao Wang, and Jian Xue. Multimodal emotional talking face generation based on action units. IEEE Transactions on Circuits and Systems for Video Technology, 35(5):4026-4038, 2025. doi: 10. 10.110//CSV.2222.55333..
Xingpei Ma, Jiaran Cai, Yuansheng Guan, Shenneng Huang, Qiang Zhang, and Shunsi Zhang. Play- mate: Flexible control of portrait animation via 3d-implicit space guided diffusion. arXiv preprint 2tli:2502.0720333 2025.
Rang Meng. Xingyu Zhang, Yuming Li, and Chenguang Ma. Echomimiev2: Towards striking, simplified, and semi-body human animation. In CVPR, pp. 5489-5498, 2025.
Fatemeh Nazarieh, Zhenhua Feng, Diptesh Kanojia, Muhammad Awais, and Josef Kittler. Por- traittalk: Towards customizable one-shot audio-to-talking face generation. arXiv preprint arXiv:2412.07754, 2024.
13

---

Published as a conference paper at ICLR 2026
K R Prajwal, Rudrabha Mukhopadhyay, Vinay P. Namboodiri, and C.V. Jawahar. A lip syne expert is all you need for speech to lip generation in the wild. ACM MM, 2020. doi: 10.1145/3394171 3413532. URL http://///// /dx. doi. org. 11.....1133331111..... 1111131...3322 3413532.
Leigang Qu, Feng Cheng, Ziyan Yang, Qi Zhao. Shanchuan Lin, Yichun Shi. Yicong Li, Wenjie Wang, Tat-Seng Chua, and Lu Jiang. Vincie: Unlocking in in-context image editing from video.
arXiv preprint arXiv:2506. 10006./0944 10941, 2025a.
Leigang Qu, Haochuan Li, Wenjie Wang, Xiang Liu, Juncheng Li, Liqiang Nie, and Tat-Seng Chua. Silmm: Self-improving large multimodal models for compositional text-to-image generation. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 18497-18508, 2025b.
Leigang Qu Ziyang Wang, Na Zheng, Wenjie Wang, Liqiang Nie, and Tat-Seng Chua. Ttom: Test-time optimization and memorization for compositional video generation. arXiv preprint arXiv:25/0.07940. 2025c.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- 2022.
resolution image synthesis with latent diffusion models. In CVPR, pp.
Fei Shen and Jinhui Tang. Imagpose: unified conditional framework for pose-guided person generation. Advances in neural information processing systems, 37:6246-6266, 2024.
Fei Shen, Hu Ye, Jun Zhang, Cong Wang. Xiao Han, and Yang Wei. Advancing pose- gose-guided image synthesis with progressive conditional diffusion models. In The Twelfth International Confer- ence on Learning Representations, 2024. URL https: pasp::///penreeeee /pppreeeeeeee.. net. forum? id= rHzapPnCgT.
Fei Shen, Xin Jiang, Xin Hc, Hu Ye, Cong Wang, Xiaoyu Du Zechao Li, and Jinhui Tang. Imagdressing-vi. Customizable virtual dressing. In Proceedings of the AAAI Conference Artificial Intelligence, volume 39, pp. 6795-6804, 2025a.
Fei Shen, Cong Wang. Junyao Gao, Qin Guo, Jisheng Dang, Jinhui Tang, and Tat-Seng Chua. Long- term talkingface generation via motion-prior conditional diffusion model. In ICML, 2025b.
Shuai Shen, Wenliang Zhao, Zibin Meng, Wanhua Li, Zheng Zhu, Jie Zhou, and Jiwen Lu. Difftalk: Crafting diffusion models for generalized audio-driven portraits animation. CVPR, pp. 1982- 1991, 2023.
Michat Stypuikowski, Konstantinos Vougioukas, Sen He, Maciej Zięba, Stavros Petridis, and Maja Pantic. Diffused heads: Diffusion models beat gans talking-face generation. In CVPR, pp- 5091-5100, 2024.
Shuai Tan, Bin Ji, Mengxiao Bi, and Ye Pan. Edtalk: Efficient disentanglement for emotional talking head synthesis. In Ales Leonardis, Elisa Ricci, Stefan Roth, Olga Russakovsky, Torsten Sattler, and Güi Varol (eds.), ECCV, pp. 398 416, 2024a.
Shuai Tan, Bin Ji, Yu Ding, and Ye Pan. Say anything with any style. In AAAI, pp. 5088-5096, 2024b.
Weipeng Tan, Chuming Lin, Chengming Xu, FeiFan Xu, Xiaobin Hu, Xiaozhong Ji, Junwei Zhu, Chengjie Wang, and Yanwei Fu. Disentangle identity, cooperate emotion: Correlation-awan emotional talking portrait generation. Proceedings of the 33rd ACM International Conference on Multimedia, pp. 9987-9995. 2025.
Changli Tang, Wenyi Yu, Guangzhi Sun, Xianzhao Chen. Tian Tan, Wei Li, Lu Lu, Zejun Ma, and Chao Zhang. SALMONN: towards generic hearing abilities for large language models. In ICLR, 2024. URL https:tt///..........ttt http///ppeeeeeeeeeee//e/eeeeeeee net forum? d= 4rn (rrn7HpKK 7HKKk.
Linrui Tian, Qi Wang, Bang Zhang, and Liefeng Bo. Emo: Emote portrait alive generating pressive portrait videos with audio2video diffusion model under weak conditions. In ECCV, pp- 244-260. Springer, 2024.
14

---

Published as a conference paper at ICLR 2026
Chengyi Wang. Sanyuan Chen, Yu Wu, Ziqiang Zhang, Long Zhou, Shujie Liu, Zhuo Chen, Yanqing Liu, Huamingg Wang, Jinyu Li, et al. Neural codec language models are zero-shot text to speech synthesizers. arXiv preprint arXt::331.02211, 20233.03.
Cong Wang, Kuan Tian, Jun Zhang, Yonghang Guan, Feng Luo, Fei Shen, Zhiwei Jiang, Qing Gu, Xiao Han, and Wei Yang. V-express: Conditional dropout for progressive training of portrait video generation. 2024.
Kaisiyuan Wang, Qianyi Wu, Linsen Song, Zhuoqian Yang, Wayne Wu, Chen Qian, Ran He, Yu Qiao, and Chen Change Loy. Mead: A large-scale audio-visual dataset for emotional talking- face generation. In ECCV, pp. 700-717. Springer, 2020.
Suzhen Wang. Lincheng Li, Yu Ding, Changjie Fan, and Xin Yu. Audio2head: Audio-driven one- shot hhlk generation with natural head motion. arXiv preprint arXiv:2107.09223, 2021. 2021.
Yaohui Wang, Di Yang, Francois Bremond, and Antitza Dantcheva. Latent image animator: Learn- ing to animate images via latent space navigation. arXiv preprini 0rX1v:2203.09043, 2022.
Yuchi Wang, Junliang Guo, Jianhong Bai, Runyi Yu, Tianyu He, Xu Tan, Xu Sun, and Jiang Bian. Instructavatar: Text-guided emotion and motion control for avatar generation. In Toby Walsh, Julie Shah, and Zico Kolter (eds.), AAAI, pp. 8132-8140, 2025.
Huawei Wei, Zejun Yang. and Zhisheng Wang. Aniportrait: Audio-driven synthesis of photorealistic portrait animations, 2024.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi. Quoc V Le, Denny Zhou, et al. Chain-of- ohotthtgtt prompting clicits reasoning in large language models. In NeurIPS, pp. 24824-24837, 2022.
Y Wu, Y Deng, Yang, F Wei, Q Chen, and X Tong. Animatable 3d-aware face image generation for video avatars. arXiv preprint 0rXiv:2210.06465. 2022.
Mingwang Xu, Hui Li, Qingkun Su, Hanlin Shang, Liwei Zhang, Ce Liu, Jingdong Wang, Yao Yao, and Siyu zhu. Hallo: Hierarchical audio-driven visual synthesis for portrait image animation, 2024.
Zunnan Xu, Zhentao Yu, Zixiang Zhou, Jun Zhou, Xiaoyu Jin, Fa-Ting Hong, Xiaozhong Ji, Junwei Zhu, Chengfei Cai, Shiyu Tang, et al. Hunyuanportrai:: Implicit condition control for enhanced portrait animation. In CVPR, pp. 15909-15919 2025.
Yanfu Yan, Ke Lu, Jian Xue, Pengcheng Gao, and Jiayi Lyu. Feafa: A well-annotated dataset for facial expression analysis and 3d facial animation. In ICME Workshop, pp. 96-101, 2019.
Sejong Yang, Seoung Wug Oh, Yang Zhou, and Seon Joo Kim. If-mdm: Implicit face motion diffu- sion model for high-fidelity realtime talking head generation. arXiv preprint arXiv:2412.040000 2024.
Hongwei Yi, Tian Ye, Shitong Shao, Xuancheng Yang, Jiantong Zhao Hanzhong Guo, Terrance Wang, Qingyu Yin, Zeke Xie. Lei Zhu, et al. Magicinfinite: Generating infinite talking videos with your words and voice. arXiv preprint arXiv:2503. 05978, 2025.
Haojie Zhang, Zhihao Liang, Ruibo Fu, Bingyan Liu, Zhengqi Wen, Xuefei Liu, Jianhua Tao, and Yaling Liang. Efficient long-duration talking video synthesis with linear diffusion transformer under multimodal guidance. arXiv preprint arXhv:2411.16745 2024.
Wenxuan Zhang, Xiaodong Cun, Xuan Wang, Yong Zhang, Xi Shen, Yu Guo, Ying Shan, and Fei Wang. Sadtalker: Learning realistic 3d motion coefficients for stylized audio-driven single image talking face animation. 2023.
Zhimeng Zhang, Lincheng Li, Yu Ding, and Changjie Fan. Flow-guided one-shot talking face gen- cration with a resouuutnn audio-visuee dataset. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 3661-3670, 2021.
15

---

Published as a conference paper at ICLR 2026
Longtao Zheng, Yifan Zhang, Hanzhong Guo, Jiachun Pan, Zhenxiong Tan, Jiahao Lu, Chuanxin Tang, Bo An, and Shuicheng Yan. Memo: Memory-guided diffusion for expressive talking video generation, 2024. URL httsst/::/ wwtpab/i.org/b///////e/a.b.bbaas
04448.
Hang Zhou, Yasheng Sun, Wayne Wu, Chen Change Loy, Xiaogang Wang, and Ziwei Liu. Pose- controllable talking face generation by implicitly modularized audio-visual representation. In CVPR. pp. 4176-4186, 2021.
Zhenglin Zhou, Fan Ma, Hehe Fan, and Tat-Seng Chua. Zero-I-to-a: Zero-shot one image to ani- matable head avatars using video diffusion. In CVPR, 2025.
16

---

Published as a conference paper at ICLR 2026
A DETAILED INTRODUCTION TO FACIAL ACTION UNIT
Facial Action Units (AUs) are defined in the Facial Action Coding System (FACS), which was originally developed by Ekman & Friesen (1978). FACS provides standardized framework for describing facial muscle movements. It decomposes facial expressions into 44 individual AUs, each corresponding to specific muscle activations. These units can appear independently or in combi- nation to form complex expressions. To simplify expression modeling, the researchers redefined and selected 24 representative AUS commonly observed facial expressions (Yan et al., 2019; Gan et al., 2022). A complete list of these 24 AUs is provided in Table 5. In FEAFA Yan et al. (2019); Gan et al. (2022), each AU is annotated with a continuous value ranging from to I, where 0 in- dicates no activation and 1 indicates maximum activation. As shown Fig. 8, each AU is visually illustrated across different intensity levels, providing clear examples of activation changes across frames.
AU 2
AU17
02
04
02
04
06
08
AU
AU19
03
03
92
84
08
AU
62
AU20
AU9
AU21
93
55
07
085
AU10
AU22
02
04
05
03
01
63
DB
AU13
AU23
AU14
AU24
Figure 8: Visual examples of AU activations at different intensity levels in the FEAFA dataset. Each row corresponds one AU, and columns show variations from neutral (left) to highly activated (right).
To ensure the quality of our AU data, we an interactive AU validation tool during data construc- tion. As shown in Fig. 9, this tool displays the aligned AU values, corresponding face images, and their 2D mesh projections for each frame. Although we do not manually annotate AUs, this visual interface allows us to carefully inspect whether the automatically extracted AU values align with the visual facial changes.
B IMPLEMENTATION DETAILS OF EMOTION GUIDED GENERATION
This section provides additional details and analysis corresponding to the emotion guided genera- tion setting discussed in Section Experiment Fig. and Fig. 10, where AUHead is compared to the MEMO baseline under identical input conditions. Specifically, both models receive the same neu- tral audio and reference face image reflecting the target emotion. For AUHead, although the audio lacks emotional cues, we leverage the structure of the MEAD dataset. which includes repeated sen- tences performed with different emotional expressions. Using this property, extract sparse AU
17

---

Published as a conference paper at ICLR 2026
Table 5: Explanation of redefined AUs from the FEAFA+ dataset.
FEAFA's Definition of AUS
AU1: Left Eye Close
AU2: Right Eye Close
AU3: Left Upper Lid Raiser
AU4: Right Upper Lid Raiser
AU5: Left Brow Lowerer
AU6: Right Brow Lowerer
AU7: Left Outer Brow Raiser
AU8: Right Outer Brow Raiser
AU9: Jaw Drop
AU10: Left Jaw Sideways
AU11: Right Jaw Sideways
AU12: Left Lip Corner Puller
AU13: Right Lip Coreer Puller
AU14: Left Lip Stretcher
AU15: Right Lip Stretcher
AU16: Upper Lip Suck
AU17: Lower Lip Suck
AU18: Jaw Thrust
AU19: Upper Lip Raiser
AU20: Lower Lip Depressor
AU21: Chin Raiser
AU22: Lip Pucker
AU23: Puff
AD24: Nose Wrinkler
Keretattones
Crem prurt
Figure 9: Frame level AU verification interface. Each frame displays the annotated AU values, the original face image, and its mesh based projection, enabling cross-modal consistency checking.
sequences from emotional versions of the same sentence via OowenAAudi--e (Stage 1). These se- quences then temporally aligned with the neutral audio through linear interpolation and reshaped into 2D AU representations, which serve as AU guidance input to AUHead.
After generation, extract the same set of keyframes from both models for visual comparison. It important to note that the reference image plays dominant role in talking head generation, as emphasized in the MEMO framework (Zheng et al., 2024). Nevertheless, AU representations offer complementary guidance that modulates subtle expression dynamics. They help the model gen- erate more natural and emotionally aligned movements. Compared MEMO, AUHead produces clearer lip shapes, more accurate brow motions, and realistic eye activity. Moreover, AUHead with emotional AU guidance shows clearer and more emotionally rich expressions than the neutral AU setting, further highlighting the benefit of AU features during generation.
C PROMPT TEMPLATE
In Stage 1, fine-tune the Qwen-Audio-Chat model using supervised instruction-followinn data. Each prompt consists of an audio file and a text instruction. The text part describes how the audio is processed (including the frame rate), lists the definition and order of all AUs, explains the format of AU values, and finally asks the model to classify the emotion and predict the AU sequence for each frame. Below is real example prompt used in training:
18

---

Published as a conference paper at ICLR 2026
Neutral
AUHead
Angry
MEMO
Angry
AUHead
Neutral
AUHead
Surprise
MЕMO
Surprise
AUHead
Figure 10: Facial animation comparison under neutral audio using AU from Qwen-Audio-CHi Chat.
Owen-Audio-Ce Auaio-Chat Input Example
("messages";
("zole": "user", "audio"; "data/XEAD "uata/ Audio audio, split waveform Train//Taan//013 sur_3_014, wav", "content"; *For each frames 3200 samples Éps). Each produces a 24-dimensioral AU vactor, with aomponents AD0 to A323 representing facial muscle activations in this fixed orders ADD lett eye closure; AC1 right eye closure; AU2 left lid raise; AU3 right lid raise; AD4 left brow lower; AUS right brow loverr AUE left brew raise; AU7 right brow reiser AD8 jaw-driven mouth opening; AU9 lower lip slide (left); AU10 lower 1ip alide (right1; Aull left
raise; AU12
sight lip cerner raise; AU13 left 1ip corner streteh; A014 right lip corner stretch; faw threst;
upper lip raise; AU19 lowar 1ip depress; A326 chir raise; A021 1p picker; A022 check puff; and AU23 nose wrinkle, Bach AD value
upper lip suck; A016
lip suck;
between. and and must be fornatted to two decimal places ithat is, write enly the decimal point and two digits for example, \*,12\" for 0.12), for each audio segment, record only the AUS that are activated along with their values; for example, [10,
only AU2 0.12 and ADl is 0.10 while the others remain untriggered. What is the emction of the current audio? current audic?"), ("role": "assistant", "content"; "surprise, [1[2, 1.0), (3, 1.0), [5, 1.0), 17, 1.0), 18, 0.52], [13, 0.17], [19, 0.11ll, [12, 1.01, 13, 1.01, [6, 1.0), 17, 1.0), 118, 0.191. [19, 0.0811, 1)
(1, .101]
sequence
1)
19

---

Published as a conference paper at ICLR 2026
This prompt enables the model to perform both emotion classification and AU sequence regression in unified framework.
D ADDITIONAL QUALITATIVE RESULTS
AniPortrait
AbHurinal
Echomimic
Flat Emsstio
Emition
HalloVI
Abmormal
ADHOETOJ
Alworricll
AUHead (HalloVI)
MЕMО
Flat Emoticn
AUHead (MEMO)
CREMA
MEAD
Figure 11: Qualitative comparison with SOTA methods on MEAD and CREMA.
Fig. 11 presents an extended visual comparison between AUHead and multiple SOTA methods on the MEAD and CREMA datasets. This figure complements the qualitative results shown in the main paper by including more models and a wider range of cases. We also annotate noticeable artifacts in the baseline outputs, such as blurry textures, unnatural expressions, or incorrect mouth shapes. In contrast, AUHead consistently produces clearer, more expressive, and visually coherent results, highlighting the benefits of AU-guided generation.
In addition to still-frame comparisons, we provide supplementary video demonstrations in the het___suuuppleeettrrrr_ee_._
contains two folders. The first, compare_w. wi th aac__STTA_eethooe includes video comparisons between AUHead and existing SOTA methods using identical audio and reference images. These clips offer a more intuitive view of AUHead's superior temporal consistency and expression quality. In addition to the video demon- strations, we further provide static frame comparisons extracted from these videos (see Figs. 14, representative snapshots of the dynamic results,
directory named
16, 13, 15, and 17). These visualizations allowing for a clearer inspection of fine-grained details. Across diverse subjects and emotions, AU- Head produces frames with more natural and expressive facial dynamics, particularly in the subtle motion of the eyes and brows. Compared with basclines that often exhibit artifacts such as blur- riness or rigid expressions, AUHead consistently delivers higher-quality outputs that preserve both emotional fidelity and identity stability. The second folder, Emotio ion, shows emotional genera- tion results using neutral audio combined with AU features and reference images corresponding to different emotions. These examples demonstrate that AUHead is capable of generating expressive talking- videos with strong emotional fidelity and stable identity preservation.
E THE USE OF LARGE LANGUAGE MODELS
We declare that large language models (LLMs) were employed to assist with the refinement this manuscript, specifically, for grammar checking, language polishing, and improving the clarity and fluency of the text. Additionally, LLMs were used in a limited capacity for minor debugging and syntactic correction of code snippets included in the work.
20

---

Published as a conference paper at ICLR 2026
AnPortrolt
Wov2Lip
Echonimio
MENO
Soctakker
Notol
Anger
Figure 12: Qualitative comparison with SOTA methods on MEAD: sults subject W009 with Angry expression.
Antertiilt
WorLUp
Echonimic
MENO
Spatakee
Halet
Feor
Figure 13: Qualitative comparison with SOTA methods on MEAD: sults subject M030 with Fear expression.
21

---

Published as a conference paper at ICLR 2026
monnimio
MENO
Gootaker
Hotel
Anger
Figure 14: Qualitative comparison with SOTA methods on MEAD:
sults subject M003 with angry expression.
AnPertoiit
WaxZLip
Echonimic
MENO
Soctolker
Helle1
Dagusst
Hallo2
Figure 15: Qualitative comparison with SOTA methods CREMA: results of subject 1033 with Disgusted expression.
22

---

Published as a conference paper at ICLR 2026
Antorttal
Wor2Lip
Echonimio
MENO
Soctakker
Motio
Anger
Figure 16: Qualitative comparison with SOTA methods CREMA: results of subject 1062 with Angry expression.
AnPortriit
WorZLip
Echonimic
MENO
Soctaker
Hello
Happy
Audio2Heod
Figure 17: Qualitative comparison with SOTA methods CREMA: results of subject 1089 with Happy expression.
23


Reviews:
[{"review_id": "wTDhqtxORK", "reviewer": "Reviewer_SgZG", "summary": "This paper presents AUHead, a two-stage framework for generating realistic and emotionally expressive talking head videos conditioned on speech and a reference image. The key idea is to employ Facial Action Units as an interpretable intermediate representation bridging audio and visual modalities, addressing the difficulty of producing fine-grained emotional expressions in prior works.\nIn Stage 1 (Understanding), an Audio Language Model (Audio-Qwen-Chat) extracts emotion aware AU sequences from speech through two steps: (1) a spatio-temporal AU tokenization strategy that compresses dense AU vectors, and (2) an emotion-then-AU Chain-of-Thought process that first predicts emotion categories before generating detailed AU sequences.\nIn Stage 2 (Controllable Generation), an AU-guided diffusion model synthesizes videos via three modules: (1) AU Representation, which upsamples and projects AU sequences into structured 2D facial maps (LMK or RoM) to ensure spatial realism; (2) Context-Aware AU Embedding, using temporal convolution to maintain motion consistency; and (3) AU Vision Interaction, which introduces AU-conditioned cross-attention and a classifier-free AU guidance mechanism for flexible control between expression fidelity and visual quality.", "questions": "1. Does the chosen set of 24 AUs cover all the major emotional expressions present in the MEAD and CREMA datasets?\n\n2. In the spatial-temporal AU tokenization, the AU sequences are downsampled to 5 fps and encoded as (index, intensity) pairs. Could the authors explain how this 5 fps rate was determined and whether alternative rates were considered?\n\n3. Please provide more technical details on the AU to 2D representation mapping (LMK and RoM).\n \n4. The user study involves a small number of raters. Could the authors elaborate on the selection process and whether they plan to expand this evaluation in future work?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The supplementary visualizations are exceptionally clear and persuasive, providing strong qualitative evidence of the method’s superiority.\n\n2. The two-stage pipeline is well justified, and adopting AUs as the intermediate feature is a sensible and effective choice.\n\n3. As I know, this is the first work to leverage a large audio-language model to predict AU sequences from speech, enabling disentangled emotion-aware representations for talking-head generation.\n\n4. Stage 2 effectively maps AUs to 2D spatial priors (landmark or mesh) and integrates temporal AU embeddings with AU–vision cross-attention adapters. \n\n5. This paper is well-orgnized and well written.", "weaknesses": "1. The AU regression in Stage 1 achieves an MAE of around 0.2, which is comparable to inter-annotator variability. However, the paper does not explicitly discuss how this residual AU prediction error might influence the final video generation quality. This limitation is acceptable, but a short analysis would further clarify the robustness of the pipeline.\n\n2. The AU sequences are downsampled to 5 fps to fit within the ALM’s context window, which may constrain the modeling of subtle, high-frequency facial movements.\n\n3. The user study is small in scale, this is not sufficiently rigorous for strong perceptual claims and risks bias.\n\n4. Some implementation details are briefly described and could be clarified. It would be useful to explain how the AU sequences are converted into 2D facial representations (landmarks and RoM).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "204VlDaWYM", "reviewer": "Reviewer_U6Er", "summary": "This paper presents AUHead, a two-stage framework for audio-driven talking head generation that aims to improve the expressiveness and controllability of facial animations. Instead of directly generating video from audio and a reference image, the proposed method first extracts Action Unit (AU) sequences from the input speech using an audio language model (ALM), leveraging its capability to capture emotional and articulatory cues. In the second stage, a pretrained diffusion model is enhanced with an AU adapter module that integrates AU embeddings, audio features, and the reference image through cross-attention mechanisms. Experiments on benchmark datasets (MEAD and CREMA) show that AUHead generates videos with improved facial expressiveness and temporal smoothness compared to existing methods.", "questions": "* Does AUHead support generating videos from the same audio segment guided by different emotion labels? If not, will inaccurate emotion recognition by the ALM lead to incorrect emotional expressions in the generated results?\n* Can the core method of AUHead be equally applied to more recent frameworks such as Sonic[1] and WanS2V[2]?\n* Could the experimental evaluation be expanded to more comprehensively demonstrate the effectiveness of the proposed method?\n* The videos provided in the supplementary material are all shorter than 10 seconds. How does AUHead perform on audio clips longer than 10 seconds? Could the authors provide corresponding results?\n\n[1]Ji, Xiaozhong, et al. \"Sonic: Shifting focus to global audio perception in portrait animation.\" Proceedings of the Computer Vision and Pattern Recognition Conference. 2025.\n\n[2]Gao, Xin, et al. \"Wan-s2v: Audio-driven cinematic video generation.\" arXiv preprint arXiv:2508.18621 (2025).", "limitations": "", "rating": 2, "confidence": 4, "soundness": 1, "presentation": 2, "contribution": 2, "strengths": "* Enhanced Facial Expressiveness: By using Action Units (AUs) extracted from audio as an intermediate representation, AUHead generates facial animations with richer and more realistic emotional expressions. AUs provide structured, interpretable cues for specific facial muscle movements, enabling the model to produce more nuanced and natural-looking expressions compared to baseline methods.\n* Effective Utilization of Pre-trained Models: The method leverages a audio language model to predict AU sequences from speech. This design capitalizes on the strong audio understanding capabilities of existing pre-trained models, making the AU prediction stage more effective and avoiding the need to learn complex audio-to-AU mappings from scratch.", "weaknesses": "* The experimental setup and comparisons are not sufficiently rigorous:\n    * There has been extensive research on facial expression control in the audio-driven talking-head generation field, including but not limited to [1, 2, 3, 4, 5, 6]. However, this work does not compare with any of these methods in its experiments. What is the reason for this omission?\n    * The datasets used in this work are MEAD and CREMA, which contain only 60 and 91 identities respectively. The limited scale and quality of the experimental data significantly weaken the validity and persuasiveness of the evaluation.\n    * The comparison lacks up-to-date state-of-the-art methods. The baselines chosen are at least half a year to a year old; for instance, EchoMimic[7] has already released version V3 and the Hallo[8] series has reached V4. This work fails to discuss or compare against more recent approaches such as Sonic[9], MultiTalk[10], and WanS2V[11].\n* Insufficient experimental results: The quantitative results of this work leave room for improvement. Notably, in terms of lip-sync accuracy, it ranks only sixth on the MEAD test set—lower than the pre-trained method MEMO[12]—suggesting that the proposed AUHead may degrade the performance of the underlying pre-trained model. Moreover, the provided visual results exhibit clear issues, such as inaccurate lip movements and blurred teeth, indicating a noticeable gap compared to current leading audio-driven methods like MultiTalk and WanS2V.\n* Practical challenges arising from the core methodology: Recent advances in large video generation models have enabled effective text-guided facial expression control via implicit semantic understanding—for example, methods like InfiniteTalk and FantasyTalking2 leverage vision-language models (VLMs) to generate emotion labels and achieve expressive control without explicit supervision. In contrast, this work relies on constructing explicit Action Unit (AU) labels for emotion control, which requires large-scale supervised training data annotated with AUs. This approach faces significant practical challenges in data collection and annotation, making it less scalable and deviating from mainstream trends. As a result, its potential contribution to the broader research community appears limited.\n\n[1]Gan, Yuan, et al. \"Efficient emotional adaptation for audio-driven talking-head generation.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.\n\n[2]Tan, Shuai, et al. \"Edtalk: Efficient disentanglement for emotional talking head synthesis.\" European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2024.\n\n[3]Ma, Yifeng, et al. \"DreamTalk: When Emotional Talking Head Generation Meets Diffusion Probabilistic Models.\" arXiv preprint arXiv:2312.09767 (2023).\n\n[4]Tan, Weipeng, et al. \"Disentangle Identity, Cooperate Emotion: Correlation-Aware Emotional Talking Portrait Generation.\" arXiv preprint arXiv:2504.18087 (2025).\n\n[5]Ma, Xingpei, et al. \"Playmate: Flexible Control of Portrait Animation via 3D-Implicit Space Guided Diffusion.\" Forty-second International Conference on Machine Learning.\n\n[6]Lin, Bin, et al. \"Takin-ADA: Emotion Controllable Audio-Driven Animation with Canonical and Landmark Loss Optimization.\" arXiv preprint arXiv:2410.14283 (2024).\n\n[7]Chen, Zhiyuan, et al. \"Echomimic: Lifelike audio-driven portrait animations through editable landmark conditions.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 39. No. 3. 2025.\n\n[8]Xu, Mingwang, et al. \"Hallo: Hierarchical audio-driven visual synthesis for portrait image animation.\" arXiv preprint arXiv:2406.08801 (2024).\n\n[9]Ji, Xiaozhong, et al. \"Sonic: Shifting focus to global audio perception in portrait animation.\" Proceedings of the Computer Vision and Pattern Recognition Conference. 2025.\n\n[10]Kong, Zhe, et al. \"Let Them Talk: Audio-Driven Multi-Person Conversational Video Generation.\" arXiv preprint arXiv:2505.22647 (2025).\n\n[11]Gao, Xin, et al. \"Wan-s2v: Audio-driven cinematic video generation.\" arXiv preprint arXiv:2508.18621 (2025).\n\n[12]Zheng, Longtao, et al. \"Memo: Memory-guided diffusion for expressive talking video generation.\" arXiv preprint arXiv:2412.04448 (2024).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4qgeYRO1Zi", "reviewer": "Reviewer_xY8X", "summary": "The paper introduces AUHead, a novel framework that creates emotionally expressive, lip-synced talking-head videos by extracting facial Action Units from audio. It employs an Audio-Language Model to derive AUs and a diffusion model to animate faces, surpassing previous methods in emotional realism, lip sync accuracy, and visual coherence.", "questions": "- Does the model perform equally well on unseen audio samples not part of the training data, especially in terms of subtle emotional cues that were not explicitly included in the training set?\n- How does the model handle cases where there are multiple potential AU sequences for a given audio input? Is there any inherent ambiguity in predicting AUs, especially in more complex emotional expressions?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The idea of using AUs as a bridge for generating expressive talking-head videos is novel and could lead to more controllable and interpretable results in audio-driven face generation.\n- The disentanglement of AUs from audio before the generation process is an interesting and effective way to tackle the complexities of fine-grained facial expression modeling.", "weaknesses": "- While the model performs well in terms of lip sync and expression fidelity, the paper doesn't sufficiently address how well the model handles longer videos or more complex facial movements. Temporal consistency issues may arise in longer video synthesis.\n- The spatial-temporal tokenization approach to generate AU sequences may introduce issues related to the loss of subtle details in facial expressions, especially when reducing the sequence length to manageable tokens (as discussed in the sparse tokenization section).\n- While AUs provide a more interpretable control space compared to emotion labels, the complex tokenization and disentanglement from audio may reduce the overall interpretability of how specific AUs contribute to particular facial expressions. A clearer explanation of the model’s inner workings could help with understanding the generated results better.\n- Although the paper compares AUHead to state-of-the-art methods, further comparisons with models that also leverage other types of intermediate representations (e.g., motion priors, landmarks) could give a better perspective on AUHead’s relative strength.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JlzlWTtFQb", "reviewer": "Reviewer_kSUe", "summary": "The paper proposes AUHead, a two-stage talking head generation method that leverages ALM and  facial action units to control the speaker's emotion.", "questions": "Can the method generalize to audio and source images that are not from MEAD and CREMA?\nCan the authors provide comparisons with more recent methods?", "limitations": "", "rating": null, "confidence": 4, "soundness": 1, "presentation": 2, "contribution": 1, "strengths": "1. The talking head generation task is of practical importance. Emotional control is a important topic in talking head generation since natural talking heads exhibit emotions.\n2. The paper is easy to follow.", "weaknesses": "1. The performance is far from satisfactory. The results shown in demo are blurry. The expressions are unnatural.  All audio and source images shown in demo are from lab dataset MEAD[1] and CREMA[2], which cannot indicate that the model can generalize to more diverse inputs. In short, the performance is far from SOTA and is unacceptable for publication.\n2. The paper lacks comparisons with recent emotional talking head methods, including EDTalk[3], EAT[4]. Lack of comparisons makes it difficult to validate the effectiveness of the proposed method.\n3. Utilizing ALM and AU to control emotion is of little significance now. Recent methods, like omnihuman1.5[5], can control emotion with diverse text. Diverse text achieves more powerful control than AUs.\n\n[1] Wang, Kaisiyuan, et al. \"Mead: A large-scale audio-visual dataset for emotional talking-face generation.\" European conference on computer vision. Cham: Springer International Publishing, 2020.\n\n[2]  Cao, Houwei, et al. \"Crema-d: Crowd-sourced emotional multimodal actors dataset.\" IEEE transactions on affective computing 5.4 (2014): 377-390.\n\n[3] Tan, Shuai, et al. \"Edtalk: Efficient disentanglement for emotional talking head synthesis.\" European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2024.\n\n[4] Gan, Yuan, et al. \"Efficient emotional adaptation for audio-driven talking-head generation.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.\n\n[5] Jiang, Jianwen, et al. \"Omnihuman-1.5: Instilling an active mind in avatars via cognitive simulation.\" arXiv preprint arXiv:2508.19209 (2025).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ynyHnJXqos", "reviewer": "Reviewer_rPph", "summary": "The paper proposes the AUHead method for the talking head generation task with fine-grained emotion control. This method is divided into two stages: in the first stage, an audio-language model (ALM) is used to predict Action Units sequences from audio; in the second stage, AU and Audio are fed into the diffusion model as conditions for video generation.​", "questions": "Please refer to the weakness.", "limitations": "", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper proposes a method of using ALM for fine-grained emotion prediction, which is relatively rare in related works.​\n- The paper conducts rich and comprehensive experiments.​\n- The paper is well written and presents content clearly.​", "weaknesses": "- Although using ALM for emotion prediction is novel, the necessity of using ALM is questionable. The AU sequences used in the paper are only 24-dimensional, which is obviously a low-dimensional vector. There are many works on 3D talking head generation based on audio-to-mesh/3DMM that have achieved good results (many of which specifically model emotion), and such scenarios are similar to the problem mentioned in this paper, but even more difficult. The authors need to use experiments to demonstrate that the traditional audio embedding + regression method cannot predict AU sequences ideally. In addition, fine-tuning ALM has significantly higher inference cost and training difficulty, which limits this method to a certain extent.​\n- The paper claims that the model has \"speech understanding\" ability, but it is hard to be convinced that AU sequence prediction is equivalent to \"speech understanding\".​\n- The \"AU-based disentanglement guidance strategy\" proposed in the paper is not original. In fact, many multi-conditional generation tasks adopt the same or similar CFG variants, such as MagicInfinite[1] (eq 7.).\n\n[1] Yi, Hongwei, et al. \"Magicinfinite: Generating infinite talking videos with your words and voice.\" arXiv preprint arXiv:2503.05978 (2025).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uScOFOry6k", "reviewer": "Reviewer_xaec", "summary": "This paper presents AUHead, a two-stage framework for audio-driven emotional talking head generation. The first stage leverages a fine-tuned Audio-Language Model (Qwen-Audio-Chat) to disentangle facial Action Units (AUs) from audio using a spatial–temporal tokenization and Chain-of-Thought (CoT) based “emotion-then-AU” reasoning process. The second stage introduces an AU-driven diffusion generation framework, including AU-to-2D mapping, context-aware temporal embedding to synthesize identity-consistent and emotionally expressive faces. Experiments on MEAD and CREMA datasets show strong quantitative and qualitative performance improvements over state-of-the-art baselines (e.g., MEMO, HalloV1/V2), especially in emotional realism and AU alignment. Overall, the paper is technically novel, methodologically well motivated, and experimentally solid, offering an interpretable bridge between audio and visual modalities.", "questions": "1.You are encouraged to expand the user study with a larger and more diverse set of participants to further validate the perceptual quality claims.\n2.Can the AU guidance be manually adjusted at inference time to control or edit facial expressions—for example, by modifying AU values to vary the intensity or type of emotion?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1.Introducing AUs as an intermediate control effectively bridges speech understanding and video generation.\n2.Stage 1 pioneers the use of an ALM for AU generation, which is a highly insightful idea for a generalizable control signal. The “emotion-then-AU” CoT design and spatial temporal AU tokenization are elegant, showing how ALMs can perform emotion reasoning beyond conventional recognition tasks.\n3.Stage 2's AU-driven controllable generation framework enhances emotional expressiveness while maintaining identity and lip sync.\n4.The paper is clear, includes all implementation details, and provides an anonymous open-source link, which supports reproducibility.", "weaknesses": "1．The user study involved only 4 participants who are identified as \"colleagues from our laboratory.\" This sample size is too small, and the lack of participant independence introduces a severe bias, which significantly diminishes the persuasiveness of the user preference results for a top-tier conference.\n2．While the intuition behind using AUs as intermediate control is strong, the paper could better clarify why ALM-based AU prediction is preferred over simpler audio–AU regression networks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)