Long-Term TalkingFace Generation via Motion-Prior Conditional Diffusion Model
Authors: Fei Shen
Cong Wang
Junyao Gao
Qin Guo
Jisheng Dang
Jinhui Tang
Tat-Seng Chua
Venue: ICML 2025 poster (2025)
Keywords: Diffusion Model
TalkingFace
Pose
Primary area: applications->computer_vision

Abstract: Recent advances in conditional diffusion models have shown promise for generating realistic TalkingFace videos, yet challenges persist in achieving consistent head movement, synchronized facial expressions, and accurate lip synchronization over extended generations. To address these, we introduce the \textbf{M}otion-priors \textbf{C}onditional \textbf{D}iffusion \textbf{M}odel (\textbf{MCDM}), which utilizes both archived and current clip motion priors to enhance motion prediction and ensure temporal consistency. The model consists of three key elements: (1) an archived-clip motion-prior that incorporates historical frames and a reference frame to preserve identity and context; (2) a present-clip motion-prior diffusion model that captures multimodal causality for accurate predictions of head movements, lip sync, and expressions; and (3) a memory-efficient temporal attention mechanism that mitigates error accumulation by dynamically storing and updating motion features. We also introduce the {TalkingFace-Wild} dataset, a multilingual collection of over 200 hours of footage across 10 languages. Experimental results demonstrate the effectiveness of MCDM in maintaining identity and motion continuity for long-term TalkingFace generation.

Long-Term Term TalkingFace Generation via Motion-Prior Conditional Diffusion Model
Fei Shen' Cong Wang² Junyao Gao
Qin Guo
Jisheng Dang²
Jinhui Tang Tat-Seng Chua
Abstract
et al., 2023). However, the complexity of facial movements, including head, lip, and expression motions, presents chal- lenges, along with the need to maintain identity consistency across extended sequences.
Recent advances in conditional diffusion models have shown promise for generating realistic Talk- ingFace videos, yet challenges persist in achieving consistent head movement, synchronized facial expressions, and accurate lip synchronization over extended generations. To address these, in- troduce the Motion-priors Conditional Diffusion Model (MCDM), which utilizes both archived and current clip motion priors to enhance mo- tion prediction and ensure temporal consistency. The model consists of three key elements: (1) archived-clip motion-prior that incorporates historical frames and reference frame pre- serve identity and context; (2) present-clip cotion-prion diffusion model that captures multi- modal causality for accurate predictions of head movements, lip sync, and expressions; and (3) a memory-efficient temporal attention mechanism that mitigates еггоr accumulation by dynamically storing and updating motion features. We also introduce the MalkingFFaee-Wile dataset, multi- lingual collection of over 200 hours of footage across 10 languages. Experimental results demon- strate the effectiveness of MCDM in maintaining identity and motion continuity for long-term Talk- ingFace generation.
Early methods (Vougioukas et al., 2020; Wang et al., 2021b; Hong et al., 2022a; Chan et al., 2022; Guo et al., 2024) use GANs (Goodfellow al., 2014; Mirza & Osindero, 2014) to synthesize facial motions onto reference image through two-step process: decoupling motion features from audio and mapping them onto intermediate representations like facial landmarks (Yang et al., 2023), 3DMM (Sun et al., 2023), or HeadNeRF (Hong et al., 2022b). Despite their promise, GAN-based methods suffer from training insta- bility and inaccuracies in motion extraction, often leading to artifacts like blurriness and flickering that compromise video realism. Recent diffusion models (Wei et al., 2024; Shen et al., 2025a; Tian et al., 2024; Shen et al., 2025b;d; Guo et al., 2024; Zheng et al., 2024; Jiang et al., 2024) have improved TalkingFace generation by enhancing video real- ism through multi-step denoising that preserves conditional imput information. These methods typically use a Reference UNet (Hu, 2024) to encode identity features and integrate audio via cross-attentie However, reliance static audio features and weak correlations between audio and motion complicate the decoupling of identity and motion cues, often resulting in artifacts like motion distortion and flickering, especially in long-term generation.
While some methods (Wang al., 2024b; Ma al., 2024; Yang et al., 2024) improve long-term stability by introducing motion constraints like facial landmarks and emotion tags, these constraints often overly bind poses to the reference image, limiting expression diversity. Models trained with driven landmark fail to learn natural audio-driven motion patterns, reducing audio-visul synergy. Additionally, static emotion tags cannot capture dynamic shifts, leading to rigid, inauthentic animations over extended sequences. Besides, some approaches (Xu et al., 2024; Chen et al., 2024) in- ject brief motion reference frames, usually fewer than five over 0.2 seconds, which is insufficient to establish coherent motion, resulting in random, less dynamic movements.
1. Introduction
TalkingFace generation (Tan et al., 2024; Peng et al., 2024; Ye et al., 2024: Ji al., 2021; Tan al., 2023; Kim al., 2018; Liang et al., 2022; Ye et al., 2023; Pumarola al., 2018; Vougioukas et al., 2020) aims create realistic and expressive videos from reference face and audio, with ap- plications in virtual avatars, gaming, and filmmaking (Shen
'Nanjing University of Science and Technology "Nanjing Uni- versity Tongji University 'Peking University Sun Yat-sen Univer- sity National University of Singapore. Correspondence to: Jinhui Tang jjinhuitang @ nju.t..edu.cn...
Proceedings of the 4nnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
In this paper, we propose the Motion-priors Conditional Diffusion Model (MCDM) to address the challenges in

---

Motion-Prion Conditional Diffusion Model
achieving long-term consistency in TackingFace generation. The MCDM comprises three key modules: the archived-clip motion-prior, the present-clip motion-prior diffusion model, and memory-efficient temporal attention mechanism. Un- like conventional reference UNet-based identity learning, the archived-clip motion-prior introduces historical frames along with a reference frame via frame-aligned attention, enhancing identity representation and creating a cohesive fa- cial context over extended sequences. Then, the present-clip motion-prior diffusion model leverages multimodal causal- ity and temporal interactions to effectively decouple and predict motion states, including head, lip, and expression movements, ensuring a clear separation between identity and motion features and promoting temporal consistency across frames. To support long-term stability, we devise memory-efficiint temporal attention that dynamically stores and updates historical motion features, integrating them with current motion cues via a memory update mechanism. This structure reduces error accumulation often observed in diffusion- based long-term TalkingFace generation, enabling more stable and consistent outputs. Additionally, we present the TalkingFace- Wild dataset, a high-quality, multilingual video dataset with over 200 hours of footage in 10 languages, offering valuable resource for further research in Talking- Face generation. Our main contributions are summarized as follows:
GAN for video synthesis. GANimation (Pumarola al., 2018) models facial motion via continuous manifolds, en- hancing expression dynamies. SadTalker (Zhang et al., 2023) integrates ExpNet and Pose PoseVAE to refine motion rep- resentations within the Face Vid2Vid (Wang et al., 2021b) framework. DaGAN (Hong et al., 2022a) introduces self- supervised geometric learning to capture dense 3D motion fields. While effective, GAN-based methods suffer from ad- versarial training instability and motion inaccuracies, often resulting in artifacts that degrade realism.
Eertusion-Baseed Methods. Diffusion models (Rombach et al., 2022; Shen & Tang, 2024; Wang et al., 2024a; Shen et al., 2025c) have gained traction in TalkingFace gener- ation, producing high-quality, diverse outputs. AniPor- trait (Wei al., 2024) maps audio to 3D facial structures, generating temporally coherent videos with expressive de- tail. MegActor-E (Wang et al., 2024b) synchronizes lip movements, expressions, and head poses using a reference UNet (Hu, 2024) and facial loss functions to enhance fi- delity. Hallo (Xu et al., 2024) and EchoMimic (Chen et al., 2024) leverage limited motion reference frames to improve expression diversity and pose alignment. However, reliance short-term frame histories (2-4 frames) compronises long- meng-termiie motion consistency, while increased frame depen- dencies escalate computational costs. Additionally, static audio features and restricted references fail to capture nat- ural motion variations, leading to artifacts such as motion distortion and rigid expressions in extended sequences.
We propose MCDM to enhance robust identity consis- tency and support temporal consistency in long-term TalkingFace generation.
Unlike prior work, our approach introduces motion priors from both archived and present clips to enhance long-term motion prediction and identity consistency. By leveraging historical frames and memory eeffore-fffcieent temporal attention, MCDM improves motion continuity while maintaining real- ism in TalkingFace generation.
MCDM leverages archived-clip priors for identity- aware context, present-clip priors for disentangling identity and motion, and memory-efficient temporal attention to integrate historical and current motion fea- tures with reduced error accumulation.
3. Method
MCDM achieves state-of-ther -frt performance on mul- tiple TalkingFace benchmarks, demonstrating superior identity preservation and temporal consistency under long-term generation.
Task Definition. Given reference image, audio, and op- tional facial landmarks, TalkingFace generation aims to produce temporally coherent and realistic videos. The key challenges include maintaining consistent identity over time, achieving natural head movements, and ensuring expressive and precise lip alignment with audio cues. However, ex- isting methods often encounter limitations such error accumulation, inconsistent identity preservation, subopti- mal audio-lip synchronization, and rigid expressions.
2. Related Work
GAN-Based Methods. GAN-based approaches (Kim et al., 2018; Zhou et al., 2020; Pumarola et al., 2018; Vougioukas et al., 2020; Zhang et al., 2023: Wang et al., 2021b; Hong et al., 2022a; Chan et al., 2022: Guo et al., 2024) for TalkingFace generation extract motion features from dio or visual inputs and map them to intermediate repre- sentations such as facial landmarks (Yang et al., 2023), 3DMM (Sun et al., 2023), or HeadNeRF (Hong et al., 2022b). MakeltTalk (Zhou et al., 2020) employs LSTMs to predict landmarks from audio, followed by a warp-based
3.1. Overall Framework
To address the above challenges, we introduce MCDM, framework centered on denoising UNet resembling Stable Diffusion v1.5 (SD v1.5) 1 tailored to denoise multi-frame
2

---

Motion-Priee Prior Conditional Diffusion Model
$3.2 Archived-Clip Motion-Prior Module
93.3 Present-Clip Motion-Priot Diffusion Model
VAE Emcoder
Patchily Layer
Frane-kiggee Allesben
Patchify Laver
Aadie
Landlaark
Video
Encoder
Archived Clip
Fizame
Excaoe
Laaade
FILM
Decoder
Video
Mattinedaal Cannal Trandiemee Blick
Andiive Cipp
Audio
Present Clip Matiou-Pries DMI
Image
Tempural Interaction Tranefarner Black
Encoder
Encoder
Audio
1
Figure 1. Our MCDM architecture. On the upper, the archived-clip motion leverages trame-alignenn aligned attention with archived-clip. enhancing identity coherence over extended sequences. On right, the present-clip motion-prion diffusion model uses multimodal causality and temporal interactions to decouple and predict motion states, covering head, lip, and expression movements while maintaining a clear separation identity and motion features.
noisy latent inputs under conditional guidance. As illus- trated in Figure 1, unlike standard UNet architectures, each Transformer block in MCDM incorporates four attention layers. The first layer, a self-attention, mirrors that in SD v1.5. The second and third layers are parallel cross atten- tion (spatial-wise), designed for distinct interactions: the archived-clip reference attention layer, which integrates mo- tion priors from archived clip encoded by the archived-clip motion- motion-prior prior module (Section 3.2), and the present-chi -lip ref- erence attention, which engages with present clip priors from the present-clip motion-prior diffusion model (Sec- tion 3.3). The fourth layer, the memory-efficied temporal attention (Section 3.4), is a temporal-wer wise self attention that dynamically updates and merges archived motion features with current motion features, effectively mitigating error accumulation.
and archived clip Xarch [ where b, h, w, and a represent the batch size, channels, height, width, and the number of archived frames, respectively. First, the frozen VAE encoder extracts latent features from both the reference and archived frames, resulting in fa R x and fo E Rbxax4x xx respectivelyy Next, the learnable patchify layers, consisting of 2D convolu- tions followed by flattening operations, transform these latent features into tokens, yielding F₂ E R& xmxxd and Fa E Rbxaxmxd where m and d denote the token length and embedding dimension.
In the frame-ala alie-el attention, we adopt frame-wiss com- putation approach improve efficiency and adaptability for long temporal sequences. For each archived frame i [1, a]. the Key K; is derived from the reference tokens Fz. while the Value VÚ is derived from the tokens of the corresponding archived frame F%:
3.2. Archived-Clip Motion-Prior Module
K; K V Wv. where WH Rdd and Wy E Rdxd are learnable projec- tion matrices for the Key and Value. The attention for each frame i is then computed as:
(I)
Motivation. Existing methods typically use the past 2 frames to guide the denoising network for generating tem- porally consistent videos. However, this limited history frame is insufficient for maintaining long-term consistency, and incorporating more frames exponentially increases com- putational demand, making impractical for real-world applications. To overcome these limitations, we propose an archived-clip motion prior that integrates long-term histori- cal frames and reference frame into the denoising UNet via conditional frame-aligned attention, enhancing identity representation and establishing motion context.
QK1
Attention K, V,) = Softmax
Ve.
(2)
where Q E Rexd represents learnable query tokens, with n denoting the number of queries. Aggregating the outputs across all frames yields the final output Far E x where each frame's attended tokens reflect both the static reference and dynamic temporal information.
Architecture. As illustrated Figure 1, the archived-clip motion-prior consists of two frozen VAE encoders. two learnable patchify layers, and frame-aligned attention mechanism. Given reference frame X R& xexhxw
3.3. Present-Clip Motion-Prion Diffusion Model
Motivation. Motion information is typically driven either by landmark signals from driving video or directly by
3

---

Motion-Prior Conditional Diffusion Model
audio cues. The landmark-driven approach guides refer- ence image movements but limits the natural diversity of head motions and expressions. In contrast, audio-driven methods rely solely on audio cues, often lacking sufficient guidance for realistic head movement. To address these lim- itations, we propose the present-clip motion-prion diffusion model, which first predicts motion states, including head, lip. and expressions motions, rather than directly generating TalkingFace videos.
Arakleed Chty Motten Miaatry
Z4
Mr
Fe
Z
Faz
Figure 2. The overview of memory-efficient temporal attention. It can dynamically update and integrate historical motion features with current ones.
Architecture. We aim to predict motion in head, lip, and expressions lip movements, conditioned on audio and image tokens. As shown in Figure (right), we begin by extracting feature tokens from the audio encoder, image encoder, head encoder, lip encoder, and express encoder.
ditioning, incorporating landmark guidance when available, while effectively leveraging multimodal interactions for ac- curate motion state predictions.
Audio Encoder: Audio sequence tokens are extracted from the input audio via frozen Wav2Vec model (Baevski et al., 2020).
3.4. Memory-Efficiee Temporal Attention
Image Encoder: Image tokens are extracted from the refer- ence frame using a frozen CLIP (Radford et al., 2021) and are replicated along the temporal dimension to align with audio features.
Motivation. For long-term TalkingFace generation, current methods primarily adopt either fully or semi-autoregressive strategies: the former generates one frame per iteration, while the latter produces a fixed-length clip. However, due to GPU memory limitations, relying on restricted frame history for extrapolation often results in error accumulation, as limited prior motion information undermines consistency over extended sequences. Therefore, we propose a memory- efficient temporal attention to dynamically update and inte- grate historical motion features with current ones, reducing error accumulation.
Head Encoder: Head tokens are extracted from reference landmark video through a frozen Landmark Guider 2: no- tably, these tokens are optional, allowing simulation of con- ditions with or without reference video guidance.
Lip and Express Encoders: Lip and expression tokens are extracted from the target video using a custom-traine encoder. Details of the lip and express encoders are provided in the supplementary material.
Architecture. AnimateDiff (Guo et al., 2023) demonstrates that the temporal layer in self-attention ensures smooth temporal continuity and consistency of appearance details across frames. We replace traditional self-attention with fast attention (Choromanski et al., 2020) in the temporal layer to enhance temporal continuity and manage memory efficiently, allowing the accumulation of extensive historical motion information for consistent long-sequence generation. As shown in Figure 2, let Fac and Fpe denote the output features of the archived-clip and present-clip motion-prior modules, respectively, and let Ze represent the noisy latent feature at time step t. These features undergo reference at- tention, yielding refined representations Fre and Fre which capture spatial- motion characteristics. Fref is then input into the motion memory update mechanism, which ag- gregates motion across frames, producing the update feature My. The memory update mechanism is defined as follows,
We then pass the audio and image tokens through a feature- wise linear modulation (FiL.)) layer (Perez et al., 2018) to adaptively learn multimodal correlation tokens. These tokens, along with the timestep I, and noise-added tokens for head, lip. and expression movements, are prepended to the input sequence. This composite input is fed into an I-- layer structure consisting of multimodal causal transformer block (Peebles & Xie, 2023) and temporal interaction transformer block (Hu, 2024), with added noise in facial motion tokens acting as the supervision. The training loss Tpriior for the present-clip motion-prior diffusion model eg is defined as:
Leprior Ee,Fp ||c- co (2t, 2
(3)
Without landmark guidance, F, represent multimodal in- teraction tokens from audio and the reference frame. 2t represent noise-added tokens for head, lip, and expression movements at timestep t. With landmark guidance, Fp addi- tionally include landmark tokens. 2₁ represent noise lip and expression tokens. This design allows flexible con-
(1) Initialization: At the first frame, the memory M1 is ini- tialized with Free since no prior motion information exists:
M1 Frs
(4)
(2) Memory Update: For each frame f. the memory My is updated by combining the current feature pref with the
ac
4

---

Motion-Prion Conditional Diffusion Model
previous memory M as:
German
43%%
Polish
M, = a a) Pae
(5)
4.8%
8.4%
French
Rovilan
Num of langunges
Statisties of our TalkingFice Wild dataset
10
a Creatian
Num of identiiiss 3,450
where a E [0, I] controls the balance between past and cur- rent frames. This fixed memory update mechanism avoids storage bottlenecks of historical information. We then con- catenate Fee with M, along the temporal dimension, creat- ing Fm, which integrates past and current motion. Fm is processed through Fast Attention along the temporal axis to capture dependencies across frames, with the lower half of the resulting feature map used as the output Fin
91%
Chinew
Num. of video cliss
31.3k
1.5%
Arablic
Daglish
Total hours
Avg. duration
241.6
27.8s
1.8%
Sponiish
Table 1. Statistics of our TalkingFace- Wild dataset. We release TalkingFace dataset that well-balanced across 10 languages.
regions, and low resolution. To mitigate these, we devel- oped a custom data processing pipeline for high-quality TalkingFace data, detailed in the following subsection.
3.5. Training and Inference
Training, Our training process is divided into three stages, each with specific learning objectives. Each stage is super- vised using standard MSE loss (Rombach et al., 2022).
Additionally, mostly methods (Wang et al., 2024b; Xu et al., 2024; Jiang et al., 2024) employ proprictary datasets for supplementary training and testing. Similarly, sourced variety of TalkingFace videos from YouTube using tar- geted keyword queries (e.g.. "nationality." "interview," "di- alogue") across different languages and contexts. From Table 1, we collect a new high-quality dataset, TalkingFace- Wild, covering 10 languages and totaling over 200 hours after processing through our data pipeline. To assess the generalization capability of models, we also constructed an open-set test collection of 20 diverse portrait images and 20 audio clips.
Stage1. The archived-clip motion-prion -rroor is trained to hance identity representation and establish a robust facial motion context across extended sequences. The present-clip reference attention and memory-efficient temporal attention modules remain frozen during this stage.
Stage2. The present-clip motion- prior diffusion model is trained to predict the motion states of facial expressions, lip, and head movements. To simulate scenarios without driving video, we randomly drop the entire landmark clip.
Data Processing, First, we detect scene transitions in raw videos using PyScene and trim each clip to a max- imum duration of 30 seconds. Next, we apply face detec- tion (Guo al., 2021) to exclude videos lacking complete faces or containing multiple faces, using the bounding boxes to extract talking head regions. Third, an image quality as- sessment model (Su et al., 2020) filters out low-quality and low- oow-resouution clips. Fourth. SyncNet (Prajwal et al., 2020) assesses audio-lip synchronization, discarding clips with misaligned audio. Finally, we manually inspect a subset to verify audio-lip synchronization and overall video quality, ensuring precise filtering. In addition, ensure fair com- parison, we report results trained independently on each of the previously mentioned datasets.
Stage3. The full motion-priors conditional diffusion model is trained for generating stable and consistent long-term TalkingFace videos. Only the present-clip reference and memory-efficient temporal attentions are trained.
Inference. The present-clip motion-prior diffusion model first predicts distinct motion tokens based on the given con- ditions (either with or without landmark guidance). Land- marks are not used by default unless specified. Subsequently, MCDM utilizes these motion tokens, alongside a single reference image and audio input, to generate the video se- quence. For the initial archived clip, we initialize it using the reference image and then progressively update the motion memory to ensure temporal consistency.
Metrics. We utilize a comprehensive set of metrics to assess the quality of generated videos and audio-lip synchroniza- tion. Fréchet Inception Distance (FID) (Heusel et al., 2017) evaluates individual frame quality by comparing feature distributions from a pre-trained model. Fréchet Video Dis- tance (FVD) (Unterthiner et al., 2019) quantifies the distribu- tional distance between real and generated videos, providing an overall assessment of video fidelity. Sync-C and Sync- D (Chung & Zisserman, 2017) evaluate lip synchronization from content and dynamic perspectives, with higher Sync- C and lower Sync-D scores indicating superior alignment with audio. Structural Similarity Index (SSIM) (Wang et al.,
4. Experiments
4.1. Experimental Settings
Datasets. The HDTF dataset (Zhang et al., 2021) comprises 410 videos with over 10,000 unique speech sentences, var- ied head poses, and movement patterns. Following prior work (Chen et al., 2024; Tian et al., 2024; Xu et al., 2024), we split HDTF into training and testing sets with 9:1 ratio. The CelebV-HQ dataset (Zhu et al., 2022) includes 35,666 clips (3-20 seconds each) across 15,653 identities, totaling roughly 65 hours. Both datasets present quality issues, such as audio-lip misalignment, facial occlusions, small facial
5

---

Motion-Prion Conditional Diffusion Model
HDTF
CelehV-HQ
Method
FID
FVD.
Sync-Ct
Sync-D,
SSIM
E-FID,
FID.
FVD4
Syne-D, Sync-Ct
SSIM
E-FIDD
Audio2Head
76,08
1417.65
3.16
17.62
0.572
3.81
127.30
1882.64
1.96
17.36
0.391
8.42
V-Express
57.14
1152.29
5.05
11.68
0.706
1.83
98.07
1465.26
3.71
13.41
0.514
5.18
AniPortrait
54.81
1072.63
5.40
11.39
0.727
1.95
94.25
1260.74
3.98
12.88
0.536
4.91
SadTalker
52.77
956.24
5.73
10.65
0.736
1.87
88.22
1055.49
4.05
11.20
0.565
4.66
Hallo
37.29
616.04
6.33
8.64
0.774
1.67
72.46
907.60
6.48
8.61
0.620
2.93
EchoMinic
31.44
595.17
6.96
8.59
0.782
1.64
71.47
893.28
6.70
8.45
0.637
2.81
MegActor-E:
31.37
586.10
6.87
8.55
0.778
1.62
70.82
875.21
6.77
8.32
0.634
2.74
MCDM (Ours)
26.45
543.28
7.49
8.04
0.824
1.51
67.29
784.53
7.25
7.84
0.662
2.31
Table Quantitative comparisons on HDTF and CelebV-HQQ MCDM achieves the top results across metrics, with best in bold.
Method
FID FVD] Sync-CT Syne-D DJ SSIM E-FID
comparison on the HDTF (Zhang et al., 2021) and CelebV- HQ (Zhu et al., 2022), illustrating the overall superior perfor- mance of diffusion based methods compared to GAN-based methods. Our proposed MCDM achieves the best scores across all metrics, outperforming existing diffusion- approaches. Specifically, MCDM achieves superior lip- syne accuracy, reflected in higher Sync-C C and lower Syne-D scores, outperforming methods like EchoMimic (Chen et al., 2024) and MegActor-2 (Yang et al., 2024), which show notable declines in synchronization quality. MCDM's out- standing SSIM and E-FID scores also highlight its ability to generate visually appealing, temporally consistent content with precise lip synchronization.
Audio2Head
87.21 1836.25
2.32
13.92
0.613
3.12
V-Express
62.18 1324.57
5.45
9.04
0.674
2.81
AniPortrait
56.11 954.91
6.37
8.29
0.706
2.60
SadTalker
52.77 847.20
6,94
7.92
0.724
2.49
Hallo
51.35 792.38
6.85
7.65
0.728
2.35
EchoMimic
49.20 751.44
7.06
7.18
0.737
2.31
MegActor-22
48.57 724.40
7.22
7.14
0.745
2.29
MCDM (Ours)
42.08 656.71
7.84
6.69
0.779
1.97
Table 3. Quantitative comparisous on
TalkingFace-Vi Wild. MCDM achieves significant advantage over other methods.
2004) measures structural consistency between ground truth and generated videos, while E-FID (Deng et al., 2019) pro- vides a refined image fidelity evaluation based on Inception network features.
Table 3 summarizes the quantitative performance on the pro- posed TalkingFace-Wii dataset. Consistent with results on HDTF (Zhang et al., 2021) and CelebV-HQ (Zhu et al., 2022), MCDM surpasses all competing SOTA methods across evaluation metrics, demonstrating marked improve- ments in visual quality and temporal consistency. Achieving the best FID, FVD, and E-FID of 1.97, MCDM shows strong capability in generating high-fidelity TalkingFace videos under diverse conditions, effectively maintaining temporal coherence across audio, expressions, and lip syn- chronization.
The experiments are conducted on computing platform equipped with NVIDIA V100 GPUs. Training is performed in three stages, with each stage con- sisting of 30,000 iterations and a batch size of 4. Video data is processed at resolution of 512 512. The learning rate is fixed at x 10-6 across all stages, and the AdamW optimizer is employed to stabilize training. Each training clip comprised 16 video frames. In the archived-clip motion- prior module, we set a 16, m 256, and n 16. In the present-elip motion-prior diffusion model, the number of layers L is set to 8, and the weighting factor in Eq. 5 is configured to 0.1 to balance the influence of prior mo- tion information. This setup is chosen to optimize long-term identity preservation and enhance motion consistency within generated TalkingFace videos.
Implementations.
Qualitative Evaluation. Figure provides a qualita- tive comparison of our method against other SOTA ap- proaches. Compared V-Express (Wang et al., 2024b) and EchoMimic (Chen et al., 2024), our approach shows superior head and lip synchronization, benefiting from the audio-visual consistency introduced by motion priors. Ad- ditionally, unlike Hallo (Xu et al., 2024) and MegActor- Σ (Yang et al., 2024), Our method accurately captures sub- facial actions, including blinks and expression nuances through the archived-clip. while better preserving identity consistency. Overall, our approach demonstrates the best visual results.
4.2. Main Results
We compare our method with several SOTA meth- ods, including Audio2Head (Wang et al., 2021a), V-
Express (Wang al., 2024b), AniPortrait (Wei et al., 2024), SadTalker (Zhang et al., 2023), Hallo (Xu al., 2024), EchoMimic (Chen et al., 2024), and MegActor- (Yang et al., 2024), from quantitative, qualitative, and user study. Unless otherwise specified, all methods do not use land- marks to ensure a fair comparison.
User Study. The quantitative and qualitative compar- isons underscore the substantial advantages of our proposed MCDM in generating consistent TalkingFace videos. To further evaluate video quality, we conducte user study, fo- cusing on identity consistency, motion synchronization, and overall video quality. We randomly selected 10 cases, shuf-
Quantitative Evaluation. Table 2 presents a quantitative
6

---

Motion-Priee Prior Conditional Diffusion Model
GT
Erril Syeidpc
V-Expros
Imerited
Hallo
Repiniie
MegActor- Echomimic
40
Figure 3. Qualitative comparison on HDTF and CelebV-HIQ. Our method achieves the best generation results, particularly in identity consistency and motion detail.
A: Audio2Head V-Express
C: AniPortrait
SadTalkerr
E: Hallo
F: EchoMimic
G: MegActor-E
H: Ours
Identity Consistency
Motion Synchronization
Video Quality
0.8
0.8
0.8
Rank-3
0.6
0.6
Rank-3
0.6
0.4
0.2
0.2
0.2
F H User study results of identity consistency, motion synchronization, and video quality. Higher values indicate better
E
HI
F
II
Figure 4.
performance.
Method
FIDJ, FVD., Sync-Ct Sync-DJ, SSIM E-FID.
Archived-Clip Motion-Prior: The results in Table 4 show that removing historical frame information (w/o F.) sig- nificantly degrades performance across metrics, under- scoring the importance of the archived-clip motion-prior, To further assess the effect of F, on long-term generation, we visualized frames 30, 300, 1800, 3600, and 7200 with corresponding SSIM scores, as shown in Figure 5. Fig- ure 5(a) indicates that without the archived-clip (w/o F.). identity consistency worsens with frame progression, result- ing in visible artifacts and inconsistencies head, mouth, and expression. Figure 5(b), the SSIM scores highlight error accumulation increases with frame count, showing rapid decline in (w/o F.), while (w/ F.) remains stable at higher value. These findings validate the effectiveness of the archived-clip motion-prior in preserving both identity and temporal coherence over extended sequences.
w/o F.
46,25
708.93
7.37
7.05
0.749
2.25
w/o Fpc
45.63 684.20
7.49
6.97
0.7.8
2.13
w/o MTA
44.27
671.05
7.62
6.84
0.771
2.04
Ours
42.08 656.71
7.84
6.69
0.779
1.97
Table
Ablation results on the TalkingFace- Wild datasct.
fled the generated videos from each method, and recruited 20 participants (10 male, 10 female) to provide rank-3 pref- erences. From Figure our method consistently achieved the highest scores across all metrics in the user preference evaluation. This user study highlights the significant advan- tage of our approach in user-centric TalkingFace generation.
4.3. A Ablation Results
We conduct an ablation study to assess the impact of each component in our method. Table 4 shows the results: w/o F, omits historical frame information, w/o Fpc adds an audio attention module for audio feature input, and w/o MTA applies a standard temporal attention module.
Present-Clip Motion-Prior. Similarly, excluding the present-clip motion prior and injecting audio information directly via audio attention (w/o Fpx) leads drop in performance across all metrics. This decline highlights the
7

---

Motion-Prion Conditional Diffusion Model
Pome Ferrer yon
Freyy 18
Preay 1619
Framy-2 72
0.777
-M wco MTA
Refirmne
0.301
0.365
V Visaalization resalts of different frumees
0.55
0.1
33
0.5
077
0.9
Dilicrent Value
Figure 7. SSIM results for different a values. Our method per-
580M
0.77
forms comparably well when the a value is smaller than 0.9.
0.73
14/ Fg
0.69
w/o Fg
0.6
Method FID. FVD. Sync-C" Sync-D] SSIM1 E-FID.
30
300
1926
3600
7200
Number of Frames
(b) SSTM scores of different frames
B1
42.49 668.24 24
7.69
6.78
0.771
2.02
B2
47.12 721.17
7.30
6.84
0.732
2.29
Figure Visualization results and SSIM scores during long- term generation. We find that w/ F, offers a distinct advantage in maintaining both identity and contextual consistency.
Ours
42.08 656.71
7.84
6.69
0.779
1.97
Table 5. More results of variant MCDM.
Reference
Generatal Images
the effect of different values in Eq. 5, which control the update rate, on the model's 's SSIM performance, as shown in Figure 7. We observed that as a increases, SSIM grad- ually declines. When is below 0.9, our approach signifi- cantly outperforms the w/o WT A configuration. However, at a 0.9, the performance is weaker than w/o WTA, A, due to the excessive accumulation of historical frame informa- tion and reduced proportion of the present clip. Conse- quently, we set a = 0.1 as the default value in this paper.
(a) Heatmap Results of Lip
Fi (b) Clusters Resalts of Expression
Gl Fj
Figure 6. Lip heatmap and expression cluster. We find that w/ F, effectively tracks the lip region and conveys expressions.
More Results. Table 5 evaluates different design variants. In BI, Q-Former (Li et al., 2023) replaces frame-aligned at- tention, while in B2, Reference UNet (Hu, 2024) substitutes VAE with Reference UNet, omitting archived-clip informa- tion. Results show that frame-aligned attention outperforms Q-Former by effectively capturing temporal context and integrating long-term dependencies. Additionally, using a frozen VAE with a trainable patchify layer proves to be an efficient alternative to the conventional Reference UNet.
effectiveness of the plesent-clip motion- prrioo leveraging multimodal causality and temporal interactions decouple and predict motion states, including expressions, lip move- ment, and head motion (see Table 4). To further validate this decoupling capability, we visualize heatmaps of the predicted lip tokens, as shown in Figure 6(a), where the present-clip motion-prior accurately localizes and tracks lip motion. For expression decoupling, t-SNE (Van der Maaten & Hinton, 2008) visualization of expression tokens reveals tighter clustering within each of the eight distinct emotion categories when using the present-clip motion-prior, indicat- ing improved separation of emotional content from audio input.
5. Conclusion
We presented the Motion-priors Conditional Diffusion Model (MCDM) to address the challenges of long-term TalkingFace generation by achieving robust identity con- sistency and motion continuity. MCDM integrates three key innovations: an archived-clip motion-prior to enhance identity representation, a present-elip motion-prior diffu- sion model for accurate motion prediction, and memory- efficient temporal attention to mitigate error accumulation over extended sequences. Additionally, we introduced the TalkingFace- wild dataset, offering over 200 hours of multi- lingual video data across diverse scenarios. Experimental results demonstrate the effectiveness of MCDM, setting new benchmarks long-term TalkingFace generation.
Memory-Efficee Efficient Temporal Attention. Following the standard approach (Hu, 2024), we replace the proposed memory-efficient temporal attention with conventional tem- poral attention by directly summing Fres and Fres_ As shown in Table 4, this modification significantly degrades performance across all metrics. This drop in quality is pri- marily due to the absence of an update mechanism, which introduces gaps between the archived clip and the present clip, compromising video smoothness. Next, we analyzed
8

---

Motion-Prior Conditional Diffusion Model
Acknowledgements
resolution audio-driven portrait image animation. arXiv
preprint arXiv:2410.077/1, 2024.
This work supported by the Major Research Program of Jiangsu Province (Grant No. BG2024042). We would also like thank Silicon Intelligence* for their support and collaboration.
Deng. Y., Yang, J., Xu, S., Chen, D., Jia, Y., and Tong. X. Accurate 3d face reconstruction with reakly-supervised learning: From single image to image set. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition workshops, pp. 0-0, 2019.
Impact Statement
Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Farde- D., Ozair, S., Courville, A., and Bengio, Y. Generative adversarial nets. Advances in neural informa- tion processing systems, 27, 2014.
This paper presents the MCDM model, designed to enhance identity and temporal consistency in long-term Tace generation. While MCDM contributes the advancement of generative modeling, we recognize the potential ethical concerns, including the risks of misuse for creating decep- tive content or spreading misinformation. We emphasize the importance of transparency in AI development and support the integration of detection frameworks to mitigate these risks. In alignment with ongoing efforts in responsible AI, we aim to ensure that the benefits our work are balanced with its ethical implications, promoting safe and construc- tive applications in society.
Guo, J., Deng. J., Lattas, A., and Zafeiriou, S. Sample and computation redistribution for efficient face detection. arXiv preprint arXiv:2105.0477
2021.
Guo, J., Zhang, D., Liu, X., Zhong, Z., Zhang, Y., Wan, P., and Zhang, D. Liveportrait: Efficient portrait animation with stitching and retargeting control. arXiv preprint arXiv:2407.0m 066,, 2024.
Guo, Y., Yang. C., Rao, A., Liang, Z., Wang, Y., Qiao, Y., Agrawala, M., Lin, D., and Dai, B. Animatediff: Animate your personalized text-to-image diffusion models without specific tuning. arXiv preprint arXhv:2307.044 04725, 2023.
References
Baevski, Zhou, Y., Mohamed, A., and Auli, M. wav2vec 2.0: A framework for self-supervised learning of speech representations. Advances in neural information process- ing systems, 33:1 2449-12460, 2020.
Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., and Hochreiter, S. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Chan, E. R., Lin, C. Z., Chan, M. A., Nagano, K., Pan, B., De Mello, S., Gallo, O., Guibas, L. J., Tremblay, J., Khamis, S., et al. Efficient gcometry-aware 3d generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 16123-16133, 2022.
Hong, F.-T., Zhang, L., Shen, L., and Xu, D. Depth- aware generative adversarial network for talking head video generation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 3397- 3406, 2022a
Chen, Z., Cao, J., Chen, Z., Li, Y., and Ma, C. Echomimic: Lifelike audio-driven portrait animations through editable landmark conditions. arXiv preprint arXiv:2407.081333 2024.
Hong, Y., Peng, B., Xiao, H., Liu, L.., and Zhang, J. Head- nerf: real-time nerf- berf-based parametric head model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 20374-20384, 2022b.
Choromanski, K., Likhosherstov, V., Dohan, D., Song, X., Gane, A., Sarlos, T., Hawkins, P., Davis, J., Mohiuddin, A., Kaiser, L.. et al. Rethinking attention with performers. arXiv preprint arXiv:2009. 00000099./779.. /4794, 2020.
Hu, L. Animate anyone: Consistent and controllable image- to-video synthesis for character animation. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8153 8163, 2024.
Chung. J. S. and Zisserman, A. Out of time: automated lip syne in the wild. In Computer Vision-ACCV 2016 Workshops: ACCV 2016 International Workshops, Taipei, Taiwan, November 20-24, 2016, Revised Selected Papers, Part II 13, pp. 251-263 Springer, 2017.
Ji, X., Zhou, H., Wang, K., Wu, W., Loy, C. C., Cao, X., and Xu, F. Audio- emotional video portraits. In Proceedings of the IEFF/CVF conference on computer vision and pattern recognition, pp. 14080-14089, 2021.
Jiang, J., Liang, C., Yang. J., Lin, G., Zhong, T., and Zheng, Y. Loopy: Taming audio-driven portrait avatar with long-term motion dependency. arXiv preprint arXiv:2409.08 02634, 2024.
Cui, J., Li, H., Yao, Y., Zhu, H., Shang, H., Cheng, K., Zhou, H. Zhu, S., and Wang, Hallo2: Long-duration and high-
hittps://wwer jiiji

---

Motion-Prior Conditional Diffusion Model
Kim, H., Garrido, P.. Tewari, A., Xu, W., Thies, J., Niessner, M., Pérez, P., Richardt, C., Zollhöfer, M., and Theobalt, C. Deep video portraits. ACM transactions graphics (TOG), 1-14 2018.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G.,
Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763. PMLR, 2021.
Langley, P. Crafting papers on machine learning. In Langley, P. (ed.), Proceedings of the 17th International Conference on Machine Learning (ICML 2000). pp. 1207-1216, Stan- ford, CA, 2000. Morgan Kaufmann.
Razzhigaev, A., Shakhmatov, A., Maltseva, A., Arkhip- kin, V., Pavlov, I., Ryabov, I., Kuts, A., Panchenko, A., Kuznetsov, A., and Dimitrov, D. Kandinsky: an improved text-to-image synthesis with image prior and latent diffu- sion, arXiv preprint arXiv: /0.0333000.00222. 2023.
Li,J. 1 Li, D., Savarese, S., and Hoi, S. Blip-2: Bootstrapping language-iman image pre-training with frozen image encoders and large language models. In International conference on machine learning, pp. 19730- 19742. PMLR, 2023.
Rombach. R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition, pp. 10684-10695, 2022.
Liang, B., Pan, Y., Guo, Z., Zhou, H., Hong, Z., Han, X., Han, J., Liu, J., Ding, E., and Wang, J. Expressive talking head generation with granular oudio-visuai control. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3387-3396, 2022.
Shen, F. and Tang, J. Imagpose: A unified conditional framework for pose-guided person generation. Advances in nearal information processing systems, 37:6246-6266, 2024.
Ma, Y., Liu, H., Wang, H., Pan, H., He, Y., Yuan, J., Zeng, A., Cai, C., Shum, H.-Y., Liu, W., et al. Follow-your- emoji: Fine-controllable and expressive freestyle portrait animation. urXiv preprint arXiv:2406.0 01900, 2024.
Shen, F., Ye, H., Zhang, J., Wang, C., Han, X., and Yang, W. Advancing pose-guided image synthesis with pro- gressive conditional diffusion models. arXiv preprint arXiv:2310.06313, 2023.
Mirza, M. and Osindero, S. Conditional generative adver- sarial nets. arXiv preprint arXiv:1411.1788 2014.
Shen, F., Du, X., Gao, Y., Cao, Y., Lei, X., and Tang. J. Imagharmony: Controllable image editing with consistent object quantity and layout. 2025a.
Peebles, W. and Xie, S. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, pp. 4195-4205, 2023.
Shen, F., Jiang, X., He, X., Ye, H., Wang, C., Du, X., Li, Z.. and Tang. J. Imagdrresinng-ll Customizable virtual dressing. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp. 6795-6804, 2025b.
Peng, Z., Hu, W., Shi, Y., Zhu, X., Zhang, X., Zhao, H., He, J.. Liu, H., and Fan, Z. Synctalk: The devil is in the syn- chronization for talking head synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 666-676, 2024.
Shen, F., Ye, H., Liu, S., Zhang. J., Wang, C., Han, X., and Wei, Y. Boosting consistency in story visualization with rich-contextual conditional diffusion models. In Proceed- ings of the AAAI Conference on Artificial Intelligence, volume 39, pp.
Perez, E., Strub, F., De Vries, H., Dumoulin, V., and Courville, A. Film: Visual reasoning with a general con- ditioning layer. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018
2025c.
Shen, F., Yu, J., Wang. C., Jiang, X., Du, X., and Tang. J. Imaggarment-l: Fine-grained garment gener- ation for controllable fashion design. arXiv preprint arXiv:2504113176 2025d.
Prajwal, K., Mukhopadhyay, R., Namboodiri, V. P., and Jawahar, C. A lip syne expert is all you need for speech to lip generation in the wild. In Proceedings of the 28th ACM international conference on multimedia, pp. 484- 492, 2020.
Su, S., Yan, Q., Zhu, Y., Zhang, C., Ge, X., Sun, J., and Zhang, Y. Blindly assess image quality in the wild guided by a self-adaptive hyper network. In Proceedings of the IEEE/VV conference on computer vision and pattern recognition, pp. 3667-3676, 2020.
Pumarola, A., Agudo, A., Martinez, A. M., Sanfeliu, A.,
and Moreno Moreno-Noguer, F. Ganimation: Anatomically facial animation from a single image. In Proceedings of the European conference on computer vision (ECCV), pp- 81 818-88 08-833, 2018
Sun, X., Zhang, L., Zhu, H., Zhang, P., Zhang, B., Ji, X., Zhou, K., Gao, D., Bo, L., and Cao, X. Vividtalk: One- shot audio-driven talking head generation based on 3d hybrid prior. arXiv preprint arXiv:2312.00
2023.
10

---

Motion-Prion Conditional Diffusion Model
Tan, S., Ji, B., and Pan, Y. Emmn: Emotional motion memory network for audio-driven emotional talking face generation. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, pp. 22146-221555 2023.
Xu, M., Li, H., Su, Q.. Shang, H., Zhang, L., Liu, C., Wang. J., Van Gool, L.., Yao, Y., and Zhu, S. Hallo: Hierarchical audio-driven visual synthesis for portrait image animation. arXiv preprint arXiv:2406. 08801, 2024.
Yang. S., Li, H., Wu, J., Jing, M., Li, L., Ji, R., Liang, J., Fan, H., and Wang. J. Megactor-sigmar Unlocking flexible mixed-modal control in portrait animation with diffusion transformer. arXiv preprint arXiv:2408. /4975, 2024.
Tan, S., Ji, B., and Pan, Y. Flowvqtalker: High-quality emotional talking face generation through normalizing flow and quantization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 26317-26322 2024.
Yang. Z.. Zeng, A., Yuan, C., and Li, Y. Effective whole- body pose estimation with two-stages distillation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4214-421222 2023.
Tian, L., Wang, Q.. Zhang, B., and Bo, L. Emo: Emote por- trait seiive-generating expressive portrait videos with au- dio2video diffusion model under weak conditions. arXiv preprint arXiv:2402.17488 2024.
Ye, Z., Jiang, Z., Ren, Y., Liu, J., He, J., and Zhao, Z. Geneface: Generalized and high-fidelity audio-driven 3d talking face synthesis. arXiv preprint arXiv:2301.1342 2023.
Unterthiner, T., van Steenkiste, S., Kurach, K., Marinier, R., Michalski, M., and Gelly, S. Fvd: A new metric for video generation. 2019.
Ye, Z., Zhong, T., Ren, Y., Yang. J., Li, W., Huang, J., Jiang, Z., He, J., Huang, R., Liu, J., et al. Realšd-portrait One- shot realistic 3d talking portrait synthesis. arXiv preprint 2024.
Van der Maaten, L. and Hinton, G. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008.
urXiv:2401.085
Vougioukas, K., Petridis,, S., and Pantic, M. Realistic speech- driven facial animation with gans. International Journal of Computer Vision, 12865):1388-1411 2020.
Zhang, W., Cun, X., Wang. X., Zhang, Y., Shen, X., Guo, Y., Shan, Y., and Wang. F. Sadtalker: Learning realistic 3d motion coefficients for stylized audio-driven single image talking face animation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8652 8661, 2023.
Wang, C., Tian, K. Guan, Y., Zhang, J., Jiang, Z., Shen, F., Han, X., Gu, Q., and Yang, W. Ensembling diffusion models via adaptive feature aggregation. arXiv preprint wrXi:22455.17082 2024a.
Zhang, Z. Li, L., Ding, Y., and Fan, C. Fllo- Flow-guided one- shot talking face generation with a high- audio- visual dataset. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pp. 3661-3670, 2021.
Wang, C., Tian, K., Zhang. J.. Guan, Y., Luo, F., Shen, F,, Jiang. Z., Gu, Q., Han, X., and Yang, W. V-express: Con- ditional dropout for progressive training of portrait video generation, arXiv preprint arXiv:2406.0 0254001..11,, 2024b.
Wang, S., Li, L.., Ding, Y., Fan, C., and Yu, X. Audio2head: Audio-driven one-shot talking- head generation with nat- ural head motion. arXiv preprint 1002992 2021a.
Zheng, L., Zhang, Y., Guo, H. A., Pan, J., Tan, Z., Lu, J., Tang, C., An, B., and YAN, S. MEMO: Memory- guided and emotion-aware talking video generation, 2024. URL https: /opeeeev//pennrviiee.. net forum? id=CggWRFqxhD
Wang, T.-.., Mallya, A., and Liu, M.-Y. One-shot free-view neural talking-head hhaad synthesis for video conferencing. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10039-10049, 2021b.
Zhou, Y., Han, X., Shechtman, E., Echevarria, J., Kaloger- akis, E., and Li, D. Makelttalk: speaker-aware talking- head animation. ACM Transactions On Graphics (TOG), 39(6):1-15, 2020.
Wang, Z., Bovik, A. C., Sheikh, H. R., and Simoncelli, E. P. Image quality assessment: from error visibility to struc- tural similarity. IEEE transactions on image processing, 13(4):600-612, 2004.
Zhu, H., Wu, W., Zhu, Ji, L., Tang, S., Zhang, L., Liu, Z., and Loy, C. C. Celebv-hq: A large-scale video facial attributes dataset. In European conference on computer vision, pp. 650-667, Springer, 2022.
Wei, H., Yang, Z., and Wang, Z. Aniportrait: Audio-driven synthesis of photorealistic portrait animation. arXiv preprint arXiv:2403.1 17694, 2024.

---

Motion-Prion Conditional Diffusion Model
Supplementary Material
This supplementary material offers more detailed exploration of the experiments and methodologies presented in the main paper. Section A introduces a comprehensive set of symbols and definitions to enhance understanding. Section B delves into the implementation details of our proposed method. Section C provides additional experimental results. Sections D and respectively discuss comparisons with concurrent works and future work.
A. Some Notations and Definitions
The notations and definitions used throughout this paper are summarized in Table
Table 6.
Main notations and definitions used in this paper.
Notation
Definition
20
Target image
Timestep
Zt
Noisy data at step t
c
Pose or audio condition
Gaussian noise Diffusion model
Eo
Xref
Reference frame
Xarch
Archived clip
Fr
Feature of the reference frame
Fa
Feature of the archived clip
Fac
Feature archived-clij motion prior
Fp
Feature of present-clip motion prior
Frof
Feature of archived-cil reference attention
Fref
Feature of present-clip reference attention
pc
B. More Details
B.1. Archived-Clip Motion-Prior Module
VAE Encoder. Existing TalkingFace methods (Wang et al., 2024b; Xu et al., 2024; Chen al., 2024) often rely Reference UNet (Hu, 2024) to inject and learn identity features from reference frame. However, training a standalone Reference UNet demands substantial parameters and incurs significant computational costs, restricting its applicability to efficient generation tasks. In contrast, our approach employs a frozen VAE to encode reference images, streamlining the generation process via latent feature projection. By leveraging the pretrained encoding capabilities of the VAE, our method not only preserves identity consistency but also significantly enhances computational efficiency. As demonstrated in Tables 2 and of the main manuscript, the proposed VAB-based approach outperforms traditional Reference UNet methods in identity preservation. Moreover, Reference UNet exhibits considerable limitations in integrating archived frames, rendering it less effective for long-sequees sengseeqeence generation. Our archived-clip motion-prior module seamlessly incorporates archived frames, enabling high-qualit long-term TalkingFace generation. Results in Table 7 of the main manuscript further highlight the robustness and effectiveness of our method.
Frage-Aligned Attention. We propose frame-aligned attention to align the reference frame with the archived frames to incorporate additional archived frames. The frame-aligned attention overcomes the limitations of traditional Q-Former (Li al., 2023) preserving temporal consistency and identity integrity. Unlike Q-former, which primarily tailored for shoutesequence multimodal alignment, frame-aligned attention dynamically aligns feature tokens between reference and archived frames, enabling precise modeling of static identity features and dynamic temporal dependencies. This method achieves superior performance in TalkingFace generation, as evidenced in Table 7 of the main manuscript. Additionally, frame-aligned attention employs frame-wist attention to optimize computational efficiency, making it highly effective for long-sequence tasks. In contrast, Q-Former relies on global attention, concatenating features from all frames, leading to computational costs that scale linearly with sequence length and insufficient temporal modeling. The frame-aligned attention
12

---

Motion-Prior Conditional Diffusion Model
following videos, Consistency, Synchronization.
category: Identity Quality.
Identity Consistency: Look for how well the person's facial features, skin tore, ovenal the same throughout the video, without nolceaare changes.
Motion Synchronization: Focus on whether the up movements, head movements, and
Sacial expressions match the audo and seem natural over lime.
sharp resolution taxtures, smooth transition between ttmees
Quality: clrtyy.
Please select the top from the following:
Ampp
100.09
C.mp4 198.8
D.mp4 163.01 KB
E.mp4
Gumpé 193.72
H.mp4
148.47
Identity Consistency
Mation Synchronization:
Quality:
Figure Results of different expression. Our proposed MCDM demonstrates remarkahly realistic performance across four common emotional audio categories.
seamlessly integrates archived frames, significantly improving both temporal consistency and identity preservation.
B.2. Present-Clip Motion-Prior Diffusion Model
Lip and Express Encoders. Lip motion is driven by speech content, while emotions influence expression motion. The entanglement of these components often results in inconsistencies, such smiling expression paired with tense lip movements, undermining the realism and naturalness of the generated results. We propose the present-clip motion-prior diffusion model, which disentangles lip and expression motion features from lip and express encoders and predicts them independently. The training process involves constructing a pseudo-labeled dataset of paired videos with consistent expressions but varying lip motions, or vice versa, to facilitate disentangled learning. A 4-layer Transformer-basee lip encoder and express encoder are used to extract speech-driven lip motion and emotion- driven expression features separately. These features are integrated into a motion decoder to reconstruct facial motion, including landmarks and local deformations, The training is optimized with total loss function comprising self-reconstruction. gross-rcconstruction. and consistency losses, ensuring effective disentanglement and accurate motion reconstruction.
Temporal Interaction Transformer Block. In text-to-image (T21) generation, Kandinsky (Razzhigaev et al., 2023) has demonstrated the effectiveness of predicting image features to improve model quality. However, maintaining temporal consistency is crucial for TalkingFace generation, where prior T2I models primarily focus on predicting features for static conditions (c.g., image) and lack the capability to model motion continuity across sequences. To address this limitation, we propose the present-clip motion-prics aation diffusion model, which presents a temporal interaction transformer block to ensure
13

---

Motion-Prion Conditional Diffusion Model
Method
30
300
1800
3600
7200
V-Express (Wang et al., 2024b) Vanilla (Ours)
1.43
15.7
N/A
N/A
N/A
0.97
8.6
54.1
122.5
264.6
MCDM (Ours)
0.75
5.7
27.3
46.9
74.5
Table Results of inference speed. As the number frames increases, the advantage our method becomes more pronouneed. "N/A" denotes that the GPU encountered an overflow issue and was unable to complete the task. The unit is minutes.
temporal coherence. Specifically, multimodal correlation tokens and noise-injected decoupled motion features are extracted using corresponding encoders and a feature-wise linear modulation (FiLM) layer (Perez et al., 2018). These tokens are concatenated along the token dimension and represented as a 4D tensor æ E IR f where 6, f. n, and denote batch size, temporal length, token count, and token dimension, respectively. During processing, the temporal dimension f is reshaped into the batch size b within a multimodal causal transformer block (Peebles & Xie, 2023), allowing independent frame-wish wise processing. After passing through the multimodal causal transformer block, the features are reshaped back into 4D tensor to preserve temporal relationships. The temporal interaction transformer further reshapes the token dimension into the batch size b to learn and maintain temporal consistency for TalkingFace generation before reshaping it back, ignoring the temporal dimension. The temporal interaction transformer block employs multiple self-attention modules to guide self-attention along the temporal dimension f. effectively capturing motion features.
B.3. Memory-Efficient Temporal Attention
Fast Attention. In memory-efficiet temporal attention, both self-attention (Guo et al., 2023) and fast attention (Choromanski et al., 2020) are viable options. However, fast attention is particularly suited for capturing long-term temporal dependencies. As sequence length increases, traditional mechanisms like cross-attention face scalability challenges, resulting in higher computational and memory costs. Fast attention, with its optimized computation, allows efficient modeling of extended temporal relationships. making it ideal for tasks requiring consistent and coherent motion across large number of frames while maintaining computational feasibility and temporal fidelity.
B.4. User Study
As shown in Figure 8, we conducted user study to comprehensively evaluate the quality of the generated videos, focusing on three key metrics: identity consistency, motion synchronization, and video quality. The detailed procedure is follows:
Data Preparation. We randomly selected 10 samples from the test set, where each sample includes one input and its corresponding target video. For each sample, videos were generated using eight different methods, resulting in a total of 80 video clips (10 samples x 8 methods). These videos were randomly shuffled to ensure anonymity and fairness during the evaluation.
Evaluation Rules. We recruited 20 participants, comprising 10 males and 10 females, from diverse backgrounds with adequate visual perception abilities. Before the evaluation, participants were given detailed instructions on the evaluation criteria:
Identity Consistency: Assess whether the facial identity remains consistent across frames.
Motion Synchronization: Evaluate the alignment between lip motion and audio input.
Video Quality: Consider the naturalness, smoothness, and realism of the video.
Participants were asked to rank the videos using the rank-3 rule, selecting the top three videos from each set and ordering them by preference (1 being the best, 3 being the third best). To ensure accurate evaluations, participants were allowed rewatch videos as needed. The study followed ethical principles outlined in relevant guidelines. All participants provided informed consent prior to the study, and their feedbbck anonymized to protect privacy.
C. More Results
Comparisons in Inference Speed. To further evaluate inference speed, we conducted an additional experiment, as shown in Table 7. Using the representative Reference-basee architecture V-Express (Wang al., 2024b) as baseline, we kept all other
14

---

Motion-Prior Conditional Diffusion Model
Settings
FID
FVD.
Sync-Ct Sync-DI SSIM
F-FID
2
45.72 704.26
7.39
7.03
0.754
2.19
4
44.53 694.61
7.55
6.93
0.767
2.08
8
43.68 678.49
7.68
6,81
0.773
2.03
16
42.08 656.71
7.84
6.69
0.779
1.97
Table Results of different archived frames. We observed that increasing the number of archived frames significantly improves the quality of the generated results.
Reference
Fearful
Happy
Angry
Disgusted
RO
RC
RO
RO
RO
Figure 9. Results of different expression. Our proposed MCDM demonstrates remarkably realistic performance across four common emotional audio categories.
settings of MCDM unchanged and replaced fast attention (Choromanski et al., 2020) with the commonly used self-attention (Vanilla) (Guo et al., 2023) and our proposed MCDM. We measured the time required to generate 30, 300, 1800, 3600, and 7200 frames (in minutes). The results show that for 30 and 300 frames, V-Express (Wang et al., 2024b) significantly lags behind both Vanilla (Ours) and MCDM (Ours). Notably, as the number of frames increases, the Reference-based V-Express encounters GPU memory overflow, rendering it unable to handle more extensive sequences. contrast, our proposed MCDM demonstrates increasing efficiency advantages over Vanilla as the frame count grows. Specifically, for 7200 frames, our method achieves nearly 4x speedup compared Vanilla.
Influence of Different Archived Frame Numbers. To investigate the influence of archived clip length, conducted experiments using 2, 4, 8, and 16 frames while keeping all other settings fixed. As presented in Table 8, increasing the number of archived frames leads to notable improvements in both identity consistency and temporal coherence. This result highlights the critical role of archived frames and further demonstrates the effectiveness of our proposed archived-clip motion-prior module.
Results of Different Emotional Audio. To comprehensively evaluate the capability of our proposed MCDM in audio disentanglement tasks, select four classic emotional audio categories inputs: 'fearful,' 'happy,' 'angry." and 'disgusted.' These audio clips drive single reference image to generate the corresponding facial dynamics, as illustrated in Figure 9, The results demonstrate that MCDM achieves remarkably lifelike performance across all four emotional audio types, with outstanding detail fidelity. Subtle variations in eye expressions and mouth movements further highlight the model's ability to capture intricate facial features and emotional nuances.
Comparisons with T2I Prior Diffusion Model. To validate the effectiveness of the proposed present-clip motion-prior diffusion model, we compare its performance with the Kandinsky (Razzhigaev et al., 2023). Kandinsky's results are obtained by independently predicting features and then averaging them. As shown in Table the present-clip motion-prior diffusion model significantly outperforms Kandinsky in generating prior features. This improvement is attributed to the model's ability to capture temporal consistency, whereas Kandinsky only predicts static image features. This inherent conflict with the temporal consistency required for TalkingFace generation limits Kandinsky's effectiveness in modeling motion sequences.
Results of Long-Tere Term TalkingFace Generation. Figure 10 illustrates the long-term TalkingFace generation results the proposed MCDM. The generated sequences demonstrate consistent identity preservation throughout the extended duration.
15

---

Motion-Prioo Conditional Diffusion Model
Methods
Cosine Similarity?
Kandinsky 0,656
MCDM (Ours)
0.947
Table Results of prior model. We compute the average cosine similarity of motion features, revealing that MCDM demonstrates significant advantage in predicting motion features.
Reference
Frame 30
Frame 300
Frame 1800
Frame 3600
Frame 7200
Figure 10. Results of long-term TalkingFace generation. MCDM maintains long-term TalkingFace consistency and enables natural head pose transitions.
Additionally, the transitions in head poses appear smooth and natural, further validating the effectiveness of MCDM.
D. Discussion of Concurrent Work
Alongside our work, there are two notable concurrent studies, Loopy (Jiang et al., 2024) and Hallo2 (Cui et al., 2024) for TalkingFace generation. Since both are concurrent works without available code, we limit our discussion to their methodologies. Loopy (Jiang et al., 2024) introduces inter- and intra-clip temporal modules to capture long-term motion dependencies, focusing on natural and unconstrained motion without spatial constraints. Hallo2 (Cui al., 2024) extends portrait animation long- long-duration videos by employing patch-drop and Gaussian noise augmentations to address temporal artifacts and improve consistency. In contrast, our proposed MCDM introduces the Archived-Clij Motion-Prior and Present- Clip Motion-Par Prior mechanisms, explicitly disentangling identity and motion features while leveraging memory-efficient temporal attention. Unlike Loopy's 's emphasis on naturalness or Hallo2's augmentation-based strategies, MCDM achieves superior identity preservation and temporal coherence, particularly in long-frame frame scenarios, while maintaining computational efficiency.
E. Future Work
In our experiments, the proposed motion -prioo conditional diffusion model (MCDM) effectively demonstrates that combining archived and present motion priors significantly improves identity consistency and temporal coherence in long long-sequence TalkingFace generation tasks. This enhancement leads to notable improvement in the overall quality of the generated videos. MCDM the first approach to explicitly focus on motion priors while eliminating dependency on Reference UNet, thereby addressing GPU memory constraints. Additionally, its simple and modular design ensures ease of reproduction. Nonetheless, several directions for future exploration remain. These include extending the method to tasks such as "animate anyone" for dance generation, full-body digital human synthesis, and, in particular, scenarios involving multiple reference images. In these contexts, the continued reliance on Reference UNet presents limitation, marking important area for future research.
16


Reviews:
[{"review_id": "QRXzbdL9kw", "reviewer": "Reviewer_Vnrq", "summary": "The paper introduces the Motion-priors Conditional Diffusion Model (MCDM) for long-term TalkingFace generation, leveraging archived historical frames and present-clip motion priors through a memory-efficient temporal attention mechanism to achieve robust identity preservation, synchronized expressions, and accurate lip-audio alignment while releasing the multilingual TalkingFace-Wild dataset and demonstrating state-of-the-art performance across benchmarks.\n\n## update after rebuttal\nThanks for your detailed responses. I appreciate the additional experiments, qualitative comparisons, and clarifications, which have improved the quality of the paper. Given these improvements, I am happy to maintain my original rating and support the paper's acceptance.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer Vnrq:\n\nThank you for your thoughtful and encouraging review. We appreciate your recognition of our work as **“an interesting idea”** with **“rich and comprehensive experiments.”** We have carefully considered all your suggestions. Please find our point-by-point responses below.\n\n**Q1: Ablation Study on Historical Frame Length**\n\n**Response:** Thank you for the insightful question. We conducted an ablation study on the number of archived frames, as shown in Table 8 of the supplementary material, covering 2, 4, 8, and 16 frames. Additionally, we extended the study to 24 frames. Due to memory constraints, further increases were infeasible.\n\n| **Method**   | **FID ↓** | **FVD ↓** | **Sync-C ↑** | **Sync-D ↓** | **SSIM ↑** | **E-FID ↓** |\n|--------------|-----------|-----------|--------------|--------------|------------|-------------|\n| MegActor-Σ   | 48.57     | 724.40    | 7.22         | 7.14         | 0.745      | 2.29        |\n| Ours-2       | 45.72     | 704.26    | 7.39         | 7.03         | 0.754      | 2.19        |\n| Ours-4       | 44.53     | 694.61    | 7.55         | 6.93         | 0.767      | 2.08        |\n| Ours-8       | 43.68     | 678.49    | 7.68         | 6.81         | 0.773      | 2.03        |\n| Ours-16 | 42.08 | 656.71| 7.84    | 6.69     | 0.779  | 1.97 |\n| **Ours-24**  | **41.74** | **642.35**| **7.96**     | **6.55**     | **0.782**  | **1.94** |\n\nThe experimental results show that even with only 2 frames, our method outperforms MegActor-Σ across all metrics, indicating strong robustness under limited historical context. Furthermore, increasing the number of archived frames consistently enhances identity preservation and motion quality. We will highlight this ablation more clearly in the revised manuscript.\n\n\n**Q2: Lack of Qualitative Comparison with Related Works [1][2], Recent Open-Sourced Hallo2, and Other Methods**\n\n**Response:** Thank you for the valuable suggestion. We have added qualitative comparisons between our proposed MCDM and both open-source methods (SadTalker, EchoMimic, and Hallo2) and closed-source methods (Loopy [1] and EMO [2]), using the official audio and image inputs provided on the project pages of Loopy and EMO. Please refer to the following demo links: **Loopy**: [Demo 1](https://files.catbox.moe/d54l8i.mov), [Demo 2](https://files.catbox.moe/zezhpy.mov) and **EMO**: [Demo 1](https://files.catbox.moe/77kqif.mov), [Demo 2](https://files.catbox.moe/1p4qo4.mov).\n\nOur method demonstrates stronger identity consistency and better motion synchronization compared to SadTalker, EchoMimic, and Hallo2, while achieving visual quality comparable to Loopy. Additionally, it generates more natural expressions and overall enhanced visual realism compared to the original EMO outputs.\n\n\n\n**Q3: On the Distinction Between GAN-based and Diffusion-based Methods**\n\n**Response:** Thank you for the helpful suggestion. We have revised the introduction to improve the logical flow and clarify the distinction between GAN-based and diffusion-based methods. GANs often suffer from training instability and artifacts such as flickering, especially in long-form generation. In contrast, diffusion models provide more stable training and improved visual realism via multi-step denoising and stronger identity preservation, making them better suited for this task. We have added this clarification and cited relevant works in the revised version.\n\n\n**Q4: On Pose Constraints and Expression Diversity**\n\n**Response:** Thank you for the constructive feedback. (1) Our original intent was to highlight that when the reference image and driving pose come from different identities, strong pose constraints (e.g., facial landmarks) may cause identity distortion or unnatural warping. (2) While facial expressions are generally independent of head pose, over-constraining pose can inadvertently limit expression diversity. For example, methods like EchoMimic use dense 3D landmarks from tools such as MediaPipe, which may implicitly encode facial details and constrain expression variation. We have clarified this point in the revised introduction.\n\n\n**Q5: Misuse of the Term “Static Audio Feature”**\n\n**Response:** Thank you for pointing this out. The term “static audio feature” was a misphrasing. Our intended meaning was to describe the limitation of relying solely on audio features without incorporating complementary cues such as lip or head motion. We have clarified this wording in the revised manuscript.\n\n\n**Q6: Redundancy in Contribution Statements**\n\n**Response:** We agree with the reviewer’s observation. Contributions 2, 3, and 4 all describe technical components of our framework. We have merged them into a single contribution for improved clarity.\n\n**Again, thank you so much for helping us improve the paper!  Please let us know if you have any further questions. We are actively  available until the end of this rebuttal period. Looking forward to  hearing back from you!**", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "Yes.", "experimental_designs_or_analyses": "Yes.", "relation_to_broader_scientific_literature": "It improves the performance of long-term talking head video generation by achieving robust identity preservation, synchronized expressions, and accurate lip-audio alignment.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "Overall, this paper presents an interesting idea and is well-structured, with carefully designed figures. The experiments are rich and comprehensive, and the work meets the standard of ICML. However, I still have several concerns:\n\n1) Ablation Study: The archived motion prior module defaults to using 16 frames of historical information, but the impact of using shorter or longer histories on long-term consistency is not explored. For instance, would 4 frames be sufficient, or would more than 16 frames yield better results? A thorough ablation study on this aspect would strengthen the work.\n\n2) Comparison with Related Work: Although the two related papers [1][2] do not release their code, it would still be valuable to provide qualitative comparisons with them. Since both papers offer demo videos on their official websites, comparisons could be made using their audio and images at least at a qualitative level to better position this work among existing approaches.\n\n3) Concurrent Work: The authors mention that they do not compare with the concurrent method [3] due to the unavailability of code. However, I noticed that [3] has publicly released both code and model weights on GitHub. Including a comparison with [3] would make the evaluation more complete and convincing.\n\n4) Video Comparisons with Open-Sourced Methods: In the supplementary video, there are no comparison demos with existing open-sourced methods such as SadTalker, EchoMimic, and so on. Including video comparisons would provide a clearer understanding of the advantages and limitations of the proposed method and help readers better assess its performance.\n\nReferences:  \n[1] Loopy: Taming Audio-Driven Portrait Avatar with Long-Term Motion Dependency  \n[2] EMO: Emote Portrait Alive -- Generating Expressive Portrait Videos with Audio2Video Diffusion Model under Weak Conditions  \n[3] Hallo2: Long-Duration and High-Resolution Audio-driven Portrait Image Animation", "other_comments_or_suggestions": "The writing of the Introduction section needs refinement:\n\n1. The transition and logical flow between the second and third paragraphs, especially the use of logical connectors, should be improved and supported with sufficient evidence. For example, \na) The relationship between GAN-based methods and Diffusion-based models needs to be clarified to ensure coherence. \nb) In line 041, the claim that \"facial landmarks overly bind poses to the reference image\" is unclear — this motion constraint should naturally allow different poses across frames. \nc) In line 043, the assertion that overly binding poses limits expression diversity is also questionable. Facial expressions are largely independent of head pose — humans can maintain a fixed pose while still displaying a wide range of expressions. \nd) Additionally, the term \"static audio feature\" is ambiguous — audio features extracted from a time window inherently contain dynamic information.\n\n2. Contributions 2, 3, and 4 should be merged, as they all pertain to technical details and belong to the same category of contributions.", "questions_for_authors": "No.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethics_expertise_needed": ["Inappropriate Potential Applications & Impact  (e.g., human rights concerns)", "Privacy and Security"], "ethical_review_concerns": "The research focuses on facial applications, which raises concerns about privacy, security, and human rights, as facial images can be easily modified without individuals' consent."}}, {"review_id": "lO1fanVPXv", "reviewer": "Reviewer_S3hq", "summary": "In this paper proposed the Motion-Priors Conditional Diffusion Model (MCDM), a diffusion-based framework designed for improved long-term motion generation to resolve the key challenges in existing methods, including lack of identity consistency, limited expression diversity, audio-motion synchronization issues, and error accumulation in long-term facial motion generation.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer S3hq:\n\nThank you for your detailed and constructive review. **We appreciate your recognition of our contributions, including the TalkingFace-Wild dataset and the proposed motion-prior framework.**  We address your concerns as follows.\n\n**Q1: Ambiguity in Audio-Motion Correlation**\n\n**Response:** Thank you for pointing this out. The original sentence \"*Reliance on static audio features and weak correlations between audio and 'motion' complicate the decoupling of identity and motion cues...*\".  'motion' refers specifically to **head motion**, not lip motion. Following [1-2], audio and head motion are only weakly correlated. We have clarified this distinction in the revised text.\n\n[1] Wang et al., \"V-express: Conditional dropout for progressive training of portrait video generation\"\n\n[2] Chen et al., \"Audio-driven Talking-Head Generation with Rhythmic Head Motion\" .\n\n**Q2: Missing Complexity and Efficiency Analysis for MTA**\n\n**Response:** Thank you for the detailed feedback. Standard temporal self-attention requires storing all key-value pairs, leading to quadratic memory growth and potential OOM issues in long sequences (e.g., V-Express in Table 7 is marked N/A due to this).\n\nIn contrast, our Memory-Efficient Temporal Attention (MTA) maintains a fixed-size memory via incremental updates, enabling stable and efficient long-term modeling.\n\nFurthermore, we provide a comparison (T=4, batch size=1):\n\n| **Method**        | **FLOPs (G) ↓** | **Training GPU Memory (GB) ↓** | **FID ↓** |\n|------------------|------------------|-------------------------------|------------|\n| Self-Attention     | 6.7              | 28.7                          | 46.29      |\n| **MTA (Ours)**     | **3.1**          | **16.4**                      | **42.08**  |\n\nMTA reduces FLOPs by **54%** and GPU memory by **43%**, while also improving FID. We will include this result in the final version.\n\n**Q3: Lack of Explanation for Multimodal Causality**\n\n**Response:** Thank you for highlighting this point. We apologize for the lack of clarity. By *multimodal causality (MC)*, we refer to modeling head motion, lip motion, and expression tokens jointly using a unified transformer conditioned on audio and reference image features. These conditioning tokens are prepended to the sequence and influence all motion tokens uniformly. Unlike modality-specific denoising, our model adopts a shared causal transformer to perform unified denoising across all motion cues. To quantify its effectiveness, we conducted an ablation study:\n\n| **Method**   | **FID ↓** | **FVD ↓** | **Sync-C ↑** | **Sync-D ↓** | **SSIM ↑** | **E-FID ↓** |\n|--------------|-----------|-----------|--------------|--------------|------------|-------------|\n| w/o MC       | 46.03     | 702.18    | 7.42         | 7.03         | 0.754      | 2.19        |\n| **w/ MC**    | **42.08** | **656.71**| **7.84**     | **6.69**     | **0.779**  | **1.97**    |\n\nThese results demonstrate that multimodal causality modeling leads to consistent improvements in identity preservation, synchronization, and overall generation quality.\n\n**Q4: Missing Comparison with Reference-Based and Seq2Seq Methods**\n\n**Response:** Thank you for the suggestion. As noted in Q2, reference-based methods (e.g.,EchoMimic) segment long videos to mitigate OOM, but often lead to identity inconsistency between clips. Seq2Seq models (e.g., SadTalker and EMO) lack memory mechanisms and are prone to error accumulation. Our unified memory-based framework avoids these issues by maintaining and updating a fixed-size memory for consistent long-term generation. We have added qualitative comparisons in the [link 1](https://files.catbox.moe/77kqif.mov), and [link 2](https://files.catbox.moe/1p4qo4.mov). which demonstrate superior identity consistency and visual quality.\n\n**Q5: Concerns About the User Study Design**\n\n**Response:** We would like to clarify that in our user study, the mapping between methods and filenames (e.g., A.mp4, B.mp4) was randomly shuffled for each evaluation task, and the correspondence was stored using a dictionary to ensure fair randomization and avoid naming bias.\n\nThe initial study involved 20 participants, which aligns with common practice in prior works [3]. To enhance statistical reliability, we extended the study to 40 participants. We made every effort to expand the subject pool within the available timeframe. The results are available at: [link3](https://files.catbox.moe/dm8rio.pdf).\n\n[3] Zhang et al., \"SadTalker: Learning Realistic 3D Motion Coefficients for Stylized Audio-Driven Single Image Talking Face Animation\".\n\n**In the end, thanks a lot for your detailed comments and thank  you for helping us improve our work! We appreciate your thoughts on our  work and we would be more than happy to discuss more during or after the  rebuttal to explore more in this direction. Please let us know if you  have any further questions. We are actively available until the end of  this rebuttal period.**", "claims_and_evidence": "In this paper claim that the weak correlation between audio and motion complicates the decoupling of identity and motion cues.\n\nHowever, audio and lip motion are strongly correlated. The author did not clearly whether “motion” refers to lip motion, head motion, or both, which creates ambiguity in evaluating the claim.", "methods_and_evaluation_criteria": "The proposed method is well aligned with the problem of speaking face generation.\nThe evaluation criteria (dataset, metrics) are adequate and widely used in previous work.\nHowever, the lack of efficiency evaluation weakens the memory-efficiency claim.\nThe lack of direct comparison with reference video-based methods makes it difficult to evaluate the benefits of the archived clip motion first.\n\nMemory-efficient attention lacks complexity analysis to justify the efficiency claim.\nThe concept of \"multi-causality\" is mentioned, but there is no explanation of how it exists.\nEven experimentally, there was no quantitative analysis of how much improvement was achieved when causality was broken.", "theoretical_claims": "Memory-efficient attention lacks complexity analysis to justify the efficiency claim.\n\nThe concept of \"multi-causality\" is mentioned, but there is no explanation of how it exists.\nEven experimentally, there was no quantitative analysis of how much improvement was achieved when causality was broken.", "experimental_designs_or_analyses": "The authors introduce Memory-Efficient Temporal Attention (MTA) to improve long-term stability.\nHowever, the authors does not provide the evidences that MTA is more memory-efficient than existing approaches.\n\nNo parameter comparison: The paper does not compare the parameter count of MTA with previous temporal attention mechanisms.\nNo memory usage evaluation: There is no quantitative measurement of GPU memory usage or computational cost (e.g., FLOPs, VRAM consumption).\nNo baseline comparison: The paper does not include comparisons with existing self-attention methods to demonstrate MTA’s efficiency.", "relation_to_broader_scientific_literature": "This paper tried to address several issues that arise in existing Talking Face Generation. \n\nTo address motion distortion and flickering that occur during long-term generation in GAN and diffusion models, we propose the Motion-priors Conditional Diffusion Model (MCDM). \n\nHowever, existing papers also try to preserve temporal information by using seq2seq specialized models to preserve temporal information well. Is there a reason why this discussion is missing?", "essential_references_not_discussed": ".", "other_strengths_and_weaknesses": "* Strengths:\n\nThe authors proposed named TalkingFace-Wild, a multilingual dataset with over 200 hours of high-quality video, is a valuable contribution to the research community.\n\nThe proposed Archived-Clip Motion-Prior Module effectively integrates long-term motion context, which is a promising direction for improving motion coherence.\n\n* Weaknesses:\n\nMemory efficiency is not empirically validated: No direct comparisons of parameter count, computational complexity (FLOPs), or VRAM consumption are provided.\n\nThe user study is limited in number of subjects (20 subjects), which raises concerns about the statistical reliability of the subjective evaluations.\n\nThe paper does not compare its approach to reference video-based motion modeling methods, which are commonly used to improve long-term consistency.", "other_comments_or_suggestions": "The explanation of “Multimodal Causality” in Section 3.2 is somewhat vague. It would help to provide a more concrete formulation or a visual representation to clarify how causality is modeled between audio and facial motion.\n\nThe authors claim that the Memory-Efficient Temporal Attention module is to be more efficient, but the paper does not provide direct comparisons with standard self-attention in terms of computational cost or memory usage. Including FLOPs or VRAM usage in Table 7 would strengthen the claim.", "questions_for_authors": "Memory-Efficient Temporal Attention:\n\n-\tThe authors mentioned Memory-Efficient Temporal Attention as a motivation, pointing out the GPU memory limitation of existing methods, but no quantitative evaluation was done. Can you provide a table comparing the efficiency of this method with existing methods? \n\nUser Study:\n\n-\tThe paper conducts a user study with 20 subjects, which may raise concerns about the statistical reliability of the results.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cE9hLOcllJ", "reviewer": "Reviewer_kTry", "summary": "This paper looks at generating long-form videos of talking faces with a view to improving the overall expressivity and naturalness of the head motion / facial expression accompanying the speech. A diffusion model is used for generation that is conditioned on various cues, which include historical motion context from other videos, local motion context for the current sequence, an encoding of the lips and facial expression, and optionally the facial landmarks to help guide the motion generation.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer kTry:\n\n**Thank you very much for  the detailed comments. We address your concerns as follows.**\n\n**Q1: Lack of Clarity in Technical Descriptions**\n\n**Response:** Thank you for the helpful comments. We address each point below:\n\n1. **Memory Cost of Past Frames vs. Archived Sequences:**\n   Standard temporal self-attention stores all key-value pairs, resulting in quadratic memory growth with the number of frames. In contrast, our Archived-Clip Motion-Prior Module integrates with Memory-Efficient Temporal Attention (MTA), which maintains a fixed-size memory via incremental updates, avoiding GPU memory overflow in long-term generation. Please see our experimentalanalysis in Q2 of R2 (Reviwer S3hq).\n2. **Landmark Token Drop:**\n   Yes, we replace dropped landmark tokens with zeros.\n3. **Typo in Line 299:**\n   Sorry, the symbol “\\alpha” was a typo, it should be the variable name “a” and is unrelated to Section 3.4.\n4. **Interpretation of Figure 5(a):**\n   We agree with your observation. Our intent was not to suggest identity drift. As stated in line 375, the absence of archived-clip priors leads to visible artifacts and inconsistencies in head, mouth, and expression, not identity change. We will clarify this wording in the revised version.\n\n**Q2: Issues in Subjective Evaluation Design**\n\n**Response:** Thank you for your thoughtful comments regarding the user study design. We address each concern below:\n\n1. **Rank Selection Protocol:**\n   Following prior works [1,2], we adopted a ranking-based evaluation strategy to effectively capture relative user preferences. While this approach may introduce some cognitive load, participants were allowed unlimited time and could freely replay videos to ensure fair and consistent judgments.\n   \n   [1] Ma et al., \"Follow Your Emoji: Fine-Controllable and Expressive Freestyle Portrait Animation\", SIGGRAPH Asia 2024.\n   \n   [2] Gao et al., \"FaceShot: Bring Any Character into Life\", ICLR 2025.\n2. **Time Control:**\n   There was no strict time limit. Participants were allowed to complete the task at their own pace.\n3. **Randomization and File Naming:**\n   We confirm that the method-to-letter mapping (e.g., A.mp4, B.mp4) was independently randomized for each session. The correspondence was stored using a dictionary to avoid fixed associations and mitigate bias.\n4. **Addition of MOS Study:**\n   Thank you for your  thoughtful advice.  Following your suggestions, we have additionally conducted a Mean Opinion Score (MOS) study with 20 participants. Each participant rated video quality on a 1–5 scale (5 = Excellent, 1 = Bad). The results are summarized below:\n\n| **Method**       | **MOS Score ↑** |\n|------------------  |------------------|\n| Audio2Head       | 1.6              |\n| V-Express        | 2.6              |\n| AniPortrait      | 2.9              |\n| SadTalker        | 3.2              |\n| Hallo            | 3.8              |\n| EchoMimic        | 3.7              |\n| MegActor-∑       | 3.5              |\n| **Ours (MCDM)**  | **4.3**          |\n\nThe experimental results demonstrate that our method achieved the highest MOS score, demonstrating strong user preference in terms of identity consistency and visual quality.\n\n**Q3: Writing and Terminology Issues**\n\n**Response:** Thank you for the reminder. We have fixed all mentioned issues, including terminology corrections, grammar errors, and formatting. We also reviewed the entire paper to correct other typos, table labels, and figure captions for consistency and clarity.\n\n**Q4: Concerns on Expressiveness and Visual Artifacts**\n\n**Response:** Thank you for the valuable feedback. We acknowledge that minor artifacts (e.g., around the chin or eyes) may appear in difficult cases involving large head movements, this remains an open challenge in the community. **Our work takes a meaningful step forward by proposing a memory-based generation framework and releasing a 200-hour multilingual dataset to help mitigate these issues.** Both quantitative and qualitative results validate the motivation of our approach, showing that our method consistently outperforms prior baselines in expressiveness and coherence. **More visual comparisons, including EMO (closed-source, official demo inputs), are provided in [link 1](https://files.catbox.moe/77kqif.mov) and [link 2](https://files.catbox.moe/1p4qo4.mov), demonstrating superior identity consistency and visual quality.**\n\n**Q5: Literature Framing and Terminology**\n\n**Response:** Thank you for the suggestion. We agree that referring to recent deep learning methods as “early” may be misleading. We have revised the introduction to more clearly distinguish early non-neural approaches from modern neural-based methods.\n\n**Lastly, thank you so much for helping us improve the paper and  appreciate your open discussions! Please let us know if you have any  further questions. We are actively available until the end of this  rebuttal period. Looking forward to hearing back from you!**", "claims_and_evidence": "Some aspects of the expressiveness is problematic in the example video.  For example, the head motion in the long-form sequence when the speaker rotates their head to their left and then back to the camera looks slow and seemingly unnatural.  Also, the eyes are staring and look unnatural.  The speech articulation looks very good, but I find other aspects of the selected videos do not support the claim that the work is significantly more expressive.", "methods_and_evaluation_criteria": "The authors curate a new dataset for benchmarking, which they will release.  Other standard datasets have been used.  The approach is tested against a number of state of the art methods.  Also, standard metrics are used to measure performance.", "theoretical_claims": "There are no theoretical claims in the paper.", "experimental_designs_or_analyses": "I have questions about the subjective evaluation.   Specifically:  1) are viewers expected to watch all eight sequences and select the top three?  This is a placing a high load on the viewers to keep track of which is the best three, and going back and forth to compare.  2) How long are viewers given to complete the task?   2) The text mentions randomization — can I confirm that the video labeled as A.mp4 is not always the same system that generates the A sequences?  The text seems to suggest that the sequences are randomized, but the filename shown (A.mp4) should not be the same for a given system each time as this risks introducing bias.", "relation_to_broader_scientific_literature": "Using the library of archived motion sequences to condition the generator is novel.  Similar work has been done encoding the expressiveness and eyes separately, but the application here is different from what I have seen.", "essential_references_not_discussed": "Modern literature is cited, but I am not sure it is fair to call the cited examples in the introduction “early methods”.  You can go back to the late 1990s to find work on creating expressive talking faces (albeit not deep learning / neural based methods).  The paper does cite the relevant recent literature.", "other_strengths_and_weaknesses": "Strengths:\n+ The objective metrics suggest the approach compares favorably against all baseline on all three datasets.\n+ The code and data will be released.\n\nWeaknesses:\n- There are artifacts in the output from the proposed system, especially around the chin.  I observed this in two of the examples.\n- See the questions regarding the subjective evaluation.  Why not do something more typical, like MOS?", "other_comments_or_suggestions": "Throughout the paper you use TalkingFace as one word — you should use “talking face” when discussing talking faces in general.\n\nIn Figure 1 for the present-clip motion-prior diffusion model:  you refer to the “express encoder”, which should read “expression encoder”.\n\nLine 132: “a archived clip” > “an archived clip”.\nLine 151:  “limited history frame” > “limited frame history”.\nLine 182: “and express encoder” > “and expression encoder”.\nLine 196: “Lip and Express Encoders” > “Lip and Expression Encoders”.\nLine 198: “lip and express encoders” > “lip and expression encoders”.\nLine 316: remove the underline.\n\nAppendix A is useful.", "questions_for_authors": "Q1:  The paper states that conditioning on too many past frames for the generated sequence is computationally expensive.  Why is it more expensive than conditioning on sequences from the archive?\n\nQ2:  If the landmark tokens are dropped, are they simply replaced with zeros?\n\nQ3:  On line 299: for the archived-clip motion-prior module, the text states \\alpha=16.  There is no \\alpha mentioned in the description of this module in Section 3.2.  Rather this appears in the description of the memory-efficient temporal attention in Section 3.4.\n\nQ4:  For the example showing the importance of Fa the text reads:  “Figure 5(a) indicates that without the archived-clip (w/o Fa), identity consistency worsens with frame progression“ — this does not look like a shift in identity, but rather image artifacts being introduced.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "N/A"}}]

Author rebuttal:

Meta-review: This work makes contributions in: 1). a ​motion-prior diffusion framework that enhances ID consistency and lip-sync quality, 2). a ​memory-efficient attention mechanism, and 3). ​TalkingFace-Wild, a valuable 200H multilingual dataset. The method is well-validated, and the released data benefits the community. The rebuttal successfully addressed concerns, with two reviewers raising their scores. All three reviewers now recommend acceptance. After reading the whole review process and careful discussion, the AC recommends ​Accept.

Authors are required to:
- Incorporate all new materials from the rebuttal into the paper.
- Guarantee dataset release before the camera-ready due.
Decision: Accept (poster)