MEMO: Memory-Guided Diffusion for Expressive Talking Video Generation
Authors: Longtao Zheng
Yifan Zhang
Hanzhong Guo
Jiachun Pan
Zhenxiong Tan
Jiahao Lu
Chuanxin Tang
Bo An
Shuicheng YAN
Venue: Accepted by TMLR ()

Abstract: Recent advances in video diffusion models have unlocked new potential for realistic audio-driven talking video generation. However, maintaining long-term identity consistency, achieving seamless lip-audio synchronization, and producing natural, audio-aligned expressions in generated talking videos remain significant challenges. To address these challenges, we propose Memory-guided EMOtion-aware diffusion (MEMO), an end-to-end audio-driven portrait animation approach to generate identity-consistent and expressive talking videos. Our approach is built around two key modules: (1) a memory-guided temporal module, which enhances long-term identity consistency and motion smoothness by developing causal motion memory to store information from an extended past context to guide temporal modeling; and (2) an emotion-aware audio module, which replaces traditional cross attention with multi-modal attention to enhance audio-video interaction, while detecting emotions from audio to refine facial expressions via emotion-adaptive layer norm. Extensive quantitative and qualitative results demonstrate that MEMO generates more realistic talking videos across diverse image and audio types, outperforming state-of-the-art methods in overall quality, lip-audio synchronization, identity consistency, and expression-audio alignment. Our model and video demos are available at https://memoavatar.github.io.

Published in Transactions on Machine Learning Research (03/2026)
MEMO: Memory-Guided Diffusion for Expressive Talking Video Generation
Longtao Zhengle Yifan Zhang2*1, Hanzhong Guo², Jiachun Pan?, Zhenxiong Tan',
Jiahao Lu2, Chuanxin Tang,, Bo Ant-2, Shuicheng Yan2.3
Nannyan Technological University
2 Skyywork AI National University of Singapore
"quual contribution Prroject lead
Reviewed on OpenReview:
https: // openreview. net/ pormf id= MFu
Abstract
Recent advances in video diffusion models have unlocked new potential for realistic audio- driven talking video generation. However, maintaining long-term identity consistency, achieving seamless lip-audio synchronization, and producing natural, audio-aligned expres- sions in generated (alking videos remain significant challenges. To address chese challenges, propose Memory-guided EMOtion-aware diffusion (MEMO), end-to-end audio- driven portrait animation approach to generate identity-consistent and expressive talking videos. Our approach is built around two key modules: (1) memory-guided temporal mod- ule, which enhances long-ter termtermmm identity consistency and motion smoothness by developing causal motion memory to store information from an extended past context to guide tem- poral modeling; and (2) an emotion-aware audio module, which replaces traditional cross attention with multi-modal attention to enhance audio-video interaction, while detecting emotions from audio to refine facial expressions via emotion-adaptive layer norm. Exten- sive quantitative and qualitative results demonstrate that MEMO generates more realistic talking videos across diverse image and audio types, outperforming satte-of-theaaae art meth- ods in overall quality, lip-andio synchronization, identity consistency, and expression-audio alignment. Our model and video demos are available at https:/ / ///mmmvvaaaarrr.maaa .iirrr.it............ io.
1 Introduction
Audio-driven talking video generation [42; 57; 66] has gained significant allention due broad impact on areas like virtual avatars and digital content creation, offering transformative possibilities in entertainment, education, and o-commerce. However, compared to text-to-vidco [17; 45; 43] imago-to-vides generation [3]. audio-driven talking video generation presents unique challenges. It requires not only generating synchronized lip movements from the audio, but also preserving long-term identity consistency of the reference image and producing natural expressions that align with the emotional tone of the audio. Balancing these demands while ensuring generalization across diverse audio and images makes this task especially challenging.
Recent advances video diffusion models [57; 65; 9; 12] have enabled more realistic audio-driven talking video generation. Most of these methods autoregressively generate videos with a fixed context window, conditioning the past 2-4 generated frames. They typically use cross attention incorporate audio guidance for video generation. Additionally, some incorporate fixed and predefined emotion label for the whole video to specify the emotion of the generated vidco [66; 56]. These approaches face three major challenges: long- term identity consistency, lip-audio synchronization, and natural expressions aligned with the audio. First, conditioning on a limited number of motion frames can lead to temporal error accumulation (i.e., drifting), especially when these frames contain artifacts (cf. Figure 4). Socond, the use of cross attention limits the level of interaction between audio and video streams. Third, applying a fixed emotion label throughout the video prevents the facial expressions from accurately reflecting the dynamic emotional shifts inherent in the

---

Published in Transactions on Machine Learning Research (03/2026)
R Reference Net
Reference
M Motion frames
image flow
Motion frame flow
VAE Encoder
Video flow
Audio flow
eference image
motion flow
Memorr update
Memory guide
Noisy tents
Diffusion Net
VAE Decoder
+
Wav2Vec
Audio 1||b:///|/
patiol module
Emotion Detection
motiion-waar audio module Memnry-guided
M Meroory
temporal module
Figure 1: MEMO is structured with Reference Net and a Diffusion Net. The core innovations of MEMO reside in two key modules within the Diffusion Net: the memory-guided temporal module and the emotion-aware audio module. These modules work in tandem to deliver enhanced audio-video synchro- nization, sustained identity consistency, and more natural expression generation.
audio. As a result, existing methods struggle with long-term consistency, lip-audio synchronization, and expression-audio alignment.
In this paper, we propose Memory-guided EMOtion-aware diffusion (MEMO), end-to-end audio-driven portrait animation approach. As shown in Figure 1, MEMO builds two key modules: (1) memory- guided temporal module and (2) an emotion-aware audio module. To ensure consistent facial identity and smooth transitions across long-duration videos, MEMO develops a memory-guided temporal module (cf. Section 4.1) that maintains a causal motion memory over an extended history of generated frames. This allows the model to use long-term history information to guide temporal modeling through linear attention, resulting in better identity consistency and less error accumulation that may occur in existing diffusion methods (cf. Figure 4). To improve líp-audio synchronization and align facial expressions with audio emotions, MEMO introduces an emotion-aware audio module (cf. Scction 4.2). This module replaces audio cross-attention in previous dilfusion methods with a more dynamic multi-modal al attetioon mechanism, enabling better interaction between audio and video during the diffusion process. Meanwhile, by dynamically detecting segment-level emotion cues from the audio, this module helps to subtly refine facial expressions via enotion-adaplive layer normalization, enabling the generation of expressive talking videos.
Extensive quantitative results and human evaluations demonstrate that our approach consistently outper- forms state-of-the-art methods in overall quality, lip-audio synchronization, expression-audio alignment, iden- tity consistency, and motion smoothness (cf. Table 1 and Figure 9). Additionally, diverse qualitative results highlight MEMO's strong generalization across various types of audio, images, languages, and head poses (cf. Figures 5-8), further showcasing the effectiveness of our method. Lastly, ablation studies further vali- date the distinct contributions the memory-guided temporal module, which enhances long- long-term identity consistency and motion smoothness (cf. Figures and 4), and the emotion-aware audio module, which significantly improves lip-audio alignment and expression naturalness (cf. Figures and 14).
Our method offers two main advantages previous talking video methods (e.g., EMO [57], Hallo2 [12], and Loopy [25]): (i) MEMO introduces novel memory-guide temporal module equipped with causal motion memory mechanism, effectively enhancing long-tern term temporal consistency and mitigating error accumulation; and (ii) MEMO introduces a new emotion-aware audio module that dynamically delects segment-level audio emotion to guide multi-modal attention, thereby enhancing lip-audio synchronization and expression-aus alignment in talking videos. To our best knowledge, both the causal memory update scheme and the audio emotion awareness mechanism are novel for this field. Our model and code are publicly available on https::/memoavatarr .gi github... io to support future research.
2

---

Published in Transactions on Machine Learning Research (03/2026)
2
Related Work
Audio-driven talking vidco generation aims to synthesize realistic and synchronized talking videos given driving audio and a reference face image. Early approaches only focused on learning lip-audio mapping while keeping other facial attributes static [55; 8; 42; 10; 67], which cannot capture comprehensive facial expressions. To resolve this, later research used intermediate motion representations (e.g., landmark coordinates, 3D facial mesh, and 3D morphable models) and decomposed the generation process into two stages, i.e., audio to motion and motion to video [76; 53; 68; 60; 9; 62). However, they often generate inaccurate intermediate representations from audio, which restricts the expressiveness and realism of videos.
Recent end-to-end methods, like EMO [57] and Hallo [65], can generate vivid portrait videos by fine-tuning
pre-trained diffusion models [45]. However, most of these methods used 2-4 motion frames [57; 65; 12] as tem- poral conditions for autoregressive generation of long videos. However, such a limited frame history can result in error accumulation over time when artifacts appear in the past 2-4 frames. Recently, Loopy [25] increased the number of motion frames to reduce this dependence, and used a temporal segment module to model cross-clip relationships. However, it still uses limited motion frames, whereas our proposed memory-guided linear attention module allows utilizing a broader range of motion frames to provide more comprehensive temporal guidance, thus mitigating error accumulation and enhancing long-term identity consistency. More- over, unlike prior methods that use cross attention to integrate audio features [57; 65; 12], we introduce multi-modal attention to improve lip-audio synchronization. Additionally, our method dynamically detects fine-grained emotion in the audio, which adaptively guides generation with emotion-adaptive layer normal- ization, leading to enhanced expression-audio alignment over prior emotion-aware methods [56; 66]. More related studics are provided in Appendix C.
3 Problem and Preliminaries
Problem statement. Given a reference image and audio as inputs, audio-driven talking video genera- tion [42; 57] aims to output a vivid video that closely aligns with the input audio and authentically replicates real human speech and facial movements. This task is challenging because it requires seamless lip-audio syn- chronization, realistic head movements, long-term identity consistency, and natural expressions that align with audio. Most existing diffusion- approaches [57; 65; 9] struggle with issues such as error accumula- tion, inconsistent identity preservation over time, limited lip-audio synchronization, unnatural expressions, and poor generalization.
Latent diffusion models and rectified flow loss. Our method is built upon the Latent Diffusion Model (LDM) [45]. framework designed to efficiently learn generative processes in a lower-dimensional latent space rather than directly operating on pixel space. During training, LDM first employs a pre- trained encoder E(-) to map high-dimensionall images into a compressed latent space, producing latent features 20 = El). Then, following the principles of Denoising Diffusson Probabilistic Models (DDPM) [22], Gaussian noise e is progressively added to the latent features over t discrete timesteps, resulting in noisy latent features = = vaizo + v1- aye, where o is variance schedule controlling how much noise is added. The diffusion model is then trained to reverse this noise-adding process by taking the noisy latent representation as input and predicting the added noise e. The objective function for training can be expressed as: -= V,,, (o.loo/0. - <[]3], where eş represents the noise prediction made by the diffusion network, and e represents conditioning information such as audio, or motion frames the context of talking video generation.
Subsequently, Stable Diffusion 3 (SD3) [15] refines this process by incorporating rectified flow loss [32], which modifies the traditional DDPM objective to:
L E, Bi... - -e12),
(1)
where X(t) 1/(1 t)2 and 2, is reparamcterized using linear combination as 2, = (1 t)=o + fc. This formulation leads both better training stability and more efficient inference. In light of these advantages, we adopt the rectified flow loss from SD3 in our training.
3

---

Published in Transactions on Machine Learning Research (03/2026)
Current frames
Motion frames
Video
Audio
Emotion Detection
h
m
Emo-AdaNorm
Emo-AdaNorm
h
K h
Vh
K y72
Vm
Self-Attention
Self-Attention
Update
(Eq. 3-4)
Emo-AdaNorn
Emo-AdaNorm
Memory
KV H
M KV
Emo-aware Multi-Modal Attention
MK
LayerNorm
LayerNorm
FeedForward
FeedForward
A
Memory guided
linear attention (Eq. 2)
out
Figure 2: Memory-guided temporal module.
Figure 3: Emotion-aware audio module.
4 Method
As illustrated in Figure MEMO is an end-to-end audio-driven diffusion model for generating identity- consistent and expressive talking videos. Similar to previous diffusion-based approaches [57; 65]. MEMO is built around two main components: Reference Net and Diffusion Net. The main contributions of MEMO lie in two key modules within the Diffusion Net: the memory- guidded temporal module (cf. Section 4.1), and the emotion-aware audio module (cf. Section 4.2), which work together to achieve superior audio- video synchronization, long-term identity consistency, and natural expression generation. Morcover, MEMO introduces a decomposed training stralegy (ef. Section 4.3), along with a new data processing pipeline (cf. Section 4.4).
4.1 Memory-Guided Temporal Module
Most cxisting diffusion methods [57; 65; 9] generate talking vidcos in an autoregressive manner by dividing the audio into clips of 12-16 frames and using the past 2-4 generated frames to guide the generation of the next video clip. Although this strategy can model short-term dependencies, it often struggles with maintaining long-term consistency. If artifacts generated in the 2-4 conditioned motion frames, these errors tend to accumulate generation progresses, leading to visual distortions that degrade both video quality and identity consistency (e.g., Hallo2 [12) in Figure 4). To address these issues, we propose memory-guided temporal module for long-term consistency. This module consists of two key designs: (1) memory-guided linear attention, and (2) causal memory update.
Memory-guided linear attention. To alleviate temporal error accumulation, our main intuition is that leveraging longer motion history can provide richer temporal context than solely relying on the most recent 2-4 frames. Inspired by this, propose to increase the number of motion frames to better guide temporal modeling, However, previous approaches use self-attention [57; 25) to model temporal relationships between frames, which requires storing all key-value pairs. As the number of motion frames incrcases, GPU memory overhead also increases. Therefore, it is challenging for self-allention to effectively utilize the history beyond its limited-length context window. To address this issue, we replace self-attention with linear attention [27] for temporal modeling. Specifically, denoting query as Q, key as K, and value as V, the output of lincar attention for i-th frame out, +(Q)) V (/(())) E/= o(K,), where f is the number of frames and the activation function (we use softmax in this work).
4

---

Published in Transactions on Machine Learning Research (03/2026)
To incorporate more motion frames to guide generation, we construct a motion memory to record the past generated frames. As shown in Figure 2, let the latent features of motion frames be E R/xdd and the latent features of current frames be h E Rxxdd where d is the dimension of latent features. Linear attention processes these latent features via learnable matrices, which transform them into queries (Qn), keys (Kn, m), and values (Vn: Vm). We then define the motion memory for the past I frames as two matrices: MKV = El= ((Km)) and M = E'= @(Km,s,, which occupy constant GPU memory irrespective of I. Based on the memory, the output of the memory-guided linear attention is formulated as:
out (QN)) (HKV + MxV) ((Q)) (Hx + Mx)
(2)
where HKV = (KK)V and Hx - 6(K))
Causal memory update. After each generation of f frames, we update the memory M by incorporating information from these newly generated frames. In formal, the memory update when adding the latest a frames to the memory with b motion frames is:
Mot 7 7 +
(3)
Mgt + 7 + E ((Kn)))
(4)
Here, y is a frame-level decay factor that modulates the influence of motion frames, with more recent frames exerting greater impact, reflected through the exponentiation by i. Such a decay scheme is critical, as a unified positional encoding across different video subsegments infeasible. Instead, we use causal memory decay to provide implicit positional encoding, which enables more effective memory updates capture long-term dependencies.
4.2 Emotion-Aware Audio Module
Previous talking video methods face two key limitations in handling audio conditions. First, the cross- attention commonly used in prior approaches [57; 65; 9] provides limited interaction between audio and video modalities, restricting their effectiveness in achieving better lip-audio synchronization. Second, ex- isting emotion-aware techniques [66; 56] typically rely on a fixed emotion label for the entire video, which fails to capture the dynamic emotional shifts in audio, causing misaligmment between the generated facial expressions and the audio's emotion. To address these challenges, we propose a new emotion-awant audio module to improve lip-audio synchronization and align facial expressions with the underlying audio emotion. Specifically, our audio module distinguished by the following three core designs: (1) multi-modal allention, (2) audio emotion-aware diffusion, and (3) emotion decoupling training.
Multi-modal attention. Our emotion-aware audio module replaces the traditional cross attention with more dynamic multi-modal attention mechanism. Specifically, cross attention aligns video and audio by conditioning the process of video features v on audio features a, which can be formalized as minimizing the loss function (, = ((,.)) (-tt)oooo((()lllel [C(t) -)]]] In contrast, we explore multi-modal attention, which jointly processes both video and audio inputs by minimizing the loss function Le (v,,-s((o,)) [eel a) ell)). Specifically, as shown in Figure 3, we use two modality-specific branches for audio and video, within which an intermediate step concatenates their outputs into a unified token sequence for multi-modal self- attention. This enables better video-audio interaction during the diffusion process, thus achieving enhanced lip-audio synchronization.
Audio emotion-aware diffusion. To enhance expression-audio alignment, our audio module seeks to dynamically detect audio emotions to guide audio-video interaction. To this end, we train new emotion detector to extract emotion from audio (See Appendix E for more implementation details and robustness discussions of the audio emotion detector), recognizing eight distinct emotions: angry, disgusted, fearful, happy, neutral, sad, surprised, and others. To improve the robustness of emotion labels, we conduct detection the segment level. Each segment's emotion determined by the most frequently detected emotion across all its frames, where each frame's emotion is evaluated using audio features from a 3-second sliding window centered on that frame.
5

---

Published in Transactions on Machine Learning Research (03/2026)
We then incorporate the segment-level audio emotion via emotion-adaptive layer normalization (Emo- AdaNorm) for expression-audio alignment. As shown in Figure 3, the detected emotion of each segment is first projected into an emotion embedding e, which is then integrated into each layer via Emo-AdaNorm to guide multi-modal attention. Specifically, e is processed through a lincar modulation [40] to produce a scaling factor a and a shifting factor B. These factors adaplively modulate the LayerNorm (LN) putputs, i.e., EmooAdaNorm(z)) = (I+)) thereby conditioning the normalization explicitly on the detected emotion labels. This approach transforms Eq. into the following emotion-conditioned flow loss:
Li R (oro - -((²).
During inference, we use classifier free guidance (CFG) [21] to control the effect of emotion on the output. The emotion-awan aware output computed as
Eo(Ue, ale) = (1 + w)ea(v, (lee) - wes(ve. a),
(6)
where w is the CFG scale to control the influence of the emotion condition. Eq. 6 is simplified to highlight CFG for emotion, but in practice, CFG is similarly applied to both audio and the reference image. It should be noted that the overall emotional tone of the generated talking video is largely inferred from the facial expression in the reference image. The audio emotion serves more as subtle adjustment to enhance the emotional expression in response to audio cues, rather than completely overriding the facial expression provided by the reference image.
Emotion decoupling training. To improve the effect of audio emotion in talking videos, we develop an emotion decoupling training strategy to separate the expression in the reference image from the audio emotion. Specifically, for training video clips sourced from MEAD [61]-which provides both speaker identity and emotion labels we avoid using reference image from the same video clip. Instead, we randomly select a reference image of the same person but with a different emotion. This encourages a better disentanglement emotion-aware module
between the reference image's expression and the audio-indueee emotion, enabling to better refine facial expressions in alignment with the audio. Our method also supports replacing the detected audio emotion label with a manually specified emotion label, if desired.
4.3 Training Strategy Decomposition
The training of MEMO is divided into two progressive stages, each with specific objectives.
Face domain adaptation. We initialize Reference Net and the spatial module of Diffusion Net with the weights of SD 1.5 [45]. In this stage, we adapt Reference Net, the spatial attention modules of Diffussio Net, and the original text cross-attention module to the face domain with the rectified flow loss (cf. Eq. 1), ensuring these components capture facial features effectively.
Stage 1:
Stage 2: Emotion-ddcoupled robust training. We then integrate the cmotion-aware audio modulc and memory-guided temporal module into the Diffusion Net. Initially, we perform a warm-up training phase for the newly added modules, keeping the modules in Stage fixed. After the warm-up, all modules are jointly trained. In this stage, we use the emotion-conditionee flow loss (cf. Eq. 5) and scale up the dataset to include all processed data for more comprehensive training. Here, we adopt the emotion decoupling training strategy (cf. Section 4.2) only when the training video clips are sourced from MEAD. Moreover, we found that some noisy data persisted even after applying our data processing pipeline (ef. Section 4.4), making diffusion training unstable and leading to biased model optimization. To mitigate this, we further develop robust, training strategy that. filters out data points with loss values suddenly exceeding specific large value (0.1 in our case), as the emotion-conniitooned flow loss in our method typically converges and fluctuates around 0.03. Please refer to Appendix D for more implementation details.
4.4 Data Processing Pipeline
We collect comprehensive set of open-source datasets, such as HDTF [73], VFHQ [63], CelebV-HQ [77]. MultiTalk [54], and MEAD [61], along with additional data collected by ourselves. The total duration of
6

---

Published in Transactions on Machine Learning Research (03/2026)
Reference
Hallo2
Ours
Audio -
Reference
Hallo2
Ours
Audio *
Reference
Hallo2
Ours
Audio *
Start frames
Middle frames
End frames
Figure MЕMO generates long talking videos with improved identity consistency, lip-audio alignment, and motion snoot.n contrast, existing methods (e.g., Hallo2 [12]) are prone to drifting during autoregressive generation, especially when the last 2-4 generated frames used temporal conditions contain artifacts, leading to inconsistent identity. Please refer to the supplementary material for video demos.
these raw videos exceeds 2,200 hours. However, as rllustrated in Appendix F, we find that the overall quality of the data is poor, with numerous issues such as lip-audio misalignment, missing heads, multiple heads, occluded faces by subtitles, extremely small face regions, and low resolution. Directly using these data for model Training results in unstable training, poor convergence, and terrible generation quality.
To further obtain high-quality talking head data, we developed a dedicated data processing pipeline for talking head generation. The pipeline consists of five steps: First, perform scene transition detection and trim video clips to length of less than 30 seconds. Second, videos with no faces, partial faces, multiple heads, and use the resulting bounding boxes to extract talking heads. Third, we use an Image Quality Assessment model [51] to filter out low-quality and low-resolution videos. Fourth, we apply SyncNet [42] to remove videos with lip-andio synchronization issues. Lastly, we manually assess the lip-audio synchronization and overall video quality for subset of the data ensure more accurate filtering. After completing the entire pipeline, the total duration of our processed high-quality videos about 660 hours.
apply face detection, filtering out
5 Experiments
5.1
Experimental Setup
Evaluation benchmarks. We create two distrr-iit (OOD) datasets to evaluate MEMO's perfor- mance and generalization. For the first OOD dataset, we sample 150 video clips from the VoxCeleb2 [37]

---

Published in Transactions on Machine Learning Research (03/2026)
Reference
Generated frames
Referance
Generated frames
Speach
Portat
Singing
Soupphire
Digital Ang
Rap
Figure The generated vidcos with various types of driving audio. Please refer to the supplementary material for video demos.
Figure 6: The generatod vidcos with various types of reference images. Please refer to the supplementary material for video demos.
test set, with videos of various celebrities. Similarly, we create the second OOD dataset with 150 clips across more diverse set of audio, backgrounds, ages, genders, and languages.
Evaluation metrics. We adopt suite of metrics to evaluate the overall quality and lip-audio synchro- nization of the generated videos. FVD [58] measures overall video quality with the distance between the distributions of real and generated videos. FID [20] evaluates the quality of individual frames by com- paring feature distributions extracted from a pre-trained model. For lip-audio synchronization, we report Sync-C and Syne-D [11] using a pre-trained discriminator model [42]. To evaluate identity consistency, we compute the cosine similarity between face embeddings (Identity). To assess facial expressiveness, we use Expression-FID (E-FID), following [57].
Baselines. We compare method against several state-of-the-art baselines with publicly available model checkpoints. The baselines include both two-stage methods with intermediate reprosentations and end-to- end diffusion methods. V-Express [60], AniPortrait [62] and EchoMimic [9] are two-slage methods using intermediate representations like landmarks, while Hallo [65] and Hallo2 [12] are recent end-to-end diffusion- based models.
Role of training data vs. architecture. We acknowledge that stronger training data can partially contribute to overall gains. MEMO is trained using officially released checkpoints, each trained with its own (often undisclosed) data. As most baselines do not release complete training data and code, fully controlled retraining under one unified pipeline is infeasible. To reduce this confounder, all module-wise ablations in Tables 2 and 3 are trained and evaluated with the same setting to isolate the contributions of the proposed momory and emotion modules.
curated dataset, while compared baselines evaluated
5.2 Quantitative Results
Performance on two OOD test sets. Table reports the quantitative results on two OOD test sets. Our method consistently outperforms all baselines terms of FVD, FID, and Syne-D metrics, indicating better video quality and lip-audio synchronization. These results also demonstrate the improved gencralization abilities of MEMO to unseen identities and audio. In addition to achieving the best or comparable results Sync-C C and E-FID, MEMO shows clear advantages in identity consistency across two test sets.
Human evaluation. To better benchmark the quality of generated talking vidcos, we conduct human studies based on five subjective metries in several challenging scenarios, e.g., singing, rap, and multi-lingual talking video generation. Specifically, our analyses are based on the overall quality, motion smoothness, expression-audio alignment, lip-audio synchronization, and identity consistency. As shown in Figure 9, our method achieves the highest scores across all criteria in human evaluations, much higher than compared methods. This further demonstrates the effectiveness of our approach.
8

---

Published in Transactions on Machine Learning Research (03/2026)
Reference
Generated framas
Referance
Generated frames
Engitsh
Lef 00
Murdamin
Left 30
Spanieh
AKY
AKT
Front
Japainese
Right 30
Koreen
Right 80
Figure 7: The generated videos on driving audio with different languages. Please refer the supplemen- tary material for vidco demos.
Figure 8: The generated videos with reference images of different head poses. Please refer the supplemen- tary matcrial for vidoo demos.
Table 1: Quantitative results of vidco quality, lip-audio synchronization, identity consistency, and expres- siveness on two OOD lest datasets. MEMO consistently outperforms existing talking video baselines. For F-FID, s_" denotes omission where the face-recognition model cannot reliably identify enough faces to yield a statistically reliable estimate.
VoxCeleb2 test set
Collected OOD dataset
Method
FVDL
FID
Syne-Ct Ct
Syne-DI
Identityt
E-FID,
FVDJ
FID,
Syne-CT
Syne-Di
Ideutity"
E-FIDJ
SudTallerr [08]
397.0
258.7
0.83
AniPortrait [62]
333.2
45.5
2.8
11.0
0.80
9.5
238.7
31.2
3.5
10.5
(1300
6.3
V-Express [60]
118.9
58.9
6.4
8.2
0.82
315.2
16.7
6.1
9.5
0.87
[63]
0.75
0.88
7.3
Hallo2 [12]
302.0
41.6
0.77
7.3
223.1
29.8
5.5
0.88
6.9
EchoMinie [s]
293.9
13.8
3.7
10.1
0.67
223.9
39.9
4.5
9.8
0.89
6.9
MEMO (Ours)
251.3
31.7
7.1
0.85
6.2
161.1
24.9
5.7
9.2
0.92
6.1
5.3
Qualitative Results
Long-duration talking video generation. Figure and Figure 11 demonstrate that MEMO can generate long-duration videos while consistently maintaining the subject's facial features and expression fidelity over thousands of frames. The resulting videos exhibit smooth motion and high temporal coherence, showcasing the robustness of our approach for long video synthesis. These capabilities make MEMO a superior choice over existing methods for applications requiring extended video content with stable generation quality.
Generalization different types of audio. We evaluate MEMO on various audio types, including speeches, songs, and raps. In Figure 5, MEMO consistently generates synchronized lip movements across diverse audio types. It performs well with both expressive songs, which demand muanced emotional align- ment, and raps, which require rapid lip-andio synchronization. This verifies MEMO's strong generalization to various types of driving audio.
Generalization to different styles reference images. Figure shows MEMO's performance on challenging reference images of diverse styles, such as portraits, sculpture, and digital art images. Despite these styles deviating significantly from our training data, MEMO maintains robust generation quality without producing noticeable artifacts, demonstrating its generalizability to OOD images.
9

---

Published in Transactions on Machine Learning Research (03/2026)
EEpprees
AniPortrail
EchaMimic
MEMO
Overal Quality
2f
Molion Smoothnes
Expressien-Aure Audio Align.
0.4
Win rate
Audio-Lip up Sync.
Identily Consistancy
0.0
Identity Consistency
Audio-Lip Sync.
Expression-AUude Align.
Overall Quality Motion Smoothness
Preference Proportion
Figure 10: Human preferences on the number of Figure Human preferences among MEMO and base- motion frames (f), where 16+f indicates lines in terms of five metrics.
text beyond 16 frames.
Reference
1k frames
10k frames
20k frames
AKT
AKT
AKT
AKT
AKT
AKT
AKT
E
E
EE
Z
CE
EE
EE
Figure 11: MEMO can generate long-durates videos with alleviated error accumulation and maintain iden- tity consistency. Please refer to the supplementary for video demos.
Generalization to multilingual audio. Figure demonstrates that MEMO achieves robust generalization across multilingual audio inputs, such English, Chinese, Spanish, Japanese, and Korean. Despite most of our training data being in English, our approach effectively generates lip movements synchronized with the given multilingual audio while capturing rich, realistic facial expressions.
Generalization to different head poses. Figure 8 shows the generated videos of MEMO with reference images of varying head poses, including frontal views and multiple side angles. This demonstrates that our method can generate realistic talking videos across different angles while maintaining consistency in facial appearance and expression. In Figure 6, we also demonstrate that MEMO generalizes to different head poses for more challenging reference images. While enhancing head movements is not the focus of our work, method can still generate talking videos with bel head motion diversily compared to previous open-source approaches (cf. Figure 12). This increased motion diversity enhances the naturalness and expressiveness of the generated videos.
5.4 Ablation Studies
To evaluate the effects of method components, we conduct ablation studies based on quantitative metrics, human evaluation, and qualitative analysis. More discussions provided in Appendices A and B.
10

---

Published in Transactions on Machine Learning Research (03/2026)
Generated
Motion
Generated
Motion
V-Express
ART
Hallo
EchoMimic
AKT
ANT
AAE
ANT
1
Ours
Figure 12: MEMO can generate talking videos featuring a wider range smooth head movements and more emotional facial expressions, illustrated both visualization and heatmaps.
Noutral
Angry
Happy
Sad
Overall Quality
Multi-modal attention
Cross attention
Motion Smoothness
Reference
Expression-Audee Audio
Audio Align.
Audio-Lip Sync.
Identity Consistency
Reference
Preference Proportion
Figure 13: Human preferences between multi-modal attention and cross attention for integrating audio conditions the audio module.
Figure 14: Ablation of emotion decoupled
training.
Effects of memory module. We evaluate the impacts of our memory module through both human eval- uation and quantitative ablations. In Figure 10, longer memory siguificantly improves temporal coherence, overall quality, motion smoothness, identity consistency, and lip-audio alignment, while short motion frames lead worse performance. This demonstrates the effectiveness of our memory-guided temporal module and also explains why our method alleviate temporal error accumulation in Figure 4, whereas Hallo2 is prone to error accumulation. For quantitative ablations, we control the history information available to the memory module and compare three settings: memory with only the initial reference image, memory with only the most recently generated frames, and full memory. As shown in Table 2, using the full memory yields the best FVD, Sync-C, and head-motion statistics.
Effects of multi-modal attention. We further investigate the impact of the multi-modal attention through human evaluations. Results in Figure 13 underscore the effectiveness of multi-modal attention over cross attention in terms of the overall video quality and lip-audio alignments.
Effects of cmotion-aware scheme. To investigate the effects of our emotion-aware scheme (i.e., emotion- aware diffusion and imolion decoupling training), we conduct ablations through both quantitative metrics and qualitative analysis. In Figure 14, using the same reference image and different emotion labels, we compare the frames generated by our method with and without emotion decoupling training the same audio moment. The results demonstrate that our emotion-aware module with emotion decoupling training can effectively refine facial expressions to align with the specified emotion labels. This finding suggests that,
11

---

Published in Transactions on Machine Learning Research (03/2026)
Table 2: Controlled quantitative ablation of the memory mod- ule on the collected OOD datasct.
Table 3: Controlled quantitative ablation of the emotion module on the collected OOD dataset.
Method
FVD4
Sync-Ct
Head Motiont
Reference image only
175.2
4.32
1.54
Method
FVD
FID4
LPIPS!
Recent frames only
165.9
5.67
4.03
No emotion
163.0
25.0
0.268
Full memory (Ours)
161.1
5.74
4.19
Ours
161.1
24.9
0.263
with our detected emotion labels, MEMO can generate facial expressions that match the audio emotion. Moreover, this comparison also validates the necessity of our emotion decoupling training. Quantitatively, we mamually control the emotion label to static to test visual quality metrics to validate the effectiveness of the emotion module. As shown Table 3, the emotion module consistently improves FVD, FTD, and LPIPS over the variant without emotion conditioning. While these improvements are modest automatic metrics, they are consistent with the stronger user preference observed in our human study, suggesting that the emotion-aware design improves both perceptual expressiveness and overall generation quality. We do not report off-the-shelf emotion-classfier scores because, in our experiments, such classifiers are unreliable on synthesized videos and do not provide a stable proxy.
6 Conclusion
This work has presented MEMO for talking video generation. MEMO effectively alleviates temporal error accumulation and enhances long-term identity consistency via a new memory-guides lemporal module, while generating videos with high lip-audio and expression-audic alignment via a new emotion-aware audio module. There are several promising directions remaining as future work below.
Future work: (1) Integrating Diffusion Transformer backbones [39] could enhance generation quality and expressiveness of talking videos. (2) Incorporating a dedicated head motion conditioning mechanism could enable more dynamic head motion. (3) Extending MEMO to broader applications, such as talking body generation, could unlock new possibilities for human video generation.
Ethics Statement
We acknowledge that audio-driven talking video generation has strong positive potential in education, ac- cessibility, virtual assistants, and entertainment, but also carries serious misuse risks. The same capability can be abused for impersonation, non-consensual synthetic media, fraud, harassment, and political misin- formation. These risks are especially severe when real identities, voices, or sensitive contexts are involved. We restrict the release of our model and code to research use and explicitly prohibit malicious, misleading, defamatory, or privacy-infringee
applications. We require users to obtaín proper authorization for all inputs (e.g., reference images and audio) and to comply with applicable laws, personality rights, and copyright regulations. For downstream deployment, strongly recommend adding provenance signals (e-g., water- marking/content content credentials), maintaining auditable usage logs, and including human review before public dissemination of generated videos. Safeguards reduce but do not eliminate misuse. In addition, model be- havior may vary across demographics, languages, and acoustic conditions due to data bias. We therefore encourage carcful risk assessment, transparent disclosure of synthetic content, and continued research on detection and responsible release practices.
Acknowledgments
This research is supported by the Ministry of Education, Singapore, under its MOE AcRF Tier 2 Award MOE- SDDE-22PP2222--000
Any opinions, findings and conclusions or recommendations expressed in this ma- terial are those of the author(s) and do not reflect the views of the Ministry of Education, Singapore. In addition, this research is supported in part by NUS Start-up Grant A-0010106-00-00 and by the National Natural Science Foundation of China under Grant No. 62320106007.
12

---

Published in Transactions on Machine Learning Research (03/2026)
References
[4] Adaeze Adigwe, Noć Tits, Kevin El Haddad, Sarah Ostadabbas, and Thierry Dutoit. The emotional voices database: Towards controlling the emotion dimension in voice generation systems. arXiv preprint arXiv:1806. 0.0114, 2018.
[2] Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vec 2.0: A framework for self- supervised learning of speech representations. In Advances in Neural Information Processing Systems, volume 33, 2020.
pp.
[3] Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint wxxi:2311.15112, 2023.
[4] Dmitry Bogdanov, Minz Won, Philip Tovstogan, Alastair Porter, and Xavier Serra. The MTG-Jamendo dataset for automatic music tagging. In International Conference on Machine Learning Workshop, 2019.
[5] Felix Burkhardt, Astrid Paeschke, Miriam Rolfes, Walter F Sendlmeier, Benjamin Weiss, et al. A database of German emotional speech. In Interspeech, 2005.
[6] Salih Firat Canpolat, Zuhal Ormanoglu, and Deniz Zeyrek. Turkish emotion voice database (TurEV-DB). In LREC Workshop Language Resources and Evaluation Conferene, pp. 368 (58-375,, 2020.
7 Fabio Catania. Speech emotion recognition in Italian using wav2vec 2. Authorea Preprints, 2023.
[8] Lele Chen, Zhiheng Li, Ross K. Maddox, Zhiyao Duan, and Chenliang Xu. Lip movements generation glance.
In European Conference on Computer Vision, pp. 520-535, 2018.
(9] Zhiyuan Chen, Jiajiong Cao, Zhiquan Chen, Yuming Li, and Chenguang Ma. Echomimic: Lifelike audio-driven portrait animations Uhrough editable landmark conditions. arx preprinl wxXi::2407.88166 2024.
[10] Kun Cheng, Xiaodong Cun, Yong Zhang, Menghan Xia, Fei Yin, Mingrui Zhu, Xuan Wang, Jue Wang, and Nannan Wang. ViteoReta, Audio-based lip synchronization for talking head video editing in the wild. SIGGRAPH Asia, pp. 1-9, 2022.
[1] Joon Son Chung and Andrew Zisserman. Out of time: Automated lip sync the wild. In Asian Conference on Computer Vision Workshops, pp. 251-263. Springer, 2017.
[12] Jiahao Cui, Hui Li, Yao Yao, Hao Zhu, Haulin Shang, Kaihui Cheng, Hang Zhou, Siyu Zlu, and Jingdong Wang. Hallo2: Long-duration and high-resolution audio-driven portrait image animation. In International Conference on Learning Representations, 2025.
[13] Kate Dupuis and Kathleen Pichora-Fuller, M. Toronto emotional speech set (Less) younger talker happy. Tochnical Report, University of Toronto, Psychology Department, 2010. Datasel.
[14] Mathilde M Duville, Luz M Alonso-Valerda and David I Ibarra-Zarate. The Mexican emotional speech database (mesd): Elaboration and assessment based on machine learning. In Annual International Conference of the IEEE Engineering in Medicine & Biology Sociely, pp.
2021.
[15] Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Muller, Harry Saini, Yam Levi, Do- minik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. In International Conference on Machine Learning, 2024.
166 Philippe Gournay, Olivier Lahaie, and Roch Lefebvre. A Canadian French emotional speech dataset. In ACM Multimedia Systems Conference, pp. 399 099-402, 2018.
[17] Yuwei Guo, Ceyuan Yang, Anyi Rao, Zhengyaug Liang, Yaohui Wang, Yu Qiao, Maneesh Agrawala, Dahua Lin, and Bo Dai. AnimateDifft Animate your personalized rext-to-image diffusion models without specific tuning. International Conference on Learning Representations, 2023.
[18] Tianyu He, Junliang Guo, Runyi Yu, Yuchi Wang, Kaikai An, Leyi Li, Xu Tan, Chunyu Wang, Han Hu, MisiangThoo Wu, et al. GAIA: Zero-shot talking avatar generation. In International Conference on Learning Representations, 2023.
13

---

Published in Transactions on Machine Learning Research (03/2026)
[19] Xianglong He, Chunli Peng, Zexiang Liu, Boyang Wang, Yifan Zhang, Qi Cui, Fei Kang, Bino Jiang, Mengyin Au, Yaugyaug Ren, et al. Matrix-game 2.0: An open-source real-time and streaming interactive world model. arXiv preprint or 1aoo:2508.19000 2025.
[0]] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Beruhard Nessler, and Sepp Hochreiter. GANs trained by a two time-scale update rule converge to a local Nash equilibrium. In Advances in Newral Information Processing Systems, volume 30, 2017.
21] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXio preprint N.6:22077112900 2022.
[2]] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusiio probabilistic models. In Advances in Newral Information Processing Systems, volume 33, pp.
2020.
[23) Philip Jackson and SJUoSG Haq. Surrey audio-visue expressed emotion (savee) database. University of Surrey: Guildford, UK, 2014.
[24] Jesin James, Li Tian, and Catherine Watson, An open source emotional speech corpus for human robot inter- action applications. In Interspeech, 2018.
[25] Jianwen Jiang, Chao Liang, Jiaqi Yang, Gaojie Lin, Tianyun Zhong, and Yanbo Zheng. Loopy: Taming andio- driven portrait avatar with long-term motion dependency. In International Conference on Learning Representa- tions, 2025.
[26] Dorota Kaminska, Tomasz Sapinski, and Adam Pelikant. Polish emotional natural speech database. In Proceed- ings of the Conference: Signal Processing Symposium, 2015.
[27] Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Trannformers are RNNs: Fast autoregressive transformers with linear attention. In International Conference on Machine Learning, pp. 5156- 5165. PMLR, 2020.
288] Leila KERKENI, Catherine CLEDER, Youssef Serrestou, and Y Raood. French emotional speech database- oréau, 2020.
229 Dejoli Landry, Qianhua He, Haikang Yan, and Yanxiong Li. ASVP-ESD: A dataset and its benchmark for emotion recognition using both speech and non-speech utterances. Global Scientific Journals, 8:1793- 1798, 2020.
300 Siddique Latif, Adnan Qayyum, Muhammad Usman, and Junaid Qadir. Cross lingual speech emotion recog- nition: Urdu vs. Western languages. In International Conference on Frontiers of Information Technology, pp. 88 93, 2018
31] Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al. Crounding dino: Marrying dino with grounded pre-training for open-set object detection. arXie preprint Xix::303 05499, 2023.
332] Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: with rectified flow. In International Conference Learning Representations, 2023.
Learning to generate and transfer data
[33] Steven R Livingstone and Frank A Russo. The ryerson audio-visual database of emotional speech and song (ravdess)): A dynamic, multimodal set of facial and vocal expressions in north american english. PloS One, 13 (5):901963 2018.
[34] Ziyang Ma, Mingjie Chen, Hezhao Zhang, Zhisheng Zheng, Wenxi Chen, Xiquan Li, Jiaxin Ye, Xie Chen, and Thomas Hain. EmoBox: Multilingual multi-corpus speech emotion recognition toolkit and benchmark. In Interspeech, 2024.
[35] Ziyang Ma, Zhisheng Zheng, Jiaxin Ye, Jinchao Li, Zhifu Gao, Shiliang Zhang, and Xie Chen. emotion2vec: Self-supervised pre-training for speech emotion representation. In Findings of the Association for Computational Linguistics, pp. 15747-15760, 2024.
366 Olivier Martin, Trene Kotsia, Benoit Macq, and Toannis Pitas. The enterface'Os '05 andio-visual emotion database. In International Conference on Data Engineering Workshops, pp. 8 8, 2006.
(37] Arsha Nagrani, Joon Son Chung, Weidi Xie, and Andrew Zisserman. Voxceleb: Large-scale speaker verification in the wild. Computer Speech & Longuage, 60:101027, 2020.
14

---

Published in Transactions on Machine Learning Research (03/2026)
[38] Kari Ali Noriy, Xiaosong Yang, and Jian Jun Zhang. Emns/tnnzcorpus: An emotive single-speaker dataset for narrative storytelling in games, television and graphic novels. arXiv preprinl arXie:2905. 19197, 2023.
[39] William Peebles and Saining Xie. Scalable diffusion models with transformers. In International Conference on Computer Vision, pp. 4195-4200 2023.
[10]
Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville. Film: with general conditioning layer. In Proceedings of the AAAI conference on artificial intelligence, 2018.
Visual reasoning
[41] Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. SDXL: Improving latent diffusion models for high-resolution image synthesis. In International Conference on Learning Representations, 2023.
[42] KR Prajwal, Rudrabla Mukhopadhyay, Vinay Namboodiri, and CV Jawahar. A lip syne expert is all you need for speech to lip generation in the wild. In ACM International Conference on Multimedio, pp. 482-492, 2020.
[33 Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXio:2204. 201:2204.06125, 2022.
(44] Ephrem Afele Retta, Eiad Almekhlafi, Richard Sutcliffe, Mustafa Mhamed, Haider Ali, and Jun Feng. A new amharic speech emotion dataset and classification benchmark. ACM Transactions on Asian and Low- Resource Language Information Processing, 22(1):1 2023.
[45] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Computer Vision and Pattern Recognition, pp. 10684-10695, 2022.
[16] Fei Shen, Cong Wang, Junyao Gao, Qin Gua, Jisheng Dang, Jinhui Tang, and Tat-Seng Chua. Long-term talkingface generation via motion-prior conditional diffusion model. arXiv preprint ar arXiv:2502.09333 2025.
[47] Jascha nonequilibriu thermodynamics. In International Conference on Machine Learning, pp. 2256 2265. PMLR, 2015.
Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using
48) Tomás Soucek and Jakub Lokoe. Transnet v2: An effective deep network architecture for fast shot transition detection. In ACM International Conference on Multimedia, pp. 1222111221, 2024.
[99 Ingmar Steiner, Mare Schröder, and Annette Klepp. The PAVOQUE corpus as a resource for analysis and synthesis of expressive speech. Proceedings of Phonetik &l Phonologie, 9, 2013.
[50] Michal Stypulkowski, Konstantinos Vougioukas, Sen He, Maciej Zięba, Stavros Petridis, and Maja Pantic. Dif- fused heads: Diffusion models beat gans on talking-face face generation. In Proceedings of the IEEE/CVE Winter Conference on Applications of Computer Vision, pp. 5091-5100, 2024.
[5]] Shaolin Su, Qingsen Yan, Yu Zhu, Cheng Zhang, Xin Ge, Jinqiu Sun, and Yanning Zhang, Blindly assess image quality in the wild guided by selfadappivve hyper network. In Computer Vision and Pallern Recognition, June 2020.
[52] Sadia Sultana, M Shahidur Rahman, M Reza Selim, and M Zafar lqbal. Sust bangla emotional speech corpus (subesco)) An audio-only emotional speech corpus for bangla. Plos one, 16(4):e0250173, 2021.
533 Xusen Sun, Longhao Zhang, Hao Zhu, Peng Zhang, Bang Zhang, Xinya Ji, Kangneng Zhou, Daiheng Gao, Liefeng Bo, and Xun Cao. Vividtalk: One-shot audio-driven talking head generation based on 3D hybrid prior. Xiv preprint 2rX.v:2312.018411 2023.
54] Kim Sung- Bin, Lee Chae- Yeon, Gihun Son, Oh Hyun-Bin, Janghoon Ju, Suekyeong Nam, and Hyun Oh. MultiTalk: Enhancing 3D talking head generation across languages with multilingual video dataset. arXiv preprint arXiv:2406.112 14272, 2024.
[55 Supasorn Suwajanakorn, Steven M Seitz, and Tra Kemelmaachr--Shiizemma Synthesizing obama: Learning lip sync from audio. ACM Transactions on Graphics, 36(4):1-13, 2017.
566 Shuai Tan, Bin Ji, and Ye Pan. Flowvqtalker: High-quality emotionnl talking face generation through normalizing flow and quantization. In Computer Vision and Pattern Recognition, pp. 26317-26327, 2024.
15

---

Published in Transactions on Machine Learning Research (03/2026)
[57] Linrui Tian, Qi Wang, Bang Zhang, and Liefeng Bo. EMO: Emole portrail alive-generating expressive portrait videos with audio2video diffusion model under weak conditions. In European Conference on Computer Vision, 2024.
588 Thomas Unterthiner, Sjoerd van Steenkiste, Karol Kurach, Raphaäl Marinier, Marcin Michalski, and Sylvain Gelly. FVD: A new metric for video generation, 2019. URL
(59) Nikolaos Vryzas, Rigas Kotsakis, Aikaterini Liatsou, Charalampos A Dimoulas, and George Kalliris. Speech emotioo recognition for performance interaction. Journal of the Audio Engineering Sociely, 66(6))457-44) -47,, 2018.
[60] Cong Wang, Kuan Tian, Jun Zhang, Yonghang Guan, Feng Luo, Fei Shen, Zhiwei Jiang, Qing Gu, Xiao Han, and Wei Yang. V-Express: Conditional dropout for progressive training of portrait video generation. arx preprint Xix:224406 02511, 2024.
[61] Kaisiyuan Wang, Qianyi Wu, Linsen Song, Zhuoqian Yang, Wayne Wu, Chen Qian, Ran He, Yu Qiao, and Chen Change Loy. MEAD: A large-scale andio-visual dataset for emotional talking-face face generation. In Europeon Conference on Computer Vision, pp. 700-717. Springer, 2020.
[2) Huawei Wei, Zejun Yang, and Zhisheng Wang. Aniorrtriit: Audio-driven synthesis of photorealistic portrait animation. arXie preprint aixi:22403. 176644
17694, 2024.
[63] Liangbin Xie, Xintao Wang, Honglun Zhang, Chao Dong, and Ying Shan. VFHQ: A high-quality dataset and benchmark for video face super-resolution In Computer Vision and Pottern Recognition, pp. 657-666, 2022.
664 Jinbo Xing, Menghan Xia, Yong Zhang, Haoxin Chen, Xintao Wang, Tien-TTin Wong, and Ying Shan. Dy- namiCrafter: Animating open-domain images with video diffusion priors. arXie preprint XXi::2910.121900 2023.
65) Mingwang Xu, Hui Li, Qingkun Su, Hanlin Shang, Liwei Zhang, Ce Liu, Jingdong Wang, Lue Van Gool, Yao Yao, and Siyu Zhu, Hallo: Hierarchical audio-driven visual synthesis for portrait image animation. arviv preprint or arxiv:2406.0m 08801,080001.00. 2024.
66) Sicheng Xu, Guojun Chen, Yu-Xiao Guo, Jiaolong Yang, Chong Li, Zhenyu Zang, Yizhong Zhang, Xin Tong, and Baining Guo. VASA-1: Lifelike audio-driven talking faces generated in real time. In Advances in Newral Information Processing Systems, 2024.
(67] Fei Yin, Yong Zhang, Xiaodong Cun, Mingdeng Cao, Yanbo Fan, Xuan Wang, Qingyan Bai, Baoynan Wu, Jue Wang, and Yujiu Yang. StyleHeat: One-shot high-resolution editable talking face generation via pre-trained stylegan. In European Conference on Computer Vision, pp. 85-101. Springer, 2022.
688 Wenxuan Zhang, Xiaodong Cun, Xuan Wang, Yong Zhang, Xi Shen, Yu Guo, Ying Shan, and Fei Wang. SadTalker: Learning realistic 3D motion coefficients for stylized audio-driven single image talking face animation. In Computer Vision and Paltern Recognation, pp. 8652 8661, 2023.
[69] Yifan Zhang and Bryan Hooi Hipa: Enabling one-step text-to-image diffusion models via high-frequency- promoting adaptation. Xrxiv preprint arXi:23111.18158 2023.
(70] Yifan Zhang, Bryan Hooi, Dapeng Hu, Jian Liang, and Jiashi Feng. Unleashing the power of contrastive self- supervised visual models via antrass-reeguarizer fine-tuning. In Advances in Neural Information Processing Systems, 2021.
[71] Yifan Zhang, Daquan Zhou, Bryan Hooi, Kai Wang, and Jiashi Feng. Expanding small-scale datasets with guided imagination. In Advances in nearal information processing systems, volume 36, pp. 76558 76618, 2023.
(22 Yifan Zhang, Chunli Peng, Boyang Wang, Puyi Wang, Qingeheng Zhu, Fei Kang, Biao Jiang, Zedong Gao, Fric Tí, Yang Liu, et al. Matrix-game: Interactive world foundation model. arXio preprint ar 250: 18701, 2025.
(33] Zhimeng Zhang, Lincheng Li, Yu Ding, and Changjie Fan. Flow-guided one-shot talking face generation with a high-resolution audio-visual dataset. In Computer Vision and Pattern Recognition, pp. 3661-3670, 2021.
[74] Jinming Zhao, Tenggan Zhang, Jingwen Hu, Yuchen Liu, Qin Jin, Xinchao Wang, and Haizhou Li. M3ed: Multi-modal multi-scene multi-label emotional dialogue database. In Association for Computational Linguistics, pp. 5699-5710, 2022.
16

---

Published in Transactions on Machine Learning Research (03/2026)
[75] Kun Zhou, Berrak Sisman, Rui Liu, and Haizhou Li. Seen and unseen emotional style transfer for voice conversion with a new emotional speech dataset. In International Conference on Acoustics, Speech and Signal Processing, pp. 920 924. IEEE, 2021.
76] Yang Zhou, Xintong Han, Eli Shechtman, Jose Echevarria, Evangelos Kalogerakis, and Dingzeyu Li. Makelttalk: Speaker- awere aware talking-heas head animation. ACM Transactions On Graphics, 39(6):1-15, 2020.
[77] Hao Zhu, Wayne Wu, Wentao Zhu, Liming Jiang, Siwei Tang, Li Zhang, Ziwei Liu, and Chen Change Loy. CelebV-HQ: large-scale video facial attributes dataset. In European Conference on Computer Vision, pp. 650-667. Springer, 2022.
17

---

Published in Transactions on Machine Learning Research (03/2026)
A More Qualitative Results
Comparisons with baselines. Figure 15 showcases comparisons between talking videos generated by MEMO and baseline models on sampled out-of-distribution (OOD) data. Specifically, some baseline models (e.g., Hallo [65] and Hallo2 [12]) tend to produce artifacts and fail to preserve the original identity and fine details. While certain methods (e.g., AniPortrait [62] and V-Express [60]) generate videos with fewer artifacts, they suffer from poor lip-audio synchronization and motion smoothness. In contrast, our method demonstrates the ability to produce more natural facial expressions and head movements that are well- aligned with the audio imput. Additionally, the videos generated by MEMO exhibit superior overall visual quality and stronger identity consistency.
More visualization of emotion-guided generation. The facial expressions of the generated talking video are influenced by both the expressions in the reference image and the emotional tone of the audio, As discussed in Section 4.2, the overall emotional tone of facial expressions is inferred mainly from the facial expression of the reference image, while our audio emotion-aware module functions mainly as a subtle ment to enhance or moderately alter the emotion when prompted by the audio. Figure 14 in Section 5.4 has demonstrated that given a fixed reference image, MEMO can refine the facial expressions of talking videos based on the given audio emotion.
In this appendix, further explore the flexibility of our method by evaluating its ability to generate expres- sive talking videos using reference images depicting the same person with different emotional expressions, such as neutral, angry, happy, and sad. To isolate the effect of reference image expressions, we set the audio emotion label to match the emotional state of each reference expression. As shown in Figure 16, our method adapts scamlessly to diverse emotional states, generating highly expressive and cmotionally consistent talk- ing videos. These results highlight the robust.ness and versatility of MEMO leveraging both reference expressions and audio cues to create emotionally nuanced talking videos.
B More Ablation Studies
Classificr-free guidance scale. By adjusting the classifter-free guidance scale, observe variations in the expressiveness of the generated faces. As shown in Figure 17, higher guidance scales lead to more pronounced emotional expressions in talking videos.
Dynamic motion frame training. Appendix D presents a dynamic motion frame training strategy as a supplementary approach. The ablation results of this strategy, evaluated through human preference studies, are shown in Figure 18. Specifically,, our method incorporating this strategy consistently receives higher human preference scores, highlighting its effectiveness in enhancing overall quality, motion smoothness, expression-audio alignment, lip-audio synchronization, and identity consistency.
Inference-time runtime and memory. Table 4 compares inference latency and GPU memory under the same hardware and inference settings. The memory mechanism adds a modest runtime overhead (+10.0 ms/step, about +2.5% compared with our variant without memory) while improving long-term consistency. It also introduces additional GPU memory for persistent memory states (about +3.2 GB peak in this setup). Importantly, this overhead is bounded with respect to generated vidco length bocause the memory uses fixed-sive states with dynamic updates.
Table 4: Inference-time runtime and GPU momory comparisons.
Method
Avg step time (ms)
Avg allocated (GB)
Peak allocated (GB)
Hallo
327.1
5.5
8.0
Ours (w/o memory)
404.0
5.8
8.2
Ours (w/ memory)
414.0
9.0
11.5
18

---

Published in Transactions on Machine Learning Research (03/2026)
Generated frames
AniPortrait
V-Express
Hallo
Hallo2
39
Ours
Figure 15: Visualization of generated vidcos on the OOD datasct. Existing methods cither have poor lip- audio synchronization (e.g,, AniPortrait [62]) or suffer from error accumulation (e.g., Hallo [65]). ). In contrast, MEMO generates talking videos with natural head motion and accurate lip-audio synchronization without artifacts. Please refer to the supplementary for video demos.
C Further Related Work
Diffusion models [47; 22; 69; 71] highly expressive generative models, demonstrating remarkable capabil- itics in image synthesis [45; 41] and video generation [17; 64; 72; 19). Stable Diffusion [45] employs U-Net architecture and generates high-resolution images in the latent space, which exlended to video domains by AnimateDiff [17] by adding temporal attention layers. These models generate images or videos based on prompts, where the text guidance from the pre-trained text encoder introduced through cross atten- tion. In the domain of talking video generation, diffusion models also show promising results in generation quality [18; 57; 62; 65; 50; 66), outperforming previous GAN-based methods (42; 76). Instead of using text
19

---

Published in Transactions on Machine Learning Research (03/2026)
Reference
Generated frames
Neutral
Angry
Happy
Sad
Figure 16: More visualization of expressive talking videos generated by MEMO based on reference images with various emotions. Please refer to the supplementary for video demos.
Overall Quality
w/ DPF
w/o DPF
Motion Smoothness
Expres Alon- Align.
Audio-Lip Sync.
Identity Consistency
Preference Proportion
scale 2
Scale
scale
soalle
Figure 17: Ablation of the classifier-free guidance scale.
Figure 18: Human preferences between MEMO variants with and without dynamic motion frames (DPF) training.
prompts, most of these ceattsioonbassee methods condition diffusion models image and audio embeddings extracted from pre-trained image encoder and audio encoder, respectively.
A recent work, MCDM [46], also proposes a motion-prior module to mitigate error accumulation in long talking video generation. While MCDM also improves temporal modeling abilities, its memory mechanism mainly focuses on motion priors and comparatively less effective in ensuring cross-frame consistency and long-ted term identity preservation. Compared MCDM, MEMO not only achieves more expressive and emotionally aligned facial animations through the emotion-aware audio module, but also addresses long-
20

---

Published in Transactions on Machine Learning Research (03/2026)
term drifting via memory-guided linear attention. This allows MEMO to maintain identity consistency and smooth motion across long video sequences.
D More Implementation Details
Both Reference Net and the spatial module of Diffusion Net are initialized with the weights of SD 1.5 [45]. The temporal module of Diffusion Net is initialized from AnimateDiff [17]. Here, the Reference Net provides identity information from reference images for spatial modeling of the Diffusion Net and offers temporal information from motion frames for temporal modeling of the Diffusion Net. For both Reference Net and Diffusion Net, we replace the text cross- atssstottention with image cross-attention. We add two projection modules to convert the audio embedding and image embedding into the dimonsions required by our attention module. The audio embedding consists of all the hidden stales from the Wav2Vec 2.0 model [2].
The training videos are center-cropped and resized to a resolution of 512 x 512 pixels. With fixed learning rate of 1c-5, train MEMO for 15k and 600k steps at training stages 1 and 2, respectively. During Stage 2, fixed number of 16 motion frames is used to compute memory states as molion context (cf. Eq. 3 and Eq. 4). Specifically,
MRv + E) Po(Kn,)) M + Σ) 7 (Knn)),
where the memory decay factor y is set to 0.9. Besides 16, the number of motion frames can be dynamically chosen from 16, 32, or 48 for training, enabling the model to handle longer context scenarios more effectively. Nevertheless, thanks to the memory update mechanism with causal history decay (cf. Section 4.1), such dynamic motion frame training is unnecessary. Moreover, emotion embeddings, reference images, audio embeddings, and motion frames are randomly dropped with probability of 5% for classifier-free inference. At inference, we set the frame rate to 30 frames per second (FPS) and employ autoregressive generation, producing 16 frames per iteration. The CFG scale is set to 3.5. When generating each sub-clip, the memory only needs to be updated once during the first denoise step, and subsequent denoise steps will not incur additional computational overhead. More details on the ablation of the CFG scale, dynamic motion frame training, and y can be found in Appendix B. Our model trained on NVIDIA A800 GPUs.
E
Audio Emotion Detection
To improve the natural expression of talking videos, we develop an emotion detection model to detect emotion labels from audio. In this appendix, we first introduce the data collection and processing strategies for audio emotion recognition, followed by the details of our emotion detector.
E.1 Dataset Collection and Processing
Dataset collection. To achieve robust emotion detection across both speech and music audio sources, we collect a large-scale dataset encompassing both speech and music segments, each annotated with emotion labels. A detailed overview of the datasets used our training process provided in Table 5. For speech audio, we collect data from a recent Speech Emotion Recognition benchmark, EmoBox [34], which incorpo- rates 23 datasets from various origins, covering 12 distinct languages. Regarding music audio, we gather data from the RAVDESS-Sone [33] and MTG-Jamendo [4] datasets, including songs with and without background music.
All data underwent. a standardized processing protocol, converted to a monophonic formal with a sampling rate of 16,000 Hz. Each utterance is uniquely annotated with an emotion label. For datasets containing lengthy samples, such as MTG-Jamendo, we divide them into shorter segments of scconds to align with the typically shorter length of other datasets, assigning the same label all segments. Each dataset was then split into training and testing sets with ratio of 3:1.
21

---

Published in Transactions on Machine Learning Research (03/2026)
Table 5: Statistics the emotion detection Dataset. The source column represents the origin of the samples, and the language column specifics the dataset's language. #Emo indicates the number of emotion categories. # Utts shows the total number of utterances. #Hrs represents the total hours of training data.
Speech Emotion Recognition datasets
Dataset
Source
Language
#Emo
#Utts
#Hrs
AESDD [59]
Act
Greek
604
0.7
ASED [44]
Act
Amharic
2,474
2.1
ASVP-ESD [29]
Media
Mix
12
13,964
18.0
CaFE [16]
Act
French
936
1.2
EMNS [38]
Act
English
8
1,181
1.9
EmoDB [5]
Act
German
535
0.4
EmoV-DB [1]
Act
English
6,887
9.5
Emozionnalmente [7]
Act
Italian
6,902
6.3
eNTERFACE [36]
Act
English
1,263
1.1
ESD [75]
Act
Mix
35,000
29.1
JL-Corpus [24]
Act
English
2,400
1.4
M3ED [74]
TV
Mandarin
7
24,437
9.8
MEAD [61]
Act
English
31,729
37.3
MESD [14]
Act
Spanish
862
0.2
Oreau [28]
Act
French
434
0.3
PAVOQUE [49]
Act
German
7,334
12.2
Polish [26]
Act
Polish
450
0.1
RAVDESS [33]
Act
English
1,440
1.5
SAVEE [23]
Act
English
7
480
0.5
SUBESCO [52]
Act
Bangla
7
7,000
7.8
TESS [13]
Act
English
2,800
1.6
TurVV-DB [6]
Act
Turkish
1,735
0.5
URDU [30]
Talk show
Urdu
400
0.3
Music Emotion Recognition Datascts
Dataset
Source
Lang
Emo
# Utts
# #Hrs
RAVDESS-Song [33]
Act
English
1,012
1.31
MTG-Jamendo [4]
Media
Mix
56
5,022
299.47
Label merging. A major challenge in integrating different datasets is aligning their label spaces, as each dataset often features distinct emotion categories. For instance, the URDU dataset [30] contains only four emotion labels: happy, sad, angry, and neutral. In contrast, the ASVP-ESD dataset [29] includes 12 emotion labels, covering less common emotions such as boredom and pain. For music emotion recognition datasets like MTG-Jamendo [4], there are 56 mood/themo tags, not all of which correspond to emotional labels, and each sample can be assigned multiple lags. These discrepancies and overlaps in category spaces across different datasets present significant challenges for emotion detection.
To establish a generalized label space, we design our module to perform an 8-class classification task, selecting labels that are both commonly recognized and easily distinguishable: angry, disgusted, fearful, happy, neutral, sad, surprised, and others. We meticulously review and map the original labels from each dataset to fit within this new label space. For instance, samples labeled as pleasure in the ASVPESD dataset are mapped the happy category due their semantic similarity. Labels that do not clearly correspond to specific emotion categorized under the others label.
22

---

Published in Transactions on Machine Learning Research (03/2026)
Table Accuracy comparison of audio emotion detection between Emotion2vec [35] detector.
and our learned emotion
Datasct
Emotion2vec
Ours
AESDD
75.84
78,52
ASED
86.20
85.23
ASVP-ESD
52.55
55.99
CaFE
73.30
100.00
EMNS
57.98
61.87
EmoDB
88.41
100.0
EmoV-DB
77.84
91.22
Emmozionalmente
66.61
71.02
eNTERFACE
28.21
32.05
ESD
94.83
99.94
-or-Cosuus
71.92
100.00
M3E
42.59
41.52
MEAD
61.74
71.45
MESD
40.65
41.12
Oreau
50.96
42.31
PAVOQUE
85.15
92.74
Polish
44.89
100.00
RAVDESS
82.36
100.00
SAVEE
83.33
100.00
SUBESCO
78.43
100.00
TESS
76.29
95.14
TurEV-DB
47.45
53.47
URDU
54.00
56.00
RAVDESS-SOns
43.58
100.00
MTG-
65.30
74.50
Total
68.78
78.26
E.2 Audio Emotion Detector
We train an 8-way classifier for our task, drawing inspiration from state-of-the-art methods in speech and music emotion detection. Our solution builds upon Emotion2vec [35], & robust universal speech emotion reprosentation model. The feature extractor employs multiple convolutional layers and transformer blocks. It is trained using a teacher-student online distillation self-supervised learning approach. The feature extractor backbone of Emotion2vec is pre-trained on a large-scale multilingual speech corpus. For our classification lask, we use the fixed Emotion2vec backbone as the feature extractor and train a 5-layer MLP as the classification head. To stabilize the training process, we apply gradient clipping, constraining the gradient updates within an l2 norm of 1.0. To enhance the model's generalization ability, we incorporate a contrastive learning technique [70].
The test accuracy of our trained emotion detector for each dataset, well the overall accuracy, is reported in Table 6. Specifically, our trained detector achieves better performance than the original Emotion2vee [35], which adopted a single linear layer after the feature extraction backbone for downstream emotion detection.
E.3 Robustness of Audio Emotion Detection
One possible concern that the potential inaccuracy in audio emotion detection may lead to unreliable emotion labels. As shown in Table 6, although our emotion detector is not perfect, has already surpassod existing open-source models in accuracy. This facilitates more reliable emotion detection and dynamic emotion adjustments in talking video generation. Moreover, detailed in Section 4.2, to enhance the
23

---

Published in Transactions on Machine Learning Research (03/2026)
Issue
No Mouth
Partial Face
Low Resolution
Audio-lip Async
Sample
Issue
Multi-heads
Sublities take up large portion
Human face under-represented
EXAMPLES OF THINGS CAN RAP ABOUTH INDIA, CANDY, KOBE BRIAANT STEAK, SPORTS, MMA, CONOR MCGREGOR, GÓNS STEVE TYEE THE CREATC
Sample
SUBSCRIBE
Figure 19: Examples of issues in the raw dataset.
robustness emotion labels, our emotion detection is performed at the segment level. Specifically, each segment's emotion determined by the most frequently detected emotion across its frames, where each frame's emotion is evaluated using audio fcatures extracted from a 3-sccond sliding window centered around that frame. This strengthens the reliability of detected emotion labels.
E.4 Robustness of Audio Emotion Detection
One possible concern that the potential inaccuracy in audio emotion detection may lead unreliable emotion labels. As shown in Table 6, although emotion detector is not perfect, it has already surpassed existing open-source models in accuracy. This facilitates more reliable emotion detection and dynamic emotion adjustments in talking video generation. Moreover, as detailed in Section 4.2, to enhance the robustness of emotion labels, our emotion detection is performed at the segment level. Specifically, each segment's cmotion determined by the most frequently detected emotion across its framos, where cach frame's emotion evaluated using audio features extracted from a 3-second sliding window centered around that frame. This strengthens the reliability of detected emotion labels.
More importantly, stated in Scction 4.2, the emotional tone of the generated vidco is primarily driven by the facial expression in the reference image, with audio-derived emotion labels providing subtle refinements. This further enhances model robustness, ensuring emotional coherence in generated videos even with noisy labels. Video demonstrations provided in the supplementary material further validate the effectiveness of our method.
F Data Processing Pipeline
We collect a comprehensive set of open-source datasets, such as HDTF [73], VFHQ [63], CelebV-HQ [77]. MultiTalk [54], and MEAD [61], along with additional data we collected ourselves. The total duration of these raw videos exceeds 2,200 hours. However, as illustrated in Figure 19, we find that the overall quality of the data is poor, with numerous issues such as líp-audio misalignment, missing heads, multiple heads, occluded faces by subtitles, extremely small face regions, and low resolution. Directly using these data for model training results in unstable training, poor convergence, and terrible generation quality.
To further obtain high-quality talking video data, we developed a dedicated data processing pipeline for talking head generation. The pipeline consists of five steps:
First, we perform scene transition detection based on TransNet V2 [48] and trim video clips length of less than 30 seconds.
24

---

Published in Transactions on Machine Learning Research (03/2026)
Sync-C Distribution
score=5
Frequency
Figure 20: Distribution of the Sync-C in the CelebV-HQ.
Second, we apply face detection based on Grounding DINO [31], filtering out videos with faces, partial faces, or multiple heads, and use the resulting bounding boxes to extract talking heads. To ensure that the cropped areas encompass more than just the human faces, we apply a scaling factor 1.1 to the bounding box regions.
Third, we use HyperIQA [51], an image quality assessment model, to filter out low-quality and low- resolution videos. We apply HyperIQA to the first frame of each video and find that when the IQA score exceeds 40, there is noticcable improvement in overall vidco quality. Therefore, selection threshold, but this threshold can be dynamically adjusted depending on the dataset quality requirements.
IQA score of 40
Fourth, we utilize SyncNet [42] to filter out videos with lip-audio synchronization issues. The SyncNet Confidence (Sync-C) metric used as the basis for filtering. Figure illustrates the confidence distribution of the CelebV-HQ dataset, where threshold of 5 is applied for filtering. This threshold, like others, can be dynamically adjusted based the dataset quality requirements.
Lastly, manually check the lip-audio synchronization and overall video quality for more accurate filtering for subset of the data. After completing the entire pipeline, the total duration of the processed high-quality videos is about 660 hours.
25


Reviews:
[{"review_id": "VDtNtPWIrj", "reviewer": "Reviewer_sG8R", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents MEMO, a diffusion-based talking-face generation system with two key components: 1. a memory-guided temporal module using linear attention over a compact memory state to improve long-term identity consistency and motion smoothness in long videos; 2. an emotion-aware multi-modal module, which replaces standard cross-attention with audio–visual multi-modal attention and modulates the UNet via emotion-adaptive LayerNorm using emotions detected from audio (or optionally user-specified labels).\n\nThe authors also construct a large, cleaned multi-dataset corpus and evaluate on VoxCeleb2 and two OOD benchmarks with both automatic metrics and human studies.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The paper claims that MEMO improves overall video quality, lip–audio synchronization, identity consistency, and expressiveness over prior talking-face methods, and Table 1 shows gains in FVD, FID, Sync-C/D, identity similarity.\n\nThe architectural claims about the two key modules are supported by mathematical descriptions and a detailed training strategy. \n\nThe claim that longer temporal context mitigates error accumulation and improves long-term consistency is supported by qualitative long-video comparisons (Figure 4) and a human study varying memory length (Figure 10). \n\nHowever, there are also some claims that are only partially backed by evidence: the paper does not provide module-wise quantitative ablations on standard metrics (e.g., FVD / Sync-C with and without the memory or emotion modules), and there is no automatic classifier-based metric for emotion consistency. \n\nBaselines are not retrained on the same cleaned dataset, which slightly weakens the fairness of the “consistently outperforms” claim.", "audience_yn": "Yes", "audience_explanation": "A diffusion-based talking-face generation system is definitely of interest for many individuals in TMLR's audience.", "requested_changes": "There are some changes that would be a nice addition to the paper. \n\n- Add clearer module-wise ablations\n- Clarify fairness of comparisons: be explicit about the training data used for MEMO vs each baseline and briefly discuss how much of the gain might come from better data rather than only the architecture. Retraining one baseline on the same data would be ideal, but even just a discussion would be a plus.\n- Include a light compute / efficiency summary", "broader_impact_concerns": "This method definitely raises some concerns on broader impact. In particular, regarding Deepfakes / impersonation / non-consensual media. A system that can generate highly realistic, emotion-controlled talking faces is inherently dual-use. It can be used for benign applications but also for impersonating real people, producing non-consensual or defamatory content, or targeted political persuasion. The paper should explicitly acknowledge these risks and discuss potential safeguards (watermarking, detection tools, usage restrictions, consent requirements).", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "n0V5LQLYGR", "reviewer": "Reviewer_koxi", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper works on audio-driven talking video generation. The authors argued previous models mostly use a fixed context window and only conditioned on fixed emotion labels. The authors suggested to use a memory guided temporal module and a emotion aware audio model to ensure the consistency on long rollouts and better emotion conditioning. The memory guidance is achieved by encoding the reference image and motion frames with a reference net as the input for diffusion net. The audio condition is achieved with a pretrained wav2vec model and newly trained emption detector. The proposed method achieved better quantitive results compared to previous state of the art and could alleviated error accumulation in long-duration video. \n\nStrength:\n- The model will be open-sourced to boost research of related community. \n- The overall design makes sense and achieved better results with reasonable increase in the complexity of model architecture.\n\nWeakness:\n- The contribution on overall methodology is incremental as the proposed method is a close follow-up of previous literatures with limited changes to the input modality and model architecture.\n- The main paper does not include quantitive ablative study on the effectiveness of each design module. Quantitive metrics will help readers better understand the necessity of each module for future model design choices.\n\nIn conclusion, the paper provides valid insights of audio-driven video generation. The author also promised to open-source their checkpoints and source code which would be beneficial for future research. However, the contribution is limited and might have limited impacts to other research community as the use of multi-modality conditioning and memory guidance have been abundant in related fields.", "claims_and_evidence_yn": "Yes", "claims_explanation": "Yes, the authors provided quantitive and qualitative evaluation of the proposed method in comparison of the previous methods.", "audience_yn": "Yes", "audience_explanation": "Yes, the improvements on long video generation and audio conditioning could be beneficial to folks working on audio-driven video generation.", "requested_changes": "Ablative study on the effectiveness of each design module. Quantitive metrics will help readers better understand the necessity of each module for future model design choices.", "broader_impact_concerns": "The video generative models with human subjects are a sensitive fields with regard to privacy and legal concerns. The authors have included an ethics statements but should still act with caution.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "dt67Dj0tbv", "reviewer": "Reviewer_bbTE", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes MEMO for expressive talking video generation that introduces a motion memory module into a latent diffusion model. Unlike prior talking-head approaches that generate frames sequentially without temporal feedback, MEMO maintains motion memory matrices that record key representations of the past frames. These are used to guide the denoising process for future frames, improving temporal consistency and expressiveness.", "claims_and_evidence_yn": "Yes", "claims_explanation": "1. The paper provides clear architectural definitions and an explicit formulation of motion memory matrices, not just a conceptual claim.\n2. Quantitative results show notable improvements in almost all metrics compared to previous baselines.\n3. Ablations demonstrate that removing motion memory causes visible temporal degradation, confirming its contribution.\n4. Visual results are consistent with quantitative metrics, where videos show smoother transitions and more expressive detail.\n5. Overall, this is a very good work.", "audience_yn": "Yes", "audience_explanation": "TMLR’s community includes researchers in topics included in this paper such as multimodal generation, speech-driven animation, and diffusion-based video synthesis.", "requested_changes": "1. The paper would be significantly strengthened by a more detailed analysis and/or visualization of the motion memory mechanism. While Section 3.2 defines the memory as two matrices that record the past f frames, the paper does not illustrate what specific temporal or expressive information these matrices capture during generation. A qualitative or quantitative visualization, such as plotting the attention/related matrix module weights over time, or showing how memory activations evolve through the diffusion steps, would make the mechanism’s contribution clearer. \n2. How does the memory mechanism affect training and inference cost? A brief analysis of runtime and memory usage would be helpful.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)