X-NeMo: Expressive Neural Motion Reenactment via Disentangled Latent Attention
Authors: XIAOCHEN ZHAO
Hongyi Xu
Guoxian Song
You Xie
Chenxu Zhang
Xiu Li
Linjie Luo
Jinli Suo
Yebin Liu
Venue: ICLR 2025 Poster (2025)
Keywords: Portrait Animation
Head Avatar
Conditional Video Generation
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: We propose X-NeMo, a novel zero-shot diffusion-based portrait animation pipeline that animates a static portrait using facial movements from a driving video of a different individual. Our work first identifies the root causes of the limitations in prior approaches, such as identity leakage and difficulty in capturing subtle and extreme expressions. To address these challenges, we introduce a fully end-to-end training framework that distills a 1D identity-agnostic latent motion descriptor from driving image, effectively controlling motion through cross-attention during image generation. Our implicit motion descriptor captures expressive facial motion in fine detail, learned end-to-end from a diverse video dataset without reliance on any pre-trained motion detectors.  We further disentangle motion latents from identity cues with enhanced expressiveness by supervising their learning with a dual GAN decoder, alongside spatial and color augmentations. By embedding the driving motion into a 1D latent vector and controlling motion via cross-attention instead of additive spatial guidance, our design effectively eliminates the transmission of spatial-aligned structural clues from the driving condition to the diffusion backbone, substantially mitigating identity leakage. Extensive experiments demonstrate that X-NeMo surpasses state-of-the-art baselines, producing highly expressive animations with superior identity resemblance. Our code and models will be available for research.

Published as a conference paper at ICLR 2025
X-NEMo: EXPRESSIVE NEURAL MOTION REENACT- MENT VIA DISENTANGLED LATENT ATTENTION
Xiaochen Zhao Hongyi Xu² Guoxian Song", You Xie Chenxu Zhang", Xiu Li,, Linjie Luo", Jinli Suo', Yebin Liu
Tsinghua University, 2 ByteDance Inc.
Implisi
Latent Motion
Descriptoe
diffusion
stopss
Figure We present X-NeMo, a diffusion- based portrait animation framework that integrates expressive 1D latent motion descriptors with deentity-disentanglee motion control through cross- attention mechanisms (left). Our method enables meticulous transfer of expressive head poses and detailed facial expressions while maintaining identity consistency, even across subjects with distinct appearances, styles and facial structures (right).
ABSTRACT
We propose X-NeMo, novel zero-shot diffusion-base portrait animation pipeline that animates static portrait using facial movements from driving video of different individual. Our work first identifies the root causes of the key issues in prior approaches, such identity leakage and difficulty in capturing subtle and extreme expressions. To address these challenges, we introduce a self-supervised training framework that distills ID identity-agnostic latent motion descriptor from driving image, effectively controlling motion through cross- attention dur- ing image generation. Our implicit motion descriptor captures expressive facial motion in fine detail, learned from a diverse video dataset without reliance on pretrained motion detectors. We further enhance expressiveness and disentangle motion latents from identity cues by supervising their learning with dual GAN decoder, alongside spatial and color augmentations. By embedding the driving motion into a 1D latent vector and controlling motion via cross-attention rather than additive spatial guidance, our design eliminates the transmission of spatial- aligned structural clues from the driving condition to the diffusion backbone, sub- stantially mitigating identity leakage. Extensive experiments demonstrate that X- NeMo surpasses saate-of-the-ar baselines, producing highly expressive anima- tions with superior identity resemblance. Our code and models will be available for research at our project page
Work done during the internship at ByteDance. Correspnninng author.

---

Published as a conference paper at ICLR 2025
INTRODUCTION
We investigate the task of portrait animation, where a static portrait is animated using head move- ments and facial expressions derived from a driving video of different subject. This task has garnered growing interest owing to its versatile applications in video conferencing, visual effects and digital agents. Building on prior research, we aim advent the field of zero-shot portrait reen- actment by synthesizing highly expressive animations while maintaining identity resemblance to the reference portraits with minimal loss.
Commencing with the pioneering works Siarohin et al. (201 a). portrait animation has primarily involved extracting motion features from a driving video followed with generative process, such as GANs Goodfellow al. (2014): Karras et al. (2019 2020) or diffusion models et al. (2020): [Song] jet al. (2020a Rombach et al. (022), conditioned on the reference appearance and derived motion features. Recent advancements in diffusion models have achieved unprecedented diversity and qual- ity in image generation, prompting us to utilize their generative capabilities (Saharia et al. (2022); A1 (222) for portrait animation. Recent approaches have tackled portrait animation as controlled image-to-video diffusion task, where the reference appearance is cross- gooos--ueried through mutual self- attention Cao et al. (2023) whereas the driving motion signal is integrated into the denoising pro- cess using frameworks like ControlNet /hhang et al. (2023b) or lighter-weight PoseGuider Hu et al.) (2023). The driving motion is represented either through explicit semantic signals such as facial landmarks Ma et al. (2024); Wei et al. (2024); Chang et al. (2024). dense pose Xu et al. (2024d) and facial template renderings Chen al. (2024a), or through implicit motion features learnt from synthetic cross-identity image pairs with aligned expression but different identities Xie et al. (2024); Yang et al.] (2024a). Despite significant progress realism and dynamics, these diffusion- based methods still struggle to capture extreme or subtle expressions and often suffer from identity drift- ing, particularly when the reference and driving identities differ substantially.
We identify two main factors contributing to the challenges in expressiveness and identity resem- blance in prior network designs. First, explicit motion descriptors like facial landmarks or blend- shapes, are often too coarse capture extreme or subtle facial motions and rely heavily on the ro- bustness and accuracy of external motion detectors. Although these descriptors do not contain RGB appearance, they encode the facial structure of the driving identity. leading undesirable identity leakage in cross-identic animations. Recent approaches [iie et al. (2024); Yang et al. (2024a) have attempted to derive implicit motion signals directly from synthetic cross-identity training image pairs generated using an off-the-shelf portrait animator (e.g.. Waan et al. (202T)). Despite substantial im- provements in expressiveness and stability, these methods remain constrained by the capacity of pre- trained portrait animators which struggle with complex expressions (e.g.. tongue protrusion, check puffing). Additionally, sharing aligned facial structures in training pairs inadvertently pass identity information onto the learnt implicit motion features. Second, prior diffusion-basee approaches often guide motion control using ppatially-aligned 2D conditions via ControlNet or PoseGuider. While ef- fective for self-driven motion, this approach encourages the diffusion backbone to take shortcut to mimic the 2D layout rather than fully leveraging semantic mappings between reference and driving images, leading to identity leakage during expression transfer across different subjects.
In this work, we propose X-NeMo, a novel portrait animation framework that enables self-supervised learning of a compact ID latent motion descriptor, facilitating effective motion control in diffusson models via cross attentions. Specifically, we introduce a motion encoder to extract a 1-D identity- agnostic motion latent from the original driving image, and modulate this controlling motion de- scriptor into the diffusion backbone via cross-attentions. By training jointly with the image gener- ator, our encoder fully leverages the motion diversity and richness embedded in our training video collections, without reliance on off-the-shelf motion detectors. We restrict the dimensionality of the latent embedding, functioning as a low-pass filter Burkov et al. (2020), and format it as a ID global motion descriptor that excludes 2D structural cues from the driving image. Furthermore, by using cross motion attentions rather than spatial additive guidance, we ensure that the backbone remains agnostic to the identity structural signals from the motion control branch. This structure-agnostic motion control enables various augmentations like color jittering and spatial transformations, pro- moting the self-supervised disentanglement of identity and motion. In addition to the diffusion loss, we incorporate a dual GAN-based decoder head and refine the learning of our motion latent space with image-level losses that capture subtle and detailed expressions. Our design effectively mitigates the aforementioned shortcut learning, and compels the network to interpret fine- tne-grained motion semantics during both motion encoding and image generation stages.
2

---

Published as a conference paper at ICLR 2025
Trained on collective of public video datasets Zhang al. (2021); Xie et al. (2022); Kirschstein et al. (202)), our method excels at faithfully capturing both extreme and nuanced facial motions and transferring them across subjects even with distinct identity attributes. We extensively evaluate model across our challenging benchmarks and X-NeMo outperforms state-of-the-art portrait anima- tion baselines both quantitatively and qualitatively. Additionally, our expressive latent motion de- scriptor serves as a unified -agnost- embedding, facilitating motion interpolation and video outpainting applications beyond portrait animation. We summarize our contributions follows,
A novel diffusion- based portrait animation pipeline, coupled with latent motion representation, achieving state-of-the-a performance in terms of motion accuracy and identity disentanglement.
A structure-gnostie motion control scheme that learns 1-D ceennttt-diientangle latent motion descriptor and modulates control into image generation via cross attentioss, effectively addressing the long-standing issues of identity entanglement and motion expressiveness loss.
set of carefully designed strategies during both training (e.g.. dual head latent supervision, augmentations and reference feature masking) and inference that substantially enhance the model performance, as supported by extensive ablation studies.
Demonstration of captivating zero-shot portrait animations and generations.
2 RELATED WORKS
GAN-based Portrait Animation. Video-driven face reenactment seeks to accurately transfer fa- cial expressions and head movements from a driving video to target image. Common approaches have primarily leveraged Generative Adversarial Networks (GANs) Goodfellow et al. (2014); Kar- ras et al. 20119 2020) to model and capture the intricate motion dynamics between source and target identities. Broadly, these methods can be categorized into two classes: The first class is based
on explicit motion representations [Siarohin et al.
(2019a b): Ren et al. (2021): I) Wang et al. (202 I);
Mallya et al. (2022); Yin et al. (2022); Gao et al. 2023); Doukas et al. (2021) Guo al. 2024a);
Zhao & Zhang (2022), such as 3D face model parameters, landmarks, or latent keypoints, which use
structured information disentangle appearance and motion, but struggle with large pose changes or dynamic expressions. The second category involves latent motion representations Burkov et al.
(2020) Liang et al. (2022); Zhou et al. (2021): Pang al. (2023); Wang et al. (2022 ;
shev et al. 2022 2024); Xu et al./ (2024c), embedding motion information in a latent space, offering improved expressiveness but relying on complex loss functions and hyperparameters to achieve identity-motion disentanglement. While more effective at transferring subtle expressions, such methods are still limited by the capability of GAN-based generators in handling extreme expres- sions and out-of-domain portrait styles. Our work follows this disentangled representation learning approach, but instead we use a Diffusion Model as the generator, which offers significantly improved generation capabilities with diverse and complex portrait styles.
Diffusion-based Portrait Animation. Diffusion models Ho et al. (0200 Song et al. (2020a b) have demonstrated strong generative capabilities, with Latent Diffusion Models (LDMs) Rombach ct al. (2022) further advancing its efficiency by operating in a dower-dimensional latent space. Re- cent works Liu et al. (2024); Xu et al. (2024a); Han et al. (2023); Varanka et al. (2024); Paskaleva (et al. (2024) have explored adapting pre-trained LDMs Al 2022) for conditional portrait generation, by mapping reference images and driving signals into the text embeddings (e.g.. using CLIP Khan- delwwl et al. (2022)) and injecting them into cross-attention layers. While effective for coarse- level facial expression editing, these methods still struggle with appearance and motion consis- tency in portrait video animation. Hu et al. (2023): Xu et al. 20: Chang et al. (2024) designed for human body animation have shown that the combination of dual U-Net with mutual self- attention Cao et al. (202)) and temporal module Guo et al. (2024b) is able to maintain motion smoothness with consistent appearance. This framework has been extended to portrait animation in several works Tian et al. (2024); Xie et al. (2024); Wei et al. (2024); ( Xu et al.] (2024b): Yang et al. (2024a); Wang et al. 2022)): Ma et al. (2024): Chen et al. (2024b). often using ControlNet Zhang et al. (2023b) or PoseGuider Hu et al. (2023) for motion control. During training, they rely on explicit representations like facial keypoints Ma et al. (2024): : Wei et al. (2024). facial mesh render- ings Chen et al. (2024a), or synthetic cross-identity portraits Xie et al. (2024); Yang et al. (2024a). In contrast, our method learns a latent motion representation jointly with our diffusion backbone, and incorporates motion control with cross-attentions, effectively preventing identity leakage.
3

---

Published as a conference paper at ICLR 2025
Training
Inference
Driving Video (Vo,)
Augmentations
App Enconler
Mation Eneuner
fape
Motion Encoder
frts
Umerd
Emot
Ze
D
Decoder
GAN
X-NeMo
Aggan
Ltdm
Spucial
Temporal
Figure 2: Overview of X-NeMo. We leverage a pretrained diffusion model D as the rendering backbone and incorporate a reference network module R for appearance conditioning, along with temporal modules for cross-frame consistency. For motion control, train a latent motion em- bedding mot encoded from the driving image Tp after applying spatial and color augmentations. Alongside the relative translation and scaling fris of the face bounding box from reference In and driving image DD we integrate the latent motion conditions into the diffusion backbone using newly inserted cross-attention layers. Besides the original diffusion loss Lidm. we supervise the learning of our latent motion embedding with jointly trained GAN decoder head (coupled with an appear- ance latent embedding fapp) using image-level losses Lgan. During inference, we derive the latent motion codes directly from each driving frame, allowing us to synthesize expressive and precise animations while strictly maintain identity resemblance to the reference image.
METHOD
Given single portrait as the reference image IR. objective is to generate head animation sequence {RR=>D,} of length I, conditioned on driving video JD where i = = 1,..., denotes frame index. The generated frames {IR->D,) aim to preserve the identity features and background content depicted in Ia while accurately replicating the head pose and facial expressions featured in each corresponding driving frame TD While portrait animation algorithms generally trained as a frame reconstruction task over video datasets, the J and Tp may feature distinct identities during inference, enabling cross-identity motion transfer.
For our task, we harness the generative capabilities of pre-trained Latent Diffusion Models Al (2022) for image generation. Although our method shares some network modules with prior diffusion- based approaches (Section 3.T), it innovates on motion control by addressing the root causes behind the loss of expressiveness and identity resemblance. We introduce our learning framework that achieves fine-graine fine-grained, Sientity-agnosiee plicit motion descriptor (Section 3.2). To assist the self-supervised learning of motion and identity disentanglement, we present set of carefully designed training strategies (Section 3.3) Figure provides an overview of our training and inference pipeline.
motion control through cross-attention to co-learned im-
3.1 PRELIMINARIES
Latent Diffusion Model. Facilitated by a pretrained auto-encoder, latent diffusion models Rom- bach al. (2022) class of diffusion models al. (2020); Song et al. (2020a b) that synthe- size desired samples in the image latent space, starting from Gaussian noise 27 ~ N(0, 1) and re- fining through 7' denoising steps. During training, latent representations of images progressively
4

---

Published as a conference paper at ICLR 2025
corrupted by Gaussian noise c, following the Denoising Diffusion Probabilistic Model (DDPM) framework Ho et al. (2020). A UNet-based denoising backbone network D containing intervened is trained to learn the reverse denoising process.
layers of convolutions and self-/cross-attentionn
Portrait Animation. Recently line of work Tian et al. (2024); Xie et al. (2024); Wei et al. (2024);
Xu et al. (2024b); Yang et al.] (2024a); Wang et al. (2024); Ma et al. (2024); Chen et al. (002bb) have explored leveraging the generative power of pretrained LDM, such Stable Diffusion Al (2022), for portrait animation. While exhibiting slight algorithmic variations, these methods gener- ally employ similar components to transfer driving motions onto the reference image. Specifically, a reference network R. CCoo et al. 2023), sharing the same architecture with the UNet 'D, extracts reference features of identity appearance and background which are then cross-queried by the UNet self-attention blocks. Motion control is achieved through an additional module, often formatted in ControlNet Zhang et al. (2023b) or weightt PoseGuider Hu et al. (2023). translating driving conditions into 2D apatially-alignes offsets additive to the UNet features. To maintain consistency across animated frames, temporal modules Guo et al. (202)), which incorporates cross-frame atten- tions, are intervened with the spatial transformer blocks.
While effective to some extent, prior methods often fall short in expressiveness and suffer from identity leakage in the generated animations. First, expressiveness is limited by the coarse granu- larity of the driving motion conditions, such as facial landmarks or synthetic training images Xic et al. (2024), which fail to capture complex and subtle expressions like frowning or puckering. Sec- ond, while prior approaches mostly address appearance leakage, they overlook the leakage of 2D facial structure and spatial layout embedded in the driving conditions, whether through landmarks or synthesized images. ControlNet- aligned offsets within the UNet's intermediate features. This reliance spatial alignment causes the UNet to bypass the need to interpret semantic correspondences between the reference and driving faces, resulting in undesirable identity drift during cross-identity animation at inference.
mechanisms transform these motion conditions into spatially-
3.2 PIPELINE WITH
IMPLICIT MOTION CONTROI.
As shown in Figure we follow the existing UNet-based latent diffusion framework AT 2022
integrating both the reference network and temporal modules. However, our key innovation lies in novel motion control module, designed to tackle challenges in motion expressiveness and identity consistency, particularly during cross-identity reenactments. A core design principle of our approach to distill motion directly from the original driving images, while ensuring the image generation backbone operates independently of any appearance or structural clues from the motion control path.
Latent Motion Descriptor. For motion extraction, we employ an image encoder Emot- to learn an implicit latent representation, fmot, that captures facial motions across varying levels of granularity. Similar to the approaches in Wang et al. (2022) 2023), we formulate the motion latent representation Smot as a low-dimensional T-D globul descriptor. The motion encoder Emot consists of intervened feature extraction and self-attention, followed by MLP layers, which encode the motion into a ID latent vector, thereby eliminating spatial positional information (i.c., image structure) along the encoding process. Following the information bottleneck principle Tishby lt al. (2000). we employ a larger network capacity (i.e., the reference net R) and higher feature dimensions (i.e., multi-scale feature maps) for appearance modeling, while using smaller network capacity (Emot) and lower feature dimension ( (mmot) for motion encoding. This design, function- ing as a low-pass bottleneck filter, encourages the emergence of disentangled representations that effectively capture key semantics of facial motion without entangling with appearance information. Furthermore, unlike previous methods that rely on pretrained motion extractor as the driving condi- tions (c.g., facial landmarks). our latent motion representation is optimized jointly with the diffusion generator during training. a result, this allows our model to progressively learn and refine the motion distribution as the diffusion model is trained on more diverse and expressive video data like NerSemble Kirschstein et al. (2023). With that, our approach enhances the expressiveness of the generated animations, as the model adapts to more complex and nuanced facial motions.
layers of convolution-bass
Cross-Attention Control. To exert motion control on UNet using our latent motion descriptor, one possible approach would be to use a ControlNet- like module to guide the denoising process after transforming the ID latent code fmot into 2D spatially-aligner decoder. However, this would contradict our design goal for identity disentanglement. (Figure ((aigureBaa)))) (() Since Smot is intentionally free of 2D structural information, transforming it into spatial control
control map via StyleGAN-like
5

---

Published as a conference paper at ICLR 2025
Refevence
Driving
wie croso-atin
Our
Reference
Driving
wro mngmmmmmittioonn
Ours
(a)
(b)
Figure 3: Qualitative ablation study factors affecting identity consistency. (a) Replacing our mo- tion cross-attentions with control module using spatially additive guidance leads to severe leakage of the driving identity's facial structure. (b) Training without our color and spatial augmentations results in noticeable appearance leakage and identity drift.
map demands additional input regarding the reference identity's 's structure, thereby violating our principle that the motion control path should remain agnostic to identity-specific features. Instead the UNet should resort to the reference net for relevant identity-related information.
Instead, we adopt cross- cross-attention conditioning mechanism, which has proven effective across var- ious control modalities Rombach et al. (2022): Tian et al. (2024); Ruiz et al. (202)) This allows direct injection of the latent motion embedding into the UNet without adding spatial bias. Specifi- cally, we insert motion-attention layers performing cross attention with the latent motion code Smot after each spatial transformer blocks in the backbone. This cross-attention scheme integrates the 1D motion embedding globally into the generation process, encouraging the UNet to interpret the mo- tion condition and establish semantic correspondences between the reference and driving identity.
3.3 TRAINING STRATEGIES
For training, we randomly sample two distinct frames from a video as the reference IR and driving Id image, respectively. The model is then trained to denoise the latent map of the target image Ip at timestep t, with the diffusion loss defined as follows,
Lidm ((,.11 [|le ca( Grefs fmot }
where Eg denotes the trainable parameters in the backbone D. and fmots Cref represent the driving motion and reference features, respectively, extracted by Fmot and R. The training process is structured into three stages. The first stage is the image pretraining stage, where the backhone UNet and reference net are taken into training. The second stage additionally incorporates the motion encoder Emor and the newly integrated motion-attention layers into the optimization, forming an end-to-end encoder-generator structure. Lastly, we train the temporal modules to ensure cross- frame coherence. After the three-stage training process (intended as warm-up), we finetune the entire pipeline end-to-end, with all trainable modules optimized simultaneously.
However, straightforward self-supervised training of the entire framework does not inherently dis- entangle identity from facial motion. The UNet may inadvertently reconstruct the target image by borrowing appearance features from the driving image or encoding identity information into the la- tent motion descriptor fmot. Additionally, when Lg and Ip share similar expressions, the model may distill motion signals from the reference image hindering independent control over facial mo- tion and identity, particularly in cross-identity reenactments. To address these issues, propose several training strategies to fully leverage the potential of our network design in Section[.22
Color/Spatial Augmentation. To suppress identity information leakage from the motion control branch, we reduce the appearance and structural consistency between the driving and target images using both color and spatial augmentations. Specifically, we apply color jittering, random scaling within 30%, and piecewise affine transformations, to the driving image ID. altering the facial ap- pearance and shape while preserving motion semantics. We also perform face-centered cropping to enhance spatial disparity between the driving and target, promoting the motion encoder Emot to focus on the facial movements and capture nuanced expressions. As shown in Figure 3)b), these augmentations effectively guide the motion encoder Fmmot towards learning identity- mo-

---

Published as a conference paper at ICLR 2025
Reference
Driving
wio GAN hrad
Ours
Reference
Driving
wie RFM
Ours
hall
M
ail
im
Figure 4: Qualitative ablation study on factors influencing motion expressiveness.cr (a) Without the dual GAN head, training solely with the diffusion loss hinders the motion encoder's ability to learn detailed and local motion patterns. (b) Our reference feature masking (RFM) strategy facilitates the transfer fine-level facial expressions, such as the wrinkles at the nasal region.
tion representation. To account for the disrupted head translation due face-centered cropping, we construct triplet frts = (Ar/s.. Ay/s,, 8r,Sr, sa/ sr). where (Ar, Ay) denotes the 2D relative distance between the face centers the I and Ip, and sa/s, reflects the change in bounding box scale. This triplet is processed through fully connected layers and fused with latent motion embedding fmot.
Owing to our design of motion control via latent motion embedding and cross-attentions, we sub- stantially improves facial motion disentanglement from identity structure through spatial augmen- tations. contrast, applying these augmentations with a ControlNet-like mechanism, which relies heavily on aligned spatial control signals, would degrade both robustness and accuracy.
Dual-Head Latent Supervision. In our early experiments, observe that motion control training, while effective in capturing coarse facial motions, converges slowly and struggles to depict subtle and fine- graiina motions like frowning and puckering (see Figure 4a)). The latent motion embedding, acting as a low-pass filter, tends to model low-frequency movements first. Additionally, the diffusion loss used during training assigns equal weight every "pixel" in the latent noise, leading the model to prioritize a smooth motion space over capturing local, detailed expressions. To address this, we introduce a dual GAN-based head guide the learning of the latent motion embedding, enhancing the model's ' attention to fine-grained facial expressions.
Following Burkov al. (020);; Wang et al. (2022) 2023), we employ convolutional feature extrac- tor network to encode the reference image into an appearance latent embedding favp. Together with our motion latent code fmot. these embeddings modulate a StyleGAN generator Karras et al.) (2020)
(i.e., the GAN-head decoder) generate RGB image, co-trained with our diffusion- motion control. Its training losses, collectively denoted as Lgans are formulated image space, including weighted L1 reconstruction loss, adversarial loss, feature matching loss Burkov al. (2000), and VGG perceptual losses Simonyan & Zisserman (2014); Cao et al. (2018). Focused on structural variations more than pixel-wise differences, these image-level losses guide the latent space learning with detailed and local motion modes. Furthermore, since the GAN head contains much fewer train- able parameters than the diffusion backbone, it converges faster, boostrapping the motion encoder and aiding the learning of motion attention layers under a well-distributed motion latent embedding.
Reference Feature Masking. In line with our strategies for identity disentanglement in the motion control branch, we also aim to mitigate motion leakage from the appearance reference network. When Ia and Ip exhibit similar expressions, even just partially, the backbone network is likely to utilize the high- nigh-dimensional multi-scale appearance features bypassing the intended reliance our compact 1D latent motion descriptor. While such motion leakage does not impede training data fitting during self-driven training, it hampers the learning of effective and expressive motion control (Figure (b))
"shorteut" for motion reference,
Inspired by Masked Image Modeling He et al. (2022). we introduce reference feature masking to mitigate motion leakage in appearance features. Specifically, we apply 30% uniform random mask- ing to the appearance feature maps from the reference net R. The masked feature maps are flattened and used as reference keys and values for the self-attention layers within the UNet backbone. This balances the strength between appearance and motion signals, ensuring that subtle driving expres- sions are effectively transferred without being overshadowed by the reference expressions.

---

Published as a conference paper at ICLR 2025
Reference
Driving
PD-FGC
LivePormit
FYE
AniPormait
X-Portrait
Ours
Figure 5: Qualitative comparisons. Among all the methods, X-NeMo achieves the most accurate transfer of intricate expressions and emotional subtleties while demonstrating the highest identity resemblance, regardless of the characteristic differences between the reference and driving identities.
4 EXPERIMENT
4.1 IMPLEMENTATION DETAILS
We train our model on a combination of talking head datasets (HDTF-Zhang et al. (2021). Vie et al. )2022) and facial expression dataset (NerSemble Kirschstein et al. (2023), uniformly pro- 512 512 resolution. The training is conducted on Nvidia A100 GPUs using the Adam W optimizer Yao et al. (2021) with learning rate of Le 5. We use a batch size of 64 for appearance and motion control training, and a batch size of 16 for the tempo- ral module using 24-frame video sequences. During inference, implement the prompt traveling technique Tseng et al. (2022) to enhance temporal smoothness in long video generation.
cessed at 25 fps and cropped
For evaluation, compile benchmark of 100 in-the-wild reference portraits DeviantArt (024)); Midjourney (2024): |eexls (2024), representing a broad spectrum of facial structures, appearances and styles. Additionally collect 100 test videos from DFEW Jiang al. (2020) featuring emo- tionally expressive clips, alongside 200 licensed videos showcasing diverse range of emotions, head poses, and facial expressions. Please also refer our supplementary video for more results.
4.2 EVALUATIONS AND COMPARISONS
In our evaluation, we compare our method against state-of-the-arn video-driven portrait animation baselines, including X-Portrait Xie et al. (2024), AniPortrait Wei et al. (2024). Follow-your follow-your-Emoji (FYB) et al. (002)), and Echomimic Chen al. (2024b). We also assess recent non-diffusion- based methods, including LivePortrait Guo et al. (2024a) which uses implicit neural landmarks. EmoPortraits EimoPortraits/Doobyssee et al. 2024), GAN-based expressive portrait animation method, is excluded from our comparisons due to the lack of inference code. For fair comparisons, we finetune AniPortrait, X-Portrait, and PD-FGC using our dataset, while utilizing the released pretrained models for the remaining methods, given the unavailability of their training code. We assess performance in both self and cross reenactments, with all metrics computed resolution of 256 x 256 (the resolution which PD-FGC trained).
et al. (2023) that employs latent motion representation, and
Self Reenactment. For each test video, we utilize the first frame the reference image, generating the entire sequence with subsequent frames acting as both the driving image and the ground truth target. We evaluate the performance by computing the L1, structural (SSIM), and perceptual (LPIPS) image losses to assess both image quality and motion accuracy. Our method, X-NeMo, consistently outperforms all baseline methods, shown in our numerical comparisons (Table
8

---

Published as a conference paper at ICLR 2025
Table 1: Quantitative comparison. Our method achieves superior numerical results than all the
baselines in both self-driven and cross-driven reenactments.
Self-Reenactment
Cross-Reenactment
Method L14
SSIMI
LPIPS,
ID-SIM
AED/APD4
EMO-SIM
FVD
PD-FGC
0.085
0.728
0.291
0.604
0.045/3.95
0.49
441.8
LivePortrait
0.074
0.770
0.236
0.702
0.055/6.61
0.48
279.2
X-Portrait
0.063
0.793
0.209
0.695
0.041/4.07
0.52
237.3
FYE 0.075
0.741
0.249
0.725
0.062/4.49
0.41
360.9
AniPortrait
0.057
0.812
0.198
0.713
0.043/4.14
0.46
290.5
Ours
0.055
0.826
0.168
0.787
0.039/3.42
0.65
152.8
Cross Reenactment. Our method empowers the creation of captivating and expressive anima- tions across diverse portraits even driven by in-the-wild videos with distinct identity features (Fig- ure
Our qualitative comparisons (Figure 5) demonstrate that X-NeMo surpasses all the baselines by a significant margin in identity similarity, expression accuracy and perceptual quality. GAN-based baselines suffer from blurriness and distortions under large head motions and when ap- plied to out-of-domain portraits. For both exaggerated (e.g.. cheek puffing, sticking out the tongue) and subtle facial expressions (e.g.. biting the lip), all other methods struggle to faithfully capture and transfer these facial motion details. Additionally, our method excels in preserving identity resem- blance, regardless of the structural difference between the reference and driving faces, while severe identity drift occurs in other ceffusiio-basse baselines relying on spatially aligned control signals.
4 D
For quantitative assessment, video-level evaluation uses FVD Unterthiner et al. (0188) metrics. To evaluate the image-level quality, we employ three metrics to evaluate identity similarity, expres- sion/head pose accuracy, and emotion consistency respectively. Specifically, utilize the ArcFace score Deng et al. (2019) to measure the cosine similarity of identity features (ID-SIM). Motion accuracy is calculated as the average I. difference between extracted facial blendshapes (AED) and head poses (APD) of the driving and generated images using MediaPipe Lugaresi et al.) (2019). However, since blendshapes provide only coarse motion estimation, we further employ a pretrained emotion encoder, EmoNet Toisoul et al. (202I), to assess emotion accuracy. Specifically, we calcu- late the mean value of concordance correlation coefficients and Pearson correlation coefficients for both valence and arousal to measure the emotion similarity (EMO-SIM) (MM...M... The emotion score reflect model's performance fine-go rreeggained expression control, as emotion recognition highly sensitive to micro-expressions. Our method numerically surpasses all competitors, demonstrating the superior capabilities afforded by our novel motion control design (Table
Applications. Our latent motion descriptor acts as a unified representation for both motion comprehension and generation, supporting tasks beyond portrait animation, including (emotion- conditioned) portrait video outpainting and latent motion interpolation. With our expressive, identity-agnoste motion embedding, we are able to generate long-range expressive videos while consistently preserving the identity across diverse portraits. For more details and visual results, please refer supplemental paper (Section. E and accompanying video.
4.3 ABLATION STUDIES
We ablate individual design choices by removing them from full training pipeline. We validate the function of dual-head supervision motion expressiveness by removing the GAN decoder from co-training ("w/o GAN head"). Even with extended training, the motion encoder Emat struggles with detailed motions in the absence of image-level loss guidance FFig- ure a)). as reflected by a substantial reduction in both expression and emotion metrics (Table 2 We further validate the importance of joint training by pretraining Emot with GAN losses and then freezing it while training the rest the model sorely with the diffusion loss ("w/o joint training"). Both Table 2 and the supplemental video demonstrate that joint training elicits stronger motion
Table 2: Quantitative ablation.
Method w/o GAN head w/o joint training
ID-SIM1
AED/APD
EMO-SIMT
0.789
0.045/4.64
0.43
0.782
0.040/3.49
0.52
w/o RFM w/o augmentation
0.791
0.039/3.41
0.62
0.724
0.042/3.63
0.50
w/o cross-attn
0.697
0.040/3.55
0.48
Ours
0.787
0.039/3.42
0.65

---

Published as a conference paper at ICLR 2025
representation from the encoder, leveraging the diffusion model's superior generative capacity the standalone GAN decoder. Additionally, we assess the role of reference feature masking ("w/o RFM")) in enhancing motion accuracy. Without it, the network shows stronger bias to the reference expressions at certain local regions(Figure440)) 4)), yielding a lower emotion score (Table
We assess the efficacy of augmentation and cross-attention control in motion-identity disentangle- ment. When training without augmentations ("w/o augmentation"), the generations often exhibit noticeable identity leakage from the driving subject in both appearance and face structure (Fig- ure b)),, as confirmed by the drop in identity similarity score (Table 2). We also compare our method baseline where the motion latent transformed into 2D control map via an upsam- pling decoder and applied to the UNet using ControlNet ("w/o cross-attn"). While effective for coarse motion control, its reliance on spatially-alignel additive controls lead to reduced identity re- semblance (Figure underscoring the necessity of our atructure-agnostic motion control design.
We leverage classifier-free guidance (CFG) Ho & Salimans| (2022) to steer inference towards more expressive motion transfer. While straightforward for the conditional generation. we find the optimal practice by using fully masked appearance features and the motion latent fref.mot moo extracted from reference image IR as the negative prompts. As illustrated in Figure 6 this CFG configuration enables the network to better distinguish between conditional appearance and motion features, facilitating more accurate and semantic motion transfer. Our CFG is formulated as
Original
Figure 6: Ablation on different CFG configurations.
Eø(Ze, Cref m (1 w)Ea (Zt, Cref: fmot) wea(Ze, 0, foof
where 3.5 is the CFG scale and és is the final composed noise estimate.
DISCUSSION AND CONCLUSION
We present X-NeMo, novel diffusion-basee portrait animation framework that effectively disen- tangles motion and identity, achieving substantial improvements in generating expressive, identity- preserved animations from diverse portraits. At its core, we introduce self-supervised learning framework that integrates latent motion representations with structure-agnostic motion control through cross-attentions, enhanced by carefully- demonstrate high-quality animation results on a wide range portraits and expressive driving videos, validating the efficacy of our approach. We believe method offers valuable insights into the field and opens avenues for numerous downstream tasks.
training and inference strategies. We
Limitations and Future Work. Our method is trained solely on real human talking and expression videos. Con- sequently, out-of-domain portraits with non-human appearances, such 3D car- toon characters, may cxhibit artifacts like blurred eyes. Additionally, it might strug- gle with exaggerated expressions absent from the training data (see Figure 7 However, our scalable framework, free from reliance on pre-trained motion detectors, enables generalization various styles and motions with more training data. Our motion control represents a general scheme, with
Driving
X-NeMo
Driving
N-NoMo
Figure Failure cases.
which we aim to integrate into video diffusion backbones Yang et al. (2024b) Zheng al. (2024) in the future, for smoother and more dynamic results.
Ethics Statement. Our work aims to improve portrait animation from technical perspective and not intended for malicious use. However, we recognize the potential for misuse like generating fake videos. Therefore, synthesized images and videos should clearly indicate their artificial nature.
ACKNOWL L ED This paper is supported by the NSFC project No.62125107.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Stability AI. Stable diffusion v1.5 model card. http://wwgiiigiecccco/////eaaaaa vI-5, 2022.
Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models large datasets. arXiv preprint arXi::231.15117,, 2023.
Adrian Bulat and Georgios Tzimiropoulos. How far we from solving the 2d & 3d face alignment problem?(and) a dataset of 230,000 3d facial landmarks). In Proceedings of the IEEE international conference on computer vision, pp. 1021-1030, 2017.
Egor Burkov, Igor Pasechnik, Artur Grigorev, and Victor Lempitsky. Neural head reenactment with latent pose descriptors. In CVPR, 2020.
Mingdeng Cao, Xintao Wang, Zhongang Qi, Ying Shan, Xiaohu Qie, and Yinqiang Zheng. Mas- actrl: Tuning- free mutual self-attention control for consistent image synthesis and editing. arXiv preprint ar arXiv:2304. 08465, 2023.
Qiong Cao, Li Shen, Weidi Xie, Omkar M. Parkhi, and Andrew Zisserman. VGGFace2: A dataset for recogmising faces across pose and age. In International Conference on Automatic Face and Gesture Recognition, 2018.
Di Chang, Yichun Shi, Quankai Gao, Hongyi Xu, Jessica Fu, Guoxian Song, Qing Yan. Yizhe Zhu, Xiao Yang, and Mohammad Soleymani. Magicpose: Realistic human poses and facial expressions retargeting with identity-aware diffusion. In ICML, 2024.
Ken Chen, Sachith Seneviratne, Wei Wang. Dongting Hu, Sanjay Saha, Md Tarek Hasan, Sanka Rasnayaka, Tamasha Malepathirana, Mingming Gong, and Saman Halgamuge. Anifacediff: High-fidelity face reenactment via facial parametric conditioned diffusion models. arXiv preprint arXiv:2406. 13272, 2024a.
Zhiyuan Chen, Jiajiong Cao, Zhiquan Chen, Yuming Li, and Chenguang Ma. Echomimic: Life- like audio-driven portrait animations through editable landmark conditions. arXiv preprint arXiv:2407.081166, 2024b.
Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafciriou. Areface: Additive angular margin loss for deep face recognition. In CVPR, pp. 4690-4699, 2019.
DeviantArt. deviantart. htnp://wwwwddevaatttr..cm 2024.
Michail Christos Doukas, Stefanos Zafeiriou, and Viktoriia Sharmanska. Headgan: One-shot neural head synthesis and editing. In ICCV, 2021.
Nikita Drobyshev, Jenya Chelishev, Taras Khakhulin, Alcksei Ivakhnenko, Victor Lempitsky, and Egor Zakharov. Megaportraits: One-shot megapixel neural head avatars. In ACM MM, 2022.
Nikita Drobyshev, Antoni Bigata Casademunt, Konstantinos Vougioukas, Zoe Landgraf, Stavros Petridis, and Maja Pantic. Emoportraits: Emmotion-ehhaneed multimodal one-shot head avatars. In CVPR, 2024.
Patrick Esser, Robin Rombach, and Bjom Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recogni- fion, pp. 12873-1222 12883, 2021.
Yue Gao, Yuan Zhou, Jinglu Wang, Xiao Li, Xiang Ming, and Yan Lu. High-fidelity and freely controllable talking head video generation. In CVPR, 2023.
Ian Goodfellow, Jean Pouget-Abadie, Mchdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In NeurlPS, 2014.
Jianzhu Guo, Dingyun Zhang, Xiaoqiang Liu, Zhizhou Zhong, Yuan Zhang, Pengfei Wan, and Di Zhang. Liveportrait: Efficient portrait animation with stitching and retargeting control. arXiv preprint N0M1:2407.031660 2024a.
11

---

Published as a conference paper at ICLR 2025
Yuwei Guo, Ceyuan Yang, Anyi Rao, Yaohui Wang. Yu Qiao, Dahua Lin, and Bo Dai. Animatediff: Animate your personalized text-to-image diffusion models without specifie tuning. arXiv preprint arXiv:2307.0477 04725, 2023.
Yuwei Guo, Ceyuan Yang. Anyi Rao, Zhengyang Liang, Yaohui Wang, Qiao, Maneesh Agrawala, Dahua Lin, and Bo Dai. Animatediff: Animate your personalized text-to-image diffu- sion models without specific tuning. In ICLR, 2024b.
Yue Han, Jiangning Zhang, Junwei Zhu, Xiangtai Li, Yanhao Ge, Wei Li, Chengjie Wang, Yong Liu, Xiaoming Liu, and Ying Tai. A generalist facex via learning unified facial representation. arXiv preprint arXiv:2401.0055// 2023.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In CVPR, pp. 16000-16009, 2022.
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance.
arXiv preprint
arXiv:2207.12598, 2022.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. NeurlPS, 33: 1840-6851, 2020.
Li Hu, Xin Gao, Peng Zhang, Ke Sun, Bang Zhang, and Liefeng Bo. Animate anyone: Consistent and controllable image-to-video synthesis for character animation. arXiv preprint arXiv:2311.77117, 2023.
Xingxun Jiang, Yuan Zong, Wenming Zheng. Chuangao Tang, Wanchuang Xia, Cheng Lu, and Jiateng Liu. Dfew: A large-scale database for recognizing dynamic facial expressions in the wild. In Proceedings of the 28th ACM international conference on multimedia, pp. 2881-2888 2020.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In CVPR, 2019.
Tero Karras, Samuli Laine, Mila Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila, Analyz- ing and improving the image quality of StyleGAN. In CVPR, 2020.
Apoorv Khandelwal, Luca Weihs, Roozbeh Mottaghi, and Aniruddha Kembhavi. Simple but effee- tive: Clip embeddings for embodied ai. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18829-1488,, 2022.
Tobias Kirschstein, Shenhan Qian, Simon Giebenhain, Tim Walter, and Matthias NieBner. Nersem- ble: Multi-view radiance field reconstruction of human heads. ACM Transactions on Graphics, 2023.
Borong Liang, Yan Pan, Zhizhi Guo, Hang Zhou, Zhibin Hong, Xiaoguang Han, Junyu Han, Jingtuo Liu, Errui Ding, and Jingdong Wang. Expressive talking head generation with granular audio- visual control. In CVPR, 2022.
Renshuai Liu, Bowen Ma, Wei Zhang, Zhipeng Hu, Changjie Fan, Tangjie Ly, Yu Ding, and Xuan Cheng. Towards a simultancous and granular identity-expressin control in personalized face generation. CVPR, 2024.
Camillo Lugaresi, Jiuqiang Tang, Hadon Nash, Chris McClanahan, Esha Uboweja, Michael Hays, Fan Zhang, Chuo-Ling Chang, Ming Guang Yong, Juhyun Lee, al. Mediapipe: A framework for building perception pipelines. arXiv preprint 1906 08200, 2019.
Yue Ma, Hongyu Liu, Hongfa Wang, Heng Pan, Yingqing He, Junkun Yuan, Ailing Zeng, Chengfei Cai, Heung- Yeung Shum, Wei Liu, et al. follow-your-eman Fine-controllable and expressive freestyle portrait animation. arXiv preprint arXiv:2406.01990 2024.
Arun Mallya, Ting-Chun Chun Wang, and Ming- Yu Liu. Implicit warping for animation with image sets. NewriPS, 2022.
Midjoumey. midjoumey. https//www..mljouneeeccoom 2024.
12

---

Published as a conference paper at ICLR 2025
Youxin Pang, Yong Zhang, Weize Quan, Yanbo Fan, Xiaodong Cun, Ying Shan, and Dong-ming Yan. Dpe: Disentanglement pose and expression for general video portrait editing. In CVPR, 2023.
Reni Paskaleva, Mykyta Holubakha, Andela Ilie, Saman Motamed, Luc Van Gool, and Danda Paudel. A unified and interpretable emotion representation and expression generation. In CVPR, 2024.
Pexels. pexels. https://wwwpexxls..om/// 2024.
Yurui Ren, Ge Li, Yuanqi Chen, Thomas H Li, and Shan Liu. Pirenderer: Controllable portrait image generation via semantic neural rendering. In ICCV, 2021.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In CVPR, pp. 10684-10695, 2022.
Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-lo-image diffusion models for subject-driven generation. In Pro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 22500- 22510, 2023.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. NeurIPS, 35:36479-36494, 2022.
Aliaksandr Siarohin, Stéphane Lathuilière, Sergey Tulyakov, Elisa Ricci, and Nicu Sebe. Animating arbitrary objects via deep motion transfer. In CVPR, 2019a.
Aliaksandr Siarohin, Stéphane Lathuilière, Sergey Tulyakov, Elisa Ricci, and Nicu Sebe. First order motion model for image animation. NeurlPS, 2019b.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint 1rXivv1409.15555 2014.
Jiaming Song. Chenlin Meng. and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020a.
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456. 2020b.
Linrui Tian, Wang. Bang Zhang, and Liefeng Bo. Emo:
Emote portrait alive-generating expres- sive portrait videos with audio2video diffusion model under weak conditions. In ECCV, 2024.
Naftali Tishby, Fernando C Pereira, and William Bialek. The information bottleneck method. arXiv preprint physicx/0004057, 2000.
Antoine Toisoul, Jean Kossaifi, Adrian Bulat, Georgios Tzimiropoulos, and Maja Pantic. Es- timation of continuous valence and arousal levels from faces naturalistic conditions. Nature Machine Intelligence, 2021. URL https: hots://wwwr /www nature wwt./rticcaeeeeeeeeeooeeeoo 842256-020- 022-5-6-22-00000-000
Jonathan Tseng, Rodrigo Castellon, and C. Karen Liu. Edge: Editable dance generation from music, 2022.
Thomas Unterthiner, Sjoerd Van Steenkiste, Karol Kurach, Raphael Marinier, Marcin Michalski. and Sylvain Gelly. Towards accurate generative models of video: A metric & challenges. arXiv preprint arXiv:1812.01717, 2018.
Tuomas Varanka, Huai Huai-Qian Khor, Yante Li, Mengting Wei, Hanwei Kung, Nicu Sebe, and Guoying Zhao. Towards localized fine- ine-grained control for facial expression generation. arXiv preprint arXiv:2407.20177 2024.
13

---

Published as a conference paper at ICLR 2025
Cong Wang, Kuan Tian, Jun Zhang, Yonghang Guan, Feng Luo, Fei Shen, Zhiwei Jiang, Qing Gu, Xiao Han, and Wei Yang. V-express: Conditional dropout for progressive training of portrait video generation. arXiv preprint 2400:2440602211,, 2024.
Duomin Wang, Yu Deng, Zixin Yin, Heung- Yeung Shum, and Baoyuan Wang. Progressive dis- entangled representation learning for fine-grained controllable talking head synthesis. In CVPR, 2023.
Kaisiyuan Wang, Qianyi Wu, Linsen Song, Zhuoqian Yang. Wayne Wu, Chen Qian, Ran He, Yu Qiao, and Chen Change Loy. Mead: A large-scale audio-visual dataset for emotional talking- face generation. In European Conference on Computer Vision, pp. 700-717. Springer, 2020.
Ting-Chun Wang, Arun Mallya, and Ming- Yu Liu. One-shot free-view neural talking-head synthesis for video conferencing. In CVPR, 2021.
Yaohui Wang, Di Yang, Francois Bremond, and Antitza Dantcheva. Latent image animator: Learn- ing to animate images via latent space navigation. ICLR, 2022.
Huawei Wei, Zejun Yang, and Zhisheng Wang. Aniportrait: Audio-driven synthesis of photorealistic portrait animation. arXiv preprint XxXv::2403.16694 2024.
Lianginn Xie, Xintao Wang. Honglun Zhang, Chao Dong. and Ying Shan. Vfhq: high-quality dataset and benchmark for video face super-resolution. In The IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2022.
You Xie, Hongyi Xu, Guoxian Song, Chao Wang, Yichun Shi, and Linjie Luo. X-portrait: Expres- sive portrait animation with hierarchical motion attention. SIGGRAPH, 2024.
Chao Xu, Yang Liu, Jiazheng Xing, Weida Wang, Mingze Sun, Jun Dan, Tianxin Huang, Siyuan Li, Zhi-Qi Cheng, Ying Tai, et al. Facechain imagineed:: Freely crafting high-fidelin diverse talking faces from disentangled audio. In CVPR, 2024a.
Mingwang Xu, Hui Li, Qingkun Su, Hanlin Shang. Liwei Zhang. Ce Liu, Jingdong Wang, Luc Van Gool, Yao Yao, and Siyu Zhu. Hallo: Hierarchical audio-driven visual synthesis for portrait image animation. arXiv preprint orXiv:24060 200.00040 2024b.
Sicheng Xu, Guojun Chen, Yu-Xiao Guo, Jiaolong Yang, Chong Li, Zhenyu Zang, Yizhong Zhang, Xin Tong, and Baining Guo. Vasa-1: Lifelike audio-driven talking faces generated in real time. arXiv preprint arXi::2404.1066
2024c.
Zhongcong Xu, Jianfeng Zhang. Jun Hao Liew, Hanshu Yan, Jia-Wei Liu, Chenxu Zhang, Jiashi Feng, and Mike Zheng Shou. Magicanimate: Temporally consistent human image animation using diffusion model. In CVPR, 2024d.
Shurong Yang, Huadong Li, Juhao Wu, Minhao Jing, Linze Li, Renhe Ji, Jiajun Liang, and Haoqiang Fan. Megactor: Harness the power of raw video for vivid portrait animation. arXiv preprint arXiv:2405.20881 2024a.
Zhuoyi Yang. Jiayan Teng. Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiaohan Zhang, Guanyu Feng, et al. Cogvideox: Text-to-video diffusion models with expert transformer. arXiv preprint
240.:2200000072 2024b.
Zhewei Yao, Amir Gholami, Sheng Shen, Mustafa Mustafa, Kurt Keutzer, and Michael Mahoney. Adahessian: An adaptive second order optimizer for machine learning. In proceedings of the AAAI conference on artificial intelligence, volume 35, pp. 10665-10673, 2021.
Fei Yin, Yong Zhang, Xiaodong Cun, Mingdeng Cao, Yanbo Fan, Xuan Wang, Qingyan Bai, Baoyuan Wu, Jue Wang, and Yujiu Yang. Stylcheat: One-shot high-resolution editable talking face generation via pre-trained stylegan. In ECCV, 2022.
Jianrong Zhang, Yangsong Zhang. Xiaodong Cun. Shaoli Huang, Yong Zhang, Hongwei Zhao, Hongtao Lu, and Xi Shen. T2m-gpt: Generating human motion from textual descriptions with discrete representations. In CVPR, 2023a.
14

---

Published as a conference paper at ICLR 2025
Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control text-to-image diffusion models. ICCV, pp. 1336-3847, 2023b.
Zhimeng Zhang, Lincheng Li, Yu Ding, and Changjie Fan. Flow-guided one-shot talking face generation with high-resolution audio-visual dataset. In CVPR, pp. 3661-3670, 2021.
Jian Zhao and Hui Zhang. Thin-plate spline motion model for image animation. In CVPR, 2022.
Zangwei Zheng, Xiangyu Peng. Tianji Yang, Chenhui Shen, Shenggui Li, Hongxin Liu, Yukun Zhou, Tianyi Li, and Yang You. Open-sora: Democratizing efficient video production for all, March 2024.
Hang Zhou, Yasheng Sun, Wayne Wu, Chen Change Loy. Xiaogang Wang, and Ziwei Liu. Pose- controllable talking face generation by implicitly modularized audio-visual representation. In CVPR, 2021.
A TRAINING AND INFERENCE DETAILS
GAN Head Training Losses. Following Burkov et al. 2020)) train our dual GAN decoder a self-supervised manner to reconstruct In using a combination of losses. Specifically, a L reconstruction loss is employed to minimize pixel-wise L1 distance:
Lreeon I - TR-xDll
Additionally, two perceptual losses, LvGg and LyGGFaces are applied based on L matching of ConvNet activations from a VGG-19 model Simonyan & Zisserman (2014) pretrained for ImageNet classification and a VGGFace model Cao al. (2018) trained for face recognition:
Logg
(4)
N
Lugel
||VGG ((I))
VGG (IR+D)]lI.
(5)
where denotes the number of feature layers in each respective pre-trained VGG model. An adver- sarial generative loss Teado is applied using co-trained discriminator D, while feature matching loss Ii is calculated as the T. distance between discriminator feature maps at different layers:
N
Lim Σ (DD(I D'(RRDD)
(6)
The overall learning objective for the GAN head is then formulated as:
Lgan = Ladv X, Arecoon + Augg fogg + Augos Eivet + A fm
where =1.0, 33--2, A0gg/=6e-3 and A1m=00.0
Inference Performance. During inference, we use 25 DDIM steps Song et al. (2020a) with classifier-free guidance (CFG) scale of 3.5. For generating a I-second video at 25 frames per second, the process takes approximately 20 seconds and requires 24 GB of memory.
Encoder Architecture. Emot takes the classical feature alignment network Bulat & Tzimiropou- los (2017) the backbone. with additional attention layer added at both its input and output to enhance feature extraction capabilities. Finally, it outputs 1D vector through two MLP layers. The appearance encoder for extracting Sapp is implemented ResNet50.
15

---

Published as a conference paper at ICLR 2025
B VIDEO-LEVEL PERFORMANCE ANALYSIS
As demonstrated in our comprehensive quantitative (Table D and qualitative evaluations(Figure and video demos), X-NeMo consistently outperforms all baseline methods in motion expressive- ness and temporal coherence. This strong video-level performance stems primarily from our robust single-frame reenactment capability. Specifically, our motion encoder captures fine-scale facial mo- tion within individual frames, while the temporal motion module focuses on cross-frame consis- tency, supported by the SVD-VAE decoder Blattmann et al. (2023) and prompt traveling Cao et al. 2023) during inference.
Unlike prior methods, which rely on pre-trained motion control signals such as explicit landmarks (AniPortrait Wei al. (2024) or implicit landmarks (X-Portrait Xie et al. (2024)), approach utilizes a learnable motion representation that is optimized jointly with the diffusion generator. Pre- trained motion signals are often unstable and prone to jitter across frames, and they fail to fully capture the complex facial expressions present in diverse datasets. By jointly optimizing motion representation with the generator, X-NeMo inherently achieves better expressiveness and more ef- fective disentanglement of motion and identity.
However, straightforward self-supervised training does naturally disentangle identity from facial motion. As shown in Table critical components like Dual Head Supervision, Cross-Attention, and Augmentations are essential for achieving effective disentanglement. Furthermore, Joint Training and RFM enhance motion extraction, capturing subtle micro- excro-expressions (not captured by AKD but reflected the EMO-SIM metric) and enabling vivid and expressive reeactment results.
C MORE ABLATIONS
We provide additional visual ablations on some network and training hyperparameters.
Reference
Driving
latent size
latent size 512
latent size 1024
Figure 8: We ablate the effect of different sizes of latent motion embedding in capturing fine-grained intricate expressions.
Motion Latent Embedding Size, We evaluate the dimensionality of our motion latent embedding by comparing 128, 512, and 1024-dimensional latent codes for fmot. In typical driving scenarios, all three configurations perform similarly in replicating facial expressions with minimal differences. However, as shown in Figure g reducing the embedding size to 128 diminishes the ability to capture subtle, intricate expressions, while increasing it to 1024 provides negligible improvements. Thus, we select the 51 312-dimesio embedding as it balances compactness with motion expressiveness.
Reference Feature Masking Ratio. We assess the effectiveness of our reference feature masking
strategy across different masking ratios, ranging from 0%. 30%, 75%, to 95%. As shown in Figure this strategy enhances the transfer of detailed facial expressions: however, excessively high masking ratios impede the model's ability to capture fine motion details and maintain identity consistency.
16

---

Published as a conference paper at ICLR 2025
Reference
Driving
75%
Figure 9: Qualitative comparison of different ratios of reference feature masking indicates that 30%
achieves the most accurate capture of driving facial motions.
Table 3: More quantitative ablations.
Method
ID-SIM
AED/APD
EMO-SIM
FVD
Ours
0.787
0.039/3.42
0.6531
152.8
Temporal motion encoder
0.786
0.039/3.42
0.6526
151.5
Simultaneous training
0.774
0.039/3.45
0.6398
155.2
This is likely because when the reference image's appearance is too heavily obscured during training, the motion encoder compensates by encoding appearance information, reducing the capacity of the motion embedding for expressing dynamic movements. In practice, we found that masking ratios between 20% and 50% achieve optimal results, with 30% used in implementation.
Single-Frame Motion Extraction We conducted an experiment where latent motion sequences were extracted using multiple driving frames instead of a single image. Specifically, we imple- mented pipeline with temporal motion encoder trained end-to-end using 16-frame video clips. As shown in Tab. 3 the differences across all metrics are minimal. However, training with video clips significantly reduces training efficiency, as the batch size for multi-frant frame training is reduced by factor of 16 compared to single-frame training. Consequently, we opted for the more efficient single-frane frame pipeline.
Regarding why temporal encoding yields negligible gains for portrait animation, we believe this is due the structured nature of facial video data and the capability our motion encoder, which learns directly and implicitly from smooth RGB image signals. Facial videos exhibit strong struc- tural consistency and clear motion patterns, enabling fine-grained motion extraction on a frame-by- frame basis. This ensures temporal continuity without requiring multi-frame inputs. Additionally, in our self-driven training setup, the diffusion generation network accurately reconstructs the target image using single driving frame, often disregarding motion cues from neighboring frames.
Simultaneous Training As mentioned in Sec. 3.3 before the end-to-end fine-tuning, our method conducts a three-stage training process. To ablate this, we implemented baseline that directly trains the whole network simultaneously. As shown in Tab. the differences across all metrics are not remarkable. Nonetheless, due to the high GPU cost of simultaneous training and small batch size, it's much slower for this baseline to converge. Hence in practice, adopt the stepwise training strategy for training efficiency.
D MORE RESULTS
Please refer supplemental video for expressive demo cases.
17

---

Published as a conference paper at ICLR 2025
E APPLICATIONS
Our latent motion embedding, trained jointly with the diffusion backbone, offers a compact, identity- agnostic, yet expressive representation for diverse range of facial motions. Beyond the primary portrait animation task, we showcase its broader applications as a unified motion representation, enabling seamless motion interpolation, video outpainting and conditioned generation.
Latent Motion Interpolation
Intent Mation Interpolation
Driving
Reference
Figure 10: Latent motion interpolation. We derive latent motion codes from a few driving keyframes (top) and apply onto diverse portraits with linearly interpolated motion embeddings (bottom).
Latent Motion Interpolation. Owing to our smooth and identity-agnostie latent motion space, we able extract keyframe expressions from different videos, linearly interpolate the latent embed- dings and apply them across diverse portraits, as showcased in Figure. TO This interpolation yields smooth and natural expression transitions, maintaining motion coherence across different portraits and appearance consistency with the reference images. These results underscore the robustness and identity disentanglement of our motion latent embedding.
Portrait Video Outpainting and Generation. By leveraging motion latent embedding unified representation for motion comprehension and generation, we showcase its application in video outpainting. Specifically, we adopt the approach from T2M-GPT Zhang et al. (202aa) to to- kenize temporal latent motions by training a Dector-Quantize VAE model Esser et al. (2021) with learnable codebook (4096 entries of 8-dimension code) that downsamples the temporal dimen- sion by a factor of 4. This allows us to represent frames of motion with T/4 discrete motion tokens, where T' is the training sequence length (we T 128), facilitating the use of GPT-like frameworks for Long-sequence motion generation. In Figure. [2] we train a GPT2-small network that extends preceding motions derived from driving video with extrapolated motions. The results show natural and expressive generated sequences, thanks to the strong representation power of our latent motion embedding. Moreover, as more facial video datasets containing multimodal annota- tions (e.g., text and audio) become available, our method can seamlessly extend to multimodal facial video generation within a unified framework. As an example, we illustrate emotion-condiiioned por- trait video generation in Figure II trained with MEAD dataset Wang et al. (2020).
18

---

Published as a conference paper at ICLR 2025
Canditio:: "Angry'
Condition: "Heppy'
Figure 11: Emotion-conditioned portrait video generation.
Doviin
Motion Outpuining
Reforence
Driving
Motion Outpointing
Reforence
Figure 12: Portrait video outpainting. Starting from a sequence of driving motion, our model is capable of extrapolating into long video sequence with consistent identity attributes.
19


Reviews:
[{"review_id": "qEDaBgOqM9", "reviewer": "Reviewer_jJFM", "summary": "This paper proposes a novel portrait animation framework that extracts identity-free motion through a specially designed module and injects the motion using cross-attention, while utilizing GAN to enhance the accuracy of motion capturing. Extensive experiments demonstrate the effectiveness of this approach.", "questions": "1.For non-human data, existing methods find it challenging to crop the face. If you were to apply these methods to such data, what kind of solution would you propose?\n2.Could you simply train a model with added OOD data to provide results for non-human cases?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.This paper is well written, easy to follow.\n2.This paper proposes a new portrait animation pipeline that effectively addresses the longstanding issues of identity entanglement and loss of motion expressiveness.\n3.Extensive experiments demonstrate the effectiveness of this method.\n4.Great work! The motivation and experimental results for each component are solid. The demo in the supplementary materials also looks very impressive; (if it isn’t cherry-picked)", "weaknesses": "1.Since the motion model is trained, could it struggle to adapt to out-of-distribution (OOD) motions, could you provide extreme or unusual facial expressions to demonstrate robustness?\n2.In the results provided in the paper and the demo, the facial features of the driving and reference are quite similar. Could you provide more examples where facial features (such as eyes, mouth, nose, etc.) or face position or head pose are inconsistent?\n3.As stated in W2, I also cannot tell if this paper truly addresses the issue of identity leakage, as it appears that most of the features in the driving and reference images are quite similar, could you provide more convincing prof or experiments?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xon6tUhPKk", "reviewer": "Reviewer_82UL", "summary": "This work presents a portrait animation method that diverges from traditional landmark-based approaches. It leverages a latent motion descriptor enhanced by a low-pass filter and incorporates motion priors through cross-attention, eliminating the reliance on aligned pose information. To develop a robust and fine-grained motion descriptor, the method includes a GAN head and employs techniques such as data augmentation and masked modeling.", "questions": "1. In the quantitative comparisons, do the results for AniPortrait and X-Portrait come from the officially released weights or from weights that were retrained on your training dataset? If the latter, why do the results for both methods perform poorly in cases involving tongue motion, considering that the NerSemble dataset should include training samples with tongue motion? This is particularly concerning for X-Portrait, which is also a non-landmark method.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This work proposes a feasible solution to address the limitations of previous portrait animation methods that rely on explicit motion descriptors or the integration of motion information through PoseGuider and ControlNet.\n2. This study demonstrates strong visual performance across various samples, showcasing its robust capabilities in motion transfer and stability.\n3. This work includes comprehensive comparisons with prior methods and an ablation study to validate the proposed techniques.", "weaknesses": "1. A temporal evaluation of spatially aligned motion injection versus attention-based motion injection is recommended. Intuitively, spatially aligned motion injection is expected to provide better temporal consistency due to its stronger spatial priors.\n2. Additional analysis and experiments are needed to clarify why X-NeMo achieves such high levels of temporal consistency. Other methods, such as LivePortrait and X-Portrait, also include a stage for training temporal modules, yet they still exhibit some flickering in their results. The paper mentions only temporal modules and prompt traveling as means to achieve temporal consistency, but I remain unclear about the source of X-NeMo's superior performance in this regard.\n\nIf these concerns are adequately addressed, I will consider raising my score to \"accept.\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jG9ZZYMROm", "reviewer": "Reviewer_qbkt", "summary": "The paper proposes X-NeMo, a diffusion-based portrait animation framework. It tries to address identity leakage and capturing diverse expressions. The method involves extracting a 1D identity-agnostic latent motion descriptor from the driving image, using cross-attention for motion control in image generation. It is trained end-to-end with a dual GAN decoder and spatial/color augmentations.", "questions": "Please see the weakness.", "limitations": "", "rating": 3, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "+ The paper is well-structured.\n+ The problem of portrait animation with high expressiveness and identity preservation is important.\n+ The use of a 1D latent motion descriptor and cross-attention for motion control is reasonable.", "weaknesses": "- What is the definition of zero-shot here? Firstly the model is trained. Secondly in the inference several reference images are provided. Thirdly the description of zero-shot is missing.\n- The method has three training stages. What does it mean by end-to-end learning as described in several places in the paper? It seems each components are trained separately.\n- The approach to get identity-agnostic feature is only to augment the images with color jitter, scaling and affine transformation. Such augmentations do not remove the identity information but only change the basic appearance of the facial images. The reviewer also believes such simple augmentations are not innovative.\n- The app encode in Fig 2 seems to be not described.\n- While the method shows plausible performance on the tested datasets, the generated video seem not to be tested with widely-used metrics like FVD.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XEY0fowcBP", "reviewer": "Reviewer_tmTm", "summary": "The paper introduces X-NeMo, a novel zero-shot, diffusion-based portrait animation framework that animates static portraits using head movements and facial expressions from a driving video of another subject. The authors identify key challenges in existing approaches, such as identity leakage and difficulty in capturing subtle and extreme facial expressions. To overcome these issues, X-NeMo employs a fully end-to-end training process that extracts a 1D identity-agnostic motion descriptor from the driving image, controlling the motion in the generated animation through cross-attention rather than traditional spatial guidance. This technique mitigates the transmission of identity clues from the driving video, reducing identity leakage and improving expressiveness. X-NeMo learns facial motion from diverse video datasets without relying on pre-trained motion detectors, using a dual GAN-based decoder and various augmentations to disentangle motion from identity cues. By embedding motion in a 1D latent vector and leveraging cross-attention, the model avoids transferring spatial structures, thereby preserving identity resemblance in the animated portraits. Experiments demonstrate that X-NeMo surpasses state-of-the-art methods, producing highly expressive animations while maintaining the subject’s identity.", "questions": "None", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1.\tThe design of the implicit 1D latent motion descriptor and its integration through cross-attention offers a perspective on addressing identity leakage and expressiveness, overcoming the shortcomings of explicit motion descriptors.\n2.\tX-NeMO better captures subtle and extreme expressions in the process of portrait animation compared to previous models.", "weaknesses": "1.\tX-NeMo addresses the portrait animation task in the image level, and derive latent motion codes from each driving frame without the perception of the whole driving video. This setting neglect the video continuity, and may not capture the subtle coherence of expressions in a driving video.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)