# LIA-X: Interpretable Latent Portrait Animator

Yaohui Wang, Di Yang, Xinyuan Chen, Francois Bremond, Yu Qiao, Antitza

Dantcheva

<figure><img src="image_1.png" /></figure>

## To cite this version:

Yaohui Wang, Di Yang, Xinyuan Chen, Francois Bremond, Yu Qiao, et al., LIA-X: Interpretable Latent Portrait Animator. 2026. hal-05492556

HAL Id: hal-05492556

https://inria.hal.science/hal-05492556v1

Preprint submitted on 4 Feb 2026

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
# LIA-X: Interpretable Latent Portrait Animator

Yaohui Wang¹ Di Yang² Xinyuan Chen¹ François Brémond² Yu Qiao¹* Antitza Dantcheva²

¹Shanghai Artificial Intelligence Laboratory ²Inria, Université Côte d'Azur

https://wyhsirius.github.io/LIA-X-project/

<figure><img src="image_3.png" /><figcaption>Figure 1. **Portrait Animation.** We show results of three portraits animated by LIA-X using the video of Gal Gadot (top right). Given a source portrait (1st column), LIA-X allows for editing the source portrait based on the learned semantics to align it with the initial driving frame in terms of head pose and facial expression (2nd column). The animated sequences (2nd-5th columns) are then obtained by applying motion transfer process on the edited portraits.</figcaption></figure>

## Abstract

We introduce LIA-X, a novel interpretable portrait animator designed to transfer facial dynamics from a driving video to a source portrait with fine-grained control. LIA-X is an autoencoder that models motion transfer as a linear navigation

of motion codes in latent space. Crucially, it incorporates a novel Sparse Motion Dictionary that enables the model to disentangle facial dynamics into interpretable factors. Deviating from previous 'warp-render' approaches, the interpretability of the Sparse Motion Dictionary allows LIA-X to support a highly controllable 'edit-warp-render' strategy, enabling precise manipulation of fine-grained facial semantics in the source portrait. This helps to narrow

<footnote>*Corresponding author.</footnote>
initial differences with the driving video in terms of pose and expression. Moreover, we demonstrate the scalability of LIA-X by successfully training a large-scale model with approximately 1 billion parameters on extensive datasets. Experimental results show that our proposed method outperforms previous approaches in both self-reenactment and cross-reenactment tasks across several benchmarks. Additionally, the interpretable and controllable nature of LIA-X supports practical applications such as fine-grained, user-guided image and video editing, as well as 3D-aware portrait video manipulation.

# 1. Introduction

With the remarkable development of deep generative models [10, 13, 32], techniques for video generation have been largely advanced. Portrait animation, a domain-specific video generation task that aims to transfer facial dynamics from a driving video to a portrait image, has also received increasing attention due to its wide applications in entertainment, e-education, and digital human creation.

Towards accurately transferring facial dynamics, one straightforward strategy is to leverage pre-computed explicit representations such as facial landmarks [5, 21, 39], 3DMM [6, 19], optical flows [17, 23] and dense poses [50] as motion guidance. Such an approach has been widely used in both GAN-based and recent diffusion-based methods [5, 21]. However, the generated quality of these methods relies heavily on the performance of the off-the-shelf feature extractors, which strongly restricts their usability in more challenging, real-world scenarios.

Self-supervised learning-based techniques have also been proposed to tackle this problem. Previous methods explored learning either explicit structures such as 2D/3D keypoints [11, 28, 29, 40, 58] or implicit motion codes [45, 48] in an end-to-end manner to model facial dynamics. They typically follow a 'warp-render' strategy to conduct portrait animation based on the computed optical-flow fields. While these methods have achieved promising results in both motion transferring and identity preservation, their performance still drops significantly when large variations exist between the source and driving data in terms of head poses and facial expressions.

Towards addressing this issue, we introduce LIA-X, a novel framework that learns interpretable motion semantics to enable a highly controllable 'edit-warp-render' animation strategy. Deviating from the standard 'warp-render' pipeline, LIA-X allows users to first leverage the learned motion representations to align source portrait with the initial driving frame before applying the final motion transfer (see Fig. 1).

Specifically, LIA-X is designed as a self-supervised autoencoder that does not rely on any explicit structure representations.

Inspired by sparse dictionary coding [24], we incorporates a novel Sparse Motion Dictionary - a set of motion vectors in autoencoder to capture the underlying motion distribution. The sparsity constraint encourages the model to use a minimal set of these motion vectors to reconstruct the training images, endowing the motion vectors with enhanced interpretability compared to the dense motion dictionary proposed in prior work [45, 48]. The animation process is then modeled as a linear navigation of these motion codes to produce optical-flow fields for warping the source portrait. Interestingly, we found that these interpretable motion vectors can also be directly leveraged at the inference stage to manipulate source portrait, supporting fine-grained edits of facial attributes (e.g., eyes and mouth) as well as 3D-aware transformation (e.g., yaw, pitch and roll) in both image and video domains.

In addition, we have analyzed the scalability of proposed LIA-X framework and successfully train large-scale models with up to 1 billion parameters using a diverse mixture of public and internal talking head datasets. Experimental results demonstrate that scaling up the LIA-X significantly improves performance across several benchmarks. Given the faster inference speed of autoencoder-based models compared to diffusion-based approaches, we believe the scalable and interpretable design of LIA-X can serve as a valuable complement to current state-of-the-art generative models, enabling efficient and highly controllable video generation. In summary, the key contributions of this work are:

* We propose a novel portrait animator, LIA-X, an autoencoder that incorporates an interpretable Sparse Motion Dictionary to enable controllable portrait animation via an 'edit-warp-render' strategy.
* We analyze the scalability of the LIA-X and demonstrate that our design can be scaled up to achieve superior performance.
* Extensive experiments show that LIA-X outperforms state-of-the-art methods across several datasets, while also supporting a various of applications such as image and video editing, and 3D-aware portrait manipulation.

# 2. Related Work

Portrait animation has seen significant advancements in recent years, driven by the remarkable progress in deep generative models for video generation [1–3, 7–9, 12, 14, 20, 22, 26, 30, 31, 34–36, 41–44, 46, 47, 51, 53, 56]. Previous methods have explored learning structure representations either based on conditional generation approaches [4, 38, 39, 52, 55, 58] relying on off-the-shelf extractors, or self-supervised learning strategies to learn representations such as 2D/3D keypoints [11, 28, 40, 58], motion regions [29], and depth maps [16] in an end-to-end manner. Self-attention mechanism [54] has also been studied to im-
<figure><img src="image_4.png" /><figcaption>Figure 2. Overview of LIA-X. LIA-X consists of an encoder E, a generator G that includes an optical-flow generator Gf and a rendering network Gr. LIA-X is trained using a self-supervised learning strategy. To obtain an interpretable motion dictionary Dm, a sparsity constraint is incorporated into the training objective, encouraging the network to use a minimal number of vectors in Dm to reconstruct each driving image.</figcaption></figure>

prove cross-identity generation quality.

More recent techniques [45, 48] have proposed to model motion transfer as a linear navigation of learned motion codes, which has proven effective in implicitly capturing 2D and 3D representations to capture complex facial dynamics. Diffusion-based models [49] have also been shown to be effective for portrait animation, due to their powerful generalization abilities pretrained on large-scale datasets.

However, one key limitation still exists across all the existing methods, they lack an effective mechanism to align the source portrait with the initial driving frame in terms of head poses and facial expressions. This makes their performance drop dramatically when there are large variations between the source and driving data, further limiting their usability in more general real-world scenarios. Different from previous approaches, LIA-X incorporates an interpretable motion dictionary, which allows users to edit source portrait before animation in a controllable manner. With this unique capacity, we found LIA-X can be further used in several tasks such as portrait animation, image and video editing, and 3D-aware portrait manipulation.

## 3. Preliminary

Latent Image Animator (LIA) [45, 48] is designed as an autoencoder consisting of an encoder E, an optical-flow generator Gf and a rendering network Gr aiming to transfer motion of a talking head to a still portrait via self-supervised learning. LIA models motion as linear navigation of motion code in latent space, and follows the 'warp-render' strategy

to generate optical-flow fields via Gf and render the animated result via Gr.

**Linear Navigation.** LIA models motion transferring as learning transformations from source to driving image $x_s \to x_d$. It proved that for any given image, there exists an 'implicit reference image' $x_r$, and the transformation can be modeled as $x_s \to x_r \to x_d$ in an implicit manner. The transformation is modeled as motion code $z_{s \to d}$ in latent space, and with the help of the reference space, it can be represented as a linear navigation from $z_{s \to r}$ along a path $w_{r \to d}$, denoted as

$$z_{s \to d} = z_{s \to r} + w_{r \to d}, \quad (1)$$

where $z_{s \to r}$ indicates the transformation from source image to reference image, and is obtained via $E(x_s) = z_{s \to r}$. To learn $w_{r \to d}$, a learnable motion dictionary $D_m$ consisting of a set of orthogonal motion vectors $\{\mathbf{d}_1, ..., \mathbf{d}_M\}$ is proposed. Any linear path in the latent space can be represented as

$$w_{r \to d} = \sum_{i=1}^{M} a_i \mathbf{d}_i, \quad (2)$$

where $A_{r \to d} = \{a_1, ..., a_M\}$ indicates the magnitude for each motion vector, and is obtained via $\mathcal{FC}(E(x_d)) = A_{r \to d}$. The linear navigation from any $x_s$ to $x_d$ can be represented as

$$z_{s \to d} = z_{s \to r} + \sum_{i=1}^{M} a_i \mathbf{d}_i. \quad (3)$$
**Image Animation.** Once $z_{s→d}$ is obtained, optical flows are generated via $G_f(z_{s→d}) = \phi_{s→d}$. The source image $x_s$ is warped based on $\phi$ and the warped features are rendered through $G_r$ to produce the generated image

$$x_{s→d} = G_r(\mathcal{T}(\phi_{s→d}, x_s)), \quad (4)$$

where $\mathcal{T}$ indicates the warping operation. In practice, multi-scale optical flows are generated to warp multi-scale feature maps of $x_s$.

**Learning.** The objective of self-supervised learning consists of three parts, an $L_1$ reconstruction loss, a VGG-based perceptual loss and an adversarial loss between $x_{s→d}$ and $x_d$

$$\mathcal{L}(x_{s→d}, x_d) = \mathcal{L}_{recon}(x_{s→d}, x_d) + \lambda \mathcal{L}_{vgg}(x_{s→d}, x_d) + \mathcal{L}_{adv}(x_{s→d}). \quad (5)$$

**Inference.** LIA proposed a unified formulation for self-reenactment and cross-reenactment at the inference stage

$$z_{s→t} = (z_{s→r} + w_{r→s}) + (w_{r→t} - w_{r→1}), \quad t \in \{1, ..., T\}, \quad (6)$$

For self-reenactment, where $w_{r→s} = w_{r→1}$, Eq. 6 can be simplified as

$$z_{s→t} = z_{s→r} + w_{r→t}, \quad t \in \{1, ..., T\}, \quad (7)$$

which is the same as the training stage.

For cross-reenactment, where $w_{r→s} \neq w_{r→1}$, Eq. 6 can be rewritten as

$$z_{s→t} = \underbrace{z_{s→s}}_{\text{reconstruction}} + \underbrace{(w_{r→t} - w_{r→1})}_{\text{motion difference}}, \quad t \in \{1, ..., T\}, \quad (8)$$

where the animation process is modeled as *linearly navigating the source image along the driving motion direction in the latent space*.

# 4. Methodology of LIA-X

In this section, we proceed to introduce our proposed LIA-X, including the general model architecture, the Sparse Motion Dictionary, as well as the interpretable and controllable animation capabilities.

## 4.1. Architecture

LIA-X follows the general architecture design of LIA, containing an encoder $E$, an optical-flow generator $G_f$, and a rendering network $G_r$. However, deviating from previous methods [28, 29, 58] that rely on simple and small-scale networks, we design the LIA-X architecture to be more scalable by incorporating advanced residual blocks inspired by StyleGAN-T [27] in both the encoder and generators.

## 4.2. Sparse Motion Dictionary

While the original motion dictionary in LIA contains certain semantic meanings, we observed that it is difficult to disentangle and independently control individual factors such as mouth movements, eyebrow expressions, etc. Multiple semantics are entangled and hard to leverage for controllable image and video editing tasks.

Inspired by *sparse dictionary coding*, we propose a Sparse Motion Dictionary to improve the interpretability of the motion representations. We found that this simple improvement is extremely effective in enhancing the interpretability of the motion vectors, which in turn enables more controllable animation.

Specifically, we introduce a sparse penalty $S(\cdot)$ on the motion coefficients $A_{r→d}$ to encourage the model to use a minimal number of motion vectors to reconstruct the images during self-supervised training. The overall learning objective of LIA-X is then:

$$\mathcal{L}(x_{s→d}, x_d) = \mathcal{L}_{recon}(x_{s→d}, x_d) + \lambda_1 \mathcal{L}_{vgg}(x_{s→d}, x_d) + \mathcal{L}_{adv}(x_{s→d}) + \lambda_2 S(A_{r→d}), \quad (9)$$

where $\lambda_1$ and $\lambda_2$ are coefficients to balance the losses, and we implement $S(\cdot)$ as the $L_1$ norm.

## 4.3. Controllable Inference

While Eq. 8 enables successful motion transfer from the driving video to the source portrait, the high-quality generation relies on a strong requirement - the source and initial driving frames should have similar head poses and facial expressions. Ideally, portraits with frontal pose and neutral expression usually result in the best performance. However, in various real-world applications, this requirement cannot always be satisfied. With the interpretable Sparse Motion Dictionary, LIA-X is able to address this issue by using an "edit-warp-render" strategy.

Users are first allowed to edit the source portrait using the corresponding motion vectors, aligning the head pose and facial expression to be as similar as possible to the driving image. Then, the animation process can be formulated as:

$$z_{s→t} = \underbrace{z_{s→\mathcal{E}(s)}}_{\text{editing}} + \underbrace{(w_{r→t} - w_{r→1})}_{\text{motion difference}}, \quad t \in \{1, ..., T\}, \quad (10)$$

where $\mathcal{E}(\cdot)$ indicates the editing operation on the source image. This allows LIA-X to better handle large variations between the source and driving data, leading to higher-quality and more controllable portrait animation.
<figure><img src="image_5.png" /><figcaption>(a) Video 1</figcaption></figure>

<figure><img src="image_6.png" /><figcaption>(b) Video 2</figcaption></figure>

<figure><img src="image_7.png" /><figcaption>(c) Roll</figcaption></figure>

<figure><img src="image_8.png" /><figcaption>(c) Right Rotation</figcaption></figure>

<caption>**Figure 5. 3D-aware Video Manipulation.** We demonstrate the use of learned 3D-aware semantics to manipulate a real-world video. The original video (a) has been successfully rotated to left (b) and right (c) respectively, without using any explicit 3D representation, and the identity of the original video subject has also been well-preserved throughout these manipulations.</caption>

## 5. Experiments

In this section, we will first describe the experimental setup, including implementation details and dataset information. Then, we will present a qualitative analysis on the sparsity, interpretability, and controllability of the proposed LIA-X framework. We will also show comparative results between LIA-X and state-of-the-art methods such as FOMM, TPS,

DaGAN, LIA, MCNet, X-Portrait, and LivePortrait, on the task of portrait animation. Finally, we will quantitatively evaluate LIA-X and compare it with the prior work on two important tasks, *self-reenactment* and *cross-reenactment*.

**Implementation details.** We build LIA-X based on the implementation of the original LIA [45, 48]. To scale the model to larger sizes and prevent training instability, we
<figure><img src="image_9.png" /><figcaption>Figure 6. **Image Editing.** We show image editing capacities of LIA-X. Fine-grained semantic attributes, such as *open/close mouth*, *frown/raise eyebrows*, *open/close eyes*, *pout*, and *smile* can be successfully controlled by manipulating the corresponding motion vectors.</figcaption></figure>

designed novel residual blocks for both the optical flow
generator $G_f$ and the rendering network $G_r$. Our scal-
ing strategy focuses on increasing the number of channels,
the depth of residual blocks, as well as using a larger mo-
tion dictionary compared to the original LIA. With these
architectural enhancements, the largest model size of LIA-
X reaches around 1 billion parameters. We train the entire
LIA-X model using 8 A100 GPUs, and apply gradient accu-
mulation to increase the effective batch size when training
the larger-scale models.

**Datasets.** To scale the training data, we mix 4 publicly available datasets, i.e., VoxCelebHQ [48], TalkingHead1KH [40], HDTF [57] and MEAD [37]. Additionally, we

include 1 internally collected dataset. In total, our training
dataset contains 0.5 million talking head sequences, com-
prising around 94 million frames and 55,000 different iden-
tities. Experiments show that this dataset scaling strategy
enables LIA-X to achieve outstanding performance in gen-
eralizing to unseen portrait images.

**5.1. Sparsity Analysis**

To demonstrate the effectiveness of our proposed Sparse
Motion Dictionary, we visualized the motion coefficient
vectors $A_{r \to s}$ for two different videos, as shown in Fig. 3.
We compare the results of models with and without using
the sparsity constraint in the motion dictionary.
<figure><img src="image_10.png" /><figcaption>Figure 7. **Qualitative Comparison on Cross-reenactment.** LIA-X's ability to edit the source portrait using interpretable motion vectors before animation allows it to better adjust for initial misalignments with the driving frame. As shown, this editing capability enables LIA-X to significantly outperform other approaches, especially when there are large variations in head pose and facial expression between the source and driving data.</figcaption></figure>

It can be clearly observed that the model without the sparse motion dictionary activates almost all the motion vectors, indicating a lack of selectivity for different input data. The semantics are entangled within each motion vector. In contrast, for the model with the Sparse Motion Dictionary, sparsity can be clearly observed in $A_{r→s}$ - only a few vectors are active, while the contributions of the others can be largely omitted. These results clearly prove the effectiveness of the constraint in improving the sparsity of the motion dictionary.

## 5.2. Interpretability and Controllability Analysis

While we have proven the ability to obtain a Sparse Motion Dictionary, it is crucial to understand whether each vector in the dictionary is interpretable and controllable. To investigate the semantic meanings of the motion vectors, we linearly manipulate the vector $d_i$ using:

$$z_{s→\mathcal{E}(s)} = z_{s→s} + a_i d_i, \quad (11)$$

where we set $a_i$ as a small perturbation ranging from -0.5 to 0.5 with a step of 0.1. Surprisingly, we found that the semantics are well-disentangled in the motion dictionary and can be easily manipulated. Almost all the activated vectors correspond to human-understandable meanings. Fig. 4 shows examples of LIA-X performing 3D-aware manipulations such as yaw, pitch and roll, without relying on any explicit 3D representation during training or inference.

**Image Editing.** We further utilize Eq. 11 to manipulate more motion vectors and demonstrate the results in Fig. 6. Besides the 3D-aware semantics, LIA-X can control various fine-grained attributes like 'open/close mouth', 'frown/raise eyebrows', 'open/close eyes', 'pout', 'smile', etc. All these interpretable vectors are automatically disentangled through self-supervised learning. With such a powerful editing capability, LIA-X can be readily used as a portrait editing tool, allowing users to linearly combine different semantic operations to achieve complex editing tasks.

**Video Editing.** Once we can edit a single portrait, extending the editing to the video level becomes feasible.
<table><caption>Table 1. **Quantitative Evaluation for Self-Reenactment.** We compare the performance of LIA-X against state-of-the-art methods on two different resolutions across two datasets.</caption><thead><tr><th rowspan="3">Method</th><th colspan="5">VoxCelebHQ</th><th colspan="5">TalkingHead-1KH</th></tr><tr><th>L1 ↓</th><th>LPIPS ↓</th><th>SSIM ↑</th><th>PSNR ↑</th><th>FID ↓</th><th>L1 ↓</th><th>LPIPS ↓</th><th>SSIM ↑</th><th>PSNR ↑</th><th>FID ↓</th></tr><tr><td colspan="10" style="text-align:center;">256 × 256 resolution</td></tr></thead><tbody><tr><td>FOMM [28]</td><td>0.046</td><td>0.27</td><td>0.66</td><td>22.40</td><td>12.67</td><td>0.040</td><td>0.100</td><td>0.72</td><td>23.31</td><td>30.39</td></tr><tr><td>DaGAN [16]</td><td>0.044</td><td>0.110</td><td>0.69</td><td>23.04</td><td><b>9.13</b></td><td>0.036</td><td>0.088</td><td>0.77</td><td>24.95</td><td>25.50</td></tr><tr><td>TPS [58]</td><td>0.043</td><td>0.112</td><td>0.70</td><td>23.24</td><td>10.82</td><td>0.037</td><td>0.089</td><td>0.77</td><td>24.56</td><td>28.05</td></tr><tr><td>MCNet [15]</td><td>0.040</td><td>0.176</td><td>0.72</td><td>23.73</td><td>18.63</td><td><b>0.030</b></td><td>0.097</td><td><b>0.79</b></td><td>25.70</td><td>28.06</td></tr><tr><td>LIA-X</td><td><b>0.036</b></td><td><b>0.095</b></td><td><b>0.73</b></td><td><b>24.82</b></td><td>10.74</td><td>0.035</td><td><b>0.086</b></td><td>0.78</td><td><b>26.26</b></td><td><b>24.71</b></td></tr><tr><td colspan="10" style="text-align:center;">512 × 512 resolution</td></tr><tr><td>X-Portrait [49]</td><td>0.110</td><td>0.302</td><td>0.56</td><td>16.99</td><td>19.92</td><td>0.058</td><td>0.134</td><td>0.63</td><td>19.46</td><td>41.19</td></tr><tr><td>LivePortrait [11]</td><td>0.087</td><td>0.264</td><td>0.67</td><td>17.45</td><td>12.90</td><td>0.052</td><td>0.120</td><td>0.73</td><td>20.26</td><td>39.98</td></tr><tr><td>LIA [48]</td><td>0.052</td><td>0.211</td><td>0.68</td><td>22.14</td><td>21.86</td><td>0.049</td><td>0.165</td><td>0.72</td><td>23.37</td><td>44.64</td></tr><tr><td>LIA-X</td><td><b>0.040</b></td><td><b>0.160</b></td><td><b>0.75</b></td><td><b>24.39</b></td><td><b>12.50</b></td><td><b>0.035</b></td><td><b>0.115</b></td><td><b>0.80</b></td><td><b>26.07</b></td><td><b>38.93</b></td></tr></tbody></table>

<table><caption>Table 2. **Quantitative Evaluation for Cross-Reenactment.** We compare the performance of LIA-X against state-of-the-art methods on the constructed dataset for cross-reenactment scenarios.</caption><thead><tr><th>Method</th><th>ID Similarity ↓</th><th>Image Quality ↑</th></tr></thead><tbody><tr><td>FOMM [28]</td><td>0.262</td><td>37.08</td></tr><tr><td>DaGAN [16]</td><td>0.272</td><td>39.30</td></tr><tr><td>TPS [58]</td><td>0.216</td><td>38.27</td></tr><tr><td>MCNet [15]</td><td>0.252</td><td>37.88</td></tr><tr><td>X-Portrait [49]</td><td>0.217</td><td>55.41</td></tr><tr><td>LivePortrait [11]</td><td>0.243</td><td>51.41</td></tr><tr><td>LIA-X</td><td><b>0.206</b></td><td><b>58.74</b></td></tr></tbody></table>

<table><caption>Table 3. **Scalability analysis on VoxCelebHQ.**</caption><thead><tr><th>Model</th><th>L1 ↓</th><th>LPIPS ↓</th><th>SSIM ↑</th><th>PSNR ↑</th></tr></thead><tbody><tr><td>Base (0.05B)</td><td>0.043</td><td>0.171</td><td>0.72</td><td>23.62</td></tr><tr><td>Middle (0.3B)</td><td><b>0.040</b></td><td><b>0.16</b></td><td>0.74</td><td>24.31</td></tr><tr><td>Large (0.9B)</td><td><b>0.040</b></td><td><b>0.16</b></td><td><b>0.75</b></td><td><b>24.39</b></td></tr></tbody></table>

<table><caption>Table 4. **Scalability analysis on TalkingHead-1KH.**</caption><thead><tr><th>Model</th><th>L1 ↓</th><th>LPIPS ↓</th><th>SSIM ↑</th><th>PSNR ↑</th></tr></thead><tbody><tr><td>Base (0.05B)</td><td>0.042</td><td>0.13</td><td>0.77</td><td>24.98</td></tr><tr><td>Middle (0.3B)</td><td><b>0.035</b></td><td><b>0.113</b></td><td>0.79</td><td>25.84</td></tr><tr><td>Large (0.9B)</td><td><b>0.035</b></td><td>0.115</td><td><b>0.80</b></td><td><b>26.07</b></td></tr></tbody></table>

Given a real-world talking head video, we first apply Eq. 11 on the initial frame to manipulate the target semantics. Then, we use Eq. 10 to transfer the motion to the edited first frame. Fig. 5 demonstrates the results of using LIA-X to rotate the portrait in a video, where the identity is well-preserved while the head pose is seamlessly changed.

**Portrait Animation.** We also qualitatively compare LIA-X with state-of-the-art methods on the challenging task of cross-identity portrait animation, as shown in Fig. 7. The visualizations clearly showcase the effectiveness of LIA-X's editing stage, which can significantly improve the generated results by aligning the source portrait with the initial driving frame. This editing capability allows for a more controllable animation process compared to previous techniques. In contrast, other state-of-the-art methods struggle when the source and driving data have large variations in head poses, facial expressions, and identities. Their performance degrades dramatically in such cross-identity scenarios, whereas LIA-X maintains high-quality and controllable animation results by leveraging its interpretable motion representations and the tailored "edit-warp-render" pipeline. This comparison highlights a key advantage of our proposed framework - its ability to handle large discrepancies

between the source and driving data through the initial editing stage, before seamlessly applying the motion transfer.

## 5.3. Quantitative Evaluation

We quantitatively compare LIA-X with state-of-the-art methods on two important tasks, *self-reenactment* and *cross-reenactment*. To ensure a fair comparison, we train LIA-X at two different resolutions, 256×256 and 512×512, and compare with the corresponding methods.

**Self-Reenactment.** We evaluate our method on the validation sets of VoxCelebHQ [48] and TalkingHead-1KH [40], which contain 483 and 25 videos respectively. We reconstruct each video sequence by using the first frame as the source image and the entire video as the driving data. We report results using several metrics, L1, LPIPS, SSIM, PSNR, and FID. As shown in Tab. 1, on both high and low resolutions, LIA-X outperforms both GAN-based and diffusion-based state-of-the-art methods across all the metrics, demonstrating the effectiveness of our proposed architecture design and training strategy.

**Cross-Reenactment.** To construct the validation set for cross-reenactment, we select 70 videos from the HDTF
dataset [57] as the driving data, and for each video, we randomly select 2 images from the AAHQ dataset [18] as the source data, resulting in a total of 140 videos. Since there is no ground truth data for this task, we evaluate the results using two metrics, *Identity Similarity* and *Image Quality*. Identity Similarity measures the average embedding difference between each generated frame and the source portrait, while Image Quality is computed following [33] to indicate the generated image quality. As shown in Tab. 2, LIA-X outperforms other methods in both metrics, proving its ability to naturally transfer motion while preserving the original identities.

## 5.4. Ablation study on scalability

To verify the effectiveness of scalability, we conduct an ablation study on the model size. We train three variations of LIA-X with 0.05B, 0.3B, and 0.9B parameters, respectively, while keeping the same training configuration. The three models follow the same architectural design and only differ in the number of residual blocks, channel numbers, and motion dictionary size. The results in Tab. 3 and 4 clearly demonstrate that scaling is effective for improving model performance. However, we note that the improvement from 0.3 billion to 0.9 billion parameters becomes relatively minor. We hypothesize that this could be due to the current dataset size not being large enough to fully support training such a large-scale model. To further push the limits of LIA-X's scalability and performance, we believe the involvement of even larger and more diverse training datasets would be necessary.

# 6. Limitations and future work

While our proposed method has achieved promising performance, there are still limitations need to be tackled in future work. Firstly, current model can only be used on fixed resolution, designing novel technique to allow dynamic resolution may further improve the performance. Secondly, our model follows the convolutional network architecture which may have limitations to further scale up. DiT-based [25] architecture should be explored in future work for scalability.

## 7. Conclusion

In this work, we have introduced LIA-X, a novel portrait animator that incorporates an interpretable Sparse Motion Dictionary. This enables LIA-X to support a highly controllable 'edit-warp-render' animation strategy. Furthermore, we have analyzed the scalability of the LIA-X architecture, demonstrating its ability to achieve outstanding results when scaling up to larger model sizes, up to 1 billion parameters. Extensive evaluations show that LIA-X outperforms SOTA methods across several benchmarks, while also supporting diverse applications like fine-grained image/video editing and high-quality

portrait animation. We envision our technique serve as a valuable complement to current generative approaches and provide a novel way for interpretable video generation.

## References

[1] Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with latent diffusion models. In CVPR, 2023. 2

[2] Tim Brooks, Janne Hellsten, Miika Aittala, Ting-Chun Wang, Timo Aila, Jaakko Lehtinen, Ming-Yu Liu, Alexei A Efros, and Tero Karras. Generating long videos of dynamic scenes. 2022.

[3] Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, Clarence Ng, Ricky Wang, and Aditya Ramesh. Video generation models as world simulators. 2024. 2

[4] Caroline Chan, Shiry Ginosar, Tinghui Zhou, and Alexei A Efros. Everybody dance now. In ICCV, 2019. 2

[5] Di Chang, Yichun Shi, Quankai Gao, Jessica Fu, Hongyi Xu, Guoxian Song, Qing Yan, Xiao Yang, and Mohammad Soleymani. Magicdance: Realistic human dance video generation with motions & facial expressions transfer. arXiv preprint arXiv:2311.12052, 2023. 2

[6] Hao Chen, Yaohui Wang, Benoit Lagadec, Antitza Dantcheva, and Francois Bremond. Joint generative and contrastive learning for unsupervised person re-identification. In CVPR, 2021. 2

[7] Haoxin Chen, Yong Zhang, Xiaodong Cun, Menghan Xia, Xintao Wang, Chao Weng, and Ying Shan. Videocrafter2: Overcoming data limitations for high-quality video diffusion models. In CVPR, 2024. 2

[8] Xinyuan Chen, Yaohui Wang, Lingjun Zhang, Shaobin Zhuang, Xin Ma, Jiashuo Yu, Yali Wang, Dahua Lin, Yu Qiao, and Ziwei Liu. Seine: Short-to-long video diffusion model for generative transition and prediction. In ICLR, 2023.

[9] Aidan Clark, Jeff Donahue, and Karen Simonyan. Adversarial video generation on complex datasets. arXiv preprint arXiv:1907.06571, 2019. 2

[10] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In NIPS, 2014. 2

[11] Jianzhu Guo, Dingyun Zhang, Xiaoqiang Liu, Zhizhou Zhong, Yuan Zhang, Pengfei Wan, and Di Zhang. Livepor-trait: Efficient portrait animation with stitching and retarget-ing control. arXiv preprint arXiv:2407.03168, 2024. 2, 8

[12] Yuwei Guo, Ceyuan Yang, Anyi Rao, Zhengyang Liang, Yaohui Wang, Yu Qiao, Maneesh Agrawala, Dahua Lin, and Bo Dai. Animatediff: Animate your personalized text-to-image diffusion models without specific tuning. International Conference on Learning Representations, 2024. 2
[13] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. *Advances in Neural Information Processing Systems*, 33:6840–6851, 2020. 2

[14] Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P Kingma, Ben Poole, Mohammad Norouzi, David J Fleet, et al. Imagen video: High definition video generation with diffusion models. *arXiv preprint arXiv:2210.02303*, 2022. 2

[15] Fa-Ting Hong and Dan Xu. Implicit identity representation conditioned memory compensation network for talking head video generation. In *ICCV*, 2023. 8

[16] Fa-Ting Hong, Longhao Zhang, Li Shen, and Dan Xu. Depth-aware generative adversarial network for talking head video generation. 2022. 2, 8

[17] Yijun Li, Chen Fang, Jimei Yang, Zhaowen Wang, Xin Lu, and Ming-Hsuan Yang. Flow-grounded spatial-temporal video prediction from still images. In *ECCV*, 2018. 2

[18] Mingcong Liu, Qiang Li, Zekui Qin, Guoxin Zhang, Pengfei Wan, and Wen Zheng. Blendgan: Implicitly gan blending for arbitrary stylized face generation. In *Advances in Neural Information Processing Systems*, 2021. 9

[19] Wen Liu, Zhixin Piao, Jie Min, Wenhan Luo, Lin Ma, and Shenghua Gao. Liquid warping gan: A unified framework for human motion imitation, appearance transfer and novel view synthesis. In *CVPR*, 2019. 2

[20] Xin Ma, Yaohui Wang, Xinyuan Chen, Gengyun Jia, Ziwei Liu, Yuan-Fang Li, Cunjian Chen, and Yu Qiao. Latte: Latent diffusion transformer for video generation. *Transactions on Machine Learning Research*, 2025. 2

[21] Yue Ma, Hongyu Liu, Hongfa Wang, Heng Pan, Yingqing He, Junkun Yuan, Ailing Zeng, Chengfei Cai, Heung-Yeung Shum, Wei Liu, et al. Follow-your-emoji: Fine-controllable and expressive freestyle portrait animation. *arXiv preprint arXiv:2406.01900*, 2024. 2

[22] Willi Menapace, Aliaksandr Siarohin, Ivan Skorokhodov, Ekaterina Deyneka, Tsai-Shien Chen, Anil Kag, Yuwei Fang, Aleksei Stoliar, Elisa Ricci, Jian Ren, et al. Snap video: Scaled spatiotemporal transformers for text-to-video synthesis. In *CVPR*, 2024. 2

[23] Katsunori Ohnishi, Shohei Yamamoto, Yoshitaka Ushiku, and Tatsuya Harada. Hierarchical video generation from orthogonal information: Optical flow and texture. In *AAAI*, 2018. 2

[24] Bruno A Olshausen and David J Field. Emergence of simple-cell receptive field properties by learning a sparse code for natural images. *Nature*, 381(6583):607–609, 1996. 2

[25] William Peebles and Saining Xie. Scalable diffusion models with transformers. *arXiv preprint arXiv:2212.09748*, 2022. 9

[26] Masaki Saito, Eiichi Matsumoto, and Shunta Saito. Temporal generative adversarial nets with singular value clipping. In *ICCV*, 2017. 2

[27] Axel Sauer, Tero Karras, Samuli Laine, Andreas Geiger, and Timo Aila. StyleGAN-T: Unlocking the power of GANs for fast large-scale text-to-image synthesis. 2023. 4

[28] Aliaksandr Siarohin, Stéphane Lathuilière, Sergey Tulyakov, Elisa Ricci, and Nicu Sebe. First order motion model for image animation. In *NeurIPS*, 2019. 2, 4, 8

[29] Aliaksandr Siarohin, Oliver Woodford, Jian Ren, Menglei Chai, and Sergey Tulyakov. Motion representations for articulated animation. In *CVPR*, 2021. 2, 4

[30] Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, Devi Parikh, Sonal Gupta, and Yaniv Taigman. Make-a-video: Text-to-video generation without text-video data. In *ICLR*, 2023. 2

[31] Ivan Skorokhodov, Sergey Tulyakov, and Mohamed Elhoseiny. Stylegan-v: A continuous video generator with the price, image quality and perks of stylegan2. In *CVPR*, 2022. 2

[32] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In *International Conference on Learning Representations*, 2021. 2

[33] Shaolin Su, Qingsen Yan, Yu Zhu, Cheng Zhang, Xin Ge, Jinqiu Sun, and Yanning Zhang. Blindly assess image quality in the wild guided by a self-adaptive hyper network. In *CVPR*, 2020. 9

[34] Yu Tian, Jian Ren, Menglei Chai, Kyle Olszewski, Xi Peng, Dimitris N. Metaxas, and Sergey Tulyakov. A good image generator is what you need for high-resolution video synthesis. In *ICLR*, 2021. 2

[35] Sergey Tulyakov, Ming-Yu Liu, Xiaodong Yang, and Jan Kautz. MoCoGAN: Decomposing motion and content for video generation. In *CVPR*, 2018.

[36] Carl Vondrick, Hamed Pirsivash, and Antonio Torralba. Generating videos with scene dynamics. In *NIPS*, 2016. 2

[37] Kaisiyuan Wang, Qianyi Wu, Linsen Song, Zhuoqian Yang, Wayne Wu, Chen Qian, Ran He, Yu Qiao, and Chen Change Loy. Mead: A large-scale audio-visual dataset for emotional talking-face generation. In *ECCV*, 2020. 6

[38] Ting-Chun Wang, Ming-Yu Liu, Jun-Yan Zhu, Guilin Liu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. Video-to-video synthesis. In *NeurIPS*, 2018. 2

[39] Ting-Chun Wang, Ming-Yu Liu, Andrew Tao, Guilin Liu, Jan Kautz, and Bryan Catanzaro. Few-shot video-to-video synthesis. In *NeurIPS*, 2019. 2

[40] Ting-Chun Wang, Arun Mallya, and Ming-Yu Liu. One-shot free-view neural talking-head synthesis for video conferencing. In *Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition*, 2021. 2, 6, 8

[41] Yaohui Wang. *Learning to Generate Human Videos*. Theses, Inria - Sophia Antipolis ; Université Cote d'Azur, 2021. 2

[42] Yaohui Wang, Piotr Bilinski, Francois Bremond, and Antitza Dantcheva. G3AN: Disentangling appearance and motion for video generation. In *CVPR*, 2020.

[43] Yaohui Wang, Piotr Bilinski, Francois F Bremond, and Antitza Dantcheva. ImaGINator: Conditional Spatio-Temporal GAN for Video Generation. In *WACV*, 2020.

[44] Yaohui Wang, Francois Bremond, and Antitza Dantcheva. Inmodegan: Interpretable motion decomposition generative adversarial network for video generation. *arXiv preprint arXiv:2101.03049*, 2021. 2

[45] Yaohui Wang, Di Yang, Francois Bremond, and Antitza Dantcheva. Latent image animator: Learning to animate images via latent space navigation. In *ICLR*, 2022. 2, 3, 5
[46] Yaohui Wang, Xinyuan Chen, Xin Ma, Shangchen Zhou, Ziqi Huang, Yi Wang, Ceyuan Yang, Yinan He, Jiashuo Yu, Peiqing Yang, et al. Lavie: High-quality video generation with cascaded latent diffusion models. arXiv preprint arXiv:2309.15103, 2023. 2

[47] Yaohui Wang, Xin Ma, Xinyuan Chen, Antitza Dantcheva, Bo Dai, and Yu Qiao. Leo: Generative latent image animator for human video synthesis. arXiv preprint arXiv:2305.03989, 2023. 2

[48] Yaohui Wang, Di Yang, Francois Bremond, and Antitza Dantcheva. Lia: Latent image animator. IEEE Transactions on Pattern Analysis and Machine Intelligence, pages 1–16, 2024. 2, 3, 5, 6, 8

[49] You Xie, Hongyi Xu, Guoxian Song, Chao Wang, Yichun Shi, and Linjie Luo. X-portrait: Expressive portrait animation with hierarchical motion attention. 2024. 3, 8

[50] Hongyi Xu, Guoxian Song, Zihang Jiang, Jianfeng Zhang, Yichun Shi, Jing Liu, Wanchun Ma, Jiashi Feng, and Linjie Luo. Omniavatar: Geometry-guided controllable 3d head synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12814–12824, 2023. 2

[51] Wilson Yan, Yunzhi Zhang, Pieter Abbeel, and Aravind Srinivas. Videogpt: Video generation using vq-vae and transformers. arXiv preprint arXiv:2104.10157, 2021. 2

[52] Zhuoqian Yang, Wentao Zhu, Wayne Wu, Chen Qian, Qiang Zhou, Bolei Zhou, and Chen Change Loy. Transmomo: Invariance-driven unsupervised video motion retargeting. In CVPR, 2020. 2

[53] Sihyun Yu, Jihoon Tack, Sangwoo Mo, Hyunsu Kim, Junho Kim, Jung-Woo Ha, and Jinwoo Shin. Generating videos with dynamics-aware implicit generative adversarial networks. In ICLR, 2022. 2

[54] Gong Yuan, Zhang Yong, Cun Xiaodong, Yin Fei, Fan Yanbo, Wang Xuan, Wu Baoyuan, and Yang Yujiu. Toontalker: Cross-domain face reenactment, 2023. 2

[55] Egor Zakharov, Aliaksandra Shysheya, Egor Burkov, and Victor Lempitsky. Few-shot adversarial learning of realistic neural talking head models. In ICCV, 2019. 2

[56] David Junhao Zhang, Jay Zhangjie Wu, Jia-Wei Liu, Rui Zhao, Lingmin Ran, Yuchao Gu, Difei Gao, and Mike Zheng Shou. Show-1: Marrying pixel and latent diffusion models for text-to-video generation. International Journal of Computer Vision, pages 1–15, 2024. 2

[57] Zhimeng Zhang, Lincheng Li, Yu Ding, and Changjie Fan. Flow-guided one-shot talking face generation with a high-resolution audio-visual dataset. In CVPR, 2021. 6, 9

[58] Jian Zhao and Hui Zhang. Thin-plate spline motion model for image animation. In CVPR, 2022. 2, 4, 8