Animate-X: Universal Character Image Animation with Enhanced Motion Representation
Authors: Shuai Tan
Biao Gong
Xiang Wang
Shiwei Zhang
DanDan Zheng
Ruobing Zheng
Kecheng Zheng
Jingdong Chen
Ming Yang
Venue: ICLR 2025 Poster (2025)
Keywords: Animation
Anthropomorphic
Video Generation
Pose
Primary area: generative models
TL;DR: A universal animation framework based on LDM for various character types (collectively named  X), including anthropomorphic characters.

Abstract: Character image animation, which generates high-quality videos from a reference image and target pose sequence, has seen significant progress in recent years. However, most existing methods only apply to human figures, which usually do not generalize well on anthropomorphic characters commonly used in industries like gaming and entertainment. Our in-depth analysis suggests to attribute this limitation to their insufficient modeling of motion, which is unable to comprehend the movement pattern of the driving video, thus imposing a pose sequence rigidly onto the target character. To this end, this paper proposes $\texttt{Animate-X}$, a universal animation framework based on LDM for various character types (collectively named $\texttt{X}$), including anthropomorphic characters. To enhance motion representation, we introduce the Pose Indicator, which captures comprehensive motion pattern from the driving video through both implicit and explicit manner. The former leverages CLIP visual features of a driving video to extract its gist of motion, like the overall movement pattern and temporal relations among motions, while the latter strengthens the generalization of LDM by simulating possible inputs in advance that may arise during inference. Moreover, we introduce a new Animated Anthropomorphic Benchmark ($\texttt{$A^2$Bench}$) to evaluate the performance of $\texttt{Animate-X}$ on universal and widely applicable animation images. Extensive experiments demonstrate the superiority and effectiveness of $\texttt{Animate-X}$ compared to state-of-the-art methods.

Published as a conference paper at ICLR 2025
ANIMATE-X: UNIVERSAL CHARACTER IMAGE ANI- MATION WITH ENHANCED MOTION REPRESENTATION
Shuai Tan Biao Gong", Xiang Wang", Shiwei Zhang",
Dandan Zheng', Ruobing Zheng', Kecheng Zheng', Jingdong Chen', Ming Yang'
I Ant Group Allibbba Group
(tanshuai2001, a. bia.. www.bbaai..ggg...oooo gongggggmml..ccoom
[xiaclac wx, zhangjin. zww.ealibaba-ine com, {yuandan. zdd, zhengruebing. zrb, zhengkecneng... zkc, wttp://agnnnngnnnnnnnooo.. m.. wargggggggggggr.....oo
Project Page: https: //wusr///uaaraaae http://uucaaraaaaadearaaaaaaaaae acideaaayiygyyygiggb.b..... io/ hwtp:iiii/nniiieeeeemem Animate-X:
BA B BAJG BEIE IE
Y U
YOU
OUOU YOU
Figure I: Animations produced by Animate- which extends beyond human to anthropomorphic characters with various body structures, egg., without limbs, from games, animations, and posters.
ABSTRACT
Character image animation, which generates high-quality videos from reference image and target pose sequence, has seen significant progress recent years. However, most existing methods only apply to human figures, which usually do not generalize well on anthropomorphii characters commonly used in industries like gaming and entertainment. Our in-depth analysis suggests to attribute this limitation to their insufficient modeling of motion, which unable to compre- hend the movement pattern of the driving video, thus imposing pose sequence rigidly onto the target character. To this end, this paper proposes An i ma te-X, universal animation framework based on LDM for various character types (col- lectively named X), including anthropomorphic characters. To enhance motion representation, we introduce the Pose Indicator, which captures comprehensive motion pattern from the driving video through both implicit and explicit manner. The former leverages CLIP visual features a driving video to extract its gist of motion, like the overall movement pattern and temporal relations among motions, while the latter strengthens the generalization of LDM by simulating possible in- puts in advance that may arise during inference. Morcover, we introduce new Animated Anthropomorphic Benchmark (A²Bench) to evaluate the performance of AnimteexX universal and widely applicable animation images. Extensive experiments demonstrate the superiority and effectiveness of Animate- e com- pared sate-of-the-ar methods.
Work done during internship at Ant Group. Project lead and corresponding author.

---

Published as a conference paper at ICLR 2025
INTRODUCTION
Character image animation Yang et al. (2018); Zablotskaia al. (2019b) is compelling and chal- lenging task that aims to generate lifelike, high-quality videos from a reference image and target pose sequence. A modern image animation method shall ideally balance the identity preservation and motion consistency, which contributes to the promise of broad utilization Hu et al. (2023); Xu et al. (2023a); Chang et al. (2023a); Jiang et al. (2022). The phenomenal successes of GAN Good- fellow al. (2014); Yu et al. (2023); Zhang al. (2022b) and generative diffusion models Ho al. (2022; 2020); Guo et al. (2023) have reshaped the performance of character animation generation. Nevertheless, most existing methods only apply to the human-specific character domain. In practice, the concept of "character" encompasses a much broader concept than human, including anthropo- morphic figures in cartoons and games, collectively referred as X, which are often more desirable in gaming. film, short videos, etc. The difficulty in extending current models to these domains can be attributed two main factors: (1) the predominantly buman-centeren nature of available datasets, and (2) the limited generalization capabilities of current motion representations.
The limitations are clearly evidenced for non-human characters in Fig. To replicate the given poses, the diffusion models trained on human dance video datasets tend to introduce unrelated hu- man characteristics which may not make sense to reference figures, resulting in abnormal distor- tions. In other words, these models treat identity preservation and motion consistency as confticting goals and struggle to balance them, while motion control often prevails. This issue is particularly pronounced for non-human anthropomorphic characters, whose body structures often differ from human anatomy-suue
as disproportionately large heads or the absence of arms, as shown in Fig. 1. The primary cause is that the motion representations extracted merely from pose conditions are hard to generalize to a broad range of common cartoon characters with unique physical characteristics, leading to their excessive sacrifices in identity preservation in favor of strict pose consistency, which is an unsensible trade-off between these conflicting goals.
To address this issue, the natural approach is enhance the flexibility of motion representations without discarding current pose condition, which can prevent the model from making unsensible trade-offs between overly precise poses and low fidelity to reference images. To this end, we iden- tify two key limitations of existing methods. First, the simple 21) pose skeletons, constructed by connecting sparse keypoints, lack of image-level details and therefore cannot capture the essence of the reference video, such as motion-induced deformations (e.g., body part overlap and occlusion) and overall motion patterns. Second, the self-driven reconstruction strategy aligns reference and pose skeletons by body shape, simplifying animation but ignoring shape differences during infer- ence. These inspire us to design the new Pose Indicator from both implicit and explicit perspectives.
In this paper, we propose Animate-X for animating any character X. Sparked by generative diffu- sion models Rombach et al. (2022). we employ a 3D-UNet Blattmann et al. (2023) as the denoising network and provide it with motion feature and figure identity as condition. To fully capture the gist of motion from the driving video, we introduce the Pose Indicator, which consists of the Implicit Pose Indicator (IPP) and the Explicit Pose Indicator (EPI). Specifically, IPI extracts implicit motion- related features with the assistance of CLIP image feature, isolating essential motion patterns and re- lations that cannot be directly represented by the pose skeletons from the driving video. Meanwhile, EPI enhances the representation and understanding of the pose encoder by simulating real-world misalignments between the reference image and driven poses during training, strengthening the ability to generate explicit pose features. With the combined power of implicit and explicit features, An i mate-X demonstrates strong character generalization and pose robustness, enabling general x character animation even though it is trained solely on human datasets. Moreover, we introduce a new Animated Anthropomorphic Benchmark (A'Bench), which includes 500 anthropomorphic characters along with corresponding dance videos, to evaluate the performance of Animate-1 on other types of characters. Extensive experiments on both public human animation datasets and A Bennh demonstrate that Animate-2 X outperforms state-of-the-- offtthe-arttt methods in preserving iden- tity and maintaining motion consistency in animating x. Main contributions summarized as follows:
We present Animate-X, which facilitates mange-conditioned pose-guided video genera- tion with high generalizability, particularly for attractive anthropomorphic characters. To the best of our knowledge, this is the first work to animate generic cartoon images without the need for strict pose alignment.
2

---

Published as a conference paper at ICLR 2025
The rethinking about the motion inspire to propose Pose Indicator, which extracts motion representation suitable for anthropomorphic characters in both implicit and explicit manner, enhancing the robustness of Animate-X.
Since the popular datasets only contain human video with limited character diversity, we present a new A'Bench specifically for evaluating performance on anthropomorphic char- acters. Extensive experiments demonstrate that our An i ma t--X outperforms the compet- ing methods quantitatively and qualitatively on both A Be and current human anima- tion benchmark.
2 RELATED WORK
2.1 DIFFUSION MODELS FOR IMAGE/VIDEO GENERATION
In recent years, diffusion models Song et al. (2021); Ho et al. (2020) have demonstrated strong gen- erative capabilities, pushing image generation technique towards daily productivity tool Nichol et al. (2022); Ramesh et al. (2022); Mou et al. (2023); Huang et al. (2023); Zhang et al. (2023a); Liu et al. (2023). Pioneering works such as DALL-E 2 Ramesh et al. (2022) and Imagen Saharia et al. (2022) have showcased the extraordinary potential of diffusion models for high-quality image synthesis. Notable contributions, including Stable Diffusion Rombach et al. (2022), have well bal- anced scalability and efficiency, making diffusion-based image generation accessible and versatile across various applications. On the video generation front, diffusion models are making amazing progress Singer al. (2023); Wang et al. (2023a; 2024c); Wu et al. (2023); Chai et al. (2023); Cey- lan et al. (2023); Guo et al. (2023): Zhou et al. (2022); An et al. (2023); Xing et al. (2023); Qing al. (2023); Yuan et al. (2023); Tan et al. (2024e); Gong et al. (2024); Wei et al. (2024a;b); Tan et al. (2024a); Shi et al. (2024). These methods joint spatio-temporal modeling to generate realistic motion dynamics and ensure temporal consistency, marking a substantial step forward in generative models for video content. In this work, we aim to tackle the character-centerr task, a dedicated of conditional video generation. Our approach enables the transformation of static images into dynamic animations by conditioning on desired motion. This innovation bridges the gap between image and video generation, highlights the versatility and adaptability of diffusion models in creating engaging visual narratives.
image animation
2.2 POSE-GUIDED CHARACTER MOTION TRANSFER
Character image animation aims to transfer motion from the source character to the target iden- tity Zhang et al. (2024); Chang et al. (2023b), which has experienced an impressive journey to improve animation quality and versatility. Early works Li et al. (2019); Siarohin et al. (2019b; 2021b); Zhao & Zhang (2022b); Tan et al. (2024b); Wang et al. (2022); Tan et al. (2024d:c; 2023); Pan et al. (2024) predominantly utilize Generative Adversarial Networks (GANs) to generate ani- mated human images. However, these GAN-based models are often confronted by the emergence of various artifacts in the generated outputs. With the advent of diffusion models, researchers Shen al. (2024); Zhu al. (2024) explored how to go beyond GANs. One effort is Disco Wang et al. (2023b), which leverages ControlNet Zhang et al. (20236) to facilitate human dance generation, demonstrating the potential of diffusion models in generating dynamic human poses. Following this, Animcen Xu et al. (2023b) and Animate Anyone Hu et al. (2023) introduce transformer- temporal attention modules Vaswani (2017), enhancing the temporal consistency of animations and resulting in more smooth movement transitions. Sparked by the linear time efficiency of Mamba Gu & Dao (2023); Gu et al. (2021) conceptually merges the merits of parallelism and non-locality, Unianimate Wang et al. (2024b) resorts to it resorts to Mamba for efficient temporal modeling.
While these approaches have improved the realism of the animations, a notable limitation remains: most current methods require strict alignment between reference image and driving video. This restricts their applicability in the scenarios where poses cannot be easily extracted, such as anthropo- morphic characters, often resulting in bizarre and unsatisfactory outputs. In contrast, our approach adopts a robust and flexible motion representation to mitigate the dependence on pose alignment. This enables the generation of high-quality animations even in cases where previous methods strug- gle with non-alignable poses. In this manner, our method enhances the versatility and applicability of character image animation across broad range of contexts (x character).
3

---

Published as a conference paper at ICLR 2025
Implicit Pose Indicator (IPI)
DiiRoss
Ref. Image I"
Driven Video Lir
Festare
Transformer Encoder
Cross Attentonn
Qasy
EPI
IPI
ft
P
Nubic
Lensable Query
Mage Quea Un
Image Condition
Explicit Pose Indicator (EPI)
Sunple
Eg
1 (videa)
Pose Transforaaiion
Tranffemed
Skyp
Deall
Peol
Poss
Pooc
1
Animated Video
C
Spatial Attn.
-amm fre -Agg 1AR
-foce
Aonalioe
Pose 1
Ae!
Motion Altn.
Temporal
Figure 2: (a) The overview of our Animate-X. Given reference image I", we first extract CLIP image feature fü and latent feature f" via CLIP image encoder & and VAE encoder The proposed Implicit Pose Indicator (IPI) and Explicit Pose Indicator (EPI) produce motion feature ʃi and pose feature fas respectively. fe is concatenated with the noised input along the channel dimension, then further concatenated with fr along the temporal dimension. This serves the input to the diffusson model Eg for progressive denoising. During the denoising process, f and fa provide appearance condition from I" and motion condition from F1.F- At last, a VAE decoder D is adopted to map the generated latent representation 2n the animation video. (b) The detailed structure of Implicit Pose Indicator. (c) The pipeline of pose transformation by Explicit Pose Indicator.
3 METHOD
In this work, aim generate animated video that maintains consistency in identity with reference image and body movement with a driving video - Different from previous works, primary objective is to animate general characters beyond human, particularly like anthropo- morphic ones, which has broader applications entertainment industry.
3.1 PRELIMINARIES OF LATENT DIFFUSION MODEL
A diffusion model (DM) operates by learning probabilistic process that models data generation through noise. To mitigate the heavy computational load of traditional pixel-based diffusion models in high-dimensional RGB spaces, latent diffusion models (LDMs)) Rombach al. (2022) propose to shift the process into a lower-dimensional latent space using pre-trained variational autoen- coder (VAE) Kingma (2013). It encodes the input data into compressed latent representation an- Gaussian noise is then incrementally added to this latent representation several steps, reducing computational requirements while maintaining the generative capabilities the model. The process be formalized as:
q(z 2.-1) = NiZz; v - 11-84111,3,1,,
(1)
where 3, E (0, 1) represents the noise schedule. As E 1,2, T increases, the cumulative noise applied to the original 2g intensifies, causing 2, progressively resemble random Gaussian noise.

---

Published as a conference paper at ICLR 2025
Compared to the forward diffusion process, the reverse denoising process py aims to reconstruct the clean sample Zu from the noisy input z,. We represent the denoising step zt 1 (zt) as follows:
Po (z 112)) (2(11: (10 1), Eo f)),
in which (ze 1) refers to the estimated target of the reverse diffusion process and the process typically is achieved by diffusion model Eş with the parameters 0. To model the temporal dimen- sion, the denoising model Eg is commonly built on a 3D-UNet architecture Blattmann et al. (2023) in video generation methods Hu al. (2023); Wang al. (2023c). Given the input conditional guidance c, they usually use an L2 loss to reduce the difference between the predicted noise and the ground-trute -ruth noise during the optimization process:
L = Ee ||e Es Ez t, e)]
(3)
once the reversed denoising stage is complete, the predicted clean latent is passed through the VAE decoder to reconstruct the predicted video in pixel space.
3.2 POSE INDICATOR
To extract motion representations, previous works typically detect the pose keypoints via DW- Pose Yang et al. (2023) from the driven video 1 and further visualize them as pose image TP, which are trained using self-driven reconstruction strategy. However, it brings several limitations as mentioned in Sec. I: (1) The sole pose skeletons lack image-level details and are therefore un- able to capture the essence of the reference video, such as motion-induced deformations and overall motion patterns. (2) The self-driven reconstruction training strategy naturally aligns the reference and pose images in terms of body shape, which simplifies the animation task by overlooking likely body shape differences between the reference image and the pose image during inference. Both limitations weaken the model develop deep, holistic motion understanding, leading inade- quate motion representation. To address these issues, we propose Pose Indicator, which consists of Implicit Pose Indicator (IPI) and Explicit Pose Indicator (EPI).
Implicit Pose Indicator (IPI). To extract unified motion representations from the driving video in the first limitation, we resort to the CLIP image feature f ((I..)) extracted by CLIP Image Encoder. CLIP utilizes contrastive learning to align the embeddings of related images and texts, which may include descriptions of appearance, movement, spatial relationships and etc. Therefore, the CLIP image feature is actually a highly entangled representation, containing motion patterns and relations helpful to animation generation. As presented in Fig. 2 (a), we introduce a lightweight extractor P which is composed of N stacked layers of cross-attention and feed-forward networks (FFN), In cross attention layer, we employ f as the keys (K) and values (V). Consequently, the challenge becomes designing an appropriate query (Q), which should act guidance for motion extraction. Considering that the keypoints od extracted by DWPose provide a direct description of the motion, we design a teassforrrr-bbssee encoder to obtain the embedding 9p. which is regarded as an ideal candidate for Q. Nevertheless, motion modeling using sole sparse keypoints is overly simplistic, resulting in the loss of underlying motion patterns. To this end, we draw inspiration from query transformer architecture Awadalla et al. (2023); Jaegle et al. (2021) and initialize a learnable query vector qi to complement sparse keypoints. Subsequently, we feed the merged query 4m = 4r + qi and f into P and get the implicit pose indicator ft, which contains the essential representation of motion that cannot be represented by the simple 2D pose skeletons.
Explicit Pose Indicator (EPI). To deal with the second limitation in the training strategy, we pro- pose EPI, designed to train the model to handle misaligned input pairs during inference. The key insight lies in simulating misalignments between reference image and pose images during training while ensuring the motion remains consistent with the given driving video 1 Therefore, we ex- plore two pose transformation schemes: Pose Realignment and Pose Rescale. As shown in Fig. 2 (b), in the pose realignment scheme, we first establish pose pool containing pose images from the training set. In each training step, we first sample the reference image I" and the driving pose Il following previous works. Additionally, we randomly select an align anchor pose I annchr from the pose pool. This anchor serves reference for aligning the driving pose, producing the aligned pose 12 P However, since the characters we aim to animate are often anthropomorphic char- acters, whose shapes can significantly differ from human, such as varying extremely short legs, even the absence of arms (as shown in Fig. I and Fig. 5), relying solely
ratios,
5

---

Published as a conference paper at ICLR 2025
on pose realignment is insufficient to capture these variations for simulation. Therefore, we further introduce Pose Rescale. Specifically, we define a set of keypoint rescaling operations, including modifying the length of the body, legs, arms, neck, and shoulders, altering face size, even adding or removing specific body parts and etc. These transformations are stored in a rescale pool. After ob- taining the realigned poses TP "realign" we apply a random selection of transformations from this pool with certain probability on them, generating the final transformed poses H, (additional examples of transformations are provided in the Appendix A). Note that we set the probability of À E [0, 1] to apply the pose transformation, and with a probability of 1 X, the pose image remains unchanged. Subsequently, T is encoded to the explicit feature fe via a Pose Encoder.
3.3 FRAMEWORK AND IMPLEMENT DETAILS
In light of the success of previous works Hu et al. (2023); Zhang et al. (2024), Animate -X follows the main framework, which consists of several encoders for feature extraction and 3D-UNet Wang et al. (2023a;c); Blattmann et al. (2023) for video generation. As shown in Fig. 2, given a reference image 1", employ the pretrained CLIP Image Encoder & Radford al. (2021) to extract ap- pearance feature J from I". To reduce the parameters of the framework and facilitate appearance alignment, we exclude the Reference Net presented in most of the previous works Hu et al. (2023); Zhang et al. (2024); Zhu et al. (2024). Instead, a VAE encoder & is utilized to extract the latent representation f from I", which is then directly used as part of the input for the denoising network eg following Wang et al. (2024b). For the driven video [i,,, we detect the pose keypoints p" and CLIP feature 1d via a DWPose Yang et al. (2023) and CLIP Image Encoder Φ Subsequently, IPI and EPI introduced in Sec. 3.2 extract the implicit latent f. and explicit latent fe. respectively. The explicit fe is first concatenated with the noised latent to obtain the fused features along the channel dimension, which is further stacked with fo along the temporal dimension, resulting combined features Smerge. Then, the combined features are fed into the video diffusion model cp for jointly appearance alignment and motion modeling. The diffusion model eg comprises multiple stacked lay- ers of Spatial Attention, Motion Attention and Temporal Attention. The Spatial Attention receives inputs from fmerge and f.' and fuses the identity condition from I" with the motion condition from Id through cross-attention (CA), producing an intermediate representation x. To further enhance motion consistency, the implicit representation f; is fed into the Motion Attention module, along with x in the form of a residual connection, resulting in the representation x' = x + CA(x, ft). Inpsired by the linear time efficiency of Mamba Gu & Dao (2023) in long sequence processing, employ it as Temporal Attention module to maintain the temporal consistency.
Training and Inference. To improve the model's robustness against pose and reference image misalignments, we adopt two key training schemes. First, we set a high transformation probability (over 98%) in the EPI, enabling the model to handle wide range of misalignment scenarios. Second, we apply random dropout to the input conditions at a predefined rate Wang et al. (2024b). After that, while the reference image and driven video are from the same human dancing video during training, in the inference phase (Fig. 9 (b)), Animate-X can handle an arbitrary reference image and driven video, which may differ appearance.
3.4 A'BENCH
The main task of our Animate-X is to animate an anthropomorphic char- acter with vivid and smooth mo- tions. However, current publicly avail- able datasets Jafarian & Park (2021); Zablotskaia et al. (2019a) primarily focus on human animation and fall short in capturing a broad range of anthropomorphic characters and cor- responding dancing videos. This gap makes these datasets and benchmarks unsuitable for quantitatively evaluat- ing different methods in anthropomor- phic character animation.
Figure 3: Examples from our A'Bench.

---

Published as a conference paper at ICLR 2025
Method
PSNR* 1 SSIM
1.1
LPIPS 4
HID 4
FID-VID 4
FVD.
Aooee-AniaateAnoonn Corporation (2024)
9.86
0.299
1,58-044
0.626
50.97
75.11
1367.84
MimicMotion Zhang et al. (2024) (ArXiv24)
10.18
0.318
1.51F004
0.622
122.92
129.40
2250.13
Cott Peng et al. (2024) (ArXiv24)
10.88
0.379
1.38E-04
0.572
68.15
81.05
1652.09
MusePose Tong et al. (2024) (ArXi244)
11.05
0.397
1.27E-04
0.549
100.91
114.15
1760.46
Unianimate Wang et al. (2024b) (AXX124)
11.82
0.398
1.24E-04
0.532
48.47
61.03
1156.36
Animate-X X
13.60
0.452
1.02E-04
0.430
26.11
32.23
703.87
Table 1:
Quantitative comparisons with SOTAs on A'Bench with the rescaled pose setting.
"PSNR "" means using the modified metric Wang et al. (2024a) to avoid numerical overflow.
Method
PSNR=1
SSIM
[
LIT
LPIPS
FID
FID
FVD..
FOMM Starohin al. (2019a) (NauiPS19)
10.49
0.363
1.47E-04
0.613
183.18
147.82
2535.12
MRAA Siarohin al. (2021a) (CVPR211)
12.62
0.420
1.09E-04
0.556
161.57
196.87
3094.68
LIA Wang et al. (2022) (CCL222)
13.78
0.445
9.70E-05
0.497
105.13
78.51
1813.28
DreamPose Karras
(2023) (CCV223)
7.76
0.305
2,28E-04
0.534
277.64
315.58
4324.42
MagicAnimate Xu al. (2023a) (CVP134)
11.90
0.396
1.17E-04
0.523
117.09
117.54
2021.93
Anymtt Corporation (2024) (CVPR29)
11.56
0.360
1.27E-04
0.532
37.82
59.80
1117.29
MimicMotion Zhang et al. (2024) (AXX)24)
12.66
0.407
1.07F-04
0.497
96.46
61.77
1368.83
ControlNeXt Peng et al. (2024) (ArXic24)
12.82
0.421
1.02E-04
0.472
46.66
59.41
1152.96
MusePose Tong (2024) (ArXi224)
12.92
0.438
9.90E-0S
0.470
80.22
87.97
1401.96
Animate-X
14.10
0.463
8.92F-05
0.425
31.58
33.15
849.19
Table 2: Quantitative comparisons with existing methods on A in the self-driven setting. Underline means the second best result.
To bridge this gap, we propose the Animated Anthropomorphic character Benchmark (A' to comprehensively evaluate the performance of different methods. Specifically, we first provide a prompt template to GPT-4 OpenAl (2024) and leverage it to generate 500 prompts, each of which contains a textual description of an anthropomorphic character. Please refer to Appendix B.2 for details. Inspired by the powerful image generation capability of KLing AI Technology (2024), we feed the produced prompts into its Text-TTo Imale module, which synthesizes the corresponding anthropomorphic character images according the given text prompts. Subsequently, the Image- To- Video module is employed to further make the characters in the images dance vividly. For each prompt, we repeat the process for 4 times and filter the most satisfactory image-video pairs as the output corresponding this prompt. In this manner, collect 500 anthropomorphic characters and the corresponding dance videos, as shown in Fig. 3. Please refer to Appendix B for details.
EXPERIMENTS
4.1 EXPERIMENTAL SETTINGS
Dataset. We collect approximately 9,000 hu-
man videos from the internet and supplement
Ref Image la Driven pa
Align pa Aligned PC
Our
this with TikTok dataset Jafarian & Park (2021)
Figure 4: The illustration of comparison settings,
and Fashion dataset Zablotskaia et al. (2019a)
for training. Following previous works Hu al. (2023); Zablotskaia al. (2019a); Jafarian Park (2021), we use 10 and 100 videos for both qualitative and quantitative comparisons from Tik and Fashion dataset, respectively. We additionally experimented on 100 image-video pairs selected from the newly proposed Bh introduced in Sec 3.4. Please note that, to ensure fair comparison, the data in the B'Ber.ch are not included in the training set to train our model. The data are only used to evaluate the quantitative results and provide interesting reference image cases.
Evaluation Metrics. We assess the results using evaluation metrics in Appendix B.I, including PSNR Hore & Ziou (2010), SSIM Wang al. (2004), L1, LPIPS Zhang et al. (2018), which are widely-used image metrics for measuring the visual quality of the generated results. In addition, we introduce FID Heusel al. (2017), FID-VID Balaji et al. (2019) and FVD Unterthiner et al. (2018) to quantify the discrepancy between the generated video distribution and the real video distribution.
4.2 EXPERIMENTAL RESULTS
Quantitative Results. Since our Animate-X primarily focuses on animating the anthropomor- phic characters, very few of which, if not none, can be extracted the pose skeleton accurately by
7

---

Published as a conference paper at ICLR 2025
Ref. Image
Ref.
[|CU222]
LIA
Moore-AA [CVPR24)
MusePose
[AXXw24]
Unianimate
[ArXw24]
MimicMotion
[ArXw24]
ControlNeXt
[ArXiv24]
Figure Qualitative comparisons with state-of-the of-the-artttttt methods.
DWPose Yang et al. (2023). It naturally leads to a misalignment of the input reference image with the driving pose images. To compute quantitative results in this case, set up new comparison setting. For each case in Be (i.e., a reference image I" and pose pa as shown in Fig. 4), we randomly select one human's pose image 1e and align the anthropomorphic character's pose pa to it, such that the aligned pose P retains the movements of pa but has the same body shape (fat/thin, tall/short, etc.) as p². Ultimately, we take the anthropomorphic character La and the aligned driving pose image PK as inputs the model, generating results that allow to calculate quantitative metrics with the original anthropomorphic character dancing video in A A In this setting, we compare our method with Animate Anyone Hu al. (2023), Unianimate Wang al. (2024b), Mim- ieMotion Zhang et al. (2024), ControlNeXt Peng et al. (2024) and MusePose Tong et al. (2024), which also use pose images (e.g., pb in Fig. 4) as input. The results of Animate Anyone Hu et al. (2023) are obtained by leveraging the publicly available reproduced code Corporation (2024). Tab. presents the quantitative results, where An inimate-- markedly surpasses all comparative methods in terms of all metrics. It is worth noting that, do not use A'Bench as training data to avoid overfitting and ensure fair comparisons, in line with other comparative methods.
Following previous works which evaluate quantitative results in self-driven and reconstruction man- ner, we additionally compare our method with (a) GAN-based image animate works: FOMM Siaro- hin al. (2019a), MRAA Siarohin al. (2021a), LIA Wang et al. (2022). (b) Diffusion model-based image animate works: DreamPose Karras et al. (2023), MagicAnimate Xu et al. (2023a) and present the results in Tab. 2, which indicates that our method achieves the best performance across all the metrics. Moreover, we provide the quantitative results on the human dataset (TikTok and Fashion) in Tab. 10 and Tab. 11, respectively. Please refer Appendix D.2 for details, Animate-xx reaches the comparable score to Unianimate and exceeds other SOTA methods, which demonstrates the superiority of An A on both anthropomorphic and human benchmarks.
Qualitative Results. Qualitative comparisons of anthropomorphic animation are shown in Fig. 5. We observe that GAN-based LIA Wang al. (2022) does not generalize well, which can only work on specific dataset like Siarohin et al. (2019b). Benefiting from the powerful generative capabili- ties of the diffusion model, Animate Anyone Hu et al. (2023) renders a higher resolution image, but accurate reference pose motion. Although
the identity of the image changes and do not generate MusePose Tong et al. (2024). Unianimate Wang et al. (2024b) and MimicMotion Zhang et al. (2024) improve the accuracy of the motion transfer, these methods generate unseen person, which is not
8

---

Published as a conference paper at ICLR 2025
Unianimate
Ref. Image
Unianimate
Ours
Ref. Image
Figure 6: Qualitative comparisons with Unianimate in terms of long video generation.
Method
Moore-AA
MimicMotion
ControlNeXt
MusePose
Unianimate
Animate-X
Identity preservation f
60.4%
14.8%
52.0%
31.3%
43.0%
98.5% %
Temporal consistency
19.8%
24.9%
36.9%
43.9% 40.3%
81.1% 79.3%
93.4%
Visual quality f
27.0%
17.2%
40.4%
95.8 %
Table 3: User study results.
the desired result. ControlNeXt combines the advantages of the above two types of methods, so maintains the consistency of identity and motion transfer to some extent, yet the results are some- what unnatural and unsatisfactory, e.g., the ears the rabbit and the legs of the banana in Fig. In contrast, Animate X ensures both identity and consistency with the reference image while gener- ating expressive and exaggerated figure motion, rather than simply adopting quasi-static motion of the target character. Further, ates a woman out of thin air who dances according to the given pose images. Ani aerimmaee- animates the reference image in cute way while preserving appearance and temporal continuity, and it does not generate parts that do not originally exist. In summary, An Animate-X excels in maintaining ap- pearance and produeing precise, vivid animations with high temporal consistency. Please refer to Appendix D.I for details.
present some long video comparisons in Fig. 6. Unianimate gener-
User Study. To estimate the quality of our method and SOTAs from human perspectives, we con- duct blind user study with 10 participants. Specifically. we randomly select 10 characters from A'Bench and collect 10 driving video from the website. For each of methods tested, 10 anima- tion clips are generated, resulting in total of 60 clips. Each participant is presented two results generated by different methods for the same set of inputs and asked choose which one is better in terms of visual quality, identity preservation, and temporal consistency. This process is repeated C times. The results are summarized Tab. 3, where our method noticeably outperforms other methods in all aspects, demonstrating its superiority and effectiveness. Details in Appendix C.
4.3 ABLATION STUDY
Ablation on Implicit Pose Indicator. To analyze the contributions of Implicit Pose Indicator, remove it from Animate-X as w/o IPI and compare with Baseline and Animate-X. From the first row of Fig. 7, we observe that Baseline generates person whose appearance is appreciably distinct from the reference image. With the help EPI, this problem is mildly mitigated. However, due the absence of IPI, compared to Ours, there are still strange things and human-like hands

---

Published as a conference paper at ICLR 2025
Image
Baseline
Image
Baseline
Figure 7: Visualization of ablation study on IPI and EPI.
appearing, as indicated by the blue circle. For more detailed analysis about the structure of IPI, we set up several variants: (1) remove IPI: w/o IPI. (2) remove learnable query: w/o L.Q (3) remove DWPose query: w/o DQ. The quantitative results are shown in Tab. 4. It can be seen that removing the entire IPI presents the worst performance. By modifying the IPI module, although improves on the w/o IPI, it still falls short of the final result of Animate e-x, which suggests that current IPI structure is the most reasonable and achieves the best performance.
Ablation Explicit Pose Indica- for. We demonstrate the visual re- sults of ablating EPI setting in the second row of Fig. 7 by removing EPI. Without EPI, although the ap- pearance of the panda is preserved thanks IPI, the model incorrectly treats the panda's cars as arms and foreibly stretches the legs to match the length of the legs in the pose image indicated by red circles. In contrast, these issues are com- pletely resolved by the assistance of EPI. further conduct more detailed ablation experiments for different pairs of pose transformations by (I) removing the entire EPI: w/o EPI. (3) remove Pose Realignment: w/o Realignment. (2) removing Pose Rescale: w/o Rescale; From the results dis-
Method w/o IPI
PSNR
TSIM T
LIT
LPIPS
FID
13.30
0.433
1.35E-04
0.454
32.56
64.31
893.3]
w/o
13.48
0.445
1.76E-04
0.454
28.24
42.74
754.37
w/o DQ
13.39
0.445
1.01E-04
0.456
30.33
62.34
913.33
w/o EPI
12.63
0.403
1.80-004
0.509
42.17
58.17
948.25
Realign
12.27
0.433
1.17E-04
0.434
34.60
49.33
860.25
w/o Rescale
13.23
0.438
1.21E-04
0.464
27.64
35.95
721.11
Animate-X
13.60
0.452
1.02F-04
0.430
26.11
32.23
703.87
Table Quantitative results of ablation study.
played in Tab. 4, we found that Pose Realignment contributes the most. It suggests that simulating misalignment case in inference is the the key factor.
In summary, we draw conclusions: (1) IPI facilitates the preservation of appearance and prevents the generation of content that does not exist in the reference image like human arms. (2) EPI prevents the forced alignment of pose image that is not naturally aligned with the reference image during animation, thus avoiding the unintended animation of parts that should remain static like the panda's ears shown in Fig. 7. Please refer Appendix D.5 for details.
5 CONCLUSIONS
In this study, we present Animate-X, a novel approach to character animation capable of gen- eralizing across different types of characters named x. address the imbalance between identity preservation and movement consistency caused by the insufficient motion representation, we intro- duce the Pose Indicator, which leverages both implicit and explicit features to enhance the motion understanding of the model. In this way, Animate-2 demonstrates strong generalization and ro- bustness, achieving general X character animation. The proposed framework showcases significant improvements
state-of-the-ar methods in terms of identity preservation and motion consis- tency, as evidenced by experiments on both public datasets and the newly introduced A'Bench, which features anthropomorphic characters. Limitation and ethical considerations Appendix E.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Jie An, Songyang Zhang, Harry Yang, Sonal Gupta, Jia-Bin Huang, Jiebo Luo, and Xi Yin. Latent- shift: Latent diffusion with temporal shift for efficient text-to-video generation. arXiv preprint arXiv:2304.0847, 2023.
Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, et al. Openflamingo: An open- source framework for training large autoregressive vision-language models. arXiv preprint arXiv:2308.0133 2023.
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, al. Qwen technical report. arXiv preprint MXii:2309.166600 2023.
Yogesh Balaji, Martin Renqiang Min, Bing Bai, Rama Chellappa. and Hans Peter Graf. Conditional GAN with discriminative filter generation for text-to-video synthesis. In IJCAI, volume 1, pp. 2, 2019.
Ankan Kumar Bhunia, Salman Khan, Hisham Cholakkal, Rao Muhammad Anwer, Jorma Laakso- Mubarak Shah, and Fahad Shahbaz Khan. Person image synthesis via denoising diffusion model. In CVPR, pp. 9968-5976, 2023.
Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with latent diffusion models. In CVPR, pp. 22563-22575, 2023.
Zinelabidine Boulkenafet, Jukka Komulainen, and Abdenour Hadid. Face anti-spoofing based on color texture analysis. In 2015 IEEE international conference on image processing (ICIP), pp. 2636-2640. IEEE, 2015.
Duygu Ceylan, Chun-Har P Huang, and Niloy J Mitra. Pix2video: Video editing using image diffusion. In ICCV, pp. 1-206-23217, 2023.
Wenhao Chai, Xun Guo, Gaoang Wang, and Yan Lu. Stablevideo: Text-driven consistency-aware diffusion video editing. In ICCV, pp. 23040-23050, 2023.
Di Chang, Yichun Shi, Quankai Gao, Jessica Fu, Hongyi Xu, Guoxian Song, Qing Yan, Xiao Yang, and Mohammad Soleymani. Magicdance: Realistic human dance video generation with motions & facial expressions transfer. arXiv preprint arXiv:2311 1. 12052, 2023a.
Di Chang, Yichun Shi, Quankai Gao, Hongyi Xu, Jessica Fu, Guoxian Song, Qing Yan, Yizhe Zhu, Xiao Yang. and Mohammad Soleymani. Magicpose: Realistic human poses and facial expressions retargeting with identity-aware diffusion. In Forty-first International Conference on Machine Learning. 2023b.
Moore Threads Corporation. Moore-AnimateAnyynne 2024. URL https: com/ wwwpee/hrraad//oooeennnaaaaa
Biao Gong, Siteng Huang, Yutong Feng, Shiwei Zhang, Yuyuan Li, and Yu Liu. Check locate rectify: A training-free layout calibration system for text-to-image generation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 6624-6634, 2024.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. NeurIPS, 27, 2014.
Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752, 2023.
Albert Gu, Karan Goel, and Christopher Ré. Efficiently modeling long sequences with structured 2021.
state spaces. arXiv preprint
Yuwei Guo, Ceyuan Yang, Anyi Rao, Yaohui Wang. Yu Qiao, Dahua Lin, and Bo Dai. Animatediff: Animate your personalized text-to-image diffusion models without specific tuning. arXiv preprint arXiv.2307.04725, 2023.
11

---

Published as a conference paper at ICLR 2025
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. NeurIPS, 33: 6840-6851, 2020.
Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik Kingma, Ben Poole, Mohammad Norouzi, David J Fleet, et al. Imagen video: High definition video generation with diffusion models. arXiv preprint arXiv:2210.02303, 2022.
Alain Hore and Djemel Ziou. Image quality metrics: Psnr vs. ssim. In 2010 20th international conference on pattern recognition, pp. 2366-2369. IEEE, 2010.
Li Hu, Xin Gao, Peng Zhang, Ke Sun, Bang Zhang, and Liefeng Bo. Animate anyone: Consistent and controllable image-to-video synthesis for character animation. arXiv preprint arXiv:2311.17117, 2023.
Lianghua Huang, Di Chen, Yu Liu, Yujun Shen, Deli Zhao, and Jingren Zhou. Composer: Creative and controllable image synthesis with composable conditions. ICML, 2023.
Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianx- ing Wu, Qingyang Jin, Nattapol Chanpaisit, Yaohui Wang, Xinyuan Chen, Limin Wang, Dahua Lin, Yu Qiao, and Ziwei Liu. VBench: Comprehensive benchmark suite for video generative mod- els. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024.
Andrew Jaegle, Felix Gimeno, Andy Brock, Oriol Vinyals, Andrew Zisserman, and Joao Carreira. Perceiver: General perception with iterative attention. In International conference on machine learning, pp. 4651-4664 PMLR, 2021.
Yasamin Jafarian and Hyun Soo Park. Learning high fidelity depths dressed humans by watching social media dance videos. In CVPR, pp. 17753-12722, 2021.
Yuming Jiang, Shuai Yang, Haonan Qiu, Wayne Wu, Chen Change Loy, and Ziwei Liu. Text2human: Text-driven controllable human image generation. ACM Transactions on Graphics, 41(4):1-11,200 2022
Johanna Karras, Aleksander Holynski, Ting-Chun Wang, and Ira Kemelmaches-SShizermnn Dream- pose: Fashion video synthesis with stable diffusion. In ICCV, pp. 22680-22690, 2023.
Diederik Kingma. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 2013.
Yining Li. Chen Huang, and Chen Change Loy. Dense intrinsic appearance flow for human pose transfer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3693-3702, 2019.
Ming Liu, Yuxiang Wei, Xiaohe Wu, Wangmeng Zuo, and Lei Zhang. Survey leveraging pre-trained generative adversarial networks for image editing and restoration. Science China Information Sciences, 6615):151101, 2023.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017.
Chong Mou, Xintao Wang, Liangbin Xie, Jian Zhang, Zhongang Qi, Ying Shan, and Xiaohu Qie. T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models. arXiv preprint arXiv:2302.08453, 2023.
Alexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam. Pamela Mishkin, Bob Megrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. In ICML, pp. 16784-16804, 2022.
OpenAL. Chatgpt-4o. 2024. URL https::////// httpps://chhae /httttcaatteeeeeeeeeeeeeeccc openai com/ chat..
12

---

Published as a conference paper at ICLR 2025
Ye Pan, Shuai Tan, Shengran Cheng, Qunfen Lin, Zijiao Zeng, and Kenny Mitchell. Expressive talking avatars. IEEE Transactions on Visualization and Computer Graphics, 2024.
Bohao Peng, Jian Wang, Yuechen Zhang, Wenbo Li, Bing-Chang Yang, and Jiaya Jia. Controlnext: Powerful and efficient control for image and video generation. arXiv preprint arXiv:2408.0600 2024.
Zhiwu Qing, Shiwei Zhang, Jiayu Wang, Xiang Wang, Yujie Wei, Yingya Zhang, Changxin Gao, and Nong Sang. Hierarchical spatio-temporal decoupling for text-to-video generation. arXiv preprint arXiv:2312.04483 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICMI., pp. 8748-8763, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 1(2):3, 2022.
Yurui Ren, Xiaoqing Fan, Ge Li, Shan Liu, and Thomas H Li. Neural texture extraction and distri- bution for controllable person image synthesis. CVPR, pp. 13535-13544, 2022.
Robin Rombach, Andreas Blattmann, Dominík Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In CVPR, pp. 10684-10695, 2022.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily I. Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Bureu Karagol Ayan, Tim Salimans, et Photorealistic text-to-image diffusion models with deep language understanding. NeurIPS, 35:36479-36494, 2022.
Fei Shen, Hu Ye, Jun Zhang, Cong Wang, Xiao Han, and Yang Wei. Advancing pose-guided image synthesis with progressive conditional diffusion models. In The Twelfth International Conference on Learning Representations, 2024. httpt:::: /oppnevveeeeeee.e net / forum? Id=rHzapPnCgT.
Shuwei Shi, Biao Gong, Xi Chen, Dandan Zheng, Shuai Tan, Zizheng Yang, Yuyuan Li, Jingwen He, Kecheng Zheng, Jingdong Chen, et Motionstone: Decoupled motion intensity modulation with diffusion transformer for image-to-video generation. arXiv preprint arXiv:2412.0558
2024.
Aliaksandr Siarohin, Stéphane Lathuilière, Sergey Tulyakov, Elisa Ricci, and Nicu Sebe. First order motion model for image animation. NeurlPS, 32, 2019a.
Aliaksandr Siarohin, Stéphane Lathuilière, Sergey Tulyakov, Elisa Ricci, and Nicu Sebe. First order motion model for image animation. Advances in neural information processing systems, 32, 2019b.
Aliaksandr Siarohin, Oliver J Woodford, Jian Ren, Menglei Chai, and Sergey Tulyakov. Motion representations for articulated animation. In CVPR, pp. 13653-13662, 2021a.
Aliaksandr Siarohin, Oliver J Woodford, Jian Ren, Menglei Chai, and Sergey Tulyakov. Motion rep- resentations for articulated animation. Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 13653-13668 2021b.
Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, et al. Make-a-vin vivieo:: Text-to-video generation without tixt-video data. ICLR, 2023.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In ICLR, 2021.
Shuai Tan, Bin Ji, and Ye Pan. Emmn: Emotional motion memory network for audio-driven emo- tional talking face generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 22146-22156, 2023.
13

---

Published as a conference paper at ICLR 2025
Shuai Tan, Biao Gong, Yutong Feng, Kecheng Zheng, Dandan Zheng, Shuwei Shi, Yujun Shen, Jingdong Chen, and Ming Yang. Mimir: Improving video diffusion models for precise text un- derstanding, arXiv preprint arXiv:2412.03085, 2024a.
Shuai Tan, Bin Ji, Mengxiao Bi, and Ye Pan. Edtalk: Efficient disentanglement for emotional talking head synthesis. arXiv preprint 24X1v:240.001677, 2024b.
Shuai Tan, Bin Ji, Yu Ding, and Ye Pan. Say anything with any style. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 5088-5096, 2024c.
Shuai Tan, Bin Ji, and Ye Pan. Flowvqtalker: High-quality emotional talking face generation through normalizing flow and quantization. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 26317-26327, 2024d.
2024d.
Shuai Tan, Bin Ji, and Ye Pan. Style2talker: High-resolution talking head generation with emotion style and art style. In Proceedings of the AAAI Conference Artificial Intelligence, volume 38, pp. 107795087, 2024c.
Kuaishou Technology. Kling ai. 2024. URL httpthttt////ktt
kuai shou. com.
Zhengyan Tong, Chao Li, Zhaokang Chen, Bin Wu, and Wenjiang Zhou. Musepose: pose-driven image-to-video framework for virtual human generation. arxiv, 2024.
Thomas Unterthiner, Sjoerd Van Steenkiste, Karol Kurach, Raphael Marinier, Marcin Michalski, and Sylvain Gelly. Towards accurate generative models of video: new metric challenges. arXiv preprint arXiv: wwwv:8112.0 111.01111..17,, 2018.
A Vaswani. Attention all you need. Advances in Neural Information Processing Systems, 2017.
Jiuniu Wang, Hangjie Yuan, Dayou Chen, Yingya Zhang, Xiang Wang, and Shiwei Zhang. Mod- elscope text-to-video technical report. arXiv preprint arXiv:2308.0657 2023a.
Tan Wang, Linjie Li, Kevin Lin, Chung-Ching Lin, Zhengyuan Yang, Hanwang Zhang, Zicheng Liu, and Lijuan Wang. Disco: Disentangled control for referring human dance generation in real world. arviv e-prints, pp. arXiv-2307, 2023b.
Tan Wang, Linjie Li. Kevin Lin, Shung-Ching Lin, Zhengyuan Yang, Hanwang Zhang, Zicheng Liu, and Lijuan Wang. Disco: Disentangled control for referring human dance generation real world. In ICLR, 2024a.
Xiang Wang, Hangjie Yuan, Shiwei Zhang, Dayou Chen, Jiuniu Wang, Yingya Zhang, Yujun Shen, Deli Zhao, and Jingren Zhou. Videocomposer: Compositional video synthesis with motion con- trollability. NeurIPS, 2023c.
Xiang Wang, Shiwei Zhang, Changxin Gao, Jiayu Wang, Xiaoqiang Zhou, Yingya Zhang, Luxin Yan, and Nong Sang. Unianimate: Taming unified video diffusion models for consistent human image animation. arXiv preprint arXiv:2406.0r 188, 2024b.
Xiang Wang, Shiwei Zhang, Hangjie Yuan, Zhiwu Qing, Biao Gong, Yingya Zhang, Yujun Shen, Changxin Gao, and Nong Sang. recipe for scaling up text-to-video generation with text-free videos. In CVPR, 2024c.
Yaohui Wang, Di Yang, Francois Bremond, and Antitza Dantcheva. Latent image animator: Learn- ing to animate images via latent space navigation. arXiv preprint arXiv:2203.09043, 2022.
Zezheng Wang, Zitong Yu, Chenxu Zhao, Xiangyu Zhu, Yunxiao Qin, Qiusheng Zhou, Feng Zhou, and Zhen Lei. Deep spatial gradient and temporal depth learning for face anti-spoofing. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 5042- 5051, 2020.
Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing, 13(4):600- 612, 2004.
14

---

Published as a conference paper at ICLR 2025
Yujie Wei, Shiwei Zhang, Zhiwu Qing, Hangjie Yuan, Zhiheng Liu, Yu Liu, Yingya Zhang, Jingren Zhou, and Hongming Shan. Dreamvideo: Composing your dream videos with customized sub- ject and motion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6537-6549, 2024a.
Yujie Wei, Shiwei Zhang, Hangjie Yuan, Xiang Wang, Haonan Qiu, Rui Zhao, Yutong Feng, Feng Liu, Zhizhong Huang, Jiaxin Ye, al. Dreamvideo-2: Zero-shot subject-driven video customiza- tion with precise motion control. arXiv preprint arXiv:2410.1383) 2024b.
Jay Zhangjie Wu, Yixiao Ge, Xintao Wang, Stan Weixian Lei, Yuchao Gu, Yufei Shi, Wynne Hsu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. Tunc-a-video: One-shot tuming of image diffusion models for text-to-video generation. ICCV, pp. 7623-7633, 2023.
Zhen Xing, Qi Dai, Han Hu, Zuxuan Wu, and Yu-Gang Jiang. Simda: Simple diffusion adapter for efficient video generation. arXiv preprint arXiv:2308.0977 037000 2023.
Zhongcong Xu, Jianfeng Zhang, Jun Hao Liew, Hanshu Yan, Jia-Wei Liu, Chenxu Zhang, Jiashi Feng, and Mike Zheng Shou. Magicanimate: Temporally consistent human image animation using diffusion model. arXiv preprint uIXi:2311.166499 2023a.
Zhongcong Xu, Jianfeng Zhang. Jun Hao Liew, Hanshu Yan, Jia-Wei Liu, Chenxu Zhang, Jiashi Feng, and Mike Zheng Shou. Magicanimate: Temporally consistent human image animation using diffusion model. In arXiv, 2023b.
Ceyuan Yang. Zhe Wang, Xinge Zhu, Chen Huang, Jianping Shi, and Dahua Lin. Pose guided human video generation. In ECCV, pp. 201-216, 2018.
Zhendong Yang, Ailing Zeng, Chun Yuan, and Yu Li. Effective whole-body pose estimation with two-stages distillation. In ICCV, pp. 4210-4220, 2023.
Wing-Yin Yu, Lai-Man Po, Ray CC Cheung, Yuzhi Zhao, Yu Xue, and Kun Li. Bidirectionally deformable motion modulation for video-based human pose transfer. In ICCV, pp. 7502-7512, 2023.
Zitong Yu, Chenxu Zhao, Zezheng Wang, Yunxiao Qin, Zhuo Su, Xiaobai Li, Feng Zhou, and Guoying Zhao. Searching central difference convolutional networks for face anti-spoofing. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 5295- 5305, 2020.
Hangjie Yuan, Shiwei Zhang, Xiang Wang, Yujie Wei, Tao Feng, Yining Pan, Yingya Zhang, Ziwei Liu, Samuel Albanie, and Dong Ni. Instructvideo; Instructing video diffusion models with human feedback. arXiv preprint arXiv:2312. 2023.
Polina Zablotskaia, Aliaksandr Siarohin, Bo Zhao, and Leonid Sigal. Dwnet: Dense warp-based network for pose-guided human video generation. arXiv preprint 191001900 2019a.
Polina Zablotskaia, Aliaksandr Siarohin, Bo Zhao, and Leonid Sigal. Dwnet: Dense warp-based network for pose-guided human video generation. arXiv preprint arXiv: 1910.0919,,, 2019b.
Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control text-to-image diffusion models. In ICCV, pp. 3836-3847, 2023a.
Lymin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control text-to-image diffusion models, 2023b.
Pengze Zhang, Lingxiao Yang, Jian-Huang Lai, and Xiaohua Xie. Exploring dual-task correlation for pose guided person image generation. In CVPR, pp. 7713-7722, 2022a.
Pengze Zhang, Lingxiao Yang, Jian-Huang Lai, and Xiaohua Xie. Exploring dual-task correlation for pose guided person image generation. In CVPR, pp. 7713-7722, 2022b.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features perceptual metric. In CVPR, pp. 586-595, 2018.
15

---

Published as a conference paper at ICLR 2025
Yuang Zhang, Jiaxi Gu, Li- Len Wang, Han Wang, Junqi Cheng, Yuefeng Zhu, and Fangyuan Zou. Mimicmotion: High-quality human motion video generation with confidence-awe aware pose guid- ance. arXiv preprint arXiv:2406.19680, 2024.
Jian Zhao and Hui Zhang. Thin-plate spline motion model for image animation. In CVPR, pp. 3657-3666, 2022a.
Jian Zhao and Hui Zhang. Thin-plate spline motion model for image animation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6657-3666, 2022b.
Daquan Zhou, Weimin Wang. Hanshu Yan, Weiwei Lv, Yizhe Zhu, and Jiashi Feng. Magicvideo: Efficient video generation with latent diffusion models. arXiv preprint arXiv:2211.11015 2022.
Shenhao Zhu, Junming Leo Chen, Zuozhuo Dai, Yinghui Xu, Xun Cao, Yao Yao, Hao Zhu, and Siyu Zhu. Champ: Controllable and consistent human image animation with 3d parametric guidance. In European Conference Computer Vision (ECCV), 2024.
16

---

Published as a conference paper at ICLR 2025
APPENDICES
A NETWORK DETAILS
Due to space constraints in the main paper, we only present brief overview of the EPI process. Here, in Fig. we provide more detailed explanation of the pose transformation in EPI, along with additional case examples. First, we sample a driving pose JP and then randomly select an anchor pose 7 anchor from the pose pool (two examples are shown in Fig. 8). The driving pose Ie is aligned to the anchor pose lanctorr resulting in the aligned pose reatign. Next, we apply several rescaling operations randomly chosen from the rescale pool to further modify the aligned pose 7 realign By combining different rescaling options, we can obtain multiple transformed poses F. However, it is important to note that in each training step, only one anchor pose Lanchor and one rescaling combination are selected, so only one transformed pose I used for training. As shown in the Fig. 8, the transformed pose I retains the same motion as the sampled pose IP but has body shape similar the anchor pose IP lanchor. This simulates scenarios during inference where there are body shape differences between the reference image and the driving pose, enabling the model to generalize to such cases.
In the experiments, we use the visual encoder of the multi-modal CLIP-Huge model Radford et al. (2021) in Stable Diffusion v2.1 Rombach al. (2022) to encode the CLIP embedding of the refer- ence image and driving videos. The pose encoder, composed of several convolutional layers, follows a similar structure to the STC-encoder in VideoComposer Wang et al. (2023c). For model initial- ization, employ a tre-trained video generation model Wang et al. (2024c), done in previous approaches Xu et al. (2023a); Hu et al. (2023); Zhu et al. (2024): Wang et al. (2024b). The experi- ments carried out using 8 NVIDIA A100 GPUs. During training, videos resized resolution 768x512 12 pixels, and we feed the model with uniformly sampled video segments of 32 frames ensure temporal consistency. We the Adam optimizer Loshchilov & Hutter (2017) with learning rates of 5e-7 for the implicit pose indicator and 5e-5 for other modules. For noise sampling, DDPM Ho al. (2020) with 1000 steps applied during training, In the inference phase, we adjust the length of the driving pose to align roughly with the reference pose and used the DDIM sampler Song et (2021) with 50 steps for faster sampling.
spatial
Same
Some
Bedy Shops
Pode RESUE
Alighed Fooe Fastee
poss
Seket severd
Transformed
presss free
nescat pea
paspled
Anchor posa F.
Pose Rescals
Select one enchor from poss pool
Aligned paNk Pratuan
Body Shape
Same
Transformed
Figure More example for EPI.
17

---

Published as a conference paper at ICLR 2025
Ref. Image
Driven Video
Ref. Image
Driven Video
Animate-X -X
Animate-X -
Animated Video
Animated Video
(a) Train
(b) Inference
Figure 9: The difference of training and inference pipeline. During training, the reference image and the driven video come from the same video, while in the inference pipeline, the reference image and the driven video can be from any sources and appreciably different.
B BENCHMARK DETAILS
B. EVALUATION METRIC
We employ several evaluation metrics to quantitatively assess our results, including PSNR, SSIM, L1, LPIPS, FID, FID-VID and FVD. The detailed metrics are introduced follows:
PSNR is a measure used to evaluate the quality of reconstructed images compared to the original ones. It is expressed decibels (dB) and higher values indicate better quality. PSNR is commonly used in image compression and restoration fields.
SSIM assesses the similarity between two images based their luminance, contrast, and structural information. It considers perceptual phenomena affecting human vision and thus provides better correlation with perceived image quality than PSNR.
The metric refers to the mean absolute difference between the corresponding pixel val- ues of two images. It quantifies the average magnitude of errors in predictions without considering their direction, making it useful for measuring the extent of differences.
LPIPS is perceptual distance metric based on deep learning. It evaluates the similarity between images by analyzing the feature representations of image patches and tends to align well with human visual perception, making it suitable for tasks like image generation.
FID is used assess the quality of images generated by generative models (like GANs) by comparing the distribution of generated images to that of real images in feature space (extracted by pretrained CNN). Lower FID values suggest that the generated images are more similar to real images.
FID-VID extends the FID metric video data. It measures the quality of generated videos by comparing the distribution of generated video features to real video features, providing insights into the temporal aspects of video generation.
FVD is another metric for evaluating video generation, similar to FID. It measures the distance between the feature distributions of real and generated videos, taking both spatial and temporal dimensions into account. Lower FVD indicates that generated videos are closer to real regarding visual quality and dynamics.
18

---

Published as a conference paper at ICLR 2025
Below template:: cute anthropomarphic (objrct], cule, anthrepamorphic, with arms and legs, standing, daneing. [scason], [proviacc). /spceifie lacation), high quality.
replace
[|
are examples:
cute onthropmmopphic mobile phone, cute, anthropomorphii,
legs, standing, dancing, speiing, cute authropomorphic sonthern potate, cais, anthrepomerphic, with armss and legs, standing, dancing, during the
Lizoning. plains, high quality.
Chinese New Year, Harbin, the lce and Snow World, nigh quality.
anthrepomorphic
bottle. cule. unthoopomorphit with legs, standing, dancing, in by seuside, quality.
use your imagination generate similar sentences, sentence such as everyday Homms furnitur,, fruits, natural creatures, etc.
object vary widely.
cute anthoopomopphic thane has, cute, anthepomorphic, hands and feet, stonding, dancing, spring, Guangsongg restaurant, high quality
ropmopohpp sts, cute, anthropomopphic hands standing, dancing, summer, Zieejinng, living room, high quality
A lovely unthropomorphic cocennt, cuée, anthropomorhhic, with hands and feet, standing, dancing, summer, Ilainan, in the coconut orchard, high quality
cute anthropomorphic panda, cute, anthropomorpie phino, hands feet, standing, dancing, spring, Sichuan, zoo, high quality
O KLLing Al Text22mage
O Image
O
KLing Al Text2Image
KLing Al Xmdge2Video
O
K1 ing Al 2Video
Text2Image
KLing Al
KLing Al Vidge
Text2Image
Figure 10: Detailed pipeline for building B'Bench based large-scale pretrained models, includ- ing Open-CharGPT 4o and KLing AI.
B.2 DATA DETAILS
The detailed process for constructing A'Bench outlined Fig. 10. We initially provide GPT- 4o with template that clearly specifies the demand to generate 'anthropomorphized" images. The images were required to be cute, with arms and legs, standing, dancing, and of high quality. To allow for variety of image outputs, we left the fields for 'object', 'season', 'province", and 'specific
19

---

Published as a conference paper at ICLR 2025
Video Frames
More cases
Watereolar Palating
Cyherpunk Style
2
Ukiyo-F
Pixel Art
Black&Whic White
Figure 11: More styles in A Bench.
location' empty. For the key factor influencing diversity and relevance, i.e., 'object', we provide selectable range, such as everyday items, furniture, fruits, and natural creatures. To help GPT-4o better understand our intent, we additionally provide two examples, where the prompts had already been proven to generate satisfactory images by text-to-image module of KLing AI. Thanks to the text understanding and generation capabilities of GPT-4o, collect 500 prompts for image generation. We then fed these 500 prompts into the text-to-image module of Keling AI, obtaining corresponding anthropomorphic characters images. Based these images, further generate videos of them dancing using the image-to-video module of Keling AI. this way, we collect 500 pairs of images and videos of anthropomorphic characters, forming A²Bench.
Moreover, we add style trigger words such as "Watercolor Painting", "Cyberpunk Style", "Yan Gogh", "Ukiyo-E", "Pixel Arr" and on. The results are presented Figure 11, which further enhances the diversity and complexity of A²Bench.
Since most current animation methods Wang al. (2024b); Hu al. (2023); Zhang al. (2024) take pose image sequence as motion source, we also provide our A with additional pose images. To achieve this, we employ DWPose Yang al. (2023) to extract pose sequences from the videos. However, since DWPose is trained on human data, it does not accurately extract every pose the dancing video of the anthropomorphic character, so after extraction, we manually screen 100 videos with accurate poses, and view them as test videos for calculating quantitative metrics. Fig. displays several examples, which include anthropomorphic characters of plants, animals, food, furniture, etc. For images and videos where pose extraction is not feasible, we take them as key sources of reference images in our qualitative demonstrations. This will inspire the community to animate wider range of interesting cases. We also anticipate that these data could serve as an important resource for future pose extraction algorithms tailored to anthropomorphic datasets, making them accessible for broader use.
20

---

Published as a conference paper at ICLR 2025
Ref. Image
Ref. Pose
ControlNeXt
MimicMotion
Moore-AA
MusePose
Unianimate
Ours
Figure 12: Visualization of cases the user study
C
USER STUDY
In Fig. 12, we present examples shown to participants for evaluation in our user study. To obtain genuine feedback reflective of practical applications, the ten participants in iment come from diverse academic backgrounds. Since many of them do not major in computer vision, we provide detailed explanations for each question assist their judgments.
study exper-
Identity Preservation: By comparing the reference image with the two generated videos by different methods, determine which video's 's character more closely resembles the character in the image.
21

---

Published as a conference paper at ICLR 2025
Temporal Consistency: Evaluate the motion changes of the character within the video and compare which video exhibits more coherent movement.
Visual Quality: Compared to the previous two questions, this one involves more subjective judgment. Participants should assess the videos comprehensively based on visual content (c.g., flashes, distortions, afterimages). motion effects (c.g., smoothness, physical logic), and overall plausibility.
D ADDITIONAL EXPERIMENTAL RESULTS
D.1 MORE QUALITATIVE RESULTS
In the main paper, we present qualitative comparison results between our method and the state-of- the-art (SOTA) methods under a cross-driven setting human-like character, where our approach demonstrates outstanding performance. Considering that the other methods are primarily self-driven and trained on human characters, making them more suitable for inference in such settings, we additionally provide comparison results under a seff-reconstruction setting on Tiktok and Abench. As shown in Fig. 17, when there appreciably difference between the reference pose and the reference image, the GAN-based LIA Wang et al. (2022) produces noticeable artifacts. Thanks to the powerful generative capabilities of diffusion models, diffusion- based models generate higher-quality results. However, MusePose Tong et al. (2024) and MimicMotion Zhang et al. (2024) generate awkward arms and blurry hands, respectively, while ControlNeXt Peng et al. (2024) synthesizes incorrect movements. Only Unianimate Wang et al. (2024b) can obtain results comparable to ours. Yet, when the reference image is a non-human character, even in a self-driven setting with the same training strategy Unianimate, their results still show distorted heads. Fig. 18 provides results of more comparison results, including MRAA Siarohin et al. (2021a), MagicAnimate Xu et al. (2023a) and Anoore-nmimaeennonne Corporation (2024). In contrast, our method consistently generates satisfactory results for both human and anthropomorphic characters, demonstrating its ability to drive character and highlighting strong generalization and robustness.
D.2 MORE QUANTITATIVE RESULTS
Tab. 10 and Tab. 11 presents the quantitative results on TikTok Jafarian & Park (2021) and Fash- ion Zablotskaia et al. (2019a) dataset, which suggests the superiority of methods over the comparison SOTA methods. Only Unianimate achieves comparable performance; however, our method is appli- cable wider range of characters and various unaligned pose inputs, demonstrated in Tab. 1. This addresses the main issue that this paper aims to solve: developing a universal character image animation model.
D.3 ROBUSTNESS
Our method demonstrates robustness to both input X character and pose variations. On the one hand, shown in Fig. our approach successfully handles inputs from diverse subjects, including characters vastly different from humans, such as those without limbs, as well as game characters those generated by other models. Despite these variations, satisfactory results without crashing, showcasing its robustness to the input reference images. On the other hand, as illustrated in Fig. 13, even when the pose images exhibit body part omissions (highlighted by the red circles), our method correctly interprets the intended motion and generates coherent results for the reference images. This highlights the robustness of our approach different pose images.
method consistently produces
D.4 A
Difficulty Level. We add the difficulty level split for Animate-2 X. As shown in Figure 14, categorize the videos in A'Bench into three difficulty levels: Level 1, Level 2, and Level 3. The classification is based
their appearance characteristics. First, classify characters that have body shapes and other appearance features similar to humans, as shown in the first row of Figure 14, into the easiest, Level 1 category. These characters are generally simpler to drive, produce fewer
22

---

Published as a conference paper at ICLR 2025
Pose
Results
Ref.
Pose
Results
Ref.
Pose
Results
Ref.
Pose
Results
Ref.
Figure 13; Visualization of the robustness of Animate X.
artifacts, and have better motion consistency. In contrast, characters that maintain more distinct structural features from humans, such as dragons and ducks in the third row of Figure 14, are classi- fied into the most difficult Level 3 category. These characters often preserve their original structures (e.g., duck's webbed feet and wings), which makes balancing identity preservation and motion consistency more challenging. To ensure identity preservation, the consistency of motion may be compromised, and vice versa. Additionally, images involving interactions between characters, ob- jects, environments, and backgrounds are also placed in Level 3, as they increase the difficulty for the model to distinguish the parts that need to be driven from those that do not. Videos in between these two categories, like those in the second row of Figure 14, are classified as Level 2. These characters often strike good balance between anthropomorphism and their original form, making
23

---

Published as a conference paper at ICLR 2025
Level I
Level
Level 3
Figure 14: Difficulty levels in A'Bench.
Model-Level
PSNR
SSIM
LI
LPIPS
FID
FID-VID
FVD 1
Unimatee-Xlevell
13.96
0.461
9.67E-05
0.418
24.24
31.37
681.53
Animate-X-level
13.74
0.457
9.82E-05
0.429
26.12
32.19
693.63
Animate-X-levelh
13.17
0.442
1.11E-04
0.437
27.34
35.64
721.41
UniAnimatc-levell
11.93
0.413
1.14E-04
0.521
42.39
52.14
1120.45
2niAnimate-eeve222
11.89
0.408
1.20E-04
0.526
46.27
58.53
1147.34
IniA
10.91
0.379
1.35E-04
0.549
56.58
65.39
1204.53
Table 5:
User study results.
Method
PSNR* f
SSIM
LI
LPIPS 4
FID-VID
FVD
w/o
13.30
0.433
1.35E-04
0,454
32.56
64.31
893.31
w/o LQ
13.48
0.445
1.76E-04
0.454
28.24
42.74
754.37
w/o DQ
13.39
0.445
1.01-00
0.456
30.33
62.34
913.33
PA
13.25
0,436
1.11E-04
0.464
27.63
46.54
785.36
KV Q
13.34
0.443
1.17E-04
0.459
26.75
42.14
785.69
w/o EPI
12.63
0.403
1880-04
0,509
42.17
58.17
948.25
w/o Add
13.28
0.442
1.56E-04
0,459
34.24
52.94
804.37
w/o Drop
13.36
0.441
1.94E-04
0,458
26.65
44.55
764.52
w/o BS
13.27
0.443
1,08E-04
0.461
29.60
56.56
850.17
w/o NF
13.41
0,446
1.82E-04
0,455
29.21
56.48
878.11
w/o AL
13.04
0.429
1.04E-04
0.474
27.17
33.97
765.69
w/o Rescalings
13.23
0.438
1.21E-04
0.464
27.64
35.95
721.11
w/o Realign
12.27
0.433
1.17E-04
0.434
34.60
49.33
860.25
Animate-X
13.60
0.452
1.02E-04
0.430
26.11
32.23
703.87
Table 6: Quantitative results of ablation study.
them casier to animate with better motion consistency than Level 3 characters and more interesting results than Level characters. We evaluate the results of Animate and UniAnimate for each subset. As shown in Tab. as the difficulty increases, each evaluation result shows decline.
Motivation of T21+12V for Bench.. The choice to use T2I models stems from a clear need: cur- rent T2V models often struggle with imaginative and logically complex inputs, such "personified refrigerators" or "human-like bees". T2I models offer strict logic and imagination these scenar- ios, allowing generate reasonable cartoon characters the ground-truth. To prove this point, as
24

---

Published as a conference paper at ICLR 2025
shown in Table 7, assess the semantic accuracy of Benehh using CLIP scores, which are com- monly used to evaluate whether the semantic logic of images and text is strictly aligned (i... Does the generated Mhuman-like bee" maintain the visual essence of bee while seamlessly incorporating human-like features, such as hands and feet?). We also add other metrics from Vßench Huang et al. (2024), such Background Consistency, Motion Smoothness, Aesthetic Quality and Image Quality, to assess the spatial and temporal consistency of the videos in A'Bench. For comparison, we also evaluate the publicly available TikTok and Fashion datasets using the same metric. As shown in Table 7, AeBench achieves the highest level of strict logical alignment. A'Bench outperforms TikTok dataset in all aspects and achieve comparable scores to Fashion dataset, where both TikTok and Fashion are collected from real-world scenarios. It demonstrates that the video generated by our method has the same level of spatial and temporal consistency as the real videos.
Furthermore, we input the images from A'Bench into a multimodal large language model (MLLM) with logical reasoning, such as QWen Bai al. (2023), to conduct a logical analysis of the visual outputs generated by the T2I model. The results, shown in Figure 15, reveal that the image descriptions answered by the MLLM closely aligns with our input prompts, which verifies again that the data in A'Bench maintains strict logic.
Table 7: Quantitative results of different benchmarks. The best and second results for each column are bold and underlined, respectively.
CL.I
Background Consistency
Motion Smoothness
Aesthetic Quality
Image Quality
Benchmark
Score
TikTok
26.92
94.10 %
99.05 %
55.14 %
62.54 %
Fashion
20.18
98.25 %
99.45 %
49.62 %
49.96 %
A'Bench
33.24
96.66 %
99.39 %
69.86 %
69.32 %
antrrppmerpii levdy play, Eaiiigg
13N wnhrppesppkk www.diiggg.ooooo daaciay
A www.rrerrpeeertlicommmmm saceeekkrreeee bi
dasday
The retjetia tee Inape bo iime pa
The chatatte via welky
Thinng
SIL SEC
Figure 15: Prompts, generated images by T2I in Be and logical answers from QWen.
D.5 MORE ABLATION STUDY
In the main paper, we present the results of the primary ablation experiments for IPI and EPI. In this section, we supplement those results with additional ablation experiments to further demonstrate the contribution of each individual module.
Ablation on Implicit Pose Indicator. For more detailed analysis about the structure of IPI, we set up several variants: (1) remove IPI: w/o IPI. (2) remove Iearnable query: w/o LQ. (3) remove DWPose query: w/o DQ. (4) set IPI and spatial Attention to Parallel: PA. (5) set CLIP features as Q and DWPose as K, V in IPI: KV Q. The quantitative results are shown in Tab. 6. It can be seen that removing the entire IPI presents the worst performance. By modifying the IPI module, although it improves on the w/o IPI, it still falls short of the final result of An Animate-2 X, which suggests that our current IPI structure is the most reasonable and achieves the best performance.
Since IPT embedded An mate-X in the form of residual connection, i.e., x = x (T)) also explore the impact of the weight a of IPI on performance as illustrated in Fig. 16, as increases from to 1, all metrics show a stable improvement despite some fluctuations. The best performance is achieved when is set to 1, we empirically set a to I in the final configuration.
Ablation on Explicit Pose Indicator. We conduct more detailed ablation experiments for different pairs of pose transformations by (1) removing the entire EPI: w/o EPI; (3)&(3) removing adding and dropping parts; canceling the change of the length of (4) body and should: w/o BS; (5) neck and face: w/o NF; (6) arm and leg: w/o AL; (7) removing all rescaling process: w/o Rescalings; (8)
25

---

Published as a conference paper at ICLR 2025
1.0
00
100
00
4.4
087
0.44
0.8
067
0.r
of*
Normalized Values
0.6
9.59
Rodd
0.4
0.24
20
233
011
PVD-3DInccetiion
LPIPS
0.08
PSNR
0.0
L1
4
00
0.0
0.1
0.2
0.3
0.4
0.6
0.7
0.9
1.0
Wights for IPI
Figure 16: Ablation study on the weight a of Implicit Pose Indicator. To better visualize the impact of a on performance, we normalize all the values to the range of 0 to 1.
remove another person pose alignment: w/o Realign. From the results displayed in Tab. 6, we found that each pose transformation contributes compared to w/o EPI, with aligned transformations with another person's s pose contributing the most. It suggests that maintaining the overall integrity of the pose while allowing for some variations the most important factor, and EPI also learns the overall integrity of the pose. The final result indicates that all the transformations together achieve the best performance.
To explore the effect of different probabilities À of using pose transformation for EPI on the model performance, we set as 100%, 98%, 95%, 90% and 80% for the ablation experiments on two datasets. The results presented in Tab. suggest that high À performs better A'Bench, i.e., it performs better when the reference image and pose image are not aligned, but harms performance on the Tik dataset, i.e., when the reference image and pose image a relatively low X, e.g., 90%, would be in this case perform better. It is reasonable that in the case of strict alignment, we expect the pose to provide strictly accurate motion source, and thus need to reduce the percentage À of pose transformation. However, in the non-strictly aligned case, we expect the pose image to provide an approximate motion trend, so we need to increase X.
strictly aligned. In contrast,
Since the anchor poses are chosen from the entire training set, we further conduct the statistical analysis for rescaling ratio. First, we randomly sample driven pose IP and then traverse the entire pose pool, treating each pose in the pool as an anchor pose to calculate the rescaling ratio. We repeat this process 10 times. Finally, we divide the range from 0.001 to 10 into 10 intervals, counting the proportion of rescaling ratios that fell within each interval. We analyze the proportions of other important parts like shoulder length, body length, upper arm length, lower arm length, upper leg length, lower leg length. As shown in Tab. 8, the overall distribution covers wide range (from 0.001 to 10.0), which allows the model to learn poses of various characters, encompassing non- human subjects.
E DISCUSSION
E. LIMITATION AND FUTURE WORK
Although our method has made remarkable progress, it still has certain limitations. Firstly, its ability model hands and faces remains insufficient, limitation commonly faced by most currett
26

---

Published as a conference paper at ICLR 2025
Interval
Shoulder Length
Body Longth
Upper Length
Length
Upper Log Length
Log Longth
0.001 0.1)
0.14%
0.05%
0..1
1.52%
5.73%
4.04%
3.22%
0.59%
4.60%
0..3
12.21%
18.57%
15.28%
7.63%
4.26%
5.65%
00.5.
15.33%
16,93%
12.97%
7.54%
12.02%
9.61%
10.7. 1.0)
20.07%
18.48%
17.15%
11.35%
24.86%
19.53%
[100 1.5)
22.09%
18.63%
17.50%
15.38%
27.90%
24.89%
[15.2]
10.07%
8,34%
7.93%
11.73%
14.31%
14.47%
[..03300
6.52%
16. 16%
33.0
6.28%
18.40%
2.73%
4.30%
16.0,
2.43%
0.37%
6.37%
8.47%
1.45%
0.85%
Table 8: Statistical analysis for rescaling ratio.
AeBenh
TikTok Jafarian & Park (2021)
Method
SSIM
FID4
FID-VID
FVD
SSIM
FID.
FID-VID
FVD4
100% %
0.452
26.11
32.23
703.87
0.802
55.26
17.47
138.36
98 %
0.448
26.93
37.67
775.24
0.797
55.81
16.28
129.48
95%
0.447
27.46
39.21
785.55
0.804
52.72
14.61
124.92
90%
0.444
27.15
38.03
775.38
0.806
52.81
14.82
139.01
80%
0.442
29.13
47.93
803.97
0.802
54.51
14.42
133.78
Table 9: Quantitative results for different probabilities of using pose transformation.
Method
LI
PSNR f
PSNR*
SSIM
LPIPS 1
FVD L
FOMM Siarohín et al. (2019a) (NesiPPS19)
3.61E-04
17.26
0.648
0.335
405.27
MRAA Siarohin et al. (2021a) (CVPR21)
3.21E-04
18.14
0.672
0.296
284.82
TPS Zhao & Zhang (2022a) (CVPR22)
3.23E-04
18.32
0.673
0.299
306.17
DreamPose Karras
(2023) (осе23)
6.88E-04
28.11
12.82
0.511
0.442
$51.02
DisCo Wang et al. (2024a) (CVP234)
3.78E-04
29.03
16.55
0.668
0.292
292.80
MugicAnimate Xu al. (2023a) (CVPR24)
3.13E-04
29.16
0.714
0.239
179.07
Animate Anyone Hu al. (2023) (CVPR34)
29.56
0.718
0.285
171.90
Champ Zhu et al. (2024) (#CCV24)
2.94E-04
29.91
0,802
0.234
160.82
Unianimate Wang al. (2024b) (AxXiv24)
2.66E-04
30.77
20.58
0.811
0.231
148.06
MusePose Tong et al. (2024) (ArXiv24)
3.86E-04
17.67
0.744
0.297
215.72
MimicMotion Zhang et al. (2024) (AXXW24)
5,85E-04
14.44
0,601
0,414
232.95
ControlNeXt Peng et al. (2024) (Arxiv24)
6,20E-04
0.615
0.416
326.57
Animate-X
2.70E-04
30.78
20.77
0.806
0.232
139.01
Table 10: Quantitative comparisons with existing methods on TikTok dataset.
Method
PSNR
PSNR* f
SSIM
LPIPS J
FVD J
MRAA Siarohin al. (2021a) (CVPR21)
0.749
0.212
253.6
Zhao Zhang (2022a) (CVP922)
0.746
0.213
247.5
DPIN Zhang et al. (2022a) (CVPR22)
24.00
0.907
0.060
215.1
NTED Ren et al. (2022) (CVPR22)
22.03
0.890
0.073
278.9
PIDM Bhunia et (2023) (CVPR23)
0.713
0.288
119..4
DBMM Yu et al. (2023) (CCV23)
24.07
0.918
0.048
168.3
DreamPose Karras et (2023) (CCV2))
0.885
0.068
238.7
DreamPose w/o Finetune Karras et al. (2023) acev25)
34.75
0.879
0.111
279.6
Animate Anyone Hu et al. (2023) (CVPR34)
38.49
0.931
0.044
81.6
Unianimate Wang et al. (2024b) (AXX024)
37.92
27.56
0.940
0.031
MimicMotion Zhang et al. (2024) (Axi24))
27.06
0.928
0.036
118.48
Animate-X
36.73
27.78
0.940
0.030
79.4
Table 11: Quantitative comparisons with existing methods the Fashion dataset. "w/o Finetune" represents the method without additional finetuning on the fashion dataset.
27

---

Published as a conference paper at ICLR 2025
TikTok
A Bench
Ref. Image
Ref. Pose
LIA
MusePase
Unianimate
MimicMotion
ControlNeett
Ours
Figure 17: Visualization comparison on TikTok dataset and A Bench.
(CVPR 22)
MagicArimate (CVPR 24)
Moore ArimatokAyone (CVPR 24)
Ours
Figure 18: Comparison with more SOTAs A3Bench.
generative models. While our IPI leverages CLIP features extract implicit information such as motion patterns from the driving video, mitigating the reliance on potentially inaccurate hand and face detection by DWPose, there is still a gap between our results and the desired realism. Secondly, due to the multiple denoising steps in the diffusion process, even though we replace the transformer with a more efficient Mamba model for temporal modeling, Animate X still cannot achieve real- time animation. In future work, aim to address these two limitations. Additionally, we will focus on studying interactions between the character and the surrounding environment, such as the background, as a key task to resolve. As for A 4'Bench, creating 3D models and rendering them with predefined actions using tools like Blender and Maya a superior approach for developing a character benchmark, which is also part of our future work.
28

---

Published as a conference paper at ICLR 2025
E.2 ETHICAL CONSIDERATIONS
Our approach focuses on generating high-quality character animation videos, which can be applied in diverse fields such as gaming, virtual reality, and cinematic production. By providing body move- ment, our method enables animators to create more lifelike and dynamic characters. However, the potential misuse of this technology, particularly creating misleading or harmful content on digital platforms, is a concern. While greatly progress has been made in detecting manipulated anima- tions Boulkenafet et (2015); Wang al. (2020); Yu et al. (2020), challenges remain in accurately identifying increasingly sophisticated forgeries. We believe that our animation results can contribute to the development of better detection techniques, ensuring the responsible use of animation tech- nology across different domains.
29


Reviews:
[{"review_id": "SClEUBMPLQ", "reviewer": "Reviewer_aHUH", "summary": "This work presents an animation framework capable of animating anthropomorphic characters, along with an accompanying benchmark for animated anthropomorphic characters. Specifically, the framework introduces an Implicit Pose Indicator and an Explicit Pose Indicator to provide rich pose guidance.", "questions": "Please address the concerns in the weakness.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The visual results of Animate-X demonstrate notable improvements across various characters compared to existing animation methods.\n2. Comprehensive experiments and ablation studies are presented.", "weaknesses": "1. No video samples from A2Bench are provided; only selected frames are shown in the paper. Given that the generated videos still struggle with maintaining strict logic and good spatial and temporal consistency, I question the rationale for using T2I + I2V to generate benchmark videos. Additionally, the benchmark lacks detailed information, such as video length and frame rate. Were any additional motion prompts used to generate videos from images? If so, what is their diversity and complexity?\n2. The necessity of a pose pool and the selection of an anchor pose image need clarification. What operations are involved in the \"align\" process, specifically regarding translation and rescaling? Why not use random translation and rescaling instead of relying on an anchor pose image?\n3. The effectiveness of the Implicit Pose Indicator (IPI) is also in question. The motivation for the IPI is that sparse keypoints lack image-level details, while IPI aims to retrieve richer information. However, Tables 7 and 8 indicate that Animate-X achieves comparable performance to Animate-Anyone and UniAnimate on human videos. This suggests that the IPI does not provide any benefits for human animation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pQl1ffz15F", "reviewer": "Reviewer_mbHE", "summary": "This paper focuses on the animation of non-human characters, addressing two main issues:\n1)Sole pose skeletons lack image-level details.\n2)Pose alignment in the self-driven reconstruction training strategy.\nTo resolve these issues, the paper introduces a Pose Indicator, comprising an Implicit Pose Indicator and an Explicit Pose Indicator. Experimental results demonstrate that the proposed Animate-X achieves effective performance in character animation.", "questions": "See Weakness. If the authors can address all my concerns, I am willing to raise the score.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1.The authors introduce  A2Bench, which is helpful for the evaluation of character animation.\n2.Both qualitative and quantitative experiments are conducted to evaluate the performance of the proposed method.", "weaknesses": "1.Some parts of the writing can be quite confusing, words and sentences are bad orgnized. For example, in P5 L260, what exactly is in the pose pool? And how is it aligned with the reference?\n2.The dataset includes 9,000 independently collected videos. Could you analyze these videos, and did other baselines use the same data for training? If not, could this lead to an unfair comparison?\n3.The authors first identify the weaknesses of previous methods as a conflict between identity preservation and pose control. They further expand on this point by highlighting two specific limitations: the lack of image-level details in sole pose skeletons and pose alignment within the self-driven reconstruction training strategy. However, while the authors clearly state that differences in appearance between characters and humans can negatively impact animation, learning image-level details seems to contradict their viewpoint  \"sole pose skeletons lack image-level details\", making this contribution appear more like a forced addition.\n4.Additionally, the visualization in Figure 7 provided by the authors also supports w3. The inclusion or exclusion of the IPI appears to have minimal impact on the motion of the Ref image, and with IPI, part of the foot in the Ref image is even missing. This raises doubts about the effectiveness of the IPI module and seems inconsistent with the authors' stated motivation.\n5.Pose augmentation has already been widely explored in existing methods, such as MimicMotion, which makes the innovation in this paper insufficient.\n6.This paper lacks comparisons with similar methods, such as MimicMotion, which makes the experimental results less convincing.\n[1]MimicMotion: High-Quality Human Motion Video Generation with Confidence-aware Pose Guidance", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eKL6l2G2Ks", "reviewer": "Reviewer_feUz", "summary": "This paper highlights that character animation models trained exclusively on human-only datasets struggle to learn motion patterns from driving videos, often leading to overfitting on the driving pose and poor generalization to anthropomorphic characters.\n\nTo address this issue, the authors propose a novel character animation framework called Animate-X, which incorporates two Pose Indicators. The Implicit Pose Indicator extracts motion and integrates it with CLIP features, while the Explicit Pose Indicator supports an augmentation pipeline during training that encourages the model to learn motion from misaligned pose sequences.\n\nAdditionally, a new benchmark is established for evaluating anthropomorphic characters. Experiments across multiple datasets demonstrate the effectiveness of the proposed method for animating anthropomorphic characters.", "questions": "- Could the authors provide more details on the construction of the pose pool and alignment pool, such as the pool sizes and how poses are selected from the training set?\n    \n- Comparing the results in Table 4 and Table 1, Animate-X outperforms the baselines even without pose augmentation (EPI). Could the authors provide a deeper analysis of why the Implicit Pose Indicator (IPI), with fewer parameters, outperforms the reference network?\n    \n- What happens if the reference pose differs significantly from the candidates in the pose pool and alignment pool? The authors should provide a robustness analysis for this scenario and consider adding a difficulty level split for A2Bench.\n    \n- Could aligning the driving pose to a \"standard\" one in the pose pool further improve generation quality?\n    \n- In the supplementary materials, the authors show results in various styles, yet most styles in A2Bench are in \"3D render style.\" Would it be possible to add a \"style trigger word\" in the prompt template to diversify and strengthen the benchmark?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper introduces a new augmentation method that enhances pose robustness for character animation techniques.\n    \n- A novel module is proposed to integrate the driving pose with the reference image without relying on a reference network.\n    \n- A new benchmark is established for evaluating anthropomorphic characters.\n    \n- The quality of animation results is good, even reference characters do not have leg or arm.", "weaknesses": "- The paper lacks a detailed analysis of the construction of the augmentation pool, making it difficult to reproduce the method.\n    \n- There is insufficient in-depth analysis of the model design, such as why the Implicit Pose Indicator (IPI) outperforms the reference network, which has more learnable parameters.\n    \n- Most styles in the A2Bench benchmark are \"3D render style\"; the benchmark should include a wider variety of visual styles.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "drcNMai9Lx", "reviewer": "Reviewer_ESK8", "summary": "This paper proposed Animate-X, a universal animation framework based on diffusion models. The key insight of this work is that existing image animation frameworks are only focused on the human images and fail to extract the movement pattern of the driving video, leading to the rigid retargeting of the driving pose to the target reference image. The authors propose two pose indicators to address this issue, which can capture comprehensive motion patterns from the driving video. The implicit pose indicator helps retrieve relevant features from the driving video, while the explicit one simulates the unaligned driving poses in the inference stage. To evaluate the approaches, the authors also propose a new benchmark which contains in-the-wild and unmatched driving sequence/reference image pairs. Experiments show that the proposed method outperforms state-of-the-art methods.", "questions": "Does this model still use any input videos in the inference stage? I am asking this question because there are no input driving videos in the “Animating anthropomorphic characters” section of the supplementary materials. Could the author explain the inference setting? If there is a corresponding driving video, it is better to also include them into the results.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The motivation of this work is clear, it comes from an in-depth analysis of the failure cases of existing works. The alignment between the driving signal and reference image is critical to the fidelity of character animation. The authors propose an effective method to tackle this problem.\n2. The experimental results, especially the video results, are reasonable and interesting. The proposed method shows state-of-the-art performance and outperforms baselines in animating in-the-wild characters. This indicates that the training data is well utilised and shows that the proposed method helps improve the generalisation ability of the animation model.\n3. The evaluation benchmark is valuable to the research community. It can help follow-up works measure their methods comprehensively.", "weaknesses": "1. The backbone of this work remains unchanged, it is quite similar to the prior works like your reference AnimateAnyone and MagicAnimate, which makes this work a straightforward extension of existing works and thus reduces the contribution of this paper.\n2. Leveraging driving videos to boost the animation performance has been already explored in a few prior works like [1]. The implicit pose indicator is also a similar design which aims to extract comprehensive motion patterns to improve the animation performance.\n3. The explicit pose indicator is a little bit confusing because I think this module is an augmentation of the driving pose sequences. Therefore, the novelty of the proposed method is not very significant. It is reasonable that the augmentation can break the strong correspondence between the driving video and motion representation. What is the advantage of this training time rescale augmentation and over the test time pose alignment? Are there any ablation studies about this? \n4. From the results of the animation of anthropomorphic characters, the example of a banana shows that although the animation result looks like a banana, the motion precision is decreased. Therefore, I think the implicit pose indicator could harm the motion precision. The authors could conduct more experiments to study this issue.\n\n[1] X-portrait: Expressive portrait animation with hierarchical motion attention", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)