FaceShot: Bring Any Character into Life
Authors: Junyao Gao
Yanan SUN
Fei Shen
Xin Jiang
Zhening Xing
Kai Chen
Cairong Zhao
Venue: ICLR 2025 Poster (2025)
Keywords: portrait animation
diffusion model
Primary area: generative models

Abstract: In this paper, we present ***FaceShot***, a novel training-free portrait animation framework designed to bring any character into life from any driven video without fine-tuning or retraining.
We achieve this by offering precise and robust reposed landmark sequences from an appearance-guided landmark matching module and a coordinate-based landmark retargeting module.
Together, these components harness the robust semantic correspondences of latent diffusion models to produce facial motion sequence across a wide range of character types.
After that, we input the landmark sequences into a pre-trained landmark-driven animation model to generate animated video.
With this powerful generalization capability, FaceShot can significantly extend the application of portrait animation by breaking the limitation of realistic portrait landmark detection for any stylized character and driven video.
Also, FaceShot is compatible with any landmark-driven animation model, significantly improving overall performance.
Extensive experiments on our newly constructed character benchmark CharacBench confirm that FaceShot consistently surpasses state-of-the-art (SOTA) approaches across any character domain.
More results are available at our project website https://faceshot2024.github.io/faceshot/.

Published as a conference paper at ICLR 2025
FACESHOT: BRING ANY CHARACTER INTO LIFE
Junyao Gao'; Yanan Sun ; Fei Shen?, Xin Jiang', Zhening Xing" Kai Chen" Cairong Zhao?
Tongji University, SShanghai AI Laboratory, Nanjing University of Science and Technology.
junyacgac, zhaocairong @tongji edu. en, {feishen, xinjiang) jnjust .nd s sunyanan, xingzhening, chenkai I @aiiap..oss org.en.
Input
Generated Frames
Input
Generated Frames
Figure I: Visualization results of our FaceShot. Given any character and any driven video, FaceShot effectively captures subtle facial expressions and generates stable animations for each character. Especially for non-human characters, such as emojis and toys, FaceShot demonstrates remarkable animation capabilities.
ABSTRACT
In this paper, we present FaceShot, a novel training-frre portrait animation frame- work designed to bring any character into life from any driven video without fine-tuning retraining. We achieve this by offering precise and robust reposed landmark sequences from an appearance-guil -ugiddd landmark matching module and a coordinate-based landmark retargeting module. Together, these components harness the robust semantic correspondences of latent diffusion models pro- duce facial motion sequence across wide range of character types. After that, we input the landmark sequences into a pre- treitrained landmark-driven animation model to generate animated video. With this powerful generalization capabil- ity, FaceShot can significantly extend the application of portrait animation by breaking the limitation of realistic portrait landmark detection for any stylized character and driven video. Also, FaceShot is compatible with any landmark- driven animation model, significantly improving overall performance. Exten- sive experiments on our newly constructed character benchmark CharacBench confirm that FaceShot consistently surpasses ttate-of-the-e (SOTA) approaches across any character domain. More results are available at our project website httpps//faacesott22224 github. io/ to/faccshot/
INTRODUCTION
"I wish my toys could talk"-many people make this wish on birthdays or at Christmas, hoping for the companionship from their "imaginary friends". Achieving this usually requires a bit of "magic".
Work done during internship in Shanghai AT Lahoratory. ccorresponding authors.

---

Published as a conference paper at ICLR 2025
seen with the talking teddy bear in Tea or the three chipmunks in Alvin and The Chipmunks" Behind these productions, making this "magic" a reality often requires specialized equipment and significant manual effort for character modeling and rigging. In this work, to bring any character into life for every person, we propose a novel, training-free portrait animation framework. As shown in Figure even for emojis and toys, which have totally different facial appearances from humans, proposed framework demonstrates remarkable performance in making these characters alive.
Portrait animation (Guo al., 2024 Ma et 2024 Xie et al. 2024 Yang al. 2024 Wei et al.) 2024 Niu et al., 2024; Wang et al. 2021] Zeng et al. 2023) has demonstrated impressive results with
the recent ad advancements in generative models. such as Generative Adversarial Networks (GANs)
(Goodfellow et al.] (2000) Donahue et al. 2016;; Odena et al., 2017 Radford 2013) and diffusion models (Rombach et al. 2022; Nichol et al. 2021 Saharia al., 2022, Ho et al. 2020) Song et al. 2021a). However, these methods depend on facial landmark recognition, and their performance are constrained by the generalization capability of facial landmark detection models (Zhou al. [0033] Yang et al. 2023 For non non-human characters, such as emojis, animals and toys, which often exhibit significanlly different facial features compared to human portraits, always resulting in landmark recognition failures due the supervised training paradigm and limited datasets. As shown in Figure 2] the unaligned target facial features for non-human character lead to disrupted animation results; the animation models even generate human mouth the wrong position for dog. In addition, Xie et al. (2024); Yang et al. (2024) indicate that these methods cannot control subtle facial motions, leading to inconsistent animation results.
To address the above propose FaceShot, a novel portrait animation framework capable of animating any character from any driven video without the need for training. As demonstrated in Figure [ FaceShot produces vivid and stable animations for various characters, particularly for non-human characters. This is achieved through three key components: (1) the appearance-guiee landmark matching module, (2) the coordinate-based landmark retargeting module, and (3) a handmark-driver animation model.
limitations,
Target
X-Portrait
Follow Your Emoji
AniPontrait
FaceShot
Figure Visual results generated from current portrait animation methods and our FaceShot. Previous methods apparently retain the target human's 's appearance. In contrast, the result of FaceShot both aligns the dog's facial features and captures the target human's expression.
In our landmark matching module, we inject appearance priors into diffusion features and leverage their strong semantic correspondences to match the landmarks. For the second component, we introduce theoretical algorithm to capture subtle facial motions between frames and generate the landmark sequence aligned with the driven video. Finally, for the third component, we input the landmark sequence of the reference character into a pre-trained landmark-driven animation model to animate the character. As shown in Figure FaceShot provides the reasonable animation result by offering the precise landmarks of non- non-human character. Furthermore, FaceShot is compatible with any landmark-driven portrait animation model as a plugin, improving their performance non-human characters, with experimental analysis in Section
Moreover, address the absence of benchmark for character animation, establish CharacBench that contains 46 diverse characters. Qualitative and quantitative evaluations on CharacBench demon- strate that FaceShot excels in animating characters, especially in non-human characters, outperforming existing portrait animation methods. Additionally, ablation studies validate the effectiveness and su- periority of our framework, providing valuable insights for the community. Furthermore, we provide the animation results of FaceShot from non-hu non-human human driven videos, bringing a potential solution to the community for open-domain portrait animation.
The main contributions of this paper are as follows:
httes://enunnkkpppcaaoo////////a the ftt_aaa___ehhcccpppns__________
2

---

Published as a conference paper at ICLR 2025
We propose FaceShot, novel portrait animation framework capable of animating any character from any driven video without the need for training.
FaceShot generates precise reposed landmark sequences for any character and any driven video, bringing a potential solution to the community for open-domain portrait animation.
FaceShot be seamlessly integrated model, further improving its performance.
plugin with any landmark-driven animation
We establish CharacBench, benchmark with diverse characters for comprehensive evalua- tion. Experiments on CharacBench show that FaceShot outperforms SOTA approaches.
2 RELATED WORK
Portrait Animation. Early portrait animation methods primarily relied on GANs (Goodfellow et al.
2020 to generate portrait animation through warping and rendering techniques (Drobyshev et al.
2022 Starohin et al.) 2019; Hong et al.
2022, Wang et al. 2021 Zhao & Zhang 2022). Recent
advancements in latent diffusion models (LDMs) (Rombach et al. 2022, Ramesh et al. 2022 Shen
al. 2023 Gao al., 2024, Shhn al.
2024b; Wang al. 2024a [hhnn et al.] 2022a Li al.
2024a Shen & Tang. 2024) have improved the quality and efficieccy of image generation. Building
on this, some methods (Xie et al. 2024; Yang et al., 2024) learn the identity-free expression by constructing paired data, demonstrating impressive performance. However, current data collection pipelines face challenges in constructing paired data for diverse domains, particularly for non-human characters, which limits the generalization ability of these methods. Additionally, other approaches (Wei et al. 2024 Niu et af.] 2024; Ma et al. 2024 Shen et al.) (0255 tend to utilize highly scalable conditions, such as facial landmarks, to enhance motion control. Naturally, these methods depend on the facial landmark recognition, which also restricts their applicability to non non-human characters. To break these limitations and bring any character into life, we focus on providing precise reposed landmark sequences within landmark-dnig -diveen portrait animation in this work.
Facial Landmark Detection Facial landmark detection aims to detect key points in given face. Traditional methods (Cootes et al. 2000; 2001; Dollár et al. 2010 Kowalski et al., 2017) often construct a shape model for each key point and perform iterative searches to match the landmarks. With the development of deep networks, some methods (Sun et al. [013 Zhou et al., 2013 Wu et al.) 2018; 2017) select a series of coarse to fine cascaded networks to perform direct regression on the landmarks. Another trend, |Huang et al. (2020); Zhou et al. (2023); Merget et al. (2018); Kumar et al. (2020) predict the heatmap of each point for indirect regression, improving the accuracy of landmark detection. Recently. Yang et al. (2023); Xu et al. (2022); Li et al. (2024b) collect larger datasets and train larger models for more generalize landmark detection. However, due to the supervised training paradigm and limited dataset, these methods are difficult perfectly detect the landmark of non-human characters. In our paper, we turn to a training-free landmark matching module through the strong semantic correspondence and the generalization in diffusion features (Tang al. 2023] Hedlin et al. 2024 Luo et al. 2024), aiming to provide precise landmarks for non-human characters.
Image Video Generation Image to video (12V) generation has gained significant attention in recent years due to its potential in various applications, such as image animation (Dai et al. 2023
Gong et al.) 2024 Ni al.) [0023] Guo al., (202)) and video synthesis (Blattmann al.] 2023b
Wang et al. 2024b Ruan et al. 6023). Since diffusion models demonstrate the powerful image
generation capabilities, Zhang et al. (2024); [Shi et al. (2024); Xing et al. (l.23) Ma et al. (0024)
achieve image animation by inserting temporal layers into pre-trdi 2D UNet and fine-tuning
with video data. Furthermore, some methods (Zhang et al. [2023a] Blattmann et al. (023a) have
constructed their own 12V models and performed full training with large amounts of high-quality
data, demonstrating strong competitiveness. In our work, we utilize MOFA- Video (Niu et al. 2024)
our base animation model.
3 METHOD
The framework of FaceShot is depicted in Figure 3 of diffusion models in Section[..] We then explain the three key components of our framework in detail in Section sppearance-guider landmark matching, coordinate-batt based landmark retargeting, and character animation model.
We first introduce the foundational concepts
3

---

Published as a conference paper at ICLR 2025
Appearance
cecraaannneegdiddd guided Landmark Matching
Driving Video
Driving Sequencee
Reference Pipeline
Torget Pipeline
Landmark Matching
inyer time stop
DDΣM Inversion
& fur
Arimation Model
briving Pipaline
Target Images
Landmark Retorgeting
1-th
Reference
Appearance Guidonce
Coordinate- Landmark Reforgeting
Reference
Sequence
Figure 3: The FaceShot framework first generates precise facial landmarks for the reference character with appearance guidance. Next, a moordinate-based landmark retargeting module generates the landmark sequence based on driving video. Finally, this sequence is fed into animation model to animate character.
3.1 PRELIMINARY
In FaceShot, we utilize Stable Diffusion (SD) (Rombach et al. 2022) as the base model for land- mark matching, which consists of Variational Auto-Encoder (VAE) (Kingma (013), CLIP text encoder (Radford et al. (021). and a denoising U-Net (Ronneberger et al. 2015). Compared to pixel-based diffusion models, SD uses the VAE encoder encode the input image into latent representation z = {x)) The VAE decoder 'D then reconstructs the image by decoding the latent representation: x (z)).
To train the denoising U-Net eg, the objective typically minimizes the Mean Square Error (MSE) loss each time step I, as follows:
L = N(,,), /loooo/.....lll//.....o. c
(1)
where z'= Vā'zz + VI- is the noisy latent at time step t, a² := II a=1 a and a := 1 8
with 3r represent the forward process variances. e' denotes the added Gaussian noise, and the text condition, processed by the U-Net's cross-attention module.
Moreover, the Denoising Diffusion Implicit Model (Song al. 2021a) (DDIM) enables the inversion of the latent variable 2o to 21 in deterministic manner. The formula is as follows:
-V
aati Eg (z- , I,,,d,
where e' represents the additional image prompt. our implementation, we utilize the latent space features at 1-th time step and I-th layer of U-Net from DDIM inversion for landmark matching.
3.2 FACESHOT: BRING ANY CHARACTER INTO LIFE
Reviewing the remarkable variance in perfor- mance caused by inaccurate or accurate facial landmarks Figure 2, A well-generalized land- mark detector is necessary for Landmark-driver portrait animation model to bring any character into life. Prior landmark detectors (Yang al.)
2023 Xu et al.) 2022: Zhou et al. 2023) have
either curated more diverse public datasets or introduced new loss functions during training
to improve the generalization of landmark de- tection. However, within supervised training paradigm, these detectors struggle to generalize to non-human characters, resulting inaccurate re- sults for portrait animation. To address this, we propose appearance-guides landmark match- ing module that generalizes to any character to generate precise landmarks. In addition, to cap- ture the subtle movements in driving videos, we offer a coordinate- based landmark retargeting module. Finally, character animation model is employed our base model animate reference characters.
Reference image (a) w/o appearance (b) w/ appearance
Figure 4: Visualizations of point matching with (w/, highlighted in red box) or without (w/o) appearance guidance using an anime diffusion model.
4

---

Published as a conference paper at ICLR 2025
Appearance-guided Landmark Matching, Tang al. )023); Hedlin et al. 2024): Luo al. (2024)
demonstrate the strong semantic correspondence and generalization between diffusion features, where simple feature matching can map the point p' on the reference image Lef to semantic similar point on the target image har- However, appearance discrepancies across different domains often result in mismatches, shown in Figure (a), where the points the left eye and right incorrectly matched. A natural solution is to inject prior appearance knowledge through inference using domain-specifte diffusion model. As shown in Figure (b), points are correctly matched when inferred on an anime diffusion model.
Since fine-tuning diffusion model for each reference image is costly, inspired by -d-Atpteer (Ye et al., (023), we utilize image prompts to provide appearance guidance. Specifically, we treat the reference image Tref and the target image tar as image prompts, denoted as éyef and cear- We then apply the DDIM inversion process obtain deterministic diffusion features fref and ftar from fref and tar at time step t and the 1-th layer of the U-Net:
ftar = Filca(ziar t 1, Gre))); fref Fi(co(=fe) 1-1 1, Garl),
where a/ar and 245/ are iteratively sampled by Eq. 2 from zfar = Eltar) and zles (Ires) using the text prompt e =a photo of a face" and the image prompts Cres and carr respectively. Fj denotes the function that extracts the output feature at the 1-th layer the U-Net.
After obtaining the diffusion features fref E R1xC BIxCxxhixa, xhixw; and fear E II we upsample them to Hee E I xlICxxXXW and flar E [x1xCIxHX to match the resolutions of lref and Lar- To improve performance and stability, construct the average feature of the i-th landmark point plar from target images to match the corresponding point Pres in the reference image, as follows:
k
Prej = arg min deas
Σ
Plar fres
k
Praf
J=1
where fp( E R1xC; represents the diffuusion feature vector at point p, deos denotes the cosine distance and Pref refers to points in the reference feature map. Finally, denote the matched landmark points of the reference image as L (pref number facial landmarks.
......,N, I,..,, where N represents the
Appearance Gallery. We introduce ap- pearance gallery G [Ge, Gm. Gnz Geb, Gol, which is collection of five prior compo- nents eyes, mouth, nose, eyebrows, and face boundary-across various domains, with each domain containing k images. For a reference image Ire/. reconstruct the target image heer = [G G G G Gjjl by matching Tref with the closest domain the appearance gallery G. thereby explicitly reducing the ap- pearance discrepancy between the reference and target images, shown in Figure 5
Appearance Gallery mouth nose
face boundary
eyes
eyebrows
Appearance
Coordinate-based Landmark Retargeting. Currently, Niu al. (2024); Wei al. (2024); Ma et al. (2024) utilize 3D Morphable Models (3DMM) (Booth et al., (016) generate the landmark sequence of the reference image by applying 3D face parameters. However, 3DMM- based methods often struggle to generalize to non-hu non-human character faces due to the limited number of high-quality 3D data and their inability to capture subtle expression movements (Retsinas) et al. 2024). As shown in Figure 6, the head shapes of the 3D face are not well aligned with the input images, and subtle movements, such as eye closures, absent in the i-th frame. Therefore,
Matching
Reference
Target Images
Figure 5: Illustration of our appearance gallery. We output the closest domains for each reference image to reduce the appearance discrepancy.
5

---

Published as a conference paper at ICLR 2025
fail
fail
Reference
Driving
MOFA Video
X-Portrait
Face. Vid2Vid
FADM
Follow Emoji
MegActor
FaceShot
Portrait
Figure 7: Qualitative comparison with SOTA portrait animation methods. Slash boxes represent that the method fail generate animation for this character.
we propose coordinate-based landmark retargeting module designed to generate a retargeted land- mark sequence Lref that can stably capture the subtle movements from driving video based on transformations in rectangular coordinate systems.
Our module consists of two stages, which respectively retarget the global motion of entire face and lo- cal motion of different facial parts from driving sequence image. In the first stage, the global motion from the 0-th to the m-th driving frame is defined the trans- lation AO = O O rotation Alari the corresponding global rectangular coordinate systems (O%re 0% and (O (. Specifically, the global rectangular coordinate system con- structed by the origin O and angle 0, which are calculated from the endpoints of face boundary. Afterward, the global coordinate system of reference image m-th frame can be calculated from those of O-th frame's 's as follows:
Ground truth
reference
3DMM
dri and 0e:: 0ur of
Reference
3D face
0-th frame
i-th frame
Figure 6: Visualizations of 3D face and retargeting results using 3DMM.
Om OPeg AOdris 0ne Opef + Adars.
(5)
Finally, we transfer the coordinates of the landmark points from (OP OPer) (Ome,. 0m))) repre- senting the global motion of the entire reference face.
In stage two, the local motion involves two processes: the relative motion and point motion, applied to five facial parts: eyes, mouth, nose, eyebrows, and face boundary. The relative motion is similar the global motion, but the part-specific coordinate systems are calculated from the endpoints of each part. Furthermore, to constrain each part within reasonable facial range, we scale A0 as
beef AOari where b represents the distance from the origin the boundary of each part. Next, bae

---

Published as a conference paper at ICLR 2025
model the point motion follows:
Pres
Pari [0]
0,4 Pref
[0],
Pari
Pres
0,2
(6)
Paril (0)
0.2
Pari
0.
where Pref and m. Pari represent the coordinates in the part-specific coordinate system for the m-th frame and i-th point. This simple yet effective design enables us capture both global and local, obvious and subtle motions into landmark sequence Lref (Liee ,,.,,, M} for any character, where M represents the number of video frames.
Character Animation Model. After obtaining the reference landmark sequence Lrefs it can be applied to any landmark-driven animation model animate the character portrait. Specifically, L treated as an additional condition for the U-Net, either injected via ControlNet-lie structure (Niu al. 2024) or incorporated directly into the latent space (Hul 2024 Wei et al. 2024). This enables the model to precisely track the motion encoded in the landmark sequence while preserving the
character's visual identity. Moreover, this flexible condition can be scamlessly extended to various architectures, enhancing scalability across diverse animation tasks.
4 EXPERIMENTS
4.1
IMPLEMENT DETAIL
In this work, employ MOFA-Vider (Niu al. 2024). Stable Video Diffusion (Blattmann et al. 2023a) (SVD)-based landmark driven animation model, as our base character animation model. For appearance-guide guided landmark matching, utilize Stable Diffusion v1.5 along with the pre-trained weights of IP-Adapter (Ye et al. (023) to extract diffusion features from the images. Specifically, we set the time step 301, U-Net layer I 6, and the number of target images A 10. Additionally, following MOFA- Video, we use N = 68 keypoints (Sagonas et al., (016)) as facial landmarks and M frames for animation. More details are shown Appendix.
Evaluation Metrics. Following Xie et al. (2024); Ma et al. (2024), we employ four met- rics to evaluate identity similarity, high- and low-level image quality and expression accu- racy. Specifically, we utilize ArcFace score (Deng et al. 2019a) that calculates average co- sine similarity between source and generated videos as identity similarity. We also employ HyperlQA (Zhang al.] 2023b) and LAION Aesthetic (Schuhmann et al.) 2022) for evalu- ating image quality from low- and high-level. Moreover, we conduct the expression evaluation following the steps of Point-Tracking in Mim- icBenct
Reference
Driving
Result
Character Benchmark. To comprehensively evaluate the effectiveness and generalization ability of portrait animation methods towards characters, build CharacBench that comprises 46 characters from various domains, such as animals, emojis, toys and anime characters. Characters CharacBench are collected from Internet by following the guideline of ensuring that the characters do not resemble human facial features as much as possible. Moreover, we consider videos of human head from RAVDESS (Livingstone Russo 2018) our driving videos.
Figure 8: Visualizations of character animation from non-human driving videos.
4.2 COMPARISON WITH SOTA METHODS
Qualitative Results. We compare proposed FaceShot with SOTA portrait animation methods, including MOFA- NNu et al. 2024). X-Portrait (Xie et al. 2024). VaceVid (Wang et

---

Published as a conference paper at ICLR 2025
Table Quantitative comparison between FaceShot and other SOTA methods on CharacBench. The best result marked in bold, and the second- best performance is highlighted in underline. Symbol indicates that there are some failure cases in these methods, we report the values of these methods only for reference.
Metrics
User Preference
Methods
ArcFace
IlyperQQA f
Aesthetic
Point- Track-Tn
Motion t
Identity t
Overall
VaceVid2Vid
0.525
33.721
4.267
6.944
3.58
3.83
4.52
FADM
0.633
39.402
4,522
6,993
1.93
2.04
1.96
X-Portrait
0.490
52.357
4.754
7.301
1.47
1.57
Follow Your Emoji
0.612
52.056
4.906
6.960
6.91
6.67
6.74
AniPortrait
0.634
55.951
4,928
6.367
5.84
5.64
5.39
MegActor*
0.613
40.191
4,855
7.183
6.53
6.75
6.26
LivePortrair*
0.893
53.587
5.092
7.474
7.33
7.11
MOFA-Video
0.695
52.272
4.952
14.985
3.27
3.04
3.18
FaceShot
0.848
53.723
5.036
6.935
8.14
8.27
Follow Your Emoji (Ma al. 2024), LivePortrait (Guo al. 2024). and MegActor (Yang et al. 2024). Visual comparisons are presented in Figure 7 where fail indicates that the method was unable to generate animation for the character. As AniPortrait (Wei) et al., (204)) fails with most non non-human characters, we only provide its quantitative results. We observe that most methods such as MOFA- Video, X-Portrait, FaceVid2Ve and Follow Your Emoji, are influenced by the human prior in the driving video, resulting in human facial features appearing on character faces. contrast, FaceShot effectively preserves the identity of reference characters through precise landmarks provided by our proposed appearance-guider landmark matching module. Furthermore, while most methods struggle to retarget the motions like eye closure and mouth opening, coordinated based landmark retargeting module enables FaceShot capture subtle movements.
2021), FADM Zenn et al., 2023
Beyond its effective character animation capability, FaceShot can also animate reference characters from non- non-human driving videos, extending the application of portrait animation from duman--elatee videos to any video as shown Fig. 8 This demonstrates its potential for open-domain portrait animations.
Quantitative Results. We conduct a quantitative comparison on the met- rics mentioned in Section 4.1 Please note that some methods, such as Live- Portrait, MegActor, and AniPortrait, fail to generate animations for certain characters when they are unable to de- tect the face. Therefore, for a fair com- parison, we report the failure rate for these methods as follows: AniPortrait (39.13%) MegActor (36.50%) and LivePortrait (16.67%), and we calcu- late their metric values on success- ful characters only for reference pur- poses. Based Table FaceShot demonstrates significantly superior performance across various metrics compared other methods on CharacBench. Specifically, FaceShot achieves the highest score in terms of ArcFace (0.848), demonstrating the effectiveness of the precise landmarks generated by the appearance-guide
FaceShot
Uni-Pose
STAR
DIFT
Figure 9: The visualizations of landmarks detection on differ- ent characters in CharacBench using DIFT. STAR, Uni-Pose and FaceShot.
landmark matching module in preserving facial identity. FaceShot achieves superior HyperIQA (53.723) and Aesthetic (5.036) scores, indicat- ing better image quality. Additionally, the coordinate-ban based landmark retargeting module contributes to the competitive point tracking score (6.935), highlighting its ability to handle motion effectively. important to note that our method has achieved significant improvements across all metrics compared to the base method, MOFA- Video, further demonstrating the effectiveness of our proposed FaceShot.
User Preference. Additionally, randomly selected 15 case examples and enlisted 20 volunteers to evaluate each method across three key dimensions: Motion, Identity, and Overall User Satisfac- tion. Volunteers ranked the animations based on these criteria, ensuring fair and comprehensive
8

---

Published as a conference paper at ICLR 2025
comparison between the methods. As shown in Table FaceShot achieves the highest scores in Motion, Identity, and Overall categories, demonstrating its robust animation capabilities across diverse characters and driving videos.
4.3 ABLATION STUDIES
Appearance-guided Landmark Matching, To evaluate the effectiveness of our appearance-guided landmark matching module, we compare with SOTA unsupervised algorithm DIFT (Tang et al. 2023) and supervised algorithms Uni-Pose (Yang et al. (202) and STAR (Zhou et al., (203) on CharacBench. Specifically, we recruited volunteers to annotate the landmarks of images in CharacBench as ground-truth values for calculating the corresponding Normalized Mean Error (NME) value.
Table Albation studies of appearance guided landmark matching module with supervised SOTA methods Uni-Pose and STAR and unsupervised method DIFT CharacBench. Best result marked in bold, and the second- best performance is highlighted in underline.
Methods
NME 4
ArcFace f
HyperlQA t
Aesthetic 1
STAR
24.530
52.849
0.829
4.989
Uni-Pose
13.731
53.685
0.851
5.025
DIFT 11.448
53.506
0.843
5.023
FaceShot 8.569
53.723
0.848
5.036
As illustrated in Table 2 FaceShot achieves the lowest NME on CharacBench. The visual landmark results are shown in Figure FaceShot accurately detects the landmarks on the non-human characters, whereas others fail to match the positions of the eyes and mouth. Furthermore, we observe that Faceshot also achieves the best scores in Table highlighting the necessity precise landmarks of landmark-drin driven portrait animation models.
Coordinate-based Land- mark Retargeting. To evaluate the effectiveness of our landmark retargeting module, we provide a com- parison with SOTA land- mark retargeting methods, including Deep3D (3DMM) (Deng et al., 2019b), Ever- thing's Talking (Song et al. bb and FreeNet (Zhang ct al. 2020). where Ever- thing's Talking models the Bézier Curve as the mo- tion controller and FreeNet
0-th 0-th frame
1-th frame
Driving
FaceShot
FreeNet
Everything's Talking
Deep3D
Figure 10: Visual results of landmark retargeting between FaceShot and other methods.
trains a parameterized network for retargeting.
Table 3: Albation studies coordinate-basse landmark retargeting module with SOTA methods Deep3D, Everthing's Talking and FreeNet. Best result is marked in bold.
Everthing's 's Talking
Metric
FaceShot
Deep3D
FreeNet
Point-
6.935
8.282
8.382
8.272
Tracking
As shown in Figure 10 due the precision loss of fitting Bezier curve, Everthing's Talking always generates the inaccurate retargeting. FreeNet and 3DMM have strict requirements for the distribution facial features, making it unable to adapt non-human characters. In contrast, module can

---

Published as a conference paper at ICLR 2025
precisely capture the subtle motion such mouth opening, eye closure, and global face movement. Additionally, FaceShot also achieves the lowest point-tracking score in Table demonstrating its effectiveness in capturing the consistent face movements.
As Plugin. Experiments Scction 4.2 have demonstrated that FaceShot can significantly improve the perfor- mance of the landmark-drivee method MOFA-Video (Niu et al., 2024). To further verify effectiveness as plugin for landmark -diveen anima- tion models, we input the reposed landmark sequences generated by FaceShot into AniPortrait's (Wei et al., 2024) pipeline. As shown in Figure inaccurate landmarks in original AniPortrait often result in distortions, producing results that resemble real humans. In contrast, FaceShot can provide precise and robust landmarks for characters, leading to harmonious and stable animation
Reference
Driving
AniPortrait
w/ FaceShot
Figure 11: Visual results of AniPortrait with or without FaccShot as plugin to generate animation results.
results.
CONCLUSION
In this paper, introduced FaceShot, training-free portrait animation framework that animates any character from any driven video. By leveraging semantic correspondence in latent diffusion model features, FaceShot addresses the limitations of existing landmark-driven methods, enabling precise landmark matching and landmark retargeting. This powerful capability not only extends the application of portrait animation beyond traditional boundaries but also enhances the realism and consistency of animations in landmark-driven models. FaceShot is also compatible with any andmark-drivee animation model as plugin. Additionally, experimental results on CharaeBench, benchmark featuring diverse characters, demonstrate that FaceShot consistently outperforms current SOTA methods.
Future Work. Although FaceShot shows strong performance, future work could focus enhancing appearance guided landmark matching by refining semantic feature extraction from latent diffusion models, partieularly for complex facial geometries. Furthermore, parameterizing landmark retargeting could offer more precise control over facial expressions, improving the adaptability of FaceShot across diverse character types and styles.
ETHICS STATEMENT
In developing FaceShot, training-free portrait animation framework that animates any characters from any driven video, we are dedicated to upholding ethical standards and promoting responsible AI use. We acknowledge potential risks, such as deepfake misuse unauthorized media manipulation. and stress the importance of applying this technology in ways that respect privacy, consent, and individual rights. Our code will be publicly released encourage responsible use in areas like entertainment and education, while discouraging unethical practices, including misinformation and harassment. We also advocate for continued research on safeguards and detection mechanisms to prevent misuse and ensure adherence to ethical guidelines and legal frameworks.
ACKNOWLEDGMENTS
This project is supported by the National Key R&D Program of China (No. 20222D0161600) and the National Natural Science Fund of China (No. 62473286).
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15122 2023a.
Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 2023b.
James Booth, Anastasios Roussos, Stefanos Zafeiriou, Allan Ponniah, and David Dunaway. A 3d morphable model learnt from 10,000 faces. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5543-5552, 2016.
Tim Cootes, ER Baldock, and J Graham. An introduction to active shape models. Image processing and analysis, 328:223-2444 2000.
Timothy Cootes, Gareth J. Edwards, and Christopher Taylor. Active appearance models, JEEE Transactions pattern analysis and machine intelligence, 23(6):681-683 2001.
Zuozhuo Dai, Zhenghao Zhang, Yao Yao, Bingxue Qiu, Siyu Zhu, Long Qin, and Weizhi Wang. Animateanything: Fine-grained open domain image animation with motion guidance. arXiv e-prints, pp. arXiv-2311, 2023.
Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou. Arcface: Additive angular margin loss for deep face recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 4690-4699, 2019a.
Yu Deng. Jiaolong Yang, Sicheng Xu, Dong Chen, Yunde Jia, and Xin Tong. Accurate 3d face reconstruction with weakly-supervised learning: From single image image set. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, pp. 0-0, 2019b.
Piotr Dollár, Peter Welinder, and Pietro Perona. Cascaded pose regression. 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 1078-1085. IEEE, 2010.
Jeff Donahue, Philipp Krähenbühl, and Trevor Darrell. Adversarial feature learning. arXiv preprint 1605.0260, 2016.
Nikita Drobyshev, Jenya Chelishev, Taras Khakhulin, Aleksei Ivakhnenko, Victor Lempitsky, and Egor Zakharov. Megaportraits: One-shot megapixel neural head avatars. In Proceedings of the 30th ACM International Conference on Multimedia, pp. 2663-2671, 2022.
Yao Feng, Haiwen Feng, Michael J Black, and Timo Bolkart. Learning an animatable detailed 3d face model from in-the-wild images. ACM Transactions on Graphics (ToG), 40(4):1-13, 2021.
Junyao Gao, Yanchen Liu, Yanan Sun, Yinhao Tang, Yanhong Zeng, Kai Chen, and Cairong Zhao. Styleshot: A snapshot any style. arXiv preprint 2xXi:2407.00114. 2024.
Litong Gong, Yiran Zhu, Weijie Li, Xiaoyang Kang, Biao Wang, Tiezheng Ge, and Bo Zheng. Atomovideo: High fidelity image-to-vides generation. arXiv preprint arXiv:2403.0/800, 2024.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the 2020.
ACM,
Jianzhu Guo, Xiangyu Zhu, Yang Yang, Fan Yang, Zhen Lei, and Stan Li. Towards fast, accurate and stable 3d dense face alignment. European Conference on Computer Vision, pp. 152-168.
Springer, 2020.
Jianzhu Guo, Dingyun Zhang, Xiaoqiang Liu, Zhizhou Zhong, Yuan Zhang, Pengfei Wan, and Di Zhang. Liveportrait: Efficient portrait animation with stitching and retargeting control. arXiv preprint arXiv:2407.00 03168, 2024.
11

---

Published as a conference paper at ICLR 2025
Yuwei Guo, Ceyuan Yang, Anyi Rao, Zhengyang Liang, Yaohui Wang, Yu Qiao, Maneesh Agrawala, Dahua Lin, and Bo Dai. Animatediff: Animate your personalized text-to-image diffusion models without specific tuning. arXiv preprint arXiv:2307.04725. 2023.
Eric Hedlin, Gopal Sharma, Shweta Mahajan, Xingzhe He, Hossam Isack, Abhishek Kar, Helge Rhodin, Andrea Tagliasacchi, and Kwang Moo Yi. Unsupervised keypoints from pretrained diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern 2024.
Recognition, pp.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In NeurIPS, 2020.
Fa- Ting Hong, Longhao Zhang, Li Shen, and Dan Xu. Depth-aware generative adversarial network for talking head video generation. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 3397-3406, 2022.
Li Hu. Animate anyone: Consistent and controllable image-to- viiee synthesis for character animation. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 2024.
Xiche Huang, Weihong Deng, Haifeng Shen, Xiubao Zhang, and Jieping Ye. Propagationnet: Propa- gate points to curve to learn structure information. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7265-7274, 2020.
Diederik P Kingma. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6111 2013.
Marek Kowalski, Jacek Narunicc, and Tomasz Trzcinski. Deep alignment network: A convolutional neural network for robust face alignment. In Proceedings of the IEEE conference computer vision and pattern recognition workshops, pp. 88-97, 2017.
Abhinav Kumar, Tim K Marks, Wenxuan Mou, Ye Wang, Michael Jones, Anoop Cherian, Toshiaki Koike-Akino, Xiaoming Liu, and Chen Feng. Luvli face alignment: Estimating landmarks' location, uncertainty, and visibility likelihood. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 8236-8246, 2020.
Jiaxing Li, Hongbo Zhao, Yijun Wang. and Jianxin Lin. Towards photorealistic video colorization via gated color-guided image diffusion models. In Proceedings of the 32nd ACM International Conference Multimedia, pp. 10891-10900 2024a.
Yaokun Li, Guang Tan, and Chao Gou, Cascaded iterative transformer for jointly predicting facial landmark, occlusion probability and head pose. International Journal of Computer Vision, 132(4): 1242-1255 1257, 2024b.
Steven R Livingstone and Frank A Russo. The ryerson audio-visual database of emotional speech and song (ravdess): A dynamic. multimodal set of facial and vocal expressions in north american english. PloS one, 13(5)ee019339,, 2018.
Grace Luo, Lisa Dunlap, Dong Huk Park, Aleksander Holynski, and Trevor Darrell. Diffusion hyperfeatures: Searching through time and space for semantic correspondence. Advances in Neural Information Processing Systems, 36, 2024.
Yue Ma, Hongyu Liu, Hongfa Wang, Heng Pan, Yingqing He, Junkun Yuan, Ailing Zeng, Chengfei controlllll and expressive
Cai, Heung- Yeung Shum, Wei Liu, et al. freestyle portrait animation. arXiv preprint arXiv:2406. N.X::2406.0190000 2024.
Daniel Merget, Matthias Rock, and Gerhard Rigoll. Robust facial landmark detection via a fully- convolutional local-globe context network. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 781-790, 2018
Haomiao Ni, Changhao Shi, Kai Li, Sharon X Huang, and Martin Renqiang Min. Conditional image-to-video generation with latent flow diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 18444-18455 2023.
12

---

Published as a conference paper at ICLR 2025
Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXi::2112.10741)) 2021.
Muyao Niu, Xiaodong Cun, Xintao Wang, Yong Zhang, Ying Shan, and Yinqiang Zheng. Mofa- video: Controllable image animation via generative motion field adaptions in frozen image-to-video diffusion model. arXiv preprint 24ti::2400.20222. 2024.
Augustus Odena, Christopher Olah, and Jonathon Shlens. Conditional image synthesis with auxiliary classifier gans. In International conference on machine learning, pp. 2642-2651. PMLR, 2017.
Alec Radford. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprini arXiv: /5| 1.06434, 2015.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 9748-8763. PMLR, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint 06122,, 1(2):3, 2022.
George
Retsinas, Panagiotis P Filntisis, Radek Danecek, Victoria F Abrevaya, Anastasios Roussos, Timo Bolkart, and Petros Maragos. 3d facial expressions through analysis-by-neural-syyttt In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2490-2501, 2024.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pp. 10684-10695, 2022.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. Medical Image Computing and Computer-Assisterr Intervention-MICCO 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part IIl 18, pp. 234-22444444 241. Springer, 2015.
Ludan Ruan, Yiyang Ma, Huan Yang. Huiguo He, Bei Liu, Jianlong Fu, Nicholas Jing Yuan, Qin Jin, and Baining Guo. Mm-diffuusio: Learning multi-modal diffusion models for joint audio and video generation. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 10219-10228, 2023.
Christos Sagonas, Epameinondas Antonakos, Georgios Tzimiropoulos, Stefanos Zafeiriou, and Maja Pantic. 300 faces in-the-wild challenge: Database and results. Image and vision computing, 47:
3-18, 2016.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in neural information processing systems, 155334799-36999 2022.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systemx, 35:25278-25294, 2022.
Fei Shen and Jinhui Tang. Imagpose: A unified conditional framework for pose-guided person generation. In The eiiht Annual Conference on Neural Information Processing Systems, 2024.
Fei Shen, Hu Ye, Jun Zhang, Cong Wang. Xiao Han, and Wei Yang. Advancing pose-guided image synthesis with progressive conditional diffusion models. arXiv preprint arXiv:2310.06311 13, 2023.
Fei Shen, Xin Jiang, Xin He, Hu Ye, Cong Wang, Xiaoyu Du, Zechao Li, and Jinghui Tang. Imagdressing-vl: Customizable virtual dressing. arXiv preprint arXiv:2407. 12..007..775,,. 2024a.
13

---

Published as a conference paper at ICLR 2025
Fei Shen, Hu Ye, Sibo Liu, Jun Zhang, Cong Wang, Xiao Han, and Wei Yang. Boosting consis- tency in story visualization with rich-contextual conditional diffusion models. arXiv preprint arXiv:2407.02482, 2024b.
Fei Shen, Cong Wang, Junyao Gao, Qin Guo, Jisheng Dang, Jinhui Tang, and Tat-Seng Chua Long-term talkingface generation via motion-prior conditional diffusion model. arXiv preprini arXiv:2502.09533, 2025.
Xiaoyu Shi, Zhaoyang Huang, Fu-Yun Wang, Weikang Bian, Dasong Li. Yi Zhang, Manyuan Zhang, Ka Chun Cheung, Simon See, Hongwei Qin, et al. Motion-i2v: Consistent and controllable image-to-video generation with explicit motion modeling. ACM SIGGRAPH 2024 Conference Papers, pp. 1-11, 2024.
Aliaksandr Siarohin, Stéphane Lathuilière, Sergey Tulyakov, Elisa Ricci, and Nicu Sebe. First order motion model for image animation. Advances in neural information processing systems, 32, 2019.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. ICLR, 2021a.
Linsen Song, Wayne Wu, Chaoyou Fu, Chen Qian, Chen Change Loy, and Ran He. Everything's talkin': Pareidolia face reenactment. arXiv preprint arXiv:2104.0306/ 2021b.
Yi Sun, Xiaogang Wang, and Xiaoou Tang. Deep convolutional network cascade for facial point detection. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 3476-3483, 2013.
Luming Tang, Menglin Jia, Qianqian Wang, Cheng Perng Phoo, and Bharath Hariharan. Emergent correspondence from image diffusion. Advances in Neural Information Processing Systems, 36: 1363-1389, 2023.
Cong Wang. Kuan Tian, Yonghang Guan, Jun Zhang, Zhiwei Jiang, Fei Shen, Xiao Han, Qing Gu, and Wei Yang. Ensembling diffusion models via adaptive feature aggregation. arXiv preprint arXiv:2405. 2024a.
Ting-Chun Wang, Arun Mallya, and Ming-Yo Liu. One-shot free-view neural talking-head synthesis for video conferencing. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pp. 10039-10049, 2021.
Xiang Wang, Hangjie Yuan, Shiwei Zhang, Dayou Chen, Jiuniu Wang. Yingya Zhang, Yujun Shen, Deli Zhao, and Jingren Zhou Videocomposer: Compositional video synthesis with motion controllability. Advances in Neural Information Processing Systems, 36, 2024b.
Huawei Wei, Zejun Yang, and Zhisheng Wang. Aniportrait: Audio-driven synthesis of photorealistic portrait animation. arXiv preprint arXiv:2403. 17694, 2024.
Wayne WU, Chen Qian, Shuo Yang, Quan Wang, Yici Cai, and Qiang Zhou. Look at boundary: A bownd face alignment algorithm. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2129-2138, 2018.
Yue Wu, Tal Hassner, KangGeon Kim, Gerard Medioni, and Prem Natarajan. Facial landmark detection with tweaked convolutional neural networks. IEEE transactions on pattern analysis and machine intelligence, 40(12):3067-3074 2017.
You Xie, Hongyi Xu, Guoxian Song, Chao Wang, Yichun Shi, and Linjie Luo. X-portrait: Expressive portrait animation with hierarchical motion attention. In ACM SIGGRAPH 2024 Conference Papers, pp. 1-11, 2024.
Jinbo Xing, Menghan Xia, Yong Zhang, Haoxin Chen, Xintao Wang, Tien-Tsin Wong, and Ying Shan. Dynamicrafter: Animating open-domain images with video diffusion priors. arXiv preprint arXiv:2310.12190, 2023.
Lumin Xu, Sheng Jin, Wang Zeng, Wentao Liu, Chen Qian, Wanli Ouyang, Ping Luo, and Xiaogang Wang. Pose for everything: Towards category-agnosiee pose estimation. In European conference computer vision, pp. 398-416. Springer, 2022.
14

---

Published as a conference paper at ICLR 2025
Jie Yang, Ailing Zeng, Ruimao Zhang, and Lei Zhang. Unipose: Detecting any keypoints. arXiv preprint arXiv:2310.085.0 2023.
Shurong Yang, Huadong Li, Juhao Wu, Minhao Jing, Linze Li, Renhe Ji, Jiajun Liang, and Haoqiang Fan. Megactor: Harness the power of raw video for vivid portrait animation. arXiv preprint arXiv:2405.20851 2024.
Hu Ye, Jun Zhang, Sibo Liu, Xiao Han, and Wei Yang. Ip-adapter: Text compatible image prompt adapter for text-to-image diffusion models. arXiv preprint arXiv:2338..000 06721, 2023.
Bohan Zeng, Xuhui Liu, Sicheng Gao, Boyu Liu, Hong Li, Jianzhuang Liu, and Baochang Zhang
Face animation with an attribute-guides diffusion model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 628-628-6666 2023.
Jiangning Zhang, Xianfang Zeng, Mengmeng Wang, Yusu Pan, Liang Liu, Yong Liu, Yu Ding, and Changjie Fan. Freenet: Multi-identity face reenactment. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 5326-5335, 2020.
Shiwei Zhang, Jiayu Wang, Yingya Zhang, Kang Zhao, Hangjie Yuan, Zhiwu Qin, Xiang Wang, Deli Zhao, and Jingren Zhou. 12vgen-xl: High-quality image-to-video synthesis via cascaded diffusion models. arXiv preprint arXiv:2311.041455 2023a.
Weixia Zhang, Guangtao Zhai, Ying Wei, Xiaokang Yang, and Kede Ma. Blind image quality assessment via vision-language correspondence: A multitask learning perspective. In Proceddigss of the IEEF/CVF conference on computer vision and pattern recognition, pp. 14071-14081, 2023b.
Yiming Zhang, Zhening Xing, Yanhong Zeng, Youqing Fang, and Kai Chen. Pia: Your personalized image animator via plug-and-play modules in text-to-image models. In Proceedings of the IEFF/CVF Conference on Computer Vision and Puttern Recognition, pp- 7747-7756, 2024.
Zhimeng Zhang, Lincheng Li, Yu Ding, and Changjie Fan. Flow-guided one-shot talking face generation with high-resolution audio-visual dataset. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3661-3670, 2021.
Jian Zhao and Hui Zhang. Thin-plate spline motion model for image animation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3657-3666, 2022.
Erjin Zhou, Haoqiang Fan, Zhimin Cao, Yuning Jiang, and Qi Yin. Extensive facial landmark localization with coarse-to-fine convolutional network cascade. In Proceedings of the IEEE international conference on computer vision workshops, pp. 386-391, 2013.
Zhenglin Zhou, Huaxia Li, Hong Liu, Nanyang Wang, Gang Yu, and Rongrong Ji. Star loss: Reducing semantic ambiguity in facial landmark detection. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp.
2023.
15

---

Published as a conference paper at ICLR 2025
APPENDIX
A IMPLEMENTATION DETAILS
In FaceShot, we use single H800 generate animation results. And we have included total of 46 images and 24 driving videos in CharacBench, with each video consisting of 110 to 127 frames, All videos (.mp4) and images (Jig) are processed into resolution of 512 x 512. Following MOFA-Video for human face and driving video, we utilized Facial Alignment Network (FAN) implemented in facexlib annotating algorithm detect the landmarks. And for non-human characters, perform manual annotating as even the SOTA face landmark detection methods still fail on these characters.
B METHODS
Appearance Matching in Appearance Gallery. As mentioned in Section 3.2] the purpose of the appearance gallery is reduce appearance discrepancies by matching the reference image to the closest target domain. In detail, the reference image is first cropped into five facial parts i.e., eyes, mouth, nose, eyebrows and face boundary as:
Tref [Trefes Tref,, Tref.as Tref.eb, Tref.fb);
each facial part includes specific number of landmarks, listed in the Table
Table 4: Specific landmark number for each facial part.
eyes
mouth
nose
eyebrows
face boundary
12
20
10
17
Next, each part is matched to the closest target domain in the appearance gallery by calculating the average CLIP image score:
G = arg max CLIP-S(Iref.r) JED
pi
where p E {e, m. n. eb. fb} and D represents the domains in part & denotes the number of target images in given domain and CLIP- denotes the clip image score. Finally, the target images are formulated as Iar G G G Gob Gp)
Settings in Landmark Retargeting. The angle and the origin of the rectangular coordinate sys- tem are calculated using two endpoints pd1 and p as follows:
Face Boundary
Nosc
Mouth<inner)
p 2 [0]
+
p0)
p=1[1]
[1] pda [2[1]]
2
pe [1-111->p(111 1 [0) pr [0]
arctan
Fyebrow
Eye
Mouth(outer)
And the indices of the endpoints within each part of every frame are fixed, as illustrated in the Figure [2] For better understanding of our coordinate-basee landmark retargeting module, we provide an illustration of this module Fig-
Figure 12: Illustration of the endpoints in each part, marked with circle.
ure K
C EXPERIMENTS
Choices of Time Step t and Layer To extract the diffusion feature that best fits facial instances, conduct detailed experiments the selection of time steps and layer 7 of U-Net. Specifically,
16

---

Published as a conference paper at ICLR 2025
Stage Global motion
Driving
Reference
Stage 2: Local motion
Reference
Driving
Driving
Reference
frame
frame
Illustration of our coordinate brased landmark retargeting module. Specifically, our module consists of two stages: global motion and local motion retargeting, which aim capture global and local positional changes of the entire face and individual facial parts separately.
Figure 13:
we test different combinations of and 300W (Sagonas al. 2016). a widely used facial dataset, and report NME as the quantitative result. As shown in Figure 14 we achieve the best NME value when 301 and 6, which are used the basic settings of our paper.
Choice of Target Number k. As mentioned in Section 3.2 we use the averaged diffusion feature the i-th point of target images to improve matching performance. We eval- uate different values of k 1,5, 10, 15, 50, 1000 on the 300W dataset and report the NME in T Results indi- cate that increasing & significantly improves performance when < 10. However, for & 10. the time cost in-
233
231
creases exponentially with diminishing performance gains. Based on this observation,
2et
set A 10 for our exper-
iments. Please note that this analysis is solely aimed at determining the number of target images. However, the features of the target images will be pre-stored as local data, which is considered additional time overhead during inference.
201
3DMM Modeling. Following our base model MOFA- Video, we adopt Deep3D (Deng et al., 2019b) as the 3DMM method in Figure 6 Deep3D employs a deep
Figure 14: Heatmap of NME values for time step and layer of the U-Net.
network to predict 3D coefficients (coeff) at each frame of driven videos, instead of iterative fitting. Additionally, we provide the 3D modeling results of DECA (Feng al. 202 1. Deep3D and 3DDFAv2 (Guo et al. 2020) on non-human characters and driven videos in Figure 15 Our observations reveal that none of these 3DMM methods accurately generate precise 3D models of non-human characters or capture subtle movements in driven sequences, such as eye closure.
Visualization of Appearance Guidance. We provide the cosine similarity distribution during inference. As depicted in Figure [6] with prior appearance knowledge, the similarity between reference points and unrelated target points becomes smaller, reducing the probability of mismatching.
17

---

Published as a conference paper at ICLR 2025
Target
Figure 17: Visual results of landmarks human faces.
3DDFAv2
Drivio
Deep3D
Self
DECA
Cress
Figure 18: Self and cross identity driving results on HDTE.
Figure 15: Modeling results of different 3DMM methods.
Human Evaluations. To evaluate the effectiveness of FaceShot on human faces, we first provide the visual land- mark results on 300W in Figure 17
by appearance
0.15
wyl appeerance
Frequency
Furthermore, we present the animation results for self- identity and cross-identity driving on traditional facial video datasets HDTF (Zhang et al. (2021) in Figure 18 FaceShot perform well on these real human faces, showing its robustness.
0.1
0.05
Cosine Similanty
Time Efficiency. We present a time analysis of each step in FaceShot for processing varying numbers of frames a single H800 GPU, as shown in the Table
Figure 16: Cosine similarity distribution with or without appearance guidance.
Driving Detection: Detecting the landmark sequence of the driving video using the landmark detector from MOFA-Video. The detector used is FAN from facexlib.
Detecting the target image landmarks using the appearance-guiin guided landmark matching module. The time cost of landmark matching remains almost identical regardless of the number frames because the matching is required only once for the target image, irrespective of the number of frames in the driving video. The 0.8-second time includes both the DDIM inversion and the argmin operation in Eq.
Landmark Matching:
Landmark Retargeting: Retargeting landmark motion using coordinate- based landmark retargeting module. As no model parameters are required, our retargeting modules can generate precise landmark sequences with very low time cost.
Table 5: Experiments on the number & of target images. Best NME result is marked in bold.
Metric
10
15
50
1000
NME
12.801
7.009
6.343
6.267
6.252
6.104
Table 6: Time analysis of each step FaceShot for processing varying numbers frames.
Driving Detection
Landmark Matching
Landmark Retargeting
frames
50
1.817(s)
0.860(s)
0.382(s)
100
3.562(s)
0.858(s)
0.751(s)
In conclusion, FaceShot introduces only 119ms additional time overhead when used plugin for MOFA-Video (for 50 frames). This minimal time cost negligible compared to the inference time of diffusion-based models (approximately 80 seconds for 50 frames). As shown in Table 7 FaceShot
18

---

Published as a conference paper at ICLR 2025
achieves the low time cost among diffusion- methods, including AniPortrait, FADM, Follow Your Emoji, MegActor, X-P X-Portrait, and MOFA-Vine Video.
Table Time analysis SOTA methods inference on frames. Symbol represents GAN-based method.
Methods
Time
Methods
Time
Vid2Vidd
4.308(s)
MegActor
174.189(s)
LivePortrait'
4s321(s)
X-Portrait
132.702(s)
AniPortrait
99.977(s)
MOFA- Video 79.421(s)
FADM
88.368(s)
FaceShot 79.540(s)
Follow Your Emoji 112.830(s)
19


Reviews:
[{"review_id": "OqKODLJkeX", "reviewer": "Reviewer_fAcM", "summary": "FaceShot achieves precise landmark matching and robust motion transfer by introducing an appearance-guided landmark matching module and a relative motion transfer module. The landmark matching module utilizes the semantic correspondences of a latent diffusion model to extract consistent and robust landmarks from various types of driving videos. Subsequently, the motion transfer module effectively conveys the relative motion information of these landmarks to the target character, enabling diverse and realistic character animation without needing fine-tuning or retraining.", "questions": "1. Is the purpose of the appearance gallery to match the target image with a composite reference image (from different images)?\n2. What is the structure of the LandmarkMotionTransfer module? Can you describe the working principle of this part?\n3. On the CABench benchmark proposed by the authors, Faceshot achieved SOTA performance. Is this a test benchmark for the same identity (character)? Besides qualitative experiments, how is the effect of cross-identity motion measured? What are the quantitative results for same-identity and cross-identity driving on traditional facial video datasets, such as voxceleb1 or 2, or HDTF?\n4. Is the 3DMM using an iterative fitting algorithm based on LSFM? If so, it seems intuitive that it might not perform well on video sequences. Additionally, how many iterations does it take? Have algorithms like DECA, Deep3D, 3DDFAv2, etc., been tested for performance on video sequences?\n5. Which algorithm is used for annotating the landmarks in the reference images?\n6. How is the stability and robustness of landmarks prediction with training-free?\n7. In Figure 7, is the poor driving effect of the eyes in the first row and fifth row due to the difficulty in detecting round eyes?\n8. How is the time efficiency? What is the specific time for each step, and what is the frame rate for animation driving?\nI am willing to improve my score after addressing any questions.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1.FaceShot offers a potential solution for portrait animation in open domains. Traditional methods often fail with non-human characters, such as emojis, animals, and toys, due to their facial features significantly differing from humans. This results in landmark detection failures and compromised animation quality. FaceShot addresses this by providing accurate landmarks, making animations for non-human characters more reasonable and effective.\n2.FaceShot can be integrated as a plugin into other landmark-based animation-driven models, enhancing its scalability across various animation tasks.\n3.FaceShot achieves precise landmark matching by leveraging semantic correspondences within latent diffusion models.", "weaknesses": "1. I like this application scenario as it effectively utilizes the correspondence between the features and positions of diffusion for landmark detection. However, without training, the effectiveness seems limited. See Question 7.\n2. The cross-domain expression driving has been effectively improved, but the pose driving remains limited. However, this is a common drawback of 2D methods. See Question 2.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "89CtddmvNn", "reviewer": "Reviewer_QeSM", "summary": "This paper introduces an innovative training-free framework, FaceShot, designed to animate any human or non-human character using any driven video. The FaceShot framework is built on three key components: 1) an appearance-guided landmark matching module, 2) a relative landmark motion transfer module, and 3) a character animation model. Together, these components enable precise landmark detection and stable animation generation across diverse character types. In addition, the authors present CABench, a standardized benchmark dataset specifically created to evaluate distribution-agnostic portrait animation techniques. The techniques introduced by FaceShot represent a significant advancement in the field of open-domain portrait animation, overcoming limitations in existing methods and expanding animation capabilities to a broader range of characters", "questions": "1. How to match I_{tar} with the closest domain in the appearance gallery G, and what evaluation metric is used? This step is not indicated in the pipeline.\n\n2. In the relative landmark motion transfer section, I have three questions. First, how to get the angle of the global rectangular coordinate system? Second, how to determine the origin and angle of the local rectangular coordinate system of each part of the face? Third, when motion occurs, the origin and angle of different local coordinate systems in the reference and target image will also change. How to deal with this? \n\n3. CABench is considered a significant contribution of this paper, but the dataset details are missing. It would be nice to provide a detailed list of the character types and their quantities included in CABench, specifying the specific emotions and intensity distributions of the driving videos, and the number of videos for each emotion. Additionally, what is the total number of images and video frames in the dataset, and do you need any preprocessing operations performed on the dataset, including the resolution and format of the images?\n\n4. When FaceShot is a plugin for landmark-driven animation models, will it introduce additional computation and time overhead?\n\n5. Can Faceshot achieve real-time interaction?\n\n6. In the pipeline, it would be nice to give some example text prompts in the  “Appearance Guided Landmark Matching” step.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. Broad Character Adaptability: FaceShot incorporates an \"appearance-guided landmark matching module\" that enables it to handle a wide variety of characters, including non-human characters like emojis, toys, and animals, without requiring retraining. This innovation significantly extends the model's applicability, allowing it to generate stable animations across diverse domains.\n\n2. Stability in Animation Generation: FaceShot demonstrates high stability in handling non-human characters, accurately capturing subtle expression changes like eye blinks and mouth movements to produce smooth and coherent animations. This stability is mainly due to the \"relative motion transfer module,\" which ensures continuity of facial features in the landmark sequence generation, resulting in consistent animation effects.\n\n3. Introduction of the CABench Dataset: FaceShot not only offers a novel animation generation method but also introduces the CABench dataset, providing a standardized tool for evaluating animation performance on non-human characters. This dataset includes various character types and emotion-driven videos, offering a more representative testing environment for future research and filling a gap in the diversity of characters in current datasets.", "weaknesses": "Many details are missing. Without specifying the experimental setup, including the device used and the details of the training parameters (such as the learning rate), it is unclear how much time overhead is introduced by incorporating the appearance-guided landmark matching module and the relative motion transfer module. Additionally, the resolution of the generated videos has not been mentioned. See questions for details.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XporKHHvLf", "reviewer": "Reviewer_x66b", "summary": "The paper proposed a training free framework for animating any input character (human or non-human) from any human face driven video, by predicting facial landmarks for the input character. Extensive qualitative results showcase the effectiveness of the proposed method.", "questions": "1. Though reported in Table 2, how long does it take for generating a single frame or the same number of frames in inference compared with other methods. e.g.  LivePortrait (with the same number of inputs)?\n2. In Figure 12, why the numbers of generated landmarks with the proposed method are not consistent across different figures?\n3. Some black or purple texture artifacts are witnessed on the generated images of the pink figure shown in Figure 1 (on the second row and second colume), which is more severe in its corresponding video (at the very end of the website, the 5th video from left to right). Is that because of the proposed method itself cause I haven't seen such issue on other compared methods?\n4. Are there any generated videos longer than five seconds?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed training-free architecture is easy to follow and and the objective as in Eq(4) is straightforward.\n2. The introduced Appearance Gallery consisting of rich semantic meaningful regions benefits facial expression&motion modeling, matching and transfer. \n3. The qualitative results look good and promising, especially the comparisons in Fig.12.", "weaknesses": "It seems that the proposed method does not significantly outperform LivePortrait in terms of both qualitative and quantitative results, as shown in Table1 and videos from Anonymous website.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)