FaceShot: Bring Any Character into Life

Authors: Junyao Gao, Yanan SUN, Fei Shen, Xin Jiang, Zhening Xing, Kai Chen, Cairong Zhao
Primary area: generative models
Keywords: portrait animation, diffusion model

Abstract:
In this paper, we present ***FaceShot***, a novel training-free portrait animation framework designed to bring any character into life from any driven video without fine-tuning or retraining.
We achieve this by offering precise and robust reposed landmark sequences from an appearance-guided landmark matching module and a coordinate-based landmark retargeting module.
Together, these components harness the robust semantic correspondences of latent diffusion models to produce facial motion sequence across a wide range of character types.
After that, we input the landmark sequences into a pre-trained landmark-driven animation model to generate animated video.
With this powerful generalization capability, FaceShot can significantly extend the application of portrait animation by breaking the limitation of realistic portrait landmark detection for any stylized character and driven video.
Also, FaceShot is compatible with any landmark-driven animation model, significantly improving overall performance.
Extensive experiments on our newly constructed character benchmark CharacBench confirm that FaceShot consistently surpasses state-of-the-art (SOTA) approaches across any character domain.
More results are available at our project website https://faceshot2024.github.io/faceshot/.

Decision: accept (poster)
Reviewer scores: 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1730003969065

thread_type:
official_review

<thread_object>
creator: reviewer_x66b

<content>
summary:
The paper proposed a training free framework for animating any input character (human or non-human) from any human face driven video, by predicting facial landmarks for the input character. Extensive qualitative results showcase the effectiveness of the proposed method.

strengths:
1. The proposed training-free architecture is easy to follow and and the objective as in Eq(4) is straightforward.
2. The introduced Appearance Gallery consisting of rich semantic meaningful regions benefits facial expression&motion modeling, matching and transfer. 
3. The qualitative results look good and promising, especially the comparisons in Fig.12.

weaknesses:
It seems that the proposed method does not significantly outperform LivePortrait in terms of both qualitative and quantitative results, as shown in Table1 and videos from Anonymous website.

questions:
1. Though reported in Table 2, how long does it take for generating a single frame or the same number of frames in inference compared with other methods. e.g.  LivePortrait (with the same number of inputs)?
2. In Figure 12, why the numbers of generated landmarks with the proposed method are not consistent across different figures?
3. Some black or purple texture artifacts are witnessed on the generated images of the pink figure shown in Figure 1 (on the second row and second colume), which is more severe in its corresponding video (at the very end of the website, the 5th video from left to right). Is that because of the proposed method itself cause I haven't seen such issue on other compared methods?
4. Are there any generated videos longer than five seconds?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730424977263

thread_type:
official_review

<thread_object>
creator: reviewer_qesm

<content>
summary:
This paper introduces an innovative training-free framework, FaceShot, designed to animate any human or non-human character using any driven video. The FaceShot framework is built on three key components: 1) an appearance-guided landmark matching module, 2) a relative landmark motion transfer module, and 3) a character animation model. Together, these components enable precise landmark detection and stable animation generation across diverse character types. In addition, the authors present CABench, a standardized benchmark dataset specifically created to evaluate distribution-agnostic portrait animation techniques. The techniques introduced by FaceShot represent a significant advancement in the field of open-domain portrait animation, overcoming limitations in existing methods and expanding animation capabilities to a broader range of characters

strengths:
1. Broad Character Adaptability: FaceShot incorporates an "appearance-guided landmark matching module" that enables it to handle a wide variety of characters, including non-human characters like emojis, toys, and animals, without requiring retraining. This innovation significantly extends the model's applicability, allowing it to generate stable animations across diverse domains.

2. Stability in Animation Generation: FaceShot demonstrates high stability in handling non-human characters, accurately capturing subtle expression changes like eye blinks and mouth movements to produce smooth and coherent animations. This stability is mainly due to the "relative motion transfer module," which ensures continuity of facial features in the landmark sequence generation, resulting in consistent animation effects.

3. Introduction of the CABench Dataset: FaceShot not only offers a novel animation generation method but also introduces the CABench dataset, providing a standardized tool for evaluating animation performance on non-human characters. This dataset includes various character types and emotion-driven videos, offering a more representative testing environment for future research and filling a gap in the diversity of characters in current datasets.

weaknesses:
Many details are missing. Without specifying the experimental setup, including the device used and the details of the training parameters (such as the learning rate), it is unclear how much time overhead is introduced by incorporating the appearance-guided landmark matching module and the relative motion transfer module. Additionally, the resolution of the generated videos has not been mentioned. See questions for details.

questions:
1. How to match I_{tar} with the closest domain in the appearance gallery G, and what evaluation metric is used? This step is not indicated in the pipeline.

2. In the relative landmark motion transfer section, I have three questions. First, how to get the angle of the global rectangular coordinate system? Second, how to determine the origin and angle of the local rectangular coordinate system of each part of the face? Third, when motion occurs, the origin and angle of different local coordinate systems in the reference and target image will also change. How to deal with this? 

3. CABench is considered a significant contribution of this paper, but the dataset details are missing. It would be nice to provide a detailed list of the character types and their quantities included in CABench, specifying the specific emotions and intensity distributions of the driving videos, and the number of videos for each emotion. Additionally, what is the total number of images and video frames in the dataset, and do you need any preprocessing operations performed on the dataset, including the resolution and format of the images?

4. When FaceShot is a plugin for landmark-driven animation models, will it introduce additional computation and time overhead?

5. Can Faceshot achieve real-time interaction?

6. In the pipeline, it would be nice to give some example text prompts in the  “Appearance Guided Landmark Matching” step.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730605676744

thread_type:
official_review

<thread_object>
creator: reviewer_facm

<content>
summary:
FaceShot achieves precise landmark matching and robust motion transfer by introducing an appearance-guided landmark matching module and a relative motion transfer module. The landmark matching module utilizes the semantic correspondences of a latent diffusion model to extract consistent and robust landmarks from various types of driving videos. Subsequently, the motion transfer module effectively conveys the relative motion information of these landmarks to the target character, enabling diverse and realistic character animation without needing fine-tuning or retraining.

strengths:
1.FaceShot offers a potential solution for portrait animation in open domains. Traditional methods often fail with non-human characters, such as emojis, animals, and toys, due to their facial features significantly differing from humans. This results in landmark detection failures and compromised animation quality. FaceShot addresses this by providing accurate landmarks, making animations for non-human characters more reasonable and effective.
2.FaceShot can be integrated as a plugin into other landmark-based animation-driven models, enhancing its scalability across various animation tasks.
3.FaceShot achieves precise landmark matching by leveraging semantic correspondences within latent diffusion models.

weaknesses:
1. I like this application scenario as it effectively utilizes the correspondence between the features and positions of diffusion for landmark detection. However, without training, the effectiveness seems limited. See Question 7.
2. The cross-domain expression driving has been effectively improved, but the pose driving remains limited. However, this is a common drawback of 2D methods. See Question 2.

questions:
1. Is the purpose of the appearance gallery to match the target image with a composite reference image (from different images)?
2. What is the structure of the LandmarkMotionTransfer module? Can you describe the working principle of this part?
3. On the CABench benchmark proposed by the authors, Faceshot achieved SOTA performance. Is this a test benchmark for the same identity (character)? Besides qualitative experiments, how is the effect of cross-identity motion measured? What are the quantitative results for same-identity and cross-identity driving on traditional facial video datasets, such as voxceleb1 or 2, or HDTF?
4. Is the 3DMM using an iterative fitting algorithm based on LSFM? If so, it seems intuitive that it might not perform well on video sequences. Additionally, how many iterations does it take? Have algorithms like DECA, Deep3D, 3DDFAv2, etc., been tested for performance on video sequences?
5. Which algorithm is used for annotating the landmarks in the reference images?
6. How is the stability and robustness of landmarks prediction with training-free?
7. In Figure 7, is the poor driving effect of the eyes in the first row and fifth row due to the difficulty in detecting round eyes?
8. How is the time efficiency? What is the specific time for each step, and what is the frame rate for animation driving?
I am willing to improve my score after addressing any questions.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732515416736

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Comments

comment:
Thank all reviewers for reviewing and providing constructive feedbacks to our paper. We also deeply appreciate the reviewers’ acknowledgment of:
* Open-domain adaptability (reviewers fAcM, QeSM)
* Scalability as a training-free plugin (reviewers fAcM, QeSM)
* Stable and precise results for landmarks and animation (reviewers fAcM, QeSM, x66b)

Please note that, we put the rebuttal pdf both in the Appendix of revised manuscript and the supplemental materials. In the rebuttal pdf, we included 9 figures:

* Figure R1: Modeling results of different 3DMM methods.
* Figure R2: The landmark visual results on CABench.
* Figure R3: Illustration of the reasons why the eyes in the 1st and 5-th rows of Figure 7 do not change. The reason is that the driving eyes do not change instead of poor driving quality.
* Figure R4: Eye driving case, closed eyes, for the 1st and 5-th characters in Figure 7.
* Figure R5: Illustration of our relative landmark motion transfer module.
* Figure R6: Comparisons with LivePortrait.
* Figure R7: Texture artifacts in other diffusion-based methods.
* Figure R8: Illustration of why texture artifacts occur.
* Figure R9: Illustration of the endpoints in each part, marked with a red circle.

We respond to each reviewer below to address concerns. Please take a look and let us know if further clarification / discussion is needed.
Also, we will include all these discussions in the next version and release codes, models and data of FaceShot.
</content>
</thread_object>

</thread>


Metareview:
metareview:
All reviewers agree to accept the paper. Reviewers appreciate the novel solution, broad adaptability, and promising results. Please be sure to address the reviewers' comments in the final version.