Playmate: Flexible Control of Portrait Animation via 3D-Implicit Space Guided Diffusion
Authors: Xingpei Ma
Jiaran Cai
Yuansheng Guan
Shenneng Huang
Qiang Zhang
Shunsi Zhang
Venue: ICML 2025 poster (2025)
Keywords: Diffusion
Portrait Animation
Transformer
Audio-driven
Primary area: applications->computer_vision

Abstract: Recent diffusion-based talking face generation models have demonstrated impressive potential in synthesizing videos that accurately match a speech audio clip with a given reference identity. However, existing approaches still encounter significant challenges due to uncontrollable factors, such as inaccurate lip-sync, inappropriate head posture and the lack of fine-grained control over facial expressions. In order to introduce more face-guided conditions beyond speech audio clips, a novel two-stage training framework Playmate is proposed to generate more lifelike facial expressions and talking faces. In the first stage, we introduce a decoupled implicit 3D representation along with a meticulously designed motion-decoupled module to facilitate more accurate attribute disentanglement and generate expressive talking videos directly from audio cues. Then, in the second stage, we introduce an emotion-control module to encode emotion control information into the latent space, enabling fine-grained control over emotions and thereby achieving the ability to generate talking videos with desired emotion. Extensive experiments demonstrate that Playmate not only outperforms existing state-of-the-art methods in terms of video quality, but also exhibits strong competitiveness in lip synchronization while offering improved flexibility in controlling emotion and head pose. The code will be available at https://github.com/Playmate111/Playmate.

Playmate: Flexible Control of Portrait Animation via 3D-Implicit Space Guided Diffusion
Xingpei Ma"
Jiaran Cai
Yuansheng Guan Shenneng Huang
Qiang Zhang Shunsi Zhang
Guangzhou Quwan Network Technology
Vivid Emotion Control
Angry
Disgusted
Contempt
Fearful
Happy
Surprised
Precise Pose Control
Lip Sync and Vivid Expression
null:
(c)
Figure 1. Playmate can generate lifelike talking faces for arbitrary identity, guided by speech audio clip and variety of optional control conditions. (a) shows the generation results under different emotional conditions using the same audio clip. The top row in (b) shows the driving images, while the bottom row shows the generated results. The poses in the generated results are controlled by the driving images, and the lip movements are guided by the driving audio. (c) demonstrates highly accurate lip synchronization and vivid, rich expressions across different style images.
Abstract
posture and the lack of fine-graine control over facial expressions. In order to introduce more face-guided conditions beyond speech audio clips, a novel two-stage training framework Playmate is proposed to generate more lifelike facial expres- sions and talking faces. In the first stage, we intro- duce decoupled implicit 3D representation along with meticulously designed motion-decoupe module to facilitate more accurate attribute disen- tanglement and generate expressive talking videos directly from audio cues. Then, in the second stage, we introduce an emotion-control module to encode emotion control information into the latent space, enabling fine-grained control over emo- tions and thereby achieving the ability generate
Recent diffusion- bassdd talking face generation models have demonstrated impressive potential in synthesizing videos that accurately match a speech audio clip with given reference identity. However, existing approaches still encounter sig- nificant challenges due to uncontrollable factors, such as inaccurate lip-syne, inappropriate head
Equal contribution 'Project lead Correspondence to: Jiaran Cai <caijiaran@e 52tt.com>, Xingpei Ma <maxing- pei@ 52tt.com>.
Proceedings of the ndnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Playmate: Flexible Control of Portrait Animation via 3D-Implicit Space Guided Diffusion
talking videos with desired emotion. Extensive experiments demonstrate that Playmate not only outperforms existing state-of-the-art methods in terms of video quality, but also exhibits strong competitiveness in lip synchronization while of- fering improved flexibility in controlling emotion and head pose. The code will be available at https///iihub.coo//aaaaaatt http:maaaee////aaaaaaeea I/Playmate.
Based on the above observations, we propose Playmate, novel two-stage training framework that leverages a 3D> Implicit Space Guided Diffusion Model to generate lifelike talking faces with controllable facial attributes. To achieve this goal, we first introduce a decoupled implicit 3D rep- resentation proposed in face-vid2vid (Wang et al., 2021) and LivePortrait (Guo et al., 2024). This implicit repre- sentation effectively disentangles multiple face attributes. including expression, lip movement, head pose, and others, thereby enabling more flexible editing and control of these attributes. Subsequently, we train an audio-conditiones diffusion transformer combined with a carefully designed motion-decoupled module, facilitate more accurate attribute disentanglement and generate expressive talking videos di- rectly from audio cues. Finally, to enhance the controllabil- ity of emotions, we introduce the emotion-control module, which utilizes DiT blocks (Peebles & Xie, 2023) to encode specified emotion conditions and integrates the encoded information into the aforementioned audio-conditioned dif- fusion model, thereby achieving flexible control over emo- tions and improving the editability of the generated talking faces. As shown in Figure 1, Playmate has advantages in audio-driven portrait animation. The main contributions of this paper can be summarized as follows:
1. Introduction
Audio-driven portrait animation techniques (Jiang et al., 2024a; Xue et al., 2024) aim to synthesize a lifelike talking face from a static image and speech audio, thereby creating realistic and expressive avatars. In recent years, with the significant development of diffusion-based models (Ho et al., 2020; Song et al., 2020; Dhariwal & Nichol, 2021; Rombach et al., 2022), this field has attracted increasing attention from both academia and industry. Consequently, these advances have unlocked substantial potential for diverse applications, including film dubbing, video generation, and interactive media.
Existing audio- portrait animation approaches can be broadly categorized into two main branches: GA based methods (Chen al., 2018; Prajwal al., 2020; Lu al., 2021; Zhang et al., 2023b; Ma et al., 2023a; Tan et al., 2024b; Ki & Min, 2023; Cheng et al., 2022; Guo et al., 2021b; Wang et al., 2024; Ye et al., 2023) and diffusion- based methods (Shen et al., 2023; Sun et al., 2023; Tian et al., 2025; He et al., 2023; Xu et al., 2024b;a; Zheng et al.,
We present Playmate, a novel framework that utilizes 3D-Implicit Space Guided Diffusion for generating talking face videos.
We meticulously designed a motion-decoupled module and trained a diffusion transformer to improve mo- tion disentanglement and generate motion sequences directly from audio cues.
2024; Cao et al., 2024; et al., 2024; Ma al., 2023b; Chen
et al., 2024; Lin et al., 2024b; Liu et al., 2024a; Sun al., 2024). The former category has primarily focused the accuracy of lip synchronization (Chung & Zisserman, 2017) and has delivered significant results in this regard. How- ever, GadeeRFF-aaeed methods often overlook the holistic coordination between audio cues and facial expressions, leading to failure in generating expressive facial dynamics and lifelike expressions. Recently, the advent of diffusion models has facilitated the generation of high-quality images and videos (Podell et al., 2023; Esser et al., 2024; Liu et al., 2024b; Lin et al., 2024a; Kong et al., 2024: Hong et al., 2022; Yang et al., 2024). Several studies have introduced diffusion models into the field of portrait animation, enabling them to excel in generating talking face videos. Nonetheless, the expression, lip-syne, and head pose of videos generated by diffusion-based methods are strongly correlated and cou- pled with audio cues, which limits flexibility in controlling specific facial attributes such as head pose and emotional expression, making it challenging to modify one attribute independently without altering the associated audio content. This dependency limits the customization and adaptability of generated animations for different applications.
A novel emotion-control module and corresponding training strategy are proposed to encode emotion infor- mation into latent space and enhance the controllability of emotions of talking head vidco.
Experiments show that our method achieves SOTA performance in terms of video quality, motion diversity and emotion controllability.
2. Related Work
2.1. GAN/NeRF- Audio-driven Portrait Animation
Talking face video generation has been a long-standing chal- lenge in computer vision and graphies. The goal is to synthe- size lifelike and synchronized talking videos from driving audio and static reference images. Early MAN/NeRF-basee approaches, such as Wav2Lip (Prajwal et al., 2020), Dinet (Zhang et al., 2023b), and VideoRe Talking (Cheng et al., 2022), primarily focused on achieving high-quality lip-sync while keeping other facial attributes static. Consequently,
2

---

Playmate: Flexible Control of Portrait Animation via 3D-Implicit Space Guided Diffusion
these methods fail to capture strong correlations between the audio and other facial attributes, such as facial expression and head movements. To address this limitation, GANi- mation (Pumarola et al., 2018) introduced an unsupervised method to generate talking videos with a specific expres- sion. EAMM (Ji et al., 2022) synthesizes emotional talking faces with augmented emotional source videos. More re- cent studies have typically employed intermediate motion representations (e.g., landmark coordinates, 3D facial mesh, and 3DMM) to generate videos from audio (Zhang et al., 2023a; Gan et al., 2023; Peng et al., 2024). However, such approaches often generate inaccurate intermediate repre- sentations, which restricts the expressiveness and realism of the resulting videos. In contrast, our framework gen- crates accurate motion representations based on diffusion transformer.
using 3DMM, sparse keypoints, or FLAME to explicitly characterize facial attributes. Furthermore, in the field of audio-driven portrait animation, several studies have intro- duced facial representation techniques to generate lifelike talking videos. Sadtalker (Zhang et al., 2023a) separates generation targets into different categories, including eye blinks, head poses, and lip-only 3DMM coefficients. Recent works such as VASA-1 (Xu et al.. 2024b), Takin-ADi Aik (Lin et al., 2024b), Dream'Talk (Ma al., 2023b), and Joy VASA (Cao et al., 2024) have begun to combine face represen- tations with diffusion models to achieve more naturalistic results. Inspired by these advancements, we similarly utilize face representation techniques to generate more natural and controllable talking videos.
3. Methodologg
2.2. Diffusiin-baase Audio-driven Portrait Animation
As shown in Figure 2, Playmate uses 3D implicit space as the intermediate representation for generating talking heads from a single static image, guided by a speech audio clip and a set of optional control signals. This section claborates on our method in detail. We begin with a brief introduc- tion of the 3D implicit space. Furthermore, we describe our meticulously designed approach for generating motion sequences directly from audio cues. Finally, we introduce our emotion control module and two-stage training strategy, which enhance the controllability of emotions of talking head vidco.
Diffusion models have shown impressive performance across various vision tasks. However, previous (Bigioi et al., 2024; Mukhopadhyay et al., 2024; Shen et al., 2023) at- tempts to utilize diffusion models for generating talking heads have only yielded neutral-emotion expressions, lead- ing to unsatisfactory results. Some of the latest methods have some optimizations for this purpose, such as EMO (Tian et al., 2025), Hallo (Xu et al., 2024a), Echomimic (Chen et al., 2024), and Loopy (Jiang et al., 2024b). EMO introduces novel framework that ensures consistency in audio-driven animations across video frames, thereby en- hancing the stability and naturalness of synthesized speech animations. Hallo contributes a hierarchical audio-driven method for animating portrait images, tackling the com- plexities of lip synchronization, expression, and pose align- ment. MEMO (Zheng et al., 2024) proposes an end-to- end audio-driven portrait animation approach to generate dlennttt-connisten and expressive talking videos. Several of the above methods can generate vivid portrait videos by fine-tuning pre-trained diffusion models. However, they usu- ally use coupled latent spaces to represent facial attributes in relation to the audio. Facial attributes such as expression and head posture are often generated directly from audio cues. This coupling limits the ability customize control over certain facial attributes, such as pose and expression. In Playmate, we leverage 3D implicit space that decoupled various facial attributes, enabling diverse and controllable facial animations while maintaining high accuracy in lip synchronization.
3.1. Expressive and Disentangled Latent Face Space Construction
Facial representation aims to construct a latent face space that exhibits high degrees of expressiveness and disentan- glement. Typically, this approach separates various aspects of facial data into distinct components, such as appearance features and motion attributes. In the field of audio-driven portrait animation, many studies have leveraged these latent spaces to generate talking heads. For example. VASA-1 1 based its model on the 3D-aid face reenactment framework from (Wang et al., 2021; Drobyshev et al., 2022), while Takin-ADA and Joy constructed their latent space based on face-vid2vid and LivePortrait, respectively. Simi- larly, we adopt and enhance the decoupled facial representa- tion proposed by face-vid2vid and LivePortrait.
LivePortrait primarily comprises an appearance feature ex- tractor F, a motion extractor M, a warping module W, and decoder G. When presented with a source image 7, and a driving image La. LivePortrait initially utilizes F to extract appearance feature f. from I., and separately captures the motion information from both I, and Ia using M. Subse- quently, it leverages the motion information extracted from Ia to animate 1, using W and G. ensuring that the animated result retains the original appearance while adopting the mo-
2.3. Facial Representation in Audio-driven Portrait Animation
Facial representation learning has been extensively studied in previous works. Various methods (Siarohin et al., 2019; Ren et al., 2021; Li et al., 2017) disentangled variables
3

---

Similarly, transfer I, expression onto I5. yielding 1,,00p = ((W(Sj,a) (,,,,)))) and the transformation is Playmate: Flexible Control of Portrait Animation via 3D-Implicit Space Guided Diffusion
Motion
Norm Motion
Noise
Proj
Audio Proj
Adapt
Proj
Modion Prej
M
Norm
ID Proj
Idendity Proj
Video
T Proj
Time Proj
Stagel flow
Wav2Vee
Self Attn
Stage2 flow
Audio
Audio Feat
Motion
Feat
Emo Controller
Diffusion Transformer
Pose MLP
Conformer
DiT block
Proj
Proj
Exp MLP
Eo
DiT block
T Proj
ID Proj
Emo Cond
Figure 2. Framework of our approach. Playmate is a two-stage training framework that leverages a 3D-Implicit Space Guided Diffusion Model to generate lifelike talking faces. In the first stage, Playmate utilizes mocoun-eecoupled module enhance attribute disentanglement accuracy and trains diffusion transformer to generate motion sequences directly from audio cues. In the second stage, we use an emotion-control module to encode emotion control information into the latent space, enabling teeiggraiedd control over emotions, thereby improving flexibility in controlling emotion and head pose.
tion cues from the driving image. The motion information expres- R3x3
formalized as:
is characterized by canonical keypoints v, E IR xxx x3
S 6,) (8) { z = (Ze, Jz,yeap = 8j (Ia, R3 + 5,) l--
sion deformations e xx3 rotation matrix translation vector R33 and a scaling factor E The transformation LivePortrait is formalized as:
Ultimately, a perceptual loss (Johnson al., 2016) is em- ployed between Li,pose and Iyssoy to enhance the disentan- glement between facial dynamics and head pose, the loss function as:
F. Sy (Ie, Rs + 6z) + t.r
(1)
Fa = 5, (Ze,s Ra + 6a) + ld.
where Fr,s represents the canonical keypoints of the source image L,, and v, and za are the source and driving 3D implicit keypoints after transformation. Additional informa- tion about this framework can be found in reference (Guo et al., 2024).
=1 V 1|2,
(4)
where V denotes the feature extractor of VGG19 (Simonyan, 2014).
3.2. Audio Conditioned Diffusion Transformer
Similar to VASA-1, we found that the original LivePortrait also suffered from poor disentanglement between facial dynamics and head pose. To address this issue, we utilize the pairwise head pose and facial dynamics transfer loss used in VASA-1 to improve its disentanglement. Specifically, let I, and 1 be two randomly sampled frames from the same video clip. Then we transfer the head pose of I onto 1; resulting in Ir,poose G(W(Ss,, (i, transformation is formalized as:
After constructing the latent face space, we can extract mo- tions by utilizing the frozen M and train the motion genera- tor, with audio cues serving the condition.
Adaptive Normalization. To achieve a more effective de- coupling of expression and head pose, we employ adaptive normalization by using different means and standard de- viations when training the motion generator. Specifically, for expression, we compute the global mean and standard deviation using the entire training dataset,
the
{
x = (I + 8,) + ti Digyome 8j (Ic.; R, + 6;) + +t--
8E. FM
11
N:
(5)
S N
5
2
N.

---

Playmate: Flexible Control of Portrait Animation via 3D-Implicit Space Guided Diffusion
where N₂ is the number of frames in the i-th video clip, and M is the total number of all training samples. For head pose, we treat each identity independently and compute the private mean and standard deviation for each sample along the time dimension, as:
outpat Exp Latent
NEP
Block
Layer harm
u
E Pijs
Get
of- En (P.,--4))2
(6)
Seff
Euso Eirleed
Albeiike
Smtc Shil
where p denotes the head pose information. By combining adaptive normalization with the transfer loss mentioned in 3.1, we achieve better decoupling of motion. We refer to this approach as the motion-decoupler module.
Input Exp Lutent
Eus Cand
Lewn
Timestop
Leket
Coodilie
Figure 3. The structure of the Emotion-control Module.
Speech Representation. Extensive research has demon- strated that pre-trained speech models, such as Wav2Vec2 (Baevski et al., 2020) and HuBERT (Hsu et al., 2021), out- perform traditional features like MFCC in performance. Similar to other methods, we utilize Wav2Vee2 as our speech encoder extract audio features. Additionally, we use self-attention module to align audio features with motion features.
transformer and training an emotion controller based on the DiT block (Peebles & Xie, 2023). As shown in Figure and Figure the emotion-control module consists of two DiT blocks. The first block receives inputs from the conformer and the emotion condition, while the second block takes the output of the first block and emotion condition as its inputs. The outputs of the second block are then passed to Exp- MLP, replacing the original conformer's output. Through the aforementioned operations, information into the features and pass them to the Exp-MLI module, thereby achieving the ability to generate talking videos with desired emotion.
Diffusion Transformer. The architecture of the Diffusion Transformer is illustrated in Figure 2. We employ four Proj modules, primarily composed of fully connected layers, to extract semantic information from four unique features, aligning this information across different modalities. Next, utilize a multilayer Conformer (Gulati et al., 2020), Posc-MLP, and a Exp-MLP in combination with the diffu- sion formulation to generate motion sequences. Diffusion models define two Markov chains: the forward chain pro- gressively adds Gaussian noise to the target data, while the reverse chain iteratively refines the raw signal from the noise. During training, we gradually transform clean motion m into Gaussian noise my following the principles of De- noising Diffusion Probabilistic Models(DDPM) (Ho al., 2020). The Diffusion Transformer is then trained to reverse this noise-adding process by taking m, and other condi- tional features as input and predicting the added noise 6. The objective function for training can be expressed as:
encode emotion control
During the second phase of training, we freeze the parame- ters of the diffusion transformer and train only the param- eters of the emotion-control module. Experimental results demonstrate that this approach effectively maintains lip- sync accuracy while integrating emotion conditions, allow- ing for precise emotion control in the generated animations.
Classifier-free guidance(CFGG. We adopt a classifier- free guidance (Ho & Salimans, 2022) approach following (Brooks et al., 2023), which has been successfully applied to image generation from multiple conditions. During training, random dropout strategy is applied to each of the input conditions to improve the model's robustness and general- ization. During inference, we apply:
Casss 1Emes (IE- (me, feas +))" (7)
where far fid are the audio feature and identity feature, to represents the noise prediction made by the Diffusion Transformer.
Eg =ē (me, f.d, + waleo (mz, Sa. 0, fid, 1) 20 (m, 0. 0, Fid. t)] welão (mi, Sa. Ses Sea, 1) - Eo (m², Sa. 0, fia, 1)),
3.3. Emotion-control Module
(8)
After completing the first training stage, we obtain a diffu- sion transformer that generates motion sequences guided by audio cues. To enhance the controllability of emotions in talking head videos, we propose the emotion-contre mod- ule. This involves fixing the parameters of the diffusion
where wa and we are the guidance scales for audio condition and emotion condition, respectively.
5

---

Playmate: Flexible Control of Portrait Animation via 3D-Implicit Space Guided Diffusion
Table 1. Quantitative comparisons of video quality and lip synchronization with state-of-the-art methods on two test datasets. The best results are in bold, and the second-best boot are underlined. Playmate consistently outperforms existing methods terms of vidco quality and identity preservation, while also exhibiting strong competitiveness in lip synchronization.
Dataset
Method
FID
FVD
Sync-C t
Sync-D L
CSIM
LPIPS 4
Hallo (Xu et al., 2024a)
30.484
288,479
7.923
7.531
0.804
0.139
Hallo2 (Cui et al., 2024)
30.768
288.385
7.754
7.649
0.822
0.138
MEMO (Zheng et al., 2024)
27.713
299.493
8,059
7.473
0.840
0.132
HDTF
Sonic (Ji et al., 2024)
29.189
305,867
9.139
6,549
0.783
0,149
JAyVASA (Cao al., 2024)
29.581
306.683
8,522
7.215
0.781
0.157
Playmate(OUus))
19.138
231.048
8,580
6,985
0.848
0.099
Hallo (Xu et al., 2024a)
46.114
288,415
6.454
8.384
0.767
0.139
Hallo2 (Cui et al., 2024)
46.185
295.532
6,509
8.358
0.761
0.144
MEMO (Zheng al., 2024)
39.224
260.498
6.569
8.193
0.782
0.130
Collected dataset
Sonic (Ji ct al., 2024)
39.069
254,959
7.972
7.124
0.762
0.139
JAYVA (Cao al., 2024)
50.314
304.621
6.858
8.194
0.713
0.163
Playmate(QUrs)
34.716
227.871
7.125
8.007
0.797
0.128
Generated Frame
Gonurated Frames
Play-motte
45
Ref Image
Hilioo
Halle
JayYAMA
Ref Image
Senti MEMO
Figure 4. Qualitative comparisons with state-of-the-art methods. Previous methods were prone generate artifacts in tooth render- ing(e.g. (a)-row 6, column 3; (b)-row 3, column 1) and lip synchronization(c.g (a)-row 4, column 7; (b)-row 2, column 7). Conversely, our approach boasts a superior decoupling capability, which allows to create more lifelike talking head videos. more comparison details, please see the Appendix.
4. Experiments
we choose HDTF (Zhang et al., 2021) as our out-of-domain dataset, which comprises about 362 different videos with original resolution of 720P or 1080P.
4.1. Experiments Setup
Datasets. We utilize mixture of datasets, including AVSpeech (Ephrat et al., 2018), CelebV-Text (Yu et al., 2023), Acappella (Montesinos et al., 2021), MEAD (Wang et al., 2020), MAFW (Liu et al., 2022), and a talking video dataset collected by us to train our Playmate. the first stage, we selected approximately 80,000 video clips from the AVSpeech, CelebV-Text, Acappella, and own dataset to train the diffusion transformer. For the second phase, selected approximately 30,000 emotionally labeled video clips from the MEAD, MAFW, and our own dataset to train the emotion control module. The duration of each training video ranges from 3 to 30 seconds. We set aside a portion of video clips from our dataset that were not involved in the training process as our out-of-distribution test Besides,
Implementation Details. our experiments, the videos are initially converted 25 fps and subsequently cropped to resolution of 256 x 256 pixels based on face landmarks extracted using InsightFace (Deng et al., 2020; Guo et al., 2021a). The final output resolution is set to 512 x 512 pixels. During preprocessing, the audios were resampled to 16kHz. The first training phase utilized four NVIDIA A A100 GPUs over a 3-day period, with models initialized from scratch. In the second phase, continued training for two days with two NVIDIA A100 GPUs, while freezing the parameters of the diffusion transformer. For all experiments, we employed the Adam optimizer (Kingma, 2014). In the inference phase, multi- multi-condition CFG is performed. The CFG scales of the
6

---

Playmate: Flexible Control of Portrait Animation via 3D-Implicit Space Guided Diffusion
Imsgge
Generated
Angry
Digustel
Conterupt
Happy
Surpe isol
Figure 5. Visualization results in different style images. Play- mate can drive a wide range of portraits, including real humans, animations, artistic portraits, and even animals.
audio Wa and the emotion condition We are set to 1.5.
Figure 6. Visualization results of emotion control. Each row shows the generation for different identity under different emo- tional conditions using the same audio clip, demonstrating the flexibility in controlling emotion of Playmate.
Evaluation Metrics. We demonstrate the superiority of our method using multiple widely recognized metrics from previous studies. Specifically, we employ Fréchet Inception Distance (FID) (Heusel et al., 2017) and Fréchet Video Dis- tance (FVD) (Unterthiner et al., 2019) to evaluate the quality of the generated data. For evaluating lip synchronization and motion fluidity, we compute the confidence score (Sync-C) and feature distance (Sync-D) using the pretrained SyneNet (Chung & Zisserman, 2017). Additionally, we compute the cosine similarity (CSIM) of identity vectors extracted using the ArcFace (Deng et al., 2019) face recognition model to evaluate identity preservation. Furthermore, we leverage Learned Perceptual Image Patch Similarity (LPIPS) (Zhang et al., 2018) to measure the feature-level similarity between the generated faces and their ground-truth faces. For the evaluation of emotion control, we employ an additional emotion classifier from (Savchenko, 2023) to calculate the emotion accuracy (Emo-A) of the generated videos.
cating superior performance in identity preservation and image quality. Additionally, our method achieves good re- sults in Syne-C and Sync-D, both of which are second-best, exhibiting strong competitiveness. Although our quantita- tive metrics for lip synchronization on the two test sets are marginally lower than Sonic's, they still outperform other compared methods. Furthermore, Sonic is a purely audio- driven algorithm that generates all features of the portrait, including lip movements, expressions, pose, etc., based audio. This indicates that its driving flexibility lim- ited. In contrast, we achieve multiple controllable portrait driving methods by constructing precise attribute disen- tanglement space, offering users various flexible driving options. The implementation complexity of this decoupling and subsequent driving approach is higher than that of a simple audio-driven method.
4.2. Results and Analysis
Figure 4 provides a visualization com- parison using open datasets. Upon analysis, previous meth- ods were prone to generate artifacts in tooth rendering(e.g) (a)-row 6, column 3; (b)-row 3, column 1) and lip synchro- nization(e.g. (a)-row 4, column 7: (b)-row 2, column 7). Conversely, our approach boasts a superior decoupling ca- pability, which allows to create more lifelike talking head videos. For more comparison details, please see the Ap- pendix. Since images cannot adequately reflect important aspects such as synchronization, naturalness, and stability, the full video comparison will be included in the supple- mentary materials or on our project page.
Qualitative Results.
Quantitative Results. We benchmark our method against SOTA audio-driven portrait animation methods, including Hallo (Xu et al., 2024a), Hallo2 (Cui et al., 2024), Joy VASA (Cao et al., 2024). MEMO (Zheng et al., 2024), and Sonic (Ji et al., 2024). As shown in Table 1, our Playmate signif- icantly outperforms other methods in terms of FID, FVD, CSIM, and LPIPS on two test datasets, while also exhibiting strong competitiveness in lip synchronization. Regarding video quality, method achieves the lowest FID and FVD scores on both test sets. Specifically, on the HDTF dataset, our FID and FVD scores are 30% and 20% lower than those of the second-bes best method, respectively, indicating superior video quality compared to other methods. For the CSIM and LPIPS metrics, we also achieve the best results, indi-
Visualization Results in Different Style Images. We fur- ther investigate the generation performance of our method
7

---

Playmate: Flexible Control of Portrait Animation via 3D-Implicit Space Guided Diffusion
Table 2. Quantitative comparisons of emotion control. The best results are in bold, and the second-best are in underlined.
Arlapt-Norm A
Methods
FID 4
FVD
LPIPS 5
Emo-A
EAMM (11 et al., 2022)
111.710
210.275
0.223
0.160
DreamTalk (Ma al., 2023b)
119.032
199.962
0.246
0.350
EDTalk (Tan et al., 2024a)
135.215
221.897
0.289
0.460
EAT (Gan al., 2023)
95.085
166.316
0.138
0.450
Playmate(Ours)
68.234
149.837
0.550
win Adapt Norma
in different style images. Given that our Playmate can di- rectly generate motion sequences from audio, it is capable of driving a wide range of portraits, including real humans, an- imations, artistic portraits, and even animals. Although this versatility may lead to characters having similar expressions, by integrating the amotion-control module and adjusting the CFG weight, we can easily achieve diverse expressions and movements. As illustrated in Figure 5, our method suc- cessfully generates a wide range of expressions and diverse movements across various style images, underscoring its robust performance.
Adapt Norm
Up
Majjt-Norm Adapt-
Emotion Control. Figure 6 shows the generation results of our method under different emotional conditions us- ing the same audio clip. These examples clearly illustrate our model's proficiency in encoding emotional signals into latent space and producing talking face animations with the desired emotion. This highlights Playmate's s effective- ness in achieving precise emotional control while main- taining natural and lifelike results. Furthermore, we com- pare our method with several existing expression control ap- proaches across various quantitative metrics on the MEAD test dataset. As presented in Table 2, Playmate achieves the highest scores in both video quality and emotion control metrics.
Figure 7. Ablation study of Adaptive Normalization. Without adaptive normalization, the generated results prone to pose jittering. inaccurate lip-sync. and strange expressions, resulting in temporal discontinuities and artifacts within the final video.
4.3. Ablation Studies
Adaptive Normalization. We analyze the effectiveness of our proposed Adaptive Normalization in decoupling ex- pressions and head pose. As shown in Figure 7, without adaptive normalization, the generated results are prone to pose jittering, inaccurate lip-sync, and strange expressions, resulting in temporal discontinuities and artifacts within the final video. After introducing adaptive normalization, where head pose feature and expression feature are normal- ized using different mean and variance values, the model can independently identify and distinguish pose feature and expression feature. Consequently, the final generated videos maintain better continuity in both pose and expression.
Table User study comparison on open dataset. The best re- sults are in bold, and the second- bccond-best&e are in underlined.
Methods
L
VD f
N
VA f
VASA (Cao et al., 2024)
2.500
2.286
1.714
1.929
Hallo (Xu et al., 2024a)
2,964
2.929
3.071
2.893
Hallo2 (Cui et al., 2024)
3.036
2.929
2.893
2.786
MEMO (Zheng et al., 2024)
3,321
3.036
3.179
3.143
Sonic (Ji et al., 2024)
3,821
3.071
3,750
3.500
Playmate(Ours)
3.750
3.857
3,464
3.643
User Study. We conducted a user study involving 10 par- ticipants who rated videos using the MOS (Mean Opinion Score) rating method, on a scale of to across four met- rics: Lip Syne (LS), Video Definition (VD), Naturalness (N), and Visual Appeal (VA). As illustrated in Table 3, Play- mate has a notable advantage in the VD and VA metrics. While the LS and N metrics are slightly lower than Sonic's, s, they still outperform those of other methods, showcasing Playmate's 's strong competitiveness.
CFG seales. By adjusting the CFG scales, we can strike balance between the quality and diversity of the gener- ated results. In Table we evaluate the selection of CFG scales for the audio and emotion conditions (represented by W. and w. in Equation (8)) within our model. Firstly, we conduct ablation experiments solely on Wa as shown in the upper part of Table 4. When w, is set to 1.5 or 2.0, Playmate achieves better evaluation metric scores. Upon comprehensive consideration, we set Wa to 1.5 and con-
8

---

Playmate: Flexible Control of Portrait Animation via 3D-Implicit Space Guided Diffusion
Table 4, Ablation study of the audio and emotion CFG scales. The top half shows the ablation results for Way and the bottom half shows the ablation results when both and We combined. The best results are in bold, and the second-best best are in underlined.
Impact Statement
This paper presents work whose goal is to advance the field of machine learning in portrait animation in partic- ular. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here. For an extensive discussion of the general ramifica- tions of talking video generation, we point interested readers towards (Xue et al., 2024; Jiang et al., 2024a).
Me
FID 1
FV13
Sync-C1
Sysc-D ]
CSIM
LPIPS
EimnAA 1
10
17.125
118.481
7.553
7.22
0.939
0,1144
7,064
0.935
0.046
2.0
8
15.968
122.718
8,111
7,049
0.916
0.064
0
2,5
8
16.857
125. 238
5.03
7,109
0.936
0,047
1.5
138.511
7.385
7.644
0.922
0.045
54.405
20
19.978
113
7,205
7.931
0.905
0.056
57,579
1.5
2,5
17.498
189.053
7,181
7.933
0.888
0.054
56,276
30
177.522
8.075
0.858
0.058
50,05
3.5
22.33
207.535
6.89
8.313
0.871
0.067
55.751
References
Baevski, A., Zhou, Y., Mohamed, A., and Auli, M. wav2vec 2.0: A framework for self-supervised learning of speech representations. Advances neural information process- ing systems, 13:49-41-22460 2020.
duct an additional ablation experiments on We. To more precisely assess the influence of we, we utilize the Emo-A metric, which helps in selecting suitable value for we. As shown in the bottom part of Table 4, when Wa is set to 1.5, setting We to 1.5 as well can achieve more balanced results across various testing metrics. Therefore, we also set we to 1.5.
Bigioi, D., Basak, S., Stypulkowski. M., Zieba, M., Jordan, H., McDonnell, R., and Corcoran, P. Speech driven video editing via an audio-conditioned diffusion model. Image and Vision Computing, 142: 142:104911, 2024.
5. Conclusion
Brooks, T., Holynski, A., and Efros, A. A. Instructpix2pix: Learning follow image editing instructions. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18402, 2023.
In summary, we present Playmate, a two-stage training framework designed to generate lifelike talking videos guided by speech audio clips and various optional control conditions. Our approach addresses critical limitations of existing methods, such as precise motion decoupling, ex- pression controllability. and lip synchronization accuracy. In the first stage, Playmate utilizes a motion motion-decoupled mod- ule to enhance attribute disentanglement accuracy and trains a diffusion transformer to produce expressive talking vidcos directly from audio cues. In the second stage, an emotion- control module is introduced to encode emotion control information into the latent space, enabling fine-grained con- trol over emotions, thereby achieving the ability to generate talking videos with desired emotion. Extensive evaluations show that Playmate consistently surpasses existing state- of-the-art approaches in video quality and facial dynamics realism, while also demonstrating strong performance in lip synchronization and enhanced control over emotion and head pose.
Cao, X., Shi, S., Zhao, J., Yao, Y., Fei, J., Gao, M., and Wang. G. Joyvasa: Portrait and animal image animation with diffusion-based audio-driven facial dy- namics and head motion generation. arXiv preprint arXiv:2411.099009 2024.
Chen, L., Li, Z., Maddox, R. K., Duan, Z., and Xu, C. Lip movements generation at a glance. In Proceedings of the European conference on computer vision (ECCV), pp. 520-535, 2018
Chen, Z., Cao, J., Chen, Z., Li, Y., and Ma, C. Echomimic: Lifelike audio-driven portrait animations through editable landmark conditions. arXiv preprint arXiv:2407.08// 2024.
Cheng, K., Cun, X., Zhang, Y., Xia, M., Yin, F., Zhu, M., Wang, X., Wang. J., and Wang, N. Videoretalking: Audio- based lip synchronization for talking head video editing in the wild. In SIGGRAPH Asia 2022 Conference Papers, pp. 1-9, 2022.
Limitations and future work. While Playmate shows sig- nificant advancements, it still has some limitations. Play- mate processes information primarily around the face area. Extending its capability to the full upper body or even the whole body could provide additional capability. When using 3D implicit representations, the lack of a more explicit 3D face model may lead to artifacts such as blurred edges during extreme head movements, texture sticking due to neural ren- dering, and minor inconsistencies in complex backgrounds. Future work will focus on enhancing the model's robustness to diverse perspectives and styles by incorporating more diverse training data, as well as improving the rendering quality of the framework through advanced techniques.
Chung, S. and Zisserman, A. Out of time: automated lip sync in the wild. In Computer Vision-ACCV 2016 Workshops: ACCV 2016 International Workshops, Taipei, Taiwan, November 20-24, 2016, Revised Selected Papers, Part I1 13, pp. 251-263. Springer, 2017.
Cui, J., Li, H., Yao, Y. Zhu, H., Shang, H., Cheng, K., Zhou, H., Zhu, S., and Wang, J. Hallo2: Long-duration and high-
9

---

Playmate: Flexible Control of Portrait Animation via 3D-Implicit Space Guided Diffusion
resolution audio-driven portrait image animation. arXiv preprint arXiv:2410.07718, 2024.
head synthesis. In Proceedings of the IEEE/CVF inter- national conference on computer vision, pp. 5784-5794, 2021b.
Deng, J., Guo, J., Xue, N., and Zafeiriou, S. Arcface: Additive angular margin loss for deep face recognition. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp. 4690-4699, 2019.
He, T., Guo, J., Yu, R., Wang. Y., Zhu, J., An, K., Li, L., Tan, X., Wang, C., Hu, H.,, et al. Gaia: Zero-shot talking avatar generation. arXiv preprint arXiv:2311.52300 2023.
Deng. J., Guo, J., Ververas, E., Kotsia, and Zafeiriou, S. Retinaface: Single-shot multi-level face localisation in the wild. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 5203-5212, 2020.
Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., and Hochreiter, S. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Ho, J. and Salimans, T. Classifier-free diffusion guidance. arXiv preprint arXiv:2207. 12598, 2022.
Dhariwal, P. and Nichol, A. Diffusion models beat gans on image synthesis. Neural Information Processing Sys- Neural Information Processing Systems, Dec 2021.
Ho, J., Jain, A., and Abbeel, Denoising diffusion proba- bilistic models. Advances in neural information process- ing systems,
Drobyshev, N., Chelishev, J., Khakhulin, T., Ivakhnenko, A., Lempitsky, V., and Zakharov, E. Megaportraits: One-shot megapixel neural head avatars. In Proceedings of the 30th ACM International Conference on Multimedia, pp- 2663-2671, 2022.
2020.
Hong, W., Ding, M., Zheng, W., Liu, X., and Tang, J. Cogvideo: Large-scale pretraining for text-to-video gener- ation via transformers. arXiv preprint arXiv:2205. 158688 2022.
Ephrat, A., Mosseri, I., Lang, O., Dekel, T., Wilson, K., Hassidim, A., Freeman, W. T., and Rubinstein, M. Look- ing to listen at the cocktail party: A speaker-independen audio-visual model for speech separation. arXiv preprint arXiv:/804.036/1 03619, 2018.
Hsu, W.-N. Bolte, B., Tsai, Y.-H. H., Lakhotia, K., Salakhutdinov, R., and Mohamed, A. Hubert: Self- supervised speech representation learning by masked pre- diction of hidden units. IEEE/ACM transactions on audio, speech, and language processing, 29:3451-3466 2021.
Esser, P., Kulal, S., Blattmann, A., Entezari, R., Müller, J., Saini, H., Levi, Y., Lorenz, D., Sauer, A., Boesel, F., et al. Scaling rectified flow transformers for high-resolution image synthesis. In Forty-first International Conference on Machine Learning, 2024.
Ji, X., Zhou, H., Wang, K., Wu, Q., Wu, W., Xu, F., and Cao, X. Eamm: One-shot emotional talking face via audio- based cmotion-aware motion model. In ACM SIGGRAPH 2022 Conference Proceedings, pp. 1-10, 2022.
Ji, X., Hu, X., Xu, Z., Zhu, J., Lin, C., He, Q. Zhang, J., Luo, D., Chen, Y., Lin, Q., et al. Sonic: Shifting focus to global audio perception in portrait animation. arXiv preprint arXiv:2411.16633 2024.
Gan, Y.. Yang, Z., Yue, X., Sun, L.,, and Yang, Y. Efficient emotional adaptation for audio-driven talking-head gen- eration. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 22634-22645, 2023.
Jiang, D., Chang, J., You, L.., Bian, S., Kosk, R., and Maguire, G. Audio-driven facial animation with deep learning: A survey. Information, 15(11):675, 2024a.
Gulati, A., Qin, J., Chiu, C.-C., Parmar, N., Zhang, Y., Yu, J., Han, W., Wang, S., Zhang, Z., Wu, Y., al. Con- former: Convolution-augmentte transformer for speech recognition. arXiv preprint arxiv:2005.08100, 2020.
Jiang, J., Liang, C., Yang. J., Lin, G., Zhong. T., and Zheng. Y. Loopy: Taming audio-driven portrait avatar with long-term motion dependency. arXiv preprint 24fi::2409026663 2024b.
Guo, J., Deng, J., Lattas, A., and Zafeiriou, S. Sample and computation redistribution for efficient face detection. arXiv preprint 044.44405 2021a.
Johnson, J., Alahi, A., and Fei-Fei, L. Perceptual losses for real-time style transfer and super-resolution, In Com- puter Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Pro- ceedings, Part II 14, pp. 694-711. Springer 2016.
Guo, J., Zhang, D., Liu, X., Zhong, Z., Zhang, Y., Wan, P., and Zhang. D. Liveportrait: Efficient portrait animation with stitching and retargeting control. arXiv preprint arXiv:2407.03168 2024.
Guo, Y., Chen, K., Liang, S., Liu, Y.J.., Bao, H., and Zhang, Ad-nerf: Audio driven neural radiance fields for talking
Ki, and Min, D. Stylelipsyne: Style-based personal- ized lip-syne video generation. In Proceedings of the
10

---

Playmate: Flexible Control of Portrait Animation via 3D-Implicit Space Guided Diffusion
IEEE/CVF International Conference on Computer Vision, pp. 22841-22850, 2023.
Montesinos, J. F., Kadandale, V. and Haro, G. A cappella: Audio-visual singing voice separation. arXiv preprint arXiv:2104.09946, 2021.
Kingma, D. P. Adam: A method for stochastic optimization. arXiv preprint
2014.
Mukhopadhyay, S., Suri, S., Gadde, R. T.,, and Shrivastava, A. Diff2lip: Audio conditioned diffusion models for lip-synchromization. In Proceedings of the IEEE/CVE Winter Conference on Applications of Computer Vision, pp. 5292-5302, 2024.
Kong, W. Tian, Q., Zhang, Z., Min, R., Dai, Z., Zhou, J., Xiong, J., Li, X., Wu, B., Zhang, J., et Hunyuan- video: systematic framework for large video generative models. arXiv preprint arXiv:2412.03660
2024.
Peebles, W. and Xie, S. Scalable diffusion models with transformers. In Proceedings of the IEEF/CVE Interna- tional Conference on Computer Vision, pp. 1195-4205, 2023.
Li, T., Bolkart, T., Black, M. J., Li, H., and Romero, J. Learning a model of facial shape and expression from 4d scans. ACM Trans. Graph., 36(6):194-1, 2017.
Lin, B., Ge, Y., Cheng, X., Li, Z., Zhu, B., Wang. S., He, X., Ye, Y., Yuan, S., Chen, L., al. Open-sora plan: Open-source large video generation model. arXiv preprint arXiv:2412.0011/ 2024a.
Peng, Z., Hu, W., Shi, Y., Zhu, X., Zhang, X., Zhao, H., He, J., Liu, H., and Fan, Z. Synctalk: The devil is in the syn- chronization for talking head synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 666-676, 2024.
Lin, B., Yu, Y., Ye, J., Lv, R., Yang, Y., Xie, R., Yu, P., and Zhou, H. Takin-ada: Emotion controllable audio-driven animation with canonical and landmark loss optimization. arXiv preprint ar arXiv:2410.1 14283, 2024b.
Podell, D., English, Z., Lacey. K., Blattmann, A., Dockhorn, T., Müller, J., Penna, J., and Rombach, R. Sdxl: Im- proving latent diffiusion models for high- resolution image synthesis. arXiv preprint arXiv:2307.01952 2023.
Liu, T., Chen, F., Fan, S., Du, C., Chen, Q., Chen, X., and Yu, K. Anitalker: animate vivid and diverse talking faces through ddentity-decouple facial motion encoding. In Proceedings of the 32nd ACM International Conference on Multimedia, pp. 6696-6705, 2024a.
Prajwal, K., Mukhopadhyay, R., Namboodiri, V. P., and
Jawahar, C. A lip sync expert is all you need for speech to lip generation in the wild. In Proceedings of the 28th ACM international conference on multimedia, pp. 484 492, 2020.
Liu, Y., Dai, W., Feng, C., Wang, W., Yin, G., Zeng, J., and Shan, Mafw: A large-scale, multi-modal, compound af- fective database for dynamic facial expression recognition in the wild. In Proceedings of the 30th ACM International Conference on Multimedia, pp. 24-32, 2022.
Pumarola, A., Agudo, A., Martinez, A. M., Sanfeliu, A., and Noreno-Noguer, F. Ganimation: Anatomically facial animation from single image. In Proceedings of the European conference on computer vision (ECCV), pp. 818-833, 2018.
Liu, Y., Zhang, K., Li, Y., Yan, Z., Gao, C., Chen, R., Yuan, Z., Huang, Y., Sun, H., Gao, J., et al. Sora: A review on background, technology, limitations, and opportunities of large vision models. arXiv preprint arXiv:240..17177. 2024b.
Ren, Y., Li, G., Chen, Y., Li, T. H., and Liu, S. Pirenderer: Controllable portrait image generation via semantic neu- ral rendering. In Proceedings of the IEEE/CVF interna- tional conference on computer vision, pp. 13759-13768, 2021.
Lu, Y., Chai, J., and Cao, X. Live speech portraits: real-time photorealistic talking-hewn head animation. ACM Transactions on Graphics (ToG), 10 2021.
Rombach, R., Blattmann, A., Lorenz, D.,, Esser, P., and Ommer, B. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition, pp. 10684-10695, 2022.
Ma, Y., Wang. S., Hu, Z., Fan, C., Lv, T., Ding, Y., Deng, Z., and Yu, X. Styletalk: One-shot talking head generation with controllable speaking styles. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37,
Savchenko, A. Facial expression recognition with adap- tive frame rate based on multiple testing correction. In Krause, A., Brunskill, E., Cho, K., Engelhardt, B., Sabato, S., and Scarlett, J. (eds.), Proceedings of the 40th International Conference on Machine Learning (ICML), volume 202 of Proceedings of Machine Learn- ing Research, pp. 301 30119-900 19129... PMLR, 23-29 Jul
pp. 1896-1904, 2023a.
Ma. Y., Zhang, S., Wang, J., Wang, X., Zhang, Y., and Deng, Z. Dreamtalk: When expressive talking head generation meets diffusion probabilistic models. arXiv e-prints. pp.
2rX 2023b.

---

Playmate: Flexible Control of Portrait Animation via 3D-Implicit Space Guided Diffusion
2023. WtRLhhtts://proceeeeee ngs v202/ 2.02/sawchenko23a.. htmm. html.
ngs mlr. mwrr.ppree prrss//
Wang, T.-C., Mallya, A., and Liu, M.-Y. One-shot free-view neural talking-head synthesis for video conferencing. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 10039- 10049, 2021.
Shen, S., Zhao, W., Meng, Z., Li, W., Zhu, Z., Zhou, J., and Lu, J. Difftalk: Crafting diffusion models for generalized audio-driven portraits animation. In Proceedings of the IEEF/CVF Conference on Computer Vision and Patlern Recognition, pp. 1982-1991, 2023.
Wang. X., Ruan, T., Xu, J., Guo, X., Li, J., Yan, F., Zhao, G., and Wang. C. Expression-awan neural radiance fields for high-fidelity talking portrait synthesis. Image and Vision Computing, 147:105075, 2024.
Siarohin, A., Lathuilière, S., Tulyakov, S., Ricci, E.,, and Sebe, N. First order motion model for image animation. Advances in neural information processing systems, 32, 2019.
Xu, M., Li, H., Su, Q., Shang, H., Zhang, L., Liu, C., Wang, J., Yao, Y., and Zhu, S. Hallo: Hierarchical audio-driven visual synthesis for portrait image animation. arXiv preprint arXiv:2406.0856 200.00801, 2024a.
Simonyan, K. Very deep convolutional networks for large- scale image recognition. arXiv preprint arXiv: 1556, 2014.
Xu, S., Chen, G., Guo, Y-X.,, Yang, J., Li, C., Zang, Z., Zhang, Y.. Tong, X., and Guo, B. Vasa-l: Lifelike audio- driven talking faces generated in real time, arXiv preprint arXiv:2404. 10667, 2024b.
Song. J., Meng, C., and Ermon, S. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020.
Xue, H., Luo, X., Hu, Z., Zhang, X., Xiang, X., Dai, Y., Liu, J.. Zhang, Z., Li, M., Yang. J., et al. Human motion video generation: A survey. Authorea Preprints, 2024.
Sun, X., Zhang, L., Zhu, H., Zhang, P., Zhang, B., Ji, X., Zhou, K.. Gao, D.. Bo, L., and Cao, X. Vividtalk: One- shot audio-driven talking head generation based on 3d hybrid prior. arXiv preprint arXiv:2312.0184. 2023.
Yang, Z., Teng, J., Zheng, W., Ding, M., Huang, S., Xu, J., Yang, Y., Hong, W., Zhang, X., Feng G., et al. Cogvideox: Text-to-video diffusion models with an ex- pert transformer. arXiv preprint arXiv:2408.0610 2024.
Sun, Z., Lv, T., Ye, S., Lin, M., Sheng, J., Wen, Y.-H., Yu, M., and Liu, Y.-j. Diffposetalk: Speech-driven driven stylistic 3d facial animation and head pose generation via diffusion models. ACM Transactions on Graphics (TOG), 43(4): 1-9, 2024.
Ye, Z.,, He, J., Jiang, Z., Huang, R., Huang, J., Liu, J., Ren, Y., Yin, X., Ma, Z., and Zhao, Z. Geneface++: Generalized and stable real-time audio-driven 3d talking face generation. arXiv preprint 0rXiv22305.00787 2023,
Tan, S., Ji, B., Bi, M., and Pan, Y. Edtalk: Efficient dis- entanglement for emotional talking head synthesis. In European Conference on Computer Vision, pp. 398-4166 Springer, 2024a.
Yu, J., Zhu, H., Jiang, I.., Loy, C. C.,, Cai, W., and Wu, W. Celebv-text: A large-scale facial tixt-video dataset. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14805-14814 14, 2023,
Tan, S., Ji, B., and Pan, Y. Flowvqtalker: High-quality emotional talking face generation through normalizing flow and quantization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 26317-26327, 2024b.
Zhang, R., Isola, P., Efros, A. A., Shechtman, E., and Wang, O. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 586-595, 2018.
Tian, L., Wang, Q., Zhang, B., and Bo, L. Emo: Emote portrait alive generating expressive portrait videos with audio2video diffusion model under weak conditions. In European Conference on Computer Vision, pp. 264-260. Springer, 2025.
Zhang, W. Cun, X., Wang, X., Zhang, Y., Shen, X., Guo, Y., Shan, Y., and Wang, F. Sadtalker: Learning realistic 3d motion coefficients for stylized audio-driven single image talking face animation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8652-8661, 2023a.
Unterthiner, T., van Steenkiste, S., Kurach, K., Marinier, R., Michalski, M., and Gelly, S. Fvd: A new metric for video generation. 2019.
Wang, K., Wu, Q. Song, I.., Yang, Z., Wu, W., Qian, C., He, R., Qiao, Y., and Loy, C. C. Mead: A large-scale audio- visual dataset for emotional talking-face face generation. In European Conference on Computer Vision, pp. 700-717. Springer, 2020.
Zhang. 7., Li, L.,, Ding, Y., and Fan, C. Fllo-guided one- shot talking face generation with a high high-resolution audio- visual dataset. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pp. 3661-3670, 2021.
12

---

Playmate: Flexible Control of Portrait Animation via 3D-Implicit Space Guided Diffusion
Zhang, Z.,, Hu, Z., Deng, W., Fan, C., Lv, T., and Ding, Y. Dinet: Deformation inpainting network for realistic face visually dubbing on high resolution video. In Proceed- ings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 3543-3551, 2023b.
Zheng, I.., Zhang, Y., Guo, H., Pan, J., Tan, 7.,, Lu, J., Tang, C., An, B., and Yan, S. Memo: Memory-guided diffusion for expressive talking video generation. arXiv preprint 2rXiv:2412.04445 2024.
13

---

Playmate: Flexible Control of Portrait Animation via 3D-Implicit Space Guided Diffusion
A. Appendix
this appendix, we provide the following materials:
1. More visual comparisons of different methods. Refer Appendix A.1 in the appendix.
2. More details of Gaussian noise augmentation. Refer to Appendix A.2 in the appendix.
3. Evaluate the accuracy of pose conditional control. Refer to Appendix A.3 in the appendix.
4. Disentanglement of facial representation. Refer to Appendix A.4 in the appendix.
Ref Image
Playmate
Hallo
Hallo2
JoyVASA
MEMO
Sonic
Figure More Visual Comparisons. Each of images represent the generation results of different methods using the same reference image and the same driving audio.
14

---

Playmate: Flexible Control of Portrait Animation via 3D-Implicit Space Guided Diffusion
A.1. More Visual Comparisons
To more comprchensively and objectively demonstrate Playmate's generation results, we compare the performance of Playmate and several other methods across wider range of images. As shown in Figure 8, we present a comparison of the effectiveness of Playmate with other current methods in generating talking videos. As evident from the illustration, Hallo and Hallo2 exhibit noticeable artifacts in video quality, characterized by insufficient clarity in teeth generation and inaccurate lip-sync, as exemplified by the images in row 4, column 3 and row 2, column 4 respectively. Joy VASA, despite utilizing a 3D-Implicit Space approach for guidance, falls short in terms of pose and facial motion, resulting in notable head distortions, as exemplified by the images in row 1, column and row 3, column MEMO, on the other hand, struggles with exposure issues, often presenting artifacts on both the face and background, as exemplified by the images in row 2, column and row 7, column 6. Finally, Sonic demonstrates good lip-syne and vivid expressions but still grapples with blurred teeth, as exemplified by the images in row 1, column 7 and row 4, column 7. In sharp contrast, our proposed method showcases the ability to generate more natural facial expressions and head movements that are well-synchronized with audio inputs. Additionally, videos produced by Playmate exhibit superior overall visual quality and stronger identity consistency.
A.2. Gaussian Noise Augmentation
We additionally utilize the speech and motion representation of the previous video frames as input to the audio conditioned diffusion transformer, aiming to maintain the continuity of the generated video. But in this incremental generation process, some contaminations of the previously generated video frames, such as the background noise and subtle distortions in facial expressions, will propagate to subsequent frames and continue to amplify artifacts. In order to enhance the resistance of our diffusion transformer to the above contaminations, we similar to Hallo2 incorporate Gaussian noise into the prev motion representations
prev
aarterre + (((0, I). (0, 50),
where Prev represents the motion representations of the previous frame, and c represents the Gaussian noise of the corresponding frame. The Gaussian noise follows the normal distribution ( I), which I denotes the identity matrix. These noise-augmented motion representations are concatenated with current noisy motions my and jointly participate the diffusion process. Specifically, each denoising step can be described as:
Eg (me, Ja, Saa, 1) = ED (nn me), S1a5,
(10)
where ég (ma, Sa. Sid, 1) represents the noise component predicted by Audio Conditioned Diffusion Transformer, and ʃ represents the conditioning inputs of audio feature and identity feature. Through this noise-augmented operation, our diffusion transformer more robust to slight changes in the motion input, thereby mitigating the impact of contaminations from previously generated video frames.
We conducted ablation study on Gaussian noise augmentation, illustrated in Figure 9. Without this augmentation strategy, once artifacts emerged in the generated image, these flaws would propagate, leading to flawed generations in subsequent images. After applying the augmentation strategy, we observed significant improvement in the quality of the generated images.
1s
5s
(N-1)s
Ns
without Gaussian Noise Augmentation
with Gaussian Noise
Augmentation
Figure 9, Ablation study of the Gaussian noise augmentation.
15

---

Playmate: Flexible Control of Portrait Animation via 3D-Implicit Space Guided Diffusion
A.3. Evaluation of Pose Control
In the field of image animation, the mean keypoint distance (AKD) and the average pose distance (APD) are commonly used evaluate the performance of pose control. We calculated the APD metric of Playmate two datasets (HDTF and dataset), shown in the table below.
Table 5. Evaluation Pose Control.
Dataset
APD 3.003"
pPDptt
APD
HDTF
1.308
1.214"
Collected dataset
3.714
1.751
1.398
A.4. Disentanglement of Facial Representation
Figure 10 shows the effective disentanglement of the latents of the face. From top to bottom: the raw generated sequence, applying generated poses with fixed initial facial dynamics, and applying generated facial dynamics with fixed initial head pose and predefined spinning poses.
Figure 10. Disentanglement between head pose and facial dynamics.
16


Reviews:
[{"review_id": "CGZc9CUf9x", "reviewer": "Reviewer_54Zx", "summary": "This work targets to generate lifelike talking videos for arbitrary identity, guided by a speech clip. Emotional and pose conditions are carefully devised to control the talking status. Specifically, a motion-decoupled module and emotion-control module are designed to enhance the performance.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "First, we would like to thank the reviewer for your careful reading and providing numerous constructive comments! Below we address the concerns mentioned in the review.\n\n**W1:The attached website does not include head pose control videos.**\n\nThank you for pointing this out. We have uploaded multiple pose control videos and emotion control videos on our anonymous project website. For the pose control videos([video1](https://playmate111.github.io/videos/pose_control_demo/female.mp4) and [video2](https://playmate111.github.io/videos/pose_control_demo/male.mp4)), the lip sync is synchronized with the audio input, whereas the pose control is achieved through various driving mechanisms(e.g., driving video, preset mode, fixed pose). Regarding the emotion control videos, we have compared our method with several approaches([video3](https://playmate111.github.io/videos/emotion_vs/male_10s.mp4) and [video4](https://playmate111.github.io/videos/emotion_vs/female.mp4)). Additionally, we conducted quantitative comparisons on emotion control accuracy and video quality, please refer to our response to Reviewer SMci's Q2.\n\n**W2:Line 215 includes some spelling mistakes. “to enhance”.**\n\nThank you very much for the reminder. We will correct these issues in the revised version and also carefully revise the full text.\n\n**Q1:The training stability of the transfer loss.**\n\nRegarding the training stability, since we use $\\mathcal{M}$ from LivePortrait as pre-trained model, which inherently possesses face attribute disentanglement capabilities, fine-tuning with transfer loss ensures both training stability and rapid convergence. We have uploaded sample image([image1](https://playmate111.github.io/src/Disentanglement.png)) on website to demonstrate disentanglement between head pose and facial dynamics. \n\n**Q2:Evaluate the accuracy of pose conditional control.**\n\nIn the field of image animation, Average Keypoint Distance (AKD) and Average Pose Distance (APD) are commonly used to evaluate pose control performance. We calculated the APD metric of Playmate on two datasets (HDTF and our dataset), as shown in the tables below. \n\n|Dataset|APD-jaw|APD-pitch|APD-roll|\n|----|:----:|:----:|:----:|\n|HDTF|${3.003}^\\circ$|${1.308}^\\circ$|${1.214}^\\circ$|\n|Our Dataset|${3.714}^\\circ$|${1.751}^\\circ$|${1.398}^\\circ$|\n\nAdditionally, we have uploaded sample pose visualization videos on our anonymous project website([video5](https://playmate111.github.io/videos/poseControl/32.mp4) and [video6](https://playmate111.github.io/videos/poseControl/2.mp4)), demonstrating that Playmate achieves good generation quality even in pure pose-driven scenarios.\n\n**Q3:In the inference stage, how does denormalization operate?**\n\nIn the inference stage, for expression, the mean and standard deviation are the same used during training, calculated from all training data. For head pose, the mean and standard deviation are optional and can be computed from user-provided driving videos. If not provided, they can either be set to default parameters(e.g., computed from forward-looking videos) or calculated from randomly selected videos in the dataset. \n\n**Q4:About the collected dataset.**\n\nOur own dataset involved about 39k clips, featuring more complex scenes, diverse styles, higher clarity, and greater challenges, which leads to worse overall performance for most approaches. We'll try our best to release the collected dataset, but we hope you understand that both we and the community need to exercise caution when releasing rich datasets. This caution stems from concerns regarding potential risks, particularly those related to individual privacy and likeness rights.\n\n**Q5:The poses do not showcase too much variability.**\n\nThank you for your interest in this section. In fact, we support multiple pose driving methods. As shown in our newly uploaded sample videos, under the same audio, different pose control conditions result in different poses generated by Playmate. We support various methods to enhance pose variability, such as using videos with significant pose variations as driving videos, or manually adjusting the rotation value of the pose.\n\n**Q6:It works surprisingly well with only two DiT blocks inserted before the MLP head for emotional control, any explanations?**\n\nThank you for your affirmation of our emotion control module. We believe its effectiveness is due to two main reasons: (1) Precise latent space construction and attribute disentanglement, enabling the model to effectively correlate emotion control with emotion features after decoupling attributes like pose and expression, leading to effective learning; (2) A two-stage training approach, where we first stabilize audio-driven training, then train the emotion control module separately while keeping most weights fixed, reducing training complexity. Training all parameters simultaneously might prioritize other labels, rendering the emotion control signals ineffective.", "claims_and_evidence": "Authors claim superior talking head performance and compare it with state-of-art approaches. However, for pose control and emotion control, the comparison and experiment validation are missing.", "methods_and_evaluation_criteria": "The overall approach is composed with two components. One is the enhanced disentanglement in latent space while the other is audio-emotion conditioned diffusion transformer. Both are adaptively modified to accomplish current function. The talking video evaluation criteria follows previous approaches and is plausible.", "theoretical_claims": "There are no theoretical claims involved in this paper.", "experimental_designs_or_analyses": "1. Authors design emotional control strategies but it seems to lack comparison with emotion-condition based state-of-art approaches.", "relation_to_broader_scientific_literature": "This approach introduces a diffusion transformer to the talking head generation field. It is interesting for the community to know such a design can enhance the talking video generation performance.", "essential_references_not_discussed": "N.A", "other_strengths_and_weaknesses": "1. The attached website does not include head pose control videos. It is hard to evaluate the pose-control performance. \n\n2. Line 215 includes some spelling mistakes. “to enhance”.", "other_comments_or_suggestions": "N.A", "questions_for_authors": "1. How is the training stability of applying head pose and facial dynamics transfer loss in equation 4? Any strategy to evaluate its effectiveness?\n\n2. Is there any possibility to evaluate the accuracy of pose conditional control? \nFor instance, facial reconstruction algorithms for the head pose angles. \n\n3. For the motion-decoupled module, authors introduce adaptive normalization. In the inference stage, how does denormalization operate? \n\n4. About the collected dataset, it seems the overall performance becomes worse for most approaches. What is the difference between the collected dataset and the HDTF dataset? How many videos are involved in the collected datasets? Will this dataset be released? \n\n5. The presented videos show high image quality, but the poses do not showcase too much variability. \n\n6. It works surprisingly well with only two DiT blocks inserted before the MLP head for emotional control, any explanations?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nE2yDuqTWL", "reviewer": "Reviewer_WRyG", "summary": "The work introduces Playmat, a diffusion transformer based talking face generation model. Playmate is able to generate talking heads (portrait animation) given reference image and audio signal, as well as an emotion signal. It splits training into two stages, first training the talking face generation model (diffusion transformer backbone), and then training an emotional control module while keeping the backbone fixed. Playmate makes use of the same 3D motion representation from face-vid2iid and Liveportrait. The results demonstrate competitive quantative and qualitative performance of the method, with good video generation quality.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful for your review and valuable comments, and we hope our response fully resolves your concerns.\n\n**Q1:precise motion decoupling... (a) confusing and (b) lacking evidence...**\n\nAbout the motion decoupling. We emphasize motion decoupling because it is the foundation for Playmate to achieve various flexible driving combinations. When multiple control conditions exist, we can provide multiple driving modes. For example, a driving video can be used to control the pose, while audio is used to drive lip movements. This is also the most significant distinction between Playmate and methods like Sonic and EMO. Sonic only takes a reference image and audio as inputs, with the generated results (lip, pose, and expressions) all driven by the audio, without allowing users to specify different driving methods. We have uploaded multiple pose control videos and emotion control videos on our anonymous project website([video1](https://playmate111.github.io/videos/pose_control_demo/female.mp4) and [video2](https://playmate111.github.io/videos/pose_control_demo/male.mp4)). The lip sync is synchronized with the audio input, whereas the pose control is achieved through various driving mechanisms(e.g., driving video, preset mode, fixed pose).\n\n**Q2:state-of-the-art lip synchronization/superior video quality...Sonic achieves superior performance in this regard...**\n\nThank you very much for the reminder. Although our quantitative metrics for lip synchronization on the two test sets are marginally lower than Sonic's, they still outperform other compared methods. Furthermore, across the remaining four comparison metrics, our method consistently exceeds the performance of all other competing approaches. We have uploaded more comparison videos with Sonic to our anonymous project website([video3](https://playmate111.github.io/videos/vs_sonic/0.mp4), [video4](https://playmate111.github.io/videos/vs_sonic/1.mp4), [video5](https://playmate111.github.io/videos/vs_sonic/2.mp4) and [video6](https://playmate111.github.io/videos/vs_sonic/3.mp4)). These videos show that, qualitatively, our lip synchronization is not far behind Sonic's. Moreover, in terms of video clarity, we are significantly better, especially in areas like the teeth. Regarding the differences from Sonic, let us briefly explain here. Sonic is a purely audio-driven algorithm that generates all features of the portrait, including lip movements, expressions, pose, etc., based on audio. This indicates that its driving flexibility is limited. In contrast, we achieve multiple controllable portrait driving methods by constructing a precise attribute disentanglement space, providing users with various flexible driving options. The implementation difficulty of this decoupling and then driving approach is higher than that of a simple audio-driven method. This is also the reason why we emphasize \"Flexible Control\" in our paper title.\n\n**Q3:About the user study.**\n\nThank you for your suggestion. This question was also raised by Reviewer SMci. Please refer to our response to Reviewer SMci's question Q4.\n\n**Q4:While quantitative results show good performance on image/video metrics...This conflicting language is misleading to the actual performance of the model.**\n\nThank you very much for the reminder. We will correct these issues in the revised version and will also carefully revise the full text.\n\n**Q5:Further, analysis that video quality is superior to that of other methods, while...**\n\nFor details on the comparison with Sonic, see the response provided in Q2.\n\n**Q6:the strength of the novelty of the work however is not proven...**\n\nThank you for your skepticism. In this response, we have added multiple tests(including responses to other reviewers), and uploaded more visual results on anonymous website, hoping that these results will dispel your doubts and concerns in this regard.\n\n**Q7:About the Adaptive Normalization.**\n\nRegarding Adaptive Normalization, our approach focuses on adapting to the dimensions of facial attributes. We apply distinct means and standard deviations for pose and expression, which provides additional prior information and reduces the learning complexity for the model. This facilitates more flexible control over the generated outputs. In the inference stage, we have the flexibility to derive these means and standard deviations from various available sources, thereby enabling more precise and controllable driving effects.\n\n**Q8:At inference, which private mean/std is used...**\n\nThis question was also raised by Reviewer 54Zx. Please refer to our response to Reviewer 54Zx's question Q3.\n\n**W1:The writing, particularly around claims and introduction of...**\n\nWe will correct these issues in the revised version and will also carefully revise the full text.\n\n**W2:the strength of the novelty of the work however is not proven in experiments in my opinion, and...**\n\nDue to the character limit on response, please refer to our response to Q6.", "claims_and_evidence": "- **precise motion decoupling**: This line is used throughout the paper however I found its explanation (a) confusing and (b) lacking evidence. Namely, it is described to decouple expression and head pose, however section 4.3 does not explicitly investigate this. Section 4.3 performs an image qualitative analysis of certain failure cases with and without the adaptive norm, however I fail to see how these disjoint artifacts related to improved pose and expression decoupling. The lack of video in the supplementary material makes further investigation impossible.\n- **state-of-the-art lip synchronization**/**superior video quality**: as shown both quantitatively and qualitatively, this is not the case, and the strength of this claim is not validated. Sonic achieves superior performance in this regard. Further, qualitative videos in the supplementary material demonstrate still that Sonic has better video quality given its overall expression and pose quality over Playmate, in my opinion.", "methods_and_evaluation_criteria": "In general, yes. The metrics and datasets follow related art, and the baselines are sound choices.", "theoretical_claims": "The paper does not make any theoretical claims.", "experimental_designs_or_analyses": "In general, the experimental designs are sound, with the exception of the following:\n- It is common practice in this field to include a user study. While quantitative results are good, and performing good on benchmarks is important, it is not the full picture. A user study enables quantifying the qualitative results of the work in terms of user preference, which is the intended use case of this technology, and is critical to evaluating the quality of the method.\n- While quantitative results show good performance on image/video metrics, they fall short in lip synchronization. This is in contrast to the repeated claim and analysis that Playmate achieves state-of-the-art performance in terms of lip synchronization (abstract/L50/col#2, conclusion/L414/col#2) - note however that caption in Figure 2 makes the alternative \"competitive\" claim. This conflicting language is misleading to the actual performance of the model.\n- Further, analysis that video quality is superior to that of other methods, while shown quantitatively, is not supported qualitatively. Primarily, the method falls short in lip sync and expression realism compared mainly to Sonic. in talking head work like this, video quality is highly correlated to the animation quality, and I am not convinced of Playmates results. Playmate also suffers from odd artifacts and expressions, as shown in the videos on the supplementary material website, often have mouth ajar, or eyes bulging.\n    - These artifacts reduce the impact of analyses towards \"precise motion decoupling\" as well\n    - This further highlights the importance of a user study", "relation_to_broader_scientific_literature": "The contributions of this work are primarily scoped to talking head generation. The techniques used in this work pull from the broader literature, however the contributions do not generalize beyond their scope. This statement is made objectively, and does not demote the work itself.", "essential_references_not_discussed": "Related art appears sound.", "other_strengths_and_weaknesses": "- **Strength**: The paper does introduce a novel orchestration of components into a novel training framework and architecture. This is a strength, and is a good combination of prior with a few new components.\n- **Strength**: the emotion control does seem to work well.\n- **Weakness*: The writing, particularly around claims and introduction of components is often embellished. The language used is often \"gratuitously grandiose\" - I use this language primarily to illustrate my meaning. This embellishment feeds the narrative of overselling the work's contributions. This is a minor concern and could be addressed in revision. For example,\n    - \"meticulously designed\": over embellishing language\n    - \"specialized Proj modules\": these are just linear layers it appears\n    - The authors often say they \"introduce\" a component, which is misleading terminology that would indicate this is a novel introduction of this technique, when it is in fact not\n- **Weakness**: the strength of the novelty of the work however is not proven in experiments in my opinion, and the qualitative results do not convince me. While emotion control seems to work well, the overall expression quality is not great and a little uncanny still.", "other_comments_or_suggestions": "see other.", "questions_for_authors": "- I am confused about you \"Adaptive Normalization\" - from the definition, using pre-computed global and private mead/std values is not adaptive, and is instead fixed. Adaptive Normalization, as the term is normally used for, involves learnable parameters in the network, as is commonly used for DiT models. Can you please explain this further, or make a distinction between the two concepts?\n    - Further, at inference, which private mean/std is used? How does this work for unseen speakers?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fhSgGkxkpO", "reviewer": "Reviewer_SMci", "summary": "This paper proposed a novel audio-driven DiT-based portrait animation pipeline with customized emotion control and driving video control. The major contributions are 1) a motion-decoupled module with perceptual loss and adaptive normalization, 2) an emotion-control module with DiT blocks, and 3) an implicit 3D decoupled face representation. The experimental results of video visualization demonstrate the superiority of the Playmate framework.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback. We appreciate your recognition of our method's innovation and applicability. Here are our responses to your comments.\n\n**Q1:However, it's unclear how the head pose and facial dynamics transfer and the perceptual loss improve the disentanglement, is there any detailed explanation for this?**\n\nOur transfer loss is primarily inspired by DPE[1] and VASA-1[2]. DPE introduces a bidirectional cyclic training strategy akin to CycleGAN's A2B2A' and B2A2B' pathways to achieve the disentanglement of pose and expression. Our approach is similar but constructs bidirectional, non-cyclic pathways. Specifically, we randomly sample two frames $I$ and $J$ from a video clip; these frames are characterized by their expression and pose attributes: $exp_i$, $pose_i$ for frame $I$, and $exp_j$, $pose_j$ for frame $J$. After applying attribute transfer, we obtain images $I_{exp_i,pose_j}$ and $J_{exp_j,pose_i}$. If the attribute transfer technique is perfect, we should obtain two portrait images with identical expressions and poses. To achieve this goal, we apply a perceptual loss to make the synthetic results appear more realistic.\n\n\\[1] Pang Y, Zhang Y, Quan W, et al. Dpe: Disentanglement of pose and expression for general video portrait editing\n\n\\[2] Xu S, Chen G, Guo Y X, et al. Vasa-1: Lifelike audio-driven talking faces generated in real time\n\n**Q2:Alongside the ablations on CFG scales (Table 2) and the Adaptive Norm (Figure 7), it would be beneficial to include a quantitative analysis of each proposed module—specifically, the perceptual loss and the emotion-control module.**\n\nThank you very much for the reminder.\n\n(1)About the transfer loss, the purpose of this loss is to achieve more precise disentanglement of facial attributes. Due to time constraints, we calculated the APD (Average Pose Distance) metric of Playmate on two datasets (HDTF and our dataset), as shown in the table below. \n\n|Dataset|APD-jaw|APD-pitch|APD-roll|\n|----|:----:|:----:|:----:|\n|HDTF|${3.003}^\\circ$|${1.308}^\\circ$|${1.214}^\\circ$|\n|Our Dataset|${3.714}^\\circ$|${1.751}^\\circ$|${1.398}^\\circ$| \n\n(2)About the emotion-control module, we have compared our method with several approaches, as shown in the table below, demonstrating the effectiveness of our expression control and achieving superior generation quality in terms of emotional expression.\n\n|Methods|FID$\\downarrow$|FVD$\\downarrow$|LPIPS$\\downarrow$|Acc(Emo)$\\uparrow$|\n|----|:----:|:----:|:----:|:----:|\n|EAMM|111.710|210.275|0.223|0.160|\n|EAT|95.085|166.316|0.138|0.450|\n|DreamTalk|119.032|199.962|0.246|0.350|\n|EDTalk|135.215|221.897|0.289|0.460|\n|Playmate|68.234|149.837|0.112|0.550|\n\n**Q3:How does the method handle scenarios in which both the audio and the driving video provide conditions for localized lip movement? Is there a mechanism to balance or integrate these dual inputs?**\n\nSince in most cases users expect to use audio to drive lip movements, when both audio and a driving video are available as inputs, we default to using audio for driving lip movements and expressions, while using the driving video to control the pose. In fact, we support multiple driving modes. For example, audio can drive lip movements, while the driving video controls the expression and pose, and vice versa. As shown in [video1](https://playmate111.github.io/videos/pose_control_demo/female.mp4) and [video2](https://playmate111.github.io/videos/pose_control_demo/male.mp4), the first row displays the reference image, the first column shows the pose control mode, and the remaining cells present the generated results. The lip sync is synchronized with the audio input, whereas the pose control is achieved through various driving mechanisms (e.g., driving video, preset mode, fixed pose).\n\n**Q4:Although the supplementary website offers visual comparisons, a user study could further substantiate the effectiveness of the proposed Playmate framework in comparison to baseline methods.**\n\nThank you for your suggestion. Due to time constraints, we conducted a user study involving 10 participants who rated videos using the MOS (Mean Opinion Score) rating method, on a scale of 1 to 5, across four metrics: Lip Sync (LS), Video Definition (VD), Naturalness (N), and Visual Appeal (VA). As illustrated in the table below, Playmate has a notable advantage in the VD and VA metrics. While the LS and N metrics are slightly lower than Sonic's, they still outperform those of other methods, showcasing Playmate's strong competitiveness.\n\n|Methods|LS$\\uparrow$|VD$\\uparrow$|N$\\uparrow$|VA$\\uparrow$|\n|----|:----:|:----:|:----:|:----:|\n|JoyVASA|2.500|2.286|1.714|1.929|\n|Hallo|2.964|2.929|3.071|2.893|\n|Hallo2|3.036|2.929|2.893|2.786|\n|MEMO|3.321|3.036|3.179|3.143|\n|Sonic|3.821|3.071|3.750|3.500|\n|Playmate|3.750|3.857|3.464|3.643|", "claims_and_evidence": "The claims are supported by qualitative visualizations such as videos and figures, as well as quantitative comparison and ablation analysis.", "methods_and_evaluation_criteria": "The two-stage framework is well-explained, and the evaluation provides a comprehensive comparison of SOTA methods in both the HDTF benchmark and the self-collected dataset. However, it's unclear how the head pose and facial dynamics transfer and the perceptual loss improve the disentanglement, is there any detailed explanation for this?", "theoretical_claims": "I have checked the theoretical formula in the method section for both perceptual loss and Adaptive Normalization.", "experimental_designs_or_analyses": "I appreciate the authors for providing both video visualizations and benchmark comparison tables. However, I still have a few minor concerns:\n\n1) Alongside the ablations on CFG scales (Table 2) and the Adaptive Norm (Figure 7), it would be beneficial to include a quantitative analysis of each proposed module—specifically, the perceptual loss and the emotion-control module.\n\n2) How does the method handle scenarios in which both the audio and the driving video provide conditions for localized lip movement? Is there a mechanism to balance or integrate these dual inputs?\n\n3) Although the supplementary website offers visual comparisons, a user study could further substantiate the effectiveness of the proposed Playmate framework in comparison to baseline methods. (Optional)", "relation_to_broader_scientific_literature": "The proposed method contributes to the portrait animation with more flexible control.", "essential_references_not_discussed": "I believe all essential references have been included already.", "other_strengths_and_weaknesses": "Please see the \"Methods And Evaluation Criteria\" and \"Experimental Designs Or Analyses\" sections.", "other_comments_or_suggestions": "In general, I believe this is a well-written paper in good shape. The existing experimental result is convincing and demonstrates expressive facial motions and head movements. The method is novel and offers flexible control.", "questions_for_authors": "Please see the \"Methods And Evaluation Criteria\" and \"Experimental Designs Or Analyses\" sections.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper presents ​Playmate, a novel DiT-based framework for portrait animation with fine-grained emotion and motion control. The work makes contributions in: 1) Motion-Decoupled Module – Leveraging perceptual loss and adaptive normalization to disentangle motion dynamics effectively. 2) ​Emotion-Control Module – Integrating DiT blocks for precise emotional expression manipulation. The rebuttal addressed some concerns, with one reviewer score raising from 2 to 3. After reading the whole review process and careful discussion, the AC can recommend ​accept. 

Authors are suggested to incorporate all new materials from the rebuttal into the paper. Especially the performance difference with Sonic is noticable both quantitatively (also in the added user study in rebuttal) and visually. The gap is noticed by both the reviewers and the AC. The author's explanation of the controllability features in this method make AC still lean towards Accept.
Decision: Accept (poster)