CyberHost: A One-stage Diffusion Framework for Audio-driven Talking Body Generation
Authors: Gaojie Lin
Jianwen Jiang
Chao Liang
Tianyun Zhong
Jiaqi Yang
Zerong Zheng
Yanbo Zheng
Venue: ICLR 2025 Oral (2025)
Keywords: Audio-driven Human Animation.+Diffusion Model.+Generative Model.+Human Video Generation
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: We propose a one-stage audio-driven talking body generation framework, CyberHost, designed to produce human videos that match the input audio with high expressiveness and realism.

Abstract: Diffusion-based video generation technology has advanced significantly, catalyzing a proliferation of research in human animation. While breakthroughs have been made in driving human animation through various modalities for portraits, most of current solutions for human body animation still focus on video-driven methods, leaving audio-driven taking body generation relatively underexplored. In this paper, we introduce CyberHost, a one-stage audio-driven talking body generation framework that addresses common synthesis degradations in half-body animation, including hand integrity, identity consistency, and natural motion.
CyberHost's key designs are twofold. Firstly, the Region Attention Module (RAM) maintains a set of learnable, implicit, identity-agnostic latent features and combines them with identity-specific local visual features to enhance the synthesis of critical local regions. Secondly, the Human-Prior-Guided Conditions introduce more human structural priors into the model, reducing uncertainty in generated motion patterns and thereby improving the stability of the generated videos.
To our knowledge, CyberHost is the first one-stage audio-driven human diffusion model capable of zero-shot video generation for the human body. Extensive experiments demonstrate that CyberHost surpasses previous works in both quantitative and qualitative aspects. CyberHost can also be extended to video-driven and audio-video hybrid-driven scenarios, achieving similarly satisfactory results.

Published as a conference paper at ICLR 2025
CYBERHOST: A ONE-STAGE DIFFUSION FRAMEWORK FOR AUDIO-DRIVEL TALKING BODY GENERATION
Gaojie Lin'; Jiianen Jiang' Chao Liang', Tianyun Zhonga! Jiaqi Yang', Yanbo Zheng'
'ByteDance, *Zhejiang University {Lingae jianwen... alan, Liangehao. o412)agmai..com
zhongtianyund edu.cn
ABSTRACT
Diffusion-based video generation technology has advanced significantly, catalyz- ing a proliferation research in human animation. While breakthroughs have been made in driving human animation through various modalities for portraits, most of current solutions for human body animation still focus on video-driven methods, leaving audio-driven taking body generation relatively underexplored. In this paper, we introduce CyberHost, a one-stage audio-driven talking body generation framework that addresses common synthesis degradations in half-body animation, including hand integrity, identity consistency, and natural motion. Cy- berHost's key designs twofold. Firstly, the Region Attention Module (RAM) maintains a set of learnable, implicit, idgntity-agnostic latent features and com- bines them with identity-speeific local visual features to enhance the synthesis of critical local regions. Secondly, the more human structural priors into the model, reducing uncertainty in generated motion patterns and thereby improving the stability of the generated videos. To our knowledge, CyberHost is the first one-stage audio-driven human diffusion model capable of zero-shot video generation for the human body. Extensive ex- periments demonstrate that CyberHost surpasses previous works in both quanti- tative and qualitative aspects. CyberHost can also be extended to video-driven and audio-video -ybrid-driven scenarios, achieving similarly satisfactory results. Video samples available at https:/cyberhost.github..i
Conditions introduce
INTRODUCTION
Human animation aims to generate realistic and natural human videos from single image and control signals such as audio, text, and pose sequences. Previous works (Prajwal al., [0200] Yin
eet al. [022] Wang et al., 20211 Ma et al. [2023] Zhang al. 2023 Chen et al. [202bb] Xu et al.)
2024b a Tian et al. 2024 Jiang et al. [024a] Wang et al. [0024]] Jiang et al. (0244b have primarily focused on generating talking head videos based varied input modalities. Among these, audio- driven methods have recently attracted significant interest, particularly those employing diffusion models (Tian et al. 2024 Xu al. 2024a b). While these methods yield impressive results, they are specifically tailored for portrait scenarios, making it challenging to extend them to half- body scenarios to achieve audio-driven talking body generation. This is because generating talking body videos involves more intricate human appearance details and complex motion patterns.
On the other end of the spectrum, some recent studies (Karras al. 2023 Wang et al. 2024b Hu et al., 2024 Zhang et al.) 2024; Xu et al. 2024c; Huang et al., 2024 Corona et al.] 2024) fo- cus on tackling video-driven body animation. Unlike audio-driven settings, these methods rely on pose conditions to provide precise. pixel-aligned body structure prior, enabling the modeling and generation of large-scale human movements and fine- ome-grained local details. Even so, accurately gen- erating detailed body parts remains challenging, as shown in Figure [ Video-driven body animation methods often require additional motion generation modules and retargeting techniques, which limit
* "Equal contribution.
I Project lead
Done during an internship ByteDance.

---

Published as a conference paper at ICLR 2025
Source Image
MimicMotion (V2V)
Ours (V2V)
Source Image
DiffGiesture +Mimic Motion (A2V)
Ours (A2V)
Figure 1: Existing body animation methods struggle to generate detailed hand and facial results in both video- driven (V2V) and audio-driven (A2V) settings. In contrast, our approach ensures hand integrity and facial identity consistency. These differences are also illustrated with videos the supplementary materials.
their practical applications. Recent works (Liao al., 2020; Wang et al. 2023c Hogue al. 2024
Corona et al., 2024) have explored the implementation of two-stage systems to achieve dudie-driven talking body generation. This generally consists of audio-to-pose module and pose-to-vides module, using poses or meshes as intermediate representations. Nevertheless, this approach faces several critical limitations: (I) The two-stage framework design increases system complexity and reduces the model's learning efficiency. (2) The poses or meshes carry limited information related to expressiveness, constraining the model's ability capture subtle human nuances. (3) Potential inaccuracies in pose or mesh annotations can diminish the model's performance. Therefore, there an urgent need to explore how optimize the generation quality of talking body video within one-stage audio-driven framework.
In this paper, we aim address one-stage talking body generation, topic that remains unexplored in current literature. The challenge lies in two aspects: 1) Details Underfitting. Unlike video-driven methods, capturing local structural details from audio signals is difficult, making it harder to ensure the integrity of body parts, such as the face and hands. Moreover, critical human body parts occupy only a small portion of the frame but carry the majority of the identity information and semantic expression. Unfortunately, neural networks often fail to spontaneously prioritize learning in these key regions, intensifying the issue of underfitting the generation of local details. 2) Motion Uncer- tainty. Unlike portrait animation, body animation encompasses higher degree of motion freedom and exhibits a weaker correlation between audio cues and limb movement patterns. Consequently, predicting body movements from audio signals introduces more substantial one-to-many problem, leading to significant uncertainty in the motion generation process. This uncertainty exacerbates instability in the generated talking body videos, thereby complicating the direct adaptation of audio- driven portrait animation techniques to half-body scenarios.
To address these two challenges. propose CyberHost, one-stage audio-driven talking body framework capable of zero-shot human videos generation. On one hand, CyberHost introduces region attention module (RAM) address the issue of underfitting local details. Specifically, RAM utilizes learnable spatio-temporal latents bank to capture common local human details from the data, such as topological structures and motion patterns, thereby ensuring the maintenance of structural details. Additionally, it integrates appearance features from local cropped images, serving as identity descriptors, to supplement the identity-specific texture details. On the other hand, to address the motion uncertainty problem, we designed tuman-prior-guided conditions to incorporate motion pattern constraints and human structural priors into the human video generation process. Specifically, for global motion. propose body movement map to constrain the motion space of the human root node, and for local motion. we introduce a hand clarity score to mitigate hand degradation caused by motion blur. Additionally, for human structure, utilize the skeleton map
2

---

Published as a conference paper at ICLR 2025
of the reference image extract pose-aligned reference features, thereby providing the model with initial pose information from the reference image.
In experiments, we validated the effectiveness of the region attention modules and human- prior-guide guided conditions, Both qualitative and quantitative experiments demonstrate that CyberHost achieves superior results compared to existing methods. Moreover, we validated the exceptional per- formance of CyberHost in various settings, including audio-driven, video-driven, and mu multimodal- driven scenarios, as well as its zero-shot video generation capability for open-set images.
We summarize our technical contributions as follows: 1) We propose the first one-stage audio- driven talking body framework enabling zero-shot human body animation without relying any intermediate representations, and validate its effectiveness across multiple scenarios. 2) We crafted region attention module (RAM) to enhance the generation quality of key local regions such as hands and faces, by including a spatio-temporal latents bank to learn shared local structural details and an identity descriptor to supplement DD-specific texture details. 3) We designed suite of human prioor-- guided conditions mitigate the instability caused by motion uncertainty in audio-driven settings.
RELATED WORK
Benefiting from the advancements in diffusion models, video generation has made significant progress in recent years. Some early works (Singer et al. 2022] Blattmann et al.) 2023a Zhou et al. 2022: He al. 2022 Wang et al. 2023a have attempted to directly extend the 2D U-Net pretrained on text-to-image tasks into 3D) to generate continuous video segments. Ani- mateDiff (Guo al. 2024) trained pluggable temporal module on large-scale video data, allowing easy application to other text-to-image backbones and enabling text-to-video generation with min- imal fine-tuning. For controllability, (Wang al.) 2023b) trained Composer Fusion Encoder to integrate multiple modalities of input as control conditions, thereby making video generation for complex scenes such as human bodies more controllable.
Video Generation.
Body Animation. Existing body animation approaches mainly (Hu et al. 2024 Wang et al. 2024b) Xu et al. [024cc Karras et al. 2023) Zhho et al., (002)) focus on video-driven settings, where con- trol signals pose sequence extract from the driving video. DreamPose (Karras et al. 2023) uses 2018) to train a diffusion model for sequential pose transfer. MagicAn- imate (Xu et al.] 2024c) extends the 2D U-Net to 3D to enhance the temporal smoothness. Ani- mateAnyone (Hu et al.,
DensePose (Güler et al.
2024) employs a dual U-Net architecture to maintain consistency between the generated video and the reference images. Some speech- speech-driven body animation works (Liao) eet al. [2020] Ginosar et al. 2019 Wang et al. 2023c Corona et al.] 2024) do exist, but they typi- cally employ a two-stage framework. Speech2Gesture (Ginosar et al. 2019) first predicts gesture sequence and then utilizes pre-trained GAN to render it to final video. Similarly, Vlogger (Corona (et al. (004)) employs two diffusion models to separately perform audio-to-mesh and mesh-to-video mapping. Two-stage methods rely on explicit intermediate representations to mitigate the training difficulty in audio-driven settings. However, the limited expressive capabilities of these intermediate representations can also constrain the overall performance. Unfortunately, the end-to-end training of a sne-stage diffusion model for audio-driven talking body generation remains unexplored.
3 METHOD
3.1 OVERVIEW
We develop our algorithm based on the Latent Diffusion Model (LDM) Blattmann et al. 2023b), which utilizes a Variational Autoencoder (VAE) Encoder (Kingma & Welling 2014) & to transform the image / from pixel space into a more compact latent space, represented as 20 = {(), to reduce the computational load. During training, random noise is iteratively added to 2p at various timesteps t E [1, T), ensuring that 27 ~ (0 1). The training objective of LDM is to predict the added noise at every timestep t:
= E (e,. [[le-
(2,, t, c)
where Eg denotes the trainable components such as the Denoising U-Net, and c represents the con- ditional inputs like audio or text. During inference, the trained model is used to iteratively remove
3

---

Published as a conference paper at ICLR 2025
Reference Net
Region Attention Module
Motion Frome
A
A
H Pand
Hummn--Prior-Guiee Conditions
Latarta DeRt
bescriptar
Leani
H
larpardl
Laber
P1
Ref. Image
Spatial Attention
Temporal Altention
Hand Clarity
Embedding
Noise Lafent
Audio Attertion
Ragion Attention
Body
P2
Denoising U-Net
W
Audio
Fix Parameter
VAE-Encoder
VAE-Decoder
W Wav2Vector
P Encoder
Add
The overall structure of CyberHost. We aim to generate video clip by driving reference image based on an audio signal. Region attention modules (RAMs) are inserted at multiple stages U-Net for fine-grained modeling of local regions. Additionally, Humaan-rior-Guidder Conditions, including the body movement map, hand clarity score and pose-aligned reference features are also introduced to reduce motion uncertainty. The reference network extracts motion from motion frames for temporal continuation.
Figure 2:
denoising
noise from noised latent sampled from Gaussian distribution. Subsequently, the denoised latent decoded into an image using the VAE Decoder D.
Our proposed CyperHost takes human reference image and speech audio clip inputs, ulti- mately generating synchronized human video. The overall architecture is illustrated Figure 2 We referenced the design of the reference
from AnimateAnyoneduu (SweaaaaaadsSSSssseeseaaa Stypulkkosski et al.
al. 2024) and TryOnDif-
fusion et al.) 2023b), as well as the motion frames from Diffused-Heads 2024) and EMO Tian al. 2024). to construct a baseline framework. Specifically, a copy of the 2D U-Net is utilized as reference net to extract reference features from the reference image and motion features from the motion frames. For audio, we use Wav2vec (Schneider et al., 2019) to extract multi-scale features. For the denoising U-Net, we extend the 2D version to 3D by integrating the pretrained temporal module from AnimateDiff (Guo al., 2024), enabling body video clips. The reference, motion frames, and audio features are fed into the denoising U-Net each resblock. They are respectively combined with latent features the spatial dimension to share the self-attention layer, in the temporal dimension to share the temporal module, and through an additional tross-attention layer to achieve this.
predict human
As shown in Figure 2] based on the baseline, proposed two key designs to address the inherent challenges of the audio-driven talking body generation task. First, to enhance the model's ability to capture details in critical human region, i.e., hands and faces, adapt the proposed region attention module (RAM), detailed in section (.2) to both the facial and hand regions and insert them into mul- tiple stages of the Denoising U-Net. RAM consists of parts: the spatio-temporal region latents bank learned from the data and the identity descriptor extracted from cropped local images. Second, reduce the motion uncertainty in half-body animation driven solely by audio, several conditions (detailed in section 3.3) have been designed to integrate global-local motion constraints and human structural priors: (1) The body movement map employed to stabilize the root movements of the body. It is encoded and merged with the noised latent, serving the input for the denoising U-Net. (2) The hand clarity score designed to prevent hand prediction degradation caused by motion blur in the training data. It is incorporated encodes the reference skeleton map, which is then integrated into the reference latent, yielding a alse-aligned reference feature. Note that the pose encoder for the body movement map and the reference skeleton map share the same model architecture, except for the first convolution layer, but do not share any model parameter.
residual into the time embedding. (3) The pose encoder
4

---

Published as a conference paper at ICLR 2025
Region Attention Module
Tops Ixwad
Triemp 1xam x
Spatio-Temp Tembo-a Cross Attention
latent
Leterts Bark
Repse:
(b/) xmxd
(blor)
Reshape
(Miw)) 7
F
Mask Predictor
F uned
unet
Cross Atto
(bf) (Jw) x
Head Act
(t/) (her) x
(bhao) I
H
jd
bx bxxxx
fxfxhx wxx.wwd
Fi unod
Figure 3: An illustration of region attention module (RAM), using the hand region example.
3.2 REGION SYNTHESIS WITH REGION ATTENTION MODULES
While the popular dual U-Net architecture effectively maintains overall visual consistency between the generated video and reference image, it struggles with generating fine- ine-grained texture details and complex motion patterns in local areas like the face and hands. This challenge is further exacerbated in the task of audio-driven human body animation due to the absence of explicit control signals. To address this issue, we meticulously designed the structure of the RAM to enhance its ability to learn local details. As shown in Figure 3 our proposed RAM comprises two key parts: spatio- temporal region latents bank and identity descriptor. The former aims to learn identity-agnostic general features, while the latter focuses on extracting identity-secciic unique features. Together, they enhance the synthesis of local human regions. subsequent sections, we apply it to the face and hands, areas that typically present significant challenges, and confirm its effectiveness.
latents bank as additional training parameters, prompting to learn shared local structural priors, including common topological structures and motion patterns. The region latents are composed of two sets of learnable basis vectors: Lspa E 1 xxxd for spatial features and Lemp E R 1x for temporal features. where and m denote the number of basis vectors and d denotes the channel dimension. We consider the combination of Lapu and Lump as pseudo 3D latents bank, endowing it with the capability to learn apatio-temporal features jointly. This capability facilitates the modeling of 3D characteristics such as hand motion. Furthermore, we constrain the basis vectors of the latents bank be mutually orthogonal to maximize its learning capacity.
Region Latents Bank, RAM enhances the model by adding a
The regional latent features are integrated into the U-Net through a spatio-temporal cross-attention as shown in Figure 3 Given the backbone feature Fin from U-Net, we apply cross attention with Lepa in the spatial dimension and with Liemp in the temporal dimension. The final output F Flatent is formulated as the sum of two attentions' result,
F Intent Attn(F
unet> Lepar Lspa) + Attn AFin(Finnn
unet:: Lnemp- Leemp)
QKT"" spa Vd
(QKT
softmax
Vspa + softmax
V temp
(3)
Va
where Attn(*, +, *) denotes the cross attention, Q, K and V are the query, key, and value, re- spectively. We aim for F Fiatent to fully utilize the spatio-temporal motion priors of the local region leamed within the 3D latents bank, refining and guiding the U-Net features through residual addi- tion. Notably, effectively focus the latents bank on feature learning for the target local region while filtering out gradient information from unrelated areas, we require regional mask to weight the residual addition process. Due to the absence of prior information on body part positions the audio-driven scenario, we employ auxiliary convolutional layers regional mask predictor. This predictor directly estimates a regional attention mask Mpred using the U-Net feature Fin During training, we region detection boxes to generate supervision signals Mg for mask predictor.
Identity Descriptor: The process of learning the latents bank is adentity- It leverages data to leam the shared local structural and motion patterns priors of the human body. However, speecffc features such as hand size, skin color, and textures are also important and cannot be overlooked. To complement this, employ regional image encoder R. to extract identity-aware
5

---

Published as a conference paper at ICLR 2025
regional features from the cropped region image I, The extracted feature is referred to as the identity descriptor. We illustrate this process in the left bottom of Figure 3 using the hand as an example.
Combining the ceentity-iddpeenddee latents bank and the identity descriptor, the final output Fin of the RAM module with locally enhancement features is obtained as follows:
Attn F unct RII,), R.I,))
(4)
F unet
(F latent + Fia) * MM pred Fiin
(5)
unet
Enhance Hand and Facial Regions Synthesis with RAM. Note that both hand and facial features common structural features and dientity-dependerr appear-
can be divided into dlentity-independell ance features. Therefore, the design principle of region attention module ensures its applicability to feature modeling for both hand and facial regions. In terms of implementation details, we use structure similar to the Pose Encoder but with deeper blocks for the Hand Encoder to enhance its feature extraction capability. The images of both hands are individually cropped and resized resolution of 128 for feature extraction, and then concatenated to form the hand identity descriptor. For the facial region, we utilize a pre-trained ArcFace (Deng et al. (019) network for identity fea- ture extraction. Considering the rich details of the face, facial images are resized to resolution of 256 for feature extraction. For Mgt. both hand and facial detect boxes are obtained by the minimal enclosing area defined by their respective key points.
Local Enhancement Supervision. The design of the latents bank and the identity descriptor aims to enhance the model's learning capacity. Additionally, signals for modeling critical local regions based on the local enhancement features from RAM. Firstly, we employ a local reweight strategy to optimize the original training objective. Specifically, we construct a mask Mg for critical regions such as the face and hands, and use to reweight the original training loss L by factor a. Moreover, synthesizing hands is more challenging than synthesizing faces due to the high flexibility of fingers. To address this, we add auxiliary losses to further enhance the synthesis of hand regions. Specifically, after each hand RAM, we pass the locally refined features Fume through several convolutional layers to predict the hand keypoints' heatmap H. aiding the model in better understanding the hand structure. Finally, our local enhancement loss function is formulated as follows:
strive to provide stronger supervisory
N
1 N
Lies (1 + * M hand
M
* Ml L +
H.
H.
(6)
where H denotes the ground truth keypoints heatmap. and N denotes the number of region attention modules in U-Net. We found that setting a 1 yielded the most stable results. Considering that the signal-to-noise ratio varies at different time steps f, we apply the local enhancement loss with a 50% probability only when & < 500.
3.3 HUMAN-PRIOR GUIDEE CONDITIONS
While RAM enhances the synthesis of critical human regions, generating half-body human mo- tion videos with audio as the sole condition can still introduce several artifacts: 1) random global body movements; 2) probabilistic blurring of hand gestures; 3) ambiguous limb structures. The first two artifacts are caused by motion uncertainty in the audio-driven generation of the talking body, while the last artifact is due to the model's insufficient prior knowledge of human structure topol- ogy. Recent video generation studies (Podell et al. 2023 Blattmann et al., 2023a) demonstrate that designing condition inputs, such as resolution and cropping parameters, can enhance the model's robustness to varied data and improve output controllability. Inspired by these studies, we intro- duce human prior information to the model and design the body movement map and hand clarity score conditions to decouple absolute body motion and hand motion blur during training, reducing the uncertainty caused by the weak correlation between audio and body motion. Additionally, we designed the pose-aligned reference feature help the model better perceive human structure.
Body Movement Map. Frequent body movements, including translations, rotations, or those caused by camera movements, are present in the talking body video data, increasing the training difficulty. To address this issue, we introduce a body movement map to serve as a control signal for the move- ment amplitude of the body root in generated videos. Specifically, we define a rectangular box

---

Published as a conference paper at ICLR 2025
representing the motion range of the thorax point over video segment. To avoid generating body movements that are overly dependent on the input movement map (which can lead to rigid motion during inference), we augment the size of the rectangular box by 100%-150% The body movement map is down-sampled and encoded through a learnable Pose Encoder and added as a residual to the noised latent. During inference, we input a body movement map of fixed size to ensure the stability of the overall generated results. Due to the augmentation during training, the audio-conditioned model can still generate natural motion, avoiding rigidity.
Hand Clarity Score. Due to exposure duration and rapid hand movements, video data often con- tains blurry hand images, which lose human region structural details, weakening the model's ability to learn hand structures and causing it to generate indistinct hand appearances. Therefore, we intro- duce a hand clarity score to indicate the clarity of hand regions in the training video frames. This score is used as a conditional input to the denoising U-Net, enhancing the model's robustness to blurry hand data during training and enabling control over the clarity of the hand images during in- ference. Specifically, for each frame in the training data, we crop the pixel areas of the left and right hands based on key points and resize them to a resolution of 128 x 128. We then use the Laplacian operator to calculate the Laplacian standard deviation of the hand image frames. A higher standard deviation typically indicates clearer hand images, and we use this value as the hand clarity score. The hand clarity score is provided to the U-Net model by residually adding the time embedding. During inference, a higher clarity score is applied to enhance the generation results for the hands.
Pose-aligned Reference Feature. Recent works (Zhu al., 2023b Hu al. 2024 Tian et al., 2024) have utilized reference net to extract and inject appearance features from the reference im- age, thereby maintaining overall visual consistency between the generated video and the reference image. However, when dealing with challenging pose reference images, the reference net struggles to accurately perceive the initial pose, especially hand gestures, which affects the quality of hand generation in the whole generated video. To address this, we leverage a pretrained model to perform pose recognition on the reference image. The results are encoded as skeleton map, which is then processed by a feature extractor and added to the original reference features. This ensures that the extracted reference features not only include the appearance information of the human body but also incorporate its topological structure information provided by the expert pose model, improving the model's 's robustness to the initial pose.
4 EXPERIMENTS
4.1 IMPLEMENT DETAILS
The training process is divided into two stages. In the first stage, two arbitrary frames from the training videos are sampled as the reference and target frames to learn the human image generation process. Training parameters include those of the reference net, pose encoder, and basic modules within the denoising U-Net. the second stage, we begin end-to-end training for generating videos from reference image and audio. All the parameters including those of the temporal layers, audio attention layers, and region attention modules are optimized. Each video clip has a length of 12 frames, with the motion frames' length set to 4. In stage one, trained 640 x 384, and in stage two, we trained at a dynamic resolution with an area approximately equal to 640 384. Each stage is trained with the learning rate set 1 We trained our model on 200 hours of video data featuring half-body speech scenarios, sourced from the internet and comprising over 10,000 unique identities. Further details are provided in the supplementary material, For quantitative evaluation, we designated 269 video segments from 119 identities as the test set.
fixed resolution of
4.2 COMPARISONS WITH STATE-OF-
Although initially designed for the audio-driven talking body generation task, our method can be casily generalized to other applications, such as video-driven body reenactment and audio-driven talking head. This allows us to validate the advanced nature and generalizability of the CyberHost framework against some of the current state-of-the-a metrics, use Fréchet Inception Distance (FID) assess the quality of the generated video frames and Fréchet Video Distance (FVD) (Unterthiner et al. 2019) to evaluate the overall coherence of the generated videos. To assess the preservation of facial appearance, calculate the cosine similarity
methods in related fields. For evaluation
7

---

Published as a conference paper at ICLR 2025
Table 1: Quantitative comparison of audio-driven talking body. denotes evaluate on vlogger
Methods
SSIM
PSNRT
FID4
FVDL
CSIM
SyncCT
SyneD/
HKCT
HKVT
DifTTED
0.667
15.48
95.45
1185.8
0.185
0.9259
12.543
0.769
0.656
14.97
58.95
1515.9
0.377
0.496
13.427
0.833
23.40
CyberHost (A2V-B)
0.691
16.96
32.97
555.8
0.514
6.627
7.506
0.884
24.73
Vlogger
0.470
0.601
11.132
0.923
9.84
CyberHost (A2V-B)
0.522
7.897
7.532
0.907
18.75
w/o Latents Bank
0.687
16.53
37.80
643.9
0.523
6.384
7.719
0,859
21.35
w/o Temp. Latents
0.681
16.62
36.75
624.1
0.489
6,468
7.573
0.870
19.44
w/o ID Descriptor
0.690
16.95
35.83
582.9
0.422
6.418
7.669
0.881
22.64
w/o Hand RAM
0.686
16.80
37.71
625.9
0.498
6.510
7.574
0.869
22.98
w/o Face RAM
0.685
16.86
35.14
612.8
0.425
6,299
7.796
0,880
24.11
w/o Local Enhancement
0.687
16.92
35.25
581.5
0.461
6.127
7.930
0,866
21.35
w/o Body Movement
0.680
16.76
39.83
668.6
0.458
6.372
7.769
0.867
27.54
w/o Hand Clarity
0.686
16.73
37.81
643.8
0.503
6.556
7.556
0.849
33.00
w/o Pose-aligned Ref.
0.683
16.66
38.32
660.0
0.487
6.498
7.684
0.870
23.18
Image
DIFTED
DiffGesture
CyberHost
Vlogger
CyberHost
Figure 4: The audio driven taking body results of CyberHost compared to other methods.
(CSIM) between the facial features of the reference image and the generated vidco frames. We use SyncC and SyncD, as proposed in (Prajwal et al. 2020), evaluate the synchronization quality between lip movements and audio signals in audio-driven settings. Additionally, Average Keypoint Distance (AKD) is used to measure the accuracy of actions in video-driven settings. Because the AKD cannot be used to evaluate hand quality in audio-driven scenarios, we compute the average of Hand Keypoint Confidence (HKC) reference metric for evaluating hand quality. Similarly, calculated the standard deviation of hand keypoint coordinates within a video segment as the Hand Keypoint Variance (HKV) metric represent the richness of hand movements.
Audio-driven Talking Body Currently, only few works such Dr2 (Wang et al. 2023c), DiffTED (Hogue et al. 2002)), and Vlogger (Corona et al.) (024) have adopted two-stage approaches to achieve audio-driven talking body video generation. However, these methods are not open- sourced, making it difficult to conduct direct comparisons. To better compare the effectiveness with the dual-stage method, we constructed a dual-stage audio-driven talking body baseline based on the current state-of-the-an audio2gesture and pose2video algorithms. Specifically, DiffGesture (Zhu et al., 2023a) on our dataset to generate subsequent driving SMPLX (Pavlakos et al. 2019) pose sequences based on input audio and an initial SMPLX pose. Finally, the SMPLX meshes were converted into DWPose (Yngg et al. 2023) key points, and MimicMotion (Zhang et al. 2024] was used for video rendering based these key points.
trained
8

---

Published as a conference paper at ICLR 2025
Table 2: Quantitative comparison with existing video-driven body reenactment methods.
Methods
SSIM
PSNR
FID4
FVDL
CSIM
AKD
DisCo Wang ct al. 2024b
0.660
17.33
57.12
1490.4
0.227
9.313
AnimateAnyone (Hu al.
2024
0.737
20.52
26.87
834.6
0.347
5.747
MimicMotion (hhang al.
2024
0.684
17.96
23.43
420.6
0.340
8.536
CyberHost (V2V-B)
0.782
21.31
20.04
181.6
0.458
3.123
Image
AnimuteAnyooe Anyone
Motic
Figure 5: Comparisons with other video-driven body reenactment results
As shown in Table L our proposed CyberHost significantly outperforms DifiTED and the two-stage baseline in terms of image quality, video quality, facial consistency, and lip-sync accuracy in the audio-driven talking body (A2V-B) setting. It is important to note that due to DiffTED generat- ing too many degraded hand results, the HKV metric can no longer reflect the diversity of hand movements. Figure also presents visual comparison between CyberHost and the two-stage base- line. Additionally, we utilized reference images and audio from 30 demos displayed on the Vlogger homepage to conduct both quantitative and qualitative comparisons with Vlogger. Notably, since most of Vlogger's s test videos exhibit minimal motion, the HKC indicator is relatively high, whereas the HKV indicator, which measures the diversity of movements, is very low, as shown in Table As depicted in F our proposed CyberHost surpasses Vlogger in both generated image quality and the naturalness of hand movements. These results demonstrate that our one-stage framework, Cy- berHost, achieves significant improvements over existing methods across multiple key dimensions of synthesis quality, as further validated by the videos provided in the supplementary materials.
Video-driven Body Reenactment We adapt our method perform video-driven human body reenactment (V2V-B) by utilizing DWPose (Yang et al., 2023) to extract full-body keypoints from videos and replace the body movement maps with a sequence of skeleton maps. As shown in Ta- ble 2 we compared our CyberHost with several state-of-the thet--rr sero-shot human body reenactment methods, including DisCo (Wang et al.) 2024b), AnimateAnyone (Hu et al. (024) and MimicMo- tion (Zhang et al., 2024). CyberHost significantly outperforms the current state-of-the-art methods in various metrics such as FID, FVD, and AKD. The visual results Figure also demonstrate that CyberHost achieves better structural integrity and identity consistency in local regions such as the hands and face compared to current saate-o-thh--eee results.
Audio-driven Talking Head. Although our framework designed for talking body, it requires only minor modifications to be adapted for audio-driven talking head (A2V-H) setting. We removed the unnecessary Hand RAMs and adjusted the cropping area of the training data to focus around face. As shown in Table we compared our method with Hallo (Xu et al. 2024a), VExpress (Wang) ctt al.) 2024a) and EchoMimic (Chen et al., 2024b). We randomly sampled 100 videos from CelebV. HQ (Zhu al. 2022) the test set. Experimental results demonstrate that CyberHost surpasses current state-of-the performance across multiple metrics, including FID, FVD, and Syne score.
4.3 ABLATION STUDY
Analysis of Region Attention Modules. As shown in Table I we conduct an ablation study to analyze the effectiveness of the Region Attention Modules. As we can see, the latents bank in RAM significantly improves metrics related to the hand quality like HKC and HKV. It can be observed that removing the temporal latents Lemp from latents bank leads to a degradation in the hand motion modeling ability, as reflected by decreased HK V, demonstrating the importance design. The identity descriptor, the other hand. is more closely associated with the CSIM metric, demonstrating its effectiveness in maintaining identity consistency. Face RAM significantly im- proves facial-related metrics such CSIM and Syne score, while Hand RAM effectively reduces
ppatial-tempora

---

Published as a conference paper at ICLR 2025
Ref
Audio)
Imagge
(a)
(b)
Figure 6: (a) Multimodal-dtie driven results (b) Audio-driven results on the open-set test images.
Table 3: Comparison with existing audio-driven talking head methods.
Methods
SSIM
PSNR
FIDJ
FVDJ
CSIM
SyncC1
SyncD4
EchoMMimic (Chen et al.
2024b)
0.619
17.468
35.37
828.9
0,411
3.136
10.378
VExpress (Wang et al. 2024a
0.422
10.227
65.09
1356.5
0.573
3.547
9.415
Hallo [u et al. 2024a
0.632
18.556
35.96
742.9
0.619
4.130
9.079
CyberHost (A2V-H)
0.694
19.247
25.79
552.6
0.581
4.243
8.658
artifacts in hand generation, thereby improving pixel quality metrics like FVD and FID. Local en- hancement supervision enhances facial consistency, lip synchronization, and hand quality.
Analysis of Human-Prior-Guudee Conditions. We also validated the effectiveness of various suman-prior-guite grior-guided conditions in Table the absence of body movement constraints, the mo- tion space of the human root is unrestricted, resulting in unstable overall generation quality. This instability negatively impacts metrics related to overall generation quality, such as SSIM, PSNR, FID, and FVD, while the richness of hand movement metrics, that is HKV. may appear artificially high. Similarly, the absence of the hand clarity score as a prior for hand motion patterns results in more erratic hand movement trajectories. Frequent artifacts contribute to artificially high HKV, and significantly impact the hand quality metric, such as HKC. The pose-aligned reference features, by leveraging the topological structure priors of the reference images, also enhance the stability of the generated results, providing significant benefits in metrics such as HKC, FID, and FVD.
4.4 MULTIMODAI--DRIVE VIDEO GENERATION
Our proposed CyberHost also supports combined control signals from multiple modalities, such as 2D hand keypoints and audio. As shown in Figure 6 (a), the hand keypoints used to control hand movements and the audio signals are used to drive head movements, facial expressions, and lip synchronization. This driving setup leverages the explicit structural information provided by hand pose templates to enhance the stability of hand generation, while significantly improving the correlation of head movements, facial expressions, and lip synchronization with the audio.
4.5 AUDIO-DRIVEN RESULTS IN OPEN-SET DOMAIN
To validate the robustness of CyberHost, tested the audio-driven talking body video generation results on open-set test images sourced from the internet and AIGC image synthesis. As shown in Figure 6 (b). our proposed method demonstrates good generalization across various characters and is capable of generating complex gestures, such as hand interactions.
5 CONCLUSION
This paper introduces a one-stage audio-driven talking body generation framework, CyberHost, de- signed to produce human videos that match the input audio with high expressiveness and realism. CyberHost features region attention module to enhance the details generation ability of key local re- human-prior-
gions and a suite of human- prior- prior-guided conditions to reduce motion uncertainty in the audio-driven setting. Extensive experiments demonstrate that CyberHost can generate stable, natural, and realistic talking body videos and achieve zero-shot human image animation in open-set domains. Addition- ally, CyberHost be extended to various human animation settings, achieving satisfactory results.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
We extend our gratitude to Dr. Pengfei Wei for assisting in setting up the two-stage audio-driven talking body video generation baseline.
REFERENCES
Chaitanya Ahuja, Dong Won Lee, Ryo Ishii, and Louis-Philippe Morency. No gestures left behind: Learning relationships between spoken language and freeform gestures. In EMNLP, pp. 1884- 1895, 2020.
Simon Alexanderson, Gustav Eje Henter, Taras Kucherenko, and Jonas Beskow. Style-controllable speech-drived gesture synthesis using normalising flows. Comput. Graph. Forum, 39(2))487-49) 2020.
Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models large datasets. arXiv preprint arXiv:231..15127, 2023a.
Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with latent diffusion models. In CVPR, pp. 22575,, 2023b.
Tsai-Shien Chen, Aliaksandr Siarohin, Willi Menapace, Ekaterina Deyneka, Hsiang- wei Chao, Byung Eun Jeon, Yuwei Fang, Hsin- Hsin-Ying Lee, Jian Ren, Ming- Yang, and Sergey Tulyakov. Panda- 70m: Captioning 70m videos with multiple cross-modality teachers. In CVPR, pp. 13320- 13331, 2024a.
Zhiyuan Chen, Jiajiong Cao, Zhiquan Chen, Yuming Li, and Chenguang Ma. Echomimie: Life- like audio-driven portrait animations through editable landmark conditions. arXiv preprint arXiv:2407.08136, 2024b.
Enric Corona, Andrei Zanfir, Eduard Gabriel Bazavan, Nikos Kolotouros, Thiemo Alldieck, and Cristian Sminchisescu. Vlogger: Multimodal diffusion for embodied avatar synthesis. arXiv preprint arX1i:2403.087777 2024.
Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou. Areface: Additive angular margin loss for deep face recognition. In CVPR, pp. 4690-4699, 2019.
Shiry Ginosar, Amir Bar, Gefen Kohavi, Caroline Chan, Andrew Owens, and Jitendra Malik. Learn- ing individual styles of conversational gesture, In CVPR, pp. 3497-3506, 2019.
Riza Alp Güler, Natalia Neverova, and Iasonas Kokkinos. Densepose: Dense human pose estimation in the wild. In CVPR, pp. 7297-7306, 2018
Yuwei Guo, Ceyuan Yang, Anyi Rao, Zhengyang Liang, Yaohui Wang, Yu Qiao, Mancesh Agrawala, Dahua Lin, and Bo Dai. Animatediff: Animate your personalized text-to-image diffu- sion models without specific tuning. In ICLR, 2024.
Yingqing He, Tianyu Yang, Yong Zhang, Ying Shan, and Qifeng Chen. Latent video diffusion models for high-fidelity long video generation. arXiv preprint (rXiv:2211.1322) 2022.
Steven Hogue, Chenxu Zhang, Hamza Daruger, Yapeng Tian, and Xiaohu Guo. Diffted: One-shot audio-driven ted talk video generation with diffusion-based co-speech gestures. In CVPR, pp- 1922-1931, 2024.
Li Hu, Xin Gao, Peng Zhang, Ke Sun, Bang Zhang, and Liefeng Bo. Animate anyone: Consistent and controllable image-to-video synthesis for character animation. In CVPR, pp. 8153-8163, 2024.
Ziyao Huang. Fan Tang, Yong Zhang, Xiaodong Cun, Juan Cao, Jintao Li, and Tong-Yes Yee Lee. Make- your-anchor: A diffusion-based 2d avatar generation framework. In CVPR, pp. 6997-7006, 2024.

---

Published as a conference paper at ICLR 2025
Jianwen Jiang. Chao Liang, Jiaqi Yang, Gaojie Lin, Tianyun Zhong, and Yanbo Zheng. Loopy: Taming audio-driven portrait avatar with long-term motion dependency. arXiv preprint 0rXiv:2409.02634, 2024a.
Jianwen Jiang. Gaojie Lin, Zhengkun Rong, Chao Liang. Yongming Zhu, Jiaqi Yang, and Tianyun Zhong. Mobileportrait: Real-time one-shot neural head avatars mobile devices. arXiv preprint arXiv:2407.05712, 2024b.
Johanna Karras, Aleksander Holynski, Ting-Chun Wang. and Ira Kemelmacher-Shhizeemaee Dream- pose: Fashion image-to-vides synthesis via stable diffusion. In ICCV, pp. 22623-22633, 2023.
Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. In ICLR, 2014.
Miao Liao, Sibo Zhang. Peng Wang, Hao Zhu, Xinxin Zuo, and Ruigang Yang. Speech2video synthesis with 3d skeleton regularization and expressive body poses. ACCV, pp. 308-323. Springer, 2020.
Bin Lin, Yunyang Ge, Xinhua Cheng, Zongjian Li, Bin Zhu. Shaodong Wang, Xianyi He, Yang Ye, Shenghai Yuan, Liuhan Chen, et al. Open-sora plan: Open-source large video generation model. arXiv preprint 2rX00:24/2.00131, 2024.
Xian Liu, Qianyi Wu, Hang Zhou, Yuanqi Du, Wayne Wu, Dahua Lin, and Ziwei Liu. Audio-driven co-speech gesture video generation. In NeurlPS, 2022.
Yifeng Ma, Shiwei Zhang. Jiayu Wang, Xiang Wang, Yingya Zhang, and Zhidong Deng. Dreamtalk: When expressive talking head generation meets diffusion probabilistic models. arXiv preprint arXiv:2312.09767. 2023.
Georgios Pavlakos, Vasileios Choutas, Nima Ghorbani, Timo Bolkart, Ahmed AA Osman, Dimitrios Tzionas, and Michael J Black. Expressive body capture: 3d hands, face, and body from single image. In CVPR, pp. 10975-10988 2019.
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhom, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952, 2023.
K. R. Prajwal, Rudrabha Mukhopadhyay, Vinay P. Namboodiri, and C. V. Jawahar, A lip sync expert is all you need for speech lip generation in the wild. In ACM, pp. 484-492, 2020.
Shenhan Qian, Zhi Tu, Yihao Zhi, Wen Liu, and Shenghua Gao. Speech drives templates: Co-speech gesture synthesis with learned templates. In ICCV, pp. 11077-11086, 2021.
Steffen Schneider, Alexei Baevski, Ronan Collobert, and Michael Auli. wav2vec: Unsupervised pre-training for speech recognition. In INTERSPEECH, pp. 3465-3469, ISCA, 2019.
Aliaksandr Siarohin, Oliver J. Woodford, Jian Ren, Menglei Chai, and Sergey Tulyakov. Motion 2021.
representations for articulated animation. In CVPR, pp.
Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, et al. Make-a-video: Text-to-video generation without text-video data. arXiv preprint arXiv:2209.14777 2022.
Michal Stypulkowski, Konstantinos Vougioukas, Sen He, Maciej Zieba, Stavros Petridis, and Maja Pantic. Diffused heads: Diffusion models beat gans on talking-face generation. In WACV, pp. 5089-5098, 2024.
Linrui Tian. Qi Wang, Bang Zhang, and Liefeng Bo. EMO: emote portrait alive generating pressive portrait videos with audio2video diffusion model under weak conditions. In ECCV, pp. 244-260 Springer, 2024.
Thomas Unterthiner, Sjoerd van Steenkiste, Karol Kurach, Raphaël Marinier, Marcin Michalski, and Sylvain Gelly. FVD: A metric for video generation. In DGS@ICLA 2019.
12

---

Published as a conference paper at ICLR 2025
Cong Wang, Jiaxi Gu, Panwen Hu, Songeen Xu, Hang Xu, and Xiaodan Liang. Dreamvideo: High-fidelity image-to-vide video generation with image retention and text guidance. arXiv preprint arXiv:2312.03016 2023a.
Cong Wang, Kuan Tian, Jun Zhang, Yonghang Guan, Feng Luo, Fei Shen, Zhiwei Jiang, Qing Gu, Xiao Han, and Wei Yang. V-express: Conditional dropout for progressive training of portrait video generation. arXiv preprint arXiv:2406.00553 2024a.
Tan Wang, Linjie Li, Kevin Lin, Yuanhao Zhai, Chung- Shing-Chig Lin, Zhengyuan Yang, Hanwang Zhang, Zicheng Liu, and Lijuan Wang. Disco: Disentangled control for realistic human dance generation. CVPR, pp. 9326-9336, 2024b.
Ting-Chun Wang, Arun Mallya, and Ming- Yung-Yulu Liu. One-shot free-view neural talking- synthesis for video conferencing. In CVPR, pp. 10039-10049, 2021.
Xiang Wang, Hangjie Yuan, Shiwei Zhang, Dayou Chen, Jiuniu Wang, Yingya Zhang, Yujun Shen, Deli Zhao, and Jingren Zhou. Videocomposer: Compositional video synthesis with motion con- trollability. In NeurlPS, 2023b.
Zhixin Wang, Ziying Zhang, Xiaoyun Zhang, Huangjie Zheng, Mingyuan Zhou, Ya Zhang, and Yanfeng Wang. Dr2: Diffusion-based robust degradation remover for blind face restoration. In CVPR. pp. 1704-1713, 2023c.
Mingwang Xu, Hui Li, Qingkun Su, Hanlin Shang, Liwei Zhang, Ce Liu, Jingdong Wang, Luc Van Gool, Yao Yao, and Siyu Zhu. Hallo: Hierarchical audio-driven visual synthesis for portrait image animation. arXiv preprint arXiv:2406.0mm 08800, 2024a.
Sicheng Xu, Guojun Chen, Yu-Xiao Guo, Jiaolong Yang. Chong Li, Zhenyu Zang, Yizhong Zhang, Xin Tong, and Baining Guo. VASA-1: lifelike audio-driven talking faces generated in real time. In NeurlPS, 2024b.
Zhongcong Xu, Jianfeng Zhang, Jun Hao Liew, Hanshu Yan, Jia-Wei Liu, Chenxu Zhang, Jiashi Feng, and Mike Zheng Shou. Magicanimate: Temporally consistent human image animation using diffusion model. In CVPR, pp. 1481-1490 2024c.
Zhendong Yang, Ailing Zeng, Chun Yuan, and Yu Li. Effective whole-body pose estimation with tto-stages distillation. In ICCV, pp. 4210-4220, 2023.
Fei Yin, Yong Zhang, Xiaodong Cun, Mingdeng Cao, Yanbo Fan, Xuan Wang, Qingyan Bai, Baoyuan Wu, Jue Wang, and Yujiu Yang. Styleheat: One-shot high-resolution editable talking face generation via pre-trained stylegan. In ECCV, pp. 85-101. Springer, 2022.
Wenxuan Zhang, Xiaodong Cun, Xuan Wang, Yong Zhang, Xi Shen, Yu Guo, Ying Shan, and Fei Wang. Sadtalker: Learning realistic 3d motion coefficients for stylized audio-driven single image talking face animation. In CVPR, pp. 8652-8661, 2023.
Yuang Zhang, Jiaxi Gu, Li-Wen Wang, Han Wang, Junqi Cheng, Yuefeng Zhu, and Fangyuan Zou. Mimicmotion: High-quality human motion video generation with confidence-awant pose guid- ance. arXiv preprint arXiv:2406. 19680, 2024.
Daquan Zhou, Weimin Wang. Hanshu Yan, Weiwei Lv, Yizhe Zhu, and Jiashi Feng. Magicvideo: Efficient video generation with latent diffusion models. arXiv preprint arXiv:2211.11016 2022.
Hao Zhu, Wayne Wu, Wentao Zhu, Liming Jiang, Siwei Tang, Li Zhang, Ziwei Liu, and Chen Change Loy. Celebv-hq: A large-scale video facial attributes dataset. In ECCV, pp. 650 667. Springer, 2022.
Lingting Zhu, Xian Liu, Xuanyu Liu, Rui Qian, Ziwei Liu, and Lequan Yu. Taming diffusion models for audio-driven co-speech gesture generation. In CVPR, pp. 10544-10553, 2023a.
Luyang Zhu, Dawei Yang, Tyler Zhu, Fitsum Reda, William Chan, Chitwan Saharia, Mohammad Norouzi, and Ira SeaelmaccherSShizemmmie Tryondiffusion: A tale of two unets. In CVPR, pp. 4606 4615, 2023b.
13

---

Published as a conference paper at ICLR 2025
A MORE EXPERIMENT RESULTS
A.1 ADDITIONAL VISUALIZATION COMPARISONS
We present supplementary visual comparison results with state-of-the-art methods the video- driven context, illustrated in Figure Additionally, provide further visual results for open-set test images in the audio-driven context in Figure g
Ref Image
DisCo
AnimateAnyone
MinicMotion
Ours
GT
Figure 7: Additional comparisons results with other video-driven body reenactment results
Ref
Audio Driven Revalts
Results
Figure Additional audio driven taking body results CyberHost on the open-set test images.
A.2 USER STUDY FOR SUBJECTIVE EVALUATION
We conducted a user study with five independent, professional evaluators. Specifically, we used the MOS (Mean Opinion Score) rating method. The specific evaluation criteria discussed in
14

---

Published as a conference paper at ICLR 2025
Table 4: User study of subjective evaluation.
denotes evaluate vlogger
Method
ID Consistencyf
Motion Naturalnesst
Video Qualityt
Lip-synct
DiffTED
1.54
1.52
1.92
1.00
Mo.
3.28
2.56
3.32
2.34
CyberHost (A2V-B)
4.40
4.28
4.10
4.64
GroundTruth
4.94
4.72
4.66
4.88
Vlogger
4.10
3.54
3.26
2.40
CyberHost (A2V-B)
4.62
4.12
4.52
4.70
Section G
As can be seen from T Cyberhost shows performance advantages across multiple
dimensions compared the baseline methods.
A.3 ABLATION STUDY OF REGION ATTENTION MODULE
To more effectively validate the effectiveness of proposed Region Attention Module structure design, we have supplemented additional ablation experiments. These include removing the spatial latents, replacing the spatial latents and temporal latents with unified 3D latents bank. As shown in Ta eliminating the spatial bank significantly affects the quality of the generated videos, while the 3D latents bank, with a 55-fold increase in parameters under the current configuration, did not provide further performance gain. We hypothesize that the spatial and temporal latents bank, which more closely aligns with the behavior of U-Net by decoupling the temporal attention and spatial attention, consequently leads to more efficient learning of local features. Additionally, the Figure 9 we conducted ablation experiments on different sizes of the latents bank in the RAM. The results indicate that increasing the size of the latents bank beyond the current configuration provides only marginal performance gains.
shown in
Table 5: More ablation results on Region Attention Module.
Method
FIDJ
FVD1
SyncCt
HKC
HKV
w/o Regional Latents Bank
37.80
643.9
6.384
0.859
21.35
w/o Spatial Latents
36.23
612.7
6.362
0.864
22.15
w/o Temp. Latents
36.75
624.1
6.468
0.870
19.44
3D Latents Bank
32.94
552.3
6.613
0.888
23,93
Spatial Latents Temp. Latents
32.97
555.8
6.627
0.884
24.73
1400
3.77
3870
190%
020
Figure Comparison of results with different spatial latents sizes.
A.4 ANALYSIS OF REGIONAL ATTENTION MASKS FROM DIFFERENT LAYERS
we employ auxiliary convolutional layers as a regional mask predictor to directly estimate regional attention mask Mped to guide the optimization of the RAM modules. Note that we learn such a mask predictor each block of the denoising U-Net. Since different layers learn different kinds of features to different roles in the network, we also investigated the accuracy
In Section 3.2
15

---

Published as a conference paper at ICLR 2025
0.586
0.1
Cross Entropy
0,066
0.055
0,066
0.047
0.047
0,048
00033
0.032
bock
black
black
back
Neek
Hlauk
block
baok
black
Bloakk
black F1
bloak 12
blaak 13
block
bluck 18
Figure 10: The accuracy of hand attention mask predictions across different layers in the denoising U-Net.
Table 6: Quantitative comparison of inserting RAMs at different layers.
Method
FID
FVD.
CSIM1
SyncC1
HKC1
top-1
39.22
664.3
0.421
6,271
0.853
top-5
36.44
594.0
0.470
6.512
0.869
top-10
33.88
573.1
0.489
6.549
0.874
All
32.97
555.8
0.514
6.627
0.884
differences in hand attention mask predictions across different layers. As shown in Figure T0 we calculated the cross-entropy between the predicted hand mask from each layer and the hand region in the generated results on a batch of test data to measure the effectiveness of each attention block. Based on the accuracy of the predicted masks from each layer, we created corresponding top- top- 5, and t0p-10 experiments (where "top" refers to lower cross-entropy) to verify whether predicting masks and inserting RAM module in fewer layers could yield better results. As shown Figure 6 We found that using RAM modules in more simplified layers not as effective using them in all layers, which may suggest that even layers with less accurate hand mask predictions still have the necessity to enhance local features learning.
A.5 ABLATION STUDY HAND CLARITY SCORE
As shown in Figure [] we show the impact on the generation results when different hand clarity be seen that it will directly affect the generation quality and dy-
score conditions are input. namic degree of gestures. As illustrated in Figure 13 the hand clarity score reflects the clarity of the hands in the training data, thereby allowing the identification training videos with varying quality of hand visuals. It enables the model to distinguish between high-quality and low-quality hand im- ages during training, thereby guiding the model to generate clear hand visuals during inference by inputting high hand clarity score.
A.6 COMPARISON WITH ANGIE
ANGIE (Luu et al. 2022 establishes a unified framework to generate speaker image sequences driven by speech audio. However, it primarily focuses on designing vector quantized motion
16

---

Published as a conference paper at ICLR 2025
extractor to summarize common co-speech gesture patterns, and co-speech gesture GPT with motion refinement to complement the subtle prosodic motion details. The overall emphasis this work is on the audio-to-motion generation aspect, utilizing a pre-existing image generator model from MRAA (Siarohin et al. 2021) for the visual synthesis. While our proposed CyberHost aims optimize both motion generation and image generation simultaneously, providing more unified algorithmic framework. In Figure [2, we provide a comparison with ANGIE. It is important to note that ANGIE is trained and tested simultaneously on the four individuals in the PATS (Ahuja et al. 2020) dataset, whereas CyberHost focuses on the driving performance for open-set images. Despite not specifically training on the PATS dataset, our generated results exhibit natural motion patterns and significantly better video clarity. Additionally, we performed a visual comparison on the (neech2Gesture(Ginnne
et al., 2019) dataset against methods such as Speech2Gesture (Ginosar et al. 2019), MoGlow (Alexanderson et al. (020), and SDT (Qian et al. 2021). Please refer to the supplementary materials for the comparison video.
B ADDITIONAL DATASET DETAILS
B. DATASET COLLECTION
Most of our data primarily originates from open-source datasets (Lin at 2024 Chen al. 2024a), as well as from open platforms that adhere to the "Creative Commons" license. such as Pexels and Pixabay. All collected videos are publicly accessible, and we employ professionals review the data sources to ensure compliance with proper licensing agreements. Our emphasis is half-body human scenes; thus, we filter out data unrelated to human subjects based on video titles and other metadata. Subsequently, through a series of data processing steps, curated collection of single- person, half-body speaking videos, ensuring that key regions, such as the hands and head, remained within the frame. All collected video data is converted into latent embeddings using Variational Autoencoder (VAE) encoder and identified via MD5 checksums. This approach eliminates the stor- age of the videos' RGB content and URLs, thereby safeguarding any personal information related to the subjects. Additionally, we have implemented strict access controls to prevent non-research personnel from accessing the data. Next, will provide detailed explanation of several key steps in our data processing pipeline.
1. Video Trimming:
We utilized PySceneDetect (https://github.cc httpp/////ggiiuu...m//rereiiiiie commBBraakhrrogghhPSSeeeet to trim shot transitions and fades in video clips. The trimming method employed is akin to SVD (Blattman et al. 2023a), but we adjusted the parameters specifically for human body scenes.
Half-body Cropping: Using open-source tools like DWPose (Yang et al. 2023), we detected human key points and cropped the body region. We excluded video clips with more than two individuals, those where the subject was too small within the frame, and those with excessive human movement. This filtering allowed us select half-body videos that met our criteria based on key points.
Exclusion of Background Motion: We applied foreground segmentation algorithms identify the background regions within frames. We calculated the per-pixel differences between frames in the background areas and excluded data with significant background movement.
4. Audio-Vides Synchronization: In the final stage of data processing, we used tools like LipSyne to verify the synchronization between video and audio lip movements, discarding any video data with incorrect audio.
Table 7: Statistics dataset scale of different methods.
Method
Clips No.
Duration
IDs No.
DifITED Hogue et al.
2024
1134
2.75h
353
Vlogger Corona et al 2024
2200h
800k
CyberHost
104k
200h
10k
17

---

Published as a conference paper at ICLR 2025
Table 8: Data Distribution in the Dataset.
Category
Subcategory
Proportion
Male
63.1%
Gender
Female
36.9%
Indoor Outdoor
85.2%
Environment
14.8%
5s
58.2%
Video Duration
5s-10s
41.8%
384
5122
45.3%
Resolution Area
>>
5122
54.7%
B.2 DATASET STATISTICS
Table 7 presents statistics on the number of clips, total duration, and number of IDs in our training data, comparing these with existing audio-driven talking body generation works (Hogue al. 2024
Corona et al. 2024). DiffTED (Hogue et al.) 2024), a -aseb method, shows lower overall generation quality due to its lesser reliance data. Both Vlogger (Corona al.] 2024) and
proposed CyberHost employ diffusion- -based methods; however, despite having significantly less training data duration and fewer IDs compared to Vlogger, CyberHost achieves noticeably higher generation performance. This suggests that our performance improvements are derived from the efficacy of our algorithmic design rather than the quantity of data used. Additionally, we analyze the dataset's s distribution across four dimensions: gender, environment, video duration, and resolution area, as shown in Table
C ETHICS IMPACTS
Cyberhost enables the creation of realistic talking body videos using audio and a single image. While this technological advancement offers significant potential for fields such as education and commercial advertising, greatly enhancing and enriching people's ' lives, also raises ethical con- cerns. Caution is required to prevent its misuse in generating offensive or potentially infringing deepfake vidcos. We have carefully considered various strategies to regulate video generation within the Cyberhost application. 1) We need to confine the usage scenarios of this technology, authorizing its application solely in legitimate areas such as education and commercial advertising. This restric- tion aims prevent the creation of illicit or infringing video content. Additionally, embedding the creator's 's digital fingerprint into the video will facilitate legal tracing. 2) We plan to embed visible watermarks in the generated videos help individuals identify Al-generated content, thereby pre- venting potential misguidance. 3) We aim to establish a thorough review system to ensure the safety of uplo-ddl images and audio within Cyberhost. For images, techniques such as pomography detection and celebrity recognition can be employed. For audio, Automatic Speech Recognition (ASR) can transcribe audio text, allowing for the analysis of potential issues using Large Lan- guage Models. This approach helps prevent the generation of videos containing offensive audio.
D CALCULATION OF HAND CLARITY SCORE
We devised the hand clarity score to mitigate the impact of hand blur data on the learning of hand structures within the dataset. Consequently, this metric aims to quantify the degree of blurriness in the hand regions of the images. This is achieved by assessing the detail richness and edge sharpness of the hand images. We use DWPose (Yang et al., 2023) to extract body keypoints and form bound- ing boxes for the left and right hands. As illustrated in Figure [3] note that the areas corresponding to the left and right hands may overlap. Subsequently, the hand images are resized to a fixed resolution 128 x 128 to ensure the statistical analysis remains meaningful.
Given a video clip with a frame length of n. as detailed Algorithm we apply a Laplacian trans- formation to hand images and calculate the standard deviation, denoted as Ileft and right This metric indicates the clarity of the hand images, with higher values representing less blurriness. Since
18

---

Published as a conference paper at ICLR 2025
the absolute standard deviation of the Laplacian can vary in scale depending the image subject, also introduce the relative standard deviation of the Laplacian, denoted i.feft and itright. Specifically, we select the frame corresponding to the 90th percentile L within this video clip as the pivot frame, which is treated as a clear-hand frame. Both the absolute and relative Lapla- cian standard deviations are considered hand clarity scores and are subsequently fed into denoising U-Net condition, enabling the model to gauge the appropriate level of blurriness for the output image. During the inference stage, a high clarity score is input into CyberHost, prompting the model to generate stable and clear hand videos.
Algorithm Hand Clarity Score Extraction
INPUT: Original video frames 1is
Extract human keypoints K from I7ri
Crop out left & right hand areas in terms of Kim from Iini and resize them to x resolution,
denoted as fleft and Fright
Calculate relative Laplacian standard variance with KernelSize = 3 from pleft denoted as L Lefft and Lright
and Fright
Imm 1mm
Assign Left and L Lightt pipeot to be the 90% percentile value of L and Light Calculate absolute Laplacian standard variance Li + Lieft/ p and the same for L return L L Lright, L and 1
right
E ADDITIONAL IMPLEMENTATION DETAILS
In the region attention module, we set m, and d to 512, 64, and 512, respectively. Consequently, this configuration results in a spatial latents Lspa of shape (1, 512, 512), and a temporal latent Lemp of shape (1, 64, 512). Both Lopa and Lump initialized using Gaussian noise, with the Gram- -chmidt process being applied to both during each forward pass to ensure orthogonality. ROPE embedding utilized to inject temporal and spatial positional information into the latents bank, respectively. As for the regional attention mask Mgred in Section 3.2 we employ MSE loss to supervise the learning process. The hand mask loss and face mask loss are weighted by 0.005 and 0.001, respectively, in the total loss. For the input audio feature, to fully leverage multi-scale audio features, we concatenate the hidden states from each encoder layer of the Wav2Vec (Schneider et al. 2019) model with the last hidden state, resulting in an audio feature vector of size 10752. To capture more temporal context, we use temporal window of size 5, constructing an input audio feature of shape (5, 10752) for each video frame. It is important to note that both the video-driven body reenactment model and the multimodal-drives video generation model require separate training processes. During the training process for video-driven body reenactment, we replace the body movement map with the full-body skeleton map. For multimodal-driven video generation, we directly draw the skeleton maps of both hands onto the body movement map.
The training process of our audio-driven talking body generation model is divided into two stages. In stage one, The training of 12 per GPU. In the second stage, We use a total of 32 A100 GPUs to train for days, with each GPU processing only one video sample. This setup allows us train with different resolutions on different GPUs, enabling convenient dynamic resolution training. We constrain these different resolutions have an area similar to the 640 384 resolution, with both the height and width being multiples of 64 to ensure compatibility with the LDM structure. During inference, we use a dual classifier-free guidance (CFG) strategy for both the reference image and audio, with scales set to 2.5 and 4.5, respectively. Specific implementation can be referenced in (Karras et al. (202).. We found that a higher reference image CFG scale leads to oversaturated images, while a higher audio CFG scale results in unstable movements. The model achieves an overall inference Real-Time Factor (RTF) of approximately 65 on a single A100 GPU.
conducted for total of 4 days on 8 A100 GPUs, with batch size
F LIMITATIONS AND FUTURE WORK
We will explore the current limitations and future work in four distinct directions.
19

---

Published as a conference paper at ICLR 2025
More Realistic. While our proposed CyberHost model aims to mitigate issues of detail underfitting and motion uncertainty in one-stage audio-driven talking body generation, there are still instances of localized degradation in certain body regions, such as hands and face. Additionally, our generated results exhibit noticeable artifacts, particularly in hand details, and the overall performance has yet to reach the Turing test level achieved by methods like EMO (Tian et al.] (024), which specialized in talking head generation. As such, enhancing the quality of key area details and improving the realism of character movement patterns represent critical areas for future work.
More Robust. CyberHost continues to face challenges in generating high-quality videos from cer- tain difficult imputs. For instance, as illustrated in Figure exaggerated body proportions in cartoon images often result in unrealistic outputs. Conversely, the driving effect is more effective for car- toon images with body proportions closer to those of real humans. Moreover, complex backgrounds resembling hand colors, such skin-toned backgrounds, are prone to generating errors. Challeng- ing human poses, such as raising both hands above the head, further complicate the generation of natural motion. Regarding the audio inputs, scenarios with complex background noises and varying tones, such as singing, pose additional challenges for the model. These conditions make it diffi- cult to generate accurate lip movements and naturally synchronized body movements. Examples of these challenging scenarios can be found in the videos provided in the supplementary material. For these challenging test images, our future work involves collecting more diverse dataset of charac- ter videos, including anime scenes and data featuring complex human poses. Additionally, will be necessary to design more effective architecture that encodes prior knowledge of human body structures into the diffusion model, thereby improving its understanding of the subjects in diverse test images. Regarding complex test audio, we aim replace Wav2Vee (Schneider et al.) 2019 9 with a more advanced audio feature extractor. Furthermore, implementing background noise reduction strategies will enhance the model's robustness to audio variations.
F Full-Body Generation. On the other hand, despite the fact that our training data does not include videos of full-body scenes, we still present the audio-driven results on full-body test images in Figure [4] It can be observed that directly testing our current model on full-body scenes leads to significant degradation of generated details in areas such as the face and hands. Additionally, the overall body proportions and movements appear quite unnatural. The primary reasons for the sub- optimal performance of the current model on full-body scenarios are twofold. Firstly, our training data primarily consists of upper- body scenes, which limits the model's ability generalize effec- tively to full-body scenarios, resulting in errors and unnatural limb generation. Secondly, at the current generation resolution (640 x 384), the face and hand regions in full-body videos occupy smaller proportion of the frame. The existing VAE and U-Net architectures have limited capability model faces and hands effectively at such low resolution. In future work, we are keen collect data that includes full-body scenarios and to increase the model's 's resolution to better support video generation in full-body scenes.
G MEAN OPINION SCORE CRITERIA
Identity Consistency:
1: Identity is not preserved at all; the generated identity does not resemble the original in any aspect.
2: Identity is faintly preserved; the output shows some resemblance to the original, but major inconsistencies are apparent.
3: Identity is moderately preserved; while the general likeness is maintained, there are notable discrepancies.
4: Identity is well-preserved; only minor differences are noticeable.
5: Identity is perfectly preserved; the output matches the input without any discernible inconsistencies.
Motion Naturalness:
Motion appears highly unnatural; movements are jerky or disjointed.
2: Motion is somewhat unnatural; movements have notable unnatural transitions.
20

---

Published as a conference paper at ICLR 2025
3: Motion has mix of natural and unnatural elements; movements generally smooth with some unnatural aspects.
Motion appears mostly natural; only minor unnatural movements are present.
5: Motion is completely natural; movements fluid and appear realistic.
Video Quality:
Video quality is very poor; heavy artifacts, blurring, and pixelation are present.
2: Video quality is below average; significant artifacts and blurring occur frequently.
3: Video quality is average; some artifacts and blurring, but generally acceptable.
4: Video quality is good; few artifacts and blurring instances are minimal.
5: Video quality is excellent; no noticeable artifacts or blurring.
Lip-Sync Accuracy:
1: Lip movements are completely out of syne with the audio.
2: Lip movements are mostly out of sync; frequent discrepancies between audio and video.
3: Lip movements are partially in sync; occasional mismatches between audio and video.
4: Lip movements are generally in syne; minor mismatches are observed.
5: Lip movements perfectly in syne with the audio.
Low
Clarity Score
Medium
Clarity
Score
High
Clarity
Score
Figure 11: Visualization of videos generated with different hand ablation scores.
21

---

Published as a conference paper at ICLR 2025
Ref Image
ANGIE
CyberHost
Ref Image
ANGIE
CyberHost
EA Erans
70.5
26- BH 10.
Fou
Ref Image
ANGIE
Joc Tike
Sc Duo
Jo- Tibe
de Co
CyberHost
Figure 12: Comparison with ANGIE in the PATS dataset.
22

---

Published as a conference paper at ICLR 2025
original
left hand
right hand
Lese Lright
5.99, 9.21
31.45, 24.50
14.78, 21.36
49.97, 18.45
Rlaft, Fright
0.18, 0.36
0.95, 0.95
0.32, 0.85
1.08, 0.73
Figure 13: Cases on cropped hand images and the calculated hand clarity score.
Ref Image
Audio-Driven Results
Ref Image
Audio-Driven Results
Figure 14: Failure cases of CyberHost in challenging scenarios.
23


Reviews:
[{"review_id": "jv0DaMHP0S", "reviewer": "Reviewer_R5UP", "summary": "The paper introduces CyberHost, an innovative one-stage audio-driven framework for generating talking body animations, addressing common issues such as hand integrity, identity consistency, and natural motion. Unlike multi-stage methods using intermediate representations like poses or meshes, CyberHost works end-to-end and supports zero-shot generation.\n\nKey innovations like Region Attention Module and the usage of Human-Prior-Guided Conditions are proposed to improve the generation quality of local human regions and to address the motion uncertainty problem.\n\nExperiments show CyberHost outperforms existing methods both qualitatively and quantitatively and works well in audio-driven, video-driven, and hybrid scenarios.", "questions": "1. Will the dataset used for training and evaluation be made publicly available? This would be valuable for reproducibility and further research by other researchers.\n\n2. Failure Cases: What are the known limitations or specific scenarios where CyberHost struggles? Highlighting these would give a more complete picture of the model’s strengths and areas for improvement.\n\n3. Full-Body Animation Scalability: Can the model be adapted for full-body animation, and if so, are there significant challenges or limitations to scaling up from half-body to full-body scenarios?\n\n4. User Study Inclusion: Could authors conduct user studies for subjective evaluations to gather human feedback on the perceived quality of the generated videos?\n\n5. DiffGesture Baseline: In the experiments section, the authors mentioned that they trained DiffGesture on the collected dataset, how did the authors get the SMPLX annotations for the collected dataset? It would also be good if the authors can quantitatively and qualitatively assess the generated SMPLX quality of the trained DiffGesture.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1. CyberHost introduces the first one-stage approach for audio-driven talking body generation, avoiding the complexity and inefficiencies of multi-stage systems that rely on intermediate representations.\n\n2. The proposed Region Attention Module component effectively enhances critical areas such as hands and faces, improving the quality of local details and maintaining identity consistency.\n\n3. By integrating motion constraints and structural priors via human-prior-guided conditions, the model mitigates the challenge of motion uncertainty, resulting in more stable and natural body animations.\n\n4. The qualitative results in the supplementary materials are impressive. Also, compared to the previous state-of-the-art audio-driven half-body generation method, VLOGGER, CyberHost produces visibly superior results.\n\n5. The paper is well-written and clearly presents its objectives, methodology, and findings.", "weaknesses": "1. Detailed Failure Analysis: The paper would benefit from a discussion of failure cases or limitations where CyberHost struggles, such as specific types of input audio or complex poses. This would provide a more balanced view of the model's capabilities.\n\n2. Scalability to Full-Body Generation: The paper focuses on half-body animation, but it does not discuss how well the architecture scales to full-body animation or if there are significant challenges in extending the framework.\n\n3. Lack of User Study for Subjective Evaluation: The paper does not include user studies or subjective evaluations to gather feedback on the perceived naturalness and quality of the generated videos. Such evaluations would provide valuable insights into how well the model meets human expectations for lifelike animation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fbd1qLPRzF", "reviewer": "Reviewer_r6Nr", "summary": "The paper introduces an end-to-end audio-driven human animation framework, which is designed to generate realistic and natural upper body human videos from a single image and control signals such as audio, ensuring hand integrity, identity consistency, and natural motion.", "questions": "1. There are issues with the injection of the codebook during inference, and the paper does not clearly explain how to accurately detect the hand position from the noisy latent space when the timestep corresponds to a higher noise level.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. Cyberhost can generate cospeech videos with very natural motions and clear hand/body structures.\n2. It employs various control training methods, including codebook , hand clarity, pose-aligned reference, and also key point supervision. Experimental results indicate that these methods effectively enhance the clarity of hands and the correctness of body structures in the generated objects.", "weaknesses": "1. The generated videos exhibit insufficient facial clarity and detail, resulting in a noticeable discrepancy between the generated object and the characteristic features of the person in the reference image.\n2. Unlike the codebook in VQ-VAE, which is specifically used for the reconstruction of designated features, the codebook in Cyberhost lacks supervisory signals during training, making it unable to ensure that the codebook effectively guides the model to generate correct hand shapes and facial features.\n3. It would be good to visualize the ablation study for the two main contribution components: “Motion codebook” and \"ID Descriptor\".", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Bt9G8eXKaC", "reviewer": "Reviewer_XLFx", "summary": "This paper proposes a one-stage audio-driven talking body video generation framework, addressing issues in half-body video generation such as blurred hand details, inconsistent identity, and unnatural motion. Specifically, it introduces a Region Attention Module (RAM) to enhance the quality of local regions. Additionally, it proposes a human-prior-guided condition to improve motion stability in generated videos. A new dataset was collected for experimentation, with results verifying the effectiveness of the proposed method and the improvements contributed by each component.", "questions": "1.\tWhen using full-body keypoints instead of the body movement map for video-driven generation, is it necessary to further fine-tune the entire model?\n2.\tHow can hand pose templates be combined within the framework to achieve multimodal-driven generation? Does this process require fine-tuning the model?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tThe proposed method demonstrates a certain degree of generalization, allowing it to adapt to multiple tasks, such as video-driven generation or multimodal-driven generation, while also enabling open-set generation.\n2.\tBased on the experimental results, the proposed method surpasses both the baseline and state-of-the-art methods across multiple metrics.", "weaknesses": "1.\tAlthough the proposed method achieves promising results overall, it introduces many components. As shown in Table 1, there are nine components, but the experiments lack in-depth analysis of these. For example, the impact of the size of the latent bank in RAM. The results of using alternatives in the Region Attention Module (RAM), such as not using spatial latents, were not examined. Additionally, the effect of not decoupling the latent bank into spatial and temporal latents—instead using a single 3D latent bank—was not investigated. Furthermore, it remains unclear what specific aspects of video information are captured by the spatial and temporal latents, lacking justification and explanation.\n3.\tThe use of the Laplacian operator to compute the hand clarity score requires justification, as the rationale behind this choice is not explicitly discussed. Additionally, the influence of the hand clarity score on the experimental results is not demonstrated in the experiments. It is essential to clarify whether this score is necessary and how it contributes to the overall performance of the proposed method.\n4.\tThe method [1] is also a one-stage audio-driven half-body video generation model, but this paper does not discuss or compare it.\n\n5.\tThe dataset used in [2] was not employed in experiments for comparison with previous methods. Additionally, the beat consistency metric [3] was not reported in the experiments.\n\n6.\tSome typos, such as in line 313 feference -> reference\n\nreference:\n\n[1] Liu X, Wu Q, Zhou H, Du Y, Wu W, Lin D, Liu Z. Audio-driven co-speech gesture video generation.\n\n[2] Qian S, Tu Z, Zhi Y, Liu W, Gao S. Speech drives templates: Co-speech gesture synthesis with learned templates.\n\n[3] Li R, Yang S, Ross DA, Kanazawa A. Ai choreographer: Music conditioned 3d dance generation with aist++", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DN4qXjuc4U", "reviewer": "Reviewer_1qwV", "summary": "This paper proposes a novel and elegant one-stage audio-driven human diffusion model. The authors primarily focus on the most challenging problems of existing body animation models, which are details underfitting and motion uncertainty. To address details underfitting, the authors introduce a region attention module, and to tackle motion uncertainty, they design a series of human-prior-guided conditions. The paper is well-written and enjoyable to read. The final video results demonstrate high-quality rendering and natural motion driving.", "questions": "1. The authors claimed that the two-stage methods are mainly limited by the capability of the pose or mesh detectors, this limitation constrains the model's ability to capture subtle human nuances. I wonder if there exists, for example, a mesh detector that provides accurate and nuanced results. What are the advantages of a one-stage method compared to a two-stage method?\n2. The authors presented various driving results, including video-driven body reenactment and multimodal-driven video generation. Was the model retrained when performing these two types of driving cases? If not, why can the body movement map be directly replaced by a skeleton or hand pose template?\n3. Is the regional mask predictor embedded in all layers? Because different layers learn different kinds of features to serve different roles in the network. Therefore, I wonder about the effectiveness of predicting regional masks in all layers. Perhaps predicting the mask from the most effective layer could perform better.\n\nConsidering the good results and novelty, I would be very willing to raise my rating if my questions are answered.\n\nNew after rebuttal: The authors provide solid extended experiments, more implementation details, and reasonable explanations about the proposed questions. Taking the good results and novelty into consideration, I think this is really a good paper. Therefore, I decide to raise my rating to 10.", "limitations": "", "rating": 10, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper addresses two important and challenging problems in the body animation field, and the proposed approaches are novel and effective.\n2. The proposed method supports multi-modalities driving\n3. The driving results show really good rendering quality and natural motion fidelity.\n4. The paper is well-organized and well-written.", "weaknesses": "1. Some details are not provided:  \na) Which specific layers of Wav2Vec features were used? (line 191)  \nb) How to constrain the basis vectors of the latents bank to be orthogonal? (line 242)  \nc) There is a lack of loss description for regional mask predictor. (line 260)  \n2. Authors claim that the hand clarity score can enhance the model's robustness to blurry hands during training and enable control over the clarity of hand images during inference. They conducted ablations on hand clarity, but they did not demonstrate to what extent this score can control hand clarity during inference. I would like to know this result.\n3. The explanation of how the proposed 'Pose-aligned Reference Feature' works has not convinced me for two reasons:  \na) Although the ablation on pose-aligned ref shows a lower HKC score compared with Cyberhost, this method was proposed to solve the case of challenging initial poses, and the authors did not demonstrate its effectiveness in that scenario.  \nb) The authors claimed that the skeleton map provides topological structure information, which improves the quality of hand generation. However, they did not explain how this structural information actually contributes to generating higher-quality hand images.  \n4. Some spelling mistakes: 'feference' should be corrected to 'reference' in line 313.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PldZUGopSd", "reviewer": "Reviewer_FbGb", "summary": "This paper introduces CyberHost, audio-driven human animation framework based on diffusion models. It addresses the less explored area of full-body human animation driven by audio signals, focusing on enhancing the generation quality of critical regions like the hands and face. The authors propose a Region Codebook Attention mechanism, along with a suite of human-prior-guided training strategies. The paper aims to bridge the gap in audio-driven human animation by improving hand clarity and overall natural motion.", "questions": "1. In Section 3.2, how was the hand heatmap estimator trained? Was it trained jointly with Equation 6 during stage 1, stage 2, or was it pretrained former to Equation 6? Also, when training the hand heatmap estimator, were all weights shared across timesteps?\n\n2. Are the Pose Encoder in the Body Movement Map and the Pose-Aligned Reference Feature shared? If they are, why are the rectangular box and human pose encoded using a shared network? What are the advantages of using a shared network compared to using different networks that share the latent space? If they are not shared, they should not be described as using the same Pose Encoder or abbreviated as \"P.\"\n\n3. Were the diffusion models initialized with pretrained weights or trained from scratch? At first, it seemed they were being trained from scratch, but in Line 191, it states, \"we extend the 2D version to 3D by integrating the pretrained temporal module from AnimateDiff.\" Could you clarify how all the components were initialized?\n\n\nSimpler Questions\n\n4. What are the dimensions of L_spa and L_temp in Latent bank?\n\n5. Starting from Line 855, how will this review system be incorporated into practical applications and future research?\n\n6. Is Laplacian standard variance sufficient for \"Hand clarity score?\"", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The authors aimed to tackle two significant challenges in audio-driven body generation and achieved progress in:\n1. Improving the synthesis quality of critical regions (hands and face)\n2. Reducing motion uncertainty caused by weak correlations.\n\nSpecifically, this paper successfully addresses the challenge of generating high-quality hand and facial features using proposed modules including RAM.\n\nIn addition, comprehensive experiments were conducted. Comparisons were made to evaluate not only audio-to-body generation methods but also video-to-video and audio-to-face methods, demonstrating its expandability.", "weaknesses": "While most parts are understandable, some details and explanations are missing. The questions regarding the missing information are listed under \"Questions.\" Additionally, as the methods utilize many well-known architectures and frameworks while introducing several modules—including the Latent Bank, Pose Encoder, Heatmap Estimator, and Mask Predictor—some missing information limits the paper’s reproducibility and clarity of the paper. If the concerns or questions listed on \"Questions\" are addressed, this paper would be worthy of a higher rating.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)