DiffTED: One-shot Audio-driven TED Talk Video Generation with Diffusion-based Co-speech Gestures

Venue: thecvf_com_CVPR_2024_Workshop_HuMoGen
Published: 2024-03-19 21:04:42

Abstract: Audio-driven talking video generation has advanced significantly, but existing methods often depend on video-to-video translation techniques and traditional generative networks like GANs and they typically generate taking heads and co-speech gestures separately, leading to less coherent outputs. Furthermore, the gestures produced by these methods often appear overly smooth or subdued, lacking in diversity, and many gesture-centric approaches do not integrate talking head generation.
To address these limitations, we introduce DiffTED, a new approach for one-shot audio-driven TED-style talking video generation from a single image. Specifically, we leverage a diffusion model to generate sequences of keypoints for a Thin-Plate Spline motion model, precisely controlling the avatar's animation while ensuring temporally coherent and diverse gestures. This innovative approach utilizes classifier-free guidance, empowering the gestures to flow naturally with the audio input without relying on pre-trained classifiers. Experiments demonstrate that DiffTED generates temporally coherent talking videos with diverse co-speech gestures.

Main paper: 
 method: 
 In this section, we introduce our DiffTED. A framework overview is shown in Fig. 1 . It consists of two main parts, video generation and a diffusion model for co-speech gesture generation. We first introduce the formulation of the problem, then discuss the video generation, and finally the diffusion model.To accomplish one-shot talking video generation from a single image and a driving speech audio, we first collect video clips of N frames and the corresponding speech audio a = {a 1 , ..., a N }. We extract keypoints, x = {p 1 , ..., p N }, from the image using a pre-trained keypoint detector from Thin-Plate Spline (TPS) motion model [30] . Keypoint sequences are normalized using the global mean, µ, and standard deviation, σ. The normalized sequences are calculated as x = (x -µ)/σ. Our gesture generation model generates the normalized keypoint sequence x conditioned on the audio sequence a and initial M normalized keypoint frames {p 1 , ..., p M }. The model uses these M keypoint frames to set the initial pose of the speaker and we also use them to interpolate between segments of longer sequences. For one-shot video generation, we take the keypoints from the source image to use as the initial keypoints. The generated keypoints are then used to drive the video generation.For generating video frames, we use the Thin-Plate Spline Motion Model [30] . To do this, we make use of its dense motion network and inpainting network. Since the key- Motivated by the success of recent diffusion models [7, 31] ,EQUATION219we propose a novel diffusion model-based approach for 220 generating co-speech gesture keypoint sequences.The goal of diffusion is given some data sample, x 0 , 222 from the real data distribution q(x 0 ), to learn a model dis-tribution, p θ (x 0 ), that approximates the real distribution.The forward, or diffusion, process is a Markov chain, 225 q(x t |x t-1 ) for t = {1, ..., T } in which Gaussian noise,N (µ, σ 2 ), following a variance schedule β 1 , ..., β T , is it-eratively added to the data sample, x 0 , eventually leading 228 to pure noise. This process is defined as:EQUATION)The reverse, or denoising, process p, then goes in the op- EQUATIONFor optimization of the network, we follow DDPM 243 [7] in optimizing the variational lower bound on nega-244 tive log-likelihood: E[-log p θ (x 0 )] ≤ E q [-log p θ (x0) q(x 1:T |x0) ]245 [7] . Eliminating constant items that do not require train-ing and adding conditioning on the contextual informa-247 tion, c, we rewrite the loss function to:L noise (θ) = 248 E q [ T t=2 D KL (q(x t-1 |x t , x 0 )∥p θ (x t-1 |x t , c))].We fur-ther follow [7] to simplify the noise loss to:EQUATIONHere, ϵ ∼ N (0, I) is Gaussian noise that the network, 252 ϵ θ (x t , c, t) is trying to predict. And with α t = 1 -β t and 253 ᾱt = t s=1 α s , the noisy keypoint sequence x t , is defined 254 as: EQUATIONEQUATION)The value of the scaling parameter, s, can be increased or 275 decreased to make a trade off between gesture diversity and 276 quality. With a larger s, diversity will increase, but the generated gesture will reduce in quality. For the experiments discussed in Sec. 4, we use s = 0.2. 
experimental settings: 
 Dataset.Our model is trained on the TED-talks dataset [18] . The training videos are downscaled to a resolution of 384×384, focusing on the upper part of the human body, and resampled to 25 FPS. Videos are in the range of 64 to 1024 frames. To train our model, we use the keypoints from the learned keypoint detector in [30] to get the ground truth keypoints for each frame. For video generation, the first image from each video clip is used.Metrics. We use five quantitative metrics to evaluate our pipeline, three for measuring the final image quality and two for measuring only the gesture sequences.• Fréchet Inception Distance (FID) [6] : Aims to measure the similarity between generated and real images, in an attempt to reflect the image quality as it would be perceived by humans.• Fréchet Video Distance (FVD) [22] : An extension of FID to videos, assessing the overall quality of generated videos by evaluating temporal coherence and image quality.• Learned Perceptual Image Patch Similarity (LPIPS) [29] : An attempt to evaluate perceptual similarity between images based on deep learning features, which corresponds well with human judgment.• Diversity (Div): To measure the diversity, we follow [31] and train an auto-encoder on the keypoints to extract features of the generated gesture sequences and measure the mean feature distance between generated gestures and the ground truth gestures.• Beat Consistency (BC): In order to determine how well the generated sequences align with the cadence of human speech, we measure the beat consistency as in [31] , but as we do not have a skeletal structure, we instead use the change in velocity of keypoints in adjacent frames to detect motion beats.Implementation Details. Because there is no existing method for one-shot video generation that can generate audio-driven co-speech gestures, we instead adapt two existing methods that generate 2D keypoints. The first method, EAMM [9] utilizes an LSTM-based architecture to learn a 2D keypoint detector. We then follow Speech2Gesture [5] to implement a 1D Unet [8, 17] to represent CNN-based models. In both EAMM and Speech2Gesture adaptations we train on our learned 2D keypoints rather than the face and skeletal keypoints from those two works. These keypoints are then used on the same TPS keypoint-driven image transformation framework. worse than the diffusion model. The FVD metric takes into consideration the temporal coherence of a video, where the EAMM and Speech2Gesture models trail behind our method.HuMoGen #11 HuMoGen #11 HuMoGen 2024 SubmissionQualitative Results. In Fig. 2 , we show several frames from a sequence to showcase gesture diversity. We show the sequence with (Fig. 2b ) and without (Fig. 2a ) the diffusion generated keypoints. The gestures shown have a wide range of motion in both the arms of the speaker. EQUATIONHere, instead of predicting the noise we directly diffuse the 389 keypoint positions, xθ (•). The noise, in the base diffusion 390 model is subsequently removed from the noisy sample, but 391 in this method, the denoised sample is instead predicted di-392 rectly. This method has been used recently in EDGE [21] 393 and MDM [20] . In these works the method is shown to on the noise produces more natural looking limbs. While, as mentioned previously predicting the denoised sample directly instead of predicting the noise shows good results in other work (EDGE [21] and MDM [20] ), this direct prediction leads to some artifacts not shown in the noise prediction model. However, with these artifacts, the method still outperforms the other baseline methods, and, potentially with additional losses, this method shows to be a promising direction for improving on this work.User Study. The metrics used to quantitatively measure the video generation aim to mimic human perception and mirror human quality assessment but leave room for improvement. As such, we conduct a user study to better validate the qualitative performance of our model. The study consists of 10 participants, who grade videos based on the quality of the generated gestures rather than the images. Specifically, we take 10 audios to generate videos for 5 different methods. The methods include the ground truth keypoints, DiffTED (our method), DiffTED but predicting keypoint position rather than the noise, the EAMMbased method [9] , and the Speech2Gesture-based method Table 3 . User study results. The ratings on naturalness of gesture, smoothness of gesture and synchrony between speech and gesture are done on a scale of 1 to 5, where 5 is the best.[5], with the order of these methods being shuffled for each 434 HuMoGen #11 HuMoGen #11 HuMoGen 2024 Submission 
: 
 . These methods, while able to produce compelling gestures, still leave the problem of transferring the gestures to images to produce videos or else are limited to the use with virtual avatars.Additionally, gesture generation methods in 3D methods are able to work on the skeleton and thus the translation to video is non-trivial. Though, the skeleton offers several advantages to gesture generation, especially when not tasked with rendering a final video, such as not taking into consideration rigid constraints such as limb length. The methods can work with angles and direction vectors and then later apply predefined lengths to limbs to generate realisticlooking skeletal representations [14, 27, 31] .When rendering videos, some method of translating the pose or 3D body must be used but this is non-trivial, especially when considering texture. Methods in 2D can inherently use actual people/bodies and operate in image space and thus do not have to perform this transfer. However, without the third dimension depth ambiguity can become an issue. This means that body size or limb length can change from frame to frame and create unrealistic gestures.Using skeletal motion in 2D can be one solution but ambiguous angles in the 2D still provide some challenges. 2D audio-driven video generation methods such as ANGIE [13] learn an unsupervised motion representation rather than the skeleton but it is limited to the front-facing upper torso of the body and has a complex network structure requiring large amounts of data and long training times.In this paper, we propose DiffTED, the first one-shot audio-driven TED-style talking video generation from a single image with diffusion-generated co-speech gestures. The existing methods [5, 12] rely on video-to-video translation [8, 23] to render end results and as such, are unable to make a one-shot video generation pipeline. We choose to create a one-shot video generation method to be able to create videos of an arbitrary person with an arbitrary speech audio, rather than be bounded by the training subjects or having to retrain for additional people. We propose instead to utilize another approach to facilitate the one-shot video generation, learned 2D keypoints of Thin-Plate Spline (TPS) mo- Existing works [5, 12, 13, 16, 28] synthesize talking video 114 from a sequence of 2D skeletons [5, 16] or 3D models 115 [12] with the rendering process being disjoint from the 116 generation of the gestures. In Speech2Gesture [5] and 117 Speech2Video [12] they generate the gestures using a GAN, 118 however, their methods suffer from a lack of diversity due 119 to problems inherent in GANs like mode collapse. Qian et 120 al. [16] use a VAE to model the distribution of gestures 121 by learning a template vector that is mapped to a gesture 122 sequence. In ANGIE [13] , they use an unsupervised mo- hands or face. There have been many approaches to the design of the co-speech gesture predicting DNNs focusing on input modality (text or audio) or architecture. Some works use the speech text, audio, or both as input [10, 27] and they may additionally include other contexts like speaker identity [27] . There have been many architectures used in cospeech generation with the use of transformers [13, 15, 19] ,RNNs [3, 12, 26] , GANs [5, 12, 14, 27] , VAEs [11, 16] , flow-based models [1, 25] and recently diffusion models [4, 31] . There has also been the recent introduction of VQ-VAE [3, 19, 24] in works to help keep diversity in the generated gestures. In DiffGesture [31] , they introduce the use of a DDPM-like model for co-speech gesture generation on the 3D keypoints of a partial 3D skeleton to try and solve the problem of generation of diverse gesture sequences. All these co-speech gesture generation methods do not pay attention to the problem of video generation after the gestures are generated. In this paper, we use a DDPM-like model on learned 2D TPS keypoints, which bridges the gap between co-speech gesture generation and one-shot video generation. 
 
 Appendix:  


Decision: The paper addresses the task of creating a video with gestures from a single input image accompanied by speech input.

Pros:
* Predicting TPS points enables seamless video generation
* Comprehensive evaluation and ablation

Cons:
* Limited novelty
* The paper lacks clarity and is missing many details.
* The authors do not show results on the faces, while they criticize other methods on this matter.

The discussion about this paper was extensive, making it challenging to reach a decision. There was a notable concern regarding the writing quality. Ultimately, the decision was made to accept the paper with a firm recommendation to address the reviewers' feedback and rectify any identified issues.

**Guidance to authors:** Resolve concerns raised by the authors, particularly ones related to the clarity of writing.

Reviews: review: title: title: A diffusion-based co-speech gesture video generation framework that enables the generalization to an unseen speaker with a single source image. I recommend acceptance.
main_review: main_review: Most of co-speech gesture video generation works require model fine-tuning on additional corresponding speaker data to generalize to a new, unseen speaker. The paper proposes a framework that supports a new speaker with a single source image. Specifically, the authors train a diffusion model to generate gesture sequences represented by general TPS keypoints. Then, a video renderer drives the source speaker image, conditioned on the generated gesture sequences. This setting is practical and has the potential to evolve into a useful product (The final video quality needs optimization).

**Pros**
* A setting that enables the generalization of a gesture model to an unseen speaker with a single source image.
* The paper explores a diffusion model to translate speech into 2D TPS keypoints, demonstrating its superior performance over LSTM-based and CNN-based models.

**Cons**
* Some blurry artifacts are present in the final video results (as discussed in Section 5). Employing a more advanced rendering module, such as Champ [1], may improve the visual quality.
* The lip movements appear to be poorly aligned with speech.

In summary, the one-shot setting is intriguing and could inspire subsequent research. The authors present a good method demonstrating the potential for constructing such a one-shot generation framework. I recommend acceptance.
title: title: Original approach to gesture generation. The paper lacks in terms of clarity and missing crucial implementation details
main_review: main_review: Pros: 
-  Unique approach for conditional gesture generation using key points, allowing straight-forward solution for video-rendering, unlike many other methods which does not cover the rendering part. 
- Comprehensive evaluations, using various metrics demonstrating different qualitative and quantitative characteristics of the results. Also, a lot of effort was made to overcome the differences between the proposed solution and the others, and to make a fair comparison (to an extent).
- Interesting ablation study. I found it interesting that predicting the noise worked better than predicting key-points positions in most metrics and it lead to an interesting discussion. 
- Qualitative results look good. 

Cons: 
- The paper is not self-contained. The proposed method strongly relies on an existing solution called TPS Motion Model, which is assumed to be known by the readers and not presented (not even in related work section). Furthermore, in section 2.1 the authors refer to specific components of TPS Motion Model, when the goal of referenced solution is not clear at this point.
- Missing details regarding the proposed solution, while less-relevant topics are overly detailed. In section 3.3, Diffusion-based TPS Keypoints Generation, the well-known diffusion process is told in details while the authors barely state any insights of their own or supply any information regarding their specific implementation. The authors didn't supply any information of how the condition was incorporated. 
- Missing citations to support statements: L137-139, L139-141. 
- Results seems a bit cherry-picked. In addition, on supplementary video, faces are blurred in claim that the solution does not deal with faces, while in the paper other solutions failure on faces are presented as disadvantage.  
- Missing experiment details: Information about the test set which was used for the evaluation is missing, and also it's unclear how the audio and initial keypoints conditions were incorporated in the compared solution, which makes the comparison fairness uncertain. 
- Missing related works: as mentioned before, TPS Motion Model is missing from the related work section, but also diffusion-based gesture generation solutions are missing, such as:
1. Anna Deichler, Shivam Mehta, Simon Alexanderson, and Jonas Beskow. 2023. Difusion-based co-speech gesture generation using joint text and audio representation. In Proceedings of the ACM International Conference on Multimodal Interaction (ICMI ’23). ACM. 
2. YANG S., WU Z., LI M., ZHANG Z., HAO L., BAO
W., CHENG M., XIAO L.: Diffusestylegesture: Stylized audio-driven
co-speech gesture generation with diffusion models. arXiv preprint
arXiv:2305.04919 (2023)

Comments/Requests for final version (if accepted): Please add information about the condition representation and incorporation in your framework, add diffusion-based gesture generation solutions citations and also explain why you couldn't compare with these solutions. 

The proposed method offers original end-to-end solution to the problem it aims to solve, and comprehensive evaluation of the results was conducted. At the same time, the paper lacks in terms of clarity, and crucial information about the method is missing. Considering both pros and cons, I've decided to rate this paper as borderline.