Speech2UnifiedExpressions: Synchronous Synthesis of Co-Speech Affective Face and Body Expressions from Affordable Inputs

Authors: han  chandra
kyra  kapsaskis
kurt  gray
aniket  bera
pooja  banerjee
aniket  guhan
dinesh  bera
  manocha
carlos  busso
murtaza  bulut
chi-chun  lee
abe  661 kazemzadeh
emily  mower
samuel  kim
sungbok  chang
shrikanth s lee
  narayanan
  iemo
junsong  yuan
nadia  magnenat
thalmann  exploit
] xian liu
qianyi  wu
hang  zhou
yuanqi  du
wayne  wu
xian  liu
yinghao  xu
rui  qian
xinyi  lin
xiaowei  zhou
bo  dai
Venue: thecvf_com_CVPR_2024_Workshop_HuMoGen
Published: 2024-03-13 19:12:14

Abstract: We present a multimodal learning-based method to simultaneously synthesize co-speech facial expressions and upper-body gestures for digital characters using RGB video data captured using commodity cameras. Our approach learns from sparse face landmarks and upper-body joints, estimated directly from video data, to generate plausible emotive character motions. Given a speech audio waveform and a token sequence of the speaker's face landmark motion and body-joint motion computed from a video, our method synthesizes the full sequence of motions for the speaker's face landmarks and body joints that match the content and the affect of the speech. To this end, we design a generator consisting of a set of encoders to transform all the inputs into a multimodal embedding space capturing their correlations, followed by a pair of decoders to synthesize the desired face and pose motions. To enhance the plausibility of our synthesized motions, we use an adversarial discriminator that learns to differentiate between the face and pose motions computed from the original videos and our synthesized motions based on their affective expressions. To evaluate our approach, we extend the TED Gesture Dataset to include view-normalized, co-speech face landmarks in addition to body gestures. We demonstrate the performance of our method through thorough quantitative and qualitative experiments on multiple evaluation metrics and via a user study, and observe that our method results in low reconstruction error and produces synthesized samples with diverse facial expressions and body gestures for digital characters. We will release the extended dataset as the TED Gesture+Face Dataset consisting of 250K samples and the relevant source code.

Main paper: 
 introduction: 
 Spoken communications are a significant component of everyday human-human interactions. Human communications through digital platforms and virtual spaces are prevalent in many applications, including online learning [27, 29, 44] , virtual interviewing [6], counseling [14] , social robotics [50] , automated character designing [33] , storyboard visualizing for consumer media [24, 48] , and creating large-scale metaverse worlds [38] . Simulating immersive experiences in such digital applications necessitates the development of plausible human avatars with expres-Figure 2 . Network architecture for synchronous synthesis of co-speech face and pose expressions. Our generator encodes all the inputs: the speech audio, the corresponding test transcript, the speaker ID, and the seed 3D face landmarks and the seed 3D poses into a multimodal embedding space. It decodes variables from this space to produce the synchronized sequences of co-speech 3D face landmarks and poses. Our discriminator classifies these synthesized sequences and the corresponding ground-truths (3D motions of the original speakers), computed directly from the videos, into two different classes based both on their plausibility and their synchronous expressions. To obtain our rendered 3D character motions, we combine the outputs of our generator with our phoneme predictor network and map them to 3D meshes.Synthesizing Co-Speech Expressions. We consider digital characters with faces and body gestures.Co-Speech Facial Expressions. Wang and Soong [47] compute controllable parameters for synthesizing talking heads with desired facial expressions using a Hidden Markov Model and MFCCs of the speech audio. Recent techniques automate the facial motions for large-scale synthesis, using generative paradigms such as VAEs [19] and GANs [42] .Karras et al. [23] train a DNN to map speech audio to 3D face vertices conditioned on learned latent features corresponding to different facial expressions. Zhou et al. [53] , learn sequences of predefined visemes using LSTM networks from audio. Cudeiro et al. [13] propose a dataset of 4D face scans and learn per-vertex offsets to synthesize the face motions from audio. Richard et al. [41] learn cospeech facial motions using dense face meshes by disentangling speech-correlated and speech-uncorrelated facial features. Sinha et al. [45] focus on adding emotional expressions to the faces. Lahiri et al. [26] focus on the accuracy of the lip movements and use an autoregressive approach to synthesize 3D vertex sequences for the lips that are synced with the speech audio. In contrast to these approaches, our facial expression synthesis method uses much sparser 3D face landmarks detected from real-world videos with arbitrary orientations and lighting conditions of the faces w.r.t. the cameras, and synthesizes facial and pose expressions that are mutually coherent.Co-Speech Gestures. We can consider co-speech gesture synthesis to be a special case of gesture stylization, where the style refers to the pose expressions that are inferred from and aligned with the speech. This line of work has been richly explored [3, 12, 21, 28, [30] [31] [32] 40] . Ginosar et al. [18] propose a method to synthesize speaker-specific co-speech gestures by training a neural network given their identities and individual gesticulation patterns. Ferstl et al. [16] additionally propose using adversarial losses in the training process to improve the fidelity of the synthesized gestures. input videos in the appendix (Sec. A). 
computing face and pose expressions: 
 We consider a reference neutral expression F ∈ R L×3 for each user, L being the number of face landmarks. To synthesize facial expressions, we compute the relative motion of each landmark w.r.t. the reference expression. Specifically, we obtain the configuration F t at time step t asEQUATIONwhere f t ∈ R L denotes the set of relative motions of the landmarks w.r.t. F at time step t.On the other hand, we assume the body joints are rigidly connected by the bones. We represent each user's body joints as 3D point vectors P ∈ R J×3 in a global coordinate space, where J is the number of joints. We consider directed line vectors connecting adjacent joints. The direction is along the path from the root (pelvis) joint to the end effectors (such as wrists). These 3D point vectors and line vectors collectively form a directed tree with J nodes and J -1 edges. We assume that the magnitudes of these line vectors correspond to the bone lengths and that these magnitudes are known and fixed. To synthesize the users' body gestures, we compute the orientations of these line vectors at each time step t in the reference frame of the global coordinate space. Specifically, for each bone b with bone length (magnitude) ∥b∥ and connecting the source joint s b (t) to the destination joint d b (t) at time step t, we compute a unit vector u t such thatEQUATIONWe do not assume any locomotion, i.e., we consider the root joint is fixed at the global origin at all the time steps.Our network architecture (Fig. 2 ) consists of a phoneme predictor to predict the lip shapes corresponding to the audio and a generator-discriminator pair to synthesize plausible co-speech face and pose expressions. We design our phoneme predictor following prior approaches [26] and provide its details in the appendix (Sec. B). Our generator follows a multimodal learning strategy. It consists of separate encoders to transform the speech audio, the text transcript, the speaker ID, the seed face landmark deltas, and the seed pose unit vectors into a latent embedding space representing their correlations. It subsequently synthesizes the appropriate face and pose motions from this multimodal embedding space. Our discriminator enforces our generator to synthesize plausible face and pose motions in terms of their affective expressions. To this end, we use the same encoder architecture for the faces and the poses as in our generator, but learned separately. We describe each of the components of our generator and discriminator. where θ text represents the trainable parameters.We also represent the speaker IDs k ∈ {0, 1} K as one-hot vectors for a total of K speakers and use a speaker EQUATIONThe face landmarks we use are based on action units [52] .We represent the sequence of 3D landmarks f 1:Ts ∈ 296 R Ts×L×3 as a spatial-temporal anatomical component (AC) ponents. We show the face landmarks graph in Fig. 3a 305 with all the intra-and inter-anatomical-component adjacen-cies marked with lines. We apply a sequence of spatial-307 temporal graph convolutions on this graph to learn from the 308 localized motions of the landmarks and obtain embeddings where θ STGCN f represents the trainable parameters. From the landmarks graph, we obtain a face anatomy graph,EQUATIONwhere we consider the nodes to represent entire anatomical components and the graph to be fully connected. To compute such a graph, we append the features of intraanatomical-component nodes in the graph into collated features l ∈ R Ts×L l ×n l D f , where L l denotes the number of anatomical components and n l denotes the number of landmark nodes within each anatomical component. We take n l to be the number of nodes in the anatomical component with the most landmarks and perform zero padding as appropriate to obtain the full collated features for the other components. This hierarchically pooled representation provides a "higher-level" view of the face and helps our network learn from the correlations between the motions of the different anatomical components. Specifically, we use another set of spatial-temporal graph convolutions to obtain the embed-EQUATIONwhere θ STGCN l represents the trainable parameters. Collectively, the landmarks graph and the face anatomy graph provide complementary information to our network to encode and synthesize the required facial expressions at both the macro (anatomy) and the micro (landmark) levels. To complete our encoding, we flatten out the features of all the anatomical components in l, i.e., reshaping such that l ∈ R Ts×L l D l , and transform them using standard convolutional layers on the flattened feature channel and the temporal channel separately. This gives us our latent space embeddings l ∈ R T ×Dl asEQUATIONwhere θ ConvSl and θ ConvTl represent the trainable parameters.For the pose representation, we consider a pose graph of the upper body with J -1 bones represented with line vectors u 1:Ts (Fig. 3b ). We consider bones connected to each other or connected through a third bone to be adjacent. We we apply separate spatial and temporal convolutions on the 366 flattened graph-convolved features ṽ ∈ R Ts×Lj Dv , as 367 v = ConvT ṽ (ConvS ṽ (ṽ; θ ConvSṽ ) ; θ ConvTṽ ) ,where θ ConvSṽ and θ ConvTṽ represent the trainable params. . We also zoom in on the eyebrow (row 2) and lip (row 3) expressions for better visualization. We observe a smile, raised eyebrows, and stretched arms (left) for the word 'excited', and frowns on the eyebrows and lips (right) for the words 'very sorry'.localized dependencies between the feature values followed by fully-connected layers capturing all the dependencies between the feature values (Fig. 3c ), asEQUATION)where θ ConvT f ê , θ ConvS f ê , and θ FC f ê represent the trainable parameters. The output f 1:T from the fully-connected layers has shape T × 3L, which we reshape into T × L × 3 to get our desired 3D face landmark sequences.We similarly synthesize the line vectors u 1:T ∈ R T ×(J-1)×3 using separate spatial and temporal convolutions on the multimodal embeddings ê, followed by fullyconnected layers (Fig. 3d ), as u 1:T = FC uê (ConvS uê (ConvT uê (ê; θ ConvTuê ) ; θ ConvSuê ) ; θ FCuê ), (13) where θ ConvT uê , θ ConvS uê , and θ FC uê represent the trainable parameters. Given the synthesized face and pose motions, we use our discriminator to determine how well their affective expressions match that of the corresponding groundtruths in the training data. We obtain our ground-truths as the 3D face landmarks and the 3D pose sequences computed from the full training video data.Our discriminator takes in the synchronously synthesized face motions f 1:T and pose motions u 1:T , and encodes them using encoders with the same architecture as our generator (Sec. 3.2.2), with only the number of input time steps being T instead of T s . This gives us the corresponding latent space embeddings l and v. Similar to our generator, we concatenate these embeddings into a multimodal embedding vector ê ∈ R T ×(Dl+Dṽ) . But different from our generator, we pass these multimodal embeddings through a fully-connected classifier network FC disc to obtain class probabilities c disc ∈ [0, 1] per sample, as Similar to the original TED dataset, we divide the 3D pose EQUATIONWe run quantitative experiments using ablated versions of 458 our method as baselines. We note that Habibie et al. [20] 459 retrain their network separately for individual speakers be- gestures, but are limited to the same set of fixed speakers.We also conducted a web-based user study to evaluate the 466 qualitative performance of our method. We use seven ablated versions of our method as baselines.The first two ablations correspondingly remove the entire The fifth and the sixth ablations correspondingly remove the "higher-level" anatomical component (AC) graphs of the faces (Eqn. 7) and the poses (Eqn. 10), leading to reduced movements. The final ablation trains the face and the pose expressions separately, learning marginal embeddings for the two modalities based on the speech but not attending to their mutual synchronization. This ablation is a direct evaluation of the co-speech motions when combining separately synthesized face and pose expressions. For completeness, we also compare with co-speech gesture synthesis methods that only synthesize body poses. We evaluate all the methods on our TED Gesture+Face Dataset. 
evaluation metrics: 
 Inspired by prior work [51] , we evaluate using four reconstruction errors and two plausibility errors (PEs). we also design the Fréchet Landmark Distance (FLD). We train an autoencoder network to reconstruct the full set of face landmarks at all time steps for all the samples in the training set of our TED Gesture+Face Dataset. To compute FLD, we then obtain the Fréchet Inception Distance [22] between the encoded features of the ground-truth and the synthesized samples.We show our quantitative evaluations in Table 1 .Comparison with Co-Speech Gesture Synthesis. Since co-speech gesture synthesis methods do not synthesize face expressions, we leave those numbers blank. For these methods, we have taken the numbers reported by Bhattacharya et al. [9] . For the method of SpeechGestureMatching [21] ,we retrain their method on the TED Gesture Dataset to report the numbers. However, we were unable to perform similar comparative evaluations with co-speech face synthesis methods as existing methods synthesize dense landmarks [23] or blendshape-like features [13] , which cannot be mapped one-to-one with our sparser face landmarks. We visualize some of our synthesized samples in Fig. 4 and provide more results in our supplementary video. We observe the synchronization between the face and the pose expressions for two contrasting emotions. We also visu- We conducted a user study in two sets to evaluate the visual quality of our synthesized motions in terms of their plausibility and synchronization. We provide an overview of the results here and elaborate on all the details in the appendix (Sec. E). The first set compares between our method and its ablations without the AC graph and the discriminator. The second set compares between our method and its abla-  
 
 Appendix:  


Decision: The paper received overall a positive score. There are concerns regarding the quality of the results and the relevance of the components used in the paper. To start with, the dataset used, along with the proposed extension, is slowly getting outdated with the introduction of larger, full-body datasets like BEAT  that also provide facial annotations.  Next, the method consists of too many moving parts and it is unclear if the overall pipeline is an overkill for the task at hand. Nonetheless, the reviewers note that the paper is very well polished and the method is technically correct in the sense that the learnings can be extended to more recent datasets. In light of these arguments, I recommend accepting the paper. 

The authors are, however, highly recommended to incorporate the reviewer feedback in the camera ready, especially regarding the qualitative evaluation.

Reviews: review: title: title: Learning Synchronous Face and Body Expressions from Monocular Video
main_review: main_review: This paper presents a novel synthesis pipeline for generating co-speech face and upper-body poses from speech audio. The model is trained over sparse face landmarks and poses joints extracted from video data. In this pipeline, the authors learn a multimodal embedding for each modality as their generator module, while the discriminator validates the outcome. Numerous ablation and quantitative metrics are shown. Qualitative comparisons are sparse, especially concerning other related works. They release an extended TED dataset encompassing gesture and face expressions and perform an evaluation study.

$\newline$

Overall Evaluation
---
This method seems as a good step towards learning more synchronous gesture generation encompassing different modalities. Yet simple, the architecture manages to obtain quantitatively nice results and support the importance of learning multimodal latent spaces. The sparsity of the input data might affect the overall results, and I might have liked to see more fine-grained expressions. I am not sure if such sparsity is well justified. I also miss some real-time performance metrics. Overall, I think is a good paper, although I would have liked to see a deeper analysis during the ablation studies to see the visual effects of each module to justify the synchronous generation. It was a nice addition to doing the evaluation study and releasing the dataset. I lean towards a positive review for the paper.

Pros:
* I am glad to see more work supporting multimodal latent spaces for gesture synthesis. It is a good idea and it should be more studied in future research.
* The network encompasses numerous modules to encode each modality, as well as paying special attention to the spatio-temporal correlation of facial and pose landmarks.
* Quantitative results are good, outperforming many of the related works also addressing speech-driven gesture synthesis.
* Following this, the authors show numerous examples, both qualitative and quantitative. The video is also well done overall and the results are well supported using an evaluation study. An extension for the TED Speech Gesture Dataset is also released.

Cons:
* I struggle to see the justification for doing a two-step AC component for the face and pose (especially the latter one). Is it to capture finer-detail arm motion? Why do you attach to such sparsity? I would have been interested to see an additional ablation result adding more joints/removing the AC2 completely, in addition to removing AC1 and AC2 (as explained in l.479-481). 
* I miss a deeper analysis of the qualitative results - I think is difficult to analyze the impact for each module (e.g. in the supplementary video). The synthesized gestures look a bit robotic - probably due to the sparsity of the input. A follow-up work increasing the density of landmarks and joints would be very nice.
* Regarding also qualitative results, I missed the comparisons with other related works.
* A comparison with diffusion-based methods is missing and it would have been important.

$\newline$

References
---
References are good.

$\newline$

Visual Results	
---
They are all mostly good.

$\newline$

Grammar and Style / Misc.:
---
The paper seems well written. No feedback from my side.
title: title: This paper introduces a new speech-driven face and body gesture synthesis model that can be trained using data captured from affordable inputs such as monocular videos.
main_review: main_review: **Overview**

This work proposes a new method for the task of facial expression and 3D body gesture generation from a speech input. Similar to many other related models in the domain of speech-driven gesture synthesis, the method proposed by this paper is trained using only in-the-wild videos. It proposes a new way of capturing sparse 3D face key points from a single-view image or video by leveraging a monocular face tracker of Zhang et al. (2016). At the core of their model is an adversarial neural network where the generator tries to synthesize 3D face and body key points that closely match the distribution of the training data. In addition, they also proposed a specific neural network design to enhance the mapping quality between the speech and the gesture and expression space. The paper also proposes an extension to the TED Speech Gesture dataset by introducing a new 3D facial landmark annotation. 

**Strengths**
- This work proposes a novel architectural design to tackle the problem of 3D motion synthesis from speech. 
- The network leverages a number of encoders to map the speech, text, speaker ID, seed landmark, and seed pose into the latent space. - The model also uses a specific graph-based neural network to spatiotemporally model the correlation between the body and face animation with the speech input.
- The proposed model quantitatively outperforms several related works for the task of speech-driven 3D gesture synthesis.
- The paper proposes a new metric to evaluate speech-driven 3D facial expression generation.
- It proposes an extension to the TED Speech Gesture dataset by also incorporating 3D face landmark annotations.
- The paper conducted an extensive ablation study to validate their proposed components.

**Weaknesses**
- The claim that the paper proposes a new direction of performing speech-driven gesture and facial expression synthesis from affordable input modalities such as monocular videos might be misleading, as several prior works have also attempted the same task.
- It’s quite difficult to assess the synthesis quality from the supplementary video, even for the ground truth. Retargeting the key points into a rigged character might not be the best solution for this task due to the noise produced by the monocular 3D trackers.
- It uses several metrics that compare the absolute value difference between the ground truth and the predicted key points. Due to the ambiguous nature of speech-driven gesture synthesis tasks, these metrics are no longer commonly used by the community.
- The paper does not compare the 3D facial landmark generation against other prior works. 
- The qualitative evaluation does not include comparisons against other relevant works.
- Many recent diffusion-based methods are not included in the evaluation.
- The paper does not fully elaborate on why the proposed action unit-based facial key point is a better choice than the key point produced by fitting a 3DMM model like in Habibie et al. (2021) or Yi et al. (2023).

**Justification**

This paper presented a novel approach to synthesizing 3D face and body gestures from speech input. However, the paper does not fully justify some of the claims and proposed components. The paper also did not conduct a qualitative comparison against prior works.