Emphasizing Semantic Consistency of Salient Posture for Speech-Driven Gesture Generation

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-22 06:24:02

Abstract: Speech-driven gesture generation aims at synthesizing a gesture sequence synchronized with the input speech signal. Previous methods leverage neural networks to directly map a compact audio representation to the gesture sequence, ignoring the semantic association of different modalities and failing to deal with salient gestures. In this paper, we propose a novel speech-driven gesture generation method by emphasizing the semantic consistency of salient posture. Specifically, we first learn a joint manifold space for the individual representation of audio and body pose to exploit the inherent semantic association between the two modalities, and propose to enforce semantic consistency via a consistency loss. Furthermore, we emphasize the semantic consistency of salient postures by introducing a weakly-supervised detector to identify salient postures, and reweighting the consistency loss to focus more on learning the correspondence between salient postures and the high-level semantics of speech content. In addition, we propose to extract audio features dedicated to facial expression and body gesture separately, and design separate branches for face and body gesture synthesis. Extensive experiments and visualization results demonstrate the superiority of our method over the state-of-the-art approaches.

Main paper: 
 related work: 
 Audio-Visual Cross-Modal Learning. Multi-modal machine learning aims to train models capable of processing and relating information from multiple modalities. Recent works [1, 2, 35] encode all the modalities into a common representation space. Language2Pose [2] learns a joint embedding of text and pose. Ahuja et al. [1] maps the learned style embedding along with audio into a joint gesture space. Trimodal [35] utilizes separate representations for different modalities and handles the alignment between speech and gesture explicitly. Compared with these methods, our method focuses on identifying the strong semantic correlations between speech audio and visual gestures to synthesize more natural-looking and vivid gesture sequences. Speech-Driven Gesture Generation. Synthesizing consistent and natural gestures in line with speech is becoming popular in the field of multi-modal generation. With the development of deep learning, recent works [3, 13, 29, 32] leverage deep neural networks to generate more natural gesture sequences. Audio2Body [32] learns the correlation between audio features and body landmarks through an LSTM network and concentrates on predicting body motion from specific music like piano and violin. S2G [13] incorporates generative adversarial learning into the regression-based prediction model to enhance the naturalness of generated results. MoGlow [3] adapts the probabilistic model to this task to learn the distribution of gesture motions and can effectively take control over gesture styles. SDT [29] learns template vectors to provide extra information for gesture generation and transform the one-to-many ambiguous regression problem into a deterministic conditional regression problem. However, these methods ignore the semantic association of different modalities and fail to deal with habitual and salient gestures. Our method utilizes joint manifold space to model the mapping function and design a salient posture detector to maintain semantic consistency of salient gestures. Anomaly Detection. Video anomaly detection aims to identify abnormal events in the video that do not match the normal behaviors. Early works [16, 18, 27, 28] focus on detecting anomalies through manually extracting features and modeling the anomalies.Recently, several deep learning-based methods [14, 23, 25, 26, 37] have achieved significant performance improvement. Reconstructionbased methods [14, 26, 37] utilize the autoencoder trained on normal datasets and detect anomalous frames which are difficult to be reconstructed well. Prediction-based methods [23, 25] predict future frames and utilize prediction error as an indicator to determine anomalies, considering the frames with large prediction errors as anomalies. Different from these methods, we design a weaklysupervised salient posture detector to identify anomaly gestures only under the weak supervision of video-level labels. and reweight the cross-modal association based on the predicted saliency score. 
method 3.1 overview and notations: 
 To fully empower the learning of semantic association between speech and gesture, we propose a novel speech-driven gesture synthesis method that emphasizes semantic consistency of salient postures, i.e., postures with large movement scope that often correspond to strong semantics. Our overall architecture is shown in Figure 2 . Our model first learns a joint manifold space for different representations of audio and body pose to explore a finer mapping between two modalities. Then, we emphasize the semantic consistency of salient postures by introducing a weakly-supervised detector to identify salient postures, and enforcing a stronger alignment for the salient postures in the joint manifold space. In addition, observing that facial expressions rely on articulation-related audio features while body gestures rely on semantic-related audio features, we extract separate audio features for face and body, and design separate branches for face and body gesture synthesis. The overall architecture of our proposed method, which consists of two branches including body synthesis branch and face synthesis branch. In the body synthesis branch, our model learns a joint manifold space of representations to enforce semantic consistency by employing a dual-path structure, which contains the upper reconstruction path and the lower speechdriven generation path. Furthermore, the salient posture detector is designed to identify salient gestures and reweight the consistency loss. We then generate synchronized facial expressions using face synthesis branch. Finally, we fuse the generated results of two branches to obtain the entire gesture sequence.Given an audio clip 𝐴 = [𝐴 1 , . . . , 𝐴 𝑇 ], speech-driven gesture synthesis aims to generate a pose sequence 𝑃 = [𝑃 1 , . . . , 𝑃 𝑇 ] synchronized with the audio, where the length of sequence is 𝑇 and 𝑃 𝑡 is the pose at the 𝑡 th frame. Each 𝑃 𝑡 is defined as the coordinates set of 𝐽 upper-body keypoints in a frame, including face, hands, and arms. The major challenge of synthesizing realistic gestures is how to establish the reasonable mapping: 𝐴 1:𝑇 → 𝑃 1:𝑇 . Due to the discrepancy of mapping property of different parts, we decompose the entire pose 𝑃 𝑡 into body part 𝑃 𝑏 𝑡 and face part 𝑃 𝑓 𝑡 . The whole gesture sequence 𝑃 can be formulated as: 𝑃 = 𝑃 𝑏 ⊕ 𝑃 𝑓 .For the generation of body poses, we first utilize two feature extractors to extract audio and pose latent codes respectively. Then, we use a pose decoder to separately generate poses from these two latent codes. The output of our pose decoder is a sequence of 𝑇 frames. We denote the pose sequence generated from pose latent codes as P𝑏 𝑝 , and the pose sequence generated from audio latent codes as P𝑏 𝑎 . Then, for facial expressions synthesis, we extract different audio features (articulation-related) and directly generate a sequence of 𝑇 frames P 𝑓 𝑎 . Eventually, by fusing the synthesis results of the two branches, we obtain the final output of the entire gesture sequence P = P𝑎 = P𝑏 𝑎 ⊕ P 𝑓 𝑎 .Due to the high randomness of body motion, our model learns a multi-modal joint manifold space between audio and body pose to explore the semantic correlation between audio and pose representations. To form the shared joint-embedding space, as shown in the upper part of Figure 2 , we employ a dual-path architecture that consists of two parallel pipelines: the reconstruction path for body pose and the speech-driven gesture generation path. The reconstruction path takes as input the real pose sequence of body part 𝑃 𝑏 , and uses a pose encoder 𝐸𝑛𝑐 𝑝 consisting of two GRUs to obtain a pose feature in the joint manifold space,EQUATIONwhere 𝑍 𝑏 𝑝 ∈ R 𝑇 ×𝐷 is the latent code of body pose and 𝐷 represents the dimension of the latent code.For the speech-driven gesture generation path, we first utilize the audio encoder 𝐸𝑛𝑐 𝑎 to encode the mel-spectrogram of audio and concatenate it with the feature code 𝑓 𝑑 extracted by the 𝐷𝑒𝑒𝑝𝑆𝑝𝑒𝑒𝑐ℎ [4] model, which is pre-trained by large numbers of audio-transcript pairs. The integration of high-dimensional representations of 𝐷𝑒𝑒𝑝𝑆𝑝𝑒𝑒𝑐ℎ can provide richer semantic information for follow-up gesture generation. We further map the concatenated feature to an audio feature in the joint manifold space using 1D UNet translation network,EQUATIONwhere 𝑍 𝑏 𝑎 ∈ R 𝑇 ×𝐷 is the latent code of audio corresponding to body part.To guarantee that 𝑍 𝑏 𝑝 and 𝑍 𝑏 𝑎 lie close to each other in the shared joint-embedding space and represent similar semantic information, we propose the consistency loss to constrain the latent codes. Similar to the cosine similarity function, the consistency loss calculates the cosine similarity between the body pose latent code and audio latent code: where 𝑍 𝑏 𝑝,𝑡 and 𝑍 𝑏 𝑎,𝑡 are the 𝑡 th value of latent codes along time axis respectively, and 𝜖 is a fairly small positive scalar.EQUATIONThen, the decoders of the reconstruction path and generation path learn to generate the pose sequences P𝑏 𝑝 and P𝑏 𝑎 from the above two latent codes,EQUATIONHere, two decoders share the same network parameters and both are denoted as 𝐷𝑒𝑐.Salient postures are the postures with a large range of motion, which are closely related to strong semantics of the speech content. Therefore, we propose a weakly-supervised salient posture detector to predict the saliency score of poses for each frame, which will then be used to reweight the consistency loss. This strategy enforces a more accurate alignment of salient pose representation and corresponding audio representation in the joint embedding. The architecture of the detector is depicted in Figure 3 . First, we utilize the inflated ConvNet [7] to extract the initial frame-level feature 𝑋 ∈ R 𝑇 ×𝐷 1 , where 𝐷 1 denotes the dimension of the initial feature. Then, we feed 𝑋 into a temporal relation module to capture long-range temporal dependencies among features and transform 𝑋 into the interaction feature 𝑌 ∈ R 𝑇 ×𝐷 1 . Then, a classifier is used to project the interaction feature into the 1D vector space to obtain the frame-level saliency score of poses 𝑆 𝑏 ∈ R 𝑇 . We use the average saliency score of 𝑡𝑜𝑝-𝑘 frames as the saliency prediction for the video sequence, and make it close to the video-sequence-level saliency label using the binary cross-entropy loss.To facilitate the network to adaptively learn which frames are salient, we train the detector under the weak supervision of the video-sequence-level saliency labels. To acquire the sequence-level labels, we first derive the resting pose (the most frequent posture) of each speaker in the entire dataset as illustrated in the S2G [13] . For a pose sequence, we calculate the 𝐿 2 distance between the pose of each frame and the resting pose. Then we can obtain the probability distribution of all distances and determine whether the sequence contains salient data points based on the triple variance criterion. Video sequences containing salient points are assigned label 1 and otherwise 0.Additionally, the ground truth of poses is pseudo and obtained using OpenPose [5] , therefore it is inaccurate to directly take the saliency points as the frame-level labels. Sequence-level saliency labels can mitigate this error and facilitate the detector and generation network to learn the exact correspondence between salient postures and speech semantics. See the comparison results in Table 6 of the ablation study.Inspired by the temporal attention architecture [34] , we design the temporal relation module (TRM) to mine the most relevant information between frames to facilitate salient posture detection. Compared with [34] , our TRM leverages the global information of a sequence instead of local information in neighbor frames to exploit long-range temporal dependencies, and capture the index prior of all frames to improve the representation capability.Specifically, we first compute the adjacency matrix for the initial frame-level feature 𝑋 to measure the affinity in the embedding space, and then normalize the matrix using softmax function along each row. We denote the normalized adjacency matrix as weight matrix 𝑊 1 , which can be formulated as:EQUATIONwhere 𝜃 and 𝜑 denote linear transformation functions.Then we incorporate the frame index prior into the network to distinguish the importance of neighbor frames to the current frame. We denote the index prior matrix as 𝐼 = [𝑖 1 , 𝑖 2 , . . . , 𝑖 𝑡 , . . . , 𝑖 𝑇 ], where 𝑖 𝑡 is the negative relative distance vector between the current frame (𝑡 th ) and all frames. Since frames closer to the current frame have more relevant information, we use the softmax function to transform the index matrix 𝐼 into another weight matrix 𝑊 2 , where closer frames are mapped to higher weights. We then concatenate 𝑊 2 with 𝑊 1 to compute the final interaction feature 𝑌 :EQUATIONwhere 𝜙 denotes the transformation functions implemented by FC layers. upper part of Figure 2 ). The reweighted consistency loss can be re-formulated as follows:EQUATIONwhere 𝑆 𝑏 𝑡 is the predicted saliency score for the body pose in the 𝑡-th frame. The reweighted consistency loss with saliency score can effectively enforce the model to focus more on learning the mapping between salient pose and the high-level semantics of audio during the joint training.3.4.1 Separate Audio Feature for Face and Body. Synthesizing facial expressions and body gestures as a whole often leads to poor lip synchronization, since different audio features are required for the generation of face and body parts: articulation-related acoustic features are required for face and semantic-related acoustic features for body. Therefore, we extract audio features dedicated to facial expressions and body gestures separately, and synthesize facial expressions and body gestures with separate branches. As shown in the lower part of Figure 2 , for the face synthesis branch, we utilize a separate audio encoder and UNet to extract the rhythmic representation of audio 𝑍 𝑓 𝑎 ∈ R 𝑇 ×𝐷 . Then, we decode 𝑍 𝑓 𝑎 to generate a sequence of 𝑇 frames of facial keypoints P 𝑓 𝑎 using a decoder, which has the same network structure as the body synthesis branch but different parameters.Face-Body Feature Alignment. Although the separate body and face branches can yield individual prediction results well aligned with audio, the final gesture sequences generated by direct concatenation of the two results tend to be inconsistent and unnatural. Therefore, we design a binary classification task at the feature level to enforce the temporal alignment between representations of the body branch and face branch. As shown in Figure 4 , from the semantic-related acoustic feature 𝑍 𝑏 𝑎 (for body gesture synthesis) and the articulation-related acoustic feature 𝑍 𝑓 𝑎 (for facial expression synthesis), we randomly sample body feature sequence 𝑍 𝑏 𝑎,𝑡 𝑏 and face feature sequence 𝑍 𝑓 𝑎,𝑡 𝑓 , which have the same length of frames 𝑇 𝑐 . Here, 𝑡 𝑏 and 𝑡 𝑓 represent the starting index of the feature sequences respectively. We denote the one-hot label as 𝑐 ∈ R 2 to distinguish whether the audio features corresponding to the body and face parts are well aligned over the temporal dimension. Then, we can obtain the labeled feature pairs including the aligned positive pairs ( if 𝑡 𝑏 = 𝑡 𝑓 , then 𝑐 = [1, 0] ) and the unaligned negative pairs ( if 𝑡 𝑏 ≠ 𝑡 𝑓 , then 𝑐 = [0, 1] ). We take the labeled feature pairs as training data to optimize the binary classifier 𝐶 to determine whether the two input features are temporally aligned. As depicted in Figure 4 , the structure of 𝐶 consists of a GRU and three FC layers. With this module, we can align the feature space of both branches by self-supervised learning, resulting in realistic gesture generation with better synchrony. Formally, the loss function of the classification task can be formulated as: where 𝐶 ( * , * ) is the classification result of classifier 𝐶.EQUATIONReconstruction Loss. The goal of our reconstruction path of the body synthesis branch is to learn the pose representation space by recovering the input body pose sequence. Here, we use 𝐿 1 loss function to measure the distance between the reconstructed body poses P𝑏 𝑝 and ground truth pose sequences 𝑃 𝑏 :EQUATIONRegression Loss. The main supervision of the training process is imposed on the pose keypoints, including the supervision of the overall keypoints as well as the individual part keypoints. The supervision on the overall pose keypoints is implemented as a 𝐿 1 regression loss between the entire gestures P𝑎 generated from the given audio and ground truth gestures 𝑃:EQUATIONBesides, we additionally apply separate constraints on face pose keypoints and body pose keypoints by measuring the distance between gestures P𝑏 𝑎 , P 𝑓 𝑎 generated from the given audio, and ground truth 𝑃 𝑏 , 𝑃 𝑓 with Huber loss (𝐻𝐿) [17] :EQUATION)Then the entire loss of face part and body part is calculated as the mean value of 𝐿 𝑓 𝑎𝑐𝑒 and 𝐿 𝑏𝑜𝑑𝑦 :EQUATIONOverall, the total training objective function is:EQUATIONwhere 𝜆 𝑟 , 𝜆 𝑟𝑒𝑔 , 𝜆 ℎ , 𝜆 𝑐𝑜𝑛 , and 𝜆 𝑐 are hyper-parameters that can be adjusted to control the relative significance of each loss term. Attention Seq2Seq [36] 54.92 0.15 7.58 S2G [13] 54.65 0.68 7.95 Joint Embedding [2] 64.56 0.13 7.52 Trimodal [35] 12.61 0.56 7.89 HA2G [24] 5.31 0.64 7.11 DiffGesture [38] 2.60 0.72 6.97 Ours 2.50 0.68 6.83 
experiments 4.1 datasets: 
 Speech2Gesture. Speech2Gesture [13] is a speaker-specific dataset with full body and face keypoints annotations. Following SDT [29] , we test our method on four speakers: Oliver, Kubinec, Luo, and Xing. The number of different videos of four speakers is 113, 274, 72, and 27, with a total length of about 25 hours. All pseudo keypoint annotations are obtained by OpenPose [5] , which contains 121 keypoints. We divide them into different parts, including 70 keypoints for the facial expression, 51 keypoints for the body pose. TED Expressive. TED Expressive dataset [24] is derived from a large-scale 3D gesture dataset TED Gesture [35] . Compared to TED Gesture only with 10 upper body key points, TED Expressive is more realistic and expressive of both body and finger movements. Through 3D pose estimator ExPose [9] extracting the pose information, TED Expressive contains the 3D coordinate annotations of 43 keypoints, including 13 upper body joints and 30 finger joints.We use the data pre-processing protocol in SDT [29] to partition videos of 15 FPS into segments with 64 frames for training. We use a 1-layer GRU as our pose feature extractor with the hidden size of 1024 and the dimension 𝐷 of our audio-pose joint embedding space is 512. For the salient posture detector module, the dimension 𝐷 1 and 𝐷 2 of initial feature and interaction feature are set to 512 and 1024. In addition, we set the value of 𝑡𝑜𝑝-𝑘 to 16. For both training and testing, we use a batch size of 32. We train our model with an Adam optimizer and the learning rate is set to 0.0001. For the hyperparameters , we empirically set 𝜆 𝑟 = 10, 𝜆 𝑟𝑒𝑔 = 10, 𝜆 ℎ = 20, 𝜆 𝑐𝑜𝑛 = 1, and 𝜆 𝑐 = 1, which work well for both datasets.4.3.1 Evaluation Metrics. 𝑳 2 Distance is commonly used to measure the distance between the generated gestures and ground truth.Fréchet Gesture Distance (FGD) is proposed by [35] to measure the distribution distance of ground truth gestures and generated ones in the latent space. Note that we use the same pose feature extractor as [29] for a fair comparison.Beat Consistency Score (BC) is originally proposed by [24, 38] to measure the beat correlation between audio and gesture. They utilize the angle changes of bones to quantify the motion beat and calculate the average distance between audio beat and its nearest motion beat as Beat Consistency Score.Pose-Sync Distance (PSD) is designed by us to evaluate the consistency of audio and generated pose sequence. Inspired by Sync-Net [10] , we train a Pose-SyncNet using a similar contrastive loss between the representations of audio and generated pose. We compute this evaluation metric using the 𝐿 2 distance between the audio embedding and pose embedding of our pre-trained Pose-SyncNet.Concretely, for a pose sequence p of 64 frames with corresponding audio a, we evenly divide the pose sequence into 𝑁 pose clips 𝑝 𝑖 of 9 frames and audio clips 𝑎 𝑖 of 0.6𝑠, and encode them using Pose-SyncNet to obtain corresponding pose feature 𝑓 𝑝 𝑖 and audio feature 𝑓 𝑎 𝑖 . Then, we calculate the PSD metric as follows:EQUATION4.3.2 Evaluation Results. We make a comprehensive performance comparison between our method and other state-of-the-art methods on both datasets using the above metrics. As shown in Table 1 and Table 2 , our method almost achieves the highest performance in all evaluation metrics, which demonstrates great superiority over existing methods. In terms of 𝐿 2 distance and FGD metrics, our method outperforms Audio2Body [32] , S2G [13] , and MoGlow [3] by a large margin, indicating that our method can produce more accurate prediction results while better maintaining the realism and diversity of the generated gestures. The incorporation of joint-embedding space and salient gesture detection can effectively enforce the semantic consistency between audio and gesture, which is critical for generating vivid and realistic co-speech gestures. Compared with SDT [29] , SEEG [21] , and HA2G [24] , our method can generate gestures with better synchronization for both the lip motions and body pose movements, resulting in the higher BC and lower PSD. This is due to the separate audio feature extraction and synthesis As shown in Fig. 5 , we visualize the generated gestures of all methods given the speech signal and compare our method with other methods. In the left case of Fig. 5 , when the speaker says the verb phrase relieve pressure, he folds his open arms down to the lower right, making a gathering movement to express the meaning of the phrase. Only our method successfully synthesizes this strong semantic-related gesture since we emphasize the semantic consistency of salient posture in our method. The naturalness of gestures generated by Audio2Body [32] , S2G [13] and SEEG [21] is not good enough, especially the realism of hand motion generated by S2G [13] is visually poor and the face shape generated by SEEG [21] is distorted and deformed. MoGlow [15] , SDT [29] , and DiffGesture [38] generate less diversity in the poses. In contrast, the distribution of poses generated by our method is much closer to the real distribution. In addition, for the right case of Fig. 5 , the speaker lowers his raised right hand when he says the phrase go down.Audio2Body [32] and MoGlow [15] generate gestures of similar appearance with a small range of motion and unnatural hands. The template of SDT [29] restricts it from generating poses with large variation, thus failing to learn strong semantic gestures. SEEG [21] and DiffGesture [38] generate the coarse pose appearance, but the generated right hand is sagging when the one of ground truth is flat. Compared with these methods, our approach can learn the salient posture and generate more realistic results.We conduct a subjective user study to compare our method with other baselines from three aspects: naturalness, synchronization, and expressiveness. Here, naturalness refers to the smoothness and realism of the generated pose movements. Synchronization means the temporal consistency between lip motion and audio. Expressiveness measures the capability of gestures to express the semantic information of speech. We collect anonymous gesture videos of all methods for four characters in the dataset and invite 25 volunteers to watch and score them based on the three metrics described above. The rating scores range from 1 to 7, with 7 being the most plausible and 1 being the least plausible. As shown in Table 3 , our method shows significant advantages over other approaches and even achieves comparable results against ground truth. Compared with Audio2Body [32] and S2G [13] , our method performs better on the gesture naturalness and synchronization due to the separate face-body synthesis framework. In addition, the integration of salient posture detection enhances the expressiveness of our method, which outperforms MoGlow [3] , SDT [29] , SEEG [21] , and DiffGesture [38] by a large margin.We conduct extensive ablation studies to justify the contribution of key components to the final performance of the proposed method.Effectiveness of key components. Table 4 summarizes the performance and effectiveness of different design components on the speaker of Kubinec. We use the baseline model for a fair comparison with other variants of our method, which only contains an audio encoder and pose decoder and directly predicts the holistic gesture sequence using the audio representation. From the results of the first and second row in Table 4 , we can see that compared with the baseline, the integration of separate synthesis for facial expressions can facilitate the significant improvement of synchronization (See the obvious increase of BC metric). In addition, our model learns a joint manifold space to exploit the inherent semantic association between speech content and gesture, which helps to achieve the lower FGD metric and effectively enhance the realism of generated gestures. As shown in the last two rows in Table 4 , the incorporation of the salient detector module can further decrease the FGD metric with +1.32 and also achieves an obvious performance boost in the PSD metric. Additionally, we visualize the predicted saliency score of an input gesture sequence in Figure 6 .Ablation of face-body feature alignment. Here, we also investigate the contribution of the face-body feature alignment design to our overall framework. From the result comparison of Table 5 on the speakers of Oliver and Kubinec, we can see that the integration of face-body feature alignment can facilitate the obvious improvement of audio-motion synchronization (see the increase of the BC metric). Meanwhile, better synchrony further helps to achieve the lower FGD metric, which indicates that the alignment module also contributes to the more realistic generated results. Impact of different salient label strategies. We report the performance of different salient label strategies using the same baseline on Oliver and Kubinec in Table 6 . Frame-level strategy means training the salient posture detector under the supervision of frame-level salient labels. We acquire the frame-level labels according to the distances between the poses of all frames in a sequence and the resting pose. As shown in Table 6 , we observe that compared to the frame-level strategy, the proposed sequence-level approach can boost performance, especially in terms of FGD and PSD metrics. This is because sequence-level weak supervision facilitates the detector and generation network to learn the exact correspondence between salient postures and speech content, which contributes to the enhancement of realism of generated gestures. 
conclusion: 
 In this paper, we propose a novel co-speech gesture generation method to enhance the learning of cross-modal association of speech and gesture. Our model learns a joint manifold space for different representations of audio and body pose to exploit the inherent association between two modalities and enforce semantic consistency using a consistency loss. Further, we introduce a weakly-supervised salient posture detector to facilitate the model to focus more on learning the mapping of salient postures and corresponding audios with highly semantic information. Extensive experiments demonstrate that the proposed method surpasses state-of-the-arts by a large margin and can effectively enhance the naturalness and fidelity of generated gestures.  
: 
 g., cooling down, which are hard to be generated. Our method can synthesize more realistic gestures than SEEG [21] by emphasizing semantic consistency of salient postures.rule-based generation approaches, which establish the deterministic correspondence between audio syllables and gesture sets. Such methods ignore the intrinsic connection between different modalities and suffer from poor naturalness. Recent data-driven methods [21, 24] achieve better performance by utilizing deep neural networks to extract audio representation of different semantic granularities, which is then decoded to generate a holistic gesture sequence. However, these methods suffer from the following major weaknesses: (1) their pipelines are straightforward and cannot effectively achieve semantic consistency between the speech content and the synthesized gestures, especially for the postures with large movement scope. (2) They treat facial expressions and body movements as a whole and simultaneously synthesize them with a single pipeline, which typically leads to poor synchronization between facial expressions and speech. Therefore, our primary goal is to enhance the correspondence between the generated gestures and the semantics in the speech content. From a human gesture study [12] , we identify one significant observation: postures with large movement scope in the sequence correspond to strong and rich semantic information of speech audio, and postures with slight movement scope correspond to weak semantics of speech audio. As illustrated in Figure 1 , the gesture sequence of opening and raising the arms, then folding and lowering them is closely related to the phase cooling down. For clarity, we define salient postures as the postures with large movement scope, i.e., the movements in which the shoulders and arms have a relatively large amplitude of motion, which are often related to strong semantics of the speech content. Salient postures in gesture sequences tend to be significant in conveying the intention and emotion of speakers, and therefore the exact correspondence relationship between salient postures and speech with strong semantics will contribute to the vivid and realistic gesture generation.Inspired by the above discussions, we propose a novel speechdriven gesture generation method by emphasizing semantic consistency of salient posture. To fully exploit the inherent semantic association between audio and gesture, we first learn a joint manifold space for the representations of audio and body pose to establish a mapping between the two modalities. Through the consistency loss, we ensure that the audio and pose features are close to each other in the shared joint-embedding space and represent similar semantic information.Furthermore, we propose to emphasize the semantic consistency of salient postures, i.e., postures with large movement scope that often correspond to strong semantics. We design a weakly-supervised salient posture detector to identify salient postures, which utilizes a temporal relation module to mine long-range temporal dependencies among the pose features and predicts framelevel saliency score. We then use the saliency score to reweight the consistency loss to enforce a stronger alignment between the salient postures and corresponding audios in the joint embedding.In addition, facial expressions, especially lip motions, mainly rely on articulation-related acoustic features; while body gestures are closely correlated with strong semantics in the speech content. Therefore, we extract audio features dedicated to facial expressions and body gestures separately, and synthesize facial expressions and body gestures with separate branches. Meanwhile, we enforce the temporal alignment between the audio features extracted in the face branch and the body branch, which effectively improves synchronization and naturalness between the face and body parts. Our main contributions can be summarized as follows:• We propose a novel speech-driven gesture generation framework with an emphasis on semantic consistency of salient posture. We introduce the joint manifold space to learn the inherent semantic association between audio and gesture modalities and enforce semantic consistency via a consistency loss. • We emphasize the semantic consistency of salient postures by introducing a weakly-supervised detector to identify salient postures, and reweighting the consistency loss based on saliency score to enforce a stronger alignment in the joint manifold space. • Observing that facial expressions rely on articulation-related audio features while body gestures rely on semantic-related audio features, we propose to extract separate audio features for face and body, and design separate branches for face and body gesture synthesis. 
 
 Appendix:  


Decision: All the reviewers have acknowledged the contributions of this paper. The paper is technically sound and a great fit for MM. The authors did a good job addressing the points raised by the reviewers. However, as noted by Reviewer r4aP, there are still unresolved issues. The authors should elaborate on the related work and discuss their contributions with respect to the existing recent approaches adequately, as they promised to do so in their rebuttal.

Reviews: review: paper_summary: paper_summary: The existing co-speech gestures generation only considers the mapping from audio to the body gestures. In this paper, the authors emphasize the semantic consistency between the speech content and postures. To achieve this, they proposed that the salient posture is highly related to some specific phrase. Therefore, they enforce semantic consistency by emphasizing the salient postures that are detected by a weekly-supervised detector. In addition, the authors treat the body language and facial expressions independently compared to the previous work. The qualitative results show the model can generate more diverse and semantic-related co-speech gestures.
strength_weakness: strength_weakness: 1. The authors take the speech content into consideration and enforce the semantic consistency between the speech and generated gestures. Although the direct alignment between posture and speech content is challenging, they improve the generation by emphasizing the salient postures.
2. The attached video demonstrates that the independently generated facial expression is of a higher quality compared with the baselines.
limitations: limitations: 1. The example of phase "cooling down" shows the correlation between speech content and salient postures. However, it is hard to say that the slight postures are trivial.
2. Although the salient posture is emphasized during training, how do you guarantee the generated salient gesture is semantically correct? For example, a rising movement might be generated with the phrase "cool down"
paper_summary: paper_summary: This paper aims to generate upper body gestures and facial expressions. The main target is generate more "Salient Posture" - gestures with  larger motion ranges. The authors introduce a pipeline in four steps: 1) learn a audio-gesture joint embedding by minimizing the cosine similiarity between audio and motion latent space. 2) label the pose with larger L2 distance with mean pose as Salient Posture, and train a classifer to learn this label. 3) use this learned classifer to generate pseudo label during training, give the predicted Salient Posture a higher weight 4) use a face and body align module for final results. Results conduct on TED and S2G shows a large improvement.
strength_weakness: strength_weakness: 1. the idea of using a weakly-supervised method to geneate semantic aware pose is more easier for scaling up for real world application.
2. experiments compare enough methods, objective results are good.
limitations: limitations: I feel postive for this paper, however, I have a lot of unclear points for the motivation and correctness of the pipeline design. 

1. motivation of joint embedding for generation. 
previous method directly learn audio2gesture, the authors add gesture reconstruction and align the latents. the reconstrcution loss and latent cosine similiarity loss is trained jointly. why this would be better? Previous methods like TalkShow, first learns the reconstruction of gesture is in a pretrained VQVAE  / AE, then map the audio to the motions latent, this speerated training seems better. 

2. what is acturally we are doing with L2-distance based Salient Posture Classifier?
Salient Posture Classifier are trained with labels based on the L2-distance to mean pose (or it is like a mean pose),  so acturally we are giving the motion with higer L2-distance a higher loss weight, if so, why we do not directly resample the training data based on their L2-distance like DisCo? or why we do not directly calculate generate results - mean pose as a loss weight? The weakly supervised idea is not so bad, but do we need it?

3. the resolution of cosine similarity calculation,
From the paper, it seems the cosine similiarity is calculated per-frame between audio and gesture latents. Is this correct? I could understand the frame-level similiarity will focus on aligning the rhythmic features but cannot get how it align semantic. As the semantic similarity may activated in particular frames, not all frames.

Minor issue:
A lot of related works are missing in the context of co-speech gesture generation such as previous MM papers. Only one new method after 2021-DiffGesture-is discussed in this paper.
paper_summary: paper_summary: The paper presents a method to generate co-speech 3D face and upper-body gestures by explicitly focusing on salient postures and generating them accurately. The authors define salient postures as those postures consisting of large movements and are assumed to be correlated with speech semantics. They propose a consistency loss to better learn the salient postures. The consistency loss relies on weak supervision of saliency provided at the clip level – a clip has a saliency value of 1 if any of its constituent frames is determined to be salient (having large movements) and 0 otherwise. The authors also separately extract articulation-related and semantic-related features from the audio to respectively guide face and body generation. They evaluate the quantitative and qualitative performance of their proposed method through various evaluation metrics, ablation studies, visualizations, and user studies.
strength_weakness: strength_weakness: 1. The end-to-end approach, particularly the proposed temporal relation module, is clearly explained. The corresponding ablation studies show the achieved benefits.

2. The reported experiments show good performance of the proposed method. The visual results have reasonable quality in terms of plausibility and expressiveness.
limitations: limitations: 1. Missing related work: the authors have not mentioned related methods for gesture generation, such as [A, B], which generate both face and body gestures with various speaker-aware style information that may subsume the notion of posture saliency, [C] which generates emotionally expressive gestures, [D], which encapsulates the notion of posture saliency through speech rhythms, and [E, F], which achieve a wide diversity in generated gestures by using a diffusion-based approach. Given the substantial body of work that the authors have not mentioned, it is hard to fully understand their proposed benefits.

[A] Habibie, Ikhsanul, Weipeng Xu, Dushyant Mehta, Lingjie Liu, Hans-Peter Seidel, Gerard Pons-Moll, Mohamed Elgharib, and Christian Theobalt. "Learning speech-driven 3d conversational gestures from video." In Proceedings of the 21st ACM International Conference on Intelligent Virtual Agents, pp. 101-108. 2021.

[B] Habibie, Ikhsanul, Mohamed Elgharib, Kripasindhu Sarkar, Ahsan Abdullah, Simbarashe Nyatsanga, Michael Neff, and Christian Theobalt. "A motion matching-based framework for controllable gesture synthesis from speech." In ACM SIGGRAPH 2022 Conference Proceedings, pp. 1-9. 2022.

[C] Bhattacharya, Uttaran, Elizabeth Childs, Nicholas Rewkowski, and Dinesh Manocha. "Speech2affectivegestures: Synthesizing co-speech gestures with generative adversarial affective expression learning." In Proceedings of the 29th ACM International Conference on Multimedia, pp. 2027-2036. 2021.

[D] Ao, Tenglong, Qingzhe Gao, Yuke Lou, Baoquan Chen, and Libin Liu. "Rhythmic gesticulator: Rhythm-aware co-speech gesture synthesis with hierarchical neural embeddings." ACM Transactions on Graphics (TOG) 41, no. 6 (2022): 1-19.

[E] Ao, Tenglong, Zeyi Zhang, and Libin Liu. "Gesturediffuclip: Gesture diffusion model with clip latents." ACM Transactions on Graphics (TOG) 42, no. 4 (2023): 1-18.

[F] Yang, Sicheng, Zhiyong Wu, Minglei Li, Zhensong Zhang, Lei Hao, Weihong Bao, Ming Cheng, and Long Xiao. "DiffuseStyleGesture: Stylized Audio-Driven Co-Speech Gesture Generation with Diffusion Models."

2. Further, there are more recent datasets for co-speech face and body generation, such as [G], which is commonly used by recent approaches. Without any results on more recent datasets, it becomes harder to understand the proposed benefits over available methods.

[G] Ghorbani, Saeed, Ylva Ferstl, Daniel Holden, Nikolaus F. Troje, and Marc‐André Carbonneau. "ZeroEGGS: Zero‐shot Example‐based Gesture Generation from Speech." In Computer Graphics Forum, vol. 42, no. 1, pp. 206-216. 2023. https://github.com/ubisoft/ubisoft-laforge-ZeroEGGS

3. Do the authors have any evidence, such as experiments or prior studies, to support their claim that faces are more correlated with articulation-based audio features and poses are more correlated with semantic-related audio features? This is a key aspect of their network design, but it is not fully justified.

4. Have the authors considered any intermediate approach between assigning the saliency labels at the frame level and at the clip level? For example, assigning the saliency value for a contiguous group of frames where there are significant movements? The approach of assigning entire clips as salient appears too loose a constraint.

5. Why does the label to determine alignment between face and body gestures (Line 506) consist of two elements? It seems only the values $[1, 0]$ and $[0, 1]$ are used, so why not use only $0$ and $1$?