Enabling Synergistic Full-Body Control in Prompt-Based Co-Speech Motion Generation

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-19 03:07:15

Abstract: Current co-speech motion generation approaches usually focus on upper body gestures following speech contents only, while lacking supporting the elaborate control of synergistic full-body motion based on text prompts, such as {\it talking while walking}. The major challenges lie in 1) the existing speech-to-motion datasets only involve highly limited full-body motions, making a wide range of common human activities out of training distribution; 2) these datasets also lack annotated user prompts. To address these challenges, we propose SynTalker, which utilizes the off-the-shelf text-to-motion dataset as an auxiliary for supplementing the missing full-body motion and prompts. The core technical contributions are two-fold. One is the multi-stage training process which obtains an aligned embedding space of motion, speech, and prompts despite the significant distributional mismatch in motion between speech-to-motion and text-to-motion datasets. Another is the diffusion-based conditional inference process, which utilizes the separate-then-combine strategy to realize fine-grained control of local body parts. Extensive experiments are conducted to verify that our approach supports precise and flexible control of synergistic full-body motion generation based on both speeches and user prompts, which is beyond the ability of existing approaches. The code is released on (link will be published upon acceptance).

Main paper: 
 introduction: 
 Co-speech motion generation [9, 10, 12, 27, 48, 56] , which generates stylized movements of human body following speech audio inputs, is among the central tasks in creating digital talking avatars. Though growing rapidly in recent years, current co-speech motion generation approaches usually focus on upper-body gestures, such as head and hands, or only support limited full-body motions, in special restricted low-body movements. One of the fundamental challenges here is that the speech signal is too weak to uniquely determine full-body motions. For example, for generating co-speech motion of a digital host for releasing a new product, both "talking while walking" and "talking while standing still" are reasonable motions. As a result, it would be meaningful to realize precise and flexible control of full-body motion for achieving natural and synergistic effects based on additional input signals to reflect user intentions, such as text prompts.On the other hand, prompt-based co-speech motion generation is a highly nontrivial task with two major reasons. On one hand, the existing speech-to-motion datasets, such as BEATX [26] , focuses on subtle hand movements yet involve fairly limited full-body motions, especially in lower body. For example, the lower body of the speaker usually remains relatively stable during talking. This makes a wide range of common human activities out of training distribution. On the other hand, these datasets also lack annotated user prompts. Furthermore, crafting of a diverse and annotated dataset at scale is extremely costly. This has significantly constrained the potential for quality and diversity in motion generation.One possible solution to deal with the data lacking issue is to augment training with text-to-motion datasets, such as AMASS [30] , which include a relatively complete set of full-body motions with vast scale and strong diversity, as well as annotated text prompts [16, 35] . Superficially, jointly training with both speech-to-motion and text-to-motion datasets could lead to the ideal model, whose key is to build a joint embedding space of speech, text, and motion. However, due to the significant distribution mismatch in motion between the two kinds of datasets, a large number of full-body motions are missing their corresponding speech signals, making building such an embedding space still a challenging task.To deal with issue, we propose SynTalker, a prompt-based cospeech motion generation approach which utilizes off-the-shelf text-to-motion datasets to augment co-speech training, meanwhile addressing the distributional mismatch challenge. For training, we propose a multi-stage approach, which utilizes motion representation pre-training and motion-prompt alignment pre-training to address the issue of motion distribution mismatch and the problem of lacking prompt annotation for speech-to-motion data. For inference, we designed a novel separate-then-combine strategy under for both input conditions and body parts, such that the separate operations map the input signal to their most proper body part to control, meanwhile the combine operations leads to the synergy among body parts. Extensive experiments show that, our approach is able to achieve significant performance in using both speech and text prompt to guide the generation of synergistic full-body motion precisely and flexibly, which is beyond the capability of the existing co-speech generation approaches.In summary, by proposing SynTalker, our main contributions are: 1) We propose the first approach to enable synergistic full-body control with general text prompts for co-speech motion generation, under the situation of lacking fully annotated datasets of speech, text, and motion; 2) We propose a novel multi-stage training approach to address the motion distributional mismatch and prompt annotation lacking challenges; 3) We propose a novel separatethen-combine approach for model inference to achieve both precise control and synergistic motion generation.The rest of the paper is organized as follows. In Section 2, we discuss the related work from two closest research areas, i.e. co-speech motion generation and text-to-motion generation. In Section 3-5, we introduce the model design, training process, and inference process of our approach in detail. In Section 6, we report experimental results. In section 7-8, we discuss limitations and future work as well concluding the paper. 
related work 2.1 co-speech motion generation: 
 Early rule-based approaches to co-speech motion generation [5, 6, 21] utilize linguistic rules to translate speech into sequences of predefined gesture segments. This process, being time-consuming and labor-intensive, requires significant manual effort in defining rules and segmenting motions. Previous generative methods often produce overly smooth motions [3, 12, 27, 40] , attributable to the use of traditional deterministic generative models, which are inadequate for many-to-many mapping problems. Despite some attempts to introduce control signals and prior information into model design [1, 22, 27, 49] , the capabilities of these models remain limited. Recent advancements have leveraged modern generative models like Diffusion [11] to tackle these challenges. For instance, DiffGesture [56] employs a diffusion model to capture the relationship between speech and gesture. Nonetheless, the weak semantic signals in audio often result in motions that are misaligned with the semantic content of the input audio. DiffuseStyleGesture [48] advances this by integrating emotional control into the gesture generation process, while Amuse [9] and EMOTE [10] explicitly extract and disentangle emotions from given conditions to provide stronger control signals. UnifiedGesture [47] additionally use reinforce learning to strength gesture. GestureDiffuCLIP [2] incorporates existing contrastive learning frameworks [41] to enable prompt-based gesture style control, offering finer-grained style controllability for end-users. However, these methods still struggle to meet diverse real-world user requirements, such as accommodating gestures while walking, due to the limited motion distribution in co-speech datasets.In parallel to co-speech motion generation problems, text-based motion generation aims to generate general motions from textual prompts. Pioneering works [7, 29, 42, 54, 55] such as Motion Diffuse and T2M-GPT utilize a diffusion-based architecture or GPT-based architechture to model the many-to-many challenges in text-tomotion generation. Subsequent studies, such as PriorMDM, TLControl, and OmniControl [38, 45, 46] , further employ trajectory and end-effector tracking to provide finer-grained control. GMD [20] introduces additional scene information during the generation of human actions, and MotionClip [41] attempts to align motions with the CLIP space [36] , enabling the capability to generate motion from images. TM2D [14] and FreeTalker [50] have explored this by learning both speech-to-motion and text-to-motion tasks simultaneously. Even though this enables a single model to switch between two tasks, it does not provide synergistic generation conditioned on both signals. 
model design: 
 In this section, we introduce our prompt-based co-speech motion generation model. We first provide an overview of the model design and the corresponding generation process. Afterwards, we provide detailed descriptions of two core modules for motion representation and conditional generation. Our model takes speech audio and the corresponding transcripts as inputs, targeting at outputting realistic and stylized full-body motions that align with the speech content rhythmically and semantically. Compared with traditional co-speech generation model, besides speech, it further allows to use a short piece of text, namely a text prompt to provide additional descriptions for the desired motion style. The full-body motions are then generated to follow the style given by both speech and prompt as much as possible.The overall model structure is illustrated in Figure 2 , which consists of three major components. The first is the motion representation module, which consists of motion encoder and decoders. We include three separate encoder and decoders to represent local body parts. The second is the conditional generation module for aligning latent motion representations with conditional inputs of speeches and prompts. The module is based on the latent diffusion model [37] , which apply diffusion and denoising steps in the latent space. The third is the conditional representation module, which consists of the speech content encoder and contrastive text encoder to obtain scalar-valued prompt and speech conditions in the diffusionbased conditional generation model. Below we dive into detailed structures of the first two modules.Motion encoding. Recent studies in motion generation have demonstrated that vector-quantized autoencoder (VQ-VAE) [43] possesses a remarkable capability for compressing motion information [2, 15, 54] . We also utilize vector quantization for motion encoding. Following [15, 31, 53] , we use a residual VQ-VAE (RVQ-VAE) as the quantization layer. To further decrease the coupling between body parts, we segment the body into three parts: upper body, fingers, and lower body, like in [2, 26] , and train a separate RVQ-VAE for each part. In details, the motion sequence M can be represented as m 1:𝑁 ∈ R 𝑁 ×𝐷 , which is firstly encoded into a latent vector sequence z 1:𝑛 ∈ R 𝑛×𝑑 with downsampling ratio of 𝑛/𝑁 and latent dimension 𝑑, using 1D convolutional encoder E; The z 1:𝑛 ∈ R 𝑛×𝑑 obtained through the encoder then enters the first quantization layer Q 1 , each vector subsequently finds its nearest code entry in the layer's codebook ẑ𝑣 . Motion decoding. Similar to motion encoding, three separated decoders are introduced for generating corresponding motions for all body parts, which are 1D convolutional decoders. During training, the motion data is encoded with motion encoders and fused with speech and prompt conditions by the diffusion-based conditional generation module, and then passed through the decoders to get the reconstructed motions. During inference, the motion encoder is not utilized, the generated motion is obtained directly from the speech and prompt conditions with the diffusion module and motion decoders.C 1 = {c 1 k } 𝐾 𝑘=1 ⊂ R 𝑑 toThe conditional generation module is based on the latent diffusion model [37] , which is a variant of diffusion models that applies the forward and reverse diffusion processes in a pre-trained latent feature space. The diffusion process is modeled as a Markov noising process. Starting from a latent gesture sequence 𝑍 0 drawn from the gesture dataset, the diffusion process progressively adds Gaussian noise to the real data until its distribution approximates N (0, 𝐼 ). The distribution of the latent sequences thus evolves asEQUATIONwhere 𝑍 𝑛 is the latent sequence sampled at diffusion step 𝑛, 𝑛 ∈ {1, . . . , 𝑁 }, and 𝛼 𝑛 is determined by the variance schedules. In contrast, the reverse diffusion process, or the denoising process, estimates the added noise in a noisy latent sequence. Starting from a sequence of random latent codes 𝑍 𝑁 ∼ N (0, 𝐼 ), the denoising process progressively removes the noise and recovers the original latent code 𝑍 0 . To achieve conditional motion generation, we train a network 𝐸 𝜃 (𝑍 𝑛 , 𝑛, 𝐴, 𝑃), the denoising network, to recover the noise-free codes based on the noisy latent motion codes 𝑍 𝑛 , the diffusion step 𝑛, the audio 𝐴, and the prompt feature 𝑃 from the joint align space. Finally, the recovered code is input into the motion decoders for motion generation. 
model training: 
 The overall training pipeline is shown in Figure We initially train a contrastive learning space between text and motion, alongside a motion auto-encoder that uses motion from both speech-to-motion and prompt-to-motion dataset for an expressive latent space. Subsequently, our co-speech latent diffusion model is trained under the guidance of an implicit label extracted from motion using the contrastive space, effectively bypassing the lack of textual motion annotations in co-speech data. During inference, we implement a separate-then-combine strategy in every diffusion step, enabling finer control over individual body parts while preserving their synergistic interaction.pre-training stage involves two tasks. The first task, motion representation pre-training, targets at training the motion encoders and decoders for all body parts based on motion data from both the speech-to-motion and prompt-to-motion datasets in order to address the issue of motion distribution mismatch. The second task, prompt-motion alignment pre-training, targets at obtaining prompt-motion aligned embedding space [34, 41] based on the prompt-to-motion dataset and address the issue of lacking prompt annotations for speech-to-motion data. In the generation model training stage, both the speech-to-motion and prompt-to-motion data are utilized jointly with the motion encoders and decoders as well as the prompt-motion alignment space to obtain the final diffusion-based generation model [42] . Below we discuss the details of the training process.The target of motion representation pre-training is to obtain the motion encoders and decoders based on motion data only, which are from the speech-to-motion and prompt-to-motion datasets. By this way, the obtained primitive motion representation space are independent of any conditional speech and prompt signals. From extensive experiments, we find that this approach effectively alleviates the motion distribution mismatch issue between speech-to-motion and prompt-to-motion datasets, which can not be addressed when directly mixing the two datasets for conditional generation model training without such a pre-training process.Concretely, utilizing all motion data from the two kinds of datasets, the motion encoders and decoders are trained via a motion reconstruction loss combined with a latent embedding loss at each quantization layer of the RVQ-VAE structures:EQUATIONwhere sg[•] denotes the stop-gradient operation, and 𝛽 a weighting factor for embedding constraint. This framework is optimized with straight-though gradient estimator [44] , and our codebooks are updated via exponential moving average and codebook reset following T2M-GPT [54] . After training, the motion encoder and decoders are frozen in the rest of the training process.The target of prompt-motion alignment pre-training is to obtain the prompt-motion alignment embedding space, which consists of the contrastive text encoder in Figure 2 and an additional contrastive motion encoder. These two encoders play the essential role to address the issue of missing prompt annotations for speech-to-motion data by employing the implicit label strategy. During downstream training, assume that the prompt annotation is needed for some speech-to-motion instance, which is lacking in the original dataset. We can directly input the motion into the contrastive motion encoder and get its corresponding embedding in the prompt-motion aligned space. It is easy to see that this motion embedding is an ideal substitution of the missing prompt embedding if the aligned space is well-trained. Concretely, motivated by [34] , we formulate this pre-training task as a contrastive learning problem. Besides the contrastive text and motion encoders, we employ an additional motion decoder, which is different from the motion encoder in our final inference model. On the premise that the latent space is a probabilistic space, this setup aims to bring the feature vectors of corresponding text and motion pairs as close as possible. The decoder then decodes these latent feature vectors into motion to calculate the reconstruction loss with real motions. The loss gradients are back-propagated to update the prompt and motion encoders. This technique has been proven to be highly effective in numerous studies [33, 34, 41] .Loss function design. We introduce the same set of sub-loss terms to [34] , and the total loss can be defined as the weighted sum formulationL CON = L R + 𝜆 KL L KL + 𝜆 E L E + 𝜆 NCE L NCE .For sub-losses, the reconstruction loss L R measures the motion reconstruction given prompt or motion input (via a smooth L1 loss). The Kullback-Leibler (KL) divergence loss L KL is to regularize the distances between motion and prompt embedding distributions as well as making them closer to the standard normal distribution. The cross-modal embedding similarity loss L E enforces both prompt 𝑧 𝑇 and motion 𝑧 𝑀 latent codes to be similar to each other (with a smooth L1 loss). A contrastive loss term L NCE additionally uses negatives prompt-motion pairs to ensure a better structure of the latent space. More detailed introductions of these loss terms are included in the appendix.After training, the contrastive text and motion encoders are frozen and utilized in the downstream generation model training.After the pre-training stage, we obtain two sets of outcomes: the motion encoders and decoders, as well as the contrastive text and motion encoders. Based on the motion encoders obtained from motion representation pre-training, we can map all motions in the global motion distribution to the same compact latent space. Utilizing the contrastive text and motion encoders from the prompt-motion alignment pre-training, for motions without prompt annotations in the speech-to-motion dataset, we can provide them with an implicit label using the contrastive motion encoder. What is essential here is that 1) the motion distribution mismatch problem is addressed for motion representations; 2) all co-speech training data have their corresponding (implicit) prompt annotations.The training of the generation model mostly follows the standard training process of denoising diffusion models [11, 37] . We train the denoising network 𝐸 𝜃 by drawing random tuples (𝑍 0 , 𝑛, 𝐴, 𝑃) from the training dataset, corrupting 𝑍 0 into 𝑍 𝑛 by adding random Gaussian noises 𝐸 to obtain 𝑍 𝑛 , applying denoising steps to 𝑍 𝑛 using 𝐸 𝜃 , and optimizing the lossEQUATIONSpecifically, the latent motion representation 𝑍 0 is encoded by the motion encoder with the RVQ-VAE structure, and the prompt embedding is obtained from the implicit labels generated by the contrastive motion encoder. Since the speech audio and speech text transcript always occur simultaneously during speech, we uniformly denote them as 𝐴 here. 𝐴 is processed through a temporal convolutional network for feature extraction and to align with the latent motion sequences in the time series. We utilize the classifier-free guidance [19] to train our model. To strengthen the understanding of the two conditional signals, speech 𝐴 and prompt 𝑃, we make the diffusion model to learn under both conditioned and unconditioned distributions during training by randomly setting conditional variables A and P = 0 for 𝜂 𝑎 and 𝜂 𝑝 . This makes the diffusion model better understand the impact of various conditional signals on the generation results. More details of the training techniques are introduced in the appendix. 
model inference: 
 Through the multi-stage training process, the generation model is obtained. However, utilizing this for conditional generation is not a straightforward task. Even though an aligned space of motion, speech, and prompt is obtained, precise control and generation still requires carefully aligning generation conditions to local body parts. To achieve this target, we introduce the separate-then-combine generation strategy for manipulating latent codes of the diffusion model for both input conditions and body parts.General diffusion-based generation process. During inference, the diffusion network leverages the sampling algorithm of DDPM [18] to synthesize motions. It first draws a sequence of random latent codes 𝑍 * 𝑁 ∼ N (0, 𝐼 ) then computes a series of denoised sequences {𝑍 * 𝑛 }, 𝑛 = 𝑁 -1, . . . , 0 by iteratively removing the estimated noise 𝐸 * 𝑛 from 𝑍 * 𝑛 . The entire process is carried out in an autoregressive manner.Sampling from 𝑝 (𝑍 0 |𝑛, 𝐴, 𝑃) is done in an iterative manner, according to [18] . In every time step 𝑛 we predict the clean sample Ẑ0 = 𝐺 (𝑍 𝑡 , 𝑛, 𝐴, 𝑃) and noise it back to 𝑍 𝑡 -1 . This is repeated from 𝑡 = 𝑁 until 𝑍 0 is achieved. Separate-then-combine for conditions. Motivated by Motion-Diffuse [55] and PIDM [4] , we extend our system to allow separated guidance to apply the effect of the conditional signal audio and prompt. To achieve this, from the dimension of conditions, we separate latent codes into the following formulation:EQUATIONwhere 𝑍 uncond = 𝑍 𝜃 (𝑍 𝑛 , 𝑛, 0, 0) is the unconditioned prediction of the model, such that both the speech and prompt conditions are set as the all-zero tensor 0. The audio-guided prediction and the prompt-guided prediction are respectively represented by 𝑍 speech = 𝑍 𝜃 (Z t , t, A, 0) -Z uncond and 𝑍 prompt = 𝑍 𝜃 (Z t , t, 0, P) -Z uncond . 𝑤 𝑎 and 𝑤 𝑝 are guidance scale corresponding to speech and prompt. Separate-then-combine for body parts. Furthermore, we extend our system to allow fine-grained style control on individual body parts.We utilize the diffusion model to generate codes for each body part based on masking. The full-body motion codes To achieve better motion quality, we add a smoothness item to the denoising direction as suggested by [55] ,Z O ∈ R O× (L×C)EQUATIONwhere ∇ denotes the gradient operator. 𝑤 𝑏𝑜𝑑 𝑦 is set to 0.01.Afterwards, the following generation procedure follows the normal diffusion generation process as discussed above. By utilizing the two separate-then-combine strategies, the control of the motion generation process can be more flexible. The separation steps effectively build more precise mapping between input conditions and body parts. On the other hand, the combine steps ensure to generate synergistic full-body motions, avoiding unnatural motions to appear in generation. 
experiments: 
 In this section, we report experimental results to verify the following questions:• Full-body synergistic generation. Is SynTalker able to generate full-body synergistic motion desirably aligned with both speech and prompt inputs, which is the first co-speech motion generation approach to achieve this functionality? • Sinlge-source conditional generation. Is SynTalker able to achieve comparable or even better performance over stateof-the-art approaches under single-source conditional generation, verifying that besides motions, SynTalker truly learns desirable representations for both speeches and prompts? • Ablation study. Is the multi-stage training strategy and the separate-then-combine inference strategy indeed essential to achieve desirable performance?Below we explore all three questions in details. More experimental results can also be found in the appendix.Dataset. Our model smartly avoids the need for annotated cospeech motion data by leveraging existing speech-to-motion and prompt-to-motion datasets. For the speech-to-motion dataset, we utilize BEATX-Standard [26] , which includes 30 hours of co-speech motion, paired with audio and transcripts. For the prompt-to-motion dataset, we employ HumanML3D [16] , which annotates motion in the AMASS dataset [30] , consisting of 14,616 annotated motion sequences and 44,970 annotations. All motions are in SMPLX format [32] and consist of 30 frames per second. Implementation Details. We utilize the RVQVAE [53] as our auto-encoder architecture, featuring resblocks in both the encoder and decoder with a downscale factor of 4. Residual quantization employs 6 quantization layers, each with a code dimension of 512 and a codebook size of 512, with a quantization dropout ratio set at 0.2. During contrastive pre-training, we establish a space with a dimension size of 256 and a batch size of 32. We also set the temperature 𝜏 to 0.1, contrastive loss weight to 0.1, and negative-filtering threshold to 0.8. Our diffusion model incorporates 8 transformer layers and is trained with a batch size of 200 and a latent dimension of 512. The number of diffusion steps is 1000. All components can be trained on a single 4090 GPU within three days.In the first experiment, we aim to verify whether our approach effectively supports the generation of synergistic full-body motions conditioned on both speech and flexibly-chosen prompts. As no previous research has addressed this specific task, we focus on evaluating the generation results of our approach using carefully designed input speeches and prompts. It's important to note that we also provide experimental analysis on GestureDiffuseCLIP [2] and FreeTalker [50] in the appendix. These works address related but distinct tasks. GestureDiffuseCLIP also supports prompt-based cospeech motion control. However, this control is limited to motions inside of the speech-to-motion dataset. In comparison, our approach supports general out-of-distribution motions. FreeTalker, on the other hand, trained a model capable of switching between speech-tomotion and text-to-motion generation tasks, but it does not produce synergistic results under both speech and prompt conditions.Table 1 : Comparison with the state-of-the-art methods on HumanML3D [16] test set. We compute standard metrics following [16] .For each metric, we repeat the evaluation 20 times and report the average with 95% confidence interval. For MDM and MLD, we report the results using ground-truth motion length.↑ FID ↓ MM-Dist ↓ Diversity ↑ Top-1 Top-2 Top-3Real motion 0.490 ±.003 0.682 ±.003 0.783 ±.003 0.001 ±.001 3.378 ±.007 10.471 ±.083 MDM [42] 0.363 ±.007 0.553 ±.008 0.662 ±.007 1.390 ±.088 4.599 ±.037 10.704 ±.066 T2M-GPT [54] 0.433 ±.003 0.615 ±.002 0.716 ±.003 0.564 ±.012 3.867 ±.008 10.558 ±.083 MLD [8] 0.429 ±.003 0.613 ±.003 0.717 ±.002 0.963 ±.029 3.898 ±.012 10.401 ±.096 MoMask [15] 0.461 ±.002 0.657 ±.003 0.760 ±.002 0.222 ±.007 3.620 ±.011 10.621 ±.096SynTalker (w/o prompt-to-motion alignment) 0.429 ±.003 0.622 ±.004 0.732 ±.004 0.509 As shown in Figure 4 , we conduct qualitative experiments to evaluate the synergistic generation results of our model. To better demonstrate that the generation results synergistically integrate both speech and text prompt guidance, we present outcomes under two distinct speech audios: one excited and the other calm. We evaluate our method using four different text prompts: sitting, waving while walking, standing on the right foot, and walking straight forward. The results show that our model produces talking motions that closely align with the input speech audio while accurately adhering to the text prompt requirements for body gestures. With the excited audio, the motions exhibit more pronounced changes compared to the calm speech. These include increased arm movements, higher arm raises, more pronounced left-right body turns with larger arm movements, and a tendency for hands to reach outward while talking. For additional results, please refer to the appendix.In the second experiment, we focus on verifying whether our approach indeed learns a desirable joint embedding space, in special for the input conditions. To achieve this purpose, we introduce two single-source conditional generation benchmarks, i.e. speech-tomotion generation without prompts and prompt-to-motion generation without speeches. By quantitative comparison with state-ofthe-art approaches under these two distinguished domains, we are able to verify whether our approach successfully distills information from both speech and prompts, meanwhile avoiding interference among them, which would be revealed by performance degeneration in single-condition generation. Note that for our approach, single-source generation is realized by the similar method utilized in generating Z speech and Z prompt in Equation 4. The implementation details of all contenders are included in the appendix. Speech-to-motion. We compare our approach with state-of-theart speech-to-motion generation approaches, whose results are cited from [26] . As shown in Table 2 , our method significantly outperforms baselines in terms of FGD [52] , BC [24] , and diversity [23] . This result provides convincing proof that our approach generates significantly desirable speech representation to support strong speech-to-motion generation. To further verify how speech representations are affected by the multi-stage training process, we also conducted ablation studies under this task. The details are discussed in Section 6.5. Prompt-to-motion. In this task, we compare our method with four state-of-the-art text-to-motion generation approaches. The results are reported in Table 1 . It can be observed that our approach could achieve comparable performance to the existing baselines, showing its effectiveness in understanding text prompts. Similar to speechto-motion, we conduct ablation studies for further justification, whose results are reported in the next subsection.In this section, we demonstrate qualitative examples of ablation study on model components and assess their contribution to the synergistic generation capability. For clarity, we demonstrate the results using a single prompt. Please refer to the appendix for additional results. As shown in the Figure 5 , given the same speech and the text prompt "a person is talking while sitting", compared to the sitting and talking motion in Figure 5 (a), removing our proposed components result in non-ideal generation results. Without implicit labeling, the model defaults to merely reacting to the textual prompt, producing only a static sitting motion. This confirms that without our proposed implicit labeling method, the diffusion model does not automatically learn to synthesize and integrate input to produce synergistic motions conditioned on both signals. Separate-then-combine strategy. Figure 5 (c) illustrates the effect of omitting the separate-then-combine strategy. Although the model learns to respond to both audio and prompt signals when trained with the implicit labeling method, the textual prompt inherently imposes different requirements on various body parts. The absence of the separate-then-combine strategy eliminates part-level guidance, leading the diffusion model to incorrectly merge multiple features. In this scenario, the model misinterprets the instruction to sit as merely lowering the arms and slightly bending the legs, rather than sitting. Motion representation pre-training. Figure 5 (d) shows the results when the joint training stage is omitted from the training process. The character shows an inclination to sit, but such motion is not represented within the limited distribution of the speech-tomotion datasets, rendering accurate generation unfeasible. Single-condition experiment ablation. Though our work focuses on the synergistic co-speech motion generation that follows audio and text motion prompts at the same time, we also evaluate the impact of our proposed methods on the single-source condition generation ability, whose results are shown in both Table 1 and 2. We first remove all our propose components for synergistic cospeech motion generation, which results in a pure co-speech motion generation model conditioned on audio signal. This base model achieves the state-of-the-art performance in pure co-speech motion generation task, which serves as a solid base for our synergistic generation. Note that our base model only performs co-speech motion generation and it is unable to operate on text-to-motion task, resulting in one less ablation result in Table 1 .We then evaluate the result with prompt-motion alignment while motion representation pre-training is removed. This removal has a positive impact on the text-to-motion generation, as prompt-motion alignment enables the model to accept both the audio and text prompt as input, which drags the model's output distribution to cospeech generation instead of pure prompt-based motion generation.Finally, we evaluate the single-condition performance when prompt-to-motion alignment is removed while motion representation pre-training is kept. This results in significant performance drop in text-to-motion generation due to the lack of solution space in text-to-motion generation. In co-speech motion generation, this results in a minor performance drop due to the solution space expanding beyond the original co-speech motion distribution, which is desirable in generating synergistic co-speech full body motion. 
limitations and future work: 
 Our method necessitates processing noise and performing parallel inferences at every diffusion step due to the separate-then-combine strategy. On a single RTX 4090, it achieves a generation rate of 10 frames per second without speed-up strategies and 200 frames per second with scheduling methods like DDIM [39] . While this is sufficient for real-time streaming generation applications, our method incurs a higher inference cost than a regular latent diffusion model with the same number of parameters, leading to increased computational demands during deployment.Although our model implements semantically correct body-level control through the separate-then-combine strategy, it primarily treats conditional inputs as signals of varying strength rather than fully comprehending them. This limitation highlights the inherent challenge in the field of motion generation: the difficulty of generative models in accurately interpreting multi-modal conditions. This difficulty underscores the need for future research to focus on the deeper comprehension of user prompts, which remains a formidable and crucial challenge. Additionally, hand gestures play a significant role in the realism and expressiveness of generated gestures. Currently, the lack of an annotated hand gesture dataset limits the ability to control gestures via textual prompts, which can see significant improvements through future related work. 
conclusion: 
 In this paper, we propose SynTalker, targeting at addressing the lack of elaborate control issue of current co-speech motion generation approaches. Our main contributions are: 1) By introducing a multistage training process, we effectively utilize off-the-shelf text-tomotion datasets to enable the diffusion model to simultaneously understand both co-speech audio signals and textual requirements. This approach allows for the generation of synergistic full-body co-speech motions; 2) A separate-then-combine strategy during the inference stage, enabling fine-grained control over different local body parts. Extensive experiments demonstrate the effectiveness of our method and show that it can achieve precise control over the generated synergistic full-body motions, surpassing the capabilities of existing methods.  
 
 Appendix:  


Decision: Overall, this paper presents a significant contribution with solid evaluations. Highlights include:

    + Unlike existing literature, it focuses on full-body gesture generation from speech-prompted information, even with limited data.
    + To achieve this, the authors propose a multi-stage training process and a unique separate-then-combine strategy during inference.
    + Extensive experiments demonstrate the effectiveness of the proposed approach.
    + The paper is an excellent fit for presentation at ACM MM, combining human motion, speech, and text.

However, as noted by the authors, there are issues in the evaluation part, which lead me to recommend a poster presentation. These issues can be summarized as follows:

    - More visual results are needed to fully understand the performance of the approach. Specifically, the authors should consider conducting a user study to present qualitative results and better demonstrate the effectiveness of the approach.
    - More comparative results are necessary. The authors should consider implementing baseline approaches to better highlight the effectiveness of their method. For instance, there is a line of work focusing on simultaneous action generation, e.g., [1]. Although such methods have not been applied to nonverbal gesture generation, to the best of my knowledge, the authors should discuss these works to provide stronger comparisons.

[1] SINC: Spatial Composition of 3D Human Motions for Simultaneous Action Generation

Reviews: review: paper_summary: paper_summary: This paper proposes the first approach to enable synergistic full-body control with general text prompts for co-speech motion generation,
under the situation of lacking fully annotated datasets of speech, text, and motion. It desgin the a novel multi-stage training approach to address the motion distributional mismatch and prompt annotation lacking challenges. In inference phase, it also proposes a novel separatethen-combine approach to achieve both precise control and synergistic motion generation.
strength_weakness: strength_weakness: 1. Significant paper quality and sufficient supplementary material.
2. Good writing and easy to understand. 
3. Simple motivation for utilizing off-the-shelf text-to-motion datasets to augment co-speech training for obtaining prompt-based co-speech motion generation.
limitations: limitations: My rating are borderline and will be adjusted after authors response. 

Although this work utilize the prompt for controling whole body more precisely for the first time, it appears to be intended for use, lacking deep insights.

1. Most problems occurs in introduction. As you mention "One of the fundamental challenges here is that the speech signal is too weak to uniquely determine full-body motions.", What's the meaning of "weak speech signal"? Although the following example can reflect the main motivation about the whole-body controls, it seems cherry-picking cases. I wonder whether give more instruction in speech  like "talking while walking" instead of only "talking" can generate the motion that is more in line with the scene. Otherwise, it easily causes the different motion like you have said in row 112-113.  I want to see some examples.  After that, this paper raises the main problem that current co-speech motion generation approaches usually focus on upper-body gestures but ignore low-body movements due to the flawback of exist datasets like BEATX. Can you point out how serious the problem is by using some statistics skills? It also help me to understand the impact of the mismatch problems.

2. The authors emphasize the proposed method can supports general out-of-distribution and synergistic motion. Authors should give more failure cases about the previous model like FreeTalker by utlizing "out-of-distribution" motions, which is missing in the supplementary videos not only just two cases in appendix. Total qualitative results only support the performance of the model, but not support the comparison. If possible, I recommend authors should give the project websites for detailed illustration.

3. More analysis of the experiments in Table 1/2. For example, for Top-1 in the Table 1, why the SynTalker is lower than others methods except for the MDM ? Similar results also has shown in other terms like FID. It seems the proposed model performance is worse compared to other SOTAs. The author should focus on the whole-body control of text prompt especially in "out-of-distribution" motions.
paper_summary: paper_summary: The paper proposes SynTalker, an approach that uses both speech and text prompts for generating co-speech motion with integrated full-body control. Distinguishing itself from previous works, this study focuses on full-body control by utilizing a multi-stage training process and a unique separate-then-combine strategy during inference. However, since no additional pair-wise text-to-body motion data is provided, the authors rely on existing datasets, which leads to less satisfactory results. Furthermore, the conditional generative model proposed in this paper lacks novelty, as it primarily involves a straightforward integration of existing motion generation techniques.
strength_weakness: strength_weakness: 1. Diverging from previous co-speech motion generation approaches focus mainly on upper body gestures, the proposed method focuses on full-body control by utilizing a multi-stage training process and a unique separate-then-combine strategy during inference.
2. Extensive experiments demonstrate the superiority of SynTalker over existing methods, showing its capability to generate nuanced and contextually appropriate motions based on both speech and text inputs
3. The figure in manuscript is helpful for paper reading and understanding.
limitations: limitations: 1. The paper title uses the phrase "Synergistic Full-Body Control," but it’s unclear how synergy is achieved between each modality or body part. From my perspective, it appears to be a simple multi-control generative model.
2. No additional pair-wise text-to-body motion data is provided, and the authors rely on existing datasets, which results in less satisfactory outcomes.
3. From the videos in the supplementary material, the visualization results are poor. It might be beneficial to try using a state-of-the-art audio motion generation model combined with a lower body generation model conditioned on the upper body. Such a comparison would be necessary.
4. I noticed that while this model also generates hand poses, the videos do not display any noticeable diversity in the hand gestures.
paper_summary: paper_summary: The paper proposes a prompt-based co-speech full body motion generation pipeline named SynTalker. The paper introduces a multi-stage training process to utilize off-the-shelf text-to-motion datasets and a novel separate-then-combine strategy for inference. Experimental results demonstrate that SynTalker achieves precise and flexible control over full-body motion, surpassing previous methods. Additionally, the video in the supplementary material shows that the generated motions are coherent and well-aligned with the rhythm of the speech.
strength_weakness: strength_weakness: 1.  SynTalker is the first approach to enable synergistic full-body control with general text prompts for co-speech motion generation. It addresses the challenge of lacking fully annotated datasets of speech, text, and motion.
2.  The proposed multi-stage training approach effectively handles the motion distributional mismatch and prompts annotation lacking challenges.
3.  The proposed inference strategy allows for precise control over different body parts while achieving synergy among them.
4.  Extensive experiments demonstrate the effectiveness of SynTalker in generating synergistic full-body motions and achieving desirable performance in both speech-to-motion and prompt-to-motion generation tasks.
limitations: limitations: The inference time is not suitable for realtime motion generation.