MDT-A2G: Exploring Masked Diffusion Transformers for Co-Speech Gesture Generation

Authors: xiaofeng  mao
zhengkai  jiang
mingmin  chi
qilin  wang
chencan  fu
jiangning  zhang
ji- afu  wu
yabiao  wang
chengjie  wang
wei  li
  mdt-
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-01 08:27:00

Abstract: Recent advancements in the field of Diffusion Transformers have substantially improved the generation of high-quality 2D images, 3D videos, and 3D shapes. However, the effectiveness of the Transformer architecture in the domain of co-speech gesture generation remains relatively unexplored, as prior methodologies have predominantly employed the Convolutional Neural Network (CNNs) or simple a few transformer layers. In an attempt to bridge this research gap, we introduce a novel Masked Diffusion Transformer for co-speech gesture generation, referred to as MDT-A2G, which directly implements the denoising process on gesture sequences. To enhance the contextual reasoning capability of temporally aligned speech-driven gestures, we incorporate a novel Masked Diffusion Transformer. This model employs a mask modeling scheme specifically designed to strengthen temporal relation learning among sequence gestures, thereby expediting the learning process and leading to coherent and realistic motions. Apart from audio, Our MDT-A2G model also integrates multi-modal information, encompassing text, emotion, and identity. Furthermore, we propose an efficient inference strategy that diminishes the denoising computation by leveraging previously calculated results, thereby achieving a speedup with negligible performance degradation. Experimental results demonstrate that MDT-A2G excels in gesture generation, boasting a learning speed that is over 6$\times$ faster than traditional diffusion transformers and an inference speed that is 5.7$\times$ than the standard diffusion model.

Main paper: 
 introduction: 
 The objective of co-speech gesture generation is to synthesize human gestures from audio and other modalities. Its significance is escalating in the development of virtual avatars and interactive technologies. In everyday life, there is a substantial demand for the efficient production of high-quality and diverse human gesture animations. Consequently, the production of high-quality and diverse gestures at an affordable cost plays a pivotal role in practical applications.With the rapid development of generative models such as Variational Autoencoders (VAEs), Generative Adversarial Networks (GANs), and the more recent Diffusion Models (DMs), numerous works [2, 3, 6, 14, 16, 21-24, 31, 34-38, 40, 41] have proposed utilizing these powerful foundational models for the task of co-speech gesture generation. Although these works have achieved good generative quality, we find the following limitations in practical applications: 1) Poor diversity. For VAE-based methods [15, 19, 38] , the goal is to learn a meaningful latent space where similar inputs are mapped close together. However, if the model relies too much on the decoder's capacity to reconstruct the data from the latent variables, it might ignore the latent variables, leading to a phenomenon known as posterior collapse. This results in a lack of diversity in the generated samples. For GAN-based methods [11, 23] , training instability and mode collapse issues are common, leading to the generator producing a limited variety of human gestures, or even identical ones.2) Inefficiency. DM-based methods [2, 3, 6, 16, 31, 34, 36, 37, 40] require multiple iterations during the generation process. Each iteration involves a forward pass through the model, resulting in typically long inference time. Besides, current DMs applied in A2G domain often struggle to effectively learn the semantic relationships between features of different modalities, leading to slow training convergence.To address above challenges, we propose MDT-A2G, a Masked Diffusion Transformer framework designed for Audio-to-Gesture task. Sepecifically, we leverage the contextual reasoning capability of mask modeling and combine it with DMs to ensure the quality and diversity of co-speech gesture synthesis. During training, the masked modeling denoising network takes multi-modal conditions and partially masked noisy features as input and outputs original gestures. As shown in Figure 1 , benefited by the masked modeling's strong semantic learning ability, our MDT-A2G can generate human gestures that are both realistic and diverse. The convergence of the learning process is also accelerated. To the best of our knowledge, this is the first attempt of mask modeling DMs in A2G task. We also propose the incorporation of a simple-yet-effective multi-modal feature fusion module into existing architecture. This module is carefully designed for fusing various modalities, simplifying the complex feature processing and fusion mechanism. To improve the efficiency of inference, we introduce an efficient inference strategy during sampling without compromising the quality and diversity of the generated gestures.Extensive qualitative and quantitative experiments demonstrate that our proposed MDT-A2G can produce high-quality, diverse and temporally-coherent human gestures in a high efficiency. We achieve state-of-the-art performance regardless of upper and full body, outperforming existing co-speech generation methods. The main contributions can be summarized as follows:• We propose MDT-A2G, a masked diffusion transformer framework designed for audio-to-gesture task, enhancing semantic understanding of diverse modalities and hastening training convergence that is 6 × faster. The field of co-speech gesture generation, which concentrates on the production of gestures that synchronize with speech audio, has witnessed the advent of numerous learning-based methodologies. Among these, the hierarchical approach proposed by Liu et al. [24] is significant, as it considers both speech semantics and the structure of human gestures. Yoon et al. [39] have also made substantial contributions by utilizing a recurrent neural network and multi-modal context to devise a translation-based approach. Liu et al. further enriched the field by introducing the BEAT dataset and the Cascaded Motion Network (CaMN) [23] . Other notable contributions include EMAGE [22] , a method that integrates Masked Gesture Reconstruction and Audio-Conditioned Gesture Generation, and TalkSHOW [38] by Yi et al., which employs an autoencoder and a VQ-VAE for the generation of face and body motions. In recent years, diffusion models, celebrated for their capacity to model intricate data distributions and perform many-to-many mappings, have been increasingly harnessed for gesture synthesis. A number of studies [1, 14, 31, 35, 36, 40, 41] have focused on the generation of co-speech gestures using diffusion models, introducing various strategies to augment motion generation and style control. Despite these advancements, existing works continue to struggle with the challenge of harmonizing diversity, authenticity, and speed in the generation of gestures from speech. 
diffusion transformers: 
 Diffusion Transformers have made significant advancements in the generation of high-quality images [4, 29] , videos [25] , and 3D models [27] . DiT [29] is among the pioneering works to explore transformer-based architectures in the field of image generation. It introduces innovative designs to train latent diffusion models by replacing the commonly used U-Net with a transformer backbone. PixArt-𝛼 [4] proposes the incorporation of cross-attention modules into the Diffusion Transformer (DiT) to inject text conditions, thereby achieving photorealistic text-to-image synthesis. DiT-3D [27] explores the use of plain diffusion transformers for 3D generation, which directly operate the denoising process on voxelized point clouds instead of latent spaces. The Masked Diffusion Transformer (MDT) [9] introduces a masked latent modeling scheme, enhancing the ability to learn contextual relations among semantic parts. Like the Masked Autoencoder (MAE) [12] , MDT also adopts an asymmetric transformer to predict masked tokens from unmasked ones while simultaneously undergoing the diffusion generation process. As a result, it achieves superior image synthesis performance with a state-of-the-art (SOTA) Frechet Inception Distance (FID) score on the ImageNet dataset and faster learning speed. In this paper, we aim to explore such a masked scheme in the domain of co-speech gesture generation. Our MDT-A2G directly operates the denoising process on the gesture and incorporates multi-modal information, including speech, emotion, and identity, to align with the gesture.The masked scheme first demonstrated its effectiveness in Natural Language Processing (NLP), with BERT-based models [8, 17] enhancing the performance of word embeddings through masked language modeling and transformer architecture. Subsequently, the Masked Autoencoder (MAE) [12] extended the masking strategy into the field of computer vision and developed an asymmetric encoder-decoder architecture with a high mask ratio. This concept of masked representation learning has been employed in other modalities as well, such as video [32] , audio [13] , and time series [7] .Most related to our work, EMAGE [22] proposes a temporal transformer to learn robust motion representation for gesture generation.In contrast, we resort to masking strategy to strengthen temporal relation learning sequence gesture, resulting faster training speed and more realistic gesture generation. 
preliminaries 3.1 gesture format: 
 We employ the BEAT dataset [23] , recognized for being the most comprehensive motion capture dataset in terms of duration and the diversity of modalities it covers. The BEAT dataset stores motion capture data in the BVH file format, articulating motion representation through Euler angles. Following DSG+ [37] , we favor using rotation matrices over Euler angles to represent joint rotations. We leverage 75 joints including 27 body joints, 48 hand joints for whole-body gesture generation and choose a subset of 14 upper-body joints, along with the 48 hand joints, for generating upper-body gestures.We consider starting the denoising process with pure noise and generating gestures using a diffusion model. We introduce then denoising diffusion probabilistic models (DDPM). DDPM defines a 𝑇 -step forward process and a 𝑇 -step reverse process. DDPM turns the present state 𝑥 0 into the previous state 𝑥 𝑡 by gradually adding random noise through the forward process. The diffusion process formulas is as follows:EQUATIONwhere 𝑥 𝑡 is the noised image at time-step 𝑡, 𝛼 𝑡 is the predefined scale factor, and N represents the Gaussian distribution. It means that the original state 𝑥 0 is transformed into 𝑥 𝑡 by gradually adding Gaussian noise.Conversely, the reverse denosing phase is designed to revert the noise-infused data back to its original structured form, effectively reconstructing the joint distribution 𝑝 𝜃 (𝑥 0:𝑇 ). This phase is defined by:EQUATIONHere, the model assumes a constant time-dependent variance Σ 𝜃 (𝒙 𝑡 , 𝑡) = 𝛽 𝑡 I. A generative model G 𝜃 is then formulated to estimate the mean of the Gaussian distribution. For scenarios requiring conditional generation, the conditional variable c is seamlessly incorporated into the model's architecture.We follow [37, 44, 49] to predict the signal itself instead of predicting 𝜖 𝜃 (𝑥 𝑡 , 𝑡) [18] . The network 𝐷 learns parameters 𝜃 based on the input noise 𝑥 𝑡 , noising step 𝑡 and conditions 𝑐 to reconstruct the original signal 𝑥 0 as :EQUATION 
methodology 4.1 problem formulation: 
 The task of co-speech gesture generation can be formulated as a sequence-to-sequence learning problem. Given an input sequence of speech features and other modal conditions, the goal is to generate a corresponding sequence of human gestures. The model needs to consider the temporal alignment between the speech and gesture sequences, as well as the incorporation of multi-modal information (e.g., emotion, speaker ID and text transcript) to generate more realistic and diverse gestures.We propose a novel framework named MDT-A2G for generating human gestures from multi-modal conditions. The overall training and sampling pipeline is illustrated in Figure 2 . Specifically, gestures are generated based on the noisy gesture 𝑥 𝑡 , audio 𝑥 𝑎 , text 𝑥 𝑡𝑥𝑡 , ID 𝑥 𝑠 , emotion 𝑥 𝑒 , and time steps 𝑡. We first obtain the input noisy gesture 𝑥 𝑡 and specific multi-modal condition. Then, we fuse the multi-modal feature to get 𝑥 𝑓 𝑢𝑠𝑒 as follows:EQUATIONwhere 𝐸 (•) represents the embedding layer. 𝐴𝑡𝑡𝑒𝑛𝑡𝑖𝑜𝑛(•) utilizes cross-local attention used in DSG+ [37] . We mask a portion of 𝑥 𝑓 𝑢𝑠𝑒 to obtain 𝑥 𝑢 , which is then fed into the transformer encoder to map it to a latent representation. We then concatenate the latent representation with multiple sets of learnable tokens 𝑥 𝑙 and then feed them into a self-attention (SA) block for feature integration. 3) Scaling-aware Accelerated Sampling Process. For the multi-modal feature extractor, we propose an innovative feature fusion strategy that integrates time embeddings with emotion and ID features. These will be further concatenated with text, audio, and gesture features, resulting in a comprehensive feature representation. Additionally, we have designed a Masked Diffusion Transformer structure to expedite the convergence of the denoising network, thereby leading to more coherent motions. Finally, we introduce a scaling-aware accelerated sampling process by utilizing diffused results from previous timesteps, resulting in a faster sampling process.Previous methods, such as DSG+, handle features from different modalities in a complex manner, repeatedly concatenating information from different conditions. We believe that this approach hinders the learning of features from different modalities. Therefore, we have simplified the feature fusion process compared to former methods. Our feature fusion strategy is simple-yet-effective, requiring only a single concatenation process. Obtaining specific conditions. The generation of human gestures is dependent on a set of specific conditions, which are extracted as follows. For audio 𝑥 𝑎 , the raw input is resampled at 16 kHz, and a suite of features including Mel Frequency Cepstral Coefficients (MFCCs), Mel spectrogram features, rhythmic features, and onset points are computed. These features are then integrated with features extracted by the pre-trained WavLM [5] model to form the final audio feature representation x𝑎 . For text 𝑥 𝑡𝑥𝑡 , word embeddings are obtained using the pre-trained FastText [26] model. A linear layer is then used to generate the final text feature x𝑡𝑥𝑡 . For ID 𝑥 𝑠 , the ID information is encoded as a one-hot vector and transformed into an ID embedding x𝑠 via a linear layer. Similarly, the emotional state 𝑥 𝑒 is encoded as a one-hot vector and transformed into an emotion feature x𝑒 . During training, the noise step 𝑡 is randomly selected from a uniform distribution ranging from 1 to 𝑇 . After position encoding, 𝑡 is processed by a MLP to yield the time feature t. The noisy gesture 𝑥 𝑡 is sampled from a standard normal distribution N (0, I) during training and sampling stages. Then, 𝑥 𝑡 is dimensionally reduced through a linear layer. During the training process, 𝑥 𝑒 and 𝑥 𝑠 is randomly masked according to a Bernoulli distribution. The temporal embedding 𝑥 𝑡 is then added to the ID and emotion features after passing through an MLP layer, resulting in the fused feature. Multi-modal Feature Fusion. The process of feature fusion involves the concatenation of the aforementioned conditions and noisy gestures along the feature dimension. This concatenated feature is then subjected to a cross-local attention module [37] , which serves to fuse the different modalities effectively, allowing the model to understand the complex relationships between speech, text, style, emotion, and gestures. This is particularly important in co-speech gesture generation, as gestures are often closely related to the content and emotion of the speech. Furthermore, the inclusion of emotion as an additional condition allows the model to generate gestures that are not only contextually appropriate but also emotionally congruent with the speech.Previous diffusion model methods on A2G tasks do not pay attention to the importance of masked modeling, which result in slow learning of the semantic correlations among different modal features and slow training convergence. Inspired by the success of masking scheme on video [32] , image [12] , we utilize this strategy on gesture generation task, leading to more coherent motion generations. We mask the input features and then feed them into multiple Transformer Blocks, which are divided into encoders, decoders, and side-interpolators. We adopt an asymmetric design that allows the encoder to only take in unmasked features, reducing computational cost. Masking Operation. Given a feature matrix 𝑥 𝑓 𝑢𝑠𝑒 ∈ R 𝑁 ×𝑑 , where 𝑑 signifies the number of channels and 𝑁 denotes the number of tokens, we randomly conceal tokens at a rate 𝜌. This results in a set of unmasked tokens, denoted as 𝑥 𝑢 ∈ R 𝑑 × N , where N = (1 -𝜌)𝑁 . To track which tokens are masked, we create a binary mask 𝑀𝐴𝑆𝐾 ∈ R 𝑁 with ones assigned to the masked positions. The unmasked tokens 𝑥 𝑢 are subsequently processed. Using only the unmasked tokens 𝑥 𝑢 helps to reduce computational overhead. Encoder. For the encoder, we only map the visible unmasked features to latent representations, which allows us to use fewer computations. Different masking rates affect the experimental results, and we will discuss them in detail in the experimental section. Side-interpolater. Inspired by [9] , the latent representationS is processed through the side-interpolator, which restores it to its original feature dimensions. As illustrated in Figure 2 , the sideinterpolator, a compact network, leverages the encoder's output to estimate the masked tokens during training, working solely with the unmasked tokens 𝑥 𝑢 . During inference, all tokens 𝑥 𝑢 are processed, creating a disparity in the number of tokens. To maintain consistency, the side-interpolator fills masked positions with a shared learnable token, generating an interpolated embedding 𝑥 𝐼 , computed as 𝑥 𝐼 = 𝑆𝐴(Concat[𝑥 𝑙 , 𝐸𝑛𝑐𝑜𝑑𝑒𝑟 (𝑥 𝑢 )]). A masked shortcut connection combines 𝑥 𝑢 and the interpolated embedding 𝑥 𝐼 into x𝑢 = (1-𝑀𝐴𝑆𝐾) •𝑥 𝑢 +𝑀𝐴𝑆𝐾 •𝑥 𝐼 . Within the side-interpolator, the unmasked features are concatenated with multiple sets of learnable latent mask tokens 𝑥 𝑙 , which assist in restoring the original feature size, before being fed into a Transformer block for feature interaction. Decoder. The decoder receives the output from the side-interpolator, which comprises the encoded latent representation along with the latent mask tokens. Ultimately, these side-interpolator outputs are passed to the decoder for reconstruction. These decoders are less deep compared to the encoders. In the training stage, the encoder, decoder, and side-interpolator work in tandem to forecast the masked features based on the unmasked features. During inference, the side-interpolator is no longer utilized.To train our networks, we employ the Huber loss as the gesture loss L𝑔:EQUATIONFollowing the DDPM denoising process, we predict the gesture x0 at each time step 𝑡, as depicted in Figure 2 . During the training process, we provide both the complete feature 𝑥 𝑓 𝑢𝑠𝑒 and the masked tokens 𝑥 𝑢 to the diffusion model. This approach ensures that the model does not overly focus on reconstructing the masked region while neglecting the diffusion training. The additional costs associated with using the masked latent embedding are minimal. This is further supported by the findings in Figure 1 , which demonstrate that MDT-A2G-B achieves faster learning progress in terms of total training hours compared to the DSG+.Previous diffusion models [31, 37] in the Audio-to-Gesture (A2G) domain have not fully explored the possibilities of skip sampling techniques similar to DDIM [30] , which can significantly expedite the sampling process. However, its direct application is problematic as it exacerbates the diffusion model's exposure bias [28] , the discrepancy between the sampled x𝑡 and the training's 𝑥 𝑡 . Assume that x𝑡 0 represents the original gesture predicted by the diffusion model at step 𝑡, to mitigate this issue, we propose the following method: Theorem 4.1. Given x𝑡 0 and the sampled x𝑡 at time t, we can compute x𝑡 -1 0 for time t-1 without relying on neural networks. The subsequent formula is employed to diminish the exposure bias:EQUATION)Here, 𝑠𝑐𝑎𝑙𝑒 is a hyperparameter greater than 1 but close to 1.The proof is detailed in Appendix A. Our approach incorporates a 1:N accelerated sampling method, where the current step is fully processed and the next N steps are expedited through skip sampling. It is crucial to note that this strategy is distinct from distillation-it maintains the network's performance during full sampling without necessitating extra training, and does not introduce additional computational overhead, thus offering a straightforward and practical solution.We have set different number of blocks for our MDT-A2G and designed variants with varying hyperparameters (e.g., number of layers and heads). We denote these variants of our model as MDT-A2G-XS (extra small), MDT-A2G-S (small), MDT-A2G-B (base), and MDT-A2G-L (large). Among them, the whole parameter size of MDT-A2G-B is comparable to that of DSG+. The specific network configurations are detailed in Appendix E. 
experiments 5.1 settings: 
 Datasets. We utilize the BEAT [23] dataset, which includes 120Hz motion capture data and audios from 30 speakers, featuring 10minute conversations and 1-minute self-talks. In addition to motion and audio, BEAT includes extra information such as text, identity, emotion annotations, and facial expressions. Following [33] , we select 1-hour audio per speaker and split the dataset 70% for training, 10% for validation and 20% for testing. Metrics. We use Frechet Gesture Distance (FGD) [39] to evaluate the quality of the generated gestures, which uses a pretrained gesture feature extractor and calculates the Frechet distance between the distribution of the features of real and generated gestures. Following [23] , we use Semantic-Relevant Gesture Recall (SRGR) [23] and Beat Alignment Score (BeatAlign) [20] to evaluate the diversity and synchrony of the generated gesture. SRGR enhances the traditional Probability of Correct Keypoint (PCK) [19] by introducing semantic relevance scoring as weights. BeatAlign is a measurement between audio and gesture beats through Chamfer Distance [20] , which reflects the similarity between audio and gesture beats. We also use Diversity Score [18] to measure the diversity of the generated gestures. The diversity score is measured of the average feature distance [18] . We use the pretrained autoencoder used for FGD.Implementation Details. The motion data is downsampled to 30 Hz from original 120 Hz and we use 300 frames for training. Following [2] , we use rotation matrix instead of the original Euler angle for rotation representation. We train our model for 120k steps with a batch size of 150 and use AdamW optimizer at a learning rate of 3e-5. All the experiments are conducted on a single NVIDIA A100 GPU.Comparison Methods. We compare our proposed method with state-of-the-art gesture generation methods, including DSG+ [37] , CaMN [23] and MDM [31] . For a fair comparison, we retain all these For MDM, we retrain it to take audio features as input. Addition to whole body gesture generation, we also conduct experiments to compare the results of upper body gesture generation.Table 1 presents the quantitative results. Our approach surpasses other methods in generating both upper body and full-body motions. While masked modeling greatly improves the realism of the generated gestures, it does not necessarily enhance the Diversity Score. It's important to note that a higher Diversity Score is not always inherently beneficial. Random noise may achieve a high Diversity Score, but it is not the target we aim for. Additionally, the inherent randomness in the sampling of diffusion models can negatively affect the SRGR metric, which focuses on the accuracy of generated gestures. However, creating realistic motions does not always require strict conformity to ground-truth movements. Nonetheless, our method effectively reduces the adverse impacts on the SRGR metric. Furthermore, we have discovered that employing masked modeling can enhance the Beatalignment score between audio and gestures, thereby improving the Beatalignment score. This approach compels the denoising network to learn the correlated features between multi-modal conditions and gestures.In the visual representation of our experimental outcomes illustrated in Figure 3 , we clearly delineate the distinctions between the competing methodologies and our proposed approach. The CaMN model employs a straightforward methodology that merely concatenates multimodal inputs. While this results in technically correct outputs, the resultant actions are notably monotonous and lack dynamic variation, as highlighted within the blue box of the figure. This method, while reliable, does not allow for the generation of nuanced or complex motion patterns, leading to a visually static presentation. On the other hand, DSG+ leverages an advanced diffusion model coupled with self-attention mechanisms, aimed at enhancing the diversity of generated actions. This model is capable of producing a wide array of actions, thereby enriching the dynamic quality of the outputs. However, this complexity comes with its own set of challenges, as there is a tendency to generate anatomically incorrect gestures or unrealistic motions, as indicated by the examples in the red box. While the ambition of DSG+ to achieve high variability is commendable, it sometimes sacrifices accuracy and realism in its generated actions. In contrast, our proposed methodology synthesizes the strengths of both the CaMN and DSG+ approaches while mitigating their weaknesses. By integrating a refined feature extraction mechanism with a robust diffusion process, our model not only ensures the Effectiveness of diffrent model scale. The impact of different scale models on the outcomes has been extensively validated in the field of image generation. However, previous methods have not explored the impact of model scalability on the audio-to-gesture task. As we utilize Masked Diffusion Transformer as backbone, which possesses excellent scalability properties, we compare the effects of different network sizes on gesture generation here. Table 4 shows the quantitative results. Effectiveness of sampling acceleration. Table 2 illustrates the impact of various acceleration ratios. Notably, at an acceleration ratio of 1:20 and 1:25. We do not observe significant increases in FGD Score and decreases in Diversity Score. Meanwhile, the SRGR remain relatively stable. Furthermore, we find that introducing the Scaling-aware Accelerated Sampling Process (𝑠𝑐𝑎𝑙𝑒 > 1) yield better results. Qualitative comparison is shown in Figure 4 . Effectiveness of multi-modal fusion and mask modeling. We conduct a quantitative comparison with and without multi-modal fusion module and mask modeling. Table 3 indicates that mask modeling reduce the Diversity score, this may be attributed to the enhanced consistency among different gestures. However, the realism of the generated gestures is significantly improved. Effectiveness of Different Feature Processing. To evaluate the impact of our proposed feature processing technique, we establish a baseline by excluding the feature processing component from our MDT-A2G. We subsequently incorporate different feature processing methods for comparison: AdaLN-Zero is the method utilized by DiT [29] . Table 6 demonstrates that our proposed multi-modal fusion module shows the best performance. Notably, the use of AdaLN-Zero can not enhance results, potentially due to the introduction of excessive conditional information. Detailed network architecture diagrams for these methods are included in the appendix.Effectiveness of different wider mask ratio. We use different wider mask ratios [10] , where "wider" refers to masking the features within the range [𝜌, 𝜌 + 0.2]. Table 5 provides a comparison of different wider mask ratios. Notably, MDT-A2G-B achieves optimal performance with a mask ratio of 40%. A higher mask ratio might limit the network to reconstructing masked features, reducing diversity. Our experiments show that using varied mask ratios, instead of a fixed one, improves model robustness and adaptability across different scenarios. As shown in Table 7e , removing a wider range of mask ratios led to an increase in FGD to 47.75. Implementing variable mask ratios helps the network learn to reconstruct and generate features under varied levels of information availability, thereby improving its ability to handle real-world data.Effectiveness of decoder depth. As shown in Table 7a , optimal results are obtained when the number of transformer blocks of decoder is set to 4. This suggests that using a deeper decoder enhances the ability to assimilate and process correlated information across different features, thereby improving overall model effectiveness.Effectiveness of Side-Interpolator. Table 7b illustrates the impact of the side-interpolator on FGD Score. Side-Interpolator helps maintain uniformity in feature processing, which is critical for the accurate reconstruction of the input data. Additionally, as depicted in Table 7f , we investigate the effect of different number of side-interpolators. Optimal results are obtained with a single side-interpolator. Increasing side-interpolators can homogenize semantic information between learnable masks and unmasked tokens, potentially reducing effective reconstruction of masked tokens, impairing feature differentiation, and degrading performance.Effectiveness of masked shortcut. Table 7c shows our investigation into how masked shortcuts impact network performance.Incorporating masked shortcuts ensures that the learnable mask parameters in the side-interpolator do not interfere with unmasked tokens, facilitating more effective information flow. This design prevents the dilution of important features during encoding, enhancing the model's overall integrity and effectiveness. Effectiveness of Full/Unmasked Features. As shown in Table 7d , we analyzed the scenario where only unmasked features are fed into the MDT, rather than both unmasked features and all features. This practice can inadvertently cause the network to focus on predicting the masked features, potentially harming performance. Balancing the masking process is crucial to optimize the network's ability to learn from both masked and unmasked features. 
conclusion: 
 We introduce the novel MDT-A2G, a masked diffusion transformer for co-speech gesture generation. Benefited by the unique mask modeling scheme, MDT-A2G enhances stronger temporal and semantic relation learning among gestures, thereby accelerating the learning process. The integration of multi-modal information has proven effective in generating more realistic gestures. Besides, our efficient inference strategy has reduced denoising computation, resulting in a 5.7× speedup with minimal performance degradation. Experimental results confirm MDT-A2G's superiority in gesture generation, with significantly faster training and inference speeds than traditional counterparts. This research has bridged a significant gap in co-speech gesture generation, opening avenues for future studies. 
limitations: 
 DiT showcases strong scalability in image generation, and we aim to explore similar scalability in the A2G domain. However, we haven't observed the benefits of MDT-A2G's scalability yet. One possible reason is insufficient training data; as network complexity grows, so should the dataset. These are hypotheses that require further validation. 
 
 Appendix:  


Decision: The manuscript introduces MDT-A2G, a novel Masked Diffusion Transformer for co-speech gesture generation, which integrates multi-modal information including text, emotion, and identity. This model enhances contextual reasoning of temporally aligned speech-driven gestures through a mask modeling scheme, resulting in coherent and realistic motions. The proposed approach boasts significant improvements in learning and inference speeds compared to traditional diffusion transformers. The manuscript should incorporate detailed responses to enhance clarity and practicality, addressing the points raised by the reviewers to improve its final version.

Reviews: review: paper_summary: paper_summary: "MDT-A2G: Exploring Masked Diffusion Transformers for Co-Speech Gesture Generation" introduces a novel approach using Masked Diffusion Transformers (MDT) for generating human gestures that align with speech. The proposed model, MDT-A2G, enhances the contextual reasoning of temporally aligned speech-driven gestures by employing a mask modeling scheme, which strengthens temporal relation learning among gesture sequences. This model integrates multi-modal information, including audio, text, emotion, and identity, and proposes an efficient inference strategy to speed up the denoising process.

* Novel Framework: Introduction of MDT-A2G, a framework leveraging masked modeling combined with diffusion transformers for co-speech gesture generation.
* Multi-modal Integration: Incorporation of text, emotion, and identity information to improve the contextual understanding of gestures.
* Efficient Inference Strategy: A sampling acceleration technique that reduces inference time while maintaining performance.
* Improved Performance: Demonstrated superior performance in generating realistic and diverse gestures with faster training and inference speeds compared to existing methods.
strength_weakness: strength_weakness: * Innovative Use of Masked Modeling: The application of masked modeling to diffusion transformers for gesture generation is novel and enhances temporal relation learning.
* Multi-modal Fusion: Effective integration of various modalities (audio, text, emotion, identity) results in more realistic and contextually appropriate gestures.
* Efficiency: The proposed sampling acceleration method significantly reduces inference time, making the model more practical for real-time applications.
* Comprehensive Evaluation: Extensive experiments and ablation studies demonstrate the model's superior performance in terms of gesture quality, diversity, and training efficiency.
limitations: limitations: * Complexity of Model: The model's complexity, with multiple components like SEAFusion and various feature extractors, might pose challenges for implementation and require significant computational resources.
* Evaluation Metrics: While the paper uses metrics like FGD, Diversity, SRGR, and BeatAlign, a more detailed analysis of how these metrics correlate with human perceptual quality could strengthen the evaluation.
* Generalization: The model is trained and tested on the BEAT dataset, which, although comprehensive, may not cover all variations in human gestures and speech patterns. Testing on additional datasets could further validate the model's robustness.
* Ablation Studies: Although thorough, the ablation studies could include more detailed analysis on the trade-offs between model complexity and performance improvements, particularly in practical deployment scenarios.
* Subjective Evaluation: Based on the findings from the GENEA Challenge, the reliance on objective metrics alone may not fully capture the perceptual quality of the generated gestures. The supplementary video suggests that the performance of MDT-A2G is not significantly different from baseline models, highlighting the need for subjective evaluation metrics and corresponding experiments to assess human perception.
* Comparative Analysis: There should be more extensive comparisons with other baseline models. Including a wider range of baseline models in the comparative analysis would provide a clearer understanding of the advantages and limitations of MDT-A2G.
paper_summary: paper_summary: The paper attempts the task of co-speech gesture generation. For this task, the authors propose leveraging the masked diffusion transformer as the backbone to improve the generation diversity and devise a sampling strategy to speedup the generation process. Specifically, the proposed MDT-A2G first fuse the timesteps, emotion, speaker ID, text transcript, audio and noisy gesture as the diffusion latents. Then masked diffusion transformer is adopted for denoising prediction. Finally, to overcome the inefficiency of diffusion models, the authors propose a new skip-sampling method to accelerate the generation.
strength_weakness: strength_weakness: 1. The paper is easy to understand. 
2. The proposed MDT-A2G improves the performance on the task of co-speech gesture generation. 
3. The authors have conduct comprehensive experiments, especially for ablation studies to evaluate the effectiveness of multiple modules.
limitations: limitations: 1. The paper has limited novelty. One of the main contributions is to utilise masked diffusion transformer in the task co-speech gesture generation. However, I haven't seen any modification or improvement compared to original MDT structure. Therefore, the training acceleration compared to DSG+ also benefits the utilisation of MDT. I am not meaning that directly utilising MDT gives sufficient reason to get rejected. Demonstrating the generalization of MDT in different domains also has the merit. But from the results in Table 1, as MDT-A2G cannot consistently exceed previous approaches, some moderate modifications on MDT or the pipeline may be necessary. 
2. Several important details are not well explained. From my interpretation, the conditions like audio, text are sequential data while speaker ID, emotion are scalars. The explanations on how to concatenate these data together are absent. Moreover, the audio and text may have different lengths. How to align these two modalities? Moreover, do the audio condition and output have the same lengths / frames? There are no notations regarding the frames of inputs/ latents / outputs. It is generally difficult to generate long-term sequences. However, the details on training duration / generation duration are also absent. Additionally, style loss and emotion loss are not well explained. 
3. I also have some minor concerns. I understand the proposed scaling-aware accelerated sampling process may be one contribution / technical novelty in this paper. However, this strategy seems to be independent from the task of co-speech gesture generation. Why not consider other acceleration sampling strategies, such DDIM / series of DPM solvers, etc? Will the proposed method have advantages over these solvers? In Table 2, when using less inference steps (acceleration ratio is equal to 1:20 / 1:25), why the performance in terms of SRGR and BeatAlign retain the similar levels? In Table 7, why the performance is not improved with the model size scales to larger scale?
paper_summary: paper_summary: They propose MDT-A2G, a masked diffusion transformer for co-speech gesture generation. the masked diffusion model takes multi-modal conditions and partially masked noisy features as input. Their experiments demonstrate that MDT-A2G can generate high-quality, diverse
and temporally coherent human gestures with high efficiency.
strength_weakness: strength_weakness: - Well-designed architecture
- Effective training framework, masked diffusion model. This can be applied to other modalities
- Comprehensive evaluations, including ablation studies
- Well-written manuscript
limitations: limitations: - Qualitative examples are great, but I wonder why some models introduced in Section 2 are not compared in the experiments. I'm not familiar with speech-to-gesture generation. I'd appreciate it if the authors would explain it. I will raise my rating if I am convinced.
- By making the code publicly available, the authors can contribute more to the community.
paper_summary: paper_summary: The paper is about the exploration of a novel Masked Diffusion Transformer (MDT-A2G) for co-speech gesture generation. It aims to improve the generation of high-quality human gestures that are synchronized with speech audio by leveraging the capabilities of the Transformer architecture. The MDT-A2G model incorporates a mask modeling scheme to enhance the learning of temporal relations among gesture sequences, leading to more coherent and realistic motions.
Additionally, the paper proposes an efficient inference strategy that speeds up the denoising process by using previously calculated results, achieving faster training and inference speeds compared to traditional diffusion transformers.
strength_weakness: strength_weakness: The paper presents a complete set of experiments and conducts ablation studies in an organized manner to analyze the outcomes resulting from the model enhancements. The final results demonstrated that the improved model outperformed other variants in the ablation studies.
limitations: limitations: I do not think that the use of a masked diffusion model here is a valid innovation point. According to the paper, in fact, any modeling between modalities that requires semantic association could potentially be improved by adding masks to force the model to pay more attention to semantic information. Alternatively, is this paper the first work in the field to add masks?
It appears as though you have first discovered an approach that could potentially be incorporated and then put in the effort to demonstrate its efficacy. This is instead of initially proving through experiments that there is a particular problem within the domain, and subsequently proposing a solution targeted at this issue. To put it another way, if the problem you are addressing is to enhance the speed of generation and the realism of gesture generation, why is it necessarily MDT? Why not some other approach?