MambaGesture: Enhancing Co-Speech Gesture Generation with Mamba and Disentangled Multi-Modality Fusion

Authors: yabiao  wang
jiangning  zhang
zhengkai  jiang
xiaofeng  mao
jiafu  wu
weijian  cao
chengjie  wang
yong 2024 liu
chencan  fu
yanhao  ge
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-29 12:54:51

Abstract: Co-speech gesture generation is crucial for producing synchronized and realistic human gestures that accompany speech, enhancing the animation of lifelike avatars in virtual environments. While diffusion models have shown impressive capabilities, current approaches often overlook a wide range of modalities and their interactions, resulting in less dynamic and contextually varied gestures. To address these challenges, we present MambaGesture, a novel framework integrating a Mamba-based attention block, MambaAttn, with a multi-modality feature fusion module, SEAD. The MambaAttn block combines the sequential data processing strengths of the Mamba model with the contextual richness of attention mechanisms, enhancing the temporal coherence of generated gestures. SEAD adeptly fuses audio, text, style, and emotion modalities, employing disentanglement to deepen the fusion process and yield gestures with greater realism and diversity. Our approach, rigorously evaluated on the multi-modal BEAT dataset, demonstrates significant improvements in Fréchet Gesture Distance (FGD), diversity scores, and beat alignment, achieving state-of-the-art performance in co-speech gesture generation.

Main paper: 
 introduction: 
 Co-speech gesture generation, the task of producing human gestures synchronized with audio and other modalities, is crucial for enhancing avatar realism in animation, film, and interactive gaming. Crafting gestures that are both realistic and diverse is a significant challenge and a focal point in contemporary research.Extensive research has been conducted in this area, leading to the development of numerous innovative approaches. Gesture generation techniques are generally divided into rule-based and datadriven methods, with the latter further categorized into statistical and learning-based approaches. This paper focuses on learningbased co-speech generation methods, which can generally be divided into two categories. 1) Autoencoder-based methods, which employ autoencoders (AEs) [27] or variational autoencoders (VAEs) [26, 48] to translate gesture generation into a reconstruction task, as shown in Figure 1 . Despite their computational efficiency, these methods are limited by their architecture, resulting in restricted gesture diversity. 2) Diffusion-based methods, recognized for producing diverse gestures, such as DiffuseStyleGesture+ [46] , which combines diffusion models with Transformer encoders and integrates multiple modalities to enhance realism and diversity. However, these methods often fail to fully exploit the rich interactions among multi-modal data, leading to less expressive gestures.Drawing inspiration from the state space model Mamba, effective in synthetic tasks, language modeling, and audio generation [10] , we recognize its potential for co-speech gesture generation. Mamba, Figure 1 : Comparison of our approach with mainstream co-speech generation methods. (a) Autoencoder (AE)-based methods [26, 27] synthesize gestures by fusing multi-modal data but inherently suffer from limited diversity due to architectural constraints. (b) Diffusion-based methods [44, 46] employ diffusion models with Transformers to generate diverse gestures but are hindered by the quadratic complexity of Transformer and often overlook intricate multi-modal correlations. (c) Our MambaGesture leverages the linear scaling and sequential data processing advantages of the State Space Model to enhance gesture diversity and effectively harness multi-modal data with disentangled feature fusion, ensuring a broader spectrum and higher realism in gesture generation.an evolution of RNNs, overcomes their limitations in parallel computation through a parallel scan algorithm and boasts linear scaling, contrasting with the quadratic complexity of traditional transformers. Our exploration into Mamba's application in co-speech gesture generation reveals its capacity to produce gestures that are both realistic and diverse. Experimentally, we find that combining Mamba's sequential modeling strengths with the contextual awareness of attention mechanisms yields the best results. We propose integrating multiple modalities (audio, text, style, and emotion) to significantly elevate the realism and diversity of generated gestures. While audio lays the foundation for gesture cues, adding style and emotion enriches the gestural output, capturing individual expressions and emotional subtleties. Previous research has often focused on audio as the primary input, neglecting the depth of information from other modalities. We argue that audio contains rich details that, when effectively disentangled and combined with text, style, and emotion, can greatly refine gesture synthesis. Our approach aims to fully exploit this rich multi-modal data, with a particular emphasis on audio disentanglement, to achieve more nuanced and contextualized co-speech gesture generation. The concept of our approach is illustrated in Figure 1 .In this work, we introduce MambaGesture, a novel framework that integrates a Mamba-based attention block, MambaAttn, with a multi-modality feature fusion module, SEAD. The MambaAttn block, depicted in Figure 2 , leverages the robust sequence modeling capabilities of the state space model Mamba. This integration within our gesture diffusion model enables the production of gestures that are both realistic and diverse. Additionally, we present the crossattention enhanced Style and Emotion Aware Disentangled (SEAD) feature fusion module. This module introduces a novel technique for disentangling audio to extract personal style and emotion. The combination of MambaAttn's advanced sequence modeling with the SEAD module empowers our framework to generate co-speech gestures that are both diverse and realistic.Our primary contributions are as follows:(1) We are the first to introduce the Mamba model to the field of diffusion-based co-speech gesture generation. Mamba's superior sequence modeling capabilities make it well-suited for tasks requiring temporal coherence and dynamic gesture representation.(2) We introduce the MambaAttn block, which enhances sequential modeling, and the SEAD module, a novel audio disentanglement approach that fuses multi-modal data. The SEAD module effectively captures personal style and emotion from speech audio, enriching the multi-modal information used for gesture generation and leading to more realistic and diverse gestures.(3) Our extensive experimental evaluation on the large multimodal BEAT dataset confirms that MambaGesture achieves state-ofthe-art performance in co-speech gesture generation. Our method outperforms existing models on several key metrics, highlighting the effectiveness of our contributions. 
related work 2.1 co-speech gesture generation: 
 Co-speech gesture generation involves producing gestures synchronized with speech audio, a challenging task due to the lack of explicit mappings between speech and gestures.Gesture generation methodologies can be broadly categorized into rule-based and data-driven approaches. The latter is further subdivided into statistical and learning-based methods [32] . Rulebased methods [4, 19, 20, 29, 35, 40] are known for generating high-quality motions but lack the flexibility and diversity of datadriven systems. Statistical systems [3, 8, 17, 22, 31, 47] model gesture distribution by analyzing the statistics rather than relying on expert-encoded rules, typically involving pre-computing conditional probabilities or assigning prior probability distributions.Recently, learning-based methods using CNNs, RNNs, and transformers have gained traction. Liu et al. [28] propose a hierarchical approach for gesture generation, considering the hierarchical nature of speech semantics and human gestures. Yoon et al. [49] treat gesture generation as a translation problem, employing a recurrent neural network that utilizes multi-modal contexts of speech text, audio, and speaker identity, incorporating an adversarial scheme to enhance realism. Liu et al. [27] introduce the BEAT dataset, featuring gestures, facial expressions, audio, text, emotions, speaker identity and semantics. Their Cascaded Motion Network (CaMN) synthesizes body and hand gestures using adversarial training across multiple modalities. DisCO [25] disentangles motion into implicit content and rhythm features, feeding the processed features into a motion decoder to synthesize gestures. EMAGE [26] employs Masked Gesture Reconstruction (MG2G) to encode body hints and Audio-Conditioned Gesture Generation (A2G) to decode pre-trained face and body latent features, generating facial and local body motions using a pre-trained VQ-Decoder. Yi et al. [48] introduce TalkSHOW, which utilizes an autoencoder for facial generation and a VQ-VAE for body and hand motion generation, with an autoregressive model predicting the multinomial distribution of future motion during inference.These methods often treat co-speech gesture generation as a reconstruction task, facing challenges in establishing mappings between speech and gestures, resulting in limited diversity. However, recent advancements in diffusion-based methods offer a promising alternative, producing gestures with high realism and diversity.Diffusion models, known for complex data distribution modeling and many-to-many mappings, are gaining popularity for gesture synthesis. Several works have used text as a condition for diffusion models to generate human motion, such as MotionDiffuse [50] , FLAME [16] , and MDM [38] . Recent research focuses on generating co-speech gestures using diffusion models. Alexanderson [1] adapts the DiffWave architecture [18] , replacing dilated convolutions with Transformers or Conformers to enhance performance. Zhu et al. [53] introduce DiffGesture, which concatenates noisy gesture sequences with contextual information in the feature channel for temporal modeling using transformers. DiffuseStyleGesture+ [46] conditions on audio, text, style, and seed gesture, employing an attention-based architecture for denoising. UnifiedGesture [43] utilizes a retargeting network to standardize primal skeletons from various datasets, expanding the data pool and employing VQ-VAE and reinforcement learning for gesture generation refinement. LivelySpeaker [52] emphasizes the importance of semantics in gesture understanding and adopts a two-stage strategy for semantic-aware and rhythm-aware gesture generation. AMUSE [7] disentangles speech into content, emotion, and personal style latent representations, using a motion VAE transformer architecture for the conditional denoising process in latent space. FreeTalker [45] generates spontaneous co-speech gestures from audio and performs text-guided non-spontaneous gesture generation. Ges-tureDiffuCLIP [2] processes text, motion, and video prompts with different CLIP models to achieve style control. DiffSHEG [5] considers expressions as cues for gestures, achieving real-time joint generation of expressions and co-speech gestures.However, most existing approaches do not consider full modalities, nor do they offer a comprehensive analysis of the interactions between these modalities, potentially leading to less diverse and realistic generated gestures. Our proposed MambaGesture introduces a novel cross-attention enhanced Style and Emotion Aware Disentangled (SEAD) feature fusion module, which cleverly disentangles style and emotion from audio inputs and integrates rich multi-modal conditions (audio, text, style, and emotion) to facilitate the generation of gestures that are both realistic and diverse.State Space Models (SSMs) [9, 11, 12, 15, 33, 37] have regained popularity in sequence modeling tasks due to their linear or near-linear scaling with sequence length, outperforming attention mechanisms with quadratic scaling. Originating from RNNs and CNNs, SSMs employ state variables to model dynamic systems, making them foundational in fields like control theory and robotics.The core state space model is represented by the equations:EQUATIONEQUATIONwhere 𝑥 (𝑡) is an N -dimensional latent state, 𝑦 (𝑡) is the output, 𝑢 (𝑡) is the input, and A, B, C, D are system parameters, with D𝑢 often acting as a skip connection.The Structured State Space Sequence model (S4), introduced by Gu et al. [11] , builds on SSMs to achieve generative modeling at scale and fast autoregressive generation. S4 uses the HiPPO matrix to construct parameter 𝐴, mitigating the challenge of gradient scaling with sequence length. Despite their advantages, SSMs' constant parameters can limit their adaptability and content awareness.Mamba [10] revolutionizes sequence modeling by maintaining linear complexity, akin to state space models (SSMs), while rivaling Transformers' capabilities. It achieves this through an innovative input-dependent selection mechanism that dynamically adjusts parameters based on the input, significantly enhancing the model's content sensitivity. Additionally, Mamba incorporates computational strategies such as kernel fusion, parallel scan, and recomputation, which collectively streamline the computational process. These features have spurred the creation of Mamba-based applications, including MambaTalk [42] , which replaces traditional attention mechanisms with Mamba blocks for efficient and high-quality gesture generation, and Motion Mamba [51] , which leverages a U-Net architecture with hierarchical temporal and bidirectional spatial Mamba blocks for advanced denoising capabilities, further augmented by a CLIP text encoder for input conditioning. The development of these applications underscores Mamba's adaptability and its burgeoning role in enhancing motion generation tasks. 
preliminary 3.1 human gesture data format: 
 Human gestures are predominantly represented using rotationbased formats, with joint rotations typically expressed in SO(3). These can be parameterized through various methods, including Euler angles, axis angles, and quaternions [54] .In this paper, we utilize the BEAT dataset [27] , noted for its extensive duration and diverse modalities. The motion capture data in the BEAT dataset is stored in BVH file format, with motion represented via Euler angles: 75 × 3 rotations + 1 × 3 root translation. This dataset includes 27 body joints and 48 hand joints. Consistent with the methodologies employed by DiffuseStyleGesture+ [44] , we prefer rotation matrices over Euler angles for joint rotations to enhance the robustness and accuracy of our gesture generation model. Consequently, we treat the root translation as a single joint, resulting in a total of 76 joints. We use all 76 joints for whole-body gesture generation and select 14 upper-body joints, along with the 48 hand joints, for upper-body gestures.Denoising diffusion probabilistic models (DDPMs) are generative models designed to approximate real-world data distributions, denoted as 𝑞(x 0 ). Introduced by Ho et al. [14] , DDPMs employ a forward diffusion process that incrementally adds Gaussian noise to data, transitioning it towards a noise distribution, and a reverse process that reconstructs the original data from the noise.The forward diffusion is a Markov chain described as:EQUATIONEQUATIONwhere 𝛽 𝑡 increases over time, making the data resemble Gaussian noise N (0, 𝑰 ).The reverse process reconstructs the data distribution 𝑝 𝜃 (𝒙 0:𝑇 ):EQUATIONEQUATIONwith Σ 𝜃 (𝒙 𝑡 , 𝑡) as a time-dependent constant. The model approximates the mean of the Gaussian distribution during the reverse process.Our approach diverges from predicting noise at each step 𝑡. Instead, we predict the clean data sample 𝒙 0 directly, following recent methodologies [34, 38, 44] , to enhance the generative model's efficiency and accuracy. 
method: 
 Our approach is structured around two pivotal components: the cross-attention enhanced Style and Emotion Aware Disentangled (SEAD) feature fusion module and the MambaAttn-based denoising network. At the core of our motion generation lies the diffusion model framework, which employs an iterative process of diffusion (adding noise) and denoising to reconstruct original gestures from a noise distribution, conditioned on audio and additional multimodal data inputs. The SEAD module is tasked with the intricate fusion of multi-modality data, while the MambaAttn-based denoising network is dedicated to the accurate prediction of gestures. The overview of our proposed MambaGesture is illustrated in Figure 2 .Our methodology introduces a progressive series of multi-modality feature fusion techniques. These methods incrementally integrate features from various modalities, evolving from simple concatenation to sophisticated, disentangled fusion.SA and SEA Fusion. Adopting the state-of-the-art DiffuseS-tyleGesture+ (DSG+) [44, 46] as our baseline, we refine the gesture generation process by conditioning on a set of modalities, including timestep 𝑡, noisy gesture 𝑥 𝑡 , and conditions 𝑐. In contrast to DSG+, which employs audio 𝑎, text 𝑡𝑒𝑥𝑡, style 𝑠, and seed gesture 𝑑, our approach introduces emotion 𝑒 as a condition while dispensing with the seed gesture. This decision is informed by the recognition that emotion plays a critical role in the natural variation of gestures.We process each modality through a series of steps to prepare the features for fusion. The timestep is encoded through position encoding and an MLP to produce the time feature 𝑓 𝑡 . The noisy gesture 𝑥 𝑡 is encoded by a linear layer to obtain the noisy gesture feature 𝑓 𝑔 . Audio features are extracted and enriched with pretrained models to form 𝑓 𝑎 , while text and style features are similarly processed to yield 𝑓 𝑡𝑒𝑥𝑡 and 𝑓 𝑠 , respectively. Emotion features are encoded to produce 𝑓 𝑒 , with both style and emotion features subject to random masking during training.The preliminary feature processing of the multi-modality data is completed before training. These preprocessed features are then utilized in the subsequent fusion module.Previous works have explored multi-modality fusion for input conditions [27, 44] . They perform fusion in a cascaded way or leverage the powerful modeling capabilities of attention mechanism [39] . However, in current diffusion-based gesture generation methods, the process of multi-modality feature fusion is often complex and lacks clarity. We first introduce the Style Aware (SA) feature fusion module, which simply uses audio feature 𝑓 𝑎 , text feature 𝑓 𝑡𝑒𝑥𝑡 and style feature 𝑓 𝑠 as conditions, and concatenates them with the noisy gesture feature 𝑓 𝑔 and time feature 𝑓 𝑡 . After concatenating all modality features, we employ a cross-local attention [36] module to capture local information within the concatenated feature, resulting in the fused multi-modality feature 𝑓 𝑓 𝑢𝑠𝑒 .Moreover, many studies overlook the role of emotion. However, emotion can significantly influence our gestures when we speak. To address this, we propose the Style and Emotion Aware (SEA) feature fusion module, building upon SA, which introduces emotion as a generation condition. The emotion feature 𝑓 𝑒 is fused through concatenation in the same way. By utilizing audio, text, style and emotion, the comprehensive conditions provide a more specific description and command, which benefits our gesture generation.SEAD Fusion. Existing works on multimodal fusion often fail to fully exploit the inherent relationships between modalities. Our Style and Emotion Aware Disentangled (SEAD-basic) module, an extension of the SEA module, disentangles style and emotion from audio input using self-supervised learning. It avoids the complexity of two-stage disentanglement methods. This module consists of three independent units that extract style and emotion features from the audio feature 𝑓 𝑎 as follows:EQUATIONwhere linear layers are used to extract style and emotion features from 𝑓 𝑎 . The extracted 𝑓 𝑠 𝑎 and 𝑓 𝑒 𝑎 are aligned with the corresponding style and emotion features using style loss L 𝑠 and emotion loss L 𝑒 to facilitate the separation of personal style and emotion information from the audio.After disentangling the audio style feature 𝑓 𝑠 𝑎 and audio emotion feature 𝑓 𝑒 𝑎 , we fuse them with the original style and emotion features to obtain enhanced representations 𝑓 ℎ 𝑠 and 𝑓 ℎ 𝑒 :EQUATIONEQUATIONwhere Linear denotes a linear layer, and Cat denotes concatenation. By combining them with the original features, we decouple style and emotion from the original audio and enhance them. The 𝒇 ′ 𝒔𝒆 is obtained by concatenating 𝒇 ′ 𝒔 and 𝒇 ′ 𝒆 , and projected to original dimension by linear layer. Besides, we present a Mamba-based component termed the MambaAttn block, which merges Mamba with its sequence modeling proficiency and employs an attention mechanism to learn global information. Our denoising architecture, MambaAttn denoiser, is composed of a stack of MambaAttn blocks and a linear layer. During the sampling phase, we predict the gesture x0 by applying the fused conditions within a cyclical denoising and diffusion procedure.Building on the SEAD-basic module, the cross-attention enhanced Style and Emotion Aware Disentangled (SEAD) feature fusion module further enhances the audio feature 𝑓 𝑔 𝑎 with the fused style and emotion feature 𝑓 ′ 𝑠𝑒 using a cross-attention mechanism:EQUATIONwhere 𝑄 represents the feature 𝑓 𝑔 𝑎 , 𝐾 and 𝑉 represent the fused style and emotion feature 𝑓 ′ 𝑠𝑒 , and 𝑑 is the dimension of the feature. The fused style and emotion feature 𝑓 ′ 𝑠𝑒 is obtained as follows:EQUATIONThe SEAD module represents the pinnacle of our fusion approach, where the audio feature 𝑓 𝑔 𝑎 is further refined through cross-attention with fused style and emotion feature 𝑓 ′ 𝑠𝑒 . This mechanism effectively integrates the audio with the style and emotion, enhancing the representational capability of the audio feature and resulting in a comprehensive fused multi-modality feature 𝑓 𝑓 𝑢𝑠𝑒 .After integrating the multi-modal conditions into a comprehensive feature representation 𝑓 𝑓 𝑢𝑠𝑒 , we move to the gesture prediction phase. Here, we employ our innovative MambaAttn-based network, MambaAttn Denoiser, to predict gestures x0 from the noisy gesture input 𝑥 𝑡 . The MambaAttn Denoiser comprises 8 MambaAttn blocks and a linear projection layer.MambaAttn Block. The MambaAttn block is a novel architectural component that combines the sequential data processing strengths of the Mamba model with the contextual awareness of the attention mechanism. This fusion creates a powerful tool for capturing the intricacies of co-speech gesture generation.The structure of the MambaAttn block, as shown in Figure 2 , includes a self-attention layer, a Mamba block, and two instances of layer normalization. The process begins with the fused feature 𝑓 𝑓 𝑢𝑠𝑒 undergoing layer normalization, which then enters the selfattention module:EQUATION)where 𝑄, 𝐾, 𝑉 = LN(𝑓 𝑓 𝑢𝑠𝑒 ), and LN denotes layer normalization. The self-attention module distills global contextual information, which is then refined by the Mamba block to enhance sequence modeling. The output is subsequently processed as follows:EQUATION)The final output of the MambaAttn blocks is then projected back to the original dimension of the gesture data through the linear layer, yielding the predicted gesture x0 .Training and Sampling. To train our networks, we use the Huber loss function as the primary metric for gesture loss L 𝑔 :EQUATION)For the style and emotion losses, L 𝑠 and L 𝑒 , we use the L1 Loss, formulated as:EQUATIONThe final loss function is a composite of the gesture, style, and emotion losses, weighted and summed as follows:EQUATIONwhere we set 𝛼 = 1 and 𝛽 = 1 for simplicity. Following the DDPM denoising paradigm, we iteratively predict the gesture x0 at each timestep 𝑡, as illustrated in Figure 2 , refining our model's ability to generate accurate and lifelike gestures. 
experiments 5.1 experiment settings: 
 Dataset. We evaluate our method using the BEAT dataset [27] , which includes human motions captured at 120Hz via a motion capture system. This dataset features extended conversation audios (approximately 10 minutes each) and brief self-talk audios (around 1 minute each) from 30 diverse speakers. It is multi-modal, offering motion, audio, text, style (identity), emotion, and facial expression annotations. The dataset covers 8 emotions: neutral, anger, happiness, fear, disgust, sadness, contempt, and surprise. Following [41] , we select one hour of audio per speaker, splitting the data into 70% for training, 10% for validation, and 20% for testing. We generate both upper body and whole body gestures for comparison with state-of-the-art methods. In ablation studies, we focus on whole body gesture generation.Evaluation Metrics. We use multiple metrics to rigorously assess the quality and diversity of the generated gestures. The Fréchet Gesture Distance (FGD) [49] measures the Fréchet distance between the feature distributions of real and synthesized gestures, similar to the FID [13] used in image generation. The gesture feature extractor, trained unsupervisedly with a reconstruction loss using L1 loss, serves as the basis for this comparison.To quantify gesture diversity, we use the Diversity Score [21] and L1 Diversity Score [23] . The Diversity Score measures the average feature distance by randomly selecting 500 features and computing the average L1 distance between them.We also incorporate Semantic-Relevant Gesture Recall (SRGR) [27] and BeatAlign [24] to evaluate the semantic relevance and synchrony of the generated gestures. SRGR, an evolution of the Probability of Correct Keypoint (PCK), measures the semantic relevance of gestures to the accompanying speech. BeatAlign assesses the temporal alignment between audio and gesture beats using Chamfer Distance, providing insight into the rhythmic harmony of the generated gestures with the spoken content.Implementation Details. We downsample motion data from 120 Hz to 30 Hz and segment it into 300-frame clips (10 seconds each). Audio data is downsampled to 16 kHz from a higher sampling rate. We compute a comprehensive set of audio features, including MFCCs, Mel spectrogram features, prosodic features, and pitch onset points (onsets). These features are concatenated with those extracted by the pretrained WavLM Large model [6] to form a rich audio feature set 𝑓 𝑎 . For text data, we employ the pretrained fastText model [30] to extract word vectors from the speech transcripts, which are then processed through a linear layer to obtain the text feature 𝑓 𝑡𝑒𝑥𝑡 . Personal style 𝑠 and emotion 𝑒 are encoded as one-hot vectors and transformed into corresponding features 𝑓 𝑠 and 𝑓 𝑒 via linear layers. The diffusion process is set to 1000 steps. We train each model for 40,000 steps with a batch size of 400, using the AdamW optimizer with a learning rate of 3 × 10 -5 . All experiments are conducted on a single NVIDIA H800 GPU, ensuring reproducibility on standard hardware configurations.We compare our proposed model with state-of-the-art gesture generation methods, including DiffuseStyleGesture+ (DSG+) [44] , CaMN [27] , and MDM [38] . We retrain these methods on our partitioned dataset to ensure a fair comparison. CaMN is retrained with audio, text, emotion, and speaker identity as conditions, while DSG+ employs seed gestures, audio, and speaker identity. MDM is conditioned solely on audio features. Our evaluation includes both whole-body and upper-body gesture generation.The results, summarized in Table 1 , demonstrate that our Mam-baGesture framework achieves the lowest FGD Score for both upper body and whole body gesture generation, indicating a high similarity between the generated gestures and the real data distribution. Specifically, our whole body gesture generation records an FGD Score of 22.11, a significant improvement over MDM's 106.56 and DSG+'s 103.15. Additionally, our method excels in Diversity Score and L1 Diversity Score, with upper-body gestures scoring 374.08 and 875.06, respectively, and whole-body gestures scoring 434.94 and 1128.79, respectively. These scores underscore the enhanced diversity of our generated gestures. Notably, diffusion-based methods such as MDM and DSG+ exhibit better diversity than CaMN, which relies on autoencoders for gesture generation. Although our method does not achieve the highest SRGR Scores, it remains competitive with CaMN. Our MambaGesture excels in BeatAlign for both upper and whole-body gestures, reflecting a more synchronized audio-gesture alignment.User Study. We conduct a user study to subjectively evaluate our proposed method against state-of-the-art methods. Fifteen participants were asked to evaluate 30 gesture samples, each containing gestures generated by four different methods using the same corresponding audio. They were then asked to select the best gesture for each of the following criteria: motion naturalness, smoothness, diversity, and semantic preservation. The preferred method for each criterion was determined based on the number of selections received, and the results are presented as percentages in Table 2 .Visualization Results. Figure 3 visualizes the experimental results for the speech transcript "... when you have to work Monday through Friday the whole week, you are very tired ...", a sentence that should elicit rich body movements. Visual comparison shows that gestures generated by CaMN and DSG+ exhibit limited motion, while our approach demonstrates rich motion, highlighting its effectiveness.Both numerical and visual results corroborate that our method generates realistic and diverse co-speech gestures, advancing the state-of-the-art in the field. To rigorously evaluate the contributions of our proposed method's components, we conduct a series of ablation studies, systematically isolating and analyzing the impact of each module.Effectiveness of Proposed Components. We establish Diffus-eStyleGesture+ (DSG+) as our baseline model and incrementally integrate our novel components: SEA, SEAD and the MambaAttn denoiser. The results, detailed in Table 3 , show that replacing DSG+'s transformer encoder with our MambaAttn denoiser significantly reduces the Fréchet Gesture Distance (FGD) Score, indicating a closer match to the ground truth gestures. This is accompanied by substantial improvements in both the Diversity Score and the L1 Diversity Score. Although the Semantic-Relevant Gesture Recall (SRGR) Score sees a marginal decrease, the BeatAlign Score improves, underscoring the MambaAttn block's robust capability for gesture generation. The integration of our SEA module, which introduces emotion as a novel condition, leads to an additional reduction in the FGD Score, with other metrics showing slight variations. The culmination of our fusion approach, the SEAD module, which disentangles style and emotion from audio features and enhances them through cross-attention, further decreases the FGD Score and notably increases the Diversity Score and L1 Diversity Score. These results robustly validate the efficacy of our proposed method.Optimal Number of Layers in MambaAttn Denoiser. An additional ablation study examines the optimal number of layers in the MambaAttn denoiser. Testing configurations of 1, 2, 4, 8, and 12 layers, we find that an 8-layer MambaAttn denoiser yields the best performance across key evaluation metrics, including the FGD Score, Diversity Score, and L1 Diversity Score. Notably, increasing the number of layers to 12 does not confer additional benefits and instead leads to decreased performance. Thus, we select an 8-layer configuration for the MambaAttn denoiser in our final model, as demonstrated by the upper part of Table 4 . This finding underscores the importance of balancing model complexity with performance, as overly complex models may suffer from diminishing returns or even performance degradation.Design Choices for MambaAttn Block. Further experimentation is conducted to refine the architectural design of the Mam-baAttn block. Initially, we consider incorporating a convolutional layer to capture local information, with self-attention to capture global information, while relying on Mamba for sequential modeling. Our experiments with various combinations of these modules, as shown in the lower part of Table 4 , indicate that the inclusion of a convolutional layer does not enhance performance. Removing either the self-attention or Mamba components from the block resulted in a significant decrease in the FGD Score, with the removal of Mamba leading to a more pronounced drop in Diversity Score and L1 Diversity Score. Especially when we remove self-attention, with only Mamba and layer norms in our blocks, it can be seen that the Mamba-only architecture also works well for generating realistic and diverse gestures. This suggests that Mamba's sequential modeling ability is crucial for generating diverse gestures. The experiments further confirm this observation.Feature Fusion Module Designs. Lastly, we evaluate various designs for the feature fusion module. The results are presented 5 . Starting with a simplified fusion approach that directly concatenates audio, text, and style features (our SA fusion module), we observe a substantial decrease in the FGD Score, albeit with a decline in diversity metrics, as this method forgoes the baseline's feature fusion strategy. By adding the emotion modality, we create our SEA module, which further improves the FGD Score. The subsequent disentanglement in our SEAD-basic module results in improvements across the FGD Score, Diversity Score, and L1 Diversity Score. The final enhancement with the fused style and emotion feature 𝑓 ′ 𝑠𝑒 in our SEAD module significantly increases both the Diversity Score and L1 Diversity Score.The detailed results of these ablation studies demonstrate the incremental benefits of each proposed component in our gesture generation framework, providing clear evidence of their individual and collective impact on the model's performance. 
conclusion: 
 This paper introduces MambaGesture, a novel framework for cospeech gesture generation that leverages the state space model Mamba and a disentangled multi-modality fusion technique. Our approach integrates the MambaAttn block, which combines Mamba's strengths in sequential data processing with the contextual understanding provided by attention mechanisms. Additionally, the SEAD module effectively disentangles style and emotion from audio features, enabling the generation of more realistic and expressive gestures. Comprehensive experiments on the BEAT dataset demonstrate that MambaGesture outperforms state-of-the-art methods across multiple metrics, validating the effectiveness of the SEAD module and MambaAttn block. Future work will address current limitations, such as the slow synthesis speed, and may explore integrating facial expressions to synthesize a complete avatar. 
 
 Appendix:  


Decision: The paper introduces MambaGesture, a novel framework for co-speech gesture generation that integrates MambaAttn, a Mamba-based attention block, with SEAD, a multi-modality feature fusion module. MambaAttn enhances temporal coherence using sequential data processing and attention mechanisms, while SEAD combines audio, text, style, and emotion modalities through disentanglement, promising more realistic and diverse gestures. Evaluated on the BEAT dataset, the approach achieves state-of-the-art performance in Fréchet Gesture Distance (FGD), diversity scores, and beat alignment. While the paper addresses concerns about novelty, it lacks clarity in presentation and did not provide code during review, promising future release. Objective scores indicate strong semantic relations in generated gestures, validated by a user study assessing motion quality, diversity, and semantic preservation. The authors' commitment to open-sourcing their code enhances transparency, though concerns persist regarding baseline model replication accuracy.

Reviews: review: paper_summary: paper_summary: This paper presents "MambaGesture", a novel framework aimed at improving the generation of co-speech gestures using a combination of Mamba-based attention blocks and a multi-modality feature fusion module. The proposed method leverages state space models for enhanced temporal coherence in gesture generation and introduces an innovative feature fusion strategy that incorporates audio, text, style, and emotion.
strength_weakness: strength_weakness: The paper is well-written.

The introduction of the SEA (Style and Emotion Aware) and SEAD (Style and Emotion Aware Disentangled) feature fusion modules is interesting.
limitations: limitations: 1. While the integration of the Mamba model is highlighted as a novel contribution, it can be seen as a straightforward replacement of traditional backbones used in similar tasks. This hinders the novelty of the proposed method.

2. The clarity of Table 2 in the paper is questionable. It seems to suggest that the proposed method incrementally adds different components starting from a baseline model BasicDSG+. However, the original BasicDSG+ framework described in the original paper is quite different from the proposed one. Even to verify the effectiveness of Mamba, it is suggested to compare it with a baseline that only differs from the proposed method in the SSM block.

3. It seems that the main improvement is from adding the Mamba, however, since the advantage of Mamba over the Transformer is still under debate (2024), why such a significant improvement is achieved?
paper_summary: paper_summary: The paper introduces MambaGesture, a novel framework integrating MambaAttn with a multi-modality feature fusion module SEAD instead of diffusion models. This is to address the limitations of current approaches in co-speech gesture generation in generating less dynamic and contextually varied gesture by considering a wide range of modalities and analyzing their interactions.
strength_weakness: strength_weakness: - The work, if motivated well, could contribute to contextually relevant co-speech gesture generation. 
- The experiments section is well panned and the results presented are  interesting
limitations: limitations: - The problem lacks novelty. Introducing the Mamba-based attention itself is not a sufficient contribution. How it achieves diversity and why existing methods are insufficient to address diversity in gesture generation is to be mentioned. I suggest the authors to better motivate the problem and explain why it is relevant to address this.
-Further, there are multiple works which used multi-modal information apart from audio in co-speech gesture generation. 
The argument "Previous research has predominantly focused on audio as the primary input, often neglecting the depth of information available from other modalities." is no longer valid with the existing literature on multimodal co-speech gesture generation. In this scenario, how will you compare your approach with them? Some references are:
 1) Jingyu Wu, Shi Chen, Shuyu Gan, Weijun Li, Changyuan Yang, and Lingyun Sun. 2023. Cultural Self-Adaptive Multimodal Gesture Generation Based on Multiple Culture Gesture Dataset. In Proceedings of the 31st ACM International Conference on Multimedia (MM '23). Association for Computing Machinery, New York, NY, USA, 3538–3549. https://doi.org/10.1145/3581783.3611705 
2)Mughal, M. H., Dabral, R., Habibie, I., Donatelli, L., Habermann, M., & Theobalt, C. (2024). ConvoFusion: Multi-Modal Conversational Diffusion for Co-Speech Gesture Synthesis. arXiv preprint arXiv:2403.17936.
3) Ghorbani, S., Ferstl, Y., Holden, D., Troje, N. F., & Carbonneau, M. A. (2023, February). ZeroEGGS: Zero‐shot Example‐based Gesture Generation from Speech. In Computer Graphics Forum (Vol. 42, No. 1, pp. 206-216).
-It would be better to add a subsection on multimodal co-gesture generation
-It will be good if user evaluation is included along with the quantitative analysis.
paper_summary: paper_summary: This paper proposes to integrate Mamba architecture into co-speech gestures task. The model combines audio, text, style, and emotion modalities as same as previous method to generate gestures, evaluated on the BEAT dataset, the objective results show a improment in a clear margin.
strength_weakness: strength_weakness: 1. sharing the result of integrating Mamba architecture in the context of gesture generation is valuable for community.
2. the desgin of experiments, inputs and output of the pipeline is simple and clear (possitive).
3. objective results are good.
limitations: limitations: As a new baseline paper for using Mamba architecture, I toward to give the positive if the authour solve the following concerns:

1. videos results seems not good. why the gestures seems not continues? like the results from some VQVAEs?
2. will author provide code finally? as a new baseline if the author do not include the codes for implementation, the contribution is limited.
paper_summary: paper_summary: "MambaGesture: Enhancing Co-Speech Gesture Generation with Mamba and Disentangled Multi-Modality Fusion" introduces a new framework for generating human gestures synchronized with speech. This framework, MambaGesture, incorporates a Mamba-based attention block (MambaAttn) and a multi-modality feature fusion module (SEAD) to enhance the realism and diversity of co-speech gestures. By leveraging the strengths of Mamba's sequential data processing and the contextual richness of attention mechanisms, the authors propose a model that significantly improves the temporal coherence and contextual relevance of generated gestures.

* MambaAttn Block: Integration of the Mamba model with attention mechanisms to enhance temporal coherence and sequence modeling for gesture generation.
* SEAD Module: Introduction of a Style and Emotion Aware Disentangled feature fusion module that effectively integrates audio, text, style, and emotion modalities to produce realistic and diverse gestures.
* Improved Metrics: Demonstration of state-of-the-art performance on the BEAT dataset, with significant improvements in Fréchet Gesture Distance (FGD), diversity scores, and beat alignment.
* Ablation Studies: Extensive ablation studies that validate the efficacy of each component within the proposed framework, highlighting the contributions of MambaAttn and SEAD modules.
strength_weakness: strength_weakness: * Innovative Integration: The combination of Mamba and attention mechanisms in the MambaAttn block provides a robust solution for temporal coherence and sequence modeling in gesture generation.
* Comprehensive Multi-Modality Fusion: The SEAD module's ability to disentangle and integrate multiple modalities, including audio, text, style, and emotion, results in more realistic and contextually appropriate gestures.
* Rigorous Evaluation: The extensive experimental evaluation on the BEAT dataset and detailed ablation studies provide strong evidence for the proposed framework's effectiveness.
* Enhanced Performance: The proposed framework outperforms existing methods on key metrics, demonstrating significant improvements in gesture realism and diversity.
limitations: limitations: * Lack of User Study: The paper does not include a user study to evaluate the perceptual quality of the generated gestures. The visual results do not clearly show a distinction from the baseline models, which is a significant omission for a gesture generation task.
* Lack of Experimental Evidence for SEAD: While the SEAD module is claimed to capture emotion and style, there is insufficient experimental evidence to support this claim. The paper lacks specific experiments that demonstrate how SEAD captures these aspects effectively.
* Baseline Implementation Issues: The supplementary material suggests that the baseline model, DiffuseStyleGesture, may not have been properly implemented or optimized, which raises concerns about the validity of the comparative results.
* Objective Metrics Focus: The reliance on objective metrics like FGD, Diversity Score, and BeatAlign, while showing performance improvements, may not fully capture the perceptual quality and effectiveness of the generated gestures. These metrics are not always indicative of real-world performance in generative tasks.
* Code and Writing Similarities: The code and writing style closely resemble those of previous works like MambaTalk and DiffuseStyleGesture, which could indicate that the proposed method is more of an incremental modification rather than a novel contribution.
* Perceptual Quality Concerns: The supplementary videos indicate that the generated gestures do not significantly differ from the baseline models, suggesting that the improvements in objective metrics do not translate to noticeable visual enhancements.