MambaTalk: Efficient Holistic Gesture Synthesis with Selective State Space Models

Authors: zunnan  xu
yukang  lin
haonan  han
sicheng  yang
ronghui  li
yachao  zhang
xiu  li
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-13 14:51:43

Abstract: Gesture synthesis is a vital realm of human-computer interaction, with wide-ranging applications across various fields like film, robotics, and virtual reality. 
Recent advancements have utilized the diffusion model to improve gesture synthesis. 
However, the high computational complexity of these techniques limits the application in reality. 
In this study, we explore the potential of state space models (SSMs).
Direct application of SSMs in gesture synthesis encounters difficulties, which stem primarily from the diverse movement dynamics of various body parts. 
The generated gestures may also exhibit unnatural jittering issues.
To address these, we implement a two-stage modeling strategy with discrete motion priors to enhance the quality of gestures.
Built upon the selective scan mechanism, we introduce MambaTalk, which integrates hybrid fusion modules, local and global scans to refine latent space representations.
Subjective and objective experiments demonstrate that our method surpasses the performance of state-of-the-art models. Our project is publicly available at~\url{https://kkakkkka.github.io/MambaTalk/}.

Main paper: 
 introduction: 
 Gesture synthesis is a critical area of research in human-computer interaction (HCI), which has very broad application prospects, such as film, robotics, virtual reality, and digital human development [24] . The task is challenging due to the variable correlation between speech and gestures, as the same spoken content can elicit markedly different gestures among speakers. Meanwhile, the generated gestures should synchronize with the speaker's rhythm, emotional cues, and intentions [31, 1, 8, 42] .Recent works in co-speech gesture generation have shown great progress [11, 46, 67, 66, 2, 62] . By introducing new datasets [71] and more modalities [70, 33] , previous work achieved end-to-end gesture generation based on RNN-based models [11, 46] . With the success of transformer in nature language processing [58] and video sequence modeling [26, 27, 75] , recent works [8, 43, 55] leverage the power of attention mechanism to generate more expressive gestures that better synchronize with speech. By further combining emotional and style related features, EMoG [69] achieve better quality gesture generation. With the development in human recognition model [37] , EMAGE [32] proposes a masked audio-gesture modeling strategy to enhance unified holistic gesture synthesis. Recently, with the development of diffusion model in generative tasks [39, 20, 40] , the latest works [74, 2, 62, 7, 65] have applied the diffusion model to gesture synthesis, significantly improving the diversity of generated gesture. DiffuseStyleGesture [63] presents a diffusion model-based approach for generating diverse co-speech gestures by incorporating cross-local attention and self-attention mechanisms, and utilizing classifier-free guidance for style control. DiffuseStyleGesture+ [66] further considers the text modality as an additional input and utilizes channel concatenation to merge the text feature with the audio feature. Deichler et al. [9] also incorporates the text modality as an additional input and employs contrastive learning to enhance the features. However, the exploration of generation for co-speech gesture sequences with low latency remains relatively uncharted, constraining its application in dynamic, interactive environments. RNN-based models often struggle with the longterm forgetting issue [54, 29] , which impairs their ability to generate long sequences of gestures effectively. Additionally, these models may produce gestures that lack variability, tending towards an average representation [36] . Transformer-based models depend heavily on subtle positional encoding to capture the order of input elements [44, 50, 73] . Meanwhile, their computational complexity, which grows quadratically with the length of the input sequence, poses a challenge for generating long sequences of gestures. For the diffusion-based model, the intricate sampling strategy and iterative process lead to high computational expenses [48] , which hinder their broad adoption in gesture generation scenarios. In the first stage, we construct discrete motion spaces to learn specific motion codes. In the second stage, we develop a speech-driven model of the latent space using selective scanning mechanisms.State space models (SSMs) have recently shown significant potential in addressing challenges related to modeling sequences with low latency [12] . Inspired by continuous state space models from control systems and enhanced by HiPPO initialization [13] , SSMs [16] show promise in addressing long-term forgetting issue. These advancements have been integrated into large-scale representation models [38, 41] . Some pioneering works have applied SSMs for tasks like language understanding [38, 41] , content-based reasoning [12] , and visual recognition [35, 57] . In our work, we further explore the potential of SSMs in co-speech gesture synthesis. We observe that directly applying the selective scan mechanism from Mamba [12] to gesture generation as a sequence modeling model would result in jittery outputs. To refine the generated gestures, we propose a two-stage modeling strategy. In the first training stage, we enhance the discrete motion priors derived from VQVAEs [53] by integrating velocity and acceleration losses. In the second stage, by utilizing motion priors from VQVAEs, we introduce individual learnable queries for different body parts, thereby alleviating the jittering issue. Meanwhile, considering that the direct application of Mamba encounters the challenge of limb movements across different body parts tending to average out, we propose a hybrid scanning approach in the second stage to enhance the motion representation in the latent space. Specifically, we refine the design of spatial and temporal modeling within latent spaces by introducing a local-to-global modeling strategy and integrating attention mechanisms along with a selective scanning approach into the framework's design. Considering the significant differences in deformation and movement patterns among different body parts [32] , we propose local and global scan modules for refining the latent space representations of the movements across various body parts. These approaches enable dynamic interaction and iterative refinement of different body parts while maintaining low latency, leading to more diverse and rhythmic gestures. Our contributions can be summarized as below:• We are the first to explore the potential of the selective scan mechanism for co-speech gesture synthesis, achieving a diverse and realistic range of facial and gesture animations. • We introduce MambaTalk, an innovative framework that integrates hybrid scanning modules (e.g., local and global scan). The integration enhances the latent space representations for gesture synthesis, thereby refining the distinct movement patterns across various body parts. • Extensive experiments and analyses demonstrate the effectiveness of our proposed method. 
co-speech gesture generation: 
 Co-speech gesture generation aims to automatically generate gestures based on speech input. Existing approaches can be broadly categorized into three groups: (i) Rule-based methods: These methods rely on pre-defined rules and gesture libraries to generate gestures based on speech features [23, 56] . While offering interpretable results, they require significant manual effort in creating gesture datasets and defining rules. (ii) Statistical models: These approaches leverage data-driven techniques to learn mapping rules between speech and gestures, often employing pre-defined gesture units [22, 25] . While overcoming the limitations of manual rule creation, these methods still rely on handcrafted features. (iii) Deep learning methods: Recent advancements in deep learning have enabled neural networks to capture the complex relationship between speech and gestures directly from raw multimodal data [70, 33, 68, 32] . This progress has established deep learning approaches, particularly recurrent neural networks (RNNs) [70, 33, 61] , transformers [4, 45] , and diffusion models [2, 74, 51, 72, 21, 6] , as the prevailing paradigm for co-speech gesture generation. However, each of these models suffers from certain limitations that hinder their performance. RNNs inherently process sequences in a serial manner, where each timestep's computation depends on the output of the previous timestep. This limits their ability to efficiently handle long sequences and introduces cumulative latency. Meanwhile, RNNs lack inherent parallelism, further restricting their potential for high-speed computation. Transformers consider all positions within a sequence at every timestep, resulting in high computational complexity, especially for long sequences. While diffusion models significantly enhance the diversity of generated outputs, the sampling process is computationally expensive. To overcome these limitations, our method investigates the capacity of selective state space models in the field of gesture synthesis. To the best of our knowledge, we are the first to apply selective state space models to the task of gesture generation.State Space Models (SSMs) are a novel class of models recently integrated into deep learning for state space transformation [15, 10] . As foundational models evolve, various subquadratic-time architectures have emerged, including linear attention, gated convolution, recurrent models, and structured state space models (SSMs), aimed at mitigating the computational inefficiencies of Transformers when dealing with lengthy sequences. However, these advancements have yet to match the performance of attention mechanisms in critical modalities like language processing. SSMs draw inspiration from continuous state space models in control systems and, when combined with HiPPO initialization [13] , as seen in LSSL [16] , show promise in tackling long-range dependency issues. However, the computational and memory demands of the state representation render LSSL impractical for real-world use. To address this, S4 [15] suggests normalizing the parameters into a diagonal structure. This has led to the emergence of various structured SSMs with diverse configurations, such as complex-diagonal structures [17, 14] , multiple-input multiple-output (MIMO) support [49] , diagonal-plus-low-rank decomposition [19] , and selection mechanisms [12] . These models have been incorporated into large-scale representation models [38, 41] .These models primarily focus on the application of SSMs to long-range and sequential data like language and speech, for tasks such as language understanding [38, 41] , content-based reasoning [12] , and pixel-level 1-D image classification [15] . Recently, some pioneering work [35, 57, 59 ] have explored their application in visual recognition. We further demonstrate that by incorporating the selective scan mechanism from mamba [12] and the discrete motion priors from VQVAEs [53] , our proposed MambaTalk is capable of matching the performance of existing popular holistic gesture synthesis models, highlighting the potential of MambaTalk as a powerful gesture synthesis model.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: Yes, we discuss the limitations in section A.4 in the Appendix part.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
method: 
 We aim to synthesize sequential 3D co-speech gestures from speech signals (e.g., audio and text) using selective state space models. However, simply applying such a model to gesture synthesis leads to severe gesture jittering issues. We also found that maintaining performance is challenging due to the significant variations in movement patterns exhibited by different body parts. To overcome these challenges, we suggest modeling the gesture space using the acquired discrete motion patterns. Subsequently, we propose to develop speech-conditioned selective state space models within this framework. This approach is designed to enhance the model's robustness against uncertainties that arise from cross-modal discrepancies. As shown in Figure 2 , our framework consists of two stages: (i) modeling the discrete gestures and facial motion spaces ( §3.2) and (ii) learning speech-conditioned selective state space models ( §3.3) to generate 3D co-speech gestures. We propose a two-stage method for co-speech gesture generation. We first train multiple VQ-VAEs for face and different parts of body reconstruction. This step learns discrete motion priors through multiple codebooks. In the second stage, we train a speech-driven gesture generation model in the latent motion space with local and global scan modules.Selective State Spaces Model. In our approach, we adopt the Selective State Spaces model (Mamba [12] ) that incorporates a selection mechanism and a scan module (S6). This model is designed to make sequence modeling, as it dynamically selects salient input segments for prediction, thereby enhancing its focus on pertinent information and improving overall performance. Unlike the traditional S4 model, which uses time-invariant matrices A, B, C, and scalar ∆, Mamba introduces selection mechanism that allows for the learning of these parameters from the input data using fully-connected layers. This adaptability enables model to better generalize and perform complex modeling tasks. Mamba operates by defining the state space with structured matrices that introduce specific constraints on the parameters, facilitating efficient computation and data storage. For each batch and each dimension, the model processes the input x t , hidden state h t , and output y t at each time step t. We have h 0 = B0 x 0 when t = 0. When t > 0, the model's formulation is as follows:EQUATIONwhere Āt , Bt , and C t are matrices and vectors that are updated at each time step, allowing the model to adapt to the temporal dynamics of the input sequence. With discretization, let ∆ denote the sampling interval, exp(∆A) denote the matrix exponential, the transformation of the system's state over one time step can be represented as follows:EQUATIONwhere (∆A) -1 denotes the inverse of matrix ∆A, I denotes the identity matrix. The scan module within Mamba is designed to capture temporal patterns and dependencies across multiple time steps by applying a set of trainable parameters or operations to each segment of the input sequence. In our framework, Mamba serves as a sequence modeling tool for decoding gesture actions across different parts of the body. By modifying the decoder's input and the range of features, we leverage Mamba to model local body features and global characteristics separately. These operations are learned during training and help the model to process sequential data.To ensure visual realism in motion animations from speech signals, we learn extra motion priors to depict accurate movements and natural expressions. Building on this concept, we propose a method to represent the gesture motion space using multiple discrete codebooks.Motion Quantization. Considering the substantial variations in deformation magnitude and periodicity among various body parts, our approach involves learning multiple codebooks tailored for the reconstruction of distinct body parts. For illustrative purposes, we detail the formulation of a single codebook. Denotes C as the dimensionality of each latent vector, N as the number of vectors in the codebook, for the codebook Z = z k ∈ R C N k=1 , we employ a set of allocated items {z k } k∈S to represent the holistic gesture motion M t . Here, S represents the chosen index sets. The element-wise quantization function Q(•) maps each item ẑt in Ẑ to its closest match z k in the codebook Z:EQUATIONwhere the codebook entries act as the foundational motion elements within the discrete motion space. To establish this, we follow [64, 32] to pre-train a CNN-based Vector Quantized-Variational Autoencoder (VQ-VAE), which comprises an encoder E, a decoder D, and a context-rich codebook Z. This is done through the self-reconstruction of gesture motions.The sequence of motions M 1:T is initially transformed into a temporal feature representation Ẑ = E(M 1:T ) ∈ R T ′ ×H×C , where H represents the count of gesture components, T ′ indicates the quantity of temporal units encoded (with P = T T ′ frames per unit). Subsequently, we derive the quantized motion sequence Z q ∈ R T ′ ×H×C by quantization function Q(•). This function Q maps each element in Ẑ to its closest corresponding entry within the codebook Z:EQUATION)Training objectives. For the training of the quantized autoencoder, we employ motion-level losses to mitigate the jittering issue of generated gestures, along with two intermediate losses at the code level:EQUATIONwhere M ′ and M ′′ means the velocity and acceleration of motion, sg(•) denotes a stop-gradient operation, L rec are Geodesic [52] loss and the last two terms are designed to refine the codebook entries. For facial motions, we utilize MSE loss for both velocity (L vel ) and acceleration (L acc ) loss.For body motions, we use L1 loss as L vel and L acc . Additionally, for the foot contact loss, we employ MSE loss as the loss function. These terms work by minimizing the distance between the codebook Z and the embedded features Ẑ. Given that the quantization function (Equation 3) is non-differentiable, we utilize the straight-through gradient estimator [53] to propagate the gradients.Overall Framework. Utilizing the acquired discrete motion prior, we establish a cross-modal mapping from speech inputs to target motion codes, enabling the generation of realistic co-speech gesture motions. In our approach to speech-driven gesture synthesis, we utilize audio sequences A = {a 1 , . . . , a N } and text sequences T = {t 1 , . . . , t N } as inputs to guide the generation of cospeech gestures G = {g 1 , . . . , g N }. Here, N signifies the total frame count, andg i ∈ R 55×6+100+4+3denotes 55 pose joints in Rot6D, R 100 FLAME parameters, R 4 foot contact labels, R 3 global translations for the i-th frame. The gesture synthesis model, comprising audio encoders E A and text encoders E T and multiple selective state space models D B for different parts of the body, is trained on the discrete motion space, conditioned on the speech, as shown in Figure 2 .Speech Feature Extraction. For audio feature extraction, two CNN-based audio feature extraction networks are employed to respectively extract amplitude and onset features. Specifically, we integrate amplitude and onset features along the channel dimension to obtain audio features f A = {f a 1 , ..., f a N }. For processing speech input words, we employ pre-trained FastText [5] to obtain word embeddings, which are then refined by linear projections to produce text features f T = {f t 1 , ..., f t N }. We further fuses features from the input modalities (e.g., audio and text features). The speaker embeddings s id are first combined with audio and text features through additive operation. By concatenating feature vectors along the channel dimension, we further apply linear transformations to determine the weight factors, and then integrating the features through an element-wise summation. The process can be formalized as:EQUATIONwhere σ denotes the softmax operation, ⊙ denotes the Hadamard product, and W T and W A represent linear mapping matrices used to adjust the dimensions of merged features. fT and fA denote the fused features.Local and Global Scans. Recognizing the diverse deformations and motion patterns in various body parts, we propose using multiple local scan modules to model the movements of different body parts (e.g., face, hand, upper and lower body) with fused multi-modal features from previous modules. To enhance the generalization of the model, we have incorporated additional learnable queries to foster the queries' ability to learn motion patterns. As shown in Figure 2 , the queries (Q) are integrated with the input's multi-modal features through a multi-head cross attention mechanism (F MHCA ). This allows the queries to learn the most relevant information from the speech input. The process can be formally defined as follows:EQUATIONwhere f face denotes the feature of facial motion. This formulation can be applied similarly to obtain f hand , f upper , and f lower by substituting the corresponding query vectors Q hand , Q upper , and Q lower respectively. Utilizing the extracted perceptual features from various body parts, we proceed to employ Mamba to extract temporal features from the sequence, which can be formalized as belows:EQUATIONwhere F face corresponds to the temporal features of facial motion. The same approach is applied to generate temporal features for the hand, upper body, and lower body by inputting their respective perceptual features f hand , f upper , and f lower into the corresponding Mamba modules.By acquiring the distinctive features of various body parts, we improve the perception of motion patterns among them using a global scan module. Initially, we combine the features along the sequence dimension. Subsequently, by utilizing self-attention mechanism(F MHSA ), we model the global information across different sequence tokens. These refined features are fed into Mamba to extract temporal perceptual information, which can be fomulized as below:EQUATIONwhere [] denotes the concatenation operation of F f ace , F hand , F upper and F lower in the dimension of the sequence. The extracted latent features are then fed into their respective VQ-Decoders to produce the final motion predictions.Training Objectives. The model's training objectives are composed by a composite loss function that harmonizes reconstruction and cross-entropy losses. This design aims to augment the accuracy of motion generation, encompassing the face, hands, upper and lower body. The latent reconstruction loss, represented by L reclatent , is quantified using the Mean Squared Loss (MSELoss). Here, z i corresponds to the true latent vectors, while ẑi are the vectors reconstructed by the model. The latent reconstruction loss is expressed as:EQUATION)where N denotes the number of frames. Concurrently, to encourage diversity in the generated motions, we optimize the cross-entropy loss for latent code class classification L cls . Specifically, we employ the Negative Log Likelihood Loss (NLLLoss), where y i represents the true class labels for each sample, and ŷi denotes the model's predicted class labels. This loss is calculated as the negative sum of the logarithm of the predicted probabilities for the correct classes:EQUATIONwhere N signifies the total number of frames, C is the total number of classes, and y ic is a binary indicator of whether class c is the correct label for sample i. The total loss L is a weighted sum of the categorical and latent reconstruction losses, with α and β serving as balance hyper-parameters:EQUATIONwhere α = 1 and β = 3 for hands, upper and lower body motion. For facial motion, we set α = 0 and β = 3. By optimizing the total loss, the model is trained to generate diverse gesture results.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Justification: Yes, we do.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
experiments setting: 
 We train and evaluate on the BEATX-standard dataset proposed by [32] . BEATX-standard contains 60 hours of data with high finger quality for 25 speakers (12 female and 13 male). The dataset comprises 1762 sequences, each with an average duration of 65.66 seconds. Each sequence includes a response to a daily inquiry. We split datasets into 85%/7.5%/7.5% for the train/val/test set.We utilize the Adam optimizer with a learning rate of 2.5 × 10 -4 . To maintain stability, we apply gradient norm clipping at a value of 0.99. In the construction of the VQVAEs, we employ a uniform initialization for the codebook, setting the codebook entries to feature lengths of 512 and establishing the codebook size at 256. The numerical distribution range for the codebook initialization is defined as [-1/codebook_size, 1/codebook_size). The codebook is solely updated during the first stage, and in the second stage of training for the speech-to-gesture mapping, the codebook remains frozen. The VQVAEs are trained for 200 epochs, with a learning rate of 2.5 × 10 -4 for the first 195 epochs, which is then reduced to 2.5 × 10 -5 for the final 5 epochs. During the second stage, the model is trained for 300 epochs. All experiments are conducted using one NVIDIA A100 GPU.To evaluate the realism of body gestures, we employ Fréchet Gesture Distance (FGD) [70] to measure the proximity of the distribution between the ground truth and generated gestures. Subsequently, Diversity [28] is quantified by computing the average L1 distance across multiple gesture clips. The synchronization between speech and motion is achieved using Beat Constancy (BC) [30] . For facial motions, we assess positional accuracy by calculating the vertex Mean Squared Error (MSE) [60] .Additionally, the difference between the ground truth and the generated facial vertices is measured using the vertex L1 difference (LVD) [68] . More details about metrics and efficiency analysis are provided in the supplementary materials.As shown in Table 1 , our method attains the highest BC, with a significant increase of 14.3% when compared to the previously top-performing method, which underscores the superior capability of MambaTalk in discerning and correlating audio-motion beats. Furthermore, our method performs competitively with EMAGE in terms of FGD, and much better than others. This emphasizes the high quality and naturalness of our generated movements, showing the ability of MambaTalk to capture real motion dynamics. This also demonstrates the authenticity of our generated motions, affirming the successful capture of inherent motion characteristics. Some results are marked as "-" because these methods can not generate facial movements. Moreover, our method outperforms previous methods in terms of MSE and LVD, with substantial improvements of 36% and 14%, respectively. These two enhancements highlight the superior accuracy and fidelity of our method in capturing fine-grained details, affirming its efficacy in synthesizing realistic and authentic facial motions. We conducted a user study to assess the visual quality of the generated co-speech 3D gestures. For each method under comparison, we produced 10 gesture samples, which were then converted into video clips for evaluation by 39 participants. In each evaluation session, participants were presented with 20 seconds video clips generated by various models. They were instructed to assess the clips across the following dimensions: (i) naturalness, (ii) appropriateness, (iii) synchrony and (iv) smoothness. For naturalness, they evaluated the similarity of the generated gestures to those made by humans, paying attention to the authenticity and smoothness of the movements. In terms of appropriateness, they considered the alignment of the gestures with the spoken content, taking into account both the explicit meaning and the underlying semantics. For synchrony assessment, they examined the timing of the gestures in relation to the speech rhythm, audio, and facial expressions to ensure a harmonious and integrated performance. For smoothness, they assessed the gestures for any abrupt stops or unnatural jerks that might indicate a lack of fluidity in motion. We mainly compared two state-of-art methods with our proposed method (with and without VQVAE): CaMN [33] , EMAGE [32] , and the ground truth. As presented in Table 2 , our method's average scores are higher than previous methods.Visualization. As depicted in Figure 3 , our approach yields gestures that exhibit enhanced rhythmic alignment and a more natural appearance. For instance, when conveying "we were", our method instructs the subject to hold both hands in front of the chest, a nuanced detail absent in both CaMN and EMAGE's outcomes, where either one or both arms hang down. Additionally, when representing "no place to", our method aligns with the ground truth by extending both arms upwards, whereas CaMN and EMAGE have their arms tucked in next to the body. In the case of "up", our generated result raises the right arm in alignment with the semantics of movement. In the context of "moving around" where our left and right arm swings may differ from the ground truth, the overall movement remains consistent.Interestingly, for "sound of gunfire", a difficult semantic for the model to learn, our method still generates the result of the character's right hand clenched in a fist and the arm bent to indicate a tense situation. For the emotion of fear expressed by "is horrible", the result of our method is similar to the ground truth, with the character's hands hanging down and face facing downward, which is a visual representation of the psychological state of panic and fear. In addition, as illustrated in Figure 3 , our generated motions exhibit not only diverse characteristics, such as the range of motion and which hands to use, but also a high degree of consistency with the ground truth.Effect of VQVAEs. We confirm the significant role of the VQVAEs. As shown in Table 2 and Table 3 , the integration of VQVAEs is essential for the functionality of our approach, contributing to the generation of gestures that exhibit smoother transitions and a more human-like quality. As demonstrated in Table 3 , the removal of the VQVAEs ("-VQVAEs") from the model is also associated with performance decline, manifesting reduction in FGD, BC, Diversity, MSE and LVD.Effect of Local Scan. We validate the effectiveness of the local scan. The ablation study is divided into two segments: (i) multi head cross attention and (ii) Mamba models for different part of bodys. As shown in Table 3 , incorporating multi head cross attention enhances our method's capability to generate gestures with higher diversity and beat constancy. The incorporation of the Mamba from local scan generate gestures characterized by greater diversity and beat constancy. Concurrently, there is an observed improvement in the FGD of the generated gestures. Effect of Different Audio Encoder. To validate the effectiveness of the audio encoder, we replace the CNN-based audio encoder with pre-trained Whisper [47] and Vav2Vec2 [3] , as listed in Table 4 .Unlike CNN-based audio encoders that are randomly initialized and trained from scratch, when using Whisper and Wav2Vec2, we initialize the encoder using pre-trained weights and fix the parameters of the feature extractor. We observe a notable enhancement in facial generation when utilizing Whisper, however, the body generation results were subpar. In contrast, while Wav2Vec2 demonstrates some improvement in body generation, it results in a substantial decline in facial generation quality. 
conclusion: 
 In this study, we propose a framework to employ the state space models in gesture synthesis. To alleviate the problem of jitter in gesture synthesis, we have implemented discrete motion priors, which enhance the effectiveness of the selective scan mechanism and lead to smoother results. We further incorporate the selective state space models with attention mechanisms to enhance the refinement of motion features in latent space. These modules capture the subtle movements and deformations of various body parts, thereby enhancing the overall quality of the generated gestures. By utilizing a linear time series modeling strategy with selective state space, our method achieves high-quality full body gesture generation with low latency. 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [NA]Justification: The work we perform makes no society impact. It is only an academic study.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: The paper poses no such risks. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: We properly cite the original owners of code, data and models. Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.• For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: Figure 4 presents the facial motion results generated by our method, showcasing the generation of facial expressions and movements with a high level of realism. Our method effectively synchronizes with the phonetic articulation of speech content, accurately reflecting the physical demands of pronunciation. For instance, when uttering "walking", "came" or "bus", our approach ensures that the mouth's movements, such as opening, correspond closely with the actual phonetic requirements. Other methods do not consistently achieve this level of accuracy in aligning with the phonetic and physical nuances of speech. Our method adeptly handles the subtleties of mouth closure and elongation required for sounds such as "in", closely aligning with the ground truth, whereas other approaches may exhibit inconsistencies in this regard. Moreover, in instances of silence, all methods, including ours, demonstrate a good capacity to learn and maintain the mouth's closed position, effectively reflecting the underlying patterns of speech and silence. Since CaMN does not specifically target the generation of facial movements, it results in a lack of variation in facial expressions throughout the process. As illustrated in Figure 5 , our methodology generates gestures that demonstrate improved rhythmic synchronization and a more lifelike appearance, effectively capturing the essence of the speaker's rhythmic patterns. For example, in the expression "actually" our method guides the individual to bring the hands inward in front of the chest, a subtle gesture not observed in the results produced by CaMN and EMAGE, where the arms are either hang limply at the sides or are splayed downward. Furthermore, in the depiction of "on the way back" our approach accurately reflects the ground truth by slightly bending down and raising one hand, while EMAGE cannot respond accurately to this and remains standing.Furthermore, our approach accurately captures the semantic essence of movements. For instance, in response to the cue "hug" our method generates an inward-circling motion of the arms, aligning perfectly with the ground truth, which is a nuanced semantic element that other methodologies neglect. Similarly, in scenarios such as "so small", the result of our method is similar to the ground truth, with the character's hand moving inward. This attention to detail ensures semantic consistency, which is lacking in other approaches where actions are not aligned with the intended meaning. We leverage the linear computational complexity of Mamba and the sequence compression capability of VQVAE within our framework, which helps in reducing computational complexity. Although there are some specialized acceleration solutions, faster solutions are necessary because when the model is integrated into the system, there is not only a delay in generating gestures, but also delays in other modules. Therefore, to evaluate the efficiency of our pipeline, we conducted a series of measurements, focusing on the runtime of individual components. We measured the runtime of various components on the NVIDIA A100 GPU in our method over three runs and presented the average results in Table 5 . We also compare our method's inference time with diffusion-based methods. The average computation time was determined based on the generation of a 31 second motion sequence, to showcase our model's low latency capabilities. The total inference time of our method is much slower than state-of-the-art diffusion-based method [63] . The result confirm that our pipeline is well-suited for applications requiring low latency gesture generation like interactive systems, where responsiveness is paramount. To evaluate the generalisable benefit of our method, we conduct experiments on a large-scale multimodal dataset known as BEAT (Body-Expression-Audio-Text) [33] . This dataset encompasses 76 hours of multimodal data collected from 30 speakers engaging in conversations across four different languages while expressing eight distinct emotions. The dataset includes conversational gestures, facial expressions, emotional cues, and semantic content, along with annotations for audio, text, and speaker identity. To facilitate a fair comparison, we follow CaMN [33] and employ approximately 16 hours of speech data from English speakers. Furthermore, we implement the conventional approach of partitioning the dataset into distinct training, validation, and testing subsets, ensuring consistency with the data partitioning scheme utilized in prior research to uphold the integrity of the comparison.To facilitate a fair comparison, we employ a total of N = 34 frame clips with a stride of 10 during the training process. The first four frames serve as seed poses, while the model is trained to generate the subsequent 30 poses, which collectively represent a duration of 2 seconds. Our models incorporate 47 joints from the BEAT dataset, comprising 38 hand joints and 9 body joints. As listed in Table 6 , our method demonstrates a significant improvement compared to the CaMN (baseline), which also validates the generalisable benefits of our approach. [70] 176.2 0.195 0.776 Audio2Gesture [28] 223.8 0.097 0.766 CaMN [33] 123.7 0.239 0.783 TalkShow [68] 91.00 -0.840 MambaTalk (ours) 51.3 0.256 0.852Currently, our approach to gesture synthesis involves using distinct modules to animate various body parts, which naturally introduces some latency. Developing a single, unified model capable of capturing the wide-ranging and intricate deformations and motion patterns characteristic of different body parts can be addressed in future research. This enhancement is anticipated to lower computational overhead and substantially reduce the processing time, thereby improving the real-time capabilities of the pipeline and ensuring a smoother and more responsive gesture generation system.Meanwhile, exploring more robust audio representations or combining various types of pre-trained audio encoders could significantly enhance the quality of gesture generation. Our findings indicate that certain encoders, such as Whisper, are particularly effective for modeling facial movements, while others, like Wav2Vec2, are better suited for modeling body movements. This approach will further improve the overall performance of the method.In addition, the issue of gesture diversity among speakers and across different cultures remains unaddressed. Addressing this gap is essential for improving the cross-cultural validity and expanding the applicability of gesture-based applications in diverse global contexts.The local scanning procedure is illustrated in Algorithm 26. We attain local modeling of various body segments by individually processing actions within distinct regions. The approach to global scanning parallels this methodology; however, the key distinction lies in the simultaneous processing of motion representations across multiple body parts. x ′ o : (B, M, E) ← SiLU(Conv1do(x)) 10: Bo : (B, M, N) ← Linear B o (x ′ o ) 11: Co : (B, M, N) ← Linear C o (x ′ o ) 12: /*: (B, M, D) ← Linear T (y ′ face + y ′ upperbody + y ′ lowerbody + y ′ hand ) + T l-1 26: Return: T lTo evaluate the realism of body gestures, we employ Fréchet Gesture Distance (FGD) [70] to measure how close the distribution between the ground truth and generated body gestures is.EQUATIONwhere µ r and Σ r denote the mean and covariance of the latent feature distribution z r for real human gestures g, while µ g and Σ g correspond to the mean and covariance of the latent feature distribution z g for the synthesized gestures ĝ. We employ an encoder based on a Skeleton CNN (SKCNN) and a Full CNN-based decoder, constituting our autoencoder's pretrained network. This network is trained on both the BEATX-Standard and BEATX-Additional datasets. The preference for SKCNN over a Full CNN encoder stems from its superior performance in capturing gesture features, evidenced by a reduced reconstruction MSE loss of 0.095, as opposed to 0.103.Subsequently, Diversity [28] is quantified by computing the average L1 distance across multiple body gesture clips. Higher Diversity signifies greater variance within the gesture clips. We compute the average L1 distance across various N motion clips using the following equation:EQUATIONwhere p t denotes the positions of joints in frame t. We assess diversity across the entire test dataset. Moreover, when calculating joint positions, translation is zeroed, indicating that L1 Diversity is exclusively concentrated on local motion dynamics.The synchronization between the speech and motion is conducted using Beat Constancy (BC) [30] . BC indicates a more precise synchronization between the rhythm of gestures and the audio's beat. We define the onset of speech as the audio's beat and identify the local minima of the upper body joints' velocity (excluding fingers) as the motion's beat. The synchronization between audio and gesture is determined using the following equation:EQUATIONwhere g and a represent the sets of gesture beats and audio beats, respectively.Turning focus to facial aspects, we gauge the positional accuracy through the calculation of vertex Mean Squared Error (MSE) [60] . This metric quantifies the average squared difference between the predicted facial landmarks and their corresponding ground truths, providing a clear indication of the facial model's accuracy:EQUATIONwhere n denotes the number of vertices, f i represents the ground truth position of the i-th vertex, fi denotes the predicted position of the i-th vertex. The sum is taken over all vertices to compute the average error.Additionally, the disparity between the ground truth and the generated facial vertices is measured by the vertex L1 difference (LVD) [68] , which measures the synchronization between speech and facial expression.EQUATIONwhere n denotes the number of vertices, f ′ i represents the ground truth speed of the i-th vertex. f ′ i denotes the speed of the i-th vertex in the generated facial expression. The sum is taken over all vertices to compute the average absolute difference.This section provides additional details about the user study for qualitative analysis. For our user study, we have randomly selected ten videos generated by different methods, each containing 20-second video clips. For each participant, we paid compensation that exceeded the local average hourly wage.The screenshot of our user study website is illustrated in the Figure 6 , which displays the template layout presented to the participants. In addition to the main trials, participants were also subjected to several catch trials. These trials involved displaying Ground Truth videos and videos with distorted motion. Participants who failed to score the GT videos higher and the distorted motion videos lower were considered unresponsive or inattentive and their data was not included in the final evaluation. Justification: Overall, the abstract and introduction provide a concise yet comprehensive summary of the paper's objectives, methods, and findings, accurately reflecting its contributions and scope.Guidelines:• The answer NA means that the and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 


Decision: The paper originally got BA, BR, R, and WA, and, after the rebuttal period, two reviewers raised the ratings, leading to final ratings of BA, WA, BR, and WA. 

Reviewers recognized the major contribution of this paper in applying state space models (SSMs) into co-speech gesture generations. However, reviewers raised several concerns, including unclear motivation of the work in applying SSMs and insufficient experiments as the paper was tested on a single dataset. Following the rebuttal and discussion period, most concerns were addressed, and two reviewers increased their ratings, resulting in three positive and one negative review.

The AC also supports accepting the paper, recognizing its contribution to applying SSMs for the co-speech gesture generation. The AC strongly suggests the authors to follow the reviewers’ feedback in the camera-ready version.

Reviews: review: paper_summary: paper_summary: This paper focuses on the problem of co-speech gesture generation, particularly aiming to address the challenges of jittery movements, long motion sequences, and holistic gesture generation (including both face and body movements). To tackle these issues, the authors present a new method that combines diffusion models with state space models and incorporates both local and global scans. The proposed method has been evaluated on the BeatX dataset.
strength_weakness: strength_weakness: Major Comment:

The main weakness of the paper is its limited evaluation. While the paper excels in generating facial, body, or holistic gestures, evaluating on only one dataset is substandard. To be competitive, the authors should consider additional standard benchmarks such as those used in the GENEA challenge [1]. There is also a long list of papers using diffusion models for co-speech gesture synthesis [1], none of which are mentioned in the paper. While I appreciate the authors' work and acknowledge the scarcity of datasets with both face and body motions, the proposed approach handles face and body separately. Therefore, at least body motion generation performance could be compared on existing datasets to demonstrate generalisable benefits.

Another weakness is the insufficient details regarding long-term motion generation, which is claimed as a key contribution. The paper lacks sufficient details on the length of the sequences considered in training and evaluation. Moreover, there is no comparison to support this claim.

Minor Comment:

The lower part of Figure 2 is helpful for understanding the workflow. However, including pseudo-code would be beneficial to increase the reproducibility of this method.

[1] https://genea-workshop.github.io/2023/challenge/
questions: questions: Why is only one dataset considered? Please explain the rationale behind evaluating the method on only one dataset. How are these results generalisable and competitive given this limitation? 

Generalizability and Competitiveness: How do you ensure that the results obtained from the BeatX dataset can be generalized to other scenarios? Discuss any potential limitations in terms of generalisability and how your method addresses them.

Training and Evaluation Sequence Lengths: Please elaborate on the length of the motion sequences considered during training and evaluation.

Impact on Co-Speech Gesture Generation Performance: Discuss how the length of motion sequences impacts the performance of co-speech gesture generation. What are the implications of using different sequence lengths, and how does your method handle long-term motion generation effectively? Including comparisons or experiments that highlight these aspects would be beneficial.
limitations: limitations: The authors have satisfactorily discussed the limitations and broader impact of their work.
paper_summary: paper_summary: The paper explores the selective scan mechanism for gesture generation.  First, it trains VQVAE to reconstruct faces and body parts using discrete latent space. Then, It uses local and global scanning mechanisms to improve the latent representations of various body parts for the purpose of gesture generation.
strength_weakness: strength_weakness: 1) The related work contains many acronyms not introduced or known to a wide community (e.g., VQVAEs, HiPPO, and LSSL). 
2) How the authors used Mamba in their work is unclear. The explanation in section 3.1 is vague. 
3) Audio feature extractor is not strong. The authors should have explored stronger speech representations that can be used to improve speech prosody.  
4) The technical contributions of the paper might be limited to implementing Mamba architecture for gesture analysis.
questions: questions: 1) Can you please dedicate section 3.1 to how you used Selective State Spaces for gesture generation? The explanation given is very generic. The reader can not follow this generic description without prior knowledge of Mamba.
2) How are the motion priors (the codebook) initialized? Also, how often is this codebook updated?
3) Why are the results reported in "EMAGE: Towards Unified Holistic Co-Speech Gesture Generation via
Expressive Masked Audio Gesture Modeling" differs from what you report in Table 1. FGD in the EMAG paper is 5.512, while in your paper, they are reported as 5.423. If you are using two different backbone models (and hence features), then how fair is this comparison if you fine-tune your models with the best parameters?
limitations: limitations: One limitation related to gesture diversity across speakers and cultures is not addressed. The mentioned limitations are mainly related to technical details, which I think are already being addressed using Transformer and Diffusion based models.
paper_summary: paper_summary: This study explores the use of state space models (SSMs) to enhance gesture synthesis, addressing challenges such as diverse movement dynamics and unnatural jittering in generated gestures. Through a two-stage modeling approach and the introduction of MambaTalk with hybrid fusion modules, the study demonstrates superior performance compared to existing models in subjective and objective experiments.
strength_weakness: strength_weakness: My concerns and suggestions of this manuscript are listed as blew:
1. The motivation of this work sounds unclear and looks a little incremental. As claimed in the abstract section 'the high computational complexity of these techniques limits the application in reality'. This statement is not clear. Does computational complexity mean more model parameters? or training time? or inference time? or more GPU memory cost? The authors didn't have a clear statement.
2. Moreover, in the abstract, 'which stem primarily from the diverse movement dynamics of various body parts.' are actually the general difficulties of the co-speech gesture generation task, not the specific one of how to employ the SSMs to this task.
3. The introduction section is not clear. The authors pay much more attention to the related works. Not the motivation and the high-level technical contributions of their work. This led me to feel that the entire work lacked technological innovation after reading the Introduction.
4. As for the methods, it is just directly imply the SSMs to this work. I cannot see any design on how to effectively solve the problem of 'computational complexity'. Actually, the technical contribution is poor, and the overall pipeline is very similar to the previous work EMAGE[1].
5. Could the authors explain why they only experimented on the BEATX dataset? As far as I know, TED[2] and TED-expressive[3] are also two commonly used co-speech gesture datasets.
6. The author did not conduct experiments to demonstrate how to effectively reduce the computational complexity by using SSMs, which directly led to the unclear motivation of this work.


[1] Liu, H., Zhu, Z., Becherini, G., Peng, Y., Su, M., Zhou, Y., ... & Black, M. J. (2023). Emage: Towards unified holistic co-speech gesture generation via masked audio gesture modeling. arXiv preprint arXiv:2401.00374.
[2] Yoon, Y., Cha, B., Lee, J. H., Jang, M., Lee, J., Kim, J., & Lee, G. (2020). Speech gesture generation from the trimodal context of text, audio, and speaker identity. ACM Transactions on Graphics (TOG), 39(6), 1-16.
[3] Liu, X., Wu, Q., Zhou, H., Xu, Y., Qian, R., Lin, X., ... & Zhou, B. (2022). Learning hierarchical cross-modal association for co-speech gesture generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 10462-10472).
questions: questions: please refer to Weaknesses
limitations: limitations: please refer to Weaknesses
paper_summary: paper_summary: This paper explores the application of state space models (SSMs) to co-speech gesture generation. The authors identify the computational challenges and jittering issues associated with the direct application of SSMs to gesture synthesis. To address these, they propose a two-stage modeling strategy with discrete motion priors and hybrid fusion modules. The first stage involves learning discrete holistic gesture priors with multiple VQVAEs, and the second stage refines latent space representations using local and global scans. Experiments demonstrate that MambaTalk outperforms state-of-the-art models in generating natural, rhythmic, and contextually appropriate gestures.
strength_weakness: strength_weakness: 1. Limited novelty. Integrating the local and global scans seems not novel. More discussion on the comparison between MambaTalk and baseline methods is needed.
2. Limited datasets for method evaluation. Only the BEATX-standard dataset is adopted to evaluate the performance of the method. How about the zero-shot quantitative comparison on other datasets, such as the BEAT dataset or the RAVDESS dataset, which take emotion into account?
questions: questions: See the "Weakness" section.
limitations: limitations: See the "Weaknesses" section. If possible, I suggest adding more visualization results and videos to the supplementary materials. Additionally, please note that the final score for this study is not solely determined by the peer reviewers' discussion. If the authors can address my main concerns, I would be willing to raise the score.

Comments: title: reply: Rebuttal by Authors
Response to authors
comment: We are deeply grateful for your thoughtful reconsideration of our manuscript and the increase in your rating (weak accept). Your feedback is instrumental in helping us refine our work, and we are pleased that our clarifications and responses have been well-received.

Thank you once again for your constructive feedback and for the opportunity to improve our manuscript based on your valuable insights.
Thank you very much for the clarification and the response, which is much appreciated, especially the point related to the comparison of the result. I have raised my ratings from 5 to 6.
We would like to express our sincere appreciation for your careful review and the time you have dedicated to evaluating our manuscript. We are grateful for the opportunity to provide a rebuttal and address the points you raised.

We respect your decision and the thorough consideration you have given to our manuscript. We are pleased that our work has been recognized with a "weak accept" rating, which we interpret as a positive endorsement of our research.

We would like to assure you that we have taken your feedback seriously and have made every effort to improve the manuscript. Should there be any further opportunities to refine our work or address any additional concerns, we are more than willing to do so.

Once again, thank you for your valuable feedback.
We are grateful for your engagement with our rebuttal and for the opportunity to address your concerns. We understand that despite our efforts to respond to your initial questions, there are still aspects of our work that have not fully met your expectations. Your feedback is invaluable to us, and we are committed to enhancing the quality of our research.

We acknowledge that our motivation and the clarity of our writing may not have been as compelling as they should be. To address these issues, we will take the following steps in our revised paper:
- **Clarification of Motivation**: We will revisit the introduction and conclusion sections to more explicitly articulate the significance and novelty of our research. We have included additional context to underscore the importance of our work within the broader scope of co-speech gesture synthesis. For the motivation of long-term and real-time motion synthesis, this approach is critical for human-computer interactive scenarios, where real-time performance is crucial, and some conversations require extended periods of speech. In this context, we integrated VQVAE with Mamba in our framework, effectively addressing the shortcoming of diffusion models (e.g., slow inference speed) through VQVAE's compression expression ability and Mamba's linear computational complexity.
- **Enhanced Writing Quality**: We will undertake a thorough review of the paper to ensure that the writing is clear, concise, and engaging. We will also seek the assistance of professional editors to refine our language and presentation.
- **Addressing Specific Concerns**: We have carefully considered each of your points and provided targeted feedback. We believe these changes have strengthened the overall coherence and persuasiveness of our arguments. If you have any questions, please feel free to raise them.

We hope that these revisions have adequately addressed your concerns and have brought our manuscript closer to acceptance. We are open to further suggestions and are willing to make additional revisions as needed to meet the standards of NeurIPS.

Thank you once again for your constructive feedback.
We would like to extend our sincere gratitude for the time and effort you have invested in reviewing our rebuttal. We appreciate the detailed feedback and understand your concerns regarding the lack of experimental results and the need for further clarification on the motivation for long-term and real-time motion synthesis.

For experimental results, we have conducted comprehensive experiments on holistic co-speech gesture synthesis (the main focus of our method), including quantitative results, qualitative results, and efficiency analysis in our initial version. In response to your feedback, we have incorporated supplementary experiment results on upper-body co-speech gesture synthesis in our rebuttal. The enhancement in our method is substantial, demonstrating improved overall performance compared to the baseline. Future experiments will focus on visualization.

For the motivation of long-term and real-time motion synthesis, we expand our discussion on the motivation behind the need for long-term and real-time motion synthesis. This approach is critical for human-computer interactive scenarios, where real-time performance is crucial, and some conversations require extended periods of speech. In this context, we integrated VQVAE with Mamba in our framework, effectively addressing the shortcoming of diffusion models (e.g., slow inference speed) through VQVAE's compression capability and Mamba's linear computational complexity. As demonstrated in Appendix A.2, our method can generate sequences with extremely low latency, which is beneficial for the application of co-speech gestures.

We look forward to your further feedback and are grateful for the opportunity to refine our work based on your valuable insights.

Thank you once again for your consideration.
Thank you for the additional information. After reviewing all the feedback and responses, I have decided to maintain my initial score. While the paper introduces some novel methodological aspects, the primary concerns remain: the lack of experimental results and the need for further clarification, particularly regarding the motivation for long-term and real-time motion synthesis.
Dear authors,

Thanks for your efforts and responses to my questions.
Although most of my concerns are addressed by the authors, I still have some main concerns about the motivation and writing of this work.
Therefore, I raise my rating from 3 to 4.
Thank you.

Best regards
Thanks to the author for providing the rebuttal. The initial score remains the same after the rebuttal.