MambaTalk: Efficient Holistic Gesture Synthesis with Selective State Space Models
Authors: Zunnan Xu
Yukang Lin
Haonan Han
Sicheng Yang
Ronghui Li
Yachao Zhang
Xiu Li
Venue: NeurIPS 2024 poster (2024)
Keywords: Gesture Synthesis
Selective State Space Models
Human Computer Interaction
Primary area: generative_models

Abstract: Gesture synthesis is a vital realm of human-computer interaction, with wide-ranging applications across various fields like film, robotics, and virtual reality. 
Recent advancements have utilized the diffusion model to improve gesture synthesis. 
However, the high computational complexity of these techniques limits the application in reality. 
In this study, we explore the potential of state space models (SSMs).
Direct application of SSMs in gesture synthesis encounters difficulties, which stem primarily from the diverse movement dynamics of various body parts. 
The generated gestures may also exhibit unnatural jittering issues.
To address these, we implement a two-stage modeling strategy with discrete motion priors to enhance the quality of gestures.
Built upon the selective scan mechanism, we introduce MambaTalk, which integrates hybrid fusion modules, local and global scans to refine latent space representations.
Subjective and objective experiments demonstrate that our method surpasses the performance of state-of-the-art models. Our project is publicly available at~\url{https://kkakkkka.github.io/MambaTalk/}.

MambaTalk: Efficient Holistic Gesture Synthesis with Selective State Space Models
Zunnan Xu", Yukang Lin", Haonan Han', Sicheng Yang, Ronghui Li, Yachao Zhang', Xiu Lit Shenzhen International Graduate School, Tsinghua University University Town of Shenzhen, Nanshan District, Shenzhen, Guangdong, P.R. China
Abstract
Gesture synthesis is a vital realm of human-computer interaction, with wide-ranging applications across various fields like film, robotics, and virtual reality. Recent advancements have utilized the diffusion model to improve gesture synthesis. How- ever, the high computational complexity of these techniques limits the application in reality. In this study, we explore the potential of state space models (SSMs). Direct application of SSMs in gesture synthesis encounters difficulties, which stem primarily from the diverse movement dynamics of various body parts. The generated gestures may also exhibit unnatural jittering issues. To address these, we implement two-stage modeling strategy with discrete motion priors to enhance the quality of gestures. Built upon the selective scan mechanism, we introduce MambaTalk, which integrates hybrid fusion modules, local and global scans refine latent space representations. Subjective and objective experiments demon- strate that our method surpasses the performance of state-of-the art models. Our project is publicly available at htttt::////// //kkkkkkka github. io/MambaTalkk
Introduction
Gesture synthesis is critical area of research in human-computer interaction (HCI), which has very broad application prospects, such as film, roboties, virtual reality, and digital human development [24] The task is challenging due to the variable correlation between speech and gestures, the same spoken content can elicit markedly different gestures among speakers. Meanwhile, the generated gestures should synchronize with the speaker's rhythm, emotional cues, and intentions [3
Recent works in co-speech gesture generation have shown great progress By introducing new datasets [7]] and more modalities (70) (33, previous work achieved end-to-end
46
66
gesture generation based on RNN-based models [T] 46). With the success of transformer in nature
language processing (58) and video sequence modeling 226
27 75,, recent works B 43 55 leverage
the power of attention mechanism to generate more expressive gestures that better synchronize with speech. By further combining emotional and style related features, EMoG [69 achieve better quality gesture generation. With the development in human recognition model [37], EMAGE 32] proposes a masked audio-ge gesttrr modeling strategy to enhance unified holistic gesture synthesis. Recently, with development diffusion model generative tasks (39 20 40), the latest works 774 62 65)
have applied the diffusion model to gesture synthesis, significantly improving the diversity of generated gesture. DiffuseStyleGestture [63] presents a diffusion model-based approach for generating diverse co-speech gestures by incorporating cross-local attention and self-attention mechanisms, and utilizing classifier- free guidance for style control. DiffuseStyleGesturee [66] further considers the text modality
additional input and utilizes channel concatenation to merge the text feature
"Work done as intern at Tecent. 'Equal Contribution.
Corresponding author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

with the audio feature. Deichler et al. [9] also incorporates the text modality as an additional input and employs contrastive learning to enhance the features. However, the exploration of generation for co-speech gesture sequences with low latency remains relatively uncharted, constraining its application in dynamic, interactive environments. RNN-based models often struggle with the long- term forgetting issue [54, (2), which impairs their ability to generate long sequences of gestures effectively. Additionally, these models may produce gestures that lack variability, tending towards an average representation [36). Transformer-based models depend heavily subtle positional encoding to capture the order of input elements 44 [50] 731. Meanwhile, their computational complexity,
which grows quadratically with the length of the input sequence, poses challenge for generating long sequences of gestures. For the diffusion based model, the intricate sampling strategy and iterative
process lead to high computational expenses 48), which hinder their broad adoption in gesture generation scenarios.
Stage I: Discrate Holistic Gesture Prior Leaming
Stage II: Latent Space Modelingg with SSMs
Codebook
was walking on the strett...
Local Scan
Globdl Scan
Our two-stage method for co-speech gesture generation with selective state space models. In the first stage, we construct discrete motion spaces learn specific motion codes. In the second stage, we develop a speech-driven model of the latent space using selective scanning mechanisms.
Figure I:
State space models (SSMs) have recently shown significant potential in addressing challenges related to modeling sequences with low latency [1].. Inspired by continuous state space models from control systems and enhanced by HiPPO initialization [3], SSMs [16] show promise in addressing long-term forgetting issue. These advancements have been integrated into large-scale representation models [38 41] Some pioneering works have applied SSMs for tasks like language understanding [38 4I], content-based reasoning [12, and visual recognition [35 57]. In work, we further explore the potential of SSMs in co-speech gesture synthesis. We observe that directly applying the selective scan mechanism from Mamba [12] to gesture generation as sequence modeling model would result in jittery outputs. To refine the generated gestures, we propose a two-stage modeling strategy. In the first training stage, we enhance the discrete motion priors derived from VQVAES [53] by integrating velocity and acceleration losses, In the second stage, by utilizing motion priors from VQVAEs, we introduce individual learnable queries for different body parts, thereby alleviating the jittering issue. Meanwhile, considering that the direct application of Mamba encounters the challenge of limb movements across different body parts tending to average out, we propose a hybrid scanning approach in the second stage to enhance the motion representation in the latent space. Specifically, we refine the design of spatial and temporal modeling within latent spaces by introducing global-to-ke lo-lclll modeling strategy and integrating attention mechanisms along with selective scanning approach into the framework's design. Considering the significant differences in deformation and movement patterns among different body parts (32), we propose local and global scan modules for refining the latent space representations the movements across various body parts, These approaches enable dynamic interaction and iterative refinement of different body parts while maintaining low latency, leading to more diverse and rhythmic gestures. Our contributions can be summarized as below:
We are the first to explore the potential of the selective scan mechanism for co-speech gesture synthesis, achieving diverse and realistic range of facial and gesture animations.
We introduce MambaTalk, an innovative framework that integrates hybrid scanning modules (c.g., local and global scan). The integration enhances the latent space representations for gesture synthesis, thereby refining the distinct movement patterns across various body parts.
Extensive experiments and analyses demonstrate the effectiveness of our proposed method.

---

2 Related Work
2.1 Co-speech Gesture Generation
Co-speech gesture generation aims to automatically generate gestures based on speech input. Existing approaches can be broadly categorized into three groups: (i) Rule-based methods: These methods rely on pre-defined rules and gesture libraries to generate gestures based on speech features [23] [5]] While offering interpretable results, they require significant manual effort in creating gesture datasets and defining rules. (ii) Statistical models: These approaches leverage data-driven techniques to learn mapping rules between speech and gestures, often employing pre-defined gesture units 22 23 While overcoming the limitations of manual rule creation, these methods still rely on handcrafted features. (iii) Deep learning methods: Recent advancements in deep learning have enabled neural networks to capture the complex relationship between speech and gestures directly from raw multimodal This progress has established deep learning approaches, particularly recurrent neural networks (RNNs) 170 33 611, transformers 45). and diffusion models [2] 74 51 72 21 o) as the prevailing paradigm for co-speech gesture generation. However, each of these models suffers from certain limitations that hinder their performance. RNNs inherently process sequences in a serial manner, where each timestep's computation depends on the output of the previous timestep. This limits their ability to efficiently handle long sequences and introduces cumulative latency. Meanwhile, RNNs lack inherent parallelism, further restricting their potential for high-speed computation. Transformers consider all positions within sequence at every timestep, resulting in high computational complexity, especially for long sequences. While diffusion models significantly enhance the diversity of generated outputs, the sampling process is computationally expensive. To overcome these limitations, our method investigates the capacity of selective state space models in the field of gesture synthesis. To the best of our knowledge, we the first to apply selective state space models to the task of gesture generation.
data [70 33 68 32
2.2 Selective State Space Models
State Space Models (SSMs) are a novel class of models recently integrated into deep learning for state space transformation [0] As foundational models evolve, various subquadratic-tine architectures have emerged, including linear attention, gated convolution, recurrent models, and structured state space models (SSMs), aimed mitigating the computational inefficiencies of Transformers when dealing with lengthy sequences. However, these advancements have yet to match the performance of attention mechanisms in critical modalities like language processing.
SSMs draw inspiration from continuous state space models in control systems and, when combined with HiPPO initialization [3), as seen in LSSL [16), show promise in tackling long-range depen- dency issues. However, the computational and memory demands of the state representation render LSSL impractical for real-world use. To address this, S4 [15] suggests normalizing the parameters into a diagonal structure. This has led to the emergence of various structured SSMs with diverse configurations, such as complex-diagonal structures [17 14 multiple-input multiple-output (MIMO)
support (49), diagonal-plus -wl-s-louu---aattee decomposition [19] and selection mechanisms [2]. These
models have been incorporated into large-scale representation models 338 41).
These models primarily focus on the application of SSMs to long-range and sequential data like language and speech, for tasks such as language understanding 38 content- based reasoning [12 and lixel-level 1-D image classification [15]. Recently, some pioneering work [35] 57 59 have explored their application in visual recognition. We further demonstrate that by incorporating the selective scan mechanism from mamba (12) and the discrete motion priors from VQVAES (53), our proposed MambaTTalk is capable of matching the performance of existing popular holistic gesture synthesis models, highlighting the potential of Mamba as a powerful gesture synthesis model.
Method
We aim to synthesize sequential 3D co-speech gestures from speech signals (e.g., audio and text) using selective state space models. However, simply applying such a model to gesture synthesis leads to severe gesture jittering issues. We also found that maintaining performance is challenging due to the significant variations in movement patterns exhibited by different body parts. To overcome these challenges, suggest modeling the gesture space using the acquired discrete motion patterns.
3

---

Subsequently, we propose to develop framework. This approach is designed enhance the model's robustmess against uncertainties that arise from cross-modal discrepancies. As shown in Figure 2, our framework consists of two stages: (i) modeling the discrete gestures and facial motion spaces ($.2) and (ii) learning speech-conditioned selective state space models ($3 3.3) to generate 3D co-speech gestures.
selective state space models within this
Stage Discrete Gesture Prior Learning
Codebook
VQVAE
VOVAE
Encoder
Decoder
Stage HI: Latent Space Modeling with SSMs
Audio Eacoder
Linear
VQVAE
Decoder
Facial Features
Local Scan
Speaker Embed
Global
Upper Features
Decuder
Speaker ID
Scan
Local Scan
VQVAE
Decoder
I Tower Fentires
VQVAE
I usually feel
Linear
Local Scan
Hand Features
Decoder
Encoder
Lineur
Layer Norm
Cross Altcation
Multi Head Self Alication
Mamba
Norm
000 Lecemable Quecies
Local Scan
Global Scan
Figure 2: We propose two-stage method for co-speech gesture generation. We first train multiple VQ-VAES for face and different parts of body reconstruction. This step learns discrete motion priors through multiple codebooks. In the second stage, we train a speech-driven gesture generation model in the latent motion space with local and global scan modules.
3.1 Preliminaries
Selective State Spaces Model. In our approach, we adopt the Selective State Spaces model (Mamba [12]) that incorporates selection mechanism and
module (S6). This model is designed to make sequence modeling, as it dynamically selects salient input segments for prediction, thereby enhancing its focus pertinent information and improving overall performance. Unlike the traditional $4 model, which uses time-invariant matrices A, B, C, and scalar A, Mamba introduces selection mechanism that allows for the learning of these parameters from the input data using fully-connected layers. This adaptability enables model to better generalize and perform complex modeling tasks. Mamba operates by defining the state space with structured matrices that introduce specific constraints on the parameters, facilitating efficient computation and data storage. For each batch and each dimension, the model processes the input 24, hidden state he, and output yr each time step t. We have by Bozo when 0. When t> O, the model's 's formulation is as follows:
h A +, V: CChe.
(1)
where Ae, Br, and Ct matrices and vectors that updated each time step, allowing the model to adapt to the temporal dynamics of the input sequence. With discretization, let A denote the sampling interval, exp( A) denote the matrix exponential, the transformation of the system's 's state over one time step can be represented as follows:
À exp(AA)
B (AA) (exp( AA) 1) AB,
(2)
h1 Ah + Brc.

---

where (AA)- denotes the inverse of matrix AA, I denotes the identity matrix. The scan module within Mamba is designed to capture temporal patterns and dependencies across multiple time steps by applying a set of trainable parameters operations to each segment of the input sequence. In our framework, Mamba serves as sequence modeling tool for decoding gesture actions across different parts of the body. By modifying the decoder's s input and the range of features, we utilize Mamba to separately model the global motion features and local motion features of different body parts. These operations are learned during training and assist the model in processing sequential data.
3.2 Discrete Gestures and Facial Motion Spaces
To ensure visual realism in motion animations from speech signals, we learn extra motion priors to depict accurate movements and natural expressions. Building on this concept, we propose method to represent the gesture motion space using multiple discrete codebooks.
Motion Quantization. Considering the substantial variations in deformation magnitude and period- icity among various body parts, our approach involves learning multiple codebooks tailored for the reconstruction of distinct body parts. For illustrative purposes, we detail the formulation of a single codebook. Denotes C as the dimensionality of each latent vector, N as the number of vectors in the N
codebook, for the codebook Z = {z E _ J kg=1' we employ a set of allocated items {z JKES to represent the holistic gesture motion M,. Here, represents the chosen index sets. The element-wise quantization function Q(-) maps each item 2, in 2 its closest match 2g in the codebook 2:
Zq = (() := arg min |2, Z 1l2
(3)
22
where the codebook entries act the foundational motion elements within the discrete motion space. To establish this, we follow 64
Variational Autoencoder (VQ-VAE) which comprises encoder E, decoder D, and context-rich codebook Z. This is done through the self-reconsttucction of gesture motions.
32 to pre-train a CNN-based Vector
The sequence of motions M is initially transformed into temporal feature representation 2 F(().T) E RT x where represents the count of gesture components, T" indicates the quantity of temporal units encoded (with P Tr frames per unit). Subsequently, we derive the quantized motion sequence Zg E *RT x XCC xC by quantization function Q(-). This function Q maps each element in 2 to its closest corresponding entry within the codebook Z:
Zq = Q (E (M..)) M = D (Za)
Training objectives. For the training of the quantized autoencoder, we employ motion-level losses to mitigate the jittering issue of generated gestures, along with two intermediate losses at the code level:
LvQ = Lree (M M, M) + (Me (M',')
M') + Cace M")
Zg ||; + 2 sg (gg
(5)
+
sg (
where M' and M" means the velocity and acceleration of motion, sg(-) denotes a stop- graddentt operation, Lre are Geodesic [52] loss and the last two terms are designed to refine the codebook entries. For facial motions, we utilize MSE loss for both velocity (L.et) and acceleration (Cace) loss. For body motions, we use L1 loss as Lvel and Lase Additionally, for the foot contact loss, we employ MSE loss as the loss function. These terms work by minimizing the distance between the codebook Z and the embedded features 2. Given that the quantization function (Equation 3) is non-differentiable, we utilize the srraight-througe gradient estimator (53 to propagate the gradients.
3.3 Speech-Driven Selective State Spaces Gesture Synthesis Model
Overall Framework. Utilizing the acquired discrete motion prior, we establish cross-modal mapping from speech inputs to target motion codes, enabling the generation of realistic co-speech gesture motions. In our approach to speech-driven gesture synthesis, we utilize audio sequences (aI,.... an) and text sequences T (4,,.., ......,,,y as inputs to guide the generation of co- speech gestures G {gi.....gh. J. Here, N significs the total frame count, and g, 150-6+100-113
5

---

denotes 55 pose joints in Rot6D, R 100 FLAME parameters, R foot contact labels, R global translations for the i-th frame. The gesture synthesis model, comprising audio encoders E and text encoders Fr and multiple selective state space models DB for different parts of the body, is trained on the discrete motion space, conditioned on the speech, as shown in Figure 2
Speech Feature Extraction. For audio feature extraction, two CNN-based audio feature extraction networks are employed respectively extract features from amplitude, raw audio and decoded tokens from Wav2vec2CTC 3 Considering that the movements of body parts are not closely linked to raw audio, the audio encoder for body parts does not utilize raw audio as input. Specifically, we integrate these features along the channel dimension to obtain audio features fA = {faa..... fa.. }. For processing speech imput words, we employ pre-trained FastText [5] to obtain word embeddings, which are then refined by linear projections to produce text features fr = [ft],.... ..... ftx J. We further fuses features from the input modalities (e.g., audio and text features). The speaker IDD embeddings are first combined with audio and text features through additive operation. By concatenating feature vectors along the channel dimension, we further apply linear transformations to determine the weight factors, and then integrating the features through an element- summation. The process can be formalized as:
WT <WT [fa + Sid 1, fr sid 1))
A = aWA [fA + s:d' 1, fr + sid 1)),
(6)
fr = WT @ FA + (1-wr) 2 fr.
fA = WA @ FA + (1- w1l-wa)@ fr, where denotes the softmax operation, O denotes the Hadamard product, and W and WA represent linear mapping matrices used to adjust the dimensions of merged features. fr and FA denote the fused features.
Global and Local Scans. Recognizing the diverse deformations and motion patterns in various body parts, we propose using global scan module and multiple local scan modules to model the movements of different body parts (c.g., face, hand, upper and lower body) with fused multi-modal features from previous modules. By acquiring the speech features from audio and text encoders, we first improve the perception of motion patterns among them using a global scan module by combining the speech features along the sequence dimension. Subsequently, by utilizing self-attention mechanism(FMnsA. we model the global information across different sequence tokens. Following previous work [32], establish set of learnable parameters (Q gQuhhl!) and employ masked motion facilitate the acquisition of global information. Then, we employ self-attention mechanisms to enhance the global information and obtain the refined features. These refined features are fed into Mamba extract temporal perceptual information. Considering the differences in representation between the body and face, we employ two independent MAMBA models to model the temporal features of the face and body, respectively. We then merge these features using a linear layer to obtain global features. The process can be fomulized as below:
Fglobal J MIISA (Qylobat)
Sspeech Mamba (fr, FA)),
Sgtobal = Mamba (aooba((goooll (ootttt)
Sglohat LLinea([[sppeetta fgtobmi)),
where [] denotes the concatenation operation of features in the dimension of the sequence. To enhance the generalization of the model, we incorporate the learnable queries to foster the queries' ability to learn motion patterns. As shown in Figure the queries from global scan are integrated with input speech features through multihead cross-attention mechanism (FMHCA). This allows queries to learn the most relevant information from the speech input. The process can be formally defined as belows:
frefine = FMHCA (ytobot (Fr, FAl):
(8)
where frefine denotes the feature of refined learnable queries. Utilizing the extracted perceptual features from various body parts, we proceed employ Mamba extract temporal features from the sequence, which can be formalized as belows:
Fjuce= = Mamba(,rrefie))
(9)
where Frace corresponds to the temporal features of facial motion. The same approach utilized to generate temporal features for the hand, upper body, and lower body by inputting their respective

---

perceptual features fhand. fupper. and flower into the corresponding Mamba modules. One distinction that we incorporate an additional self-attention layer before the Mamba layer to enhance the perception of body movements. The local latent features are then fed into their respective VQ- Decoders to produce the final motion predictions.
Training Objectives. The model's training objectives are composed of composite loss function that harmonizes reconstruction and cross-entropy losses. This design aims augment the accuracy of motion generation, encompassing the face, hands, upper, and lower body. The loss of latent reconstruction, represented by Lreclatent, is quantified using the Mean Squared Loss (MSELoss). Here, corresponds to the true latent vectors, while 2 are the vectors reconstructed by the model. The latent reconstruction loss is expressed as:
Treclatent
1 N
Σ
21 &
(10)
where N denotes the number of frames. Concurrently, to encourage diversity in the generated motions, we optimize the cross-entropy loss for latent code class classification Lote- Specifically, we employ Negative Log-Likelihood Loss (NLLLoss), where yi represents the class labels for each sample, and SJ denotes the model's predicted class labels. This loss is calculated as the negative sum of the logarithm of the predicted probabilities for the correct classes:
Lets
Yic log()se),
(11)
where N signifies the total number of frames, C' is the total number of classes, and Vic is binary indicator of whether class is the correct label for sample i. The total loss L a weighted sum the categorical and latent reconstruction losses, with and 3 serving balance hyper- parameerrst
L = aLals + LLreclatent
(12)
where a and for hands, upper and lower body motion. For facial motion, we set and B = 3. By optimizing the total loss, the model is trained to generate diverse gesture results.
4
Experiments
4.1 Experiments Setting
We train and evaluate on the BEAT2 dataset proposed by (32). BEAT2 contains 60 hours of data with high finger quality for 25 speakers (12 female and 13 male). The dataset comprises 1762 sequences, each with an average duration of 65.66 seconds. Each sequence includes response to daily inquiry. We split datasets into 33%%/75%%/754 for the train/val/test set. We follow previous work [32 to select data from Speaker 2 for training and validation to ensure fair comparison.
4.2 Implementation Details
We utilize the Adam optimizer with a learning rate of 2.5 10-4 To maintain stability, we apply gradient norm clipping at value of 0.99. In the construction of the VQVAEs, we employ uniform initialization for the codebook, setting the codebook entries to feature lengths 512 and establishing the codebook size at 256. The numerical distribution range for the codebook initialization is defined The codebook solely updated during the first stage, and in the second stage of training for the speech-to- to-getture mapping, the codebook remains frozen. The VQVAEs are trained for 200 epochs, with a learning rate of 2.5 x 10 -4 for the first 195 epochs, which is then reduced to 2.5 10 for the final 5 epochs. During the second stage, the model is trained for 100 epochs. All experiments are conducted using one NVIDIA A100 GPU.
as [1//codebook _osebooooooos______e //codebookssizee
4.3 Metrics
To evaluate the realism of body gestures, we employ Fréchet Gesture Distance (FGD) 70 to measure the proximity of the distribution between the ground truth and generated gestures. Subsequently, Diversity 28 is quantified by computing the average Ll distance across multiple gesture clips. The
7

---

synchronization between speech and motion is achieved using Beat Constancy (BC) 30] For facial motions, we assess positional accuracy by calculating the vertex Mean Squared Error (MSE) [60] Additionally, the difference between the ground truth and the generated facial vertices is measured using the vertex I.1 difference (LVD) [68]. More details about metrics and efficiency analysis are provided in the supplementary materials.
4.4 Quantitative Results
As shown in Table [ our method attains the lowest FGD and highest BC when compared to the previously cop-prformming method. This highlights the superior ability of MambaTalk in discerning and correlating the audio-motion beats. The lowest FGD also emphasizes the high quality and naturalness of our generated movements, showing the ability of MambaTalk to capture real motion dynamics. This also demonstrates the authenticity of our generated motions, affirmiin the successful capture of inherent motion characteristics. Some results are marked " because these methods can not generate facial movements. Moreover, our method outperforms previous methods in terms of MSE and LVD, with substantial improvements of 18.1 I % and 8.72%. respectively. These two enhancements highlight the superior accuracy and fidelity of our method in capturing fine-grained details, affirming efficacy in synthesizing realistic and authentic facial motions.
Table I: Quantitative results on BEAT2. FGD (Frechet Gesture Distance) multiplied by 10- BC
(Beat Constaney) multiplied by 10-1, Diversity, MSE (Mean Squared Error) multiplied by 10-7 and
LVD (Learned Vector Distance) multiplied by 10 10-5. The best results are in bold.
Methods
Venue
FGD
BC f
Diversity 1 MSE 4
LVD 4
Non-facial Gesture Synthesis
S2G 111
ICRA 2019
28.15
4.683
5.971
Trimodal
70
TOG 2020
12.41
5.933
7.724
HA2G
34
CVPR 2022
12.32
6.779
8.626
DisCo
ACMMM 2022
9.417
6.439
9.912
CaMN
33
ECCV 2022
6,644
6.769
10.86
DiffStyleGesture
63
IJCAI 2023
8.811
7.241
11.49
Holistic Gesture Synthesis
Habible al. [18
IVA 2021
9.040
7.716
8.21
8.614
8.043
TalkShow 68
CVPR 2023
6.209
6.947
13.47
7.791
7.771
EMAGE 32
CVPR 2024
5.512
7.724
13.06
7.680
7.556
Mamba'Tak (Ours)
5,366
7.812
13.05
6.289
6.897
4.5
Qualitative Analysis
User Study. We conducte user study to assess the visual quality of the generated co-speech 3D gestures. For each method under comparison, we produce 10 gesture samples, which were then converted into video clips for evaluation by 39 participants. In each evaluation session, participants are presented with 20 seconds video clips generated by various models. They are instructed assess the clips across the following dimensions: (i) naturalness, (ii) appropriateness, (iii) synchrony and (iv) smoothness. For naturalness, they evaluate the similarity of the generated gestures to those made by humans, paying attention to the authenticity and smoothness of the movements. In terms of appropriateness, they consider the alignment of the gestures with the spoken content, taking into account both the explicit meaning and the underlying semantics. For synchrony assessment, they examine the timing of the gestures in relation to the speech rhythm, audio, and facial expressions to ensure a harmonious and integrated performance. For smoothness, they assess the gestures for any abrupt stops or unnatural jerks that might indicate lack of fluidity motion. We mainly compare state-of-art art methods with our proposed method (with and without VQVAE): CaMN (3), EMAGE [32], and the ground truth. As presented in Table 2 our method's average scores are higher than previous methods.
Visualization. As depicted in Figure 3 our approach yields gestures that exhibit enhanced rhythmic alignment and more natural appearance. For instance, when conveying "we were", method

---

Table 2: User study results on naturalness (human likeness), appropriateness (the degree of consistency with the speech content), synchrony (the level of synchronization with the speech rhythm) and smoothness (the fluency of actions). The rating score range is 1-5, with 5 being the best. "Avg." denotes the average scores. f indicates the higher the better.
Methods
Naturalnesst
Appropriatenesst
Synchrony
Smoothness
Avg.
CaMN
33
3.08
3.34
3.25
3.50
3.29
EMAGE 32
3.85
4.04
3.89
4.21
3.99
Oursw VQVAE
1.24
1.24
1.18
1.29
1.24
Ours
4.04
4.00
3.91
4.35
4.08
Ground Truth
4.57
4.54
4.23
4.63
4.50
instructs the subject hold both hands in front of the chest, nuanced detail absent in both CaMN and EMAGE's outcomes, where either one or both arms hang down. Additionally, when representing "no place to". our method aligns with the ground truth by extending both arms upwards, whereas CaMN and EMAGE have their arms tucked in next to the body. In the case of "up", our generated result raises the right arm alignment with the semantics movement. around" where our left and right arm swings may differ from the ground truth, the overall movement remains consistent.
context of "moving
Ground Truth
sound of gunfire...
and police
nо place
borrible
moving around
right
CaMN
EMAGE
Figure 3: Visualization of the gestures generated by CaMN, EMAGE and our method. Unreasonable results are indicated by red boxes and reasonable ones by green boxes.
Interestingly, for "sound of gunfire", difficult semantic for the model learn, our method still generates the result of the character's right hand clenched in fist and the arm bent to indicate tense situation. For the emotion of fear expressed by "is horrible". the result of method is similar the ground truth, with the character's hands hanging down and face facing downward, which a visual representation of the psychological state of panic and fear. In addition, as illustrated in Figure 3 our generated motions exhibit not only diverse characteristics, such as the range of motion and which hands to use, but also high degree of consistency with the ground truth.
4.6 Ablation Study
Effect of VQVAEs. We confirm the significant role of the VQVAEs. As shown in Table 2 and T the integration of VQVAEs is essential for the functionality our approach, contributing to the generation of gestures that exhibit smoother transitions and more human-liae like quality. As

---

demonstrated in Table the removal of the VQVAEs (*- VQVAEs") from the model is also associated with performance decline, manifesting reduction in FGD, BC. Diversity, MSE and LVD.
Effect of Local Scan. We validate the effectiveness of the local scan. The ablation study is divided into segments: (i) multi head cross attention and (ii) Mamba models for different part of bodys. As shown in Table 3 incorporating multi head cross attention enhances our method's capability generate gestures with higher beat constancy. The incorporation of the Mamba from local scan generate gestures characterized by greater diversity. Concurrently, there is an observed improvement the FGD of the generated gestures.
Table 3: Ablation study different components of our proposed method. + denotes the lower the better, and t denotes the higher the better. FGD multiplied by 10-1, BC multiplied by 10-1 Diversity, MSE multiplied by 10-7, and LVD multiplied by 10-5.
Method
FGD
BC t
Diversity 1
MSE 4
LVD 4
Ours
5.366
7.812
13.048
0.629
6.897
VQVAES
12.051
7.447
8.462
1,316
9.235
Local Scan (FMHCA)
7.189
6.701
13.216
0.638
6.938
Local Scan (Mamba)
7.277
7.742
12.844
0.627
6,941
Global Scan (FMISA))
6.308
7.882
11.875
0.644
6.972
Global Scan (Mamba)
6.149
7.840
12.605
0.592
6.752
Effect of Global Sean. We validate the effectiveness of the global sean, as listed in Table 3 the incorporation of global scan improves the overall performance of our method. For the multi- head self-attention module in global scan, the incorporation of multi-head self-attention acquires improvement of Diversity and degradation for FGD. Additionally, the ablation results demonstrate that incorporating Mamba enhances the global scan's 's capability to generate gestures with higher diversity. The FGD of generated gestures better at the same time.
Table 4: Ablation study on different audio encoders. + denotes the lower the better, and denotes the higher the better. FGD multiplied by 10-1 BC multiplied by 10-1, Diversitt, MSE multiplied by 10-7 and LVD multiplied by 10-5
Method
FGD
BC f
Diversity 1
MSE 4
LVD 4
Ours
5.366
7.812
13.048
0.629
6.897
Whisper 47
6.791
7.515
12.617
0.537
6,445
Wav2vec2
5.343
7.956
13.164
0.973
8,452
Effeet of Different Audio Encoder. To validate the effectiveness of the audio encoder, we replace the CNN-based audio encoder with pre-trained Whisper 47 and Vav2Vec2 3 listed in Table 4 Unlike CNN- bused audio encoders that are randomly initialized and trained from scratch, when using Whisper and Wav2Vec2, we initialize the encoder using pre-trained weights and fix the parameters of the feature extractor. We observe a notable enhancement in facial generation when utilizing Whisper, however, the body generation results were subpar. In contrast, while Wav2Vec2 demonstrates some improvement in body generation, it results in a substantial decline in facial generation quality.
5 Conclusion
In this study, we propose framework employ the state space models in gesture synthesis. To alleviate the problem of jitter in gesture synthesis, we have implemented discrete motion priors, which enhance the effectiveness of the selective scan mechanism and lead smoother results. We further incorporate the selective state space models with attention mechanisms to enhance the refinement of motion features in latent space. These modules capture the subtle movements and deformations of various body parts, thereby enhancing the overall quality of the generated gestures. By utilizing linear time series modeling strategy with selective state space, our method achieves high-quality full body gesture generation with low latency.
10

---

Acknowledgements
This work was supported by the STI 2030-Major Projects under Grant 2021ZD0201404 and in part by the National Natural Science Foundation of China under Grant 62306165.
References
[1] Ao. T., Gao, Q. Lou, Y., Chen, B., Liu, L.: Rhythmic gesticulator: Rhythm-awaad aware co-speech gesture synthesis with hierarchical neural embeddings. ACM Transactions Graphics (TOG) 41(6). 1-19 (2022)
[2] Ao. T. Zhang. 7.,, Liu, I..: Gesturediffuclip: Gesture diffusion model with clip latents. arXiv preprint arXiv:2303.148: (2023)
[3] Baevski, A., Zhou, Y., Mohamed, A., Auli, M.: wav2vec 2.0: A framework for self-supervised learning of speech representations. Advances in neural information processing systems 33, 12449-12460 (2020)
[4] Bhattacharya, U.. Rewkowski, N., Banerjee, A., Guhan, P., Bera, A., Manocha, D.: Text2gestures: A transformer-based network for generating emotive body gestures for virtual agents. In: 2021 IEEE virtual reality and 3D user interfaces (VR). pp. 1-10. IEEE (2021)
[5] Bojanowski, P., Grave, E., Joulin, A., Mikolov, T.: Enriching word vectors with subword information. Transactions the association for computational linguistics 5, 135-146 (2017)
[6] Chemburkar, A., Lu, S., Feng. A.: Discrete diffusion for co-speech gesture synthesis. In: Companion Publication of the 25th International Conference on Multimodal Interaction, pp. 186-19 192 (2023)
[7] Chen, J.. Liu, Y.. Wang. J., Zeng. A., Li, Y., Chen. Q.: Diffisheg: A liffusion-basee approach for real-time specch-driven holistic 3d expression and gesture generation. arXiv preprint arXiv:2401 .04747 (2024)
[8] Chhatre, K., Danětek, R., Athanasiou, N., Becherini, G., Peters, C., Black, M.J., Bolkart, T.: Emotional speech-driven 3d body animation via disentangled latent diffusion. arXiv preprint arXiv:2312.04466 (2023)
191 Deichler, A., Mehta, S., Alexanderson, S., Beskow, J.: Diffusion- based co-speech gesture generation using joint text and audio representation. In: Proceedings of the 25th International Conference on Multimodal Interaction, pp. 755-762 (2023)
[10] Fu, D.Y., Dao, T., Saab, K.K., Thomas, A.W., Rudra, A., Ré, C.: Hungry hungry hippos: Towards language modeling with state space models. arXiv preprint arXiv:2212.14003 (2022)
[11] Ginosar, S., Bar, A., Kohavi, G., Chan, C., Owens, A., Malik, J.: Learning individual styles of conversa- tional gesture. Proceedings the TEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 3497-3506 (2019)
[12] Gu, A., Dao, T: Mamba: Linear-time sequence modeling with selective state spaces, arXiv preprint arXiv:2312.0075: (202))
[13] Gu, A., Dao, T., Ermon, S., Rudra, A., Ré, C.: Hippo: Recurrent memory with optimal polynomial projections. Advances in neural information processing systems 33, 1474-1487 (2020)
[14] Gu, A., Goel, K., Gupta, A., Ré, C.: On the parameterization and initialization diagonal state space models. Advances in Neural Information Processing Systems 35, 33971-33598 (2022)
[15] Gu, A., Goel, K.. Re. C.: Efficiently modeling long sequences with structured state spaces. International Conference on Learning Representations (2021)
[16] Gu, A., Johnson, I., Goel, K., Saab, K., Dao, T., Rudra, A., Ré, C.: Combining recurrent, convolutional, and continuous time models with linear state space layers. Advances in neural information processing systems 34, 572-585 (2021)
[17] Gupta, A., Gu, A., Berant, J.: Diagonal state spaces are as effective as structured state spaces. Advances in Neural Information Processing Systems 35, 22982 229924 (2022)
[18] Habibie, I., Xu, W., Mehta, D., Liu, L., Seidel, H.P. Pons-Moll, G., Elgharib, M., Theobalt, C.: Learning speech-driven 3d conversational gestures from video. In: Proceedings of the 21st ACM International Conference Intelligent Virtual Agents. pp. 101-108 (2021)
[19] Hasani, R., Lechner, M. Wang. T.H., Chahine, M., Amini, A., Rus, D.: Liquid structural spaee models. In: The Eleventh International Conference on Learning Representations (2022)

---

[20] He. C.. Shen, Y., Fang. C.. Xiao, F., Tang, L.. Zhang, Y. Zuo, W. Guo, Z., Li, X.: Diffusion models in low-level vision: A survey. arXiv preprint MxXiv:2406.11138 (2024)
[21] Kim, G., Li, Y., Ko, H.: The ku-ispl entry to the genea challenge 2023-a diffusion model for co-speech gesture generation. In: Companion Publication of the 25th International Conference on Multimodal Interaction, pp. 220-227 (2023)
[22] Kipp. M., Neff, M., Kipp. K.H., Albrecht, L: Towards natural gesture synthesis: Evaluating gesture units in data-driven approach gesture synthesis. In: Intelligent Virtual Agents: 7th International Conference, IVA 2007 Paris, France, September 17-19, 2007 Proceedings 7. pp. 15-28. Springer (2007)
[23] Kopp, S., Wachsmuth, L: Synthesizing multimodal utierances for conversational agents. Computer anima- tion and virtual worlds 15(1), 39-52 (2004)
[24] Kucherenko, I., Jonell, P.. Yoon, Y.. Wolfert, P.. Henter, G.E.: A large, crowdsourced evaluation of gesture generation systems on common data: The genea challenge 2020. In: 26th international conference on intelligent user interfaces. pp. 11-21 (2021)
[25] Levine, S., Krähenbühl, P., Thrun, S., Koltun, V:: Gesture controllers. In: ACM SIGGRAPH 2010 papers. pp. 1-11 (2010)
[26] Li, H., Cao, M., Cheng, X., Li, Y., Zhu, Z., Zou, Y.: G2I: Semantically aligned and uniform vidco grounding via geodesic and game theory. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 12032-12042 (2023)
[27] H., Cao. M., Cheng, X., Li, Y., Zhu, Z.. Zou. Y.: Exploiting auxiliary caption video grounding. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 38, pp. 18508-18516 (2024)
[28] Li, J., Kang. D., Pei, W., Zhe, X., Zhang. Y., He, Z., Bao, L.: Audio2gestures: Generating diverse gestures from speech audio with conditional variational autoencoders. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 11293- 11302 (2021)
[29] Li, R., Zhang, Y., Zhang. Y., Zhang, H., Guo, J., Zhang. Y., Liu, Y., Li, X.: Lodge: A coarse to finc diffusion network for long dance generation guided by the characteristic dance primitives, In; Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 1524-1534 (2024)
[30] Li, R., Yang, S., Ross, D.A., Kanazawa, A.: Ai chorcographer: Music conditioned 3d dance generation with aist++. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 13401-13412 (2021)
[31] Liu, H., Iwamoto, N., Zhu, Z., Li, Z.. Zhou, Y., Bozkurt, E., Zheng. B.: Disco: Disentangled implicit content and rhythm learning for diverse co-speech gestures synthesis. In: Proceedings of the 30th ACM International Conference on Multimedia. pp. 3764-377777 (174-3773 (2022)
[32] Liu, H., Zhu, Z., Becherini, G., Peng, Y., Su, M.. Zhou, Y., Iwamoto, N., Zheng, B., Black, MJJ.: Emage: Towards unified holistic co-speech gesture generation masked audio gesture modeling. arXiv preprint arXiv:2401.cm 100374 (2023)
[33] Liu, H. Zhu, Z.. Iwamoto, N., Peng, Y., Li, Z., Zhou, Y., Bozkurt, E., Zheng, B.: Beat: A large-scale semantic and emotional multi-modal dataset for conversational gestures synthesis. In: Computer Vision- ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part VII pp. 612 22-6630.60 Springer (2022)
[34] Liu, X.. Wu, Q. Zhou, H., Xu, Y., Qian, R., Lin, X., Zhou, X., Wu, W., Dai. B., Zhou, B.: Learning hicrarchical cross-modal association for co-speech gesture generation. In: Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 10462-10472 (2022)
[35] Liu, Y. Tian, Y., Zhao, Y. Yu, H., Xie, L.., Wang, Y., Ye, Q., Liu, Y.: Vmamba: Visual state space model. arXiv preprint w:Xiv:2401.101f (2024)
[36] Lu, S., Yoon, Y., Feng. A.: Co-speech gesture synthesis using discrete gesture token learning. arXiv preprint arXiv:2303. 12822 (2023)
[37] Lu, Y., Zhang, M., Lin, Y., Ma, A.J., Xie, X., Lai, J.: Improving pre-trained masked autoencoder via locality enhancement for person re- re-identification. In: Chinese Conference on Pattern Recognition and Computer Vision (PRCV). pp. 509-521. Springer (2022)
[38] Ma. X., Zhou, C., Kong, X., He, J. Gui, L., Neubig, G., May, J., Zettlemoyer, L.: Mega: Moving average equipped gated attention. In: The Eleventh International Conference on Learning Representations (2022)
12

---

[39] Ma, Y., He, Y., Cun, X., Wang, X.. Chen, S., X.. Chen, Q.: Follow your pose: Pose-guided text-to-video generation using pose-free videos. In: Proceedings of the AAAI Conference Artificial Intelligence. vol. 38, pp. 4117-4125 (2024)
[40] Ma. Y.. Liu, H., Wang. H., Pan, H., He, Y., Yuan, J.. Zeng. A.. Cai, C., Shum, H.Y., Liu, W., et al.: Follow- your-emoji: Fine-controllable and expressive freestyle portrait animation. arXiv preprint arXiv:2406.01 (2024)
[41] Mehta, H., Gupta, A., Cutkosky, A., Neyshabur, B.: Long range language modeling via gated state spaces. In: International Conference on Learning Representations (2023)
[42] Nyatsanga, S., Kucherenko, T., Ahuja. C., Henter, G.E., Neff, M.: A comprehensive review of data-driven co-speech gesture generation. In: Computer Graphics Forum. vol. 42, pp. 569-596. Wiley Online Library (2023)
[43] Pang. K., Qin, D., Fan, Y., Habekost, J., Shiratori, T., Yamagishi, J., Komura, T.: Bodyformer: Semantics- guided 3d body gesture synthesis with transformer. ACM Transactions on Graphics (TOG) 42(4). 1-12 (2023)
[44] Press, O., Smith, N.A., Lewis, M.: Train short, test long: Attention with linear biases enables input length extrapolation. arXiv preprint arXiv:2108. 12409 (2021)
[45] Qi, X., Liu, C., Li, L.. Hou, J., Xin, H. Yo, X.: Emotiongesture: Audio-driven diverse emotional co-speech 3d gesture generation. arXiv preprint arXiv:2305. 18891 (2023)
[46] Qian, S., Tu, Z.. Zhi, Y., Liu, W., Gao, S.: Speech drives templates: Co-speech gesture synthesis with learned templates. In: Proceedings of the IEEE/CVF International Conference Computer Vision. pp. 11077-11086 (2021)
[47] Radford, A., Kim, J.W., Xu, T., Brockman, G., McLeavey, C., Sutskever, L: Robust speech recognition via large-scale weak supervision. In: International conference on machine learning. pp. 28492-28518. PMLR (2023)
[48] Ren, Y.. Xia, X., Lu. Y., Zhang, J.. Wu, J., Xie, P., Wang, X., Xiao, X.: Hyper-sd: Trajectory segmented consistency model for efficient image synthesis. arXiv preprint arXiv:2404.1366 13686 (2024)
[49] Smith, J.T., Warrington, A., Linderman, S.: Simplified state space layers for sequence modeling. In: The Eleventh International Conference on Learning Representations (2022)
[50] Su, J., Ahmed, M., Lu, Y., Pan, S., Bo, W., Liu, Y.: Roformer: Enhanced transformer with rotary position embedding. Neurocomputing 568, 127063 (2024)
[51] Tonoli, R.I.,, Marques, I.B.Md.M,, Ueda, I.H.,, Costa, P.D.P.: Gesture generation with diffusion models aided by speech activity information. In: Companion Publication of the 25th International Conference on Multimodal Interaction. pp. 193-199 (2023)
[52] Tykkälä, T., Audras, C., Comport, A.I.: Direct iterative closest point for real-time visual odometry. In: 2011 I IEEE International Conference on Computer Vision Workshops (ICCV Workshops). pp. 2050-2056. IEEE (2011)
[53] Van Den Oord, A., Vinyals. O., et al.: Neural discrete representation learning. Advances in neural information processing systems 30 (2017)
[54] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, I.,, Gomez, A.N., Kaiser, F.., Polosukhin, I.: Attention is all you need. Advances in neural information processing systems 30 (2017)
[55] VoB, H., Kopp, S.: Aq-gt: temporally aligned and quantized gru-transformer for co-speech gesture synthesis. arXiv preprint arXiv:2305.01 (2023)
[56] Wagner, P., Malisz, 7., Kopp, S.: Gesture and speech in interaction: An overview (2014)
[57] Wang, Z., Zheng, J.Q.. Zhang, Y., Cui, G., Li, L.: Mamba-unet: Unet-like pure visual mamba for medical image segmentation. arXiv preprint ar arXiv:2402.05079 (2024)
[58] Wut, X., Li*, H., Luo, Y., Cheng, X., Zhuang, X., Cao, M., Fu, K.: Uncertainty-aware sign language vidco retrieval with probability distribution modeling. ECCV 2024 (2024)
[59] Xiao, Y.. Song. L.. Huang, S., Wang. J.. Song, S., Ge, Y. Li, X., Shan, Y.: Grootvl: Tree topology is all you need in state space model. arXiv preprint arXiv:2406.0 20t.:92206.00395511 (2024)
13

---

[60] Xing. J. Xia, M.. Zhang. Y., Cun, X., Wang. J. Wong, T.T.: Codetalker: Speech-driven 3d facial animation with discrete motion prior. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattem Recognition. pp. 12780-12790 (2023)
[61] Xu, Z., Zhang, Y. Yang. S.. Li, R., Li. X.: Chain of generation: conditional control. arXiv preprint arXiv:2312. (2023)
gesture synthesis via cascaded
[62] Yang, S., Wang, Z., Wu, Z.., Li, M.. Zhang, Z., Huang. Q. Hao, L.., Xu, S., Wu, X., Yang. C., et al.: Unifiedgesture: unified gesture synthesis model for multiple skeletons. In: Proceedings the 31st ACM International Conference on Multimedia. pp. 1033 1044 (2023)
[63] Yang, S., Wu, Z.. Li. M., Zhang. Hao, L.. Bao. W., Cheng, M., Xiao, L.: Diffuseettleeeestue: Stylized audio-driven co-speech gesture generation with diffusion models. arXiv preprint arXiv:230S. (2023)
[64] Yang, S., Wu. Li, M., Zhang. Z., Hao, L., Bao, W., Zhuang, H.: Qpgesture: Quantization- based and phase-guided motion matching for natural speech-driven gesture generation. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR. pp. 2321 23300. IEEE (June 2023)
[65] Yang. S., Xu, Z., Xue, H., Cheng, Y., Huang. S., Gong, M., Wu, Z.: Freetalker: Controllable speech and text-driven gesture generation based on diffusion models for enhanced speaker naturalness. arXiv preprint arXiv:2401 003476 (2024)
[66] Yang. S., Xue, H.. Zhang. Z., Li. M., Wu, Z., Wu, X., Xu, S., Dai, Z.: The diffusestyleggsttree entry to the genea challenge 2023. In: Procoedings of the 25th International Conference Multimodal Interaction. pp. 779-785 (2023)
[67] Yazdian, PJ.., Chen, M., Lim, A.: Gesture2vec: Clustering gestures using representation learning methods co-speech gesture generation. In: 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). pp. 3100-3107. IEEE (2022)
[68] Yi, H., Liang, H., Liu, Y., Cao, Q.. Wen, Y., Bolkart, T., Tao, D., Black, M.J.: Generating holistic 3d human motion from speech. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 469-480 (2023)
[69] Yin, L., Wang, Y. He, T., Liu, J., Zhao, W., Li, B., Jin, X., Lin, J.: Emog: Synthesizing emotive co-speech 3d gesture with diffusion model. arXiv preprint arXiv:2306. 11496 (2023)
[70] Yoon, Y., Cha, B., Lee, J.H., Jang, M., Lee, J., Kim, J.. Lee, G.: Speech gesture generation from the trimodal context of text, audio, and speaker identity. ACM Transactions on Graphics (TOG) 39(6), 1-16 (2020)
[71] Yoon, Y.. Ko, W.R., Jang, M., Loe, J., Kim, J., Lec, G.: Robots learn social skills: End-to-end learning of co-specch gesture generation for humanoid robots. In: 2019 International Conference Robotics and Automation (ICRA). pp. 4303-4309. IEEE (2019)
[72] Zhao, W., Hu, L., Zhang. S.: Diffugesture: Generating human gesture from two-person dialogue with diffu- sion models. In: Companion Publication of the 25th International Conference on Multimodal Interaction. pp. 179-185 (2023)
[73] Zhong, 7., Mi, Y., Huang, Y., Xu, J., Mu, G., Ding, S., Zhang, Guo, R., Wu, Y., Zhou, S.: Slerpface: Face template protection via spherical linear interpolation. arXiv preprint arXiv:2407. 0000000003333000 (2024)
[74] Zhu, L., Liu, X., Liu, X., Qian, R., Liu, Z., Yu, L.: Taming diffusion models for audio-driven co-speech gesture generation. arXiv preprint arXiv:2303.091 (2023)
[75] Zhuang, X., Li, H., Cheng, X., Zhu, Z., Xie, Y., Zou, Y.: Kdpror: knwwledge-dcccoppingg probabilistic framework for vidco-text retrieval. ECCV 2024 (2024)
14

---

A Appendix Supplemental material
A.1 More visualization results
Figure presents the facial motion results generated by our method, showcasing the generation of facial expressions and movements with a high level of realism. Our method effectively synchronizes with the phonetic articulation of speech content, accurately reflecting the physical demands of pronunciation. For instance, when uttering "walking", "came" or "bus", our approach ensures that the mouth's 's movements, such as opening, correspond closely with the actual phonetic requirements. Other methods do not consistently achieve this level of accuracy aligning with the phonetic and physical nuances of speech. Our method adeptly handles the subtleties of mouth closure and elongation required for sounds such as "in". closely aligning with the ground truth, whereas other approaches may exhibit inconsistencies in this regard. Moreover, in instances of silence, all methods, including ours, demonstrate good capacity learn and maintain the mouth's 's closed position, effectively reflecting the underlying patterns of speech and silence. Since CaMN does not specifically target the generation of facial movements, results in a lack of variation in facial expressions throughout the process.
Ground Truth
walking home... /' kuy
that was in /nf
...card came
at the bus stop
/keim/
/bss/
Ours
EMAGE
CaMN
Figure 4: Visualization of the facial motions generated by CaMN, EMAGE and our method. Unrea- sonable results are indicated by red and gray boxes and reasonable ones by green boxes.
As illustrated in Figure our methodology generates gestures that demonstrate improved rhythmic synchronization and more lifelike appearance, effectively capturing the essence of the speaker's s rhythmic patterns. For example, in the expression "actually" our method guides the individual to bring the hands inward in front of the chest, a subtle gesture not observed in the results produced by CaMN and EMAGE, where the arms are either hang limply at the sides or are splayed downward. Furthermore, in the depiction of "on the way back" approach accurately reflects the ground truth by slightly bending down and raising one hand, while EMAGE cannot respond accurately to this and remains standing.
Furthermore, our approach accurately captures the semantic essence of movements. For instance, in response to the cue "hug" our method generates an inward-circling motion of the arms, aligning perfectly with the ground truth, which is a nuanced semantic element that other methodologies neglect. Similarly, in scenarios such as "so small", the result of our method is similar to the ground truth, with the character's hand moving inward. This attention to detail ensures semantic consistency, which is lacking other approaches where actions are not aligned with the intended meaning.
15

---

Ground Truth
actually last year... on the way back
who was
like or 5 years
warm hug and
so small...
Ours
EMAGE
CaMN
Visualization of the gestures generated by CaMN, EMAGE and our method. Unreasonable results are indicated by red boxes and reasonable ones by green boxes.
Figure 5:
A.2 Efficiency Analysis
We leverage the linear computational complexity of Mamba and the sequence compression capability of VQVAE within our framework, which helps in reducing computational complexity. Although there are some specialized acceleration solutions, faster solutions model integrated into the system, there not only a delay in generating gestures, but also delays in other modules. Therefore, to evaluate the efficiency of our pipeline, conducted series of measurements, focusing on the runtime individual components. We measured the runtime of various components the NVIDIA A100 GPU in method average results Table 5 We also compare our method's s inference time with diffusion based methods. The average computation time determined based the generation of 31 second motion sequence, to showcase our model's low latency capabilities. The total inference time of our method is much slower than state-of-the-art diffusion-based method (63) The result confirm that our pipeline is well-suited for applications requiring low latency gesture generation like interactive systems, where responsiveness is paramount.
necessary because when the
three runs and presented the
Table 5: The time cost for generating second (average) of gestures using the method's modules.
Modules
Run Time(s)
Diffusion- method
DiffStyleGesture 63
0.64365 0.0086
Our method
Audio Encoders
0.00217 + 0.0006
Text Encoders
0.00480 0.0001
Global Scan
0.00219 + 0.0004
Local Scan
0.00676 = 0.0003
Face VQDecoder
0.00073 + 0.0001
Hand VQDecoder
0.00077 = 0.0001
Upper VODecoder
0.00106 = 0.0001
Lower VQDecoder
0.00068 = 0.0001
Total Time
0.01917 = 0.0018
16

---

A.3 Evaluation on BEAT dataset
To evaluate the generalisable benefit of method, conduct experiments large-scale multimodal dataset known as BEAT (Body-EEpressionnAAudio Text) (33. This dataset encompasses 76 hours of multimodal data collected from 30 speakers engaging conversations across four different languages while expressing eight distinct emotions. The dataset includes conversational gestures, facial expressions, emotional cues, and semantic content, along with annotations for audio, text, and speaker identity. To facilitate a fair comparison, we follow CaMN 33 and employ approximately 16 hours of speech data from English speakers. Furthermore, we implement the conventional approach of partitioning the dataset into distinct training, validation, and testing subsets, ensuring consistency with the data partitioning scheme utilized in prior research to uphold the integrity the comparison.
To facilitate a fair comparison, we employ total of N = 34 frame clips with a stride of 10 during the training process. The first four frames serve seed poses, while the model is trained to generate the subsequent 30 poses, which collectively represent duration of 2 seconds. Our models incorporate 47 joints from the BEAT dataset, comprising 38 hand joints and 9 body joints. As listed in Table 6 our method demonstrates a significant improvement compared to the CaMN (baseline), which also validates the generalisable benefits of approach.
Table 6: Comparison with state-of- toate-offthh-arr method in the term of FGD, SRGR and BeatAlign. All methods are trained on BEAT datasets. & denotes the lower the better while denotes the higher the better. The best results are in bold.
Methods
FGD
SRGR t
BeatAlign
Seq2Seq 7
261.3
0.173
0.729
Spetch2Gesture
1
256.7
0.092
0.751
MultiContext 70
176.2
0.195
0.776
Audio2Gesture 28
223.8
0.097
0.766
CaMN
33
123.7
0.239
0.783
TalkShow 168
91.00
0,840
Mamba Talk (ours)
51.3
0.256
0.852
A.4 Limitations
Currently, our approach to gesture synthesis involves using distinct modules to animate various body parts, which naturally introduces some latency. Developing a single, unified model capable of capturing the wide-ranging and intricate deformations and motion patterns characteristic of different body parts can be addressed in future research. This enhancement anticipated to lower computational overhead and substantially reduce the processing time, thereby improving the real-time capabilities of the pipeline and ensuring a smoother and more responsive gesture generation system.
Meanwhile, exploring more robust audio representations or combining various types of pre-trained audio encoders could significantly enhance the quality of gesture generation. Our findings indicate that certain encoders, such as Whisper, are particularly effective for modeling facial movements, while others. like Wav2Vec2, are better suited for modeling body movements. This approach will further improve the overall performance of the method.
In addition, the issue of gesture diversity among speakers and across different cultures remains unaddressed. Addressing this gap is essential for improving the cross-cultural validity and expanding the applicability of cesture-basede applications in diverse global contexts.
A.5
Pseudo Code
The local scanning procedure illustrated in Algorithm 25 We attain local modeling of various body segments by individually processing actions within distinct regions. The approach to global scanning parallels this methodology; however, the key distinction lies the simultaneous processing of motion representations across multiple body parts.
17

---

Algorithm Local Scanning Process
Require: token sequence T (B
Ensure: token sequence T, D)
1: model motions different body regions separately T
2: Bface M, + Linear face (T/ (T/11))
3: Zupperbody (B, X, B) Linear "pperbody (SelfAttn(T (
)
4: Elowerbody (B, M, E) +- Linear lowerbody
(SelfAttn Lowerbodyy
5: Khand (, B) <- Linear (SelfAttt(T//11" haand
process with different parts human body #/
7: for o in [face, upperbody, lowerbody, hand] do
8: X (B, E) - SiLU (CLLVCo (oovlld(()))
BB
4- Linear : (xx)
10: C. (B,M, N) + Linear (x..)
1a softplus ensures positive A. %
A. (B,M, M. E) <- log(1 expmear (x,) Parameter,))
13:
1a shape of Parameter, is (E
14:
A.: (B, N) + A, X Parameter
B. (,, - A. & B.
16: No: (B, Wo:(B,M, < SS.(A.. B,, (x)(x))
17: end for
18: get gated yo */
19: Your : (B, M. E) c Vace O U
20: Vupperoddy E. < Mupposhody
O SiLU (Zuppabody
Yiuwerbody E) < Ylowerbodly
O SiLU (Zooerbody)
22: Yiund (B Yland O SiLLU(Zu (Zhand))
residual connection */
24: Tr (B, M. D) + Linear (Ylace + Yupperbody + Ylowerbody Yhand) + Ti-1
25: Return: T,
A.6 Evaluation Metrics
To evaluate the realism body gestures, we employ Fréchet Gesture Distance (FG to measure how close the distribution between the ground truth and generated body gestures is.
FGD(g. g) || Mgl Tr Σ, + -ig - 21,
(13)
where μ4 and E, denote the mean and covariance of the latent feature distribution 2, for real human gestures g, while H4 and Mn correspond to the mean and covariance of the latent feature distribution 2g for the synthesized gestures 9. We employ encoder based Skeleton CNN (SKCNN) and Full CNN-based decoder, constituting our autoencoder's pretrained network. This network is trained and REATT-Addiiional datasets. The preference for SKCNN over Full CNN encoder stems from its superior performance in capturing gesture features, evidenced by a reduced reconstruction MSE loss of 0.095, opposed 0. 0.103.
on both the
Subsequently, Diversity 28 quantified by computing the average L.1 distance across multiple body gesture clips. Higher Diversity signifies greater variance within the gesture clips. We compute the average LI distance across various N motion clips using the following equation:
N Σ
N
Diversity = 2N(N 1)
(14)
where P; denotes the positions of joints in frame We assess diversity across the entire test dataset. Moreover, when calculating joint positions, translation zeroed, indicating that L1 Diversity is exclusively concentrated local motion dynamics.
The synchronization between the speech and motion is conducted using Beat Constancy (BC) BC indicates a more precise synchronization between the rhythm of gestures and the audio 's beat. We define the onset speech as the audio's beat and identify the local minima of the upper body joints' velocity (excluding fingers) the motion's 's beat. The synchronization between audio and
18

---

gesture is determined using the following equation:
mins, 166. - ball
BC
Σ by
exp
(15)
2x²
where g and represent the sets of gesture beats and audio beats, respectively.
Turning focus to facial aspects, we gauge the positional accuracy through the calculation of vertex Mean Squared Error (MSH) This metric quantifies the average squared difference between the predicted facial landmarks and their corresponding ground truths, providing a clear indication of the facial model's accuracy:
MSE
(h. - fA) π
(16)
where n denotes the number of vertices, fi represents the ground truth position of the i-th vertex, f: denotes the predicted position of the i-th vertex. The sum is taken over all vertices to compute the average error.
Additionally, the disparity between the ground truth and the generated facial vertices is measured by the vertex L.1 difference (68 which measures the synchronization between speech and facial expression.
LVD 1 Σ
5 P
(17)
where n denotes the number of vertices, f represents the ground truth speed of the i-th vertex. f denotes the speed of the i-th vertex the generated facial expression. The sum taken over all vertices to compute the average absolute difference.
A.7 Ethical Considerations in Crowdsourcing Research
This section provides additional details about the user study for qualitative analysis. For our user study, we have randomly selected ten videos generated by different methods, each containing 20-seconi video clips. For each participant, we paid compensation that exceeded the local average hourly wage.
The screenshot of our user study website illustrated the Figure which displays the template layout presented the participants. In addition to the main trials, participants were also subjected to several catch trials. These trials involved displaying Ground Truth videos and videos with distorted motion. Participants who failed score the GT videos higher and the distorted motion videos lower were considered unresponsive or inattentive and their data was not included in the final evaluation.
19

---

Rate your score on these videos.
In this task you presented with mutilple videos of animated virtual characters.
All videos have sound, please listen to them!
You will be asked to rate the videos based on four different criteria.
Please focus gestures and the facial expressions of the characters.
Choose your preference from right. rating score range 1-5, with being the best.
Please press play in order to start the videos. You need to watch and listen videos at least once to be able to answer.
Video
For naturalness, the generated gestures similar to those made humans, with authentic movements?
04
3
2
01
For appropriateness, do the gestures align well with the spaken content, considering both the explicit meaning and the underlying semnantic??
2
O1
synchrony, does timing gestures and facial expressions match speech rhythm and audio for harmonious and integrated performance?
3
2
4. For smoothness, da the gestures have any abrupt stops unnatural jerks that might indicate a lack of fluidity in motion?
O4
3
O
O1
Submit
Figure 6: The screenshots of user study website for participants.
20

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Overall, the abstract and introduction provide a concise yet comprehensive sum- mary of the paper's objectives, methods, and findings, accurately reflecting its contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Yes, we discuss the limitations in section A.4 in the Appendix part.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-specfiiaatinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g. if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
21

---

Answer: [Yes]
Justification: By adhering to the principles mentioned in the Guidelines, we ensures that each theoretical result is underpinned by full set of assumptions and complete, correct proofs, thus reinforcing the credibility and reliability of the paper.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proof's provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions the paper (regardless whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: In Section 4. Experiments in the main text, we report all the experiments setting, implementation details and metries, which disclose all the information needed to reproduce the main experimental results.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should be possible for other researchers to have some path to reproducing verifying the results.
22

---

reproducible, they 5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification:
The code is available.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.. cc/
public/guides
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/use//uuuude//oeuubbiiiii 7/nips cc/ wtc/publicc/ gew.ci/gg/////eeeeei Codeeobmiiiie SanSSubbissss iobbbbisssiioi ssionoolicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: In Section 4. Experiments in the main text, we report all the experiments setting and implementation details, facilitating readers' understanding of the results.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer:
[Yes]
Justification:
We report the error bars in Section A.2 Efficiency Analysis.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
23

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We report the computer resources in Section 4.2 Implementation Details, and show model runtimes in detail in Section A.2 Efficiency Analysis.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics http:///wwwrrr...coooooloccccctt
Answer: [Yes]
Justification: Yes, do.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: The work we perform makes no society impact. It is only an academic study.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We properly cite the original owners of code, data and models.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in package should be provided. For popular datasets, paperswithcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
25

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the datasct/codd/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [Yes]
Justification: Yes, we provide the details in appendix A.5 Ethical Considerations in Crowd- sourcing Research.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [Yes]
Justification: Yes, we do.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such the institution conducting the review.
26


Reviews:
[{"review_id": "i70WlFmUHS", "reviewer": "Reviewer_gANk", "summary": "This paper focuses on the problem of co-speech gesture generation, particularly aiming to address the challenges of jittery movements, long motion sequences, and holistic gesture generation (including both face and body movements). To tackle these issues, the authors present a new method that combines diffusion models with state space models and incorporates both local and global scans. The proposed method has been evaluated on the BeatX dataset.", "questions": "Why is only one dataset considered? Please explain the rationale behind evaluating the method on only one dataset. How are these results generalisable and competitive given this limitation? \n\nGeneralizability and Competitiveness: How do you ensure that the results obtained from the BeatX dataset can be generalized to other scenarios? Discuss any potential limitations in terms of generalisability and how your method addresses them.\n\nTraining and Evaluation Sequence Lengths: Please elaborate on the length of the motion sequences considered during training and evaluation.\n\nImpact on Co-Speech Gesture Generation Performance: Discuss how the length of motion sequences impacts the performance of co-speech gesture generation. What are the implications of using different sequence lengths, and how does your method handle long-term motion generation effectively? Including comparisons or experiments that highlight these aspects would be beneficial.", "limitations": "The authors have satisfactorily discussed the limitations and broader impact of their work.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper introduces novel methodological aspects for co-speech feature generation. For instance, combining diffusion models with state space models and local and global scan approaches has not been explored before, to the best of my knowledge.\n\nThe experimental results clearly demonstrate the effectiveness of the proposed approach. Moreover, the paper presents both quantitative results and qualitative results through user studies, providing a complete and strong evaluation. Additionally, the proposed approach is faster compared to the state-of-the-art methods, as presented in the appendix.\n\nHowever, there are some weaknesses that need to be addressed. Please see below for detailed comments.", "weaknesses": "Major Comment:\n\nThe main weakness of the paper is its limited evaluation. While the paper excels in generating facial, body, or holistic gestures, evaluating on only one dataset is substandard. To be competitive, the authors should consider additional standard benchmarks such as those used in the GENEA challenge [1]. There is also a long list of papers using diffusion models for co-speech gesture synthesis [1], none of which are mentioned in the paper. While I appreciate the authors' work and acknowledge the scarcity of datasets with both face and body motions, the proposed approach handles face and body separately. Therefore, at least body motion generation performance could be compared on existing datasets to demonstrate generalisable benefits.\n\nAnother weakness is the insufficient details regarding long-term motion generation, which is claimed as a key contribution. The paper lacks sufficient details on the length of the sequences considered in training and evaluation. Moreover, there is no comparison to support this claim.\n\nMinor Comment:\n\nThe lower part of Figure 2 is helpful for understanding the workflow. However, including pseudo-code would be beneficial to increase the reproducibility of this method.\n\n[1] https://genea-workshop.github.io/2023/challenge/", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**A1: Reasons for choosing BEATX**\n\nThanks for your kind advice. Our work focuses on holistic co-speech gesture generation. We chose to use this dataset because it includes global movements and the smplx sequences of the entire body (e.g., face, upper body, lower body, and hands). However, most current datasets (such as TED, TED-X, BEAT, GENEA) only include movements of specific parts of the body (e.g., upper body), not the entire body.\n\n**A2: Generalisable benefits of our method**\n\nTo further validate the generalizable benefits of our method on upper body and hand movements, we used the BEAT dataset, which includes upper body and hand movements.\nDue to time constraints, we only conducted preliminary experiment. The results are shown in the table. Our method demonstrates a significant improvement compared to the CaMN (baseline), which also validates the effectiveness of our approach. More comprehensive experiments with detailed information will be included in the revised version.\n| Method | FGD$\\downarrow$ | SRGR$\\uparrow$ | BeatAlign$\\uparrow$ |\n|--------------------------|------------------|-----------------|--------|\n| CaMN (baseline)     | 123.7   | 0.239       | 0.783   |\n| Ours   |       51.3        | 0.256             | 0.852     |\n\n**A3: Related diffusion-based work**\n\nThanks for pointing out these wonderful works. For the diffusion-based model, we have compared our method with DiffuseStyleGesture in Table 1 and Table 4, which is a similar method to DiffuseStyleGesture+ (Reproducibility Award of GENEA workshop).\nCompare to DiffuseStyleGesture, DiffuseStyleGesture+ primarily considers the text modality as an additional input and utilizes channel concatenation to merge the text feature with the audio feature. Another leading solution (Diffusion-based co-speech gesture generation using joint text and audio representation) also incorporates the text modality as an additional input and employs contrastive learning to enhance the features.\nWe will cite and further analyze the differences using all of these methods in our revised version.\n\n**A4: The length of the motion sequences considered during training and evaluation**\n\nThanks for your professional advice. However, what we want to declare in our paper is that our method can generate long sequences with low latency (with analysis in Appendix A.2).\nFor the length of the motion sequences considered during training and evaluation, we adopt a segmented modeling strategy. This strategy involves dividing the target sequence into multiple segments, each 64 frames in length, for processing. Therefore, our method can effectively handle long-term motion generation.\nThe length of the motion sequence does not have a significant impact on performance because almost all segmented sequences share the same length and are processed in a similar manner.\n\n**A5: Pseudo-code**\n\nThanks for your advice. We will include pseudo-code in our revised version. We will also make our code open-sourced if the paper get accepted.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aadxUYAfsE", "reviewer": "Reviewer_WGJa", "summary": "The paper explores the selective scan mechanism for gesture generation.  First, it trains VQVAE to reconstruct faces and body parts using discrete latent space. Then, It uses local and global scanning mechanisms to improve the latent representations of various body parts for the purpose of gesture generation.", "questions": "1) Can you please dedicate section 3.1 to how you used Selective State Spaces for gesture generation? The explanation given is very generic. The reader can not follow this generic description without prior knowledge of Mamba.\n2) How are the motion priors (the codebook) initialized? Also, how often is this codebook updated?\n3) Why are the results reported in \"EMAGE: Towards Unified Holistic Co-Speech Gesture Generation via\nExpressive Masked Audio Gesture Modeling\" differs from what you report in Table 1. FGD in the EMAG paper is 5.512, while in your paper, they are reported as 5.423. If you are using two different backbone models (and hence features), then how fair is this comparison if you fine-tune your models with the best parameters?", "limitations": "One limitation related to gesture diversity across speakers and cultures is not addressed. The mentioned limitations are mainly related to technical details, which I think are already being addressed using Transformer and Diffusion based models.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1) The paper explores Mamba for gesture generation. \n2) The objective function includes features related to gesture acceleration and velocity, which are important to capture. \n3) The paper contains a user study where even the ground truths are evaluated. I think that is the right direction for gesture-related studies.", "weaknesses": "1) The related work contains many acronyms not introduced or known to a wide community (e.g., VQVAEs, HiPPO, and LSSL). \n2) How the authors used Mamba in their work is unclear. The explanation in section 3.1 is vague. \n3) Audio feature extractor is not strong. The authors should have explored stronger speech representations that can be used to improve speech prosody.  \n4) The technical contributions of the paper might be limited to implementing Mamba architecture for gesture analysis.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**A1: Related work and Preliminaries section**\n\nWe apologize for any inconvenience caused to your reading due to the generic explanation. We will provide a more detailed explanation for related work (e.g., VQVAE, HiPPO, LSSL) in section 3.1, which acts as preliminaries of our work.\nFor the use of Mamba, we mainly discuss its application in Section 3.3, where we combine Mamba with attention mechanisms and learnable queries to model motion sequences using a local-to-global scanning strategy. In the revised version, we will supplement more details, including pseudo-code, to provide a clearer explanation of our method.\n\n**A2: Codebook**\n\nThanks for your questions. We utilize uniform initialization for the codebook, with a numerical distribution range of [-1/codebook_size, 1/codebook_size), and conduct the first training stage on the training dataset. The codebook is solely updated during the first stage, and in the second stage of training for the speech-to-gesture mapping, the codebook remains frozen.\n\n**A3: Audio representation**\n\nThanks for your valuable comments. Enhancing the audio representation can potentially improve performance, but it might also result in unfair comparisons with previous methods. In our revised version, we will include additional experiments using various audio feature extractors (e.g., wav2vec2, Whisper).\n\n**A4: Technical contribution**\n\nOur work did not simply involve implementing Mamba for gesture synthesis. We found that directly applying Mamba would cause serious shaking issues. Therefore, we explored incorporating motion priors using VQVAEs and individual learnable queries for different parts of the body.\nAdditionally, the direct application of Mamba also has the issue that the limb movements of different body parts tend to be average. Therefore, we refined the design of spatial and temporal modeling in latent spaces by proposing a local-to-global modeling strategy and incorporating them with attention and selective scan mechanism into the design of our framework. We will refine our statements and make our framework clearer (e.g., adding pseudo-code) in our revised version.\n\n**A5: Results of EMAGE**\n\nFor the difference in results, we compared the v3 version released by EMAGE on arXiv, which the authors of EMAGE claimed to be the CVPR 2024 camera-ready version.\nWe checked again and found that they later updated multiple versions of arXiv. The latest version, v5, does indeed report 5.512. We will incorporate their updated results in our revised version.\nTo ensure fairness in the comparison, we maintained consistency with them in the experimental setting. We have an advantage over EMAGE in all metrics, with significant performance improvements in BC, MSE, and LVD metrics, demonstrating enhancements of 16.72%, 36.35%, and 13.99%.\n\n**A6: Limitation section**\n\nThanks for your valuable comments. We will add this discussion to the limitations section.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EijzLrVQxv", "reviewer": "Reviewer_p3w5", "summary": "This study explores the use of state space models (SSMs) to enhance gesture synthesis, addressing challenges such as diverse movement dynamics and unnatural jittering in generated gestures. Through a two-stage modeling approach and the introduction of MambaTalk with hybrid fusion modules, the study demonstrates superior performance compared to existing models in subjective and objective experiments.", "questions": "please refer to Weaknesses", "limitations": "please refer to Weaknesses", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. This work is the first to explore the potential of the selective scan mechanism for co-speech gesture synthesis, achieving a diverse and realistic range of facial and gesture animations.\n2. The writing of this work is fine, there are no obvious typos.\n3. The workflow pipeline figure is clear to understand.", "weaknesses": "My concerns and suggestions of this manuscript are listed as blew:\n1. The motivation of this work sounds unclear and looks a little incremental. As claimed in the abstract section 'the high computational complexity of these techniques limits the application in reality'. This statement is not clear. Does computational complexity mean more model parameters? or training time? or inference time? or more GPU memory cost? The authors didn't have a clear statement.\n2. Moreover, in the abstract, 'which stem primarily from the diverse movement dynamics of various body parts.' are actually the general difficulties of the co-speech gesture generation task, not the specific one of how to employ the SSMs to this task.\n3. The introduction section is not clear. The authors pay much more attention to the related works. Not the motivation and the high-level technical contributions of their work. This led me to feel that the entire work lacked technological innovation after reading the Introduction.\n4. As for the methods, it is just directly imply the SSMs to this work. I cannot see any design on how to effectively solve the problem of 'computational complexity'. Actually, the technical contribution is poor, and the overall pipeline is very similar to the previous work EMAGE[1].\n5. Could the authors explain why they only experimented on the BEATX dataset? As far as I know, TED[2] and TED-expressive[3] are also two commonly used co-speech gesture datasets.\n6. The author did not conduct experiments to demonstrate how to effectively reduce the computational complexity by using SSMs, which directly led to the unclear motivation of this work.\n\n\n[1] Liu, H., Zhu, Z., Becherini, G., Peng, Y., Su, M., Zhou, Y., ... & Black, M. J. (2023). Emage: Towards unified holistic co-speech gesture generation via masked audio gesture modeling. arXiv preprint arXiv:2401.00374.\n[2] Yoon, Y., Cha, B., Lee, J. H., Jang, M., Lee, J., Kim, J., & Lee, G. (2020). Speech gesture generation from the trimodal context of text, audio, and speaker identity. ACM Transactions on Graphics (TOG), 39(6), 1-16.\n[3] Liu, X., Wu, Q., Zhou, H., Xu, Y., Qian, R., Lin, X., ... & Zhou, B. (2022). Learning hierarchical cross-modal association for co-speech gesture generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 10462-10472).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**A1: Computational efficiency**\n\nThanks for your question. Our computational efficiency is mainly reflected in the inference time, which has been analyzed in section A.2 of the appendix. We leverage the linear computational complexity of Mamba and the sequence compression capability of VQVAE within our framework, which helps in reducing computational complexity. Additionally, our method holds a significant advantage in training time. The training time for a single epoch of MambaTalk is only 42 seconds. In comparison, CaMN and EMAGE require 493 seconds and 83 seconds per epoch, respectively, which amounts to 1173% and 212% of MambaTalk's epoch time. When considering the total time, MambaTalk also demonstrates a faster convergence rate, as our method only requires 100 epochs.\n| Method | Time per Epoch (s) | Epochs |\n|--------------------------|------------------|-----------------|\n| CaMN     |  493(1173%) |  120 | \n| EMAGE   |    83(212%) |  400 |\n| Ours   |      42(100%) |  100 |\n\n**A2: Writing improvement**\n\nThanks for your valuable comments. The goal of our work is also to use Mamba to address the difficulties of the co-speech gesture generation task. We will refine our statement in the abstract.\nFor the introduction section, we will simplify the description of related work and move this part to the related work section to make space for our motivation and the high-level technical contributions of our work (as illustrated in A3: Comparison with EMAGE).\n\n**A3: Comparison with EMAGE**\n\nOur work does not simply involve implementing Mamba for gesture synthesis. We found that directly applying Mamba would cause serious shaking problems. Therefore, we incorporated motion priors using VQVAEs and individual learnable queries for different parts of the body. Our solution differs from EMAGE, which involves extracting motion cues from masked body joints.\nAt the same time, the direct application of Mamba also has the issue that the limb movements of different body parts tending to be average. Therefore, we refine the design of spatial and temporal modeling in latent spaces by proposing a local-to-global modeling strategy and incorporating them with attention and selective scan mechanism into the design of our framework. Our work is also the first framework based on SSMs designed for co-speech gesture synthesis. We will refine our statements and make our framework clearer (e.g., adding pseudo-code) in our revised version.\n\nFor comparison with the transformer-based method EMAGE[1], our SSM-based method also has an advantage over EMAGE in all metrics, especially in terms of BC (16.72%), MSE (36.35%), and LVD (13.99%) metrics.\nOur work is not a disruptive innovation. However, in addition to using SSMs instead of transformers, we have also developed numerous adaptive designs for SSMs to make it work.\n\n**A4: Reasons for choosing BEATX and the generalizable benefits of our method**\n\nThanks for your question. We experiment on the BEATX datasets since our work focuses on holistic gesture synthesis.\nHowever, most current datasets only include movements of specific body parts, not the entire body.\nFor example, the TED[2] and TED-expressive[3] datasets that you mentioned only focus on upper body gesture synthesis. TED includes 10 upper body joints, while TED-expressive includes 13 upper body joints and 30 finger joints.\nMoreover, TED and TED-expressive datasets are based on the 3D pose estimator ExPose for extracting gestures, which contain some errors in the ground truth.\nTherefore, we conducted experiments using another motion capture dataset called BEAT to validate the generalizable benefits of our method. This dataset contains data on upper body and hand movements to validate the generalizable benefits of our method for specific parts of body movements.\nDue to time constraints, we only conducted preliminary experiments. The results are shown in the table. Our method demonstrates a significant improvement compared to the baseline, which also validates the effectiveness of our approach. We will conduct more comprehensive experiments and include them in the revised version.\n| Method | FGD$\\downarrow$ | SRGR$\\uparrow$ | BeatAlign$\\uparrow$ |\n|--------------------------|------------------|-----------------|--------|\n| CaMN (baseline)     | 123.7   | 0.239       | 0.783   |\n| Ours   |       51.3        | 0.256             | 0.852     |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uyQCDb2tZk", "reviewer": "Reviewer_qFoz", "summary": "This paper explores the application of state space models (SSMs) to co-speech gesture generation. The authors identify the computational challenges and jittering issues associated with the direct application of SSMs to gesture synthesis. To address these, they propose a two-stage modeling strategy with discrete motion priors and hybrid fusion modules. The first stage involves learning discrete holistic gesture priors with multiple VQVAEs, and the second stage refines latent space representations using local and global scans. Experiments demonstrate that MambaTalk outperforms state-of-the-art models in generating natural, rhythmic, and contextually appropriate gestures.", "questions": "See the \"Weakness\" section.", "limitations": "See the \"Weaknesses\" section. If possible, I suggest adding more visualization results and videos to the supplementary materials. Additionally, please note that the final score for this study is not solely determined by the peer reviewers' discussion. If the authors can address my main concerns, I would be willing to raise the score.", "rating": 6, "confidence": 2, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. The paper is well-structured and clearly written.\n2. It seems that MambaTalk is the first to explore the potential of the selective scan mechanism for co-speech gesture synthesis.\n3. The methodology is thoroughly validated through extensive experiments, including both subjective and objective evaluations.", "weaknesses": "1. Limited novelty. Integrating the local and global scans seems not novel. More discussion on the comparison between MambaTalk and baseline methods is needed.\n2. Limited datasets for method evaluation. Only the BEATX-standard dataset is adopted to evaluate the performance of the method. How about the zero-shot quantitative comparison on other datasets, such as the BEAT dataset or the RAVDESS dataset, which take emotion into account?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**A1: Comparison with baseline methods**\n\nThanks for your advice.\nComparing with the baseline methods, we not only propose a method that applies a selective scan mechanism on co-speech gesture synthesis with local and global scans (a novel method not found in previous work), but we also consider the different motion patterns of different parts of the human body. We found that directly applying Mamba would cause serious shaking problems. Therefore, we leverage VQVAE and learnable queries to incorporate motion priors. Additionally, the direct application of Mamba also has the issue that the limb movements of different body parts tend to be average. Therefore, we integrate attention and selective scanning mechanisms into our framework to model spatial and temporal relationships. We will refine our statements and make our framework clearer (e.g., adding pseudo-code) in our revised version.\n\nComparing to baselines, we have an advantage on all metrics, especially on BC (16.72%), MSE (36.35%), and LVD (13.99%). Our method also has efficiency in training and inference time (Appendix A.2). The training time for a single epoch of MambaTalk is only 42 seconds. In comparison, CaMN and EMAGE require 493 seconds and 83 seconds per epoch, respectively, which translates to 1173% and 212% of MambaTalk's epoch time. When considering the total time, MambaTalk also boasts a faster convergence rate, with our method requiring only 100 epochs.\n| Method | Time per Epoch (s) | Epochs |\n|--------------------------|------------------|-----------------|\n| CaMN     |  493(1173%) |  120 | \n| EMAGE   |    83(212%) |  400 |\n| Ours   |      42(100%) |  100 | \n\n**A2: Method evaluation**\n\nThanks for your advice. Conducting a zero-shot comparison between the BEAT dataset, which uses BVH to represent gestures, and RAVDESS, represented by videos, would be challenging due to the use of SMPLX in BEATX.\nTherefore, we retrain our method using the BEAT dataset to consider emotions and verify the broad advantages of our approach for upper body movements. To incorporate emotions, we utilize embedding to convert them into style features and employ adaptive layer normalization to integrate them into the latent space of our framework.\nDue to time constraints, we only conducted preliminary experiments. The results are shown in the table. Our method demonstrates a significant improvement compared to the baseline, which also validates the effectiveness of our approach. We will conduct more comprehensive experiments and include additional details in the revised version.\n| Method | FGD$\\downarrow$ | SRGR$\\uparrow$ | BeatAlign$\\uparrow$ |\n|--------------------------|------------------|-----------------|--------|\n| CaMN (baseline)     | 123.7   | 0.239       | 0.783   |\n| Ours   |       51.3        | 0.256             | 0.852     |\n\n**A3: More visualization results**\n\nThanks for your kind advice. We will include more visualization results and videos in the supplementary materials in our revised version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper originally got BA, BR, R, and WA, and, after the rebuttal period, two reviewers raised the ratings, leading to final ratings of BA, WA, BR, and WA. 

Reviewers recognized the major contribution of this paper in applying state space models (SSMs) into co-speech gesture generations. However, reviewers raised several concerns, including unclear motivation of the work in applying SSMs and insufficient experiments as the paper was tested on a single dataset. Following the rebuttal and discussion period, most concerns were addressed, and two reviewers increased their ratings, resulting in three positive and one negative review.

The AC also supports accepting the paper, recognizing its contribution to applying SSMs for the co-speech gesture generation. The AC strongly suggests the authors to follow the reviewers’ feedback in the camera-ready version.
Decision: Accept (poster)