PyraMotion: Attentional Pyramid-Structured Motion Integration for Co-Speech 3D Gesture Synthesis
Authors: Zhizhuo Yin
Yuk Hang Tsui
Pan Hui
Venue: NeurIPS 2025 poster ()
Keywords: Co-Speech Motion Synthesis
3D
Generation
Representation Learning
Primary area: applications

Abstract: Generating full-body human gestures encompassing face, body, hands, and global movements from audio is crucial yet challenging for virtual avatar creation. Existing systems tokenize gestures frame-wise, predicting tokens of each frame from the input audio. However, expressive human gestures consist of varied patterns with different frame lengths, and different body parts exhibit motion patterns of varying durations. Existing systems fail to capture motion patterns across body parts and temporal scales due to the fixed frame-count setting of their gesture tokens. Inspired by the success of the feature pyramid technique in the multi-scale visual information extraction, we propose a novel framework named PyraMotion and an adaptive multi-scale feature capturing model called Attentive Pyramidal VQ-VAE (APVQ-VAE). Objective and subjective experiments demonstrate that the PyraMotion outperforms state-of-the-art methods in terms of generating natural and expressive full-body human gestures. Extensive ablation experiments highlight that the self-adaptiveness integration through attention maps contributes to performance.

PyraMotion: Attentional Pyramid-Structtree Motion Integration for Co-Speech Gesture Synthesis
Zhizhuo Yin', Yuk Hang Tsui", Pan Huil.2
'Hong Kong University of Science and Technology (Guangzhou) Guangzhou, Guangdong, China "Hong Kong University of Science and Technology, Hong Kong SAR cn, zyin1900connect.. hkust www.hhukttttt.eeeeeee wz..gz...d...... en, yhtsui@connect. ust panhui@hkust- edu.
Abstract
Generating full-body human gestures encompassing face, body, hands, and global movements from audio crucial yet challenging for virtual avatar creation. Exist- ing systems tokenize gestures with fixed frame-count for each token, predicting tokens of single scale from the input audio. However, expressive human gestures consist of varied patterns with different frame lengths, and different body parts hibit motion patterns of varying durations. Existing systems fail to capture motion patterns across body parts and temporal scales due to the fixed frame-count setting of their gesture tokens. Inspired by the success the feature pyramid technique in the multi-scale visual information extraction, we propose novel framework named PyraMotion and an adaptive multi-scale feature capturing model called Attentive Pyramidal VA-VEE (APVQ-VAEE VAE). Objective and subjective experiments demonstrate that the PyraMotion outperforms state-of-the-art methods in terms of generating natural and expressive full-body human gestures. Extensive ablation experiments highlight that the self-adaptiveness integration through attention maps contributes to performance.
Introduction
Among all human communication, the movements of all body gestures approach to conveying thoughts. Such non-verbal signals provide more information than voice and context, enhancing the expressiveness and vividness of the speech, thus allowing listeners to gain a more comprehensive understanding of the intentions, emotions, and motivations of the speaker 32 228]. Existing studies [T5 suggest that expressive gestures make avatars more intimate and trustworthy. In the metaverse [17), the naturalness of the avatar's 's full-body gestures, such as face micro-expressions, intricate body gestures, and movement trajectory, could impact the sense of realism, presence, and satisfaction. Therefore, generating natural full-body gestures is a valuable and challenging task that serves as a critical component for creating realistic digital humans 29
important
The most recent co-speech gesture generation work
focuses on utilizing VQ- VQ-VAE 34
method that encodes continuous motion a discrete latent space while preserving the original motion information. However, in these VQ- YQ-VAE-based methods, one shared approach is that each motion token exclusively represents static human pose in single scale with fixed frame count. Since natural gestures convey semantic information through a series motion patterns with varying durations, shown in Figure 1] this assumption severely hampers the model's ' ability to learn natural gestures that express different intricate semantic information. This assumption also constrains the ability of the token predictor model capture expressive patterns of audio and transcript representations with inconsistent durations and generate accurate corresponding motion tokens.
Inspired by widespread pyramidal multiscale design exploited in computer vision 31
20
26 We
propose the PyraMotion framework to address the above problem by adaptively capturing motion
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Me, first...
like to do relaxing.
is very important...
work from Monday to Friday
Deictics
Emblems
Iconics
Metaphorics
Figure 1: Illustration of gesture sequences with expressive motion patterns in different durations.
patterns from coarse fine granularities. Experiments validate that PyraMotion outperforms existing state-of-the-at studies on reconstructing smooth body gestures from audio and textual information.
One of the key components of PyraMotion, the Attentional Pyramidal VQ-VAE (APVQ-VABE model encodes the motion patterns different body parts at different scales into a shared discrete latent space several series of motion tokens and reconstructs motion sequence by attentively fusing these motion tokens. Experiments also demonstrate that APVQ-VAE outperforms the traditional VQ-VAE in gesture reconstruction, beat alignment, and diversity. Additionally, the attention maps reveal that different body parts exhibit distinct attention patterns to tokens across varying time durations, highlighting the necessity introduce pyramidal motion representation.
To summarize, the main contributions are as follows:
This work introduces novel framework called the PyraMotion, which generates dynamic and natural full-body gestures using multi-modal information. To the best our knowledge, this paper is the first to propose encoding motion patterns with varying temporal scale into shared discrete latent space as motion pattern tokens.
This work proposes the Attentional Pyramidal VQ-VAE (APVQ- VAE), novel model designed to tokenize motion patterns across multiple temporal scales using a shared codebook. Experiments validate the superiority of APVQ-VAB compared with vanilla VAE in capturing motion patterns at different temporal scales and effectively accommodating the diverse motion information needs of various body parts during generation.
Extensive experiments show that Pyramotion outperforms state-of-the-art methods in generating audio-driven gestures, as confirmed by
both
objective evaluations and subjective human studies.
2
Related Works
2.1
Human 3D Motion Generation Approaches
Early attempts generating human gestures involved using rule-based algorithms 4 15 appropriate gestures from a database based on input and blend motion clips with smooth algorithms. However, due to limited data and varying speech details, such workflow suffers from inconsistency between the audio and motion, and the unnatural transition among different motion clips.
to select
Deep generative models, such as Variational Autoencoders (VAEs) [16], Generative Adversarial Networks (GANs) [13], and transformers [36, 91. have shown promise in addressing the limitations of rule-based methods by capturing complex data correlations in a shared vector space. However, direct generation from a continuous latent space is sensitive to input noise, presenting challenges in practical applications. VAE-based methods also face the problem of "posterior collapse," restricting their ability to generate diverse gestures based on audio cues. These pose challenges for practical implementation in real-world scenarios.
to project motion patterns to a discrete latent codebook, transforming the motion generation problem into an autoregressive token prediction task conditioned on audio. These methods utilize VQ-VAE to capture motion patterns in single scale with fixed frame count and project them into discrete latent space, represented by motion tokens.
Recent work 39 30 1 22
has utilized VQ-VAE

---

Then, they utilize token predictor network to transform the audio and text representations into the motion token series and reconstruct human motion of each frame by applying the decoder of VQ-VAE. While the VQ-VAF enhances model robustness and generation quality, the assumption that each token represents poses of gestures in single scale limits the generation of expressive motions varying in frame counts.
To address the gap, this paper introduces variation of VQ-VAE called Attentional Pyramidal VQ- VAE (APVQ-VAE) to tokenize dynamic motions in different temporal scales, enabling the generation of more detailed and natural gesture sequences.
2.2 Audio-driven Full-body Gesture Generation
Given the complexity and variability of human motions, recent research has focused on generating gestures for specific parts of the body, such as the face [10 37 upper body (including arms, wrists, and hands) [41 [2), and lower body (including overall movements) 38 individually. While these generative models have shown promise in capturing unique motion patterns, they lack the versatility to generate movements across all body parts due to the diverse range of motion patterns among different body parts. There is need for unified framework for motion modeling.
25 6 approached generating full-body gestures by generating each part, the face, upper body, hands, lower body, and global movements first and combining these components to form full-body gestures.
Recent studies 22
However, existing methods overlook the need for different temporal information in modeling and generating motions, which lack the ability to capture motion patterns in different temporal scales and extract semantic correlations between different body parts.
In this work, we utilize the Attentional Pyramidal Token predictor to extract the coase-to-fine-graiiee pattern information within the audio and transcript embeddings. It can accurately predict the motion token series by exploiting both body part information and semantic correlations between different body parts through multi-scale temporal cross-attentive network.
3 Methodology
APVQ-VAE
TONS
Pynenda TCNK
Kornch
Cachet @) Reneh
PyraMotion
Bug Dong LEPINE
Quee
Tia
Hello thore,
Speech
Atlention
Se Nettwrok
Cemoooa and Cross Croso-Attertii Aete Mosoottto
huper
Haper
Queerr
footy
Hine
Teat
Huve
Queer
Figure 2: The Overall Workflow of PyraMotion. Stage 1: APVQ- VAE learns the discrete latent representations of motions, denoted as tokens, and reconstructs the motion from the pyradical token series via decoder. Stage 2: The PyraMotion framework is trained to predict the pyradical token series of motion from audio and reconstruct the motion via decoder in APVQ-VAE.
This section presents the development of the PyraMotion framework, as depicted in Figure 2 Initially, we formally define the problem of audio-driven gesture generation. Subsequently, we outline the workflow of the PyraMotion framework and delve into the structures of Attentional Pyramidal VQ- DAE and Pyramidal Token Predictor.
3

---

3.1 Problem Formulation
For given audio a RR sTaudio/ fps gesturess where represents the expected total frame counts of generated gesture sequences, sf represents the sample count per frame, aaadiio denotes the sample rate of the given audio, and fpSgestures denotes the frames per second of the generate gesture sequences. The task is to generate the holistic 3D gesture of the human body E
from the given audio. The dimension of is formed by L multiplied by the sum of 55 joints in Rot6D, 100 dimensions of facial expressions in FLAME parameters, foot contact labels, and 3 global translation parameters.
3.2 PyraMotion
The overall workflow of PyraMotion can be divided into two stages: 1. Training APVQ- to learn discrete latent representations of motions for encoding and reconstruction. 2. Training the Pyramidal Token Predictor to predict the discrete latent representations from audio.
In the first stage, motion representations are tokenized at various scales using multiple Temporal Convolution Networks (TCNs). These tokens are mapped to shared codebook for semantic alignment, then reconstructed to original motions by applying Transpose TCN and aggregation.
In the second stage, the goal is to predict token series from input audio and text transcripts to recon- struct corresponding motion sequences. To account for the diverse motion patterns across different body parts, the audio and text representations are transformed into four separate latent embeddings for reconstructing the face, upper body, hands, and lower body. The four latent embeddings are further processed and quantized into token series of varying temporal scale, then decoded by the attentional pyramidal decoder in the trained APVQ-VAR reconstruct the motion sequences.
3.3
Attentional Pyramidal VQ-VAE (APVQ-VAE)
Normal VQ-VAE consists of Encoder ε, Quantizer Q, and Decoder D. The Encoder projects the raw gesture sequences (g) into the latent representation space as encoded embeddings in single scale with fixed frame count, and then the Quantizer classifies the embeddings into discrete tokens according the codebook based on the embedding distance in the latent space. After that, the decoder reconstructs the motion sequences ğ from the quantized embedding f, which is generated by looking up the codebook with quantized token series.
f = -= Q(f), r, g..
(2ookup(2,9).5555 = D((f)
R Touree Z Roxd gg E R (006++100040
where T Trource, d, and C are the total frame count of the input motion sequence, the dimensions of embeddings, and the size of the codebook, respectively.
In APVQ-VAE, we utilize n TCN networks representing n levels of temporal scale to generate the encoded embedding F, which consists of n embedding series with various lengths.
[f. f,] = Egg = [TCN, (g) ..., (C),,g)
where f, E RTToouree/ 2 )xd x denotes the embedding series with i from 1 to n.
In the quantization stage, the encoded embeddings are first quantized token series Q, then formed the quantized embedding series F by looking up the codebook. All embedding series share the same codebook Z to ensure consistency in the embedding semantics.
[qn qn. = Q(F), = lookup(Z, ()
where qi E RTT "oouree/2" denotes the token series.
In the decoding stage, instead of using TCN as previous VQ-VAE- VAE-bAseee methods 22 1 we utilize the Transpose Temporal Convolution Network (TransTCN) to decode the quantized embeddings into gesture sequences using the same scale of reception field in the encoder. Thus, multiple gesture sequences are reconstructed from quantized embeddings in different temporal scales. To further enhance the reconstruction, as shown in direct reconstruction from the mean of the stacked gesture sequences with the attention attention-based residue,
Eq. 4 residual connection introduced by combining the

---

enabling the model to capture both coarse and fine- siee-grrindd temporal dynamics effectively. Further details on this design and its effectiveness are discussed in |section 4.5)
m = ( = Attention AMtenntionMMcan(F5 stack)) F Stttcks F stack) + Mcan (ean(Fstoch stock)
(4)
F stack [CransTCN (f.),... TransTCN (C,,.f.).
Notably, for each convolution operation in TCNs and TranTCNs, the output embeddings are nor- malized by dividing the kernel size shown in En. to mitigate the scale differences stemming from varying kernel sizes, thereby enhancing the stability of the training process.
Conv(f, ks, 8, s,pp/Ks
where Conw() denotes convolutional operation, ks denotes kernel size, 8 denotes stride, and p denotes padding. The loss function of APVQ-VAE VAE is as follows:
LvQ VAE VAF Cree (g, 8) + Luet (g':
+ (g".
sg F]
F + |
sg F
where Cree is Geodesic [33 loss, and Leel, Lace are LI losses. sg is the stop gradient operation and
the weight of commitment 34 loss is set to
3.4 Pyramidal Token Predictor
In the Pyramidal Token Predictor, we need to predict the corresponding pyramidal tokens series Q [q..... Pt....... 9n] from audio features and text contents.
Audio and Context Feature Fusion we employ onset o, and amplitude a and combine them as rhythmic audio features E TR'x For the input transcript text, we transform it into content features с E R using the pre-trained model 3 Then we utilize the attention mechanism to fuse the audio features and context features.
For the input audio features following previous work (2].
Q Poftmax(MMP(II.1>)) C1:r)
f..T
F1:T + - a) x C1:7
where T denotes the total frame counts of the audio after sampling and the expected motion sequences. a Rxdx <d is the element-wise attention coefficient.
Latent Generation and Token Prediction According to the previous studies [22] in the holistic gesture generation task, there is only weak correlation between the distribution of facial expression and body motion. Therefore, we propose to exploit two independent feature extraction and token prediction workflows for facial expression and body motion separately, based on independent audio- text representations fface and ghody, For the latent representation prediction, we first combine the audio-text embedding with the learned speaker embedding and project to the dimension of the hidden vector:
h Jace (f ace + Pf))
hints SAN g + Pt)
body h hints + TCAT(h hinte + pe,, phodyy
harts E Tx denotes the hidden representation of part motion, where parts can be face or body. fparts E x'd denotes the fused audio-text embedding, and P, E IR denotes the learned speaker embedding. Body hints h hints E R2xx are encoded from the masked gesture sequence g, where the first 8 frames are ground truth frames. SAN denotes a self-attentive network.
To better capture the various motion patterns of different parts of the body, the hidden representation of the body is further projected into three different latent spaces, including the upper body, lower body, and hands. Then, we construct two types of pyramidal representations to generate pyramidal latent. The first type averages the consecutive representation and the second type uses the isomorphic TCN network in the APVQ-VAE encoder. The TCN network can extract similar information as the APVQ- VAE encoder and generate pyramidal motion latent.
h upper hands
MLP(h body)
H parts mean
[h parts
parts
= (a parts (0....t-1)) 1)),-., o (h/ pants (2
parts
[h 0r h parts
parts
TCNN (hpert)) )..,, TCN,-1 (haarte)]
ton
n-1
5

---

where parts can be face, upper, lower, and hands. o denotes the average operation and meansy h ien
E RR xh denotes the ith embedding series with 2 frames.
After constructing the embedding series with different scale, a temporal aoss-att Transformer decoder TCAT is applied to capture the correlations between the above two representations. Then 1 1-layer MLP is used to project the dimensions into the reconstruction latent Hparts
Aports
MLP parts
(10)
meon
We optimize the learned reconstruction latent by applying the MSE loss:
Cperts MSELoss
[
Epptta Fverts)
(11)
where Eparts is the learned latent representation APVQ- VAE of corresponding body parts from the ground truth. After learning the mutual information between the two representations, further learned the correlation among body parts. We sum the TCN latent series all body parts together to form full-body pyramidal latent and conduct aross-attention operations with the mean latent:
H
H full Hupper
+ ton
hands + H lower
(12)
Hparts (CCAT(HPOrd) H ff/ll)
[hg Ipprtt
Parts)
meon!
where Hearts denotes the latent embedding of each part of the body in different temporal scale and parts can be upper, lower, and hands according to previous body part separation. h pants E
RT/2 xh represents the latent embedding of body parts in different scales.
After generating the latent representations of each body part, we quantize the latent representations by predicting the corresponding tokens iteratively from coarse to fine granularity. Such quantization operation projects the learned continuous latent to discrete latent space represented by tokens for better reconstruction performance.
parts MLP MLP(h (aparte + quarle), i [0, 2]
(13)
Q parts = (9h parts
parte
where grarte T/2'xxx represents the predicted possibility distribution of motion tokens for zth granularity, C denotes the codebook size. We use cross- tross-entropy loss to optimize the learned token index distribution, where Quarts is the learned token of body motion in APVQ-VAR VAE.
cls
CrossEntropy ceatiroopy@@rarra Qparte Q Qurrts)
(14)
4 Experiments
4.1 Datasets and Training
We evaluate the ability of our method to generate holistic 3D gestures from speech on diverse and expressive dataset BEATZ 23 collected from mocap equipment. This public dataset contains 76 hours of high-quality, modti-modal data captured from 30 speakers talking with eight different emotions. Following the settings of existing work [2 25), we conduct the experiments on the BEAT2-Standard Speaker2 with an 85%/7.5%% train/val/test split.
4.2 Evaluation Metrics
For the body motion generation, we adopt FGD to measure the similarity between the generated gesture and the real gestures. To evaluate the Diversity [18] of generated gestures, we calculate the LI distance between different gesture clips. In terms of audio-motion synchronization, we utilize the Beat Align [9 measurement. For the facial expression generation, we use Mean Square Error (MSE) to measure the vertex position distance and I.1 Vertex Difference (LVD) 37 to measure the L1 distance between the generated facial expression and the ground truth facial expression. The results are reported as the mean value and the standard deviation computed from times of independent runs. The significance is also reported with the p-value.

---

4.3 Comparison Methods
We compare our PyraMotion with the following classic and state-of-the art methods of talking
head generation and body gesture generation: S2G [2]. Trimodal [4].. HA2G [24], DisCo [21].
CaMN 23]. Diffusestylegesture [38], HoloGest 77. ..RA-Gesstrrr 127, AMUSE 8 MambaT-
alk 43 Habibie et al TalkSHOW 44), DifTSHEG ( EMAGE [22]. and ProbTalk [25]
SynTalker [.. For Habibie al. [14, TalkSHOW 4Q) we use their reported version, which the
upper body generation. Meanwhile, we cite the results of their full-body version reproduced by [22
4.4 Overall Comparison
In this part, we compare the overall performances of M3G with classical and state-of-the-ar audio- driven gesture generation methods. In [Table 1 Habibie et alF and TalkSHOW' denotes the reported
performance of reproduced full-body motion generation in [22] denotes the results are directly adapted from their original paper due to the same experimental settings, thus no std values are
reported. AMUSE denotes the reported performance is by the reproduced evaluation code by
ourselves.
FGD x10
BA 8A010-1
Diversity
MSEX10
LVD
S2G (CVPR 2019)
27.87
4.827 + 0.138 6.022 0.097
Trimodal (TOG 2020)
12.13
5.762 0.063 7.513 0.073
HA2G (CVPR 2022)
12.32
1.022
8.626 0.016
DisCo (ACM MM 2022)
9.484
6.439+ 0.027
9.912 +0022
CaMN (ECCV 2022)
6.967
6.628 0.018
11.18 + 0.089
DinStyleGesture (IJCAI 2023)
8.866
7.239 + 0.089
11.13 + 0,077
AMUSE' (CVPR 2024)
12.11
8,318 + 0.052
14.93 + 1.497
Syn'alker'' (MM 2024)
5.366
7.812
13.05
HoloGest" (3DV 2025)
5.341
7.957
14.15
RAG-Gesture (CVPR 2025)
8.08
7.34
11.97
Habibie et al (IVA 2021)
9.040
7.716
8,213
8.614
8.043
TalkSHOW (CVPR 2023)
6.145
6.863 + 0.008
13.12
0.156
7.791
0.044
7.77
0.052
TalkSHOW (CVPR 2023)
6.209
6.947
13.47
7.791
7.771
DifTSHE (CVPR 2024)
8.986
7.142
11.91
7.665
8.673
EMAGE (CVPR 2024)
5.643
7.707 + 0.004
12.92
+ 0.198
7.694
0.076
7.593
0.062
ProbTalk (CVPR 2024)
5.040
7.711
13.27
8.617
MambaTalk (MMM 2025)
5.366
7,812
13.05
6.289
6.897
PyraMotion
4.612
7.420 0.008
13.42
0.020 7.176 0.028
7.270
0.011
p-volue
0..005
0.0001
0.0001
< 0.001
Table 1: Overall Comparison of various methods. The best performance of each metric is in boldface fonts, and the second one in font. The sign f denotes that the larger the value, the better it is, while the sign J. is the reverse, -> means the metrics measures some expressive aspects of the motion, while higher value is not necessarily correlated the generation accuracy. The standard deviation is calculated across 10 epochs after reaching best performance.
As shown in Table our proposed PyraMotion surpasses or achieves state-of ffite-of-the--e methods across all metrics. Notably, it achieves significant improvements in reconstruction accuracy like FGD, MSE, and LVD, demonstrating that its generated full-body gestures align more closely with ground-truth motion than existing approaches. For non-deterministic metrics like Beat Align and Diversity, PyraMotion delivers comparable or suboptimal performance.
These results highlight that PyraMotion's use of Attentional Pyramidal VQ-VAE adaptively capture multi-scale patterns across body parts enables it to model broader range of motion patterns. This capability enhances the model's ability to generate higher-quality, more accurate motion sequences compared to other methods.
4.5
Ablation Analysis
Key components In this ablation experiment, we propose five variants of PyraMotion: w/o Separate Body uses unified latent embeddings for different body parts instead of separate latent
"The source code will be released GitHub after acceptance.
7

---

representations. w/o Text generates motion sequences only based on the audio signal. w/o TCN Encoder discards the TCN network in equation (12) to disable the temporal information extracting process in the token predictor; w/o Full-Body Latent discards the fusing operation of full-body latent in equation (15) and generates each body part's latent embedding only based on their own parts' features; w/o Trans TCN substitutes the TransTCN structure in equation (4) with the TCN structure in equation (2) following previous work [22]
Table 2 reported that the PyraMotion significantly outperforms or performs similarly with all variants, demonstrating the contributions of these components. Moreover, the results indicate that replacing the TransTCN structure leads to more significant decline in performance compared with the other two variants, demonstrating the indispensability of the token decoding process in the overall workflow. The absence of Full-Body latent mainly affects the body's 's reconstructing performance, which might be caused by the lack of mutual information among different body parts. The w/o TCN Encoder performs significantly worse than PyraMotion on facial reconstruction and body diversity, indicating the ability of the Pyramidal TCN Encoder to model diverse types of motion patterns and facial expressions from audio.
FGD FGD
BCX10-1
Diversity!
MSE*10701 1
LVDx10-1 4
w/o Separate Body
8.926 + 0.021 6.843 = 0.008
10.838 = 0.024
0,003
8,153 + 0.019
w/o Text
6.677 0.023
7.374 1774-00.00 0.012
12,89
+ 0.04
8.612
7,852 + 0.011
w/o TransTCN
6.178 0.05
7.132
0.005
12.869
0.05 0.83
w/o Full-Body Latent
5.152 + 0.034
7.232
- 0.002
13,308
0.062 7.440
+.000
7,518 + 0.012
w/o TCN Encoder
5.178 0.013
7.436 0.004
12.499
= 0.117 7.607
1
7.589 + 0.039
PyraMotion
4.612 0.014 7.420 0.008
13.42
0.020
7.176
0.028
7.270 + 0.011
p-valuee
0.0005
> 0.1
< 0.0001
<< 0.0001
0.0001
Table 2: Ablation Experiments for Proposed Components.
Reconstruction Performance Comparison of APVQ-VAE This section evaluates the performance of APVQ-VAE in learning and reconstructing compared the widely used VQ-VAE in existing methods. A variant of it called Mean Pyramidal VQ-VAE (MPVQ-VAE) is also introduced, which
substitutes the attentional fusing operation on the pyramidal token series with an averaging operation.
presents the joints' rotation Mean Square Error (JRMSE) for each body part compared to the ground truth sequences. The second part of the table shows the metries used in Table I to assess the reconstruction performance based on the encoded tokens in APVQ- VAE.
Table 3
Face
Upper* i "I Hands 10 '1 Lower 10"1 *1 Global [1]
FGD-W01 BC
Diversity!
MSE
LVD-
VO-VAE MPVQ-VAF VAVQ-VAF
2.100 1. 0.009
5,209 0.023 7,105 0.015 3,338 0.023 4,418 0.011
3,300 0.006
7,88%
0.014
12.452
0:009 0.24 0.002
2087 0,010
1.36 1: 0.022
29921:
0.166 4.316
0.024
2.502
0,025
4.163
0.820
1.497
0.017
7.143
0.0/2
12854
0:008 0.44 1 0.008
2011 0,012
1844 + 0.022 2,655 0,018 4,662 0.011 2,209 0,925
5.129 0.091
1,296 0,028
7.33
0,015
12864
0,008 0.279
0.08
1,525 0,019
0.0001
0.0001
0.0005
0.0001
0,0001
< <.000001
0.0001
0.0001
0.0001
0.8001
Table 3: Experiments for Reconstruction Errors.
The experimental results reveal that both our APVQ-VAE and its mean-fusing counterpart MPVQ- VAE significantly outperform conventional VQ- RQ-V in motion pattern tokenization and reconstruc- tion, confirming the superiority of modeling through pyramidal token series.
Notably, APVQ- VAE surpasses the mean- frannn version, MPVQ- VAE, underscoring the importance of its attentional fusing operation across multiple scales. To interpret this mechanism, we visualize attention maps from the best epoch (lowest JRMSE) in Figure 3 revealing distinct scale preferences across body parts. The face predominantly utilizes fine-grained features (kernel size 1), likely due to continuous mouth motions. In contrast, the lower body balances attention between kernel sizes and 8, corresponding to coarse leg dynamics. Mcanwhile, hands and upper body exhibit decreasing preference from fine to coarse scales with attention weights ranging from 0.2 to 0.3, reflecting their need for both expressive details and structural motion patterns. These findings demonstrate that the attentional fusion mechanism in APVQ- EPVQ-VAE adaptively addresses the differential reliance on multi-scale features across distinct body regions.
Efficiency Analysis In this section, will provide the theoretical analysis towards the computa- tional complexity of the proposed workflow compared with vanilla VQ-VAE.
Theoretically, the computational complexity of our APVQ-VAE is:
N
Nd
0
(2
O(PNd)
2p
8

---

Hedriy of dln lirkki
Watr chotoo Darteelnn
4M
(b) Upper Body
(c) Hands
(a) Face
(d) Lower Body
Figure 3: Attention Map Visualization
where d is the latent dimension, is the total number frames, and P is the number of pyramid layers. Thus, the time complexity of APVQ-VAE is constant multiple of vanilla VQ-VAE.
For the PyraMotion model, before the pooling operation, the data processing steps are identical with single scale version. After the pooling operation described in equation Eq. 9 the time complexity in each step is in multi-scales, which follows the rule a geometric series with regard to the increasing scale numbers. For the Temporal Cross-Attention (TCA) operation, the time complexity of its single-scale version is:
On + dd
where N is the clip length and d the hidden dimension. The multi-scale Temporal Cross-Attention operation has complexity:
2 2F
N 2D
d ))
4 ) O(2(N2(2 + O N2d)
2Nd
similar conclusion will also reached on the other multi-scale operations. Considering the total time complexity before the pooling operation, PyraMotion's 's time complexity has the same time-complexity level with less than 2x of the single-scale version. To validate these theoretical conclusions, measured the inference times for single-scale EMAGE and multi-scale PyraMotion on over minutes of motion data. The experimental results are shown Table 4
Table 4: Inference Time Comparison
Method Inference Time
EMAGE 22 - 1.143
PyraMotion 41 + 1.822 s
Selection of Pyramid Layer Number This section evaluates the influence of pyramid layer design on motion reconstruction performance. Adhering to the feature pyramid framework (20), which leverages exponentially increasing kernel sizes reduce information overlap across layers, we configure successive layers with the scale sequence [1, 2, 4, 8, 16). To preserve information density and mitigate noise introduced by the padding operation, the pyramid layer downsamples features by factor of 2, corresponding to the dilation of their kernel sizes shown in Figure 2
In this experiment, we search for the optimal number of layers balance computational efficiency with hierarchical feature extraction. The results, summarized in Table 5] show that increasing the layer number initially boosts performance but eventually leads to a decline. More layers enable PyraMotion with higher capability to represent complex motion patterns with varying durations. However, overcomplex pyramidal token series may cause overfitting and the difficulty in predicting tokens from audio, resulting in comparative reconstruction performance with far worse generation performance from audio, such as the 5 Layers variant.
1ac
Upper=ti">
Hands -
Liwe Global -13"1
FGD
BC BC
Diversity?
LYD-
Layyer
1.100 0.009 5.09 + 0.078 2.103 + 0.017
3.335 0.23
4,488
0.111
3.302
0.016
7.488 + 6.014
12,430 + 0.009 0.524 0.022
2.087
0.017
Layers
0.026
3.262
0,022
5949
0.015
2,566
0.077
4.983
0.019
6.968
0.026
1.342
0.021
12,979
0.005 0.411 0.005
1.358
0,010
Layers
1352
0.088
2993.
0.025
5.218 1 0.021
2484
0.05
5021
0.018
5.627
0.017
7,254
1 0.022
13,723
0.011 0.391 1 0.1000
1616
0.016
1.044
0.023
0.011
1.2%6
0,028
12
0.008
0JNN
2955
0.014
0.029
12.263
0.008
1.458 0.012
0.0001
0.0001
0.000
0.0001
Table 5: Experiments for Pyramid Layer Number Selection
5 Perceptual Study
To evaluate the naturalness of our generated results from the subjective perception, we conducted
A/B testing following [40] by comparing the generated motions by PyraMotion with other methods,

---

including CaMN, EMAGE, APVQ-VAE Reconstruction, and Ground Truth. For CaMN, since it does not generate the facial expression, we directly use the ground truth as its facial expression.
We randomly sampled 40 clips from the generated video and compared them pairwisely. Highteen participants took part in this study. Specifically, participants are asked answer A or B to the following questions: Which clip do you think is more natural?
3.7%
Comparion Malridi
Paroe Maggeaage 1S6
69.7
90.0%
$4.5%
10.0%
te SMADE
Pyntecies so 67
Paralaaion APRG
(a) Perceptual Study Results
(b) Qualitative Results
Figure Perceptual Study
Figure 4a illustrates that participants favor PyraMotion existing work CaMN, and EMAGE, indicating the superiority of PyraMotion in generating more natural motion from audio. Moreover, in proportion of cases, PyraMotion outperforms the APVQ-VAE Reconstruction, which generates the motion from encoded tokens in APVQ- VAE, showing the closeness of generation quality between them. GT outperforms PyraMotion without doubt, showcasing the space for improvements.
6 Limitation, Future Work
The limitations of the proposed method are follows: It cannot adaptively adjust the number of pyramid layers towards different datasets. The number of pyramid layers is a hyperparameter in this work, which might restrict its generalizability and usability towards real-world usage scenarios. 2) One observation based on the existing work, EMAGE, and our work is that overly focusing on integrating the correlation among different body parts could potentially cause a decrease due the interference of different body parts' motion distributions. While fully independent modeling of each body part would cause unnatural behavior. Thus, a self-adaptive mechanism of adjusting the ratio between these two perspectives would be potentially beneficial for future studies.
7 Conclusion
This study introduces an audio-driven holistic gesture synthesis framework named PyraMotion, method that can extract pyramidal features from audio and exploit these features to generate natural and realistic holistic 3D gestures. To achieve this, we propose the APVQ-VAL which encodes the pyramidal feature of multi-scale motion patterns into shared codebook and adaptively fuses these features for different body parts. We validate the effectiveness of APVQ-VAE by comparing its reconstruction performance with vanilla VQ-VAE We also illustrate the attention map of pyramidal features fusion in APVQ-VAE. It supports the observation of varying preferences of motion scales across body parts and interprets the source of adaptivity APVQ-VAE. Extensive experiments demonstrate the superiority of the PyraMotion framework in generating more natural and realistic holistic 3D gestures compared with current state-of-the-a methods.
Acknowledgement
This work is supported by both the Bureau of Science and Technology of Nansha District, Guangzhou, Grant No. 20222222 and HSBC Project L0562.
10

---

References
[1] Tenglong Ao, Qingzhe Gao, Yuke Lou, Baoquan Chen, and Libin Liu. Rhythmic gesticula- tor: Rhythm-aware co-speech gesture synthesis with hierarchical neural embeddings. ACM Transactions on Graphics (TOG), 11(6);1-1, 2022.
[2] Tenglong Ao, Zeyi Zhang, and Libin Liu. Gesturediffuclip: Gesture diffusion model with clip latents. ACM Transactions on Graphics (TOG), 42(4):1-18, 2023.
[3] Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. Enriching word vectors with subword information. Transactions of the association for computational linguistics, 5:135-146 146, 2017.
[4] Justine Cassell, Hannes Högni Vilhjálmsson, and Timothy Bickmore. Beat: the behavior expression animation toolkit. In Proceedings of the 28th annual conference on Computer graphics and interactive techniques, pages 477-486, 2001.
[5] Bohong Chen, Yumeng Li, Yao-Xiang Ding, Tianjia Shao, and Kun Zhou. Enabling synergistic full-body control in prompt-based co- co-speech motion generation. In Proceedings of the 32nd ACM International Conference on Multimedia, pages 6774-6783, 2024.
[6] Junming Chen, Yunfei Liu, Jianan Wang, Ailing Zeng, Yu Li, and Qifeng Chen. Diffsheg: diffusion-based approach for real-time speech-driven holistic 3d expression and gesture generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pullern Recognition, pages 7352-7361, 2024.
[7] Yongkang Cheng and Shaoli Huang. Holegest: Decoupled diffusion and motion priors for generating holisticly expressive co-speech gestures. In International Conference on 3D Vision 2025.
[8] Kiran Chhatre, Radek Dančček, Nikos Athanasiou, Giorgio Becherini, Christopher Peters, Michael J. Black, and Timo Bolkart. Emotional speech-driven 3d body animation via disentan- gled latent diffusion, 2024.
[9] Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Lukasz Kaiser. Uni- versal transformers. arXiv preprint arXiv: /00.003/7,. 2018.
[10] Yingruo Fan, Zhaojiang Lin, Jun Saito, Wenping Wang, and Taku Komura. Faceformer: Speech- driven 3d facial animation with transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18770-18780, 2022.
[11] Andrew Feng, Gale Lucas, Stacy Marsella, Evan Suma, Chung-Cheng Chiu, Dan Casas, and Ari Shapiro. Acting the part: the role of gesture avatar identity. In Proceedings of the 7th International Conference on Motion in Games, MIG 14, page 49-54, New York, NY, USA, 2014. Association for Computing Machinery.
[12] Shiry Ginosar, Amir Bar, Gefen Kohavi, Caroline Chan, Andrew Owens, and Jitendra Ma- lik. Learning individual styles of conversational gesture. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 3497-3500 2019.
[13] Ian Goodfellow, Jean Pouget- Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
[14] Ikhsanul Habibie, Weipeng Xu, Dushyant Mehta, Lingjie Liu, Hans-Peter Seidel, Gerard Pons- Moll, Mohamed Elgharib, and Christian Theobalt. Learning speech-driven 3d conversational gestures from video. In Proceedings of the 21 ACM International Conference Intelligent Virtual Agents, pages 101-108, 2021.
[15] Chien-Ming Huang and Bilge Mutlu. Robot behavior toolkit: generating effective social behaviors for robots. In Proceedings of the seventh annual ACM/IEEE international conference on Human-Robot Interaction, pages 25-32, 2012.

---

[16] Diederik P Kingma and Max Welling, Auto-encoding variational bayes. arXiv preprint drXiv:1312.6114, 2013.
[17] Lik-Hang Lee, Tristan Braud, Pengyuan Zhou, Lin Wang, Dianlei Xu, Zijun Lin, Abhishek Kumar, Carlos Bermejo, and Pan Hui. All one needs to know about metaverse: A complete survey on technological singularity, virtual ecosystem, and research agenda. arXiv preprint arXiv:2110.05352, 2021.
[18] Jing Li, Di Kang, Wenjie Pei, Xuefei Zhe, Ying Zhang, Zhenyu He, and Linchao Bao. Au- dio2gestures: Generating diverse gestures from speech audio with conditional variational autoencoders. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1129--1132,, 2021.
[19] Ruilong Li, Shan Yang, David A Ross, and Angjoo Kanazawa. Ai choreographer: Music conditioned 3d dance generation with aist++. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 13401-13412, 2021. 2021.
[20] Tsung-Yi Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2117-2125, 2017.
[21] Haiyang Liu, Naoya Iwamoto, Zihao Zhu, Zhengqing Li, You Zhou, Elif Bozkurt, and Bo Zheng. Disco: Disentangled implicit content and rhythm learning for diverse co-speech gestures synthesis. Proceedings of the 30th ACM international conference on multimedia, pages 3764-3773, 2022.
[22] Haiyang Liu, Zihao Zhu, Giorgio Becherini, Yichen Peng, Mingyang Su, You Zhou, Naoya Iwamoto, Bo Zheng, and Michael J Black. Emage: Towards unified holistic co-speech gesture generation via masked audio gesture modeling. arXiv preprint urXiv:2401.0037. 2023.
[23] Haiyang Liu, Zihao Zhu, Naoya Iwamoto, Yichen Peng, Zhengqing Li, You Zhou, Elif Bozkurt, and Bo Zheng. Beat: A large-scale semantic and emotional multi-modal dataset for conversa- tional gestures synthesis. In European conference on computer vision, pages 612-630. Springer, 2022.
[24] Xian Liu, Qianyi Wu, Hang Zhou, Yinghao Xu, Rui Qian, Xinyi Lin, Xiaowei Zhou, Wayne Wu, Bo Dai, and Bolei Zhou. Learning hierarchical cross-modal association for co-speech gesture generation. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10462-10472, 2022.
[25] Yifei Liu, Qiong Cao, Yandong Wen, Huaiguang Jiang, and Changxing Ding. Towards variable and coordinated holistic co-speech motion generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1566-15555 1576, 2024.
[26] David G Lowe. Object recognition from local scale-invariant features. In Proceedings of the seventh IEEE international conference on computer vision, volume 2, pages 1150-1157. leee, 1999.
[27] M Hamza Mughal, Rishabh Dabral, Merel CJ Scholman, Vera Demberg, and Christian Theobalt. Retrieving semantics from the deep: an rag solution for gesture synthesis. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages
2025.
[28] Catherine Pelachaud. Studies on gesture expressivity for a virtual agent. Speech Communication, 51(7):630-639, 2009.
[29] Kaede Shiohara, Xingchao Yang, and Takafumi Taketomi. Blendface: Re-designing identity encoders for face-swapping. In Proceedings of the IEEE/CVF International Conference on Computer Vision. pages 7634-7644, 2023.
[30] Li Siyao, Weijiang Yu, Tianpei Gu, Chunze Lin, Quan Wang, Chen Qian, Chen Change Loy, and Ziwei Liu. Bailando: 3d dance generation by actor-critic gpt with choreographic memory. In Proceedings of the JEFE//VV Conference on Computer Vision and Pattern Recognition, pages 11050- 11099, 2022.
12

---

[31] Keyu Tian, Yi Jiang, Zehuan Yuan, Bingyue Peng, and Liwei Wang. Visual autoregressive modeling: Scalable image generation via next-scale prediction. Advances neural information processing systems, (8844399848666 2024.
[32] Christine M Tipper, Giulia Signorini, and Scott T Grafton. Body language in the brain: constructing meaning from expressive movement. Frontiers in human neuroscience, 9:450, 2015.
[33] Tommi Tykkälä, Cédric Audras, and Andrew I Comport. Direct iterative closest point for real- time visual odometry. In 2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops), pages 2050-2056. IEEE, 2011.
[34] Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
[35] Susanne Van Mulken, Elisabeth Andre, and Jochen Müller. The persona effect: how substantial it? In People and computers XIII: Proceedings of HCT'98, pages 53-66 Springer, 1998.
[36] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[37] Jinbo Xing, Menghan Xia, Yuechen Zhang, Xiaodong Cun, Jue Wang, and Tien-Tsin Wong. Codetalker: Speech-driven 3d facial animation with discrete motion prior. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patlern Recognition, pages 12780-12790, 2023.
[38] Sicheng Yang, Zhiyong Wu, Minglei Li, Zhensong Zhang, Lei Hao, Weihong Bao, Ming Cheng, and Long Xiao. Diffusestylegesture: Stylized audio-driven co-speech gesture generation with diffusion models. arXiv preprint arXiv:2305.044 049939 2023.
[39] Payam Jome Yazdian, Mo Chen, and Angelica Lim. Gesture2vec: Clustering gestures using rep- resentation learning methods for co-speech gesture generation. In 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 3100-3100 IEEE, 2022.
[40] Hongwei Yi, Hualin Liang, Yifei Liu, Qiong Cao, Yandong Wen, Timo Bolkart, Dacheng Tao, and Michael Black. Generating holistic 3d human motion from speech. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 469-480, 2023.
[41] Youngwoo Yoon, Bok Cha, Joo-Haeng Lee, Minsu Jang, Jaeyeon Lee, Jachong Kim, and Geehyuk Lee. Speech gesture generation from the trimodal context of text, audio, and speaker identity. ACM Transactions on Graphics (TOG), 39(6):1-16, 2020.
[42] Yang Zhou, Jimei Yang, Dingzeyu Li, Jun Saito, Deepali Aneja, and Evangelos Kalogerakis. Audio-driven neural gesture reenactment with video motion graphs. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 3418-3428, 2022.
[43] Deli Zhu, Zhao Xu, and Yunong Yang. Mambatalk: Speech-driven 3d facial animation with mamba. In International Conference on Multimedia Modeling, pages 310-323, Springer, 2024.
A Training Details
The whole training process is conducted on an Ubuntu Server with GPU computing card with 32 GB VRAM and 256 GB memory. The average training time of APVQ-VAV of different body parts around 24 hours, and the PyraMotion framework takes 48 hours to achieve best performance. For the software environment, the model is deployed using Python 3.9, Py Torch 2.4.1.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The main claims made in the abstract and introduction are: 1) Identifying the research gap of existing methods, which ignores the varying motion patterns among different temporal scales and body parts. 2) This paper proposes an attentional pyramidal adaptively capture features from the varying motion patterns among different temporal scales and body parts. 3) This paper proposes a PyraMotion framework predict pyramidal motion token series from audio, thus generating motion from APVQ- VAE. 4) Extensive experiments validate the effectiveness of these proposed structures and interpret the mechanism of them.
VQ-VAE (APVQ-VAE)
These claims comprehensively reflect the contributions and scope of this paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: The limitation analysis of this work is in Section 6, which includes the inability to adaptively adjust the number of pyramid layers based on datasets, and the inability to conduct multi-modal controllability.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification: This work does not contain theoretical analysis.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
This paper disclosed the data processing of each step of the framework in the methodology section, accompanied by the mathematical operation formula. We include all the needed information in the paper and source code, and there is excluded information from the source code that might affect the main claims.
Justification:
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: In the supplemental material, we provide sufficient instructions with saved parameters to faithfully reproduce the main experimental results. And the data and code will be opensourced to github once accepted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We specify all the training test details in the Experiment section.
7. Experiment statistical significance
14

---

Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We provided the standard deviation of all experiment results by repeatedly running 5 times.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: In the paper, provided the compute resource information such as the type of compute workers, memory and time of execution.
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics httppp/s//eurrrs....cccaaaiaaaaa
Answer: [Yes]
Justification: The research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification; The paper discusses potential positive societal impacts by highlighting advance- ments in virtual avatar creation, which can enhance applications in education, telepresence, and assistive technologies, improving accessibility and human-computer interaction. Addi- tionally, the authors address negative societal impacts by acknowledging risks such as the misuse of the technology to generate deepfakes for disinformation or deceptive content.
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [Yes]
Justification: We will include necessary safeguard declarations in the released model, requir- ing that users adhere to usage guidelines restrictions access the model or implementing safety filters
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The codes of comparison methods used in the paper are properly cited.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Details of the model and codes are fully discussed as part of the submission attached in supplementary materials.
14. Crowdsourcing and research with human subjects
15

---

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [Yes]
Justification: The paper involves research with human subjects by using questionnaires. The full content of the questionnaires is attached in the supplemental material.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Question:
Answer: [Yes]
Justification: We conduct the user studies after obtaining the IRB approvals.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does involve LLMs any important, original, or non-standard components.
16


Reviews:
[{"review_id": "wZYigOr2DY", "reviewer": "Reviewer_ZG2m", "summary": "The paper presents PyraMotion, a speech-driven gesture generation method that leverages an attentional pyramid-structured motion integration mechanism. This approach captures expressive gestures by modeling motion patterns of varying durations across different frame lengths and body parts, enabling the system to generate gestures that are temporally and spatially consistent with speech.", "questions": "The primary concerns are outlined in the Weaknesses section above.\n\n- Regarding the actor Scott from the BEAT-2 dataset: was this a held-out actor during training? If not, could you please include qualitative examples from an actual held-out actor to better assess generalization?", "limitations": "Yes", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 1, "clarity": 1, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n- The paper proposes a novel technique that encodes motion patterns at varying temporal scales into a shared discrete latent space as motion pattern tokens, enabling multi-scale gesture representation.\n\n- The authors introduce a novel attentional pyramidal VQ-VAE architecture to tokenize motion across different temporal resolutions, allowing the model to capture body movements over time.\n\nWeaknesses:\n- The generated gestures often appear jittery, with broken lip-sync, and unnatural finger poses.\n\n- Figures:\n-- Figure 2 is difficult to interpret due to overlapping arrows and dense layout.\n-- Figure 3 uses inconsistent font sizes, with text hard to read.\n-- The attention map lacks a color legend, reducing interpretability of visualized patterns.\n\n- The chosen metrics: FGD, Diversity, Beat Alignment (BA), and MSE, are standard for speech-to-gesture tasks, but may not effectively capture the multi-scale motion modeling claims made in the paper. The use of task-specific or scale-sensitive metrics would support the evaluation.\n\n- The evaluation omits several recent SOTA methods, such as: HOLOGEST (Cheng et al., 2025), AMUSE (CVPR 2024). ConvoFusion (CVPR 2024), RAG-Gesture (CVPR 2025). These should be included for a comprehensive comparison.\n\n- The user study relies on older methods like CaMN, despite the availability of newer, more relevant baselines. While EMAGE (CVPR 2024) is included, more recent methods should be part of the perceptual comparison to validate the model's effectiveness.\n\n- According to Appendix A, the model requires 3 days of training, which is high compared to recent SOTA methods. This raises concerns about computational efficiency, reproducibility, and accessibility.", "was_revised": true, "final_justification": "While the rebuttal addressed some concerns, the work still lacks comprehensive SOTA comparisons, limiting the strength of its claims.", "rebuttal": "Thanks for your thorough and insightful comments. We will respond to them one by one as follows:\n\nWeakness 1: \n\nThank you for pointing out the unsatisfying performance of our method in terms of jittering, mouth movements, and finger movements. We acknowledge that there is a gap between the current performance and a satisfactory outcome. However, we would like to clarify that both the conducted user perceptual and statistical experiments have validated that our generated motion outperforms existing classical and state-of-the-art methods.\n\nWeakness 2: \n\nThank you for pointing out the flaws in our used figures. We will update them with clearer figures in the final version.\n\nWeakness 3: \n\nThank you for raising your concerns about the used metrics. We adapt these metrics following several state-of-the-art works, such as DiffSHEG, EMAGE, and ProbTalk. We agree with you that a metric measuring multi-scale motion is valuable. However, we would like to clarify that our current metrics, such as FGD, Beat Align, and MSE, are sufficient to evaluate the overall similarity, smoothness, and accuracy of the generated motion and reach the claims made in our paper.\n\nWeakness 4: \n\nThank you for listing these SOTA methods. On one hand, we would like to clarify that we adopt EMAGE and ProbTalk as our SOTA method mainly because the tasks of these two methods are holistic body motion generation, including both full body and face, while the given works are mainly full body generation without face motion generation. On the other hand, we would like to argue that both EMAGE and ProbTalk are published in the same period as most of the listed works, as state-of-the-art. Moreover, RAG-Gesture was published in June 2025, when our work had already been submitted to NeurIPS, and its code is not yet available. Thus, RAG-Gesture may not be suitable to be counted as a SOTA work before our paper.\n\nTo accomplish a more thorough comparison with the listed state-of-the-art methods, we will add comparisons with these methods in the final version of the paper. \n\nDue to the limited duration of the rebuttal, we have only recorded two of the mentioned methods. The experiment results show that our method still outperforms these state-of-the-art methods, even RAG-Gesture in some circumstances, which further validates the performance of our method.\n\n|       |  FGD  |   BA  |  Div  |  MSE  |  LVD  |\n|-------|-------|-------|-------|-------|-------|\n| HOLOGEST    | 5.347 | 7.957 | 14.15 |   -   |   -   |\n| RAG-Gesture | 8.08  | 7.34  | 11.97 |   -   |   -   |\n| Ours        | 4.612 | 7.420 | 13.42 | 7.176 | 7.270 |\n\n\nWeakness 5: \n\nThank you for your suggestions. We agree that comparing with more state-of-the-art work would strengthen the validation of our paper's effectiveness. We will add the comparison with ProbTalk in the final version.\n\nHowever, we would like to clarify that the current experiments also validate our claims. CaMN is a classic and well-performing method that has been used in user perceptual experiments in several recent works, such as EMAGE and MambaTalk. As a result, we included CaMN in our user perceptual comparison. The user perceptual results in our paper also validate that CaMN is necessary, as it outperforms EMAGE in some cases.\n\nMeanwhile, we selected EMAGE, a state-of-the-art paper published in CVPR 2024, as our recent comparison primarily because it is a VQVAE-based state-of-the-art method, which helps illustrate the difference between our proposed method and traditional VQVAE-based approaches. Since ProbTalk is also a VQVAE-based method, we will add it to the user perceptual comparison experiments.\n\nWeakness 6:\n\nThank you for your concerns about the training time. We argue that the 3-day training time does not affect the paper’s quality in terms of reproducibility and accessibility. We will release both the source code and pre-trained parameters on GitHub.\nAs for computational efficiency, the additional granularity layers cause an increase, which we discuss from Lines 281 to 287 on Page 8.\n\nTheoretically, the computational complexity of APVQ-VAE is $O(\\Sigma_{p = 1}^{P}(2^pd)(Nd/2^p)) = O(PNd^2)$, where $d$ is the latent dimension, $N$ is the total number of data frame, and $P$ is the number of pyramid layers. Thus, the time complexity of APVQ-VAE is a constant times that of vanilla VQ-VAE.\n\nFor the PyraMotion model, after the pooling operation. The time complexity of each step satisfies the form of a geometric series. For example, in the TCA operation in Equation (12), the time complexity of single scale is $O(Nd^2 + N^2d)$, where $N$ is the length of the total clip, while $d$ is the hidden dimension. For multi-sclae Temporal cross-attention operation, the time complexity is $O(\\Sigma_{p=0}^{P}(Nd^2/2^p + (N/2^p)^2d)) \\sim O(2Nd^2+4N^2d/3) < 2O(Nd^2 + N^2d)$. Thus, we can conclude that the time complexity of PyraMotion is less than 2 times that of the single-scale version of Pyramotion.\n\nTechnically, the inference time of the single-scale model EMAGE and the multi-scale model PyraMotion on more than 5 min motion is listed below. The experimental result validates the above theoretical results. \n\n|       |  Inference Time |\n|-------|-------|\n| EMAGE       |  22 s |\n| PyraMotion  |  41 s |\n\nWe will add the above analysis to our appendix in the final version.\n\nQuestion 1: \n\nThank you for your suggestion. To ensure a fair comparison with existing methods, we follow the training/test/validation split strategy used in prior works like EMAGE, ProbTalk, and CaMN, which do not include a held-out actor. We will add clarification regarding held-out actors in the final version."}, {"review_id": "eLxKo7Kuao", "reviewer": "Reviewer_QWGa", "summary": "This paper introduces PyraMotion, a new framework for generating natural and expressive full-body co-speech gestures from audio inputs. Unlike existing methods that predict frame-wise motion tokens, PyraMotion captures multi-scale motion patterns across different body parts by leveraging a pyramidal structure. Central to this approach is the proposed Attentional Pyramidal VQ-VAE (APVQ-VAE), which encodes gesture sequences at varying temporal scales into a shared discrete latent space and reconstructs gestures using an attention-based fusion mechanism. The system also includes a pyramidal token predictor that integrates audio and textual features to accurately predict motion tokens for various body regions. Objective and subjective evaluations demonstrate that PyraMotion significantly outperforms baseline methods in gesture realism, diversity, and synchronization. Ablation studies confirm the importance of its key components, particularly the attention mechanism and multi-scale design.", "questions": "1. The limitation section is relatively brief and does not show concrete examples of where the model fails or underperforms. For instance, are there specific types of speech (e.g., monotone, emotional) that the model struggles with?\n2. The paper compares mainly with prior VQ-VAE-based systems. However, recent approaches such as DiffSHEG or transformer-based generators may offer different trade-offs. Can the authors comment on these directions, and whether PyraMotion could be combined with them?\n3. Why were recent strong baselines such as FreeMotion, Emotional Speech-driven Body Animation, and DiffSHEG not included in the comparison? Please clarify the rationale for not including these baselines. If they were omitted due to incompatible inputs, unavailable code, or other technical barriers, a discussion would help justify the scope of comparison. Ideally, adding results for at least one of them would greatly strengthen the paper.", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "# Strengths\n- The paper is generally well-written and easy to follow.\n- The proposed PyraMotion framework is well-motivated, and the design of the Attentional Pyramidal VQ-VAE (APVQ-VAE) is carefully validated through quantitative and qualitative evaluations.\n\n# Weaknesses\n- While the pyramidal motion integration is new in this domain, the use of pyramidal architectures and attention mechanisms is conceptually borrowed from established techniques in computer vision.\n- While the paper compares against a number of established methods, it omits several recent and relevant approaches such as FreeMotion, Emotional Speech-driven 3D Body Animation via Disentangled Latent Diffusion, and DiffSHEG. These newer methods represent the current frontier in text/audio-to-motion synthesis and may offer strong performance baselines, particularly in diffusion-based frameworks. The absence of these baselines weakens the comparative analysis and may limit the ability to assess PyraMotion's true performance advantage.", "was_revised": true, "final_justification": "Although the rebuttal has clarified my major concerns and improved the paper, the level of novelty reamins average. Therefore, I maintain my borderline accept recommendation.", "rebuttal": "Weakness 1: \n\nThank you for your insightful comment. As stated in the introduction section, we drew inspiration from the feature pyramid to construct the pyramidal VQ-VAE for tokenizing 3D holistic motion in a multi-scale manner. But we would like to clarify that the novelties of this paper are: Identifying the single-scale tokenization research gap and proposing pyramidal-tokenization-based models such as APVQVAE and PyraMotion to address it. \n\n\nWeakness 2 / Question 3: \n\nThank you for your insightful suggestion. We would like to clarify that DiffSHEG is included as a comparison method in Line 10 of Table 1.\n\nRegarding the other two methods, there are specific reasons for not including them in our comparisons. For FreeMotion, its primary task focuses on text-to-motion generation, which produces motion from descriptive text. In contrast, our task is co-speech motion generation, where motion is generated from audio input. The transcripts used in our work represent the semantic content of the audio rather than motion descriptions. Therefore, FreeMotion may not be suitable as a direct comparison method, though we will include it in our related works section.  \n\nAs for AMUSE, in the initial stages of this paper, we primarily selected state-of-the-art methods that incorporate VQ-VAE and its variant structures. We appreciate your suggestion and will add AMUSE to our comparison methods in the final version to ensure a more comprehensive evaluation.\n\nQuestion 1: \n\nThank you for your insightful suggestion. Due to page limit considerations, we only provided a brief introduction to the paper's limitations. Here we would like to introduce one additional limitation for the final version: Our current model still underperforms existing emotion-disentangled models in generating emotional expressions based on vocal emotion. It relies heavily on the emotion distribution of the training data rather than explicitly extracting emotion representations from audio.\n\nQuestion 2: \n\nThank you for your valuable question. The core idea of PyraMotion involves tokenizing motion representations at multiple scales and adaptively combining these multi-scale features to reconstruct motion.\n\nIn contrast, diffusion-based or transformer-based methods do not include this tokenization step. These methods represent motion patterns through hidden state embeddings in latent space, which are inherently frame-wise representations.\n\nTo adapt PyraMotion's approach to diffusion-based or transformer-based methods, we could first fuse motion blocks of different lengths to create multiple motion representation sequences with varying granularities. We could then use the diffusion or transformer model to generate these multi-scale motion representation sequences before integrating them through attention mechanisms."}, {"review_id": "ExBlabP0Gz", "reviewer": "Reviewer_VSf9", "summary": "This paper proposes PyraMotion, a framework for co-speech gesture generation that integrates multi-scale motion tokenization with attention-based fusion. It introduces an adaptive pyramid-structured VQ-VAE model, called APVQ-VAE, which captures motion patterns across different temporal scales and body parts. Extensive experiments demonstrate superior performance over state-of-the-art methods in both objective metrics and subjective perception.", "questions": "For a motion sequence x with shape of (L,D), where L is the time length and D is the feature dim. What is the output if feed x into the proposed APVQ-VAE?", "limitations": "As weakness.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths:\n\n1. The APVQ-VAE enables multi-scale modeling of motion patterns, and the attention maps provide interpretable insights into how different body parts attend to different temporal resolutions.\n\n2. The paper includes thorough ablation studies, comparisons with SOTA methods, and subjective human evaluations, all supporting the effectiveness of the proposed approach.\n\n3. The authors provide sufficient visualizations through videos and conduct comprehensive quantitative experiments.\n\nWeakness:\n\n1. Line 28 is imprecise. Typically, VQ-VAE can encode T frames of motion into T/d tokens, where d is the downsampling rate. Therefore, existing VQ-VAE methods do not model only single-frame motion. It is recommended that the authors revise this statement in their rebuttal.\n\n2. The comparison with state-of-the-art (SOTA) methods for co-speech gesture generation, such as MambaTalk and SynTalker, is missing.", "was_revised": true, "final_justification": "I appreciate the authors' detailed rebuttal, which has addressed my main concerns. The work is technically thorough, with high reproducibility. It demonstrates significant innovation and has great potential. I believe that with the open-source release, it will contribute to advancing co-speech gesture generation. Therefore, I raise my rating to Accept.", "rebuttal": "Weakness 1:\n\nThank you for pointing out the imprecise statement. We will revise this statement to: \"Existing VQ-VAE methods can only model motion in a single scale, which may not be able to model the complexity of natural human motions with varying durations.\"\n\nWeakness 2: \n\nThank you for your suggestion about the comparison with the listed works. We will add the comparison with these works to the final version of our paper. Moreover, we would like to clarify that, compared with these works, the claim that our method achieves and outperforms state-of-the-art holistic generation work is still valid.\n\n|       |  FGD  |   BA  |  Div  |  MSE  |  LVD  |\n|-------|-------|-------|-------|-------|-------|\n| Mamba-Talk     |  5.366 | 7.812 | 13.05  | 6.289 | 6.897 |\n| SynTalk        |  6.413 | 7.971 | 12.721 |   -   |   -   |\n| SynTalk(noboth)|  4.687 | 7.363 | 12.425 |   -   |   -   |\n| Ours           |  4.612 | 7.420 | 13.420 | 7.176 | 7.270 |\n\nQuestion 1: \n\nThanks for raising this question. if we feed a motion sequence $x_{in} \\in \\mathbb{R}^{L\\times D}$ into APVQ-VAE, the output will also be a reconstructed motion sequence $x_{out} \\in \\mathbb{R}^{L\\times D}$. Such an input-output format is consistent with traditional VQVAE. However, the difference between APVQ-VAE and VQ-VAE is that, VQ-VAE reconstructed the $L\\times D$ motion sequence by encoding it with encoder network into a token series $T \\in \\mathbb{R}^{L_T\\times 1}$, and learns a codebook $C \\in \\mathbb{R}^{|C|\\times d}$ and reconstruct it by decoding the token series with a decoder network. However, our APVQ-VAE encode the $L\\times D$ motion sequence into a series of pyramidal token series, the token series in $p^{th}$ layer is $T_p \\in \\mathbb{R}^{(L_T/2^p)\\times 1}$. The decoder network will decode these token series and generate $p$ motion sequences, which will be attentively fused into a final motion sequence representation $x_{out} \\in \\mathbb{R}^{L\\times D}$"}, {"review_id": "nIR3z3fpuE", "reviewer": "Reviewer_JaLi", "summary": "This paper proposes a novel multi-scale pyramidal framework named PyraMotion to achieve audio-driven full-body human gestures generation. It introduces the Attentional Pyramidal VQ-VAE (APVQ-VAE), which encodes gestures at multiple temporal scales using pyramidal temporal convolution networks and reconstructs motion via attention-based fusion of multi-scale token sequences. On top of this, PyraMotion introduces a pyramidal token predictor that uses audio, text, speaker identity, and partial motion hints to predict multi-scale gesture tokens for different body parts, enabling fine-grained and coherent gesture synthesis. The experiments demonstrate the effectiveness and superior performance of the proposed framework.", "questions": "(1) Given that the proposed framework introduces additional architectural components (e.g., multi-scale TCNs), it is necessary to compare its computational complexity and runtime with previous works. Regarding the optimal number of pyramid layers, Table 4 suggests that four layers yield the best performance. However, I noticed inconsistencies between the results in Table 4 (4 Layers) and Table 3 (APVQ-VAE). Specifically, the Upper metric should be 2.612 ± 0.023, and the BC and LVD values should be ±0.015 and ±0.019, respectively. The authors are expected to clarify whether these are typographical errors or explain the discrepancy.\n\n(2) Regarding the content of the paper, the “Limitation, Future Work” section lists only one limitation, and the prefix “1)” is unnecessary. In Line 8–9, the claim that “such non-verbal signals provide more information than voice and context” may be overly absolute. The authors should either clarify the specific scenarios where this applies or rephrase the statement more cautiously. In Line 293–294, the authors should provide further details on the 18 participants in the perceptual study, such as gender, age, or research background, as these factors could influence subjective evaluation outcomes.\n\n(3) The font size in Figure 2 is too small, which significantly impacts readability and visual clarity. It is recommended to enlarge the figure to improve legibility. Additionally, whether $L_{vel}$ and $L_{acc}$ in Equation 6 are commonly used loss components in motion generation tasks requires further explanation from the authors.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths:\nThis paper presents a high-quality and well-executed framework for audio-driven full-body gesture generation. Its main contribution lies in the introduction of the Attentional Pyramidal VQ-VAE, a novel architecture that encodes motion patterns across multiple temporal scales, and its integration into the PyraMotion framework for multi-modal gesture synthesis. The paper is clearly written, well-structured, and supported by thorough experiments. The proposed method achieves state-of-the-art results across various benchmarks, demonstrating both originality and effectiveness. \n\nWeakness:\nThis model introduces architectural complexity, so it is necessary to provide PyraMotion with a comparison of computational complexity and computational time to other methods. Certain sections of the text and figures/tables necessitate additional adjustments, which will be elaborated below.", "was_revised": true, "final_justification": "The authors have addressed my concerns, and I think this solid work meets the acceptance requirements. Therefore, I keep my original score as 4.", "rebuttal": "Weakness 1: \n\nThank you for pointing out the complexity of our method. We will add an analysis comparing the time complexity of our work with other single-scale methods to the appendix in the final version.\n\nTheoretically, the computational complexity of APVQ-VAE is:\n$$O\\left(\\sum_{p=1}^{P}(2^pd)\\left(\\frac{Nd}{2^p}\\right)\\right) = O(PNd^2)$$\nwhere $d$ is the latent dimension, $N$ is the total number of frames, and $P$ is the number of pyramid layers. Thus, the time complexity of APVQ-VAE is a constant multiple of vanilla VQ-VAE.\n\nFor the PyraMotion model, after the pooling operation, the time complexity at each step follows a geometric series. For example, in the TCA operation (Equation 12), the single-scale time complexity is:\n$$O(Nd^2 + N^2d)$$\nwhere $N$ is the clip length and $d$ is the hidden dimension. The multi-scale Temporal Cross-Attention operation has complexity:\n$$O\\left(\\sum_{p=0}^{P}\\left(\\frac{Nd^2}{2^p} + \\left(\\frac{N}{2^p}\\right)^2d\\right)\\right) \\sim O(2Nd^2 + \\frac{4N^2d}{3}) < 2O(Nd^2 + N^2d)$$\nThis shows PyraMotion's time complexity is less than 2× that of its single-scale version.\n\nSpecifically, we measured the inference times for single-scale EMAGE and multi-scale PyraMotion on over 5 minutes of motion data. The experimental results validate these theoretical conclusions.\n\n|       |  Inference Time |\n|-------|-------|\n| EMAGE       |  22 s |\n| PyraMotion  |  41 s |\n\nQuestion 1: \n\nThank you for pointing out the statistical typo. For time-complexity analysis, please refer to our response to Weakness 1. Regarding the discrepancy between Table 4 and Table 3, we sincerely apologize for this oversight. After thorough verification of our local statistics, we confirm that Table 3 contains the correct values. We will immediately correct Table 4 accordingly.\n\nQuestion 2: \n\nThank you for your insightful suggestions.\n\nRegarding limitations: Due to page constraints, we initially provided only a brief discussion of limitations. We will add another key limitation in the final version: Our current model underperforms existing emotion-disentangled approaches in generating emotional expressions from vocal affect, as it relies heavily on the training data's emotion distribution rather than explicitly extracting emotion representations from audio.\n\nConcerning the claim about non-verbal signals, we appreciate your identifying the imprecise phrasing. We will modify the statement to: \"Such non-verbal signals provide distinct information that neither voice nor context can adequately express.\"\n\nRegarding participant demographics: Thank you for the suggestion. Our participant pool consisted of:\n\nAge: 12 participants (18-24 years), 6 participants (25-34 years)\n\nGender: 8 male, 10 female\n\nEducation: 5 Bachelor's, 5 MPhil students, 8 PhD students\n\nQuestion 3: \n\nThank you for your constructive comments.\n\nFor Figure 2, we will revise the figure to improve the clarity and readability of the overall structure.\nRegarding the loss terms in Equation 6:\n$L_{vel}$ appears in DiffSHEG, EMAGE, Habibie et al., etc.\n$L_{acc}$ appears in EMAGE, Habibie et al., CaMN, etc.\nWe will ensure all these references are properly included in the final version."}, {"review_id": "XmqcRR0NI2", "reviewer": "Reviewer_CqFS", "summary": "PyraMotion: Attentional Pyramid-Structured Motion Integration for Co-Speech Gesture Synthesis\nIn this paper, the authors propose to use multi-scale part separated tokens to learn the embedding of co-speech motions (gesture specifically).", "questions": "1. How fast is the runtime inference time for the algorithm? Is it streamable?\n\n2. The claim that “Existing systems tokenize gestures frame-wise” is incorrect or inaccurate.\nIn [1], [2], [3], [4], [5] and other existing work, tokens are usually generated with a 1D convolutionary network. It usually has a down-sampling rate of for example, 4, and each token has a much bigger perception field than 4. \nAnd it is also worthwhile to point out that in [2] the authors also use separate tokenization for different body parts (upper body and lower body).\nIn my own experience there’s no benefit to use separate tokenizations, which is why I mentioned above that some ablation or comparisons between tokenization strategy is needed.\n\n[1] Guo, Chuan, Yuxuan Mu, Muhammad Gohar Javed, Sen Wang, and Li Cheng. \"Momask: Generative masked modeling of 3d human motions.\" In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1900-1910. 2024.\n\n[2] Pinyoanuntapong, Ekkasit, Pu Wang, Minwoo Lee, and Chen Chen. \"Mmm: Generative masked motion model.\" In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1546-1555. 2024.\n\n[3] Pinyoanuntapong, Ekkasit, Muhammad Usama Saleem, Pu Wang, Minwoo Lee, Srijan Das, and Chen Chen. \"BAMM: bidirectional autoregressive motion model.\" In European Conference on Computer Vision, pp. 172-190. Cham: Springer Nature Switzerland, 2024.\n\n[4] Jiang, Biao, Xin Chen, Wen Liu, Jingyi Yu, Gang Yu, and Tao Chen. \"Motiongpt: Human motion as a foreign language.\" Advances in Neural Information Processing Systems 36 (2023): 20067-20079.\n\n[5] Zhang, Yaqi, Di Huang, Bin Liu, Shixiang Tang, Yan Lu, Lu Chen, Lei Bai, Qi Chu, Nenghai Yu, and Wanli Ouyang. \"Motiongpt: Finetuned llms are general-purpose motion generators.\" In Proceedings of the AAAI Conference on Artificial Intelligence, vol. 38, no. 7, pp. 7368-7376. 2024.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Pros:\n\n1. The paper is well-written and clearly articulated.\nThe figures are highly intuitive and effectively enhance understanding; for instance, Figure 2 provides an excellent and comprehensive overview of the algorithm. The algorithm is presented in a way to be easily and readily comprehensible to readers.\n\n2. The experiment section is very complete and fully supports the claim of the paper.\n\nThe paper holds a full collection of more than 10 baseline algorithms and collects a wide range of metrics such as beat alignment and diversity. A comprehensive user study is also included.\n\nThe ablation study is also well prepared, clearly showing the impact of each component in the model.\n\nCons:\n\n1. The paper would benefit from a more detailed exposition on the intuition behind the choices made for the tokenizer design.\nA comparative analysis of different tokenization strategies is needed to fully justify the proposed approach.\n\n\nWhat happens if we just use a multi-scale tokenizer without the separate encoding for different body parts?\n\nWhat happens if we just use multi-head vqvae instead of multi-scale vqvae?\n\n\n2. The contribution might be seen as incremental.\n\nWhile the proposed algorithm shows improved performance compared to the baselines, it might be seen as an application of existing tokenization strategy on a specific domain (motion generation). Multi-scale tokenization and prediction strategy has been proposed and as discussed in the paper [1].\n\nAnd the body part based tokenization strategy is designed for the motion generation problem and does not seem to be generalizable to other domains.\n\n[1] Tian, Keyu, Yi Jiang, Zehuan Yuan, Bingyue Peng, and Liwei Wang. \"Visual autoregressive modeling: Scalable image generation via next-scale prediction.\" Advances in neural information processing systems 37 (2024): 84839-84865.\n\n\n\nBased on the quality of the experiments and paper writing, I lean towards borderline acceptance. \n\nHowever, given concerns regarding the novelty of the contribution, I will not actively champion this submission.", "was_revised": true, "final_justification": "The rebuttal further address some of my concerns (please refer to my comments below.)\n\nOverall I think it's a paper that meets the minimum requirement of an accepted paper.", "rebuttal": "Weakness 1: \n\nWe appreciate the insightful comment. The intuition behind our tokenizer design choices stems from a trade-off between computational inefficiency—due to overlapping information across different tokenizer scales—and the loss of multi-scale information resulting from an overly sparse selection of scale levels. Inspired by feature pyramid networks, we opted for a power-of-2 scaling factor to balance these considerations effectively\n\nQ1: \n\nIf we just use a multi-scale tokenizer without the separate encoding for different body parts? \nResponse: Thank you for this insightful question. We actually evaluated this exact configuration in our ablation studies (Table 2, Line 1), where we observed significant performance degradation compared to using separate encodings. This performance drop likely stems from interference patterns between different body parts' motions. Our findings align with observations in prior work like EMAGE, which similarly demonstrates the importance of separate encodings for different body regions.\n\nQ2: \n\nWhat happens if we just use multi-head VQ-VAE instead of multi-scale VQ-VAE?\nResponse: Thank you for raising this important comparison. The multi-head VQVAE (as implemented in MSMC-TTS) can indeed capture richer single-scale patterns through multiple codebooks. However, the multi-head VQ-VAE is still not able to capture and fuse multi-scale motion patterns since the multi-head VQ-VAE does not contain the process of extracting motion patterns with different scales in the latent representation level, let alone fusing multi-scale features of motion patterns. The fundamental architectural difference means that multi-head VQVAE is not suitable in both capturing hierarchical motion patterns and integrating them effectively than multi-scale VQVAE.\n\nWeakness 2: \n\nThank you for your comment. We apologize for not clearly articulating our contributions to both the motion generation field and the broader research community. As mentioned in Line 35, the core idea of pyramidal tokenization in PyraMotion is inspired by the concept of pyramidal multiscale design in computer vision, drawing not only from [1] but also from feature pyramid networks [2].\nWe would like to clarify that the key novelties of this paper are:\n1. Identifying the research gap in single-scale tokenization for temporal data generation.\n2. Proposing pyramidal-tokenization-based models, such as APVQVAE and PyraMotion, to address this limitation.\nFurthermore, we argue that the single-scale tokenization gap is not unique to motion generation but also exists in other 1D temporal sequence generation tasks, such as trajectory generation and stock price prediction. Thus, we believe our proposed Attentional Pyramidal VQ-VAE has the potential for broader applicability in capturing and fusing multiscale temporal patterns across different domains.\n\nWe will incorporate an analysis of the broader implications of this work in the introduction section to better highlight its impact.\n\n[1] Tian, Keyu, Yi Jiang, Zehuan Yuan, Bingyue Peng, and Liwei Wang. \"Visual autoregressive modeling: Scalable image generation via next-scale prediction.\" Advances in neural information processing systems 37 (2024): 84839-84865.\n\n[2] Tsung-Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2117–2125, 2017.\n\nQuestion 1:\nThanks for your question. The inference time of the single-scale model EMAGE and the multi-scale model Pyramotion on a motion of more than 5 min is listed below. The real-time ratio of both our model and EMAGE is far less than 1; thus, it is streamable.\n\n|       |  Inference Time |\n|-------|-------|\n| EMAGE       |  22 s |\n| PyraMotion  |  41 s |\n\nQuestion 2:\n\nThanks for pointing out our imprecise statement. We will correct our statement to “Existing systems tokenize gestures within a single scale manner.”\n\nFor the separate tokenization issue. Thanks for sharing your precious experience about it. As I mentioned above, in our ablation experiment, we first evaluated the performance of tokenization on a unified body and recorded it in Table 2, Line 1. The experiment results show that such separation largely contributes to the final performance. Such a conclusion is also validated in the paper EMAGE. As a result, such differences in the separation strategy may be related to model types and data distributions. The detailed research about it is valuable and will contribute to the research community. We will include it in our future work analysis."}]

Author rebuttal:

Meta-review: The paper introduces a method for co-speech gesture generation that incorporates a multi-scale design for 3D motion tokenization and attention-based fusion. The reviewers highlighted the following strengths: (a) a novel architecture that encodes motion patterns at varying temporal scales into a shared discrete latent space, and (b) extensive validation through ablation studies, comparisons with multiple baselines, and a user study, all supporting the effectiveness of the proposed approach.

Identified weaknesses included: missing comparisons with some SOTA methods (e.g., MambaTalk, SynTalker, Hologest, AMUSE, ConvoFusion), lack of computational complexity analysis relative to other methods, and limited discussion of alternative tokenization strategies. In the rebuttal, the authors addressed these concerns by providing computational complexity analysis, comparisons of different tokenization strategies, and additional experiments with comparisons to Hologest, MambaTalk, SynTalk, and a concurrent work (RAG-Gesture). The rebuttal clarifications satisfied four out of five reviewers. In the end, the paper received three borderline accepts, one accept, and one borderline reject (ZG2m). The borderline reviewer remained concerned about some missing comparisons (e.g., ConvoFusion, AMUSE), while the rest of the reviewers felt that the novelty of the work outweighed this weakness.

The AC agrees that further comparisons would strengthen the paper and strongly encourages the authors to include them in the final version. At the same time, the AC concurs with the majority of reviewers that the novelty of the method should be valued over incremental SOTA improvements whose absence does not diminish the scientific contribution of this work.

Thus, the AC recommends acceptance and urges the authors to carefully incorporate all reviewer feedback from both the reviews and the author-reviewer discussion when preparing the final version of the paper!
Decision: Accept (poster)