Title: PyraMotion: Attentional Pyramid-Structured Motion Integration for Co-Speech 3D Gesture Synthesis
Venue: NeurIPS 2025
Accepted: yes
Keywords: Co-Speech Motion Synthesis

3D

Generation

Representation Learning
Abstract: Generating full-body human gestures encompassing face, body, hands, and global movements from audio is crucial yet challenging for virtual avatar creation. Existing systems tokenize gestures frame-wise, predicting tokens of each frame from the input audio. However, expressive human gestures consist of varied patterns with different frame lengths, and different body parts exhibit motion patterns of varying durations. Existing systems fail to capture motion patterns across body parts and temporal scales due to the fixed frame-count setting of their gesture tokens. Inspired by the success of the feature pyramid technique in the multi-scale visual information extraction, we propose a novel framework named PyraMotion and an adaptive multi-scale feature capturing model called Attentive Pyramidal VQ-VAE (APVQ-VAE). Objective and subjective experiments demonstrate that the PyraMotion outperforms state-of-the-art methods in terms of generating natural and expressive full-body human gestures. Extensive ablation experiments highlight that the self-adaptiveness integration through attention maps contributes to performance.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nPyraMotion: Attentional Pyramid-Structured Motion Integration for Co-Speech Gesture Synthesis\nIn this paper, the authors propose to use multi-scale part separated tokens to learn the embedding of co-speech motions (gesture specifically).\n\nSTRENGTHS AND WEAKNESSES:\nPros:\n\n1. The paper is well-written and clearly articulated.\nThe figures are highly intuitive and effectively enhance understanding; for instance, Figure 2 provides an excellent and comprehensive overview of the algorithm. The algorithm is presented in a way to be easily and readily comprehensible to readers.\n\n2. The experiment section is very complete and fully supports the claim of the paper.\n\nThe paper holds a full collection of more than 10 baseline algorithms and collects a wide range of metrics such as beat alignment and diversity. A comprehensive user study is also included.\n\nThe ablation study is also well prepared, clearly showing the impact of each component in the model.\n\nCons:\n\n1. The paper would benefit from a more detailed exposition on the intuition behind the choices made for the tokenizer design.\nA comparative analysis of different tokenization strategies is needed to fully justify the proposed approach.\n\n\nWhat happens if we just use a multi-scale tokenizer without the separate encoding for different body parts?\n\nWhat happens if we just use multi-head vqvae instead of multi-scale vqvae?\n\n\n2. The contribution might be seen as incremental.\n\nWhile the proposed algorithm shows improved performance compared to the baselines, it might be seen as an application of existing tokenization strategy on a specific domain (motion generation). Multi-scale tokenization and prediction strategy has been proposed and as discussed in the paper [1].\n\nAnd the body part based tokenization strategy is designed for the motion generation problem and does not seem to be generalizable to other domains.\n\n[1] Tian, Keyu, Yi Jiang, Zehuan Yuan, Bingyue Peng, and Liwei Wang. \"Visual autoregressive modeling: Scalable image generation via next-scale prediction.\" Advances in neural information processing systems 37 (2024): 84839-84865.\n\n\n\nBased on the quality of the experiments and paper writing, I lean towards borderline acceptance. \n\nHowever, given concerns regarding the novelty of the contribution, I will not actively champion this submission.\n\nQUESTIONS:\n1. How fast is the runtime inference time for the algorithm? Is it streamable?\n\n2. The claim that “Existing systems tokenize gestures frame-wise” is incorrect or inaccurate.\nIn [1], [2], [3], [4], [5] and other existing work, tokens are usually generated with a 1D convolutionary network. It usually has a down-sampling rate of for example, 4, and each token has a much bigger perception field than 4. \nAnd it is also worthwhile to point out that in [2] the authors also use separate tokenization for different body parts (upper body and lower body).\nIn my own experience there’s no benefit to use separate tokenizations, which is why I mentioned above that some ablation or comparisons between tokenization strategy is needed.\n\n[1] Guo, Chuan, Yuxuan Mu, Muhammad Gohar Javed, Sen Wang, and Li Cheng. \"Momask: Generative masked modeling of 3d human motions.\" In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1900-1910. 2024.\n\n[2] Pinyoanuntapong, Ekkasit, Pu Wang, Minwoo Lee, and Chen Chen. \"Mmm: Generative masked motion model.\" In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1546-1555. 2024.\n\n[3] Pinyoanuntapong, Ekkasit, Muhammad Usama Saleem, Pu Wang, Minwoo Lee, Srijan Das, and Chen Chen. \"BAMM: bidirectional autoregressive motion model.\" In European Conference on Computer Vision, pp. 172-190. Cham: Springer Nature Switzerland, 2024.\n\n[4] Jiang, Biao, Xin Chen, Wen Liu, Jingyi Yu, Gang Yu, and Tao Chen. \"Motiongpt: Human motion as a foreign language.\" Advances in Neural Information Processing Systems 36 (2023): 20067-20079.\n\n[5] Zhang, Yaqi, Di Huang, Bin Liu, Shixiang Tang, Yan Lu, Lu Chen, Lei Bai, Qi Chu, Nenghai Yu, and Wanli Ouyang. \"Motiongpt: Finetuned llms are general-purpose motion generators.\" In Proceedings of the AAAI Conference on Artificial Intelligence, vol. 38, no. 7, pp. 7368-7376. 2024.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-21"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel multi-scale pyramidal framework named PyraMotion to achieve audio-driven full-body human gestures generation. It introduces the Attentional Pyramidal VQ-VAE (APVQ-VAE), which encodes gestures at multiple temporal scales using pyramidal temporal convolution networks and reconstructs motion via attention-based fusion of multi-scale token sequences. On top of this, PyraMotion introduces a pyramidal token predictor that uses audio, text, speaker identity, and partial motion hints to predict multi-scale gesture tokens for different body parts, enabling fine-grained and coherent gesture synthesis. The experiments demonstrate the effectiveness and superior performance of the proposed framework.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\nThis paper presents a high-quality and well-executed framework for audio-driven full-body gesture generation. Its main contribution lies in the introduction of the Attentional Pyramidal VQ-VAE, a novel architecture that encodes motion patterns across multiple temporal scales, and its integration into the PyraMotion framework for multi-modal gesture synthesis. The paper is clearly written, well-structured, and supported by thorough experiments. The proposed method achieves state-of-the-art results across various benchmarks, demonstrating both originality and effectiveness. \n\nWeakness:\nThis model introduces architectural complexity, so it is necessary to provide PyraMotion with a comparison of computational complexity and computational time to other methods. Certain sections of the text and figures/tables necessitate additional adjustments, which will be elaborated below.\n\nQUESTIONS:\n(1) Given that the proposed framework introduces additional architectural components (e.g., multi-scale TCNs), it is necessary to compare its computational complexity and runtime with previous works. Regarding the optimal number of pyramid layers, Table 4 suggests that four layers yield the best performance. However, I noticed inconsistencies between the results in Table 4 (4 Layers) and Table 3 (APVQ-VAE). Specifically, the Upper metric should be 2.612 ± 0.023, and the BC and LVD values should be ±0.015 and ±0.019, respectively. The authors are expected to clarify whether these are typographical errors or explain the discrepancy.\n\n(2) Regarding the content of the paper, the “Limitation, Future Work” section lists only one limitation, and the prefix “1)” is unnecessary. In Line 8–9, the claim that “such non-verbal signals provide more information than voice and context” may be overly absolute. The authors should either clarify the specific scenarios where this applies or rephrase the statement more cautiously. In Line 293–294, the authors should provide further details on the 18 participants in the perceptual study, such as gender, age, or research background, as these factors could influence subjective evaluation outcomes.\n\n(3) The font size in Figure 2 is too small, which significantly impacts readability and visual clarity. It is recommended to enlarge the figure to improve legibility. Additionally, whether $L_{vel}$ and $L_{acc}$ in Equation 6 are commonly used loss components in motion generation tasks requires further explanation from the authors.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-06-28"}

comments: SUMMARY:
This paper proposes PyraMotion, a framework for co-speech gesture generation that integrates multi-scale motion tokenization with attention-based fusion. It introduces an adaptive pyramid-structured VQ-VAE model, called APVQ-VAE, which captures motion patterns across different temporal scales and body parts. Extensive experiments demonstrate superior performance over state-of-the-art methods in both objective metrics and subjective perception.

STRENGTHS AND WEAKNESSES:
Strengths:

1. The APVQ-VAE enables multi-scale modeling of motion patterns, and the attention maps provide interpretable insights into how different body parts attend to different temporal resolutions.

2. The paper includes thorough ablation studies, comparisons with SOTA methods, and subjective human evaluations, all supporting the effectiveness of the proposed approach.

3. The authors provide sufficient visualizations through videos and conduct comprehensive quantitative experiments.

Weakness:

1. Line 28 is imprecise. Typically, VQ-VAE can encode T frames of motion into T/d tokens, where d is the downsampling rate. Therefore, existing VQ-VAE methods do not model only single-frame motion. It is recommended that the authors revise this statement in their rebuttal.

2. The comparison with state-of-the-art (SOTA) methods for co-speech gesture generation, such as MambaTalk and SynTalker, is missing.

QUESTIONS:
For a motion sequence x with shape of (L,D), where L is the time length and D is the feature dim. What is the output if feed x into the proposed APVQ-VAE?

LIMITATIONS:
As weakness.
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 4
DATE: 2025-07-01

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces PyraMotion, a new framework for generating natural and expressive full-body co-speech gestures from audio inputs. Unlike existing methods that predict frame-wise motion tokens, PyraMotion captures multi-scale motion patterns across different body parts by leveraging a pyramidal structure. Central to this approach is the proposed Attentional Pyramidal VQ-VAE (APVQ-VAE), which encodes gesture sequences at varying temporal scales into a shared discrete latent space and reconstructs gestures using an attention-based fusion mechanism. The system also includes a pyramidal token predictor that integrates audio and textual features to accurately predict motion tokens for various body regions. Objective and subjective evaluations demonstrate that PyraMotion significantly outperforms baseline methods in gesture realism, diversity, and synchronization. Ablation studies confirm the importance of its key components, particularly the attention mechanism and multi-scale design.\n\nSTRENGTHS AND WEAKNESSES:\n# Strengths\n- The paper is generally well-written and easy to follow.\n- The proposed PyraMotion framework is well-motivated, and the design of the Attentional Pyramidal VQ-VAE (APVQ-VAE) is carefully validated through quantitative and qualitative evaluations.\n\n# Weaknesses\n- While the pyramidal motion integration is new in this domain, the use of pyramidal architectures and attention mechanisms is conceptually borrowed from established techniques in computer vision.\n- While the paper compares against a number of established methods, it omits several recent and relevant approaches such as FreeMotion, Emotional Speech-driven 3D Body Animation via Disentangled Latent Diffusion, and DiffSHEG. These newer methods represent the current frontier in text/audio-to-motion synthesis and may offer strong performance baselines, particularly in diffusion-based frameworks. The absence of these baselines weakens the comparative analysis and may limit the ability to assess PyraMotion's true performance advantage.\n\nQUESTIONS:\n1. The limitation section is relatively brief and does not show concrete examples of where the model fails or underperforms. For instance, are there specific types of speech (e.g., monotone, emotional) that the model struggles with?\n2. The paper compares mainly with prior VQ-VAE-based systems. However, recent approaches such as DiffSHEG or transformer-based generators may offer different trade-offs. Can the authors comment on these directions, and whether PyraMotion could be combined with them?\n3. Why were recent strong baselines such as FreeMotion, Emotional Speech-driven Body Animation, and DiffSHEG not included in the comparison? Please clarify the rationale for not including these baselines. If they were omitted due to incompatible inputs, unavailable code, or other technical barriers, a discussion would help justify the scope of comparison. Ideally, adding results for at least one of them would greatly strengthen the paper.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents PyraMotion, a speech-driven gesture generation method that leverages an attentional pyramid-structured motion integration mechanism. This approach captures expressive gestures by modeling motion patterns of varying durations across different frame lengths and body parts, enabling the system to generate gestures that are temporally and spatially consistent with speech.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The paper proposes a novel technique that encodes motion patterns at varying temporal scales into a shared discrete latent space as motion pattern tokens, enabling multi-scale gesture representation.\n\n- The authors introduce a novel attentional pyramidal VQ-VAE architecture to tokenize motion across different temporal resolutions, allowing the model to capture body movements over time.\n\nWeaknesses:\n- The generated gestures often appear jittery, with broken lip-sync, and unnatural finger poses.\n\n- Figures:\n-- Figure 2 is difficult to interpret due to overlapping arrows and dense layout.\n-- Figure 3 uses inconsistent font sizes, with text hard to read.\n-- The attention map lacks a color legend, reducing interpretability of visualized patterns.\n\n- The chosen metrics: FGD, Diversity, Beat Alignment (BA), and MSE, are standard for speech-to-gesture tasks, but may not effectively capture the multi-scale motion modeling claims made in the paper. The use of task-specific or scale-sensitive metrics would support the evaluation.\n\n- The evaluation omits several recent SOTA methods, such as: HOLOGEST (Cheng et al., 2025), AMUSE (CVPR 2024). ConvoFusion (CVPR 2024), RAG-Gesture (CVPR 2025). These should be included for a comprehensive comparison.\n\n- The user study relies on older methods like CaMN, despite the availability of newer, more relevant baselines. While EMAGE (CVPR 2024) is included, more recent methods should be part of the perceptual comparison to validate the model's effectiveness.\n\n- According to Appendix A, the model requires 3 days of training, which is high compared to recent SOTA methods. This raises concerns about computational efficiency, reproducibility, and accessibility.\n\nQUESTIONS:\nThe primary concerns are outlined in the Weaknesses section above.\n\n- Regarding the actor Scott from the BEAT-2 dataset: was this a held-out actor during training? If not, could you please include qualitative examples from an actual held-out actor to better assess generalization?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 1, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-07"}