Title: 🎧MOSPA: Human Motion Generation Driven by Spatial Audio
Venue: NeurIPS 2025
Accepted: yes
Keywords: 3D Human Motion Generation
Abstract: Enabling virtual humans to dynamically and realistically respond to diverse auditory stimuli remains a key challenge in character animation, demanding the integration of perceptual modeling and motion synthesis. Despite its significance, this task remains largely unexplored. Most previous works have primarily focused on mapping modalities like speech, audio, and music to generate human motion. As of yet, these models typically overlook the impact of spatial features encoded in spatial audio signals on human motion. To bridge this gap and enable high-quality modeling of human movements in response to spatial audio, we introduce the first comprehensive "Spatial Audio-Driven Human Motion" (SAM) dataset, which contains diverse and high-quality spatial audio and motion data. For benchmarking, we develop a simple yet effective diffusion-based generative framework for human "MOtion generation driven by SPatial Audio," termed MOSPA, which faithfully captures the relationship between body motion and spatial audio through an effective fusion mechanism. Once trained, MOSPA can generate diverse realistic human motions conditioned on varying spatial audio inputs. We perform a thorough investigation of the proposed dataset and conduct extensive experiments for benchmarking, where our method achieves state-of-the-art performance on this task. Our code and model are publicly available at https://github.com/xsy27/Mospa-Acoustic-driven-Motion-Generation.git


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a human motion dataset comprised of spatial audio data, named the SAM dataset. The authors benchmark SAM using a diffusion-based, spatial audio-conditioned human motion generation framework (MOSPA) and demonstrate that MOSPA can be trained to generate human motions conditioned on spatial audio inputs, using the SAM dataset. The evaluation results show that MOSPA achieves good performance on the privacy-constrained HMR task.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths.**\n1. To the best of the reviewer's knowledge, this paper is the first dataset for modeling the interaction between spatial information in audio and human motion.\n\n2. The authors experimentally showed that motion synthesis models can be trained effectively on the proposed dataset, and by reporting a variety of metrics, they established a benchmark for the proposed task.\n\n**Weaknesses.**\n\nWhile the task of synthesizing human motion from audio is being discussed, the audio-related details have not been sufficiently elaborated.\n\n- On the SAM dataset\n\n    1. Please elaborate on the audio data processing configurations (e.g., sampling rate, STFT window/hop size, etc.)\n\n    2. From the picture of the measurement environment in Figure 3, it seems that the sound was played through (stationary) speakers in the room and recorded from a microphone placed in the subject's ear. As the SAM data is comprised of the mic measurements, it seems to need a more specific description of the recording environment.\n\n        - Please find more details in the Question section.\n\n- On the MOSPA model\n\n    1. What would be the rationale behind choosing an audio feature extraction as shown in Table B9? There is no mention of the selection criteria for the selected audio features, and no mention of features that are commonly used in the audio domain.\n\n        - The task that MOSPA tackles seems to be closely related to the Sound Event Localization and Detection (SELD) problem, and many SELD studies primarily use log-mel spectrograms (for per-frequency energy) and generalized cross-correlations (for phase information) as audio features [N1, N2, N3].\n\n        - ITD (Interaural Time Difference) and the ILD (Interaural Level Difference) are the most representative features for binaural audio cues, which are not mentioned in the text.\n\n    2. It is unclear, at this moment, what the variable $\\mathbf{s}$ means by reading the text “$\\mathbf{s}$ is the sound source” in L181. However, assuming that it is meant to imply the (relative) location of the sound source, the question remains why this input is required.\n\n        - Isn't it already possible to predict the source location from binaural recordings?\n\n        - Doesn't needing to know the exact (relative) position of the sound source to synthesize motion limit its applicability?\n\n        - Regarding the results in Table 3, the authors mention that entering location information improves FID and APD performance despite lowering other performance, so are these the two most important metrics?\n\n****\n\n[N1] Adavanne, S., Politis, A., Nikunen, J., & Virtanen, T. (2018). Sound event localization and detection of overlapping sources using convolutional recurrent neural networks. *IEEE Journal of Selected Topics in Signal Processing*, *13*(1), 34-48.\n\n[N2] Wilkins, J., Fuentes, M., Bondi, L., Ghaffarzadegan, S., Abavisani, A., & Bello, J. P. TWO VS. FOUR-CHANNEL SOUND EVENT LOCALIZATION AND DETECTION.\n\n[N3] Shimada, K., Simon, C., Shibuya, T., Takahashi, S., & Mitsufuji, Y. (2024). SAVGBench: Benchmarking Spatially Aligned Audio-Video Generation. *arXiv preprint arXiv:2412.13462*.\n\nQUESTIONS:\n**Details of the SAM data.**\n\nAs SAM is comprised of recorded audio, the binaural cues may vary depending on the recording environment, which can be linked to the models’ performance. One might have questions about the following details: \n\n1. What are the coordinates (locations) and the directions of the loudspeaker's location in the room, and what is the rationale for placing it that way?\n\n2. Are the speakers/microphones omnidirectional? If not, what are their specifications?\n\n3. How does the equipment used in the recording differ from the dummy head microphone, e.g. [N4], that is commonly used for binaural recordings? Is this equipment common for ‘binaural’ recordings?\n\n4. What is shape of the room? How is the acoustics of the room (reverberant/anechoic?)\n\n**Minor Comments.**\n\n1. In L121, what does “16 randomly sampled relative locations” mean? Presumably that there were a total of 16 loudspeaker locations that were the source of the sound, from which the experiment randomly picked one to play the sound, and the audio clips in the SAM are paired with relative locations between the (stationary) loudspeakers from the (moving) human location?\n\n    - MOSPA estimates $\\mathbf{x}$ which consists of the global position ($\\mathbf{p}$). Would there be any reason for encoding the sound source position as a relative position? (Perhaps due to the data sparsity in the sound sources’ global positions?)\n\n1. Would it be possible to extract a person's position and the head orientation from the motion estimated by MOSPA?\n\n    - If so, and if the original (mono) source sound is present, it can be rendered into binaural using efficient renderers such as [N5, N6]. It would be interesting to hear how well the binaurally rendered MOSPA’s sound source (say, $\\tilde{\\mathbf{a}}$) matches with the input binaural audio ($\\mathbf{a}$). So that: $\\mathbf{a}\\stackrel{\\mathcal{G}}{\\longrightarrow}\\hat{\\mathbf{x}}_0 \\stackrel{\\text{[N5]}}{\\longrightarrow} \\tilde{\\mathbf{a}}$ and then compare the similarity between the two binaural audios, e.g., $d(\\tilde{\\mathbf{a}}, \\mathbf{a})$.\n\n****\n[N4] Gardner, B., & Martin, K. (1994, May). *HRFT Measurements of a KEMAR Dummy-head Microphone*.\n\n[N5] Lee, J. W., & Lee, K. (2023, June). Neural fourier shift for binaural speech rendering. In *ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)* (pp. 1-5). IEEE.\n\n[N6] Roman, I. R., Ick, C., Ding, S., Roman, A. S., McFee, B., & Bello, J. P. (2024, April). Spatial scaper: a library to simulate and augment soundscapes for sound event localization and detection in realistic rooms. In *ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)* (pp. 1221-1225). IEEE.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new dataset, SAM, and a novel model, MOSPA, for human motion generation using spatial sound. The proposed dataset was recorded in an indoor setting and includes sound source locations, motion genres, binaural audio, and corresponding human motion data. The proposed method extracts acoustic features and, together with genre and source location information, utilizes a diffusion model to generate motion that corresponds to the audio signals. Experimental results on the SAM dataset demonstrate that the proposed approach produces more realistic and diverse motions compared to existing music-based baselines.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- Compared to prior work on gesture generation aligned with speech or music, the proposed task of generating motion based on spatial audio offers a high degree of novelty.\n- The paper is well-written and very easy to follow.\n- The proposed dataset, SAM, provides valuable insights into how humans react to spatial audio and is likely to contribute meaningfully to the advancement of the field.\n- The experimental results demonstrate that the proposed method outperforms existing approaches both quantitatively and qualitatively, highlighting the effectiveness of the proposed acoustic features and the additional conditioning information such as genre.\n\nWeaknesses:\n- The data collection environment is limited to a single indoor setting. It is well known that directional acoustic features are highly dependent on room layout and reverberation characteristics [1, 2]. Since the proposed dataset is recorded in only one room, it remains unclear whether the proposed method would generalize well to different indoor environments or outdoor scenarios. This limitation is particularly relevant for scenarios such as “insect” or “firework,” where the input audio would naturally be expected to originate from outdoor environments.\n- The proposed model does not take language as input, which limits its controllability. Although the effectiveness of genre-based control is demonstrated, there is no mechanism for more fine-grained control over the content or style of the generated motion.\n- The paper lacks in-depth analysis regarding the performance gap between models. While the proposed method is relatively simple in structure (which is not a weakness in itself), the discussion around its performance compared to other diffusion-based methods such as EDGE, is insufficient. The authors only mention:\n> \"Their shortcomings in generated samples can primarily be attributed to model size and their strong focus on music-like audio.\"\n\nHowever, it would be desirable to include a more concrete analysis, such as which features in the proposed method were particularly effective for non-music sounds.\n\n[1] Oumi, Yusuke, et al. \"Acoustic-based 3D Human Pose Estimation Robust to Human Position.\" BMVC2204.\n\n[2] Wang, Mason, et al. \"Soundcam: A dataset for finding humans using room acoustics.\" NeurIPS2023.\n\nQUESTIONS:\n(Major) Regarding the first weakness on robustness to recording environments: even if motion capture data collection is infeasible in diverse settings, qualitative results in environments with significantly different layouts, reverberation characteristics, or outdoor conditions could considerably impact the perceived effectiveness of the method and potentially alter the overall rating.\n\n(Minor) A more detailed discussion on which features or preprocessing steps led to the performance differences between the proposed method and EDGE would greatly strengthen the paper’s technical contribution.\n\n(Minor) While the paper focuses on spatial audio as the primary input, it would be valuable to discuss the potential for incorporating language information to enable more controllable motion generation.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focus on the task of human motion generation driven by spatial audio, which aims to explore the relationship between spatial audio cues and human motion. The paper presents the Spatial Audio Motion (SAM) dataset, which contains annotations of sound source locations and different reaction intensities. The authors introduce the MOSPA, a diffusion-based model that achieves SOTA performance in generating realistic and responsive human motion.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: \n1. This paper aim to addresses the new task of spatial audio-driven human motion generation and proposes a framework to model the relationship between spatial audio cues and human motion. \n2. It propose a new SAM dataset, which offers above 8 hours of annotated motion and spatial audio data, enabling research in this area. The proposed diffusion-based model effectively captures audio-motion interplay and shows strong performance.\n\nWeaknesses: \n1. It only involve 12 participants and 27 audio scenarios may restrict diversity and generalizability. \n2. The model architecture appears relatively simple except the spatial audio feature extraction pipeline, and the absence of ablation studies makes it hard to assess the contributions of individual components. \n3. The lack of evaluation on external datasets raises concerns about robustness and real-world applicability.\n4. Its evaluation is limited to a single dataset, with limited sample size and scene coverage, and lacks ablation studies on the proposed diffusion model. There are concerns about its generalization and robustness due to the lack of testing outside of the SAM dataset. More extensive benchmarks and deeper analysis of the model would help strengthen this work.\n\nQUESTIONS:\n1. Given that the SAM dataset contains a limited set of 12 subjects and 27 spatial audio scenes, how do you account for potential biases in motion responses?\n2. Have you performed ablation studies to understand the respective contributions of MFCCs, temporal maps, and RMS energy to the final representation?\n3. Have you tested MOSPA on other publicly available audio-to-motion benchmarks, even if they lack spatial annotations? \n4. How does the model perform with spatial audio inputs that differ significantly from the training distribution in SAM?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a new task-human motion generation conditioned on spatial audio, along with an 8-hour mocap dataset (SAM) and a spatial audio conditioned motion generation model (MOSPA). Extensive experiments demonstrate the proposed MOSPA can achieve better performance compared to the baseline methods. Qualitative results show responsive motion driven by spatial audio.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n1. Human motion generation driven by spatial audio is an interesting and novel direction, with strong potential for real-world applications. A high-quality, mocap-based paired dataset is a crucial foundation for advancing this line of research. The proposed SAM dataset is collected using a professional motion capture system, with well-documented collection procedures and comprehensive dataset statistics. This dataset represents a significant and valuable contribution to the field.\n2. The model design is very clear and achieve good performance, which set a good benchmark for future research.\n3. The paper presents comprehensive experiments that effectively demonstrate the proposed method and design choices. The demos on out-of-distribution (OOD) spatial audio and simulated character control are also commendable and add practical value to the work.\n\n## Weaknesses\n1. The generated motion exhibits quite a few foot-sliding artifacts, which is worse than motion generation results in other tasks, e.g. music2dance, text2motion.\n2. The methodological contribution appears relatively minor, as the core innovation lies in the spatial audio feature extraction module. It would be worthwhile to include ablation studies on the spatial audio features to better understand their impact and validate the effectiveness of different features.\n\nQUESTIONS:\n1. In Abstract #line17, what is the “privacy-constrained HMR task”? I did not find any reference in the paper. \n2. What is the motivation for including the sound source location as an explicit input condition? From the basic principles of spatial audio, one would expect a model to implicitly infer the source location from spatial audio features. Given that explicit sound source locations are typically unavailable in real-world scenarios, why is this input still deemed necessary? How do the authors interpret the effect of the sound source location on the quality of the generated motion? For example, does the character fail to orient correctly (e.g., look in the wrong direction) without this explicit input?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}