🎧MOSPA: Human Motion Generation Driven by Spatial Audio
Authors: Shuyang Xu
Zhiyang Dou
Mingyi Shi
Liang Pan
Leo Ho
Jingbo Wang
Yuan Liu
Cheng Lin
Yuexin Ma
Wenping Wang
Taku Komura
Venue: NeurIPS 2025 spotlight ()
Keywords: 3D Human Motion Generation
Primary area: applications

Abstract: Enabling virtual humans to dynamically and realistically respond to diverse auditory stimuli remains a key challenge in character animation, demanding the integration of perceptual modeling and motion synthesis. Despite its significance, this task remains largely unexplored. Most previous works have primarily focused on mapping modalities like speech, audio, and music to generate human motion. As of yet, these models typically overlook the impact of spatial features encoded in spatial audio signals on human motion. To bridge this gap and enable high-quality modeling of human movements in response to spatial audio, we introduce the first comprehensive "Spatial Audio-Driven Human Motion" (SAM) dataset, which contains diverse and high-quality spatial audio and motion data. For benchmarking, we develop a simple yet effective diffusion-based generative framework for human "MOtion generation driven by SPatial Audio," termed MOSPA, which faithfully captures the relationship between body motion and spatial audio through an effective fusion mechanism. Once trained, MOSPA can generate diverse realistic human motions conditioned on varying spatial audio inputs. We perform a thorough investigation of the proposed dataset and conduct extensive experiments for benchmarking, where our method achieves state-of-the-art performance on this task. Our code and model are publicly available at https://github.com/xsy27/Mospa-Acoustic-driven-Motion-Generation.git

MOSPA: Human Motion Generation Driven by Spatial Audio
Shuyang Xu Zhiyang Dou Mingyi Shi', Liang Pan', Leo Ho', Jiigbo Wang?, Yuan Liu Cheng Lin", Yuexin Ma Wenping Wangi Taku Komurate
'The University of Hong Kong Shanghai Al Lab
The Hong Kong University of Science and Technology
Maaaau University of Science and Technology ShanghaiTTch University
TTexas A&M University
Figure I: We introduce a novel human motion generation task centered spatial audio-driven human motion synthesis. Top row: We curate a novel Spatial Audio-Driven Human Motion (SAM) dataset, including diverse spatial audio signals and high-quality 3D human motion pairs. Bottom row: We develop generative framework for human MOtion generation driven by SPatial Audio (MOSPA) produce high-quality, responsive human motion driven by spatial audio. We note that the motion generation results are both realistic and responsive, effectively capturing both the spatial and semantic features of spatial audio inputs.
Abstract
Enabling virtual humans to dynamically and realistically respond to diverse au- ditory stimuli remains key challenge in character animation, demanding the integration of perceptual modeling and motion synthesis. Despite significance, this task remains largely unexplored. Most previous works have primarily focused on mapping modalities like speech, audio, and music to generate human motion. As of yet, these models typically overlook the impact of spatial features encoded in spatial audio signals on human motion. To bridge this gap and enable high-quality modeling of human movements in response to spatial audio, introduce the first comprehensive Spatial Audio-Driven Human Motion (SAM) dataset, which contains diverse and high-quality spatial audio and motion data. For benchmarking, we develop a simple yet effective diffusion-based generative framework for human MOtion generation driven by SPatial Audio, termed MOSPA, which faithfully cap- tures the relationship between body motion and spatial audio through an effective fusion mechanism. Once trained, MOSPA can generate diverse realistic human
denote equal contributions and conreponningg authors.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

motions conditioned on varying spatial audio inputs. We perform a thorough investigation of the proposed dataset and conduct extensive experiments for bench- marking, where our method achieves state-of-the-a performance on this task. Our code and model are publicly available at our website.
1 Introduction
Humans exhibit varying responses to different auditory inputs within a given space. For instance, when exposed to sharp. piercing sounds, individuals are likely to cover their ears and move away in the direction opposite to the sound source. Conversely, when the sound is soft and soothing, they may approach it out of curiosity or investigate further. Therefore, generating realistic human motion for virtual characters to respond realistically to a variety of sounds in their environment is both a highly sought-after feature and is crucial for applications such as virtual reality, human-computer interaction, robotics, etc.
Unfortunately, while previous studies have extensively explored motion generation from action label [85, 29], text [93, 100, 77J, music [78, 91, 46, IJ, and speech [2, 97, 3, 102], human motion the best of our knowledge. Unlike pure audio signals, e.g., music [70, 50, 78], speech [3, 97]. the spatial audio signals not only does it encode semantics, but it also captures spatial characteristics that significantly influence body movements, requiring a specialized framework to accurately model motion responses to spatial audio stimuli.
generation driven by spatial audio remains unexplored
To address this overlooked aspect, we propose model the complex interactions between spatial audio inputs and human motion using a generative model. Since there is no such dataset tailored for this task, first introduce the SAM dataset (Spatial Audio Motion dataset), which captures diverse human responses to various spatial audio conditions. This dataset is meticulously curated to include wide range of spatial audio scenarios, enabling the study of motion conditioned on sound field variations. The SAM datasel has a total of more than 9 hours of motion, covering 27 common spatial audio scenarios and more than 70 audio clips. To ensure the diversity of the spatial audio, around 480 seconds of motion were captured for each audio clip at different positions in the character space. To ensure diverse motion responses to spatial audio, we introduce 20 distinct motion types (excluding motion genres) and 49 in total when including motion genres. We visualize samples from SAM in the top row of Fig. See Appendix for detailed statistics.
We further conduct benchmarking experiments on the proposed dataset, revealing the limitations of existing methods in this setting. To enable spatial audio-driven human motion generation, we introduce MOSPA, a simple yet effective framework tailored for this task. In real-world scenarios, human responses to sound are inherently influenced by spatial perception, intensity variations, directional cues, temporal dynamics, etc. Motivated by this, we generate motion by incorporating features extracted from the input spatial audio signals using [61]. Specifically, to capture intrinsic features across both temporal and spatial dimensions, we mainly utilize Mel-Frequency Cepstral Coefficients (MFCCs) [7] and Tempograms[28] to model the temporal characteristics of the audio. Additionally, we characterize the spatial audio by analyzing the root mean square (RMS) [61] energy. which quantifies signal intensity in audio processing.
These features enhance the effective modeling of the spatial and intensity variations of the spatial audio. To capture the distribution of spatial audio features and human motion dynamics effectively, we employ a diffusion-basser generative model that ensures strong alignment between the two modalities-humal motion and spatial audio signals. Leveraging diffusion models, MOSPA excels at modeling the complex interplay between spatial audio features and human motion. Besides, a residual feature fusion mechanism is employed to model the subtle influences of spatial audio on human movement.
Extensive evaluations on the SAM demonstrate that MOSPA achieves state-of-the-ee performance on this task, outperforming existing baselines in generating realistic and diverse motion responses to spatial audio. Our contributions are summarized as follows:
We introduce a novel task of spatial audio-conditioned motion generation and present the first comprehensive dataset SAM with over 9 hours of motion across diverse scenarios.
We conduct extensive benchmarking and propose MOSPA, diffusion- based generative frame- work tailored for modeling and generating diverse human motions from spatial audio.

---

We achieve the SOTA performance on motion generation conditioned on spatial audio. Our dataset, code, and models will be publicly released for further research.
2 Related Work
Spatial Audio. Many studies have explored spatial audio modeling 198, 24, 99, 74, 87, 44, 45]. For instance, [98] utilizes the natural synchronization between visual and audio modalities to learn models that jointly parse sounds and images without manual annotations. leverages unlabeled audiovisual data to localize objects, such as moving vehicles, using only stereo sound at inference time. [99] reason about spatial sounds with large language models. Recently, spatial audio generation has been explored from text [74] and video [42]. [87] propose method to model 3D spatial audio from body motion and speech. [37] presents a framework for spatial audio generation, capable of rendering 3D soundfields generated by human actions, including speech, footsteps. and hand- body interactions. Despite progress on spatial-audio tasks, generating human motion from spatial audio remains largely underexplored.
Conditional Motion Generation. Extensive efforts have been made into motion synthesis condition- ing on user control signals [36, 72, 11, 79, 92], text [93, 100, 77, 30, 51, 53, 14], action [85, 29, 39], music [78, 91, 46, IJ. speech [2, 68, 38, 97, 3, 102]. past trajectories [23, 9, 4, 90, 73], etc. We refer readers to [103] for detailed survey of motion generation.
Text-to-Motion. Text-to-motion generation has recently gained popularity as an intuitive and user- friendly approach synthesizing diverse human body motions. Generative pre- pre-trained transformer frameworks have been utilized for rext-to-motion generation [93, 57]. Subsequently, various genera- tion techniques have been explored, including diffusion models [77], latent diffusion models [12], autoregressive diffusion model [69, 11], denoising diffusion GANs [100], consistency models [16], and generative masked modeling [31]. Recent advancements include the integration of motion generation with large language models [41, 94] and investigations into the scaling laws for mo- tion generation [58, 22]. Recently, controllable text-to-motion generation has gained attention, enabling motion synthesis conditioned on both text prompts and control signals, e.g., target control points [84, 79].
Music-to-Motien Recent advancements in Music-to-Motie generation have been made [21, 70, 48, 78, 27, 88, 71]. DanceFormer [47] adopts a two-stage approach, generating key poses for beat synchronization followed by parametric motion curves for smooth,
movements. Bailando [70] utilizes a VQ-VAE to encode motion features via a choreographic memory module. [1] introduces diffusion-based prohabilistic model for motion generation, using a Conformer- architecture. EDGE [78] also applies a diffusion model for dance generation and editing. Furthermore, multimodal approaches incorporating language and music enhance generation quality [27, 88, 15]. Speech-to-Motion. We mainly review studies on audio-driven motion (gesture) generation [97, 13, 2, 102]. Early works are mostly based GAN models [26, 54, 65, 89], while the recent at- tempts are mainly based on the generative diffusion model [102]. For instance, [97] proposes a generative retrieval framework leveraging a large language model to efficiently retrieve semantically appropriate gesture candidates from a motion library in response to input speech. [2] introduces a co-speech gesture synthesis method by employing segmentation pipeline for temporal alignment and disentangling speech- -nttinn embeddings to capture both semantics and subtle variations.
While audio signals have been widely used in music- and speech-to- to-ttoooomoonnnoooo tasks, human motion synthesis driven by spatial audio remains largely unexplored. As result, data-driven methods are highly constrained by limited paired data. The goal of this paper is develop a comprehensive dataset and a novel approach for high-quality spatial audio-driven motion synthesis.
3 SAM Dataset
We first introduce the Spatial Audio-driven Motion (SAM) dataset designed for human motion synthesis conditioned on spatial audio. We focus on binaural audio, a common form of spatial audio that aligns with human and (most) animal perception and can be readily applied to robotic platforms. SAM consists of more than 9 hours of human motions with corresponding binaural audio, and more than 4M frames, covering 27 common spatial audio scenarios and 20 common reaction types in daily life without counting the motion genres. The majority of the audio clips are sourced from the AudioSet [25], while only small portion is extracted from publicly available YouTube videos
3

---

Table 1: Statistics of the SAM dataset. The SAM dataset encompasses 27 common daily spatial audio scenarios, over 20 reaction types excluding the motion genres. and 49 reaction types (see details in Appendix A). The number of subjects covered in SAM is 12, where of them are female and the remaining are male. It is also the first dataset to incorporate spatial audio information, annotated with Sound Source Location (SSL). The total duration of the dataset exceeds 34K seconds.
Dataset
SSL
3DJoinl
Model
Joints
Subjects
Seconds
Dance with Melody [75]
V1%
21
5640
DanceNet [104]
VIx
55
2
3472
AIST+ [48]
VI
COCO/SMPL
17/24
30
18694
PupDanceSet [60]
VIV
COCO/SMPL
17/24
132
12819
FineDance [49]
VI
SMPL hand joints
52
27
52560
SAM (Ours)
VIV
SMPL-X
55
34356
Look for sound souree and approach upon hearing miaow the left-hand side.
(b) Step away with covered upon hearing crowd yelling at the back.
(c) Run away from the sound upon hearing gunshot the right hand side.
dd) Walk towards the sound source upon hearing insect noise at the left-hand side.
(e) Start dancing upon hearing music in front of the character.
(f) Look for sound source upon hearing the phone ring at the left-hand side.
Figure 2: Visualization of samples from SAM with expected motions annotated. Red dots indicate the actor's trajectory, while the blue sphere represents the sound source. The SAM dataset ensures high diversity by encompassing a broad spectrum of audio types and varying sound source locations.
or through manual recording. More detailed information can be found in Tab. and Appendix A. Visualization results are in Fig. 2.

---

300
lndi att
so ore ande
zon hodd evenne
dodie
dere
fo
por
(i) Mocap environment.
(ii) The speakers.
Figure 3: Spatial audio-driven human mo- tion data collection setup.
Figure 4: Statistics of action duration in the dataset.
Data Capture Settings. We utilize Vicon motion capture system [56] collect motion data and spatial audio signals. The motion capture is performed in a semi-open cage having a space of approximately 5m x 10m 3m, a structure covered by rope nets, with 28 mocap cameras mounted on the ropes and vertical supports recording at a frame rate of 120 Hz; See Fig. 3. The surrounding walls are standard painted concrete, resulting in a setting that resembles typical indoor environment. In SAM, each audio clip is associated with 16 randomly sampled relative sound source locations, defined by combinations of different speakers and spatial positions relative to the subject. For each location, we capture three motion sequences corresponding to different reaction intensities:
dull, neutral, and sensitive, resulting in total of 48 motion sequences, each lasting 10 seconds. Fig. 4 shows the statistics of the approximate duration of the actions. The three motion genres define the varying degrees of responsiveness, decreasing from sensitive to dull. For instance, upon hearing largely unreactive, whereas a sensitive one may immediately flee from the sound source. The total number of action types is 49. The percentage of action types covered within the dull, neutral, and sensitive motion genres are 28.57%, 34.69%, and 36.73% respectively. To capture the binaural sound heard at the position of the actor, employ two microphones to record the audio at the ear positions of the actors separately; See the inset. The
explosion, a dull individual might remain
two microphones are connected to Deity PR-2 recorder [18] that has been synchronized with the Vicon mocap system in advance using a timecode with a frame rate of 30 FPS. With this setting, the stereo sound the position of the actor can be recorded and has accurate alignment with the corresponding motion.
Data Processing. All motion and audio clips are precisely aligned. The motions are re-targeted and converted from the original BVH format in Vicon to the SMPL-X [63] format. SMPL-X parametric 3D human body model that encompasses the body, hands, and face, comprising N = 10, 475 vertices and K = 55 joints. Given shape parameters / and pose parameters 6, the SMPL-X model generates the corresponding body shape and pose through forward dynamics. We extract the locations of the sound sources in each motion clip. The sound source locations are then transformed into the local space of the character aligned with the SMPL-X local coordinate system (a.k.a the local frame).
4 Method
We introduce MOSPA, a diffusion- probabilistic model that serves as a baseline for this novel task of spatial audio-driven human motion generation. First, we extract spatial audio features a using a feature extractor [61]. During motion generation, the extracted spatial audio feature a is combined with the sound source location and the motion genre g as conditioning inputs. These inputs passed to a denoiser G, which is trained to reconstruct the original clean motion vector Xg by denoising the given noisy motion vector X₁ at time step l. Mathematically, we have Xn = 9(x,, L; a, s,
4.1 Feature Representation
The two key vectors are the audio feature vector a and the motion vector p. We carefully designed the structure of the two vectors in MOSPA. Spatial Audio Feature Extraction. We first extract a range of audio features that capture intensity, temporal dynamics, and spatial characteristics. Inspired by [70], our feature set primarily includes Mel-
5

---

frequency cepstral coefficients (MFCCs), MFCC delta, constant-Q chromagram, short-time Fourier transform (STFT) of the chromagram, onset strength, tempogram, and beats [17, 67, 28, 61. 7, 20]. On top of these audio features, we additionally add the root mean square (RMS) energy Frms of the audio [61], and the active frames Factive defined as Factive = Frrms > 0.01 to capture the distance information of the audio. The dimension of the audio feature vector for each ear is 1136. By concatenating the features from both ears, we obtain a combined feature vector a of dimension 2272. The detailed construction of the audio vector can be viewed in Appendix B.
Motion Representation. In this paper, we focus on body motion and leave the modeling of detailed finger movements to future work. Therefore, exclude all the finger joints and retain only the first == 25 body joints of the SMPL-X model [63]. In addition to the essential translation and joint rotations required for human pose representation, we introduce the residual feature fusion mechanism [30] to incorporate the global joint positions and the velocity of the joints to capture the nuanced difference in audio and further improve the accuracy of the generated samples. Each motion vector is thus composed of the global positions p E (x3x, the local rotations E gTx( and the velocities v E R [Jxx( of the joints (including the root), where T = 240 represents the number of frames in each motion sequence. The joint rotations are represented in the 6d format [101] to guarantee the continuity of the change (Xo (Po. ro, vo), x E (Tx(Jx12)) The dimension of each motion vector is therefore 300.
4.2 Framework
Following [77. 11), the diffusion is modeled as Markov chain process which progressively adds noise to clean motion vectors Xo in time steps, i.e.
Human Moliar
() (1)
p(r,y()))
where at E (0, 1). The model then learns to gradually denoise noisy motion vector Xt in time steps, i.e. p(X)-1|x)). We directly predict the clean sample Xg in each diffusion step Xo G(x,, t; a. s. g), where is the audio features, s is the sound source lo- cation and g is the motion genre. This strategy, employed by [77, 11, 66, 84], has been proved to be more efficient and accurate than predicting the noise ér, suggested by [35]. We employ an encoder-only Transformer to reverse the diffusion process and predict the clean samples. The timestep, motion, and con- ditioning signals are each projected into the same latent dimension using sepa- rate feed-forward networks. Random masks are applied to the audio features a and the sound source location (SSL) s, after which all components are concatenated to form the complete token sequence z. The tokens are positionally embedded afterward and input into a transformer to get the output The predicted clean sample is thus extracted from the last T tokens of 2 by inputting it to another feed-forward network, where T 240 is the length of the motion; see Fig. 5.
Acoustic Field
Transformer Encoder
Pis Fimbedding
Acoustic Extraction
Lineer
MLP
Uneer
Figure 5: The framework of MOSPA. We perform diffusion-based motion generation given spatial audio in- puts. Specifically, Gaussian noise is added to the clean motion sample Xo. generating a noisy motion vector Xt. modeled as g(x,) ((x)/xt1)). An encoder transformer then pre- dicts the clean motion from the noisy motion Xt guided by extracted audio features sound source location (SSL) s, motion genre g. and timestep t.
4.3 Loss Functions
We train MOSPA using the following loss functions. A simple mean squared error (MSE) loss is applied to the original clean sample and the predicted clean sample as the main objective: E||xo
xolle. To guarantee the smooth variation on the predicted clean sample across frames, we also apply MSE loss to the rate of change of the vectors across frames: E. | 5xol1 Combining the two simple losses we have Letata = E xollž + El Sxoll.. Geometric losses, encompassing position loss and velocity loss, are also incorporated, as we rely solely joint

---

EDGE
POPDG
LODGE
Bailando
MOSPA (Ours)
Audio
Bicycle Ring
Crowd yell
Firework
Explosion
Music
Genre
Neutral
Sensitive
Sensitive
Sensitive
Dull
Expected
Step aside upon
stop
stop
away and squat down when hearing explosion
dancing upon hearing music
hearing a bicycle ring
back upon hearing crowd yell
back slowly hearing firework
Motion
Figure 6: Qualitative comparison of state-of-the-arr methods for the spatial audio-to- to-totnn task. We visualize motion results from five cases. MOSPA produces high- movements that closely correspond to the input spatial audio. We provide Expected Motion as a description for reference.
rotations and translations motion vectors to represent poses: Lyea = ( FK (xx)))l (1))FK(())) - 3FK((xo)) Furthermore, foot sliding prevented by introducing foot contact loss L that measures the inconsistency in the velocities of the foot joints between the ground truth and the predicted motions.
We also incorporate trajectory loss and joint rotation loss to underscore their importance in achieving the training objectives and accelerate the convergence of the model, defined as Lirej = E E|[trajo trajoll2 + BBjtrajo strajo & and frot El roll3 + E of, ăroll3 respectively, where traj is the trajectory vector of the motion sequence and r is the joint rotations represented in the 6d format [101]. Given that trajectory and joint rotations are inherently encoded within the motion vectors, these supplementary losses represent an overlap with the existing loss terms, effectively amplifying the emphasis on trajectory and joint rotation accuracy through increased weighting. Empirically, we observe that this implementation accelerates model convergence and facilitates correct displacement direction generation in motion sequences. In sum, the total loss is given by:
C Adata Ldata + Agea + A foot + Atraj + ArotCrot
All loss weights (A) are initialized set to 1. At epoch 5,000 of the total 6,000 training epochs, Asraj and Ayot are increased to 3, thereby intensifying the emphasis on trajectory and rotation accuracy.
4.4 Implementation Details
In MOSPA, the diffusion model is transformer-base diffusion network [11, 77, 100]. The encoder transformer is configured with latent dimension of 512, heads, and layers. We employ AdamW [55] as the optimizer with initial value of 1 x 10-1 The number of denoising steps used 1000, and the noise schedule is cosine. The training phase concludes after 6, 000 epochs. Exceeding these recommended epoch counts may degrade model quality due to overfitting. The

---

Table 2: Quantitative evaluation on the SAM, where MOSPA achieves higher alignment with the GT motion while maintaining high diversity, as reflected by the metrics. The error bar is the 95% confidence interval assuming normal distribution, and + means the closer to Real Motion the better.
R R-precision f
Method
FID 4
Diversity -> APD
Topl
Top2
Top3
Real Motion
1. +0.000
1.000
1.000#0.000
0.001
23.616
59.435
EDGE [78]
0.886+0.005 +00005
0.886
0.960 +0.003
0.977
13.993
23.099
10.196
43.882
POPDG [60]
0.762+0.006
0.886 +0,005
0.934 00.004
20.967
22. 22.536
+0.177
40.170
34.996
LODGE [50]
0.000
0.594
0.679=0.004
102.289
21.101
20.14
11.801
Bailando [70]
0.077#0.005
0.134
0.182=#.004
168.396
17.347
23.121
MOSPA 0.937=0.005,
0,984 2000000.0022 002
0.996 n00
7.981 23.575 #0.188 53.915
Table 3: Ablation study on MOSPA on the spatial audio-driven motion generation performance. The error bar is the 95% confidence interval assuming normal distribution, and + means the closer to real motions the better.
Latent
Head Diff'
R- R-precision
Genre
FID + Diversity + APD +
Dim
Num Steps
Top1
Top2
Top3
Real Motion
1.000 +0.000
+.000
+.000
0.001
23.616 10.188
59.435
512
1000
0.937=0.005
0.984
0.996#0.001
7.981
23.575
20.15.0.11 188
53.915
256
8
1000
0.891 20.005
0.9952200002
0.971 20.001
9.226
23.00720.196
55.175
512
4
1000
0.923=+0.004
0.972+00.002
0.986=#0.00
9.282
23,
56.572
512
8
100
0.930 10.0000.000 004
1.0.002
0.991
1:00001
8.456
23.351
1:0177
49.824
512
8
4
0.934 0.934-10.004 000
0.989
0.998 0.989=0.002
001
8.387
23.474 19011
49.507
512
1000
x
0.889
0.958+000003
0.977#0.002
10.930
23.150
46.807
entire training process requires approximately 18 hours on a single RTX 4090 GPU with a batch size of 128.
5 Experiments
Experiment Setup. We use our SAM dataset to evaluate the spatial audio-driven motion generation task. As detailed in Sec. 3, it contains 9 hours of human motion with paired binaural audio and corresponding sound source locations, covering 27 common spatial audio scenarios and 20 common reaction types. The dataset is split into training, validation, and test sub-datased at common ratio of 8:1:1. Consequently, the training sub-dataset comprises 2,400 motion sequences, while the validation and test sub-datasets each contain approximately 300 motion sequences. To keep fair setting [10], the motions and the audio clips are both downsampled to the frame rate of 30 FPS. The character is rotated to face the negative y-axis and initially translated to the origin in the world space in all motion sequences, and the sound source locations (SSL.) are transformed to the local space of the character in every single frame.
Baselines and Metrics. Our system is the first work to receive spatial audio as input to generate human motion results. To our best knowledge, as there no other system achieving this, we made adaptations on other audio2motion methods, such as EDGE [78], POPDG [60]. LODGE [50] and Bailando [70] by replacing their original audio input with our spatial audio feature as input. We evaluated four metrics, focusing on motion quality and diversity: 1) R R-precision. FID, Diversity These three metrics are calculated using the same setup proposed by [30]. Two bidirectional GRU are trained with hidden size of 1024 for 1, 1,500 epochs with batch size of 64 to extract the audio features and the corresponding motion features, as suggested by [30]. Detailed implementation details of the feature extractor are provided in Appendix B.2. 2) APD [19, 33] is calculated by
APD(M) = Nहे Σ
EL lis - where M = {X2} is the set of generated
motion sequences, N is the number of motion sequences in the set M, the number of frames of each motion sequence, and si Xi is state in the motion sequence x1.
8

---

5.1 Comparisons
Qualitative Results. We demonstrate the qualitative comparison in Fig. 6. For the same input spatial audio, our methods show the superiority of producing high-quality and realistic response motion. Other methods often exhibit various limitations due to their unique model characteristics. EDGE [78] and POPDG [60] demonstrate relatively strong performance among the four basclines, sharing diffusion- based foundation with MOSPA, despite differences in their encoding and decoding mechanisms. Their shortcomings in generated samples can primarily be attributed to model size and their strong focus on music-like audio. The bad performance of LODGE [50] is likely due to its specialization in long-term music-like audio, resulting deficiencies when handling short-term audio information with abrupt feature changes. Similarly, Bailando [70] faces challenges in processing rapidly changing spatial audio. More critically, due to its separate training process for upper and lower body parts, Bailando occasionally produces distorted or disjointed motions when encountering sudden changes in spatial audio. Please watch our supplementary video for more results, Furthermore, we test MOSPA on motion quality and intent alignment, demonstrating robustness to unseen spatial setups.
audio-source configurations. As shown in Fig. 8, it maintains
Quantitative Results. The quantitative results are reported in Tab. 2. MOSPA achieves the best per- formance as shown by the lowest FID value and the highest R-precision values. Also, our generated motions exhibit the closest diversity and APD [19] values compared with the Real Motion, demon- strating the effectively balanced variation and precision. Bailando [70] has the worst performance among the four baselines in practice, as illustrated by the extremely high FID. The model possibly lacks the ability to perceive commonly heard sounds other than music and also the spatial information of the audio. Our method, overall speaking, still demonstrates competitive performance in spatial audio conditioned motion generation, which is proved by the low values in precision- metrics and the high values in diversity-related metries.
User Study. We conducted a user study with 25 participants to assess the perceptual qual- ity of motion generation. Participants evaluated five models (MOSPA, EDGE, POPDG, LODGE, Bailando) alongside ground truth (GT), select- ing the best motion for: 1) Human Intent Align- ment: Does the motion align with world intent? 2) Motion Quality: Which has the high- est movement quality? 3) GT Similarity: Which best matches the GT motion? We provided GT motion and a textual description for reference. As shown in Fig. 7, MOSPA outperforms all baselines across all criteria, while LODGE and Bailando received the fewest selections, indicat- ing limitations in generating realistic, semanti- cally meaningful motions. See more details in Appendix C.
Human Intent Allgnment
Motion Quality Ground Truth Similarity
Frequency
80-
80
40
20
MOSPA (Qurs)
Edge
Lodge
DGp
Ballando
Figure 7: User study results. MOSPA outperforms other methods in intent alignment, motion quality, and similarity to ground truth. The bar chart shows the vote distribution across methods.
5.2 Ablation Study
We conducted ablation studies on the latent dimension, the number of attention heads, the diffusion step number, and the masking of motion genre, with results summarized in Tab. 3. All ablation experiments maintained consistent training epoch counts throughout.
Latent Dimension. The default latent dimension of MOSPA's 's encoder transformer is 512. In our study, reducing 256 slightly increases the APD [19] value but also degrades the R-precision and the FID, leading to an overall decline in model performance, as seen in row I and row 2 in Tab. 3. Number of Attention Heads. We reduced the number of attention heads in MOSPA "s encoder transformer from 8 to 4, observing degradation in almost all of the metrics except a slight improvement in APD [19]. This reduction compromises overall model performance without yielding significant improvements in training efficiency, as seen in row and row 3 in Tab. 3.
Number of Diffusion Steps. We evaluated MOSPA with varying diffusion step numbers, reducing it from 1000 to 100 and further to 4, as detailed in rows 1, 4, and 5 of Tab. 3. Fewer steps slightly degrade the performance as shown by the increase in FID and degradation in diversity, thereby lowering the upper limit of the power of the model.

---

Step back look forward upon hearing fireworks.
Walk back and look around upon hearing vehicle horn.
Figure 8: Test of MOSPA out-of -itto--distribttion spatial audios. Descriptions of motions are provided for reference.
Look for sound source and spproach upon hearing misow at the left hand side.
Get away from sound source wave hands when hearing noisy insects.
Figure 9: Spatial audio-driven physically simulated humanoid robot control based on [34]. Descrip- tions of expected motion are provided for reference.
Genre Masking, Masking motion genres leads to a degradation in model performance across all metrics, as demonstrated in row and of Tab. 3. Motion genres are required to provide guidance for the model on the intensity of the expected motions.
We evaluate the contribution of
Table 4: Ablation study on the effect of MFCC [17] and tempogram [28] features.
the extracted audio features by conducting an ablation study on the effectiveness of MFCC [28] fea-
R precision 1
MFCC
Tempogram
FID
Top-1
Top-2
Top-3
[17] and tempogram tures. As shown in Tab. 4, improvements in FID and precision two key metrics for assessing generative quality and correspondence demonstrate their significance model.
0.937
0. 0.984=0.00m
0.906
7.981
0.907=0.004
0.967 t.0002
0.983
353300.002
9.070
x
00.004
40.000
0.994 #00001
10.786
6 Conclusion
This introduces a novel task for enabling virtual humans to respond realistically to spatial auditory stimuli. We present comprehensive SAM dataset, capturing human movement in response to spatial audio, and propose MOSPA, a diffusion- iiffusiio-bbsed generative model with an attention. based fusion mechanism. Once trained, MOSPA synthesizes diverse, high-quality motions that adapt to varying spatial audio inputs with binaural recording. Extensive evaluations show MOSPA achieves state-of-the-art performance While MOSPA generates diverse and semantically plausible motions, it lacks physical constraints, which may lead to physically implausible artifacts. Integrating physics-based based control methods [19, 59, 76, 40, 95, 96, 62, 82] could improve motion realism and embodiment fidelity (see Fig. 9 for spatial audio-driven humanoid robot control). Body Modeling: This work focuses on body motion and omits finer mmnineg components such as hand gestures and facial expressions supported by SMPL-X [63]. Extending the model full-body motion generation [57, 52, 86, 64, 5] including hand motions remains an important direction for future research. Scene Awareness: The current framework does not incorporate awareness of surrounding environments or physical scene geometry, limiting its ability to produce scene-consistent contact-aware motions. Future extensions could integrate scene representations or affordance prediction [14, 80, 83, 8, 81] with spatial audio signals to enhance human motion generation.
this task. Limitations and Future Works. Physical Correctness:
10

---

Acknowledgements
This work is partly supported by the Innovation and Technology Commission of the HKSAR Government under the ITSP-Platform grant (Ref: ITS/335/23FP) and the InnoHK initiative (TransGP project). Part of the research was conducted in the JC STEM Lab of Roboties for Soft Materials, funded by The Hong Kong Jockey Club Charities Trust. We are grateful to Yiduo Hao, Chuan Guo, Chen Wang, Zitong Lan, and Peter Qingxuan Wu for their insightful discussions and constructive feedback, which have been helpful to the development of this research.
References
[1] Simon Alexanderson, Rajmund Nagy. Jonas Beskow, and Gustav Eje Henter. Listen, denoise, action! audio-driven motion synthesis with diffusion models. ACM Transactions on Graphics (TOG), 42(4):1-20, 2023.
[2] Tenglong Ao, Qingzhe Gao, Yuke Lou, Baoquan Chen, and I Libin Liu, Rhythmic gesticulator: Rhythm- aware co-speech gesture synthesis with hicrarchical neural embeddings. ACM Transactions on Graphics (TOG), (1(6):1-19 2022.
[3]
Tenglong Ao, Zeyi Zhang, and Libin Liu. Gesturediffuclip: Gesture diffusion model with clip latents. ACM Transactions on Graphics (TOG), 4284):1-18, 2023.
[4] German Barquero, Sergio Escalera, and Cristina Palmero. Belfusion: Latent diffusion for bchavior-driven
human motion prediction. In Proceedings of the IEEE/CVF international conference on computer vision, pages 2317-2327, 2023.
[5] Yuxuan Bian, Ailing Zeng. Xuan Ju, Xian Liu, Zhaoyang Zhang, Wei Liu, and Qiang Xu. Motioneraft: Crafting whole- body motion with plug-and-pg and-play multimodal controls. In Proceedings of the AAAI Conference Artificial Intelligence, pages 1880-1888. 2025.
[6] RB Blackman and JW Tukey. The mcasurement of power spectra dover publications. Inc, New York, 1958.
[7] Sebastian Bück and Gerhard Widmer. Maximum filter vibrato suppression for onset detection. In Proc. of the 16th Int. Conf. on Digital Audio Effects (DAFx). Maynooth, Ireland (Sept 2013), page 4. Citeseer, 2013.
[8] 7hi Cen, Huaijin Pi, Sida Peng, Zehong Shen, Minghui Yang, Shuai Zhu, Hujun Bao, and Xiaowei 7hou. Generating human motion in 3d scenes from text descriptions. In Proceedings of the IEFF/CVF conference on computer vision and pattern recognition, pages 1855-1866, 2024.
[9] Ling-Hao Chen, Jiawei Zhang, Yewen Li, Yiren Pang, Xiaobo Xia. and Tongliang Liu. Humanmac: Masked motion completion for human motion prediction. In Proceedings of the IEEE/CVF international conference on computer vision, pages 9544-9555, 2023.
[10] Ling-Hao Chen, Shunlin Lu, Wenxun Dai, Zhiyang Dou, Xuan Ju, Jingbo Wang. Taku Komura, and Lei Zhang. Pay attention and move better: Harnessing attention for interactive motion generation and 2024.
training-fiee free editing. arXiv preprint arXiv:2410.
[11] Rui Chen, Mingyi Shi, Shaoli Huang, Ping Tan, Taku Komura, and Xuelin Chen. Taming diffusion prohabilistic models for character control. In ACM SIGGRAPH 2024 Conference Papers, pages 1-10, 2024.
[12] Xin Chen, Biao Jiang, Wen Liu, Zilong Huang, Bin Fu, Tao Chen, and Gang Yu. Executing your commands via motion diffusion in latent space. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18000-18010, 2023.
[13] Qingrong Cheng, Xu Li, and Xinghui Fu. Siggesture: Generalized co-speech gesture synthesis via semantic injection with large-scale pre-training diffusion models. In SIGGRAPH Asia 2024 Conference Papers, pages 1-11, 2024.
[14] Peishan Cong, Ziyi Wang. Zhiyang Dou, Yiming Ren. Wei Yin, Kai Cheng, Yujing Sun, Xiaoxiao Long, Xinge Zhu, and Yuexin Ma. Laserhuman: language-guidee guidedd scene-aware human motion generation in free environment. arXiv preprint arXiv:2403.1333
2024.
[15] Rishabh Dabral, Muhammad Hamza Mughal, Vladislav Golyanik, and Christian Thcobalt. Mofusion: motion synthesis. In Proceedings of the IEEE/CVF conference computer vision and puttern recognition, pages 9760-9770, 2023.
framework for crmnoiiing-diffusion---
[16] Wenxun Dai, Ling-Han Hao Chen, Jingbo Wang, Jinpeng Liu, Bo Dai, and Yansong Tang. Motionlem: Real-time controllable motion generation via latent consistency model. In European Conference on Computer Vision, pages 390-408 408. Springer, 2024.

---

[17] Steven Davis and Paul Mermelstein. Comparison of parametric representations for monosyllabic word recognition in continuously spoken sentences. IEEE transactions on acoustics, speech, and signal processing, 28(4):3573366, 1980.
2025. Accessed: Feb 5, 2025. [19] Zhiyang Dou, Xuelin Chen, Qingnan Fan, Taku Komura, and Wenping Wang. C- ase: Learning conditional adversarial skill embeddings for physics-bused characters. SIGGRAPH Asia 2023 Conference Papers, pages 1-11, 2023.
[18] Deity Microphones. Pr-2.
[20] Daniel PW Ellis. Beat tracking by dynamic programming. Journal of New Music Research, 36(1):51 -00, 2007.
[21] Di Fan, Lili Wan, Wanru Xu, and Shenghui Wang. A bi-directional attention guided cross-modal network for music based dance generation. Computers and Electrical Engineering, 103:108310, 2022.
[22] Ke Fan, Shunlin Lu, Minyue Dai, Runyi Yu, Lixing Xiao, Zhiyang Dou, Junting Dong. Lizhuang Ma, and Jingbo Wang. Go zero: Towards zero-shot motion generation with million-scale data. arXiv preprint MXXiv:5507007095. 2025.
[23] Yuming Feng, Zhiyang Dou, Ling- Hao Chen, Yuan Liu, Tianyu Li, Jingho Wang, Zeyu Cao, Wenping Wang. Taku Komura, and Lingjic Liu. Motionwavelet: Human motion prediction via wavelet manifold learning. arXiv preprint arXiv:2411.16964 2024.
[24] Chuang Gan, Hang Zhao, Peihao Chen, David Cox, and Antonio Torralba. Self sepe-supe moving vehicle tracking with stereo sound. In Proceedings of the IEEE/CVF international conference on computer vision, pages 7053-7062, 2019.
[25] Jort F Gemmeke, Daniel PW Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R Channing Moore, Manoj Plakal, and Marvin Ritter. Audio set: An ontology and suman-labeled dataset for audio events. In 2017 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 776-780. IEEE, 2017.
[26] Shiry Ginosar, Amir Bar, Gefen Kohavi, Caroline Chan, Andrew Owens, and Jitendra Malik. Learning individual styles of conversational gesture. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3497-3506, 2019.
[27] Kehong Gong, Dongze Lian, Heng Chang, Chuan Guo, Zihang Jiang. Xinxin Zuo, Michael Bi Mi, and Xinchao Wang. Tm2d: Bimodality driven 3d dance generation via music-text integration. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9942 9952, 2023.
[28] Peter Grosche, Meinard Müller, and Frank Kurth. Cyclic tempogram- mid-level tempo representation for musicsignals. In 2010 IEEE International Conference on Acoustics, Speech and Signal Processing. pages 5522-5525, JEEE, 2010.
[29] Chuan Guo, Xinxin Zuo, Sen Wang, Shihao Zou, Qingyao Sun, Annan Deng, Minglun Gong, and Li Cheng. Action2motion: Conditioned generation of 3d human motions. In Proceedings of the 28th ACM International Conference on Multimedia, pages 2021-2029, 2020.
[30] Chuan Guo, Shihao Zou, Xinxin Zuo, Sen Wang, Wei Ji, Xingyu Li, and Li Cheng. Generating diverse and natural 3d human motions from text. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5152-5161, 2022.
[31] Chuan Guo, Yuxuan Mu. Muhammad Gohar Javed, Sen Wang, and Li Cheng. Momask: Generative masked modeling of 3d human motions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1900-1910, 2024.
[32] Raia Hadsell, Sumit Chopra, and Yann LeCun. Dimensionality reduction by learning an invariant mapping- 2006 JEEE computer society conference on computer vision and pattern recognition (CVPR 006), pages 1742. IEEE, 2006.
[33] Mohamed Hassan, Duygu Ceylan, Ruben Villegas. Jun Saito, Jimei Yang, Yi Zhou, and Michael J Black. Stochastic scene-aware motion prediction. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11374- 17744-113844 11384, 2021.
[34] Tairan He, Jiawei Gao, Wenli Xiao, Yuanhang Zhang, Zi Wang. Jiashun Wang, Zhengyi Luo, Guanqi He, Nikhil Sobanbab, Chaoyi Pan, et al. Asap: Aligning simulation and real-world physics for learning agile humanoid whole-body skills. arXiv preprint arXiv:2502.00 01 143, 2025.
[35] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:684006851, 2020.
[36] Daniel Holden, Taku Komura, and Jun Saito. Phase-functioned neural networks for character control.
ACM Transactions on Graphics (TOG), 36(4):1-13, 2017.
[37] Chao Huang. Dejan Marković, Chenliang Xu, and Alexander Richard. Modeling and driving human body soundtields through acoustic primitives. In European Conference on Computer Vision, pages 1 177 Springer, 2024.
12

---

[38] Yinghao Huang, Leo Ho. Dafei Qin, Mingyi Shi, and Taku Komura. Interact: Capture and modelling of realistic, expressive and interactive activities between two persons in daily scenarios. arXiv preprint arXiv:240S. 16690, 2024.
[39] Yiming Huang, Weilin Wan, Yue Yang, Chris Callison-Burch, Mark Yatskar, and Lingje Liu, Como: Controllable motion generation through language guided pose code editing. In European Conference on Computer Vision, pages 180-196. Springer, 2024.
[40] Yiming Huang, Zhiyang Dou, and Lingjie Liu. Modskill: Physical character skill modularization. arXiv preprint arXiv:2502. 14/40, 2025.
[41] Biao Jiang, Xin Chen, Wen Liu, Jingyi Yu, Gang Yu, and Tao Chen. Motiongpt: Human motion as a foreign language. Advances in Neural Information Processing Systems, 36:20067-2000 220079, 2023.
[42] Jaeyeon Kim, Heeseung Yun, and Gunhee Kim. Visage: Video-to-spatial audio generation. arXiv preprint arXiv:2506. 12199, 2025.
[43] Diederik Kingma and Jimmy Ba, Adam: A methad for stochastic optimization. arXiv preprint arXiv: 2014.
[44] Zitong Lan, Chenhao Zheng, Zhiwei Zheng, and Mingmin Zhao. Acoustic volume rendering for neural impulse response fields. arXiv preprint (tti:2411.066300 2024.
[45] Zitong Lan, Yiduo Hao, and Mingmin Zhao. Resounding acoustic fields with reciprocity, 2025.
[46] Nhat Le, Tuong Do, Khoa Do. Hien Nguyen, Erman Tjiputra, Quang D Tran, and Anh Nguyen. Con- trollable group choreography using contrastive diffusion. ACM Transactions on Graphics (TOG), 42(6): 1-14, 2023.
[47] Buyu Li, Yongchi Zhao, Shi Zhelun, and Lu Sheng, Danceformer: Music conditioned 3d dance generation with parametric motion transformer. In Proceedings of the AAAI Conference Artificial Intelligence, pages 1272-1279, 2022.
[48] Ruilong Li, Shan Yang, David A Ross, and Angjoo Kanazawa. Ai choreographer: Music conditioned 3d dance generation with aist++. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 13401-13412, 2021.
[49] Ronghui Li. Junfan Zhao, Yachao Zhang, Mingyang Su, Zeping Ren, Han Zhang. Yansong Tang, and Xiu Li. Finedance: A fine-grained chorcography dataset for 3d full body dance generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10234-10243, 2023.
[50] Ronghui Li, YuXiang Zhang, Yachao Zhang, Hongwen Zhang, Jie Guo, Yan Zhang, Yebin Liu, and Xiu Li Lodge: A coarse to fine diffusion network for long dance generation guided by the characteristic dance primitives. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1524-1534 1534, 2024.
[51] Zhouyingcheng Liao, Mingyuan Zhang, Wenjia Wang, Lei Yang, and Taku Komura. Rmd: A simple baseline for more general human motion generation via training free -aggentl- motion diffuse. arXiv preprint arXiv:2412.044:
2024.
[52] Jing Lin, Ailing Zeng, Shunlin Lu, Yuanhao Cai, Ruimao Zhang, Haoqian Wang, and Lei Zhang, Motion- x: A large-scale 3d expressive whole-body human motion dataset. Advances in Neural Information Processing Systems, 36:25268-25288 2023.
[53] Jinpeng Liu, Wenxun Dai, Chunyu Wang, Yiji Cheng, Yansong Tang, and Xin Tong. Plan, posture and go: Towards open-world text-to-motion generation. arXiv preprint arXiv:2312. 148288 2023.
[54] Xian Liu, Qianyi Wu, Hang Zhou, Yinghao Xu, Rui Qian, Xinyi Lin, Xiaowei Zhou, Wayne Wu, Bo Dai, and Bolei Zhou. Learning hierarchical cross-modal association for eo-speech gesture generation. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 10462- 10472, 2022.
[55] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. 1711v:1711.05101, 2017.
arXiv preprint
[56] Vicon Motion Systems Ltd. Vicon motion capture systems, 2025. Accessed: Feb 2025.
[57] Shunlin La, I Ling-Hao Chen, Ailing Zeng, Jing Lin, Ruimao Zhang, Lei Zhang, and Heung- Yeung Shum. Humantomato: Text-aligned wholc-body motion generation. arXiv preprint arXiv:23 /2978, 2023.
[58] Shunlin Lu, Jingbo Wang, Zeyu Lu, Ling-Hao Chen, Wenxun Dai, Junting Dong, Zhiyang Dou, Bo Dai, and Ruimao Zhang. Scamo: Exploring the scaling law in autoregressive motion generation model. arXiv preprint arXiv:2412. 241.1449..,459, 2024.
[59] Zhengyi Luo, Jinkun Cao, Kris Kitani, Weipeng Xu, et al. Perpetual humanoid control for real-time simulated avatars. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2023.
13

---

[60] Zhenye Luo, Min Ren, Xuecai Hu, Yongzhen Huang, and Li Yao. Popdg: Popular 3d dance generation with popdanceset. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26984-26993, 2024.
[61] Brian McFee, Colin Raffel, Dawen Liang, Daniel PW Ellis, Matt Vicar, Eric Battenberg, and Oriol Nieto. librosa: Audio and music signal analysis in python. SciPy, 2015:18-24, 2015.
[62] Liang Pan, Zeshi Yang, Zhiyang Dou, Wenjia Wang. Buzhen Huang, Bo Dai, Taku Komura, and Jingho Wang. Tokenhsi: Unified synthesis of physical human-scene interactions through task tokenization. arXiv preprint arXiv:2503. 19901, 2025.
[63] Georgios Pavlakos, Vasileios Choutas, Nima Ghorbani, Timo Bolkart, Ahmed AA Osman, Dimitrios Tzionas, and Michacl Black. Expressive body capture: 3d hands, face, and body from single image. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10975-10985, 2019. 2019.
[64] Huaijin Pi, Zhi Cen, Zhiyang Dou, and Taku Komura. Coda: Coordinated diffusion noise optimization for whole- manipulation of articulated objects. arXiv preprint urXiv:2505.21433 2025.
[65] Shenhan Qian, Zhi Tu, Yihao Zhi, Wen Liu, and Shenghua Gao. Speech drives templates: Co-speech gesture synthesis with learned templates. In Proceedings of the IEFF/CVF International Conference on Computer Vision, pages 11077-11066, 2021.
[66] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol. Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.061 25 1())3, 2022.
[67]
Christian Schörkhuber and Anssi Klapuri. Constant-q transform toolbox for music processing. 7th sound and music computing conference, Barcelona, Spain, pages 3-64. SMC, 2010.
[68] Mingyi Shi, Dafei Qin, Leo Ho. Zhouyingcheng Liao, Yinghao Huang. Junichi Yamagishi, and Taku Komura. It takes two: Real-time co-speech two-person's 's interaction generation via reactive auto-regressive diffusion model. arXiv preprint arXiv:2412. 02229999999 2024.
[69] Yi Shi, Jingho Wang, Xuckun Jiang, Bingkun Lin, Bo Dai, and Xue Bin Peng. Interactive character control with ruto-regressive motion diffusion models. ACM Transactions on Graphics (TOG), 43(4); 1-14, 2024.
[70] Siyao, Weijiang Yu, Tianpei Gu, Chunze Lin, Quan Wang, Chen Qian, Chen Change Loy, and Ziwei Liu. Bailando: 3d dance generation by actor-critic gpt with choreographic memory. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11050-11059, 11059, 2022.
[71] Siyao, Tianpei Gu, Zhitao Yang, Zhengyu Lin, Ziwei Liu, Henghui Ding, Lei Yang, and Chen Change Loy. Duolando: Follower gpt with off-policy reinforcement learning for dance accompaniment. arXiv preprint arXiv:2403. 18811, 2024.
[72] Sebastian Starke, Ian Mason, and Taku Komura. Deepphase: Periodic autoencoders for learning motion phase manifolds. ACM Transactions on Graphics (TOG), 41(4):1-13, 2022.
[73] Jiarui Sun and Girish Chowdhary. Comusion: Towards consistent stochastic human motion prediction via motion diffusion. In European Conference on Computer Vision, pages 18-36. Springer, 2024.
[74] Peiwen Sun, Sitong Cheng, Xiangtai Li, Zhen Ye, Huadai Liu, Honggang Zhang, Wei Xue, and Yike Guo. Both ears wide open: Towards language-drinn driven spatial audio generation. arXiv preprint arXiv:2410.10677 2024.
[75] Taoran Tang, Jia Jia, and Hanyang Mao. Dance with melody: An lstm-autoencoder approach to music- oriented dance synthesis. In Proceedings of the 26th ACM international conference on Multimedia, pages 1598-1606, 2018
[76] Chen Tessler, Yunrong Guo, Ofir Nabati, Gal Chechik, and Xue Bin Peng. Maskedmimic: Unified physics-based character control through masked motion inpainting. ACM Transactions on Graphics (TOG), 43(6): 2211221,222 2024.
[77] Guy Tevet, Sigal Raah, Brian Gordon, Yoni Shafir, Daniel Cohen-or, and Amit Haim Bermano. Human motion diffusion model. In The Eleventh International Conference on Learning Representations, 2023.
[78] Jonathan Tseng, Rodrigo Castellon, and Karen Liu. Edge: Editable dance generation from music. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 448-458, 2023.
[79] Weilin Wan, Zhiyang Dou, Taku Komura, Wenping Wang, Dinesh Jayaraman, and Lingjie Liu. Ticontrol: Trajectory and language control for human motion synthesis. arXiv preprint arXiv:2311. 1.7135,,, 2023.
[80] Jingbo Wang, Sijie Yan, Bo Dai, and Dahua Lin. Scene-aware generative network for human motion synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2021.
14

---

[81] Jingbo Wang, Yu Rong, Jingyuan Liu, Sijie Yan, Dahua Lin, and Bo Dai. Towards diverse and natural scene-aware 3d human motion synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20460-20469, 2022.
[82] Wenjia Wang. Liang Pan, Zhiyang Dou, Jidong Mei, Zhouyingcheng Liao, Yuke Lou, Yifan Wu, Lei Yang, Jingbo Wang, and Taku Komura. Sims: Simulating stylized human-scene interactions with retrieval- augmeentd script generation. arXiv preprint arXiv:2411. 2024.
[83] Zan Wang, Yixin Chen, Baoxiong Jia, Puhao Li, Jinlu Zhang, Jingze Zhang, Tengyu Liu, Yixin Zhu, Wei Liang, and Siyuan Huang. Move you say interact as you can: Language guided human motion generation with scene affordance. In Proceedings of the IEEÉ/CVF Conference on Computer Vision and Pattern Recognition, pages 433-444, 2024.
[84] Yiming Xie, Varun Jampani, Lei Zhong, Deqing Sun, and Huaizu Jiang. Omnicontrol: Control any joint at any time for human motion generation. arXiv preprint arXiv:2310.0850 2023.
[85] Liang Xu, Ziyang Song, Dongliang Wang, Jing Su, Zhicheng Fang, Chenjing Ding, Weihao Gan, Yichao Yan, Xin Jin, Xiaokang Yang. et al. Actformer: A gan-based transformer towards general action- conditioned 3d human motion generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2228 2238,, 2023.
[86] Sirui Xu, Hung Yu Ling. Yu-Xiong Wang. and Liang- Yan Gui. Intermimic: Towards universal whole-body control for physics-based ooject- interactions. In Proceedings of the Computer Vision and Puttern Recognition Conference, pages 12266-12277 2025.
[87] Xudong Xu, Dejan Markovic. Jacob Sandakly, Todd Keebler, Steven Kremn, and Alexander Richard. Sounding bodies: modeling 3d spatial sound of humans using body pose and audio. Advances in Neural Information Processing Systems, 36:44740-4444 040000-4452244 2023.
[88] Han Yang. Kun Su, Yutong Zhang, Jiaben Chen, Kaizhi Qian, Gaowen Liu, and Chuang Gan. Unimumo: Unified text, music and motion generation. arXiv preprint arXiv:2410.04534, 2024.
[89] Youngwoo Yoon, Bok Cha, Joo- Hacng Lec, Minsu Jang, Jacyeon Lee, Jachong Kim, and Geehyuk Lec. Speech gesture generation from the trimodal context of text, audio, and speaker identity. ACM Transactions on Graphics (TOG), 39(6):1116 2020.
[90] Ye Yuan and Kris Kitani. Diverse trajectory forecasting with determinantal point processes. arXiv preprint arXiv: 1900:v90000444400 049677 2019.
[91] Canyu Zhang, Youbao Tang. Ning Zhang, Ruci-Sung Lin, Mei Han, Jing Xiao, and Song Wang. Bidirec- tional autoregessive diffusion model for dance generation. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 687-696, 2024.
[92] He Zhang, Sebastian Starke, Taku Komura, and Jun Saito. Mode-adaptive neural networks for quadruped motion control. ACM Transactions on Graphics (TOG), 1714):1-1,, 2018
[93] Jianrong Zhang, Yangsong Zhang, Xiaodong Cun, Yong Zhang. Hongwei Zhao, Hongtao Lu, Xi Shen, and Ying Shan. Generating human motion from textual descriptions with discrete representations. In Proceedings of IEEE/CVF conference on computer vision and pattern recognition, pages 14730-14740. 14740. 2023.
[94] Mingyuan Zhang. Daisheng Jin, Chenyang Gu Fangzhou Hong, Zhongang Cai, Jingfang Huang. Chongzhi Zhang, Xinying Guo, Lei Yang, Ying He, et al. Large motion model for unified molti-modal motion generation. In European Conference on Computer Vision, pages 397-421. Springer, 2024.
[95] Yufei Zhang, Jeffrey O Kephart, Zijun Cui, and Qiang Ji. Physpt: Physics- pretrained transformer for estimating human dynamies from monocular videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 2305-2317, 2024.
[96] Yufei Zhang, Jeffrey Kephart, and Qiang Ji. Incorporating physics principles for precise human motion prediction. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 6164-6174, 2024.
[97] Zeyi Zhang, Tenglong Ao, Yuyao Zhang, Qingzhe Gao, Chuan Lin, Baoquan Chen, and Libin Liu. Semantic gesticulator: Semantics-aware co-speech gesture synthesis. ACM Transactions on Graphics (TOG), 43(4): 2024.
[98] Hang Zhao, Chuang Gan, Andrew Rouditchenko, Carl Vondrick, Josh McDermott, and Antonio Torralba. The sound of pixels. In Proceedings of the European conference on computer vision (ECCV), pages 570- 570-586,77 2018
[99] Zhisheng Zheng, Puyuan Peng, Ziyang Ma. Xie Chen, Eunsol Choi, and David Harwath, Bat: Learning reason about spatial sounds with large language models, arXiv preprint arXi::2002.015),, 2024.
15

---

[100] Wenyang Zhou, Zhiyang Dou, Zeyu Cao. Zhouyingcheng Liao, Jingbo Wang, Wenjia Wang, Yuan Liu, Taku Komura, Wenping Wang, and Lingjie Liu. Emdm: Efficient motion diffusion model for fast and high-quality motion generation. In European Conference on Computer Vision, pages 18-38. Springer, 2025.
[101] Yi Zhou, Connelly Barnes, Jingwan Lu, Jimei Yang, and Hao Li. On the continuity of rotation representa- tions neural networks. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5745-5753, 2019.
[102] Lingting Zhu, Xian Liu, Xuanyu Liu, Rui Qian, Ziwei Liu, and Lequan Yu. Tamingg diffusion models for audio-driven co-speech gesture generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10544-10553, 2023.
[103] Wentao Zhu, Xiaoxuan Ma, Dongwoo Ro, Hai Ci, Jinlu Zhang, Jiaxin Shi, Feng Gao, Qi Tian, and Yizhou Wang. Human motion generation: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
[104] Wenlin Zhuang, Congyi Wang. Jinxiang Chai. Yangang Wang, Ming Shao, and Siyu Xia. Music2dance: Dancenet for music-driven dance generation. ACM Transactions Maltimedia Computing, Communica- tions, and Applications (TOMM), 18(2):1-21, 2022.
16

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Claimed in abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Stated Sec. 6
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [No]
17

---

Justification: [NA]
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Stated in Sec. 5
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
18

---

Answer: [Yes]
Justification:
The dataset and the codes will be released after acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.c cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: /CodeSubmic / //nips. wttpccccpppccccceeececccccccccct /tti:///ggiiieeeeeeeuuubiiiiii ssionnollll onPolicy))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
Stated Sec. 5
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Stated in Sec. 5
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
19

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Stated in Sec. 5, with one GPU (Nvidia RTX 4090 GPU).
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that NeurIPS Code of Ethics https: : /ttp///eersp//ee/ee//ee//e cc httpppbllcctthiccctteeeeeee...ee
didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the
Answer: [Yes]
Justification: Yes.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Stated in Sec. 6.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
20

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We follow the original license of previous work.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's is creators.
13. New assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
21

---

Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [Yes]
Justification: Stated Sec. 5.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [Yes]
Justification: Research with human subjects has been known and approved by the authors' university and supervisor.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMSs if it is an important, original, or non-standard component of the methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality the research, declaration is required.
Answer: [Yes]
22

---

Justification:
Editing (e.g., grammar, spelling, word choice)
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to LLM policy (https: / beeeur/ps.cc//oofferrteeee////// for what should or should not be described.
23


Reviews:
[{"review_id": "DWQ3i1h3jr", "reviewer": "Reviewer_peZR", "summary": "This paper presents a new task-human motion generation conditioned on spatial audio, along with an 8-hour mocap dataset (SAM) and a spatial audio conditioned motion generation model (MOSPA). Extensive experiments demonstrate the proposed MOSPA can achieve better performance compared to the baseline methods. Qualitative results show responsive motion driven by spatial audio.", "questions": "1. In Abstract #line17, what is the “privacy-constrained HMR task”? I did not find any reference in the paper. \n2. What is the motivation for including the sound source location as an explicit input condition? From the basic principles of spatial audio, one would expect a model to implicitly infer the source location from spatial audio features. Given that explicit sound source locations are typically unavailable in real-world scenarios, why is this input still deemed necessary? How do the authors interpret the effect of the sound source location on the quality of the generated motion? For example, does the character fail to orient correctly (e.g., look in the wrong direction) without this explicit input?", "limitations": "Yes", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths\n1. Human motion generation driven by spatial audio is an interesting and novel direction, with strong potential for real-world applications. A high-quality, mocap-based paired dataset is a crucial foundation for advancing this line of research. The proposed SAM dataset is collected using a professional motion capture system, with well-documented collection procedures and comprehensive dataset statistics. This dataset represents a significant and valuable contribution to the field.\n2. The model design is very clear and achieve good performance, which set a good benchmark for future research.\n3. The paper presents comprehensive experiments that effectively demonstrate the proposed method and design choices. The demos on out-of-distribution (OOD) spatial audio and simulated character control are also commendable and add practical value to the work.\n\n## Weaknesses\n1. The generated motion exhibits quite a few foot-sliding artifacts, which is worse than motion generation results in other tasks, e.g. music2dance, text2motion.\n2. The methodological contribution appears relatively minor, as the core innovation lies in the spatial audio feature extraction module. It would be worthwhile to include ablation studies on the spatial audio features to better understand their impact and validate the effectiveness of different features.", "was_revised": true, "final_justification": "All of my concerns have been well addressed. I believe this work makes valuable contribution to the human motion and animation community, and should be presented in NeurIPS.", "rebuttal": "We sincerely appreciate your valuable feedback and your recognition of our work's contributions, including the introduction of a high-quality mocap-based paired dataset for spatial audio conditioned motion synthesis tasks, a clearly designed model which achieves good performance, and the comprehensive experiments we have done on the model and the dataset. We have carefully considered all the comments and weaknesses you have pointed out. Below, we address each point systematically:\n\n\n### **Foot-sliding artifacts**  \n\nWe acknowledge that foot-sliding artifacts are a well-known issue across various (kinematics-based) motion synthesis frameworks, including both AR-based approaches [8, 4] and Diffusion-based models [6, 1, 10]. We fully agree that improving physical plausibility is a valuable and promising direction for future research, especially given its significance for perceptual realism—as highlighted in prior works [7, 9, 2]. \n\nThat said, our current work prioritizes addressing what we believe are more immediate challenges in spatial-audio-driven motion generation: namely, limited training data, the alignment between spatial audio and generated motion, and overall motion quality. While physical realism remains an important goal, we consider it part of a broader research trajectory beyond the current scope:) As outlined in our Future Work section, we intend to incorporate physics-informed constraints and contact-aware refinement strategies to further enhance the physical realism of generated motions.\n\n### **Audio feature extraction**  \nThank you for your insightful comments. Although music-like audio and general non-musical audio exhibit various differences, certain common audio features—such as MFCCs and tempograms—still represent fundamental audio characteristics across domains. Consequently, inspired by prior music-to-dance research [5, 11, 3], we utilize many common audio features. Building upon these foundations, we augment the feature set to incorporate spatial information: specifically, we concatenate channel-specific features for the left and right ears to further emphasize spatial information and introduce RMS energy to implicitly represent relative audio distance. See detailed results below:\n\n| MFCC | Tempogram | FID↓ | R-1↑ | R-2↑ | R-3↑ |\n|------|-----------|------|------|------|------|\n| ✔    | ✔        | $7.981$ | $0.937^{±0.005}$ | $0.984^{±0.002}$ | $0.996^{±0.001}$ |\n| ✘    | ✔        | $9.070$ | $0.907^{±0.004}$ | $0.967^{±0.002}$ | $0.983^{±0.002}$ |\n| ✔    | ✘        | $10.786$ | $0.917^{±0.004}$ | $0.982^{±0.002}$ | $0.994^{±0.001}$ |\n\n\n### **Q: What is the motivation for including the sound source location as an explicit input condition?**  \nThank you for your insightful comments. The explicit Sound Source Location (SSL) input was originally introduced to support the character animation system by providing informative guidance for motion trajectory synthesis, particularly under acoustically challenging conditions such as strong reverberation. Moreover, incorporating SSL is both practical and justified in virtual animation and simulation environments, where such information is typically accessible and reliable.\nNevertheless, our dataset naturally accommodates scenarios without SSL input, enabling the model to function effectively under both SSL-present and SSL-absent conditions - a promising direction for embodied AI and robotics research. \n\nRegarding concerns about potential over-reliance on SSL: We conducted rigorous experiments on out-of-distribution (OOD) samples (see Fig. 8 and Sec. 5.1 of the paper). Results demonstrate that the model maintains reasonable motion generation capability using audio features outside the SAM dataset. \n\nAdditional validation includes:  \n- Training/evaluation with fully masked SSL, yielding motions generally well-aligned with ground truth  \n- Directional loss implementation (measuring angular deviation between predicted/ground-truth motion vectors), which partially addresses orientation errors when the SSL is not feasible  \n\nWe will add more experiments and discussion to the revision.  \n\n### **Writing**\nWe are sorry for the misunderstanding caused by \"privacy-constrained\". We will fix this in our revision.\n\n### **References**  \n[1] Xin Chen, Biao Jiang, Wen Liu, Zilong Huang, Bin Fu, Tao Chen, and Gang Yu. Executing your commands via motion diffusion in latent space. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 18000–18010, 2023.  \n[2] Yifeng Jiang, Jungdam Won, Yuting Ye, and C Karen Liu. Drop: Dynamics responses from human motion prior and projective dynamics. In SIGGRAPH Asia 2023 Conference Papers, pages 1–11, 2023.  \n[3] Ruilong Li, Shan Yang, David A Ross, and Angjoo Kanazawa. Ai choreographer: Music conditioned 3d dance generation with aist++. In Proceedings of the IEEE/CVF international conference on computer vision, pages 13401–13412, 2021.  \n[4] Shunlin Lu, Jingbo Wang, Zeyu Lu, Ling-Hao Chen, Wenxun Dai, Junting Dong, Zhiyang Dou, Bo Dai, and Ruimao Zhang. Scamo: Exploring the scaling law in autoregressive motion generation model. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 27872–27882, 2025.  \n[5] Li Siyao, Weijiang Yu, Tianpei Gu, Chunze Lin, Quan Wang, Chen Qian, Chen Change Loy, and Ziwei Liu. Bailando: 3d dance generation by actor-critic gpt with choreographic memory. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11050–11059, 2022.  \n[6] Guy Tevet, Sigal Raab, Brian Gordon, Yonatan Shafir, Daniel Cohen-Or, and Amit H Bermano. Human motion diffusion model. arXiv preprint arXiv:2209.14916, 2022.  \n[7] Ye Yuan, Jiaming Song, Umar Iqbal, Arash Vahdat, and Jan Kautz. Physdiff: Physics-guided human motion diffusion model. In Proceedings of the IEEE/CVF international conference on computer vision, pages 16010–16021, 2023.  \n[8] Jianrong Zhang, Yangsong Zhang, Xiaodong Cun, Yong Zhang, Hongwei Zhao, Hongtao Lu, Xi Shen, and Ying Shan. Generating human motion from textual descriptions with discrete representations. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 14730–14740, 2023.  \n[9] Yufei Zhang, Jeffrey O Kephart, Zijun Cui, and Qiang Ji. Physpt: Physics-aware pretrained transformer for estimating human dynamics from monocular videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2305–2317, 2024.  \n[10] Wenyang Zhou, Zhiyang Dou, Zeyu Cao, Zhouyingcheng Liao, Jingbo Wang, Wenjia Wang, Yuan Liu, Taku Komura, Wenping Wang, and Lingjie Liu. Emdm: Efficient motion diffusion model for fast and high-quality motion generation. In European Conference on Computer Vision, pages 18–38. Springer, 2024.  \n[11] Wenlin Zhuang, Congyi Wang, Jinxiang Chai, Yangang Wang, Ming Shao, and Siyu Xia. Music2dance: Dancenet for music-driven dance generation. ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM), 18(2):1–21, 2022."}, {"review_id": "tUI8cyD0PP", "reviewer": "Reviewer_wAmb", "summary": "This paper focus on the task of human motion generation driven by spatial audio, which aims to explore the relationship between spatial audio cues and human motion. The paper presents the Spatial Audio Motion (SAM) dataset, which contains annotations of sound source locations and different reaction intensities. The authors introduce the MOSPA, a diffusion-based model that achieves SOTA performance in generating realistic and responsive human motion.", "questions": "1. Given that the SAM dataset contains a limited set of 12 subjects and 27 spatial audio scenes, how do you account for potential biases in motion responses?\n2. Have you performed ablation studies to understand the respective contributions of MFCCs, temporal maps, and RMS energy to the final representation?\n3. Have you tested MOSPA on other publicly available audio-to-motion benchmarks, even if they lack spatial annotations? \n4. How does the model perform with spatial audio inputs that differ significantly from the training distribution in SAM?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths: \n1. This paper aim to addresses the new task of spatial audio-driven human motion generation and proposes a framework to model the relationship between spatial audio cues and human motion. \n2. It propose a new SAM dataset, which offers above 8 hours of annotated motion and spatial audio data, enabling research in this area. The proposed diffusion-based model effectively captures audio-motion interplay and shows strong performance.\n\nWeaknesses: \n1. It only involve 12 participants and 27 audio scenarios may restrict diversity and generalizability. \n2. The model architecture appears relatively simple except the spatial audio feature extraction pipeline, and the absence of ablation studies makes it hard to assess the contributions of individual components. \n3. The lack of evaluation on external datasets raises concerns about robustness and real-world applicability.\n4. Its evaluation is limited to a single dataset, with limited sample size and scene coverage, and lacks ablation studies on the proposed diffusion model. There are concerns about its generalization and robustness due to the lack of testing outside of the SAM dataset. More extensive benchmarks and deeper analysis of the model would help strengthen this work.", "was_revised": true, "final_justification": "The issues I raised have all been addressed, so I am happy to raise my score to 5. I apologize for the late reply.", "rebuttal": "Thank you sincerely for your thorough and insightful comments. We are grateful to your recognition of the research direction, as well as the proposed framework and dataset. Regarding the weaknesses you pointed out and the questions you raised, we would like to provide the following explanations:\n\n### **Audio feature extraction**  \nThank you for your valuable suggestions. Despite inherent differences between musical and non-musical audio characteristics, certain features remain effective for general sound representation, like MFCCs and tempogram. Following established music-to-dance methodologies [2, 3, 4], we employ common audio features including MFCCs and tempograms. Building upon this foundation, we augment the feature set by concatenating binaural audio features and incorporating RMS energy as a distance indicator.  \nAblation studies confirm that models trained without MFCCs or tempograms exhibit significantly reduced motion accuracy compared to ground truth:  \n\n| MFCC | Tempogram | FID↓ | R-1↑ | R-2↑ | R-3↑ |\n|------|-----------|------|------|------|------|\n| ✔    | ✔        | $7.981$ | $0.937^{±0.005}$ | $0.984^{±0.002}$ | $0.996^{±0.001}$ |\n| ✘    | ✔        | $9.070$ | $0.907^{±0.004}$ | $0.967^{±0.002}$ | $0.983^{±0.002}$ |\n| ✔    | ✘        | $10.786$ | $0.917^{±0.004}$ | $0.982^{±0.002}$ | $0.994^{±0.001}$ |\n\n### **Lack of evaluation on external datasets raises concerns about robustness and real-world applicability?**  \nTo our knowledge, the SAM dataset is the *first* motion-capture-based audio-motion paired dataset with recorded spatial audio information, while uniquely encompassing both musical and non-musical audio. Existing datasets only focus on musical content and *lack spatial localization*, making them unsuitable for spatial audio-driven motion synthesis tasks.\n\n### **Ablation studies on the proposed diffusion model and testing outside of the SAM dataset**  \nThank you for your insightful feedback. As shown in Tab. 3 of the paper, we have conducted extensive ablation studies on the configuration of the proposed diffusion model MOSPA. We have also tested the model on out-of-distribution samples which are not in the SAM dataset: *\"Furthermore, we test MOSPA on out-of-distribution audio-source configurations. As shown in Fig. 8, it maintains motion quality and intent alignment, demonstrating robustness to unseen spatial setups.\"*. We will also include additional out-of-distribution (OOD) samples in our test set to better assess generalization.\n\n### **Restrict diversity and generalizability of the dataset?**  \nAs shown in Appendix Table A4, we respectfully argue that our 27 audio scenarios cover a wide range of real-life situations—including noisy and quiet environments, sudden transient sounds, and steady background noise. To further ensure diversity and generalizability, all 12 subjects performed different motions based on both the type of sound and its spatial relation to the subject’s initial position and orientation.\n\nCompared to existing music-to-dance datasets, the SAM dataset exhibits substantially greater diversity and scale, as summarized in Table 1 of the main paper. With 12 subjects and 33,600 seconds of motion-audio pairs, SAM significantly surpasses many comparable datasets—for instance, Music2Dance [4] includes only 2 subjects, and AIST++ [2] provides 18,694 seconds of paired data. This volume supports a more comprehensive representation of varied motion behaviors.\n\nWe will continuously expand the dataset in both diversity and scale to support broader research and application needs.\n\n### **References** \n[1] Chuan Guo, Shihao Zou, Xinxin Zuo, Sen Wang, Wei Ji, Xingyu Li, and Li Cheng. Generating diverse and natural 3d human motions from text. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5152–5161, 2022.  \n[2] Ruilong Li, Shan Yang, David A Ross, and Angjoo Kanazawa. Ai choreographer: Music conditioned 3d dance generation with aist++. In Proceedings of the IEEE/CVF international conference on computer vision, pages 13401–13412, 2021.  \n[3] Li Siyao, Weijiang Yu, Tianpei Gu, Chunze Lin, Quan Wang, Chen Qian, Chen Change Loy, and Ziwei Liu. Bailando: 3d dance generation by actor-critic gpt with choreographic memory. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11050–11059, 2022.  \n[4] Wenlin Zhuang, Congyi Wang, Jinxiang Chai, Yangang Wang, Ming Shao, and Siyu Xia. Music2dance: Dancenet for music-driven dance generation. ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM), 18(2):1–21, 2022."}, {"review_id": "C3pwKO5jG2", "reviewer": "Reviewer_njb5", "summary": "This paper proposes a new dataset, SAM, and a novel model, MOSPA, for human motion generation using spatial sound. The proposed dataset was recorded in an indoor setting and includes sound source locations, motion genres, binaural audio, and corresponding human motion data. The proposed method extracts acoustic features and, together with genre and source location information, utilizes a diffusion model to generate motion that corresponds to the audio signals. Experimental results on the SAM dataset demonstrate that the proposed approach produces more realistic and diverse motions compared to existing music-based baselines.", "questions": "(Major) Regarding the first weakness on robustness to recording environments: even if motion capture data collection is infeasible in diverse settings, qualitative results in environments with significantly different layouts, reverberation characteristics, or outdoor conditions could considerably impact the perceived effectiveness of the method and potentially alter the overall rating.\n\n(Minor) A more detailed discussion on which features or preprocessing steps led to the performance differences between the proposed method and EDGE would greatly strengthen the paper’s technical contribution.\n\n(Minor) While the paper focuses on spatial audio as the primary input, it would be valuable to discuss the potential for incorporating language information to enable more controllable motion generation.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- Compared to prior work on gesture generation aligned with speech or music, the proposed task of generating motion based on spatial audio offers a high degree of novelty.\n- The paper is well-written and very easy to follow.\n- The proposed dataset, SAM, provides valuable insights into how humans react to spatial audio and is likely to contribute meaningfully to the advancement of the field.\n- The experimental results demonstrate that the proposed method outperforms existing approaches both quantitatively and qualitatively, highlighting the effectiveness of the proposed acoustic features and the additional conditioning information such as genre.\n\nWeaknesses:\n- The data collection environment is limited to a single indoor setting. It is well known that directional acoustic features are highly dependent on room layout and reverberation characteristics [1, 2]. Since the proposed dataset is recorded in only one room, it remains unclear whether the proposed method would generalize well to different indoor environments or outdoor scenarios. This limitation is particularly relevant for scenarios such as “insect” or “firework,” where the input audio would naturally be expected to originate from outdoor environments.\n- The proposed model does not take language as input, which limits its controllability. Although the effectiveness of genre-based control is demonstrated, there is no mechanism for more fine-grained control over the content or style of the generated motion.\n- The paper lacks in-depth analysis regarding the performance gap between models. While the proposed method is relatively simple in structure (which is not a weakness in itself), the discussion around its performance compared to other diffusion-based methods such as EDGE, is insufficient. The authors only mention:\n> \"Their shortcomings in generated samples can primarily be attributed to model size and their strong focus on music-like audio.\"\n\nHowever, it would be desirable to include a more concrete analysis, such as which features in the proposed method were particularly effective for non-music sounds.\n\n[1] Oumi, Yusuke, et al. \"Acoustic-based 3D Human Pose Estimation Robust to Human Position.\" BMVC2204.\n\n[2] Wang, Mason, et al. \"Soundcam: A dataset for finding humans using room acoustics.\" NeurIPS2023.", "was_revised": true, "final_justification": "The authors appropriately demonstrated the importance of acoustic features by responding to my comments with supporting experimental results. While it is true that there are some issues, such as limited diversity in recording environments and missing language captions, I believe the dataset—combining spatial audio and human motion—offers a high degree of novelty and contributes meaningfully to the advancement of the field. For this reason, I have raised my score to Accept.", "rebuttal": "We sincerely appreciate the time and expertise dedicated to evaluating our work. We are pleased to see your recognition of the SAM dataset's novelty and the methodological rigor demonstrated in our experiments and paper. Next, we provide our comprehensive responses below:\n\n### **Robustness to recording environments**  \nThank you for your insightful feedback. We fully acknowledge the importance of evaluating the method’s robustness under diverse recording environments, such as varying room layouts, reverberation characteristics, or outdoor settings, which can influence acoustic perception as well as motion generation.\n\nCurrently, we focus on indoor environments to ensure controlled and consistent data quality. We carefully designed the data collection process to minimize confounding factors. For example, during recording, speakers were positioned directly in front of the actors to reduce acoustic ambiguity and mitigate potential reverberation effects. As reported in Section 5.1, our model demonstrates strong performance on both in-distribution and out-of-distribution (OOD) samples within these indoor settings.\n\nThat said, we agree that broader environmental variation is a valuable next step. In future work, we plan to extend our data collection and evaluation to include diverse spatial environments—including outdoor scenes and complex indoor layouts.\n\n### **In-depth analysis of the performance gap between methods**\n\nWe agree that a more detailed analysis would be valuable and will include it in our revision. For example, our spatio-temporal audio conditioning leverages both the sound’s location and temporal structure, which is particularly effective for handling non-music audio events such as sudden or ambient sounds.\n\n### **Which features in the proposed method were particularly effective for non-music sounds**  \nThank you for these pertinent suggestions. We conducted ablation studies on audio feature selection by training MOSPA under two modified conditions:  \n1. Exclusion of MFCC features.  \n2. Exclusion of tempogram features.  \n\nQuantitative results demonstrate that omitting either feature set significantly degrades motion synthesis quality and accuracy relative to ground truth. Specifically, tempogram has a larger contribution to the generation of precise motion sequences. These ablation studies confirm their essential role in our audio-to-motion translation framework.  \nWe will include the aforementioned experiments and discussion in the revision once available.  \n\n| MFCC | Tempogram | FID↓ | R-1↑ | R-2↑ | R-3↑ |\n|------|-----------|------|------|------|------|\n| ✔    | ✔        | $7.981$ | $0.937^{±0.005}$ | $0.984^{±0.002}$ | $0.996^{±0.001}$ |\n| ✘    | ✔        | $9.070$ | $0.907^{±0.004}$ | $0.967^{±0.002}$ | $0.983^{±0.002}$ |\n| ✔    | ✘        | $10.786$ | $0.917^{±0.004}$ | $0.982^{±0.002}$ | $0.994^{±0.001}$ |\n\n### **Potential for incorporating language information**  \nThanks for this insightful suggestion. Yes! Integrating language control with spatial audio input represents a highly promising multimodal research direction. While our current focus is on spatial audio-driven motion synthesis, we plan to explore multimodal integration in future work."}, {"review_id": "eBQDWJuqQz", "reviewer": "Reviewer_Jmh2", "summary": "The paper presents a human motion dataset comprised of spatial audio data, named the SAM dataset. The authors benchmark SAM using a diffusion-based, spatial audio-conditioned human motion generation framework (MOSPA) and demonstrate that MOSPA can be trained to generate human motions conditioned on spatial audio inputs, using the SAM dataset. The evaluation results show that MOSPA achieves good performance on the privacy-constrained HMR task.", "questions": "**Details of the SAM data.**\n\nAs SAM is comprised of recorded audio, the binaural cues may vary depending on the recording environment, which can be linked to the models’ performance. One might have questions about the following details: \n\n1. What are the coordinates (locations) and the directions of the loudspeaker's location in the room, and what is the rationale for placing it that way?\n\n2. Are the speakers/microphones omnidirectional? If not, what are their specifications?\n\n3. How does the equipment used in the recording differ from the dummy head microphone, e.g. [N4], that is commonly used for binaural recordings? Is this equipment common for ‘binaural’ recordings?\n\n4. What is shape of the room? How is the acoustics of the room (reverberant/anechoic?)\n\n**Minor Comments.**\n\n1. In L121, what does “16 randomly sampled relative locations” mean? Presumably that there were a total of 16 loudspeaker locations that were the source of the sound, from which the experiment randomly picked one to play the sound, and the audio clips in the SAM are paired with relative locations between the (stationary) loudspeakers from the (moving) human location?\n\n    - MOSPA estimates $\\mathbf{x}$ which consists of the global position ($\\mathbf{p}$). Would there be any reason for encoding the sound source position as a relative position? (Perhaps due to the data sparsity in the sound sources’ global positions?)\n\n1. Would it be possible to extract a person's position and the head orientation from the motion estimated by MOSPA?\n\n    - If so, and if the original (mono) source sound is present, it can be rendered into binaural using efficient renderers such as [N5, N6]. It would be interesting to hear how well the binaurally rendered MOSPA’s sound source (say, $\\tilde{\\mathbf{a}}$) matches with the input binaural audio ($\\mathbf{a}$). So that: $\\mathbf{a}\\stackrel{\\mathcal{G}}{\\longrightarrow}\\hat{\\mathbf{x}}_0 \\stackrel{\\text{[N5]}}{\\longrightarrow} \\tilde{\\mathbf{a}}$ and then compare the similarity between the two binaural audios, e.g., $d(\\tilde{\\mathbf{a}}, \\mathbf{a})$.\n\n****\n[N4] Gardner, B., & Martin, K. (1994, May). *HRFT Measurements of a KEMAR Dummy-head Microphone*.\n\n[N5] Lee, J. W., & Lee, K. (2023, June). Neural fourier shift for binaural speech rendering. In *ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)* (pp. 1-5). IEEE.\n\n[N6] Roman, I. R., Ick, C., Ding, S., Roman, A. S., McFee, B., & Bello, J. P. (2024, April). Spatial scaper: a library to simulate and augment soundscapes for sound event localization and detection in realistic rooms. In *ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)* (pp. 1221-1225). IEEE.", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "**Strengths.**\n1. To the best of the reviewer's knowledge, this paper is the first dataset for modeling the interaction between spatial information in audio and human motion.\n\n2. The authors experimentally showed that motion synthesis models can be trained effectively on the proposed dataset, and by reporting a variety of metrics, they established a benchmark for the proposed task.\n\n**Weaknesses.**\n\nWhile the task of synthesizing human motion from audio is being discussed, the audio-related details have not been sufficiently elaborated.\n\n- On the SAM dataset\n\n    1. Please elaborate on the audio data processing configurations (e.g., sampling rate, STFT window/hop size, etc.)\n\n    2. From the picture of the measurement environment in Figure 3, it seems that the sound was played through (stationary) speakers in the room and recorded from a microphone placed in the subject's ear. As the SAM data is comprised of the mic measurements, it seems to need a more specific description of the recording environment.\n\n        - Please find more details in the Question section.\n\n- On the MOSPA model\n\n    1. What would be the rationale behind choosing an audio feature extraction as shown in Table B9? There is no mention of the selection criteria for the selected audio features, and no mention of features that are commonly used in the audio domain.\n\n        - The task that MOSPA tackles seems to be closely related to the Sound Event Localization and Detection (SELD) problem, and many SELD studies primarily use log-mel spectrograms (for per-frequency energy) and generalized cross-correlations (for phase information) as audio features [N1, N2, N3].\n\n        - ITD (Interaural Time Difference) and the ILD (Interaural Level Difference) are the most representative features for binaural audio cues, which are not mentioned in the text.\n\n    2. It is unclear, at this moment, what the variable $\\mathbf{s}$ means by reading the text “$\\mathbf{s}$ is the sound source” in L181. However, assuming that it is meant to imply the (relative) location of the sound source, the question remains why this input is required.\n\n        - Isn't it already possible to predict the source location from binaural recordings?\n\n        - Doesn't needing to know the exact (relative) position of the sound source to synthesize motion limit its applicability?\n\n        - Regarding the results in Table 3, the authors mention that entering location information improves FID and APD performance despite lowering other performance, so are these the two most important metrics?\n\n****\n\n[N1] Adavanne, S., Politis, A., Nikunen, J., & Virtanen, T. (2018). Sound event localization and detection of overlapping sources using convolutional recurrent neural networks. *IEEE Journal of Selected Topics in Signal Processing*, *13*(1), 34-48.\n\n[N2] Wilkins, J., Fuentes, M., Bondi, L., Ghaffarzadegan, S., Abavisani, A., & Bello, J. P. TWO VS. FOUR-CHANNEL SOUND EVENT LOCALIZATION AND DETECTION.\n\n[N3] Shimada, K., Simon, C., Shibuya, T., Takahashi, S., & Mitsufuji, Y. (2024). SAVGBench: Benchmarking Spatially Aligned Audio-Video Generation. *arXiv preprint arXiv:2412.13462*.", "was_revised": true, "final_justification": "The authors resolved all of my concerns, and I maintain a score of 5.", "rebuttal": "We sincerely thank you for your insightful feedback and for recognizing both our proposed SAM dataset and the accompanying benchmark experiments. Please find our detailed responses to your questions and concerns below.\n\n### **Audio data processing configurations**  \nThank you for your careful reading. We used a sampling rate of 30,720 Hz, an FFT window length of 2048, and a hop length of 256. The Short-Time Fourier Transform (STFT) windowing function is the Hann window [1]. We will add more details in the revision.\n\n\n### **Audio feature extraction and explicit input of exact sound source location**  \nWe sincerely appreciate your detailed feedback. Our primary focus is spatial audio-conditioned motion synthesis, for which we leverage audio representations commonly used in music-to-dance pipelines [3, 4, 5], including MFCCs and tempograms. We agree with you that SELD-related features such as log-mel spectrograms, ITD/ILD, and cross-correlations are promising alternatives [N1–N3]; we are currently exploring these options and will include results in future updates.\n\nThe explicit Sound Source Location (SSL) input was introduced to enhance the character animation system by providing directional cues for motion trajectory synthesis. Incorporating SSL is both practical and well-justified in virtual animation and simulation environments, where such spatial information is typically accessible and reliable. To address concerns about potential over-reliance on SSL, we conducted rigorous evaluations on out-of-distribution (OOD) audio samples (see Fig. 8 and Sec. 5.1). The results demonstrate that our model retains strong motion generation capabilities even when tested on audio inputs outside the SAM dataset. Additionally, we are currently conducting further experiments to generate motion directly from binaural inputs without explicit SSL supervision, and we will include these results in the revised version.\n\nFor motion evaluation, people typically prioritize FID and R-precision to assess quality and audio-motion alignment, while Diversity and APD metrics quantify distributional consistency with ground truth. As stated in the paper: “R-precision, FID, Diversity, Multimodality. These four metrics are calculated using the same setup proposed by [2]. Two bidirectional GRU are trained with a latent size of 1024 for 1000 epochs to extract the audio features and the corresponding motion features.”  \nWe will expand the feature extractor architecture description in our revision.\n\n### **Capture configurations**  \n\nThank you for your comments. To ensure data diversity in terms of distance and orientation, four speakers were positioned at the corners of the mocap area, each facing toward the center to minimize reverberation. As noted in the paper, motion capture sessions were conducted in an acoustically controlled empty room (5 m width × 10 m length × 3 m height). For each session, actors began from 16 randomly sampled initial positions and orientations. Because spatialized audio at the actor's location is essential, we placed microphones near the actors’ ears instead of using dummy head setups. In future work, we plan to further expand the dataset in both scale and diversity.\n\n### **References**  \n[1] RB Blackman and JW Tukey. The measurement of power spectra dover publications. Inc, New York, 1958.  \n[2] Chuan Guo, Shihao Zou, Xinxin Zuo, Sen Wang, Wei Ji, Xingyu Li, and Li Cheng. Generating diverse and natural 3d human motions from text. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5152–5161, 2022.  \n[3] Ruilong Li, Shan Yang, David A Ross, and Angjoo Kanazawa. Ai choreographer: Music conditioned 3d dance generation with aist++. In Proceedings of the IEEE/CVF international conference on computer vision, pages 13401–13412, 2021.  \n[4] Li Siyao, Weijiang Yu, Tianpei Gu, Chunze Lin, Quan Wang, Chen Qian, Chen Change Loy, and Ziwei Liu. Bailando: 3d dance generation by actor-critic gpt with choreographic memory. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11050–11059, 2022.  \n[5] Wenlin Zhuang, Congyi Wang, Jinxiang Chai, Yangang Wang, Ming Shao, and Siyu Xia. Music2dance: Dancenet for music-driven dance generation. ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM), 18(2):1–21, 2022."}]

Author rebuttal:

Meta-review: This paper introduces MOSPA, a diffusion-based framework for generating realistic human motions driven by spatial audio, addressing a gap in existing research that primarily focuses on non-spatial audio modalities like speech or music. The authors present the SAM dataset, the first comprehensive collection of spatial audio and corresponding human motions, featuring over 8 hours of data across diverse scenarios. MOSPA leverages spatial audio features such as MFCCs, tempograms, and RMS energy to model the relationship between audio signals and motion responses for high-quality synthesis. Comprehensive experiments validating MOSPA’s ability to generate diverse, realistic motions aligned with spatial audio inputs, including robustness tests on out-of-distribution samples.

This paper has several strengths. The task and dataset are pioneering, addressing an underexplored intersection of spatial audio perception and human motion synthesis. A new dataset SAM is introduced, which is meticulously collected with professional mocap systems, diverse audio scenarios, and clear documentation, filling a critical resource gap. Furthermore, MOSPA’s design is well-motivated, with strong empirical results demonstrating superior performance over baselines. However, the reviewer raised several concerns. The generated motions exhibit quite a few foot-sliding artifacts. The dataset is limited to 12 subjects and 27 audio scenarios in a single indoor environment, raising questions about generalizability to outdoor or acoustically varied settings. The diffusion framework itself is straightforward, lacking deeper architectural insights. The motivation for including explicit Sound Source Location (SSL) as an input condition needs clarification.

During the rebuttal phase, most concerns have been addressed by the authors. Reviewer peZR and Jmh2 questioned the over-reliance on SSL. The authors clarify that SSL aids trajectory synthesis, but OOD tests show robustness without SSL. Reviewer peZR questioned unrealistic foot motions. The authors acknowledged this as a known challenge, deferring physics-based refinements to future work. Reviewer wAmb raised concern about the dataset generalizability. The authors highlighted SAM’s scale vs. prior datasets, with plans to expand diversity.

After the rebuttal, all reviewers leaned toward acceptance. This paper stands out for its novelty and potential to catalyze research in audio-driven animation. With the introduction of a new task with real-world applications, it has the potential to inspire follow-up work in this field.
Decision: Accept (spotlight)