MoMu-Diffusion: On Learning Long-Term Motion-Music Synchronization and Correspondence
Authors: Fuming You
Minghui Fang
Li Tang
Rongjie Huang
Yongqi Wang
Zhou Zhao
Venue: NeurIPS 2024 poster (2024)
Keywords: motion-music generation
rhythmic alignment
diffusion models
Primary area: diffusion_based_models
TL;DR: We propose bidirectional contrastive rhythmic VAE and Transformer-based diffusion model for versatile motion-music generation tasks.

Abstract: Motion-to-music and music-to-motion have been studied separately, each attracting substantial research interest within their respective domains. The interaction between human motion and music is a reflection of advanced human intelligence, and establishing a unified relationship between them is particularly important. However, to date, there has been no work that considers them jointly to explore the modality alignment within. To bridge this gap, we propose a novel framework, termed MoMu-Diffusion, for long-term and synchronous motion-music generation. Firstly, to mitigate the huge computational costs raised by long sequences, we propose a novel Bidirectional Contrastive Rhythmic Variational Auto-Encoder (BiCoR-VAE) that extracts the modality-aligned latent representations for both motion and music inputs. Subsequently, leveraging the aligned latent spaces, we introduce a multi-modal diffusion Transformer model and a cross-guidance sampling strategy to enable various generation tasks, including cross-modal, multi-modal, and variable-length generation. Extensive experiments demonstrate that MoMu-Diffusion surpasses recent state-of-the-art methods both qualitatively and quantitatively, and can synthesize realistic, diverse, long-term, and beat-matched music or motion sequences. The generated motion-music samples are available at https://momu-diffusion.github.io/.

MoMu-Diffusion: On Learning Long-Term
Motion-Music Synchronization and Correspondence
Fuming You, Minghui Fang, Li Tang, Rongjie Huang, Yongqi Wang, Zhou Zhao"
Zhejiang University fumyoul3 gmail.com
Abstract
Motion-to-muse and music-to-motion have been studied separately, each attracting substantial research interest within their respective domains. The interaction between human motion and music is reflection of advanced human intelligence, and establishing unified relationship between them is particularly important. However, to date, there has been no work that considers them jointly to explore the modality alignment within. To bridge this gap, we propose novel framework, termed MoMu- MoMu-Diffusinn Diffusion, for long-term and synchronous motion-musie generation. Firstly, mitigate the huge computational costs raised by long sequences, propose a novel Bidirectional Contrastive Rhythmic Variational Auto-Encoder (BiCoR-VARE -AE) that extracts the modality-aligned latent representations for both motion and music inputs. Subsequently, leveraging the aligned latent spaces, we introduce multi-modal Transformer- based diffusion model and cross-guicee gross-guidance sampling strategy to enable various generation tasks, including cross-modal, multi- modal, and variable-lengee generation. Extensive experiments demonstrate that MoMu-Diffusion surpasses recent state-of-the-ard methods both qualitatively and quantitatively, and can synthesize realistic, diverse, long-term, and beat-matched music or motion sequences. The generated samples and codes are available at ptttps//mooee diffusiofuoooo www.ggiihh........... ioo/
Aligned Latent Space Long- Term Synthesis
opan oppe
Joint Generation
Sequence
MoMu-Diffusion
Noom
Video
Variable-1 ength Syn. Best-Matching
BiCoR-VAR
Mel-Spostiit Speotrogam
Figure 1: The pipeline of music through the novel Bidirectional Contrastive Rhythmic uuto-Encoder (BiCoR- --CC). Leveraging the aligned latent space, MoMu-Difa Diffusion facilitates both cross-mode and multi-moda generations.
MoMu-Diffusiee integrates the alignment of motion and
Introduction
Dancing to the musical beats creating variety of rhythmically synchronized music for given motion is a fundamental aspect of human creativity. Music and human motions serve as universal languages that
shared by all civilizations, transcending cultural and geographical boundaries around the world (25]. For computational methodologies, the motion-music generation poses several challenges: 1) maintaining long-term coherence typically lengthy motion- music sequences 2)
"Corresponing Author
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Table I: Comparison with the state-of-the-arr audio-visual generation works, including but not limited
to motion- music generation.
Method
Pub.
Joint Generation
Pretrain
Long-Temm Synthesis
Latent Space
Diff-Foley 23
MM-Diffusion
CVPR '3
x
LORIS
ICMI1'23
X
/
D2M
NeurlPS 19
CDCD
ICLR'23
MoMu-Diffusion
ensuring temporal synchronization and rhythmic alignment between motion and music sequences, and 3) generating realistic, diverse, and variable-length human motions or music.
Existing works usually divide the motion-music generation into two distinct tasks: motion-to- music and music-to-motion. For motion-to-music, some methods compress the conditional video frames into single image, in which the temporal information is lost [52] 53 The state-of-the- art work, LORIS 49), employs a hierarchical conditional diffusion model to generate long-term musical waveforms. However, LORIS introduces huge computational costs and training difficulties since it generates long-term musical waveforms directly. For music-to-motion the Dancing2Music (D2M) [26] framework divides the generation process into two stages: decomposing the dance into basic dancing movements with a VAE and compositing the basic movements into dance with a GAN, Nonetheless, D2M's approach of segmenting long-term music into short clips (approximately 1-2 seconds) diminishes the coherence of the synthesized motion sequences.
Motivated by the fact that human motions highly associated with music yet existing computational methods often study them in isolation, we propose a novel multi-modal framework, termed MoMu- Diffusion, to address the aforementioned challenges jointly. Firstly, to mitigate the computational costs and optimization complexities raised by long sequences, we employ a VAE to encode both motion and music sequences into latent spaces. Subsequently, to investigate the relationship between human movements and musical beats, we propose rhythmic contrastive learning. This approach involves constructing contrast pairs with a kinematic amplitude indicator, which quantifies the temporal variation in motion and is derived from the spatial motion directrogram differences as detailed in [4. Given that the motion and music sequences are interactively aligned in the latent space to discern the correlation between kinematic shifts and musical rhythmic beats, we call our model as the Bidirectional Contrastive Rhythmic VAE (BiCoR-VAE).
With the aligned latent space, we introduce a Transformer-basel diffusion model that captures long- term dependencies and facilitates sequence generation across variable lengths. Additionally, introduce a simple cross-guidance sampling strategy that integrates different cross-modal generation models, enabling multi-modal joint generation without extra training. By incorporating the BiCoR- VAE and the diffusion Transformer model, our MoMu-Diffusiin framework effectively models the long-term motion-music synchronization and correspondence, enabling potion-to-music music- to-motion, and joint motion-music samples in variable lengths. The pipeline of MoMu-Diffusion Diffusion is illustrated Figure
music generation. Moreover, MoMu- Diffusion supports generating
We have conducted extensive experiments on three motion-to-music and two music-to-motion datasets, including scenarios such dancing and competitive sports. The experimental results demonstrate that MoMu-Diffusion attains state-of-the-art performance across both objective and subjective metrics, significantly enhancing music/motion quality and cross-modal fhythmic/kinmmatic alignment. Furthermore, we have carried out abundant ablation studies to validate the efficacy of the BiCoR-VAL and the DiT architecture. comparative analysis with state-of-the-ar motion-to-
music methods CDCD (53) and LORIS 49 2D music-to-mon method D2M (26), and general video-to-an audio methods Diff-Foley
33 and MM-Diffusion 41 is presented in Table
2 Related Works
Neural Motion Synthesis. Neural motion synthesis is often associated with audio, and we focus on two audio-driven scenarios: music-to-motion generation [12 27 35 26 and co-speech gesture
generation 48 32 50 For music-to-motion, some methods 12 27 propose to retrieve the
most related music for the given motion sequence. D2M 26 generative model that designs

---

a cetpommposiiiottocoomppee
method to learn the movement units and generate music from 29 investigate synthesizing 3D motions from music. For co-speech gesture generation, DiffCiestrre 50 is a state-of-the-ar model with diffusion transformer architecture and diffusion gesture stabilizer. We study the 2D music-to-motion problem and compare the proposed MoMu-Diffusion with DiffGesture and D2M
the learned units. Besides, some methods 30
Neural Music Synthesis. Neural music Synthesis aims to generate melodious music with generative neural networks. Various generative models have been successfully applied to music synthesis
such as mansformer-bbsedd autoregressive models 21 39 VAE
2 40 6) GAN [8] 24 36], and
diffusion models [16 34]. Some efforts have been made to video- to-music which focuses on the
cross-modal temporal alignment. For example, Foley Music [13] and Audeo 4 utilize Musical Instrument Digital Interface (MIDI) representations to generate music in a non-regressive manner. D2M-GAN 52 and CDCD 53 generate video-related music by compressing the video frames into single image, in which the temporal information is neglected. LORIS [49 proposes a hierarchical conditional diffusion model to generate long-term musical waveforms.
Multi-Modal Contrastive Learning. Contrastive has been demonstrated effective in For example, Elizalde al. [0] proposed Contrastive Language-Audio Pretraining (CLAP) to learn unified latent representation for an audio or text input, facilitating the birth of text-to-audio models [31] 20).
For audio-visuee generative tasks, DiftFoley 33 uses semantic and temporal contrastive learning to promote video-to-audiy generation. In this paper. to improve the efficiency and generalization ability of our generative model, we propose the first motion-mus pretraining model with a well-designed contrastive loss to learn beat synchronization and rhythm correspondence.
3
Bidirectional Contrastive Rhythmic VAE (BiCoR-VAN VVAE)
3.1 Multi-Modality Model Architecture
Motion Variational Auto-Encoder. Let n E Jx2 be the 2D motion keypoints extracted from the corresponding video, where Tm, is the motion frames, J is the number of nodes containing the values of the u-coordinate and y-coordinate. Then, we encode the spatial positions into a latent by Zm mmn=EE(nu (m) E Im where T < T is the downsampled motion frames and d is the latent motion dimension. The encoded latent can be decoded by a decoder to obtain the reconstructed motion sequence: m' (m.)
Music Variational AutoEncoder: Music is a structured and complex audio signal, composed of various elements such as melody, harmony, rhythm, and dynamics. Some works 13
43 utilize Musical Instrument Digital Interface (MIDI) representations, which yield highly formulated results. However, processing long-term music directly from the raw waveform is computationally intensive and challenging [49]. To address this, we train a VAE on the mel-spectrogram derived from the music, coupled with high-fidelity vocoder. Let u E R be music input, where T, denotes the waveform length. We can extract the mel-spectrogram of the music input: Mel(u) E R.CxxT' where Mel() the pre-defined mel spectrooro extraction function, C. the channels, and T. < Tu is the frames. Then, an encoder is used to compress the mel-spectrogram into a latent: Zu = (a) E R xx. where T.a is the downsampled music frames and d the latent mel-spectrogram dimension. The encoded mel-sppectoggam can be decoded by decoder a' = Da (a), and subsequently the musical waveform can be obtained by a high-fidelity vocoder x' = Vaa)).
3.2 Rhythmic Contrastive Learning
Contrastive learning has proven effective for learning multi-modal representations, enhancing perfor- mance in downstream tasks (38). In the context of temporal alignment, a recent work (33) introduces temporal contrast, which seeks maximize the similarity of audio-visual pairs from the same time segment while minimizing the similarity of pairs from different segments. However, this paradigm faces limitations in long-term motion- music synthesis, as musical pieces typically correspond to numerous rhythmic beats. The random selection process for constructing negative samples risks capturing similar rhythmic sequences, which undermines the learning objective. To address it, we propose rhythmic contrastive learning, designed to align cross-modal temporal synchronization and rhythmic correspondence. Based on the motion and music VAFs, we can obtain the motion latent 2m TT.. xd and music mel-spectrogram latent 2a E I8 TgTT.xe xd respectively. To synchronize the
3

---

mirodtggra
NL
ShutBaurn
W
Pull Closer
Kinematic A Amplitude
Push Away
Best Guridunue
Motton
Zy
Motion
BiCoR Encuder
BiCoR Decoder
vocoder
Z
Spectugram
Spactngram
Audio
(a) Bidirectional Contrastive Rhythmic VAE (BiCoR-VAE)
Transformer
(0)22/200)
ZinCTe Pe. 11R.
Zn(7))
Zic
(7)727777))
2n
2,(T) PARER (1RA00
20(0)
(b) Cross- Modal Gemeration
[c] Multi- Modal Joint Generation
Figure 2: An overview of the proposed MoMu-Diffusion framework. MoMu-Diffusion contains two integral components: a bidirectional contrastive rhythmic Variational Autoencoder (BiCoR-VAl designed to learn the aligned latent space, and a Transformer-baser diffusion model responsible for sequence generation. This framework is adept at facilitating both cross-modal and multi-modal joint generations, offering a robust approach to the integrated synthesis of motion and music.
motion and music, which are often sampled differently, we employ pre-processing techniques such as evenly dropping motion frames to match the number of music frames, ensuring that T T.m Tza-
In the domain of motion- guided music, the inherent irregularity of human movements, characterized by rapid and abrupt actions, can significantly influence rhythm. To synchronize these rhythmic patterns, we employ a kinematic amplitude indicator as a basis for constructing contrastive clips within each motion-musie pair. Firstly, we extract the motion kinematic offsets [15) with the motion directogram 4 metric that quantifies the variation in motion. We denote Frr, j) as the first-order difference of j-th node in the 2D motion at temporal timestep r, and divide into K bins based on their Euclidean angles with x-axis by tan " (y/æx, æ). Then, the 2D motion directogram D(r, 0) can be expressed as the aggregate of Frr,j) across each angular bin:
D 0) = Σ (IF 1=1
f1, 0.
Φ1 S 2 K, otherwise.
where(g(0) =
(I)
The indicator function (Ø) distributes the motion nodes into K angular bins. Then, the kinematic amplitude indicator is computed by summing the bin-wise directrogram difference in each angular column:
K
Q(r) Σ m0 (r, A)] - 1 1,,))),
where D(r, k) is the directogram volume temporal timestep r and k-th bin. The kinematic amplitude value is normalized within the range of (0.1).
With the kinematic amplitude indicator established, we proceed prepare the temporal motion- music clips for contrastive rhythmic learning. For each motion-music latent pair, we randomly sample Nr motion musiin clips and divide them into Ne categories according to the clip-wise maximum kinematic amplitude values. In order to maximize the similarity of motion- music pairs from the same timestep (i.e. temporal alignment) and minimize the similarity motion- music pairs across different timesteps and rhythmic patterns, we randomly sample Ns motion-music latent clip (cfaite Ca Q(r, : rz)) (Rd, Rdd, (0, 1)) from different kinematic amplitude categories for the temporal and rhythmic alignment:
Jo ( : 2 c'aif== Pmaxx f 2, Qrrs re) = max (axx)Q(r.)))) Q(re)), (3)
where and denote the start and end timesteps of the sampled clip, respectively, and Praxx denotes the max-pooll max-pooling operation across the temporal dimension. Finally, based on the sampled motion-music clips {(c em Ne N.NG the contrastive objective can be formulated as:
2 exp(sim(c //T/ 1 exp(sim(c), Sap(sim(c..dd.// c/./T)
Lcontrase
log Ne cxp(sim )/T/ 2
log NNo exp(sim(cg, c/m)T)
(4)

---

3.3 Training Strategy
In BiCoR-VAR the goal is to learn two paired VAEs for motion and music inputs, with focus on temporal and rhythmic alignment within the low-level latent space. However, the VAE's objective to preserve fine-grained details for accurate reconstruction often conflicts with contrastive rhythmic learning's 's aim to align latent representations across modalities. This presents trade-off between representational fidelity and generative alignment, posing optimization challenges. To address we propose two-stage training strategy: initially, we train the music VAE using both a VAE loss and a GAN loss to prevent over-smoothing of the mee-sppctto subsequently, we train the motion
VAE with VAE loss and the contrastive rhythmic loss, while keeping the music VAE's parameters fixed. The insight behind this strategy is that mel-spectrogramms, with their rich and complex acoustic features, require more intricate optimization process compared to motion VAE, which deals with limited set of body joint data. An overview of EiCoR-VAE is illustrated in Figure (a).
4 Transformer-based Diffusion Model with Aligned BiCoR-VAE
Diffusion Formulation. Recent works have revealed that the U-Net architecture is not essential for diffusion probabilistic modeling, and in fact, the transformer can achieve superior performance in text-to- text-to-image generation tasks [37] [T. Additionally, the transformer architecture excels at capturing long-range dependencies within sequence data and offers flexibility for variable-length architecture for our motion-music generation framework. Concretely, our approach involves initially concatenating the noisy input with the embedded conditional inputs and the embedded diffusion timesteps along the temporal dimension. This fused input is then padded to match a specified maximum length and combined with positional embeddings prior to being processed by the DiT model. The DiT output is subsequently truncated to the original temporal length and mapped to the output latent space. To illustrate the diffusion process, let's consider the motion-to-musie task. During the forward diffusion, the latent data is gradually perturbed towards a standard Gaussian distribution according to a pre-defined schedule a1...., 0T, where is the total diffusion timesteps and a. = II Qy:
generation 45 Inspired by these findings, we opt for a Transformer-base
- 1)) (za Vaszalt (t 1), aaI), where (t) denotes the music latent at timestep 1. Then, the training objectives of our DiT-based cross-modal generation models are defined:
(5)
Lm2a = ||ee. (1), 2m) el1ă, Cazm - ||ea... (Zm (4), zn) - -elí.
(6)
where c E N0, 1) denotes the noise in diffusion procedure, Og and fm are the parameterized DiT denoisers for motion-to-muse and music-ta cosic-to-mott to-totion generation, respectively.
Conditional Generation. For the cross-modal generation such as motion-to-music and music-to- motion, we implement classifier-free guidance [5] [8] This method adeptly combines conditional
and unconditional scores to obtain a trade-off between quality and diversity. By interpreting the diffusion model output as a score function, the sampling procedure with classifier-fre guidance of motion-to to--i can be written as:
E0_ (1), L, zm. = c0, (, + s. (e0, (z)(1), L, zm) - co, (z, (1), 0))
(7)
where 8 denotes the classifier sampling scale balance the diversity and quality of synthesized samples. The diffusion model with 0 condition is achieved by randomly dropping ²m and replacing it with an embedded "null" representation. Exchanging the latent inputs enables the sampling procedure for music-to-motion generation since we have built a modality-aligned latent space.
Joint Generation with Cross Guidance. To accomplish multi-modal joint generation, we propose cross-guidanee sampling strategy. This approach leverages multiple 'expert' models and introduces a slight modification to the sampling procedure, rather than integrating multiple modalities into single model. Let T be the total diffusion steps, co. be the trained motion-to- to--isic denoising model, and co.. be the trained music-to- tottooon denoising model, perform unconditional generation before defined diffusion step To:
Po. (za(t - (t) NNzz 1o, (t). (),0), I.), where T > t> Te,
(8)
1 - Ot
1 1- 0t-1
no, (z, (2, (1),
(za(t)
c0 (2a (1),,,0)
a²
(1 at).
(9)
aas
VI Cr
1 02
5

---

Figure Motion-to-music with generation quality metries: FAD| and Diversityf.
Table 2: Motion-to-music with beat-matching metrics.
Subset Metries
AIST+ Dance
40
BCS1
CSD,,
BHS
HSD
FI
Foley
96.4
6.9
41.0
15.0
57.5
30-
Diversity_ _ORIS Diversity, Ours
CMT
97.1
6.4
46.2
18.6
62.6
D2MGAN
95.6
9.4
88.7
19.0
93.1
20
FAD_LORIS
CDCD
96.5
9.1
89.3
18.1
92.7
FAI Ours
FAD_,uurs
LORIS
98.6
90.8
13.9
94.5
10
Ours
97.5
5.2
98.6
2.8
98.1
DANCE
FE25
FESO
PS25
F550
Subset Metries
Floor Exercise-251
Floor Exercise-50s
BCSt
CSD
BHSt
HSD
FIf
BCS
CSD4
BHSt
HSD
F11
Foley
36.0
36.2
32.3
30.7
34.1
32.6
38.0
28.4
32.5
30.4
CMT
46.4
30.1
57.4
29.8
51.3
42.3
32.0
53.8
31.7
47.4
D2MGAN
45.3
27.7
58.7
30.1
51.1
41.9
29.2
54.7
32.7
47.5
CDCD
49.0
21.1
61.0
27.0
54.3
45.9
23.8
57.5
29.3
51.0
LORIS
58.8
19.4
67.1
21.1
62.7
54.7
21.6
63.8
24.5
58.9
Ours
66.6
14.3
76,9
19.1
71.4
62.7
24.0
68.1
20.2
65.3
Table 3: Results on the Floor Exercise dataset with beat-matching metrics.
Eq (8) and Eq (9) delineate the reverse process for motion-to-musie generation within the timestep range 7 2 <> T. The reverse process for music-to-motion generation can be similarly constructed. For reverse timesteps T. t 0, use the estimated clean motion/music latent to condition the generation process of music/motion with the classifier-free guidance defined in Eq (7). Given that the diffusion model adopts coarse-to-fine refinement in the reverse process, we conduct unconditional generation before Te and impose conditional generation with the cross-guidance strategy after Te as the noise in the estimated clean latent is significantly reduced. Determining the value of Te appears to be quite challenging; however, our empirical findings indicate that the joint generation maintains robust performance across broad range of values for Te (from 0.37' 0.7T). The diversity in joint generation is sustained by the unconditional process and classifier-free guidance. An overview of cross-modal generation and joint generation is shown in Figure (b) and (c).
Experiments
5.1 Motion-to-Music Generation
which contains 86.43 hours of video samples synchronized with music. This benehmark presents three demanding scenarios: AIST++ Dance (30), Floor Exercise 42]. and Figure Skating [47 46). In our experiments, each dataset randomly split with a 90%/5% proportion for training, validation, and testing. For model evaluation, we use five metrics to measure the beat-matching between synthesized music and ground-truth music [49] Beats Coverage Scores (BCS) and Beat Hit Scores (BHS). Coverage Standard Deviation (CSD), Hit Standard Deviation (CSD), and the Fl scores. Besides, use the Fréchet Audio Distance (FAD) [2 and Diversity 266 scores to evaluate the quality synthesized music. Since the quality of the Floor Exercise and the Figure Skating datasets are poor, we only conduct motion-to- to-iu generation on them with learnable motion encoder, whose architecture is derived from [51]. During sampling, employ 50 DDIM sampling steps. More experimental settings are provided in Appendix
Experimental Settings. We evaluate our method on the latest LORIS benchmark 449
46]
baselines: 1) Foley Music [13,, graph transformer framework with MIDI representations. CMT [7]. controllable music transformer model to learn the rhythmic consistency between video and mu- sic. D2M-GAN [52, a GAN-based model with vector quantized music representation. 4) certusion-basee model with an additional conditional discrete contrastive diffusion loss. 5) LORIS (49], a diffusion-bal model with hierarchical conditional mechanism, yielding state-of-the the--rttt performance on video-to-mon mu-m synthesis.
Baselines. We compare our proposed method to existing advanced
CDCD 53

---

Video Frames (Ref)
Musical Beats (Ours)
Musical Beats (GT)
Figure 4: Example of beat matching the motion-to-mon generation. The red dashes indicate the extracted musical beats. The red arrow points to the video frame at that particular moment.
Subset Metrics
Figure
Skating-255
Figure Skating-50k
BCSt
CSD
BHS1
HSD
FI
BCS
CSD
BHS
HSD
F11
Foley
36.0
36.2
32.3
30.7
34.1
32.6
38.0
28.4
32.5
30.4
CMT
46.4
30.1
57.4
29.8
51.3
42.3
32.0
53.8
31.7
47.4
D2MGAN
45.3
27.7
58.7
30.1
51.1
41.9
29.2
54.7
32.7
47.5
CDCD
49.0
21.1
61.0
27.0
54.3
45.9
23.8
57.5
29.3
51.0
LORIS
58.8
19.4
67.1
21.1
62.7
54.7
21.6
63.8
24.5
58.9
Ours
63.5
16.3
75.6
28.8
69.0
59.9
22.6
68.7
22.2
64.0
Table 4: Results on the Figure Skating with beat-matching metrics.
Main Results. The results of beat-matching are shown in Table and From these tables, we can draw the following conclusions: 1)
significantly surpasses existing state-of-the-art methods in cross-modal matthing It demonstrates the effectiveness of BiCoR -AR and the multi-modal Transformer-basee model in synchronizing kinematic and rhythmic beats. 2) MoMu- Diffusion realizes substantial improvement Beat Hit Scores (BHS), which indicates the beats in the synthesized music are closely aligned with the ground truth. For example, MoMu-Diffusinn gains 98.6 88.6% BHS on the AIST+ dancing subset, while previous methods usually gain about 90% BHS. An illustrative example of beat-matching for motion-to-mun can find the musical beats of synthesized music are aligned with the ground truth and the kinematic movements of the eference video.
is presented in Figure 4 We
The FAD and Diversity results are shown in Figure In this comparison, we focus on LORIS, the current state-of ofree-of-ii tore--oftteerrttt method in motion-te Pation-to-munn to-music generation. It is evident that consistently outperforms LORIS across these metrics, particularly in FAD scores. This superiority can be attributed to MoMu- 's architectural innovations for capturing long-term correspondence. Unlike text, music encompasses richer sequence length due its complex acoustic features, such employs mel-spectrograms
melody, rhythm, and driving beats. To address this, in place of raw waveforms, thereby mitigating sequence length. Additionally, the introduction of BiCoR-VAL facilitates modality alignment in latent spaces.
5.2 Music-to-Motion Generation
Experimental Settings. We use two datasets: AIST++ Dance 30 and BHS Dance. About 71 hours BHS Dance videos are collected from (26), which contains three dancing types: "Ballet", "Zumba", and "Hip-Hop", For model evaluation, we compute beat-matching metries between synthesized motion beats and the reference musical beats with the aforementioned five beat-matching metrics. To validate the quality of synthesized motion sequences, we use Fréchet Inception Distance (FID) [I7), Mean KL-Divergent
(Mean KLD), and the Diveristy scores. The feature extractor is based on MotionBert [5T] and trained with a classification task on the BHS Dance dataset. For the BHS Dance dataset, we exclude the BiCoR-VAR VAE since this dataset only contains the paired audio MFCC features and motion sequences without raw audio. In the generation process, the settings of the diffusion transformer model are the same motion-to-munn mo-music. More details provided in Appendix B
We compare MoMu-Diin MiMu--iffusion to two baselines: 1) D2M (6),, the state-of-the-art music-to- motion work with two-stage movement unit-based model; 2) DiffGesture (50), the state-of-the-art co-speech gesture generation work with U-Net diffusion model. Dance Revolution [19] reports better performance music-to-motion generation but is withdrawn by its authors.
Baselines.

---

Motion Frames (Ours)
Kinematic Beats (Ours)
Musical Beats (Ref)
Figure 5: Example of beat matching on the music-to-motion generation. The red dashes indicate the extracted kinematic beats of the synthesized motion. The red arrow points to the frame of the synthesized motion sequence at that particular moment.
Subset Metrics
+
Dance
BHS Dance
BCSt
CSD4
BHSt
HSD
F11
BCSt
CSD
BHS
HSDL
FIf
D2M
23.7
13.8
42.8
23.6
30.5
35.1
15.9
57.5
35.0
43.6
DiffGesture
28.5
16.7
40.4
25.7
33.4
42.8
21.3
61.1
23.9
50.3
Ours
39.2
10.2
56.3
12.0
46.2
47.9
8.4
78.5
12.1
59.5
Table 5: Results on the AIST+ Dance and BHS Dance datasets with beat-matching metrics.
Main Results. The beat-matchi matching results are detailed in Table 5 An analysis of these results reveals that MoMu-Diffusion achieves superior scores across all evaluated tasks, outperforning the state-of-the-arr music-to-motion method D2M and co-speech gesture generation method DiffGesture. This performance underscores the efficacy of our BiCoR- VAE in constructing an aligned latent space for cross-modal generation and the feed-forward diffusion model in capturing long-term correspondence. It should be noted that the metries BCS (Beats Coverage Scores) and BHS (Beat Hit Scores) are defined differently in this context compared motion-to- BCS calculates the coverage score between the kinematic beats of the synthesized motions and the musical beats of the ground-truth music, rather than the kinematic beats of the ground truth motions.
scenarios. Specifically,
The generation quality results are presented in T It is observable that better FID, Mean KLD, and Diversity scores both the AIST++ and BHS Dance datasets. demonstrates that MoMMu-Diffsuion can generate more realistic and high-quality motion sequences while maintaining the capability of diverse generations. We further present qualitative example of muuiccto--ottonn
reports
mate-mathhig in Figure 5 We can find the kinematic beats of synthesized motion are highly associated with the reference musical beats. Additionally, the generated dance exhibits high degree of diversity, encompassing lateral movements, rotations, squats, and so on.
Subset Metrics
AIST++ Dance
BHS Dance
FID
Diversityf
Mean KLD.
FID
Diversity?
Mean KLD1
D2M
17.3
46.2
14.5
11.6
55.9
7.4
DiffGesture
18.6
37.1
12.6
13.8
38.9
7.0
Ours
7.3
52.7
4.9
6.5
67.4
4.2
Table 6: Results on the AIST++ Dance and BHS Dance datasets with generation quality metrics.
5.3
Analysis and Ablation Study
User Study. We conducted user study with 20 annotators on the AIST++ Dance dataset to evaluate the generation performance. For each method, 200 samples were generated, and 20 paired samples were randomly selected for each comparison group. Annotators were asked to respond on site: "Which dance/music is more realistic and matches the music/dance better?". The human evaluation results, shown in Figure indicate that our method outperforms SOTA approaches both motion-to-musie and music-to-motion generations. Notably, a preference drop is observed when BiCoR-VAs -VE is not employed, highlighting the importance of an aligned latent space for cross-modal generation.
Motion Encoding, For motion sequence encoding, we compare the spatial position-basee method with the directional vector-based method, which learns the unit directional vectors of the given adjacency set, and reconstructs the human pose with the calculated mean bone lengths [88.. However, as shown in Table 7 (#1), the spatial position-basee -base method proved superior, likely due to the error introduced by movements that alter bone length, such squatting and bending.
8

---

Music Metrics
Motion Metries
Id
Method
FAD
Fl
FID +
Flt
#1
Ours w/ Directional Vectors
10.9
91.4
14.7
38.0
#2
Ours w/o Mel-spectrogram
12.8
95.6
9.5
41.6
#3
Ours w/o Rhythmic Contrastive Learning (RCL)
8.5
93.1
8.1
37.9
#4
Ours w/o Feed-Forward Forward Transformer (FFT)
11.0
95.8
11.6
41.4
#5
Ours (Joint Generation)
8.1
96.5
8.8
45.4
#6
Ours (Joint Generation& Variiab Length)
9.1
97.6
8.5
49.6
#7
Ours (Cross Generation)
8.9
98.1
7.3
46.2
Table Ablation study on motion- tu-ii and music- toriictt-ooee co-motion generations. the quality assessment and the Fl score the weat-matching assessment.
We use the FAD/FID
BiCoR
Ours BiCoR
Roal
Dance-to- Motion Preference
Music-to-Motion Preference
100%
100%
14.6
27.7
36.5
I
73.6
79.3
79.0
86.8
93.7
98.8
93.8
85.4
72.3
26.4
20.7
21.0
13.2
1.2
0%
0%
Figure 6: Results of human evaluation on motion-to-music and music-to-motion generations.
Music Encoding, For music encoding, we evaluated the spectrogram- -baedd method against the raw waveform-bas based method. According to Table 7 (#2), the raw waveform- method gains performance declines in both FAD and Fl metrics. This is attributed to the lengthy audio sequences introduced by the raw waveform, introducing difficulties for diffusion modeling training.
Learning Techniques. In MoMu-Diffusion, there are two key learning techniques: rhythmic contrastive learning (RCL) and Feed-Forward Transformer (FFT). From Table 7 we can observe that "Ours w/o RCL" gains clear drop on the beat-matching metric FI (#3) and "Ours w/o FFT" gains a drop on the synthesis quality metric FID/FAD (#4), respectively. "Ours w/o FFT" means we use a U-Net backbone for the diffusion model, which has been shown inferior to our FFT-based model in long sequence modeling. Equipped with both RCL and FFT, MiMuu--ffuusin ensures both cross-modal modal alignment.
generation quality and cross-
Joint Generation in Variable Length. MoMu-Diffusinn supports multi-modal joint generation in variable lengths, facilitated by "pad-and-truncate" strategy in the diffusion model and the proposed cross-guidance sampling. To validate this capability, 1000 samples with varying lengths (10-30 seconds) are generated using different Gaussian noise vectors. With a cross- cross-guidance sampling timestep T. 0.5T, Table 7 (#5, #6), we can find that for multi-modal joint generation, MoMu-Diffusion shows that MoMu-Diffusion achieves comparable performance to the conditional models with clean condition inputs and advanced performance the joint generation scenario. More ablation studies are provided in Appendix D
Conclusion
In this paper, we propose MoMu-Diffusion, the first multi-modal framework designed to learn the long-term synchronization and correspondence between human motions and music. In MoMu- Diffusion, we have two key designs: bidirectional contrastive rhythmic VAE (BiCoR-VAE) for learning modality-aligned latent spaces and Transformer-bant diffusion model for learning long- term dependencies. Through extensive experiments, we demonstrate MoMu-Diffusion' efficacy across tottion-o-muusc, music-to-motion. and joint motion-musia generations.

---

Acknowledgments
This work was supported by National Natural Science Foundation of China under Grant No. 62222211 This work was also supported by National Natural Science Foundation of China under Grant No.62072397.
References
[1] Alexanderson, S., Nagy, R., Beskow, J., and Henter, G. E. Listen, denoise, action! audio-driven motion synthesis with diffusion models. ACM Transactions on Graphics (TOG), 122, 2023.
[2] Brunner, G., Konrad, A., Wang, Y., and Wattenhofer, Midi-vae: Modeling dynamics and instrumentation of music with applications to style transfer. arXiv preprint urXiv:1809.07600 2018.
[3] Cao, Z., Hidalgo Martinez, G., Simon, T., Wei, S., and Sheikh, Y. A. Openpose: Realtime multi-person 2d pose estimation using part affinity fields. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2019.
[4] Davis, A. and Agrawala, M. Visual rhythm and beat. ACM Transactions on Graphics (TOG), 37(,):1-11, 2018.
[5] Dhariwal, and Nichol, Diffusion models beat gans on image synthesis. Advances Neural Information Processing Systems, 34:8780-8794 -7944, 2021.
[6] Dhariwal, P., Jun, H., Payne, C., Kim, J. W. Radford, A., and Sutskever, Jukebox: generative model for music. arXiv preprint arXiv:2005.003: 00341, 2020.
[7] Di, S., Jiang, Z., Liu, S., Wang. Z., Zhu, L.,, He, Z., Liu, H.,, and Yan, S. Video background music generation with controllable music transformer. In Proceedings of the 29th ACM International Conference on Multimedia, pp. 2037-2045, 2021.
[8] Dong. H.-W., Hsiao, W.-Y., Yang. L.-C., and Yang. Y.-H. Musegan: Multi-track sequential gen- erative adversarial networks for symbolic music generation and accompaniment. In Proceedings of the AAAI Conference Artificial Intelligence, volume 32, 2018
[9] Dong, H.-W., Chen, K., Dubnov, S., McAuley, J., and Berg-Kirkpatrick, T. Multitrack music transformer. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5. IEEE, 2023.
[10] Elizalde, B., Deshmukh, S., Al Ismail, M., and Wang, H. Clap learning audio concepts from natural language supervision. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5. IEEE, 2023.
[11] Esser, P., Kulal, S., Blattmann, A., Entezari, R., Müller, J., Saini, H., Levi, Y., Lorenz, D., Sauer, A., Boesel, F,, al. Scaling rectified flow transformers for high-resolution image synthesis. arXiv preprint arXiv:2403.0 03206, 2024.
[12] Fan, R., Xu, S., and Geng, W. Example-based automatic music-driven conventional dance motion synthesis. IEEE transactions visualization and computer graphics, 18(3):501-515, 2011.
[13] Gan, C., Huang, D., Chen, P., Tenenbaum, J. B., and Torralba, A. Foley music: Learning to generate music from videos. In Computer Vision -CCoo 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XI 16, pp. 758-775 Springer, 2020.
[14] Gemmeke, F., Ellis, D. P., Freedman, D., Jansen, A., Lawrence, W., Moore, R. C., Plakal, M., and Ritter, M. Audio set: An ontology and human-labeled dataset for audio events. In 2017 IEEE international conference on acoustics, speech and signal processing (ICASSP), pp- 776-780. IEEE, 2017.
10

---

[15] Grosche, P.. Müller, M., and Kurth, F. Cyclic tempogram a mid-level tempo representation for musicsignals. In 2010 IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 5522-5525. IEEE, 2010.
[16] Hawthorne, C., Simon, I., Roberts, A., Zeghidour, N., Gardner, J., Manilow, E., and Engel. J. Multi-instrument music synthesis with spectrogram diffusion. arXiv preprint arXiv:2206. 200400508,0 2022.
[17] Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., and Hochreiter, S. time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Gans trained by a two
[18] Ho. J. and Salimans, T. Classifier-free diffusion guidance. arXiv preprint arXiv: 2207. 2022.
[19] Huang, R., Hu, H., Wu, W., Sawada, K., Zhang, M., and Jiang, D. Dance revolution: Long-term dance generation with music via curriculum learning. arXiv preprint arXiv:2006.06119 2020.
[20] Huang, R., Huang, J., Yang, D., Ren, Y., Liu, L.,, Li, M., Ye, Z., Liu, J., Yin, X., and Zhao, Z. Make-an-audio: Text-to-audio generation with prompt-enhanced diffusion models. In International Conference on Machine Learning, pp. 13916-13932 PMLR, 2023.
[21] Huang, Y.-.. and Yang, Y.-H. Pop music transformer: Beat-based modeling and generation of expressive pop piano compositions. In Proceedings of the 28th ACM international conference on multimedia, pp. 180- 188, 2020.
[22] Kilgour, K., Zuluaga, M.. Roblek, D., and Sharifi, M. Fr), 'echet audio distance: metric for evaluating music enhancement algorithms. arXiv preprint arXiv: (812.0881100 2018.
[23] Kingma, D. P. and Ba, J. Adam: method for stochastic optimization. arXiv:1412.66 2014.
arXiv preprint
[24] Kumar, K., Kumar, R., De Boissiere, T., Gestin, L.,, Teoh, W. Z., Sotelo, J., De Brebisson, A.,
Bengio, Y., and Courville, A. C. Melgan: Generative adversarial networks for conditional waveform synthesis. Advances in neural information processing systems, 32, 2019.
[25] LaMothe, The dancing species: how moving together in time helps make us human. Aeon, June, 1:1, 2019.
[26] Lee, H.-Y., Yang, X., Liu, M.-Y. Wang, T.-C., Lu, Y.-D., Yang. M.-H., and Kautz, J. Dancing to music. Advances in neural information processing systems, 32, 2019.
[27] Lee, M., Lee, K., and Park, J. Music similarity-based approach generating dance motion sequence. Multimedia tools and applications, 62:895-91 912, 2013.
[28] Lee, S.-g.. Ping, W., Ginsburg, B., Catanzaro, B., and Yoon, S. Bigvgan: A universal neural vocoder with large-scale training. arXiv preprint arXiv:2206. 046688 2022.
[29] Li, J.. Yin, Y., Chu, H., Zhou, Y., Wang, T., Fidler, S., and Li, H. Learning to generate diverse dance motions with transformer. arXiv preprint arXiv:2008. 08171, 2020.
[30] Li, R., Yang, S., Ross, D. A., and Kanazawa, A. Ai choreographer: Music conditioned 3d dance generation with aist++. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 13401-13411 2021.
[31] Liu, H., Chen, Z., Yuan, Y., Mei, X., Liu, X., Mandic, D., Wang, W., and Plumbley, M. D. Audi- oldm: Text-to-audis generation with latent diffusion models. arXiv preprint arXiv:2301.12500 2023.
[32] Liu, X., Wu, Q., Zhou, H., Xu, Y., Qian, R., Lin, X., Zhou, X., Wu, W., Dai, B., and Zhou, Learning hierarchical cross-modal association for co-speech gesture generation. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pp. 10462-10472, 2022.

---

[33] Luo, S.. Yan, C., Hu, C., and Zhao, H. Diff-foley: Synchronized iileo-to-audig synthesis with latent diffusion models. Advances Neural Information Processing Systems, 36, 2024.
[34] Mittal, G., Engel, J., Hawthorne, C., and Simon, Symbolic music generation with diffusion models. arXiv preprint arXiv:2103. /6091, 2021.
[35] Ofli, F,, Erzin, E., Yemez, Y., and Tekalp, A. M. Learn2dance: Learning statistical music-to- dance mappings for choreography synthesis. IEEE Transactions on Multimedia, 14(3):747-759, 2011.
[36] Pasini, M. and Schlüter, J. Musika! fast infinite waveform music generation. arXiv preprint arXiv:2208.08700 2022.
[37] Peebles, W. and Xie, S. Scalable diffusion models with transformers. In Proceedings of the JEEE/CVF International Conference on Computer Vision, pp. 4195 4205, 2023.
[38] Radford, A., Kim, W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pp. 8748-8763, PMLR, 2021.
[39] Ren, Y., He, J., Tan, X., Qin, T., Zhao, Z., and Liu, T.-Y. Popmag: Pop music accompaniment generation. In Proceedings of the 28th ACM international conference multimedia, pp. 1198-1206, 2020.
[40] Roberts, A., Engel, J., Raffel, C., Hawthorne, C., and Eck, D. hierarchical latent vector model for learning long-term structure in music. In International conference muchine learning, pp.
PMLR, 2018
[41] Ruan, L., Ma, Y., Yang, H., He, H., Liu, B., Fu, J., Yuan, N. J., Jin, Q., and Guo, B. Mm- diffusion: Learning multi-modal diffusion models for joint audio and video generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 10219-10228, 2023.
[42] Shao, D., Zhao, Y., Dai, B., and Lin, D. Finegym: A hierarchical video dataset for fine-grained action understanding. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp. 2616-2626 2020.
[43] Su, K., Liu, X., and Shlizerman, E. Audeo: Audio generation for a silent performance video. Advances in Neural Information Processing Systems, 33:3325-3337, 2020.
[44] Tsuchida, S., Fukayama, S., Hamasaki, M., and Goto, M. Aist dance video database: Multi- genre, multi-dancer, and multi-camera database for dance information processing. In ISMIR, volume 1, pp. 6, 2019.
[45] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[46] Xia, J., Zhuge, M., Geng, T., Fan, S., Wei, Y., He, 7.,, and Zheng, F. Skating-mixer: Multimodal mlp for scoring figure skating. arXiv preprint arXiv:2203.0399 2022.
[47] Xu, C., Fu, Y., Zhang, B., Chen, Z., Jiang, Y.-G., and Xue, X. Learning to score figure skating sport videos. IEEE transactions circuits and systems for video technology, 30(12):4578-44 4590. 2019.
[48] Yoon, Y., Cha, B., Lee, J.-H., Jang, M., Lee, J., Kim, J., and Lee, G. Speech gesture generation from the trimodal context of text, audio, and speaker identity. ACM Transactions on Graphics (TOG), 39(6):1-16, 2020.
[49] Yu, J., Wang, Y., Chen, X., Sun, X., and Qiao, Y. Long-term rhythmic video soundtracker. In International Conference on Machine Learning, pp. 40339-40353, PMLR, 2023.
[50] Zhu, L., Liu, X., Liu, X., Qian, R., Liu, Z.,, and Yu, L. Taming diffusion models for audio-driven co-speech gesture generation. In Proceedings of the IEEF/CV Conference on Computer Vision and Pattern Recognition, pp. 10544-10553, 2023.
12

---

[51] Zhu, W. Ma, X., Liu, Z., Liu, L., Wu, W., and Wang. Y. Motionbert: A unified perspective on learning human motion representations. Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 15085-15099, 2023.
[52] Zhu, Y., Olszewski, K., Wu, Y., Achlioptas, P., Chai, M., Yan, Y., and Tulyakov, S. Quantized gan for complex music generation from dance videos. In European Conference on Computer Vision, pp. 182-199. Springer. 2022.
[53] Zhu, Y., Wu, Y., Olszewski, K., Ren, J., Tulyakov, S., and Yan, Y. Discrete contrastive diffusion for cross-modal music and image generation. arXiv preprint arXiv:2206.0777) 2022.
[54] Zhuang, W. Wang, C., Xia, S., Chai, J., and Wang, Y. Music2dance: Music-driven dance generation using wavenet. arXiv preprint 2002:1200..03661, 3(4):6, 2020.
13

---

Appendices
MoMu-Diffusion: On Learning Long-Term Motion-Music Synchronization and Correspondence
A Implementation Details of BiCoR-VAE
A.1 Model Configurations
For EICoR-VAE, we use an encoder-decoder architecture, which the convolutional network and spatial transformer are used. The input is downsampled by a Convld 1d downsampling layer and then forwarded to the middle block, finally upsampled by a Conv Id upsampling layer. The detailed hyper-parameters of BiCoR-Va VAE are listed in Table
A.2 Model Training
We use two-stage training strategy for BiCoR- VAE. Firstly, we train the mel-speetrogram VAE with three loss functions: reconstruction loss Lrecon, KL. loss LK and a GAN loss LGAN to prevent over-smoothed mel-spectrogram:
Lstagel = Creemn + AI + A2LGAN,
(10)
where Å is set to le-5 and X2 is 0.5. Note that the GAN loss contains two steps: it first updates the generator part (mel-spectrogram VAE) with Estagel, and then updates the additional discriminator. After training mel-spectrogram VAE, we freeze it, and train the motion VAE with the proposed contrastive rhythmic learning loss defined in Eq (4) in stage 2:
Lstage2 Crecon + λ3C L AeCoontrasts
(11)
where A is set to le-5 and X is set to 1. For training BiCoR-VAF, we use the Adam W optimizer with a learning rate of 2c-4 and training epochs of 300. We NVIDIA 4090 GPUs and it takes about 12 hours to finish. To decode the mel-spectrogram into high-fidelity music, we use the BigvGAN model 28 pretrained on the AudioSet dataset (4
B
Implementation Details of Cross-Modal Generation
B.1
Dataset
For motion-to-musiee we evaluate our method on the latest LORIS benchmark 49], which contains 86.43 hours of video samples with paired music. This benchmark incorporates three challenging scenarios:
dancing, floor exercise, and figure skating. For dancing, 1,881 25-second videos are collected from AIST+ [30]. fine-annotated subset of the dancing dataset AIST [44]. For floor exercise, 1,950 25-second and 660 50-second videos are collected from the Ginegym dataset [42]
For figure skating, 8,585 25-second and 4, 1.4 50-second videos are collected from the FisV 47 and FS1000 46) datasets.
For music-to-motion. we use two datasets: AIST++ Dance [30] and BHS Dance. About 71 hours BHS Dance videos are collected from [26]. which contains three dancing types: "Ballet", "Zumba", and "Hip-Hop"" In our experiments, each dataset randomly split with 90%/5%/5% proportion for training, validation, and testing.
Note that only the AIST++ Dance dataset used for both generations. This is because the Floor Exercise and Figure Skating datasets involved too heavy motion variation, which makes it hard for the pose extraction algorithm to extract the high-accuracy motion sequences. As for the BHS Dance dataset, it only provides the MFCC audio features without raw audio. Therefore, we can not conduct motion-to to-toni experiments on it.
and music-to-motie
B.2 Data Processing
We use mel-spectrogram as audio feature representation, We first resample the audio 16kHz. We use 80 filters with fft set 1024 and hop length set 256 while processing the mel spectrogram
14

---

Syype-Parameters
BiCoR- VAE
Hidden channels
20
Residual blocks
2
Channel multiplier
[1,2,4]
Spatial attention layers
3
Downsampling rate
2
Kernel size of Conv ld
5
Total Params
213M
Table 8: Hyper-parameters of the BiCoR- BiCoR-VAE model.
Parameters
MoMuuDiffusion
Dimension of conditional embedding
1024
Input channels
20
Dimension of Hidden representation
576
Number of attention heads
Number of Transformer blocks
4
Kernel size of Conv ld projection network Padding of Conv Id projection network
5
3
Diffusion Steps
1000
Total Params
158M
Table 9: papper-prrmmeerss of the FFT model.
using Hann window with a window size of 1024. For human motions, OpenPose is applied to extract 2D body keypoints, and can process a video at 60 fps. We use the pre-trained Body-25 model extract 25 key points of the human body, but some key points are difficult extract consistently and some are less relevant to actions. As implemented by [26], we finally choose the 14 most relevant keypoints to represent the poses, i.e., nose, neck, left and right shoulders, elbows, wrists, hips, knees, and ankles. We interpolate the missing detected keypoints from the neighboring frames so that there are no missing keypoints all extracted clips.
B.3 Model Configurations
For the denoising part, we use the Transformer backbone rather than the U-Net. The hyper-parameters of our FFT model are listed in Table 9 The FFT diffusion model is trained by the AdamW opti- mizer (23] with learning rate of 1.6e-5 and lambda linear scheduler with a warmup step of 10000. We train the diffusion model with 200 epochs for each task. takes about 2 days for 8 NVIDIA 4090 GPUs. For the Figure Skating dataset, it takes about 4 days since this dataset is large.
B.4 Evaluation Metrics: Motion-to-Music
To evaluate whether the synthesized music is aligned with the given motion, we use the improved Beats Coverage Scores (BCS) and Beat Hit Scores (BHS) to validate the rhythm correspondence and cross-modal alignment of synthesized music. The improved BCS and BHS first proposed by 26), then used for rhythmic dance-to-music validation (53] 52]. and improved by [49 for long-term rhythmic music validation. Also, we report Coverage Standard (eV) and Hit Standard Deviation(CSDD CSD) to evaluate the robustness of generative models. Finally, the F1 scores of improved BCS and BHS are also reported as an overall assessment. BCS and BHS are designed by computing matching degrees of the rhythm points from synthesized music and ground- truth music. Let N₂ be the rhythm point number of synthesized music, N₂ be the rhythm point number of ground-ttruh music, and Nm be the number of matched rhythm points, the BCS is defined / Nr, respectively. However, these metrics are not suitable for long-term music evaluations since 1) the recond-wise rhythm detection algorithm leads to an extremely sparse vector and 2) BHS can easily exceed if the rhythm points of generated music more than ground truth. Therefore use improved audio onset detection
as BCS Nn. N₂ and the BHS is defined as BHS = N
15

---

Algorithm 1: Pscudo code for cross-modal (motion-lo-music) sampling.
Input: The latent mel-spectrogram representation Za. latent motion representation Zm, the pre-trained denoiser 0g for motion-to-music, and the decoder D, for mel-spectrogram.
t T.
(t) f 10
while f do
(t) 4 sample from pa, (Eo(t - 2z) 2m)
t At
end
return (z(
algorithm avoid sparse rhythm vectors. Here is the Python code based the Librosa library: librosa.onset.ong onset detect(y=aadio, sr=sampling_ram rate, wait=1, delta=0.2, pre_avg=3, post__vv==: pre_ me__max=3, poss_mmx=3,, units= 'time'). )
For validating the quality of synthesized music, we use the Fréchet Audio Distance (FAD) and the Diveristy score. We use the pre-trained VGGish model from httpp::// thub com/ gudgud96/ to compute the FAD scores. Based on the feature extractor VGGish, we compute the Diversity score by using the average feature distance for paired samples. Specifically, the Diversity score contains inter-diversity and intra-diversity. Inter-diversity is obtained by computing the average feature distance between 200 combinations of 50 pieces of music from different motions and the intra-diversity is obtained by computing the average feature distance between all combinations of pieces of music from the same motion input.
B.5 Evaluation Metrics: Music-to-Motion
To evaluate whether the synthesized motion is aligned with the reference music, we also use these five beat-matching metries. However, since the number of musical beats is always number of kinematic beats in real-world products, we use the musical beats as the reference for evaluation, which is also consistent with previous works (26). Coneretely, Let N, be the kinematic point number of synthesized motion, N, be the rhythm point number of ground-truth music, and Nn be the number of matched points, the BCS defined BCS Nn/ N, and the BHS is defined as BHS- = / / N,, respectively. For kinematic beat extraction, we use the bin-wise directrogram difference (defined Eq (2)) as the indicator
than the
For validating the quality of synthesized motion, we use the Fréchet Inception Distance (FID) and the Diveristy score. To compute the FID score, follow the design of (26) and train a motion classifier on the BHS Dance dataset with three classification categories: "Ballet", "Zumba", and "Hip-Hop".. The motion classifier consists of MotionBert encoder [51] and classification head. The motion classifier is trained by an Adam optimizer with learning rate of Ic-4 and 100 epochs, Then, we use the trained MotionBert encoder as the feature extractor for computing the FID and Diversity scores. The definition of Diversity score here is the same as Appendix B.4
C Pseudo Codes
We provide the pseudo-codes of cross-modal generation and multi-modal joint generation in Al- gorithm [ and respectively. For the cross-modal generation, take the motion-to-muse as an example while the implementations music-to-motin are symmetrical.
D The Choice of T. in Joint Generation
We propose a simple pross-guidance sampling strategy to combine multiple cross-modal generative models for joint generation. In this process, there is a hyper- parameter T. that controls the modality fusion timestep. In Table 10, we study five variants: T. begins from 0.97 0.17 with an interval of 0.27. From these results, we can observe that employing the cross-guidance strategy in the early sampling steps is not feasible since the predicted latent representation contains too many noises. However, we can find that MoMu-Diffusiin (Te 0.7l) obtains acceptable performance,
16

---

Algorithm 2: Pscudo code for multi-modal joint sampling.
Input: The latent mel-spectrogram representation Za. latent motion representation 2m. the pre-trained denoisers 0g and f.. the decoders D2 and Dm, the cross-guidance scale the pre-defined schedule o1,, and a. II Qç.
t T.
Te f- 7/
(t) A/(0,))
2m (t) t- (0,1)
while T 2t> do
(t - At) f sample from pea (za(t [t) (t),
2m (t At) sample from pom (Zm [t] (t),mm(()))))))) (,),
t < 1 At
end
while
To 1. > 0 do
En val
za(()
Va. Co. (za (t),t)) t))
2m
5m(
vI-ai
vor 6Bm (t), 0)
var
(a At) + sample from paa (t - At) 2z (t),t,
2m ( At) sample from psm (zm At) 12m (t), ta)
t t t At
end
return Da (Ea), Dm (2m)
Music Metries
Motion Metries
Method
Joint Generation
FAD 4
F1 f
FID 4
F1f
MoMu-Diffusion (Te 0.97)
14.8
82.4
17.3
25.8
MoMu-Diffusse (Te =0.77)
10.9
95.9
9.7
37.8
MiMu--iffusion (I, =0.57)
8.1
96.5
8.8
45.4
MoMuu-iifusiin (Tc = -0.37)
7.5
90.4
9.0
42.1
MoMu-Diffusion (Te 0.17)
8.0
85.5
9.5
32.6
Table 10: Ablation study of the cross cross- gooss-guidance step T, on the AIST+ Dance dataset.
indicating that the denoising process is coarse-to-fine. Using fewer cross-guidance sampling steps (like T. 0.17) ensure the quality of generated samples, but the cross-modal alignment is omitted, leading to a low F1 score. Therefore, we use T. = 0.57 in our paper to trade off the sampling quality and cross- moddll alignment in joint generation.
E Failure Cases
Since our method predicts pose points, the deviation between points can cause abnormal length of human skeleton. Three sets of examples are shown in Figure with anomalous frames on the left and the corrected frames on the right.
We first calculated the average bone length between each pair of keypoints in the dataset, after which we post-processed the predicted keypoints. We specify a threshold (which we specify as 1.3 times the mean bone length) and correct the predicted bone lengths to the mean bone length once they exceed the threshold, an approach that significantly enhances model generation. It is worth noticing that the posc-proes can not fully address this issue but alleviate
F More Qualitative Results
We provide more qualitative results in Figure g
LO
and

---

Figure 7: Three failure cases and the corrected results.
G Used Resources and Licenses
In this paper, use several open resources, including https://github...
OpenGVLab/LORIS! All Rights Reserved), Commercial),
(All Rights Reserved), https:://gtchhb..com//uuggguuuup MotionBERT (All Rights Reserved), and https: httts:///itthut thub. -ow.oobttttteeet---ttee
(NVIDIA Source Code License, 1-Way
sethhet-audiodiisttaet
Make An Audio (All Rights Reserved), We use these resources for research purpose only.
Limitations and Boarder Impact
Due to the limited motion-min music data and computing resources, the scaling law of model is not testified in super large dataset. Besides, model depends some data pre-processing methods like mel-spectrogram extraction and keypoints extraction by OpenPose, which may lead to error accumulations.
MaMuu-Difusion promotes both neural motion and music synthesis. so it may help expand any impact that generative systems have on the broader world like copyright conflicts. We will add constraints and licenses when open-resoureing our code and pre-trae pre-trained models.
18

---

Video Frames (ref)
Musical Beats urrs))
Musical Beats (GT)
Video Frames (ref)
Musical Beats (0urs urs)
Musical Beats (GT)
Video Frames (ref)
Musical Beats (Ours)
Musical Beats (GT)
Video Frames (ref)
Musical Beats (0 w
Musical Beats (GT)
Video Frames (ref)
Musical Beats (0 urus
Musical Beats (GT)
Figure Example of beat matching on the AIST++ Dance (motion-to-musie). The red dashes indicate the extracted musical beats. The red arrow points to the video frame at that particular moment.
19

---

Video Frames (ref)
Musical Beats (Ours)
Musical Beats (GT)
Video Frames (ref)
Musical Beats (0 urs)
Musical Beats (GT)
Video Frames (ref)
Musical Beats (Ours)
Musical Beats (GT)
Video
Frames (ref)
Musical Beats (Ours)
Musical Beats (GT)
Video Frames (ref)
etx
Musical Beats (Ours)
Musical Beats (GT)
Figure Example of beat matching on the Floor Exercise (motion-toomussi). The dashes indicate the extracted musical beats. The red arrow points to the video frame at that particular moment.
20

---

OHH
10SH
Video Frames (ref)
RIZ
Musical Beats (Ours)
Musical Beats (GT)
Video Frames (ref)
Musical Beats urrs))
Musical Beats (GTT)
TOK
O
E
Video Frames (ref)
Musical Beats (0 urs)
Musical Beats (GT)
ORL CORO EC
Video Frames (ref)
Musical Beats (0 w
Musical Beats (GT)
KOSE
ee
M
Video Frames (ref)
Musical
Beats (Ours)
Musical Beats (GT)
Figure 10: Example of beat matching on the Figure Skating (motion-to-musse The dashes indicate the extracted musical beats. The red arrow points to the video frame at that particular moment.
21

---

Motion Frames (Ours)
Kinematic Beats (rus))
Musical Beats (ref)
Motion Frames (Ours)
Kinematic
Beats (Ours)
Musical Beats (ref)
Motion Frames (Ours)
Kinematic Beats (Ours)
Musical Beats (ref)
Motion Frames (Ours)
Kinematic Beats (Ours)
Musical Beats (ref)
Motion Frames (0 urs)
Kinematic Beats (Ours)
Musical Beats (ref)
Figure 11: Example beat matching on the AIST+ Dance cousic-to-motion The red dashes indicate the extracted kinematic beats of the synthesized motion. The red arrow points to the frame of the synthesized motion sequence at that particular moment.
22

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The Abstract and Section accurately reflects the paper's scope: motion-music generation and contributions: 1) BiCoR-VAR VAE and FFT model for multi-modal generation, and 2) extensive experimental evaluations.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We have discussed the limitations of our work in Appendix
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiatiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
23

---

Answer: [NA]
Justification:
Our work does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We have provided all experimental details in Section Appendix A and
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with open-source dataset instructions for how construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
24

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [No]
Justification: We cannot upload our code in the peer-reviewing process due copyright constraints. We will open-source our code and data upon publication.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://niis.cc.
cublic//uudees/CodeStbbaas www.eeesuuuuisssoooooooo.o for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ //nips. wttps/cccppppclggui.aeaeaeccuuea
Stteee///ooSSSeeissSioica www.ssssiicggooooo for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We have specified all the training and test details in Section Appendix A
B
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We have suitably reported error bars in our main results, See the CSD and HSD metrics in Table 2 and the error bars Figure 3.6
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoo bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
25

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification: We have provided sufficient information on the computer resources in Ap- pendix and
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics Ethics/https/
Answer: [Yes]
Justification: Our work conforms, every respect, with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g.. if there special consid- eration due to laws or regulations in their jurisdiction..
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We have discussed the broader impacts of our work in Appendix
Guidelines:
The answer NA means that there is no societal impact of the work performed.
26

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification:
We have described safeguards in Appendix H
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We have properly listed the licenses of the resources we used in Appendix G
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URI.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
27

---

If assets are released, the license, copyright information, and terms of use in the thcode has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssi
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do not release assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [Yes]
Justification: We have included the full text of instructions given to the participants in Section 5.3
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
supplemental material is fine, but if the main contribu-
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [Yes]
Justification: In Section 5.3 we have mentioned that we just conduct human evaluations compare different synthesized samples. The synthesized samples only include human motions and musical audios, which have no potential risks.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
28

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
29


Reviews:
[{"review_id": "nDgak8sCOg", "reviewer": "Reviewer_E4Pj", "summary": "This paper proposes a framework that enables the generation of music from motion, motion from music, or both simultaneously while maintaining synchrony between the two. To achieve this, kinematic amplitude is extracted from the motion, and motion audio clip segments corresponding to different kinetic amplitudes are sampled. Contrastive loss is applied to the audio and motion embeddings of these sampled segments to ensure that the embeddings align with each kinetic amplitude. The resulting embeddings are then used for joint diffusion, enabling audio-to-motion, motion-to-audio, or joint generation. A method for applying cross guidance is also proposed. Quantitative evaluations were conducted based on the beat information of the generated music, as well as the FID and diversity of the generated motion. A user study demonstrated that the generated outputs were preferred over those from previous models.", "questions": "Suggestion: Section 3.2 is a crucial part that explains BiCoR-VAE, but the process of sampling clips was difficult to understand. Referring to Figure 2(a) made it easier. Including an example to explain this process would make it more comprehensible.", "limitations": "-", "rating": 9, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "The paper proposes an embedding method using kinetic amplitude and contrastive learning to ensure that the intensity of motion and music/motion are well-aligned. This approach effectively leverages prior knowledge of motion. I believe this methodology can be applied to problems requiring synchronization of intensity and timing beyond just motion-music synchronization. The use of diffusion for multimodal generation, incorporating DIT and introducing a cross-guidance sampling strategy, also holds potential for future research in multimodal generation.", "weaknesses": "The proposed methodology leverages the relationship between changes in motion and beats in music as the primary element of synchronization, with evaluations mainly focused on this aspect. However, there can be instances where the beatness in motion or music is not clear (especially during certain periods). A discussion on how to handle such cases, or considerations beyond beatness, would be beneficial for achieving more natural generation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for his/her insightful comments and for appreciating the technical contributions of our paper. Here, we respond to your questions as follows:\n\n***1. How to handle the cases where the beat in motion or music is not clear and achieve/evaluate natural generation beyond beat alignment?***\n\nTraining Phase: The diffusion model's goal is to learn the conditional distribution between motion and music, encompassing not only beat alignment but also the synchronization of temporal and semantic elements. While additional alignments beyond the beat are not explicitly modeled during the pre-training phase, they are implicitly achieved through the diffusion model. The incorporation of supplementary learning objectives, such as semantic contrast as utilized in Diff-Foley, into the BiCoR-VAE could lead to challenges in hyper-parameter tuning and optimization. Therefore, we only emphasize the beat alignment in the pre-training stage and implicitly model the cases where the beat is not clear in the diffusion training. In future work, we will conduct an in-depth study on stably combining both beat and non-beat alignments during the pre-training stage. \n\nEvaluation Phase: In evaluating the quality of the generated samples, we have implemented a suite of beat-matching metrics (e.g., beat F1 score) and metrics that assess the overall generation quality (e.g., FAD). Compared to introducing additional objective metrics, we posit that the design of subjective evaluations holds greater significance in assessing motion-music generation. Drawing a parallel with the well-researched domain of text-to-speech synthesis, despite numerous objective metrics such as word error rate and speaker similarity, subjective assessments are still significant for measuring the naturalness of synthesized samples. Consequently, in the context of motion-music generation, we have conducted subjective A/B testing to compare our approach with existing state-of-the-art methodologies, as depicted in Figure 6.\n\n***2. Including an example to make the Section 3.2 easier to understand.***\n\nThanks for your suggestions. For a motion-music latent representation pair, we compute the 2D motion directogram according to Eq. (1) first, and then derive the kinematic amplitude indicator for each temporal step by aggregating the bin-wise directogram difference according to Eq. (2). Finally, we randomly sample the motion-music clips from different categories to construct the contrastive learning objective according to Eq. (4). Note that the classification of the different categories is based on the kinematic amplitude indicator values, which undergo in-clip max-pooling, as described by Eq. (3). We will include the example in Figure 2 along with its accompanying explanations to make the Section 3.2 easier to understand.\n\n\nThanks again for your valuable comments. We are happy to engage in discussion if you have any further questions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5KwFtc1yOG", "reviewer": "Reviewer_pw59", "summary": "The paper propose MoMu-Diffusion, that generates both music-to-motion and motion-to-music videos. MoMu-Diffusion achieves SOTA results and is able to generate joint distribution of music and motion instead of one way.", "questions": "1. How long can you generate the video in the joint distribution generation setting? Is that possible to use DiffCollage (Zhang et al., 2023) for longer video generation?", "limitations": "1. Maybe can explore longer video generations.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Generate joint distribution of music and motion\n2. Propose rhythmic contrastive learning to align music and motion.\n3. Demo included, reasonable evaluation metrics and great experiments.", "weaknesses": "1. I might miss it, but that would be nice if there is an ablation study of the hidden size and layers of Diffusion Transformer.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for his/her insightful comments and for appreciating the technical contributions of our paper. Here, we respond to your questions as follows:\n\n***1. Ablation study of the hidden size and layers of Transformer***\n\nWe appreciate your suggestion on the ablation study of model size. Due to the time limit, we may be not able to present comprehensive results (e.g., model params $\\textgreater$ $1$B) in rebuttal within a short response period. We implement on the AIST++ dataset and study four model configurations: the combination of hidden size in $[576,768]$ and Transformer layer in $[4,8]$:\n\n|Hidden size | Layers | Params | Beat-F1 (mo2mu) $\\uparrow$| FAD (mo2mu) $\\downarrow$| Beat-F1 (mu2mo)$\\uparrow$ | FID (mu2mo) $\\downarrow$|\n|-----------|---------|---------|---------|---------|---------|---------|\n| 576 | 4 | 158M | 98.1 | 8.9 | 46.2 | 7.3|\n|576 | 8 | 315M | 98.3 | 9.0 | 46.7 | 6.8 |\n| 768 | 4 | 281M | 98.1 | 8.7 | 47.5 | 7.5 |\n768 | 8 | 559M | 98.7 | 7.2 | 52.1 | 6.2 |\n\n\nWe can observe that there is a slow performance boost when scaling the model size. The scaling law usually suggests that the performance can be improved by scaling up the training data and model size, simultaneously. It is reasonable to hypothesize that the bottleneck for the performance of motion-music generation lies in the data size. For existing public datasets like AIST++, LORIS, and BHS Dance, we think the base model configuration (DiT: 158M, BiCoR-VAE: 213M) is matched.\n\n***2. How long can we generate the video in the joint generation setting?***\n\nIn joint generation, the length of the generated video is suggested to be less than 25 seconds, which is mainly caused by the video length of training data (i.e., AIST++). Currently, for motion-music generation, the major difficulty in achieving longer video generation is the lack of corresponding training data. In terms of algorithm design, It is reasonable to speculate that DiffCollage is effective in dealing with longer video generation (more than 1min) thanks to its parallel sampling and smaller sequence length. For the experiments in our paper, the end-to-end diffusion formulation can effectively address the problems and it is not necessary to split the sequence into several clips. In future work, we are going to build a larger dataset with longer video length and are willing to incorporate DiffCollage for a faster and longer generation.\n\nThanks again for your valuable comments. We are happy to engage in discussion if you have any further questions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RSm8c3SBdw", "reviewer": "Reviewer_1QHY", "summary": "MoMu-Diffusion is a motion-music co-generation model with an aim of improved temporal synchronization of the generated motion and music sequence, based on two major components:\n1. a bidirectional contrastive rhythmic VAE (BiCoR-VAE) that provides aligned latent space of the motion and music through joint training, and\n2. a multi-modal diffusion transformer (DiT) the model the latent space with cross-guidance sampling.", "questions": "* (line 121-122): I think it would be helpful to add a specific example of the framerate differece between motion and music frames. Currently it might read like the motion framerate is always higher (by integer multiple) than the mel spectrogram framerate.\n* Assuming that the framerate design is defined by the mel framerate (followed by the vocoder), is resampling the motion sequence to match the mel framerate trivial enough to assume that we have the time aligned motion and music frames in the pre-processing stage?", "limitations": "The authors have mentioned the limitations in this work, including limited computational and data budget to verify the idea to scaled up dataset that potentially include video clips that do not adhere to the assumptions made in the rhythmic contrastive objective.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The proposed BiCoR-VAE's rhythmic contrastive objective using a kinematic amplitude indicator is well inspired and interesting approach to extract key frames that contain motions of interest. It is based on the valid assumption that the motions in the video data likely align with the beat present in the music.\n\nThe beat matching metrics in Table 2 justifies the improvements made in the proposed approach, and the ablation studies has been done thoroughly in Table 7. Human evaluation in Figure 6 shows strong prererence to the proposed model.", "weaknesses": "I would like to see the qualitative samples of the baselines and the ablation models to the demo page as well to form the readers opinion on the claimed synchronization of this work. While the presented results in the manuscript looks convincing, having direct comparisons in the demo will be helpful for the readers to have a better idea of the improvements.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for his/her insightful comments and for appreciating the technical contributions of our paper. Here, we respond to your questions as follows:\n\n***1. Comparison with other methods on the demo page.***\n\nThanks for your suggestions. Since the illustrated video samples are pretty long, the existing demo page has exceeded 567 MB. We will work to add the generated samples of SoTA methods: LORIS in motion-to-music and D2M in music-to-motion and re-organize the demo page. Currently, you can refer to their released samples from https://justinyuu.github.io/LORIS and   https://github.com/NVlabs/Dancing2Music for a preliminary qualitative comparison.\n\n***2. Add a specific example of the different frame rates of motion and music inputs.***\n\nAssuming that we have a ten-second video and process it at 60 fps for pose sequence extraction, which results in 600 motion frames. For the corresponding audio, we resample it to 16khz and extract the mel-spectrogram with a hop length of $256$. Then, the audio frames are $10\\times16000/256=625$. Finally, to align the frames from different modalities, we evenly drop $25$ audio frames to ensure the temporal dimensions of motion and music inputs are equal. In the revised manuscript, we will add an example to make it clearer.\n\n***3. Frame rate alignment between motion and music inputs in the pre-processing stage.***\n\nDue to the different sampling rates of video and audio, it is hard to ensure the audio and visual inputs are strictly temporally aligned. We ensure that the timing error is within a small range when we implement cross-modal temporal alignment in the pre-processing stage. For example, following the example mentioned above, we denote the audio frame rate as $f_1$ and the motion sequence frame rate as $f_2<f_1$. The minimum timing error is $|1/f_1-1/f_2|$ and the maximum timing error is $|k/f_1-k/f_2|$, where $k$ is the dropping interval and is computed by $k/f_1=(k-1)/f_2$. Recalling that $f_1=16000/256$ and $f_2=60$, the range of timing error is ($0.6$ms, $17$ms). For human beings, a timing error of less than $30$ms is typically hard to capture. Also, we will improve the pre-processing techniques to reduce the timing error in future work.\n\nThanks again for your valuable comments. We are happy to engage in discussion if you have any further questions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PFtFau4c1p", "reviewer": "Reviewer_aP7b", "summary": "The authors proposed a novel framework to address the motion-to-music and music-to-motion tasks. They leveraged aligned latent spaces between motion and music, a multi-modal diffusion transformer, and a cross-guidance sampling strategy. Experiments were conducted to demonstrate that their approach outperforms their counterpart models.", "questions": "- The authors discussed the architectures of the multi-modal VAE in Section 3.1. Could the authors provide more specific implementation details, including\n  - Whether the motion vectors in shape $\\mathbb{R}^{T_m\\times J\\times 2}$ are first of all flattened to $\\mathbb{R}^{T_m\\times(J\\times 2)}$ for VAE? If not, how are they processed?\n  - What exactly are $T_a$ and $T_u$ in Line 105 for the music VAE? Does this imply that the music VAE has a lower downsampling ratio than the motion VAE in the temporal dimension?\n  - Is normalization needed for the autoencoder representations of the both modalities so that they can be further processed by the diffusion model?\n- The authors went through how they constructed the motion-music latent clip for VAE training which incorporates the kinematic amplitude indicator as depicted in Eq. 2. Could the authors further elaborate on why incorporating this is important by e.g., showing a histogram of kinematic amplitude in the dataset and how it is sampled?", "limitations": "The authors have adequately address the limitations.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper proposes several interesting concepts, including rhythmic contrastive learning to produce aligned latent spaces across different modalities and transformer-based diffusion models to address music-to-motion and motion-to-music synthesis tasks. Additionally, it offers the possibility to jointly synthesize music and motion. The user study and ablation study are also well conducted.", "weaknesses": "Please refer to the questions section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for his/her insightful comments and for appreciating the technical contributions of our paper. Here, we respond to your questions as follows:\n\n***1. More specific implementation details***\n\na) The motion vectors are flattened to $n \\in \\mathbb{R}^{T_m\\times(J\\times2)}$ first.\n\nb)  No. We use the same downsampling rate for both motion and music inputs (Table 8 in the Appendix) and align the frame rates by pre-processing techniques.\n\nc) Yes. The latent representations from different modalities are normalized before forwarding the diffusion model.\n\nThanks for your comments. We will add these explanations in the revised manuscript to make it clearer.\n\n***2. Why incorporate the kinematic amplitude indicator and how the latent clip is sampled?***\n\nWhy: In lines 111-117, we mentioned that existing audio-visual temporal contrastive pre-training methods are not feasible for motion-music alignment. For example, the temporal contrast, which seeks to maximize the similarity of audio-visual pairs from the same time segment while minimizing the similarity of pairs from different segments, may push away two clips that sampled from different time segments but shared similar rhythmic beats. That is why we incorporate the kinematic amplitude indicator to enable motion-music contrastive pre-training. As for the illustration of kinematic amplitude, you can refer to the line chart example in Figure 2(a), whose values are normalized within the range of (0,1). \n\nHow: In Figure 2(a), we have vividly illustrated the sampling process. For a motion-music latent representation pair, we compute the 2D motion directogram according to Eq. (1) first, and then derive the kinematic amplitude indicator for each temporal step by aggregating the bin-wise directogram difference according to Eq. (2). Finally, we randomly sample the motion-music clips from different categories to construct the contrastive learning objective according to Eq. (4). Note that the classification of the different categories is based on the kinematic amplitude indicator values, which undergo in-clip max-pooling, as described by Eq. (3). We will include the example in Figure 2 along with its accompanying explanations to make the Section 3.2 easier to understand.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: All reviewers find the paper interesting and novel. They unanimously agree that the paper warrants an acceptance. There are a few minor but still valuable suggestions, and I highly recommend the authors polish the paper further to make it more complete.
Decision: Accept (poster)