MoMu-Diffusion: On Learning Long-Term Motion-Music Synchronization and Correspondence

Authors: fuming  you
minghui  fang
li  tang
rongjie  huang
yongqi  wang
zhou  zhao
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-10 20:32:42

Abstract: Motion-to-music and music-to-motion have been studied separately, each attracting substantial research interest within their respective domains. The interaction between human motion and music is a reflection of advanced human intelligence, and establishing a unified relationship between them is particularly important. However, to date, there has been no work that considers them jointly to explore the modality alignment within. To bridge this gap, we propose a novel framework, termed MoMu-Diffusion, for long-term and synchronous motion-music generation. Firstly, to mitigate the huge computational costs raised by long sequences, we propose a novel Bidirectional Contrastive Rhythmic Variational Auto-Encoder (BiCoR-VAE) that extracts the modality-aligned latent representations for both motion and music inputs. Subsequently, leveraging the aligned latent spaces, we introduce a multi-modal diffusion Transformer model and a cross-guidance sampling strategy to enable various generation tasks, including cross-modal, multi-modal, and variable-length generation. Extensive experiments demonstrate that MoMu-Diffusion surpasses recent state-of-the-art methods both qualitatively and quantitatively, and can synthesize realistic, diverse, long-term, and beat-matched music or motion sequences. The generated motion-music samples are available at https://momu-diffusion.github.io/.

Main paper: 
 introduction: 
 Dancing to the musical beats or creating a variety of rhythmically synchronized music for a given motion is a fundamental aspect of human creativity. Music and human motions serve as universal languages that are shared by all civilizations, transcending cultural and geographical boundaries around the world [25] . For computational methodologies, the motion-music generation poses several challenges: 1) maintaining long-term coherence in typically lengthy motion-music sequences 2) ensuring temporal synchronization and rhythmic alignment between motion and music sequences, and 3) generating realistic, diverse, and variable-length human motions or music.Existing works usually divide the motion-music generation into two distinct tasks: motion-tomusic and music-to-motion. For motion-to-music, some methods compress the conditional video frames into a single image, in which the temporal information is lost [52, 53] . The state-of-theart work, LORIS [49] , employs a hierarchical conditional diffusion model to generate long-term musical waveforms. However, LORIS introduces huge computational costs and training difficulties since it generates long-term musical waveforms directly. For music-to-motion, the Dancing2Music (D2M) [26] framework divides the generation process into two stages: decomposing the dance into basic dancing movements with a VAE and compositing the basic movements into dance with a GAN. Nonetheless, D2M's approach of segmenting long-term music into short clips (approximately 1-2 seconds) diminishes the coherence of the synthesized motion sequences.Motivated by the fact that human motions are highly associated with music yet existing computational methods often study them in isolation, we propose a novel multi-modal framework, termed MoMu-Diffusion, to address the aforementioned challenges jointly. Firstly, to mitigate the computational costs and optimization complexities raised by long sequences, we employ a VAE to encode both motion and music sequences into latent spaces. Subsequently, to investigate the relationship between human movements and musical beats, we propose rhythmic contrastive learning. This approach involves constructing contrast pairs with a kinematic amplitude indicator, which quantifies the temporal variation in motion and is derived from the spatial motion directrogram differences as detailed in [4] . Given that the motion and music sequences are interactively aligned in the latent space to discern the correlation between kinematic shifts and musical rhythmic beats, we call our model as the Bidirectional Contrastive Rhythmic VAE (BiCoR-VAE).With the aligned latent space, we introduce a Transformer-based diffusion model that captures longterm dependencies and facilitates sequence generation across variable lengths. Additionally, we introduce a simple cross-guidance sampling strategy that integrates different cross-modal generation models, enabling multi-modal joint generation without extra training. By incorporating the BiCoR-VAE and the diffusion Transformer model, our MoMu-Diffusion framework effectively models the long-term motion-music synchronization and correspondence, enabling motion-to-music, musicto-motion, and joint motion-music generation. Moreover, MoMu-Diffusion supports generating motion-music samples in variable lengths. The pipeline of MoMu-Diffusion is illustrated in Figure 1 .We have conducted extensive experiments on three motion-to-music and two music-to-motion datasets, including scenarios such as dancing and competitive sports. The experimental results demonstrate that MoMu-Diffusion attains state-of-the-art performance across both objective and subjective metrics, significantly enhancing music/motion quality and cross-modal rhythmic/kinematic alignment. Furthermore, we have carried out abundant ablation studies to validate the efficacy of the BiCoR-VAE and the DiT architecture. A comparative analysis with state-of-the-art motion-tomusic methods CDCD [53] and LORIS [49] , 2D music-to-motion method D2M [26] , and general video-to-audio methods Diff-Foley [33] and MM-Diffusion [41] , is presented in Table 1 . 
related works: 
 Neural Motion Synthesis. Neural motion synthesis is often associated with audio, and we focus on two audio-driven scenarios: music-to-motion generation [12, 27, 35, 26] and co-speech gesture generation [48, 32, 50] . For music-to-motion, some methods [12, 27, 35] propose to retrieve the most related music for the given motion sequence. D2M [26] is a generative model that designs a "decomposition-to-composition" method to learn the movement units and generate music from the learned units. Besides, some methods [30, 54, 29, 1] investigate synthesizing 3D motions from music. For co-speech gesture generation, DiffGesture [50] is a state-of-the-art model with a diffusion transformer architecture and diffusion gesture stabilizer. We study the 2D music-to-motion problem and compare the proposed MoMu-Diffusion with DiffGesture and D2MNeural Music Synthesis. Neural music Synthesis aims to generate melodious music with generative neural networks. Various generative models have been successfully applied to music synthesis such as transformer-based autoregressive models [21, 39, 9] , VAE [2, 40, 6] , GAN [8, 24, 36] , and diffusion models [16, 34] . Some efforts have been made to video-to-music which focuses on the cross-modal temporal alignment. For example, Foley Music [13] and Audeo [43] utilize Musical Instrument Digital Interface (MIDI) representations to generate music in a non-regressive manner. D2M-GAN [52] and CDCD [53] generate video-related music by compressing the video frames into a single image, in which the temporal information is neglected. LORIS [49] proposes a hierarchical conditional diffusion model to generate long-term musical waveforms.Multi-Modal Contrastive Learning. Contrastive has been demonstrated effective in For example, Elizalde et al. [10] proposed Contrastive Language-Audio Pretraining (CLAP) to learn a unified latent representation for an audio or text input, facilitating the birth of text-to-audio models [31, 20] . For audio-visual generative tasks, DiffFoley [33] uses semantic and temporal contrastive learning to promote video-to-audio generation. In this paper, to improve the efficiency and generalization ability of our generative model, we propose the first motion-music pretraining model with a well-designed contrastive loss to learn beat synchronization and rhythm correspondence.3 Bidirectional Contrastive Rhythmic VAE (BiCoR-VAE) Music Variational AutoEncoder. Music is a structured and complex audio signal, composed of various elements such as melody, harmony, rhythm, and dynamics. Some works [13, 43] utilize Musical Instrument Digital Interface (MIDI) representations, which yield highly formulated results. However, processing long-term music directly from the raw waveform is computationally intensive and challenging [49] . To address this, we train a VAE on the mel-spectrogram derived from the music, coupled with a high-fidelity vocoder. Let u ∈ R Tu be a music input, where T u denotes the waveform length. We can extract the mel-spectrogram of the music input: a = M el(u) ∈ R Ca×Ta , where M el() is the pre-defined mel-spectrogram extraction function, C a is the channels, and T a ≪ T u is the frames.Then, an encoder is used to compress the mel-spectrogram into a latent:z a = E a (a) ∈ R Tza×d ,where T za is the downsampled music frames and d is the latent mel-spectrogram dimension. The encoded mel-spectrogram can be decoded by a decoder a ′ = D a (a), and subsequently the musical waveform can be obtained by a high-fidelity vocoder x ′ = V (a ′ ).Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes]Justification: We have discussed the limitations of our work in Appendix H.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
rhythmic contrastive learning: 
 Contrastive learning has proven effective for learning multi-modal representations, enhancing performance in downstream tasks [38] . In the context of temporal alignment, a recent work [33] introduces temporal contrast, which seeks to maximize the similarity of audio-visual pairs from the same time segment while minimizing the similarity of pairs from different segments. However, this paradigm faces limitations in long-term motion-music synthesis, as musical pieces typically correspond to numerous rhythmic beats. The random selection process for constructing negative samples risks capturing similar rhythmic sequences, which undermines the learning objective. To address it, we propose rhythmic contrastive learning, designed to align cross-modal temporal synchronization and rhythmic correspondence. Based on the motion and music VAEs, we can obtain the motion latent z m ∈ R Tzm×d , and music mel-spectrogram latent z a ∈ R Tza×d , respectively. To synchronize the motion and music, which are often sampled differently, we employ pre-processing techniques such as evenly dropping motion frames to match the number of music frames, ensuring that T zm = T za .In the domain of motion-guided music, the inherent irregularity of human movements, characterized by rapid and abrupt actions, can significantly influence rhythm. To synchronize these rhythmic patterns, we employ a kinematic amplitude indicator as a basis for constructing contrastive clips within each motion-music pair. Firstly, we extract the motion kinematic offsets [15] with the motion directogram [4] , a metric that quantifies the variation in motion. We denote F (r, j) as the first-order difference of j-th node in the 2D motion at temporal timestep r, and divide it into K bins based on their Euclidean angles with x-axis by tan -1 (y/x). Then, the 2D motion directogram D(r, θ) can be expressed as the aggregate of F (r, j) across each angular bin:EQUATIONThe indicator function 1 θ (ϕ) distributes the motion nodes into K angular bins. Then, the kinematic amplitude indicator is computed by summing the bin-wise directrogram difference in each angular column:EQUATIONwhere D(r, k) is the directogram volume at temporal timestep r and k-th bin. The kinematic amplitude value is normalized within the range of (0,1).With the kinematic amplitude indicator established, we proceed to prepare the temporal motionmusic clips for contrastive rhythmic learning. For each motion-music latent pair, we randomly sample N T motion-music clips and divide them into EQUATIONwhere r s and r e denote the start and end timesteps of the sampled clip, respectively, and P max denotes the max-pooling operation across the temporal dimension. Finally, based on the sampled motion-music clips {(c i a , c i m )} N C i=1 , the contrastive objective can be formulated as:EQUATION)In BiCoR-VAE, the goal is to learn two paired VAEs for motion and music inputs, with a focus on temporal and rhythmic alignment within the low-level latent space. However, the VAE's objective to preserve fine-grained details for accurate reconstruction often conflicts with contrastive rhythmic learning's aim to align latent representations across modalities. This presents a trade-off between representational fidelity and generative alignment, posing optimization challenges. To address it, we propose a two-stage training strategy: initially, we train the music VAE using both a VAE loss and a GAN loss to prevent over-smoothing of the mel-spectrogram; subsequently, we train the motion VAE with a VAE loss and the contrastive rhythmic loss, while keeping the music VAE's parameters fixed. The insight behind this strategy is that mel-spectrograms, with their rich and complex acoustic features, require a more intricate optimization process compared to motion VAE, which deals with a limited set of body joint data. An overview of BiCoR-VAE is illustrated in Figure 2 (a).4 Transformer-based Diffusion Model with Aligned BiCoR-VAE Diffusion Formulation. Recent works have revealed that the U-Net architecture is not essential for diffusion probabilistic modeling, and in fact, the transformer can achieve superior performance in text-to-image generation tasks [37, 11] . Additionally, the transformer architecture excels at capturing long-range dependencies within sequence data and offers flexibility for variable-length generation [45] . Inspired by these findings, we opt for a Transformer-based architecture for our motion-music generation framework. Concretely, our approach involves initially concatenating the noisy input with the embedded conditional inputs and the embedded diffusion timesteps along the temporal dimension. This fused input is then padded to match a specified maximum length and combined with positional embeddings prior to being processed by the DiT model. The DiT output is subsequently truncated to the original temporal length and mapped to the output latent space.To illustrate the diffusion process, let's consider the motion-to-music task. During the forward diffusion, the latent data is gradually perturbed towards a standard Gaussian distribution according to a pre-defined schedule α 1 , ..., α T , where T is the total diffusion timesteps andEQUATIONwhere z a (t) denotes the music latent at timestep t. Then, the training objectives of our DiT-based cross-modal generation models are defined:EQUATIONwhere ϵ ∈ N (0, 1) denotes the noise in diffusion procedure, θ a and θ m are the parameterized DiT denoisers for motion-to-music and music-to-motion generation, respectively.Conditional Generation. For the cross-modal generation such as motion-to-music and music-tomotion, we implement classifier-free guidance [5, 18] . This method adeptly combines conditional and unconditional scores to obtain a trade-off between quality and diversity. By interpreting the diffusion model output as a score function, the sampling procedure with classifier-free guidance of motion-to-music can be written as:EQUATIONwhere s > 1 denotes the classifier sampling scale to balance the diversity and quality of synthesized samples. The diffusion model with ∅ condition is achieved by randomly dropping z m and replacing it with an embedded "null" representation. Exchanging the latent inputs enables the sampling procedure for music-to-motion generation since we have built a modality-aligned latent space.Joint Generation with Cross Guidance. To accomplish multi-modal joint generation, we propose a cross-guidance sampling strategy. This approach leverages multiple 'expert' models and introduces a slight modification to the sampling procedure, rather than integrating multiple modalities into a single model. Let T be the total diffusion steps, ϵ θa be the trained motion-to-music denoising model, and ϵ θm be the trained music-to-motion denoising model, we perform unconditional generation before a defined diffusion step T c : EQUATIONµ θa (z a (t), t, ∅) = 1 √ α t (z a (t) - 1 -α t √ 1 -α t ϵ θa (z a (t), t, ∅)), σ 2 = 1 -α t-1 1 -α t (1 -α t ). (9)Floor Eq (8) and Eq (9) delineate the reverse process for motion-to-music generation within the timestep range T ≥ t > T c . The reverse process for music-to-motion generation can be similarly constructed.For reverse timesteps T c ≥ t > 0, we use the estimated clean motion/music latent to condition the generation process of music/motion with the classifier-free guidance defined in Eq (7) . Given that the diffusion model adopts a coarse-to-fine refinement in the reverse process, we conduct unconditional generation before T c and impose conditional generation with the cross-guidance strategy after T c , as the noise in the estimated clean latent is significantly reduced. Determining the value of T c appears to be quite challenging; however, our empirical findings indicate that the joint generation maintains robust performance across a broad range of values for T c (from 0.3T to 0.7T ). The diversity in joint generation is sustained by the unconditional process and classifier-free guidance. An overview of cross-modal generation and joint generation is shown in Figure 2 (b) and (c).Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [No] Justification: We cannot upload our code in the peer-reviewing process due to copyright constraints. We will open-source our code and data upon publication.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
motion-to-music generation: 
 Experimental Settings. We evaluate our method on the latest LORIS benchmark [49] , which contains 86.43 hours of video samples synchronized with music. This benchmark presents three demanding scenarios: AIST++ Dance [30] , Floor Exercise [42] , and Figure Skating [47, 46] . In our experiments, each dataset is randomly split with a 90%/5%/5% proportion for training, validation, and testing.For model evaluation, we use five metrics to measure the beat-matching between synthesized music and ground-truth music [49] : Beats Coverage Scores (BCS) and Beat Hit Scores (BHS), Coverage Standard Deviation (CSD), Hit Standard Deviation (CSD), and the F1 scores. Besides, we use the Fréchet Audio Distance (FAD) [22] and Diversity [26] scores to evaluate the quality of synthesized music. Since the quality of the Floor Exercise and the Figure Skating datasets are poor, we only conduct motion-to-music generation on them with a learnable motion encoder, whose architecture is derived from [51] . During sampling, we employ 50 DDIM sampling steps. More experimental settings are provided in Appendix B.Baselines. We compare our proposed method to existing advanced video-to-music baselines: 1) Foley Music [13] , a graph transformer framework with MIDI representations. 2) CMT [7] , a controllable music transformer model to learn the rhythmic consistency between video and music. 3) D2M-GAN [52] , a GAN-based model with vector quantized music representation. 4) CDCD [53] , a diffusion-based model with an additional conditional discrete contrastive diffusion loss. 5) LORIS [49] , a diffusion-based model with hierarchical conditional mechanism, yielding state-of-the-art performance on video-to-music synthesis. 4 . We can find the musical beats of synthesized music are aligned with the ground truth and the kinematic movements of the eference video.The FAD and Diversity results are shown in Figure 3 . In this comparison, we focus on LORIS, the current state-of-the-art method in motion-to-music generation. It is evident that MoMu-Diffusion consistently outperforms LORIS across these metrics, particularly in FAD scores. This superiority can be attributed to MoMu-Diffusion's architectural innovations for capturing long-term correspondence. Unlike text, music encompasses a richer sequence length due to its complex acoustic features, such as melody, rhythm, and driving beats. To address this, MoMu-Diffusion employs mel-spectrograms in place of raw waveforms, thereby mitigating sequence length. Additionally, the introduction of BiCoR-VAE facilitates modality alignment in latent spaces.Experimental Settings. We use two datasets: AIST++ Dance [30] and BHS Dance. About 71 hours BHS Dance videos are collected from [26] , which contains three dancing types: "Ballet", "Zumba", and "Hip-Hop". For model evaluation, we compute the beat-matching metrics between synthesized motion beats and the reference musical beats with the aforementioned five beat-matching metrics. To validate the quality of synthesized motion sequences, we use Fréchet Inception Distance (FID) [17] , Mean KL-Divergence (Mean KLD), and the Diveristy scores. The feature extractor is based on MotionBert [51] and trained with a classification task on the BHS Dance dataset. For the BHS Dance dataset, we exclude the BiCoR-VAE since this dataset only contains the paired audio MFCC features and motion sequences without raw audio. In the generation process, the settings of the diffusion transformer model are the same as motion-to-music. More details are provided in Appendix B.Baselines. We compare MoMu-Diffusion to two baselines: 1) D2M [26] , the state-of-the-art music-tomotion work with a two-stage movement unit-based model; 2) DiffGesture [50] , the state-of-the-art co-speech gesture generation work with a U-Net diffusion model. Dance Revolution [19] reports better performance on music-to-motion generation but is withdrawn by its authors. Main Results. The beat-matching results are detailed in Table 5 . An analysis of these results reveals that MoMu-Diffusion achieves superior scores across all evaluated tasks, outperforming the state-of-the-art music-to-motion method D2M and co-speech gesture generation method DiffGesture. This performance underscores the efficacy of our BiCoR-VAE in constructing an aligned latent space for cross-modal generation and the feed-forward diffusion model in capturing long-term correspondence. It should be noted that the metrics BCS (Beats Coverage Scores) and BHS (Beat Hit Scores) are defined differently in this context compared to motion-to-music scenarios. Specifically, BCS calculates the coverage score between the kinematic beats of the synthesized motions and the musical beats of the ground-truth music, rather than the kinematic beats of the ground-truth motions.The generation quality results are presented in Table 6 . It is observable that MoMu-Diffusion reports better FID, Mean KLD, and Diversity scores on both the AIST++ and BHS Dance datasets. It demonstrates that MoMu-Diffsuion can generate more realistic and high-quality motion sequences while maintaining the capability of diverse generations. We further present a qualitative example of music-to-motion beat-matching in Figure 5 . We can find the kinematic beats of synthesized motion are highly associated with the reference musical beats. Additionally, the generated dance exhibits a high degree of diversity, encompassing lateral movements, rotations, squats, and so on. User Study. We conducted a user study with 20 annotators on the AIST++ Dance dataset to evaluate the generation performance. For each method, 200 samples were generated, and 20 paired samples were randomly selected for each comparison group. Annotators were asked to respond on site: "Which dance/music is more realistic and matches the music/dance better?". The human evaluation results, shown in Figure 6 , indicate that our method outperforms SOTA approaches in both motion-to-music and music-to-motion generations. Notably, a preference drop is observed when BiCoR-VAE is not employed, highlighting the importance of an aligned latent space for cross-modal generation.Motion Encoding. For motion sequence encoding, we compare the spatial position-based method with the directional vector-based method, which learns the unit directional vectors of the given adjacency set, and reconstructs the human pose with the calculated mean bone lengths [48] . However, as shown in Music Encoding. For music encoding, we evaluated the spectrogram-based method against the raw waveform-based method. According to Table 7 (#2), the raw waveform-based method gains performance declines in both FAD and F1 metrics. This is attributed to the lengthy audio sequences introduced by the raw waveform, introducing difficulties for diffusion modeling training.Learning Techniques. In MoMu-Diffusion, there are two key learning techniques: rhythmic contrastive learning (RCL) and Feed-Forward Transformer (FFT). From Table 7 , we can observe that "Ours w/o RCL" gains a clear drop on the beat-matching metric F1 (#3) and "Ours w/o FFT" gains a drop on the synthesis quality metric FID/FAD (#4), respectively. "Ours w/o FFT" means we use a U-Net backbone for the diffusion model, which has been shown inferior to our FFT-based model in long sequence modeling. Equipped with both RCL and FFT, MoMu-Diffusion ensures both generation quality and cross-modal alignment.Joint Generation in Variable Length. MoMu-Diffusion supports multi-modal joint generation in variable lengths, facilitated by a "pad-and-truncate" strategy in the diffusion model and the proposed cross-guidance sampling. To validate this capability, 1000 samples with varying lengths (10-30 seconds) are generated using different Gaussian noise vectors. With a cross-guidance sampling timestep set to T c = 0.5T , Table 7 (#5, #6), we can find that for multi-modal joint generation, MoMu-Diffusion shows that MoMu-Diffusion achieves comparable performance to the conditional models with clean condition inputs and advanced performance on the joint generation scenario. More ablation studies are provided in Appendix D. 
conclusion: 
 In this paper, we propose MoMu-Diffusion, the first multi-modal framework designed to learn the long-term synchronization and correspondence between human motions and music. In MoMu-Diffusion, we have two key designs: bidirectional contrastive rhythmic VAE (BiCoR-VAE) for learning modality-aligned latent spaces and Transformer-based diffusion model for learning longterm dependencies. Through extensive experiments, we demonstrate MoMu-Diffusion's efficacy across motion-to-music, music-to-motion, and joint motion-music generations.Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: We have specified all the training and test details in Section 5, Appendix A and B.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes]Justification: We have suitably reported error bars in our main results. See the CSD and HSD metrics in Table 2 , 3, 4, 5 and the error bars in Figure 3, 6 .Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes]Justification: We have discussed the broader impacts of our work in Appendix H.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [Yes] Justification: We have described safeguards in Appendix H.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: We have properly listed the licenses of the resources we used in Appendix G.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA] Justification: We do not release new assets.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [Yes]Justification: We have included the full text of instructions given to the participants in Section 5.3.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [Yes] Justification: In Section 5.3, we have mentioned that we just conduct human evaluations to compare different synthesized samples. The synthesized samples only include human motions and musical audios, which have no potential risks.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.• Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: A Implementation Details of BiCoR-VAEFor BiCoR-VAE, we use an encoder-decoder architecture, in which the 1D convolutional network and spatial transformer are used. The input is downsampled by a Conv1d downsampling layer and then forwarded to the middle block, finally upsampled by a Conv1d upsampling layer. The detailed hyper-parameters of BiCoR-VAE are listed in Table 8 .We use a two-stage training strategy for BiCoR-VAE. Firstly, we train the mel-spectrogram VAE with three loss functions: reconstruction loss L recon , KL loss L KL and a GAN loss L GAN to prevent over-smoothed mel-spectrogram:EQUATIONwhere λ 1 is set to 1e-5 and λ 2 is set to 0.5. Note that the GAN loss contains two steps: it first updates the generator part (mel-spectrogram VAE) with L stage1 , and then updates the additional discriminator. After training mel-spectrogram VAE, we freeze it, and train the motion VAE with the proposed contrastive rhythmic learning loss defined in Eq (4) in stage 2:EQUATIONwhere λ 3 is set to 1e-5 and λ 4 is set to 1. For training BiCoR-VAE, we use the AdamW optimizer with a learning rate of 2e-4 and training epochs of 300. We use 8 NVIDIA 4090 GPUs and it takes about 12 hours to finish. To decode the mel-spectrogram into high-fidelity music, we use the BigvGAN model [28] pretrained on the AudioSet dataset [14] .For motion-to-music, we evaluate our method on the latest LORIS benchmark [49] , which contains 86.43 hours of video samples with paired music. This benchmark incorporates three challenging scenarios: dancing, floor exercise, and figure skating. For dancing, 1,881 25-second videos are collected from AIST++ [30] , a fine-annotated subset of the dancing dataset AIST [44] . For floor exercise, 1,950 25-second and 660 50-second videos are collected from the Ginegym dataset [42] . For figure skating, 8,585 25-second and 4,147 50-second videos are collected from the FisV [47] and FS1000 [46] datasets.For music-to-motion, we use two datasets: AIST++ Dance [30] and BHS Dance. About 71 hours BHS Dance videos are collected from [26] , which contains three dancing types: "Ballet", "Zumba", and "Hip-Hop". In our experiments, each dataset is randomly split with a 90%/5%/5% proportion for training, validation, and testing.Note that only the AIST++ Dance dataset is used for both motion-to-music and music-to-motion generations. This is because the Floor Exercise and Figure Skating datasets involved too heavy motion variation, which makes it hard for the pose extraction algorithm to extract the high-accuracy motion sequences. As for the BHS Dance dataset, it only provides the MFCC audio features without raw audio. Therefore, we can not conduct motion-to-music experiments on it.We use mel-spectrogram as audio feature representation, We first resample the audio to 16kHz. We use 80 filters with fft set to 1024 and hop length set to 256 while processing the mel spectrogram using Hann window with a window size of 1024. For human motions, OpenPose [3] is applied to extract 2D body keypoints, and can process a video at 60 fps. We use the pre-trained Body-25 model to extract 25 key points of the human body, but some key points are difficult to extract consistently and some are less relevant to actions. As implemented by [26] , we finally choose the 14 most relevant keypoints to represent the poses, i.e., nose, neck, left and right shoulders, elbows, wrists, hips, knees, and ankles. We interpolate the missing detected keypoints from the neighboring frames so that there are no missing keypoints in all extracted clips.For the denoising part, we use the Transformer backbone rather than the U-Net. The hyper-parameters of our FFT model are listed in Table 9 . The FFT diffusion model is trained by the AdamW optimizer [23] with a learning rate of 1.6e-5 and a lambda linear scheduler with a warmup step of 10000. We train the diffusion model with 200 epochs for each task. It takes about 2 days for 8 NVIDIA 4090 GPUs. For the Figure Skating dataset, it takes about 4 days since this dataset is large.To evaluate whether the synthesized music is aligned with the given motion, we use the improved Beats Coverage Scores (BCS) and Beat Hit Scores (BHS) to validate the rhythm correspondence and cross-modal alignment of synthesized music. The improved BCS and BHS are first proposed by [4, 26] , then used for rhythmic dance-to-music validation [53, 52] , and improved by [49] for long-term rhythmic music validation. Also, we report Coverage Standard Deviation(CSD) and Hit Standard Deviation(CSD) to evaluate the robustness of generative models. Finally, the F1 scores of improved BCS and BHS are also reported as an overall assessment. BCS and BHS are designed by computing matching degrees of the rhythm points from synthesized music and groundtruth music. Let N s be the rhythm point number of synthesized music, N t be the rhythm point number of ground-truth music, and N m be the number of matched rhythm points, the BCS is defined as BCS = N m /N s and the BHS is defined as BHS = N m /N t , respectively. However, these metrics are not suitable for long-term music evaluations since 1) the second-wise rhythm detection algorithm leads to an extremely sparse vector and 2) BHS can easily exceed 1 if the rhythm points of generated music are more than ground truth. Therefore we use an improved audio onset detection Algorithm 1: Pseudo code for cross-modal (motion-to-music) sampling. Input: The latent mel-spectrogram representation z a , latent motion representation z m , the pre-trained denoiser θ a for motion-to-music, and the decoder D a for mel-spectrogram. t ← T , z a (t) ← N (0, I) while t > 0 do z a (t) ← sample from p θa (z a (t -∆t)|z a (t), t, z m ) t ← t -∆t end return D a ( ẑa ) algorithm to avoid sparse rhythm vectors. Here is the Python code based on the Librosa library: librosa.onset.onset_detect(y=audio, sr=sampling_rate, wait=1, delta=0.2, pre_avg=3, post_avg=3, pre_max=3, post_max=3, units='time').For validating the quality of synthesized music, we use the Fréchet Audio Distance (FAD) and the Diveristy score. We use the pre-trained VGGish model from https://github.com/gudgud96/ frechet-audio-distance to compute the FAD scores. Based on the feature extractor VGGish, we compute the Diversity score by using the average feature distance for paired samples. Specifically, the Diversity score contains inter-diversity and intra-diversity. Inter-diversity is obtained by computing the average feature distance between 200 combinations of 50 pieces of music from different motions and the intra-diversity is obtained by computing the average feature distance between all combinations of 5 pieces of music from the same motion input.To evaluate whether the synthesized motion is aligned with the reference music, we also use these five beat-matching metrics. However, since the number of musical beats is always more than the number of kinematic beats in real-world products, we use the musical beats as the reference for evaluation, which is also consistent with previous works [26] . Concretely, Let N s be the kinematic point number of synthesized motion, N t be the rhythm point number of ground-truth music, and N m be the number of matched points, the BCS is defined as BCS = N m /N s and the BHS is defined as BHS = N m /N t , respectively. For kinematic beat extraction, we use the bin-wise directrogram difference (defined in Eq (2)) as the indicator [4] .For validating the quality of synthesized motion, we use the Fréchet Inception Distance (FID) and the Diveristy score. To compute the FID score, we follow the design of [26] and train a motion classifier on the BHS Dance dataset with three classification categories: "Ballet", "Zumba", and "Hip-Hop". The motion classifier consists of a MotionBert encoder [51] and a classification head. The motion classifier is trained by an Adam optimizer with a learning rate of 1e-4 and 100 epochs. Then, we use the trained MotionBert encoder as the feature extractor for computing the FID and Diversity scores. The definition of Diversity score here is the same as Appendix B.4.We provide the pseudo-codes of cross-modal generation and multi-modal joint generation in Algorithm 1 and 2, respectively. For the cross-modal generation, we take the motion-to-music as an example while the implementations of music-to-motion are symmetrical.We propose a simple cross-guidance sampling strategy to combine multiple cross-modal generative models for joint generation. In this process, there is a hyper-parameter T c that controls the modality fusion timestep. In Table 10 , we study five variants: T c begins from 0.9T to 0.1T with an interval of 0.2T . From these results, we can observe that employing the cross-guidance strategy in the early sampling steps is not feasible since the predicted latent representation contains too many noises. However, we can find that MoMu-Diffusion (T c = 0.7T ) obtains an acceptable performance, Algorithm 2: Pseudo code for multi-modal joint sampling. Input: The latent mel-spectrogram representation z a , latent motion representation z m , the pre-trained denoisers θ a and θ m , the decoders D a and D m , the cross-guidance scale γ, the pre-defined schedule α 1 , ..., α T and indicating that the denoising process is coarse-to-fine. Using fewer cross-guidance sampling steps (like T c = 0.1T ) can ensure the quality of generated samples, but the cross-modal alignment is omitted, leading to a low F1 score. Therefore, we use T c = 0.5T in our paper to trade off the sampling quality and cross-modal alignment in joint generation.EQUATIONSince our method predicts pose points, the deviation between points can cause abnormal length of human skeleton. Three sets of examples are shown in Figure 7 , with anomalous frames on the left and the corrected frames on the right.We first calculated the average bone length between each pair of keypoints in the dataset, after which we post-processed the predicted keypoints. We specify a threshold (which we specify as 1.3 times the mean bone length) and correct the predicted bone lengths to the mean bone length once they exceed the threshold, an approach that significantly enhances model generation. It is worth noticing that the post-processing can not fully address this issue but alleviate it.We provide more qualitative results in Figure 8 , 9, 10, and 11 In this paper, we use several open resources, including https://github.com/ OpenGVLab/LORIS(All Rights Reserved), https://github.com/NVlabs/Dancing2Music (NVIDIA Source Code License, 1-Way Commercial), https://github.com/gudgud96/ frechet-audio-distance (All Rights Reserved), https://github.com/Walter0807/ MotionBERT (All Rights Reserved), and https://github.com/Text-to-Audio/ Make-An-Audio (All Rights Reserved). We use these resources for research purpose only.Due to the limited motion-music data and computing resources, the scaling law of our model is not testified in a super large dataset. Besides, our model depends on some data pre-processing methods like mel-spectrogram extraction and keypoints extraction by OpenPose, which may lead to error accumulations.MoMu-Diifusion promotes both neural motion and music synthesis, so it may help expand any impact that generative systems have on the broader world like copyright conflicts. We will add constraints and licenses when open-resourcing our code and pre-trained models. Justification: The Abstract and Section 1 accurately reflects the paper's scope: motion-music generation and contributions: 1) BiCoR-VAE and FFT model for multi-modal generation, and 2) extensive experimental evaluations.• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 


Decision: All reviewers find the paper interesting and novel. They unanimously agree that the paper warrants an acceptance. There are a few minor but still valuable suggestions, and I highly recommend the authors polish the paper further to make it more complete.

Reviews: review: paper_summary: paper_summary: This paper proposes a framework that enables the generation of music from motion, motion from music, or both simultaneously while maintaining synchrony between the two. To achieve this, kinematic amplitude is extracted from the motion, and motion audio clip segments corresponding to different kinetic amplitudes are sampled. Contrastive loss is applied to the audio and motion embeddings of these sampled segments to ensure that the embeddings align with each kinetic amplitude. The resulting embeddings are then used for joint diffusion, enabling audio-to-motion, motion-to-audio, or joint generation. A method for applying cross guidance is also proposed. Quantitative evaluations were conducted based on the beat information of the generated music, as well as the FID and diversity of the generated motion. A user study demonstrated that the generated outputs were preferred over those from previous models.
strength_weakness: strength_weakness: The proposed methodology leverages the relationship between changes in motion and beats in music as the primary element of synchronization, with evaluations mainly focused on this aspect. However, there can be instances where the beatness in motion or music is not clear (especially during certain periods). A discussion on how to handle such cases, or considerations beyond beatness, would be beneficial for achieving more natural generation.
questions: questions: Suggestion: Section 3.2 is a crucial part that explains BiCoR-VAE, but the process of sampling clips was difficult to understand. Referring to Figure 2(a) made it easier. Including an example to explain this process would make it more comprehensible.
limitations: limitations: -
paper_summary: paper_summary: The paper propose MoMu-Diffusion, that generates both music-to-motion and motion-to-music videos. MoMu-Diffusion achieves SOTA results and is able to generate joint distribution of music and motion instead of one way.
strength_weakness: strength_weakness: 1. I might miss it, but that would be nice if there is an ablation study of the hidden size and layers of Diffusion Transformer.
questions: questions: 1. How long can you generate the video in the joint distribution generation setting? Is that possible to use DiffCollage (Zhang et al., 2023) for longer video generation?
limitations: limitations: 1. Maybe can explore longer video generations.
paper_summary: paper_summary: MoMu-Diffusion is a motion-music co-generation model with an aim of improved temporal synchronization of the generated motion and music sequence, based on two major components:
1. a bidirectional contrastive rhythmic VAE (BiCoR-VAE) that provides aligned latent space of the motion and music through joint training, and
2. a multi-modal diffusion transformer (DiT) the model the latent space with cross-guidance sampling.
strength_weakness: strength_weakness: I would like to see the qualitative samples of the baselines and the ablation models to the demo page as well to form the readers opinion on the claimed synchronization of this work. While the presented results in the manuscript looks convincing, having direct comparisons in the demo will be helpful for the readers to have a better idea of the improvements.
questions: questions: * (line 121-122): I think it would be helpful to add a specific example of the framerate differece between motion and music frames. Currently it might read like the motion framerate is always higher (by integer multiple) than the mel spectrogram framerate.
* Assuming that the framerate design is defined by the mel framerate (followed by the vocoder), is resampling the motion sequence to match the mel framerate trivial enough to assume that we have the time aligned motion and music frames in the pre-processing stage?
limitations: limitations: The authors have mentioned the limitations in this work, including limited computational and data budget to verify the idea to scaled up dataset that potentially include video clips that do not adhere to the assumptions made in the rhythmic contrastive objective.
paper_summary: paper_summary: The authors proposed a novel framework to address the motion-to-music and music-to-motion tasks. They leveraged aligned latent spaces between motion and music, a multi-modal diffusion transformer, and a cross-guidance sampling strategy. Experiments were conducted to demonstrate that their approach outperforms their counterpart models.
strength_weakness: strength_weakness: Please refer to the questions section.
questions: questions: - The authors discussed the architectures of the multi-modal VAE in Section 3.1. Could the authors provide more specific implementation details, including
  - Whether the motion vectors in shape $\mathbb{R}^{T_m\times J\times 2}$ are first of all flattened to $\mathbb{R}^{T_m\times(J\times 2)}$ for VAE? If not, how are they processed?
  - What exactly are $T_a$ and $T_u$ in Line 105 for the music VAE? Does this imply that the music VAE has a lower downsampling ratio than the motion VAE in the temporal dimension?
  - Is normalization needed for the autoencoder representations of the both modalities so that they can be further processed by the diffusion model?
- The authors went through how they constructed the motion-music latent clip for VAE training which incorporates the kinematic amplitude indicator as depicted in Eq. 2. Could the authors further elaborate on why incorporating this is important by e.g., showing a histogram of kinematic amplitude in the dataset and how it is sampled?
limitations: limitations: The authors have adequately address the limitations.

Comments: comment: Thank you for your rebuttal and clarifications from my questions. Documenting the specific example regarding the design of framerate would provide clearer understanding of this work.

I am retaining my score since my overall positive assessment has not changed. Thanks again for your rebuttal.
Thank you for your insightful comments. I will maintain my score.
Thank you for your experiments and insightful explanations. I maintain my score.