Title: MotionBind: Multi-Modal Human Motion Alignment for Retrieval, Recognition, and Generation
Venue: NeurIPS 2025
Accepted: yes
Keywords: human motion

multi-modal representation learning

cross-modal retrieval

action recognition

human motion synthesis
Abstract: Recent advances in multi-modal representation learning have led to unified embedding spaces that align modalities such as images, text, audio, and vision. However, human motion sequences, a modality that is fundamental for understanding dynamic human activities, remains largely unrepresented in these frameworks. Semantic understanding of actions requires multi-modal grounding: text conveys descriptive semantics, vision provides visual context, and audio provides environmental cues. To bridge this gap, we propose MotionBind, a novel architecture that extends the LanguageBind embedding space to incorporate human motion. MotionBind has two major components. The first one is a Multi-Scale Temporal Motion Transformer (MuTMoT) that maps motion sequences to semantically meaningful embeddings. Multimodal alignment is achieved via diverse cross-modal supervision, including motion-text pairs from HumanML3D and KIT-ML, motion-video pairs rendered from AMASS, and motion-video-audio triplets from AIST++. The second component is a Retrieval-Augmented Latent diffusion Model (REALM) that can generate motion sequences conditioned on many modalities. MotionBind achieves state-of-the-art or competitive performance across motion reconstruction, cross-modal retrieval, zero-shot action recognition, and text-to-motion generation benchmarks.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a novel approach to multi-modal human motion alignment by integrating human motion data with existing modalities like text, vision, and audio within a unified embedding space. The authors build on the LanguageBind framework to extend its capabilities to human motion, and design the Multi-Scale Temporal Motion Transformer (MuTMoT) to encode human motion sequences into semantically meaningful embeddings, and also a retrieval-augmented latent diffusion model (REALM) for motion synthesis.\n\nThe paper demonstrates that the proposed method outperforms existing models in tasks like motion reconstruction, cross-modal retrieval, zero-shot action recognition across multiple benchmark datasets.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**:\n\n1. The integration of human motion into a multi-modal framework, allowing joint reasoning across text, vision, audio, and motion, is a unique contribution to the field.\n\n2. The use of MuTMoT, a multi-scale transformer architecture, to capture motion dynamics at different temporal scales is a powerful method for encoding complex motion sequences. This, combined with the retrieval-augmented latent diffusion model (REALM), demonstrates the effectiveness of the approach.\n\n3. The paper evaluates the proposed methods across a diverse set of tasks and datasets, demonstrating the robustness of the model. The paper shows competitive performance across various benchmarks.\n\n4. The paper provides detailed descriptions of the model architectures and their components, including the MuTMoT encoder-decoder and the REALM generation model.\n\n**Weaknesses**:\n\n1. On text-to-motion generation on HumanML3D, the method does not deliver the best results in any of the metrics.\n\n2. The qualitative results in the supplementary PDF only present the results of the proposed method. A qualitative comparison with other methods should be provided in order to demonstrate the strength of the proposed method.\n\n3. While the model supports any-to-motion generation, the authors only quantitatively evaluate text-conditioned motion generation. A more comprehensive evaluation across all modalities (such as audio or video) would provide a more holistic assessment of the system's capabilities.\n\n4. The approach assumes that the encoders from LanguageBind are sufficiently expressive for all modalities, including human motion. If LanguageBind's representations do not fully capture the nuances of motion, it might hinder the effectiveness of motion retrieval and generation.\n\n5. The retrieval-augmented component of REALM depends on retrieving relevant motion sequences from a database. This could potentially be a bottleneck if the database is insufficient or out-of-distribution, leading to sub-optimal results. \n\n6. While the paper is generally clear, certain sections, particularly the mathematical formulations and model descriptions, could be simplified for better accessibility.\n\n7. The use of dataset-specific embeddings and heads seem to limit the generalizability instead of “ensuring” it, as the authors claim in line 176.\n\nQUESTIONS:\n1. In line 176 it says “This modular design ensures generalizability...” But I think the use of dataset-specific embedding modules and heads seems to limit generalizability instead of improving it. If we want to apply this method to a new, unseen dataset, wouldn’t that require re-training the embeddings and heads? I think that good generalizability means being able to generalize to a new dataset without having to re-train anything. Could the authors clarify more specifically on the generalizability of this method?\n\n2. In the abstract (line 8-9) it says: “...Multi-Scale Temporal Motion Transformer (MuTMoT), a transformer-based encoder that...”, but in the introduction (line 49) it says: “...Multi-Scale Temporal Motion Transformer (MuTMoT), a transformer-based encoder-decoder architecture designed to...” So, is it an encoder or an encoder-decoder? The authors should make the statements consistent.\n\n3. What does the \\mathbf{w} in Eq. (3) represent? How is it obtained?\n\n4. Are the MuTMoT part and the REALM part trained at the same time or separately?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses a clear gap by incorporating human motion into unified multi-modal embedding spaces, which until now have largely omitted motion data. The authors extend the LanguageBind framework to include 3D human motion, creating a shared semantic space spanning text, vision, audio, and motion. This is a notable innovation beyond prior works that aligned only motion and text. The proposed Multi-Scale Temporal Motion Transformerand the training on diverse paired datasets (language–motion, video–motion, audio–motion) enable joint semantic reasoning across all four modalities. Moreover, the introduction of REALM, a retrieval-augmented latent motion diffusion model, is an original step that allows “any-to-motion” generation – going beyond conventional text-to-motion synthesis by conditioning on arbitrary modality inputs  which is a novel capability in this domain. Overall, the idea of binding human motion into a general multi-modal embedding space and leveraging it for cross-modal retrieval/generation is fresh and significant in scope.\n\nSTRENGTHS AND WEAKNESSES:\n- The paper’s approach is technically sound and well-grounded. The MuTMoT architecture is a transformer-based encoder–decoder that hierarchically encodes motion sequences at multiple temporal scales and decodes them back to full motions. This design captures both fine-grained pose dynamics and high-level action semantics, and the decoder enables accurate reconstruction of motion from the learned latent space.\n- The authors conduct a comprehensive set of experiments covering motion reconstruction, cross-modal retrieval, zero-shot action recognition, and text-driven motion generation, evaluated on four datasets (HumanML3D, KIT-ML, AMASS, AIST++). The results are rigorous and highlight the method’s strengths.\n- The paper is written in a clear and organized manner, making it easy to understand.\n\nQUESTIONS:\nNone\n\nLIMITATIONS:\nNone", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a multi-modal human motion alignment framework. For motion modality, it introduces the MuTMoT module, a Transformer-based encoder that maps motion sequences into semantically rich embeddings. The paper then extends the LanguageBind embedding space to include human motion. Cross-modal alignment is achieved through contrastive loss applied to motion-text, motion-video, and motion-video-audio pairs. Furthermore, the paper proposes REALM, a retrieval-augmented latent diffusion model for motion synthesis. Experimental results demonstrate the effectiveness of the proposed approach.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\nThis work proposes an effective Multi-Modal Human Motion Alignment model that projects data from different modalities into a unified embedding space, achieving SOTA performance on retrieval tasks. The proposed REALM module effectively leverages information from multiple modalities to enhance the performance of motion synthesis.\n\nWeakness\n\n1.\tThe lack of failure case analysis is somewhat concerning. Could the authors provide insights or examples of failure cases?\n\n2.\tHow does REALM respond to input conditions that are unrelated to human motion, for example, an image of a blue sky or other motion-irrelevant modality data? What kind of motion sequences would it generate in such scenarios?\n\n3.\tThe model is trained on the AIST++ dataset, which associates audio with motion. As far as I know, AIST++ provides dance music audio. If the conditioning input is speech audio (such as the BEATv2[1] dataset), would REALM still be able to generate plausible motion sequences?\n\n[1] Liu, Haiyang, et al. \"EMAGE: Towards unified holistic co-speech gesture generation via expressive masked audio gesture modeling.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.\n\nQUESTIONS:\n1. Failure case analysis is currently missing from the paper. Could the authors provide a systematic analysis or categorization of common failure modes?\n\n2. The paper primarily uses the AIST++ dataset, which contains music-driven dance data. How does the model generalize to other audio domains, such as speech? Have the authors tested REALM on inputs from datasets like BEAT or EMAGE, which focus on co-speech gestures? If not, could the authors comment on potential domain gaps, and whether the model would need fine-tuning or architectural adaptation to handle such input?\n\nLIMITATIONS:\nThe authors addressed the limitations and potential negative societal impact of their work.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a unified framework for aligning human motion sequences with multiple modalities (text, video, audio) within a shared embedding space, alongside a novel generative pipeline for motion synthesis conditioned on arbitrary inputs. The key contributions are:\na) MuTMoT: a multi-scale temporal motion Transformer that hierarchically encodes and decodes 3D motion sequences;\nb) REALM: a retrieval-augmented latent diffusion model that utilizes learnable frame tokens and cross-modal conditioning to generate high-quality motion.\nThe model is evaluated across several tasks, including text-to-motion generation, motion retrieval, and zero-shot action recognition.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1.The paper proposes a modular and extensible architecture that combines multi-modal alignment, contrastive learning, and latent diffusion. The use of learnable frame-level tokens and time-aware modulation is a particularly notable design choice.\n2.The model achieves strong quantitative performance on text-to-motion benchmarks (e.g., HumanML3D), outperforming existing baselines in standard metrics.\n3.The supplementary ablation studies are reasonably thorough, covering most core components. \n4.The architecture appears broadly generalizable to other multi-modal backbones.\n\nWeaknesses:\n1.Despite claiming robust multi-modal alignment and generative capabilities, the method relies entirely on frozen, pretrained LanguageBind encoders for all non-motion modalities (text, audio, image, video). As a result, the framework lacks novel contributions toward modality-specific understanding. Moreover, only text-conditioned generation is quantitatively evaluated, while other modalities (audio, video, image) are not assessed in the main paper.\n﻿2.The supplementary generation videos exhibit noticeable artifacts, such as foot sliding and physically implausible transitions (e.g., in stands_up_from_a_laying.mp4, the subject appears to float unnaturally), which undermine the claimed motion quality.\n3.Although the model achieves strong retrieval performance, a significant portion of the improvement appears to stem from GPT-4o-based text paraphrasing augmentation. As shown in Sec. B.3, Table 3, removing this augmentation causes R@1 to drop from 69.56 to 62.74. This raises concerns that the architectural contributions alone may not fully account for the observed gains. Clarifying the role of this augmentation and evaluating performance without it would strengthen the claims.\n4.While training and inference are briefly described in the supplement, the paper lacks a clear, step-by-step explanation or diagram of the overall motion generation pipeline, which affects both clarity and reproducibility.\n\nQUESTIONS:\n1.Can you provide quantitative or user study results for motion generation from non-text modalities (e.g., video-to-motion, audio-to-motion)?\n2.Could you clarify the impact of frame-wise conditioning versus simpler global token conditioning through ablation?\n3.How are positive and negative samples selected?\n4.How sensitive is the model to the quality or relevance of the retrieved reference motions? And how are the candidate motion embeddings collected?\n\nLIMITATIONS:\n1. The training process is resource-intensive, requiring 8× RTX A5000 GPUs and ~5 days for REALM to converge, which may limit reproducibility and accessibility.\n2. During contrastive training, all non-corresponding modality-motion pairs appear to be treated as negative samples, without consideration for potential semantic similarity. This could penalize semantically related but unmatched pairs (false negatives), potentially degrading the embedding granularity and generalization ability.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}