MotionBind: Multi-Modal Human Motion Alignment for Retrieval, Recognition, and Generation
Authors: Kaleab A Kinfu
Rene Vidal
Venue: NeurIPS 2025 poster ()
Keywords: human motion
multi-modal representation learning
cross-modal retrieval
action recognition
human motion synthesis
Primary area: applications

Abstract: Recent advances in multi-modal representation learning have led to unified embedding spaces that align modalities such as images, text, audio, and vision. However, human motion sequences, a modality that is fundamental for understanding dynamic human activities, remains largely unrepresented in these frameworks. Semantic understanding of actions requires multi-modal grounding: text conveys descriptive semantics, vision provides visual context, and audio provides environmental cues. To bridge this gap, we propose MotionBind, a novel architecture that extends the LanguageBind embedding space to incorporate human motion. MotionBind has two major components. The first one is a Multi-Scale Temporal Motion Transformer (MuTMoT) that maps motion sequences to semantically meaningful embeddings. Multimodal alignment is achieved via diverse cross-modal supervision, including motion-text pairs from HumanML3D and KIT-ML, motion-video pairs rendered from AMASS, and motion-video-audio triplets from AIST++. The second component is a Retrieval-Augmented Latent diffusion Model (REALM) that can generate motion sequences conditioned on many modalities. MotionBind achieves state-of-the-art or competitive performance across motion reconstruction, cross-modal retrieval, zero-shot action recognition, and text-to-motion generation benchmarks.

MotionBind: Multi-Modal Human Motion Alignment for Retrieval, Recognition, and Generation
Kaleab A. Kinfu University of Pennsylvania Philadelphia, PA 19104, USA kinfuuupenn.en edu
René Vidal University of Pennsylvania Philadelphia, PA 19104, USA vi vidalr@upenn.en edu
Abstract
Recent advances in multi-modal representation learning have led to unified em- bedding spaces that align modalities such as images, text, audio, and vision. How- ever, human motion sequences, modality that is fundamental for understanding dynamic human activities, remains largely unrepresented in these frameworks. Semantic understanding of actions requires multi-modal grounding: text conveys descriptive semantics, vision provides visual context, and audio provides environ- mental cues. To bridge this gap, propose MotionBind, a novel architecture that extends the Bang embedding space to incorporate human motion. Motion- Bind has two major components. The first one is Multi-Scale Temporal Motion Transformer (MuTMoT) that maps motion sequences to semantically meaningful embeddings. Multimodal alignment is achieved via diverse cross-modal supervi- sion, including motion-text pairs from HumanML3D and KIT-ML, motion-video pairs rendered from AMASS, and motion-videe-aaude triplets from AIST++. The second component a Retrieval- -uggeeeeee Latent diffusion Model (REALM) that can generate motion sequences conditioned many modalities. MotionBind achieves state-of-the-art or competitive performance across motion reconstruction, cross-modal retrieval, zero-shot action recognition, and text-to-motion generation benchmarks. The code is available at: hmtp://gthhhh.ommmddaaaaaaoooann
Introduction
Recent advances in multi-modal representation learning have led to unified embedding spaces that bridge diverse data modalities. Vision- Vision-language models like CLIP [ and ALIGN [2] first demonstrated that aligning images with natural language enables powerful zero-shot visual recognition capabilitics. Building on this idea, more comprehensive models have emerged to simultancously bind multiple modalities. For example, ImageBind text, audio, depth, and inertial motion (IMU) data) into a shared embedding space. By training on naturally co-occurring data (e.g., images with captions or videos with audio), ImageBind [ can relate modalities that were never explicitly paired during training, exhibiting "emergent" cross-modal alignment. Similarly, LanguageBind (4) further generalizes this idea by using language as the central bind to align multiple modalities via contrastive learning, mapping all inputs into a common semantic space. This progress underscores the promise of joint embeddings that connect vision, language, audio, and more facilitate tasks ranging from cross- cross-modal retrieval zero-shot recognition.
K aligns six different modalities (images, video,
While prior work (5-8) has explored learning joint representations between human motion and language, human motion remains notably absent from ceneral-purpose multi-modal frameworks. Here, "human motion" refers the sequence of 3D human poses, which conveys rich information about human movements that are critical for understanding actions, behaviors, and intent. Integrating motion into these multi-modal embedding spaces is vital for grounding action semantics a broader
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Madon
Enteediing Tokes
gaoaae
goideo
gaidio
MuTMoT Decoder
Adapter
Mapter 6
plest
getoage
Spccito
Festuro-wis Modiiation
Modti-Modal Encoders
MuTMoT Encoder
MuTMoT Decoder
Desade Sadc os)
teved
Terppesi lisempping
Trandommr Erendirr
Vision
Motion
(a) Multi-modal representation learning
MuTMoT Encoder
Conditining Tokans
Refroenc Mation
Lavrobll Toooee
MulMotT Erooodrr
Latent Space
Retrieval and TerparatiCondiioningg
Difuston Proosos
M
Tex:
Temporal Downoumping
Transfamet Ercoder
MoTMor Decoder
Cooe Aletiin
Fastsme-s liwer Modiestoon
Tire- Coniinaes Orige Atneiion
Spediic Embrdddnn
Su
(c) MuTMoT architecture
(b) REALM architecture
Figure 1: Overview of our framework. (a) We align human motion with text, vision, and audio by projecting modelity-specffic embeddings (from frozen I LanguageRind) and motion embeddings (from MuTMoT) into shared space via lightweight adapters and contrastive learning. (b) REALM enables any-to-motion generation through a latent diffuusio in the MuTMoT space, using temporal conditioning where learnable frame tokens attend to conditioning input and reference motion via time-aware cross-attention. Together, these components form unified framework for cross cross-modal retrieval, recognition, and generation.
context. Effective understanding of human actions requires alignment between motion and other modalities: text conveys descriptions intent or style, vision (images or video) provides visual cues, and audio can indicate associated sounds rhythm (e.g.. in dance). Aligning motion with these modalities could enable rich semantic motion representations that would be useful for a more holistic understanding of human motion.
Beyond motion understanding, shared representation spaces have been shown support "any-to-any" generation, where a single model can generate data like images conditioned on a wide range of modalities [T] Extending this paradigm to motion would allow translating visual cues, musical rhythm, or textual instructions into plausible human motion sequences, opening the door to several applications, including embodied Al and virtual character animation. However, most prior work on motion generation has focused primarily on text-to-motion synthesis 12.55
In this work, we address these gaps by proposing MotionBind, which aligns human motion with multi-modal embeddings in unified space for Building on the LanguageBind framework, we extend its modality set to include human motion, enabling joint reasoning across text, vision, audio, and motion. More specifically, MotionBind makes the following contributions.
motion understanding and synthesis.
1. We introduce Multi-Scale Temporal Motion Transformer (MuTMoT)) encoder-decoder archi- tecture designed to encode human motion sequences into semantically meaningful embeddings aligned with other modalities. The MuTMoT encoder captures motion dynamics multiple tem- poral resolutions, producing compact representations that reflect both fine-grained pose transitions and high-level action semantics. The MuTMoT decoder reconstructs motion from the motion latent space, enabling both self-supervised learning and downstream generation.
2.
We extend LanguageBind with MuTMoT to unify human motion, text, vision, and audio within a shared embedding space that enables joint moda reasoning and retrieval. The model is trained on a diverse suite of cross-modal datasets, including HumanML.3D [16] and KIT-MI [7 for language alignment, AMASS 18 for visual alignment, and AIST++ [1 for joint visual-audio alignment.

---

3. We present a Auggeeeted Latent Diffusion Model (REALM) that can generate mo- tion sequences conditioned on many modalities by leveraging the multi-modal embeddings of MuTMoT's shared latent space. To improve realism and controllability, REALM retrieves semantically similar motion from large datahase and incorporates as reference signal during generation. The generated latent representation is then decoded by MuTMoT to produce the final motion sequence.
4. Extensive experiments on motion reconstruction, cross-modal retrieval, zero-shot action recogni- tion, and text-to-motion synthesis demonstrate that our methods achieve state-of-the-art perfor- mance across five benchmarks.
2 Related Work
Multi-Modal Representation Learning. Recent advances in multi-modal representation learning have led unified embedding spaces that align diverse modalities through large-scale contrastive training. Early efforts like CLIP [1] and ALIGN [2] established strong image-text alignment by training dual encoders hundreds of millions of (image, text) pairs, enabling zero-shot recognition and retrieval. Subsequent "foundation" models (2H22) expanded to richer visual inputs, including videos and depth maps. ImageBind 3 extended the paradigm to six modalities (vision, text, audio, depth, thermal, and IMU) by treating images as the central alignment modality. However, this image-centric strategy limits semantic grounding for modalities not naturally paired with vision. LanguageBind [4] addresses this by using frozen language encoder as the universal anchor and aligning all other modality encoders to language via contrastive learning, yielding improved zero-shot performance on tasks requiring semantic understanding. Despite these advances, human motion remains absent from current multimodal frameworks.
Languug- Representation Learning. Parallel to the advances in multi-modal models, there has been extensive research on learning joint representations of human motion and language. MotionCLIP [6] introduced a transformer-based motion autoencoder that maps motion sequences into the CLIP embedding space, leveraging both text and image supervision to inherit CLIP's semantic structure and enabling plausible motion generations for text prompts. TM2T (7] proposed a bidirectional framework that tokenizes motion into discrete units and employs a neural machine translation model to facilitate reciprocal generation between text and motion, promoting a shared latent space for both modalities. Building on these foundations, MotionGPT [23] and its successor MotionGPT-22 S integrated motion tokens into large language models, creating unified vocabulary that allows for diverse motion- language tasks, including generation and captioning, within a single framework. Despite these developments, existing models primarily focus on text-motion alignment and often overlook integration with other modalities like audio or vision.
Motion Retrieval and Action Recognition. Recent advancements in human motion understanding have emphasized the importance of embedding motion data into semantically rich latent spaces to facilitate tasks such as text-to-motic evolved from being a mere evaluation metric in text-to-motion synthesis a primary objective, with models like TMR (5) introducing contrastive learning to align motion and text embeddings effectively, By extending a generative model (TEMOS 13] TMR demonstrates that combining contrastive and reconstruction losses improves retrieval performance on benchmarks such as HumanML3D and KIT-ML. In parallel, action recognition has benefited from integrating language supervision, as seen in ActionCLIP [24], which models video-text alignment to enable zero-shot recognition without additional labeled data. Despite these advancements, existing approaches often focus on a specific pair of modalities and lack unified framework that encompasses multiple modalities.
retrieval and action recognition. Text-to-motion retrieval has
Text-to-Motion Synthesis. Text-to-motion synthesis has evolved from early GAN-based approaches, such as Tex(AAction (25), which generated short and simple actions from natural language, to more expressive and diverse generative models. Variational methods such as T2M 16 and TEMOS [B3 modeled probabilistic mappings from text to motion. Autoregressive models duced discrete motion representations through vector quantization and employed transformer-batt sequence models to generate coherent and compositional motion. More recently, diffusion models have emerged as state-of-the-a methods. For instance, MDM (6) and MotionDiffuse [27 adapted denoising diffusion models to motion synthesis, improving generation fidelity. ReMoDiffuse [15] introduced retrieval-augmented diffusion that incorporates similar motion samples during denoising, and MLD (28) utilized a latent diffusion model to improve efficiency. In our work, we train
8
12
14 2 intro-
3

---

retrieval-augiee augmented latent diffusion model on motion-tex: pairs but, through alignment with shared embedding space, we generalize motion synthesis inputs from text, image, audio, or video.
3 Methods
In this work, we present MotionBind, which learns a unified embedding space that aligns human motion with other modalities, specifically text, vision, and audio, to enable cross-modal retrieval, action recognition, and motion generation. In Section 3.1] we introduce a motion autoencoder architecture, MuTMoT, that encodes motion sequences into this shared space. In Section 3.2 discuss the multi-modal representation learning paradigm. In Section 3.3 we present REALM, retrieval-augee -aggmentedd latent diffusion model capable of synthesizing realistic motion sequences conditioned on any modality. Together, MuT'MoT and REALM form MotionBind, a framework that unifies multi-moda. alignment and generation, enabling applications such cross-modal retrieval, zero-shot action recognition, and any-to-motion synthesis from a shared semantic space. We present the overall architecture of approach in Figure
3.1 Multi-Scale Temporal Motion Transformer
In this section, we present our Multi-Scale Temporal Motion Transformer (MuTMoT), transformer- based hierarchical architecture designed to encode motion sequences into compact embeddings aligned with a shared multi-modal space, and decode them back for motion reconstruction or generation. As illustrated in Figure Ic it consists of a modular encoder-decoder design, supporting both representation learning and generation across several motion benchmarks.
3.1.1 Motion Encoding
Dateset-Specific Embedding. Since our approach integrates multiple hetcrogeneous datasets (AMASS 18), HumanML3D [66), KIT-ML [17], and AIST++ (1)), the number of joints in the pose representations varies across datasets. To handle these discrepancies, we use a dataset-specific embedding module that maps raw motion sequences into unified representation space. Given motion sequence X E R TCxx, where 7 denotes the sequence length and C.s is the dataset-specific shared latent space of dimension D. Next,
input dimension, we project the motion features prepend a learnable motion embedding token z E IR D to the motion sequence. To inject temporal order information, we add sinusoidal positional encodings E [T++1)xD to the token sequence.
E E
where X
z: X] E (T++)xx
Feature-wise Linear Modulation (FiLM). To further accommodate dataset- speusff- characteristics and improve generalization, we leverage Feature-wiss Linear Modulation (FiLM) (29). FiLM applies an affine transformation to feature activations, conditioning on the dataset source. Formally, given the embedding E, FiLM modulation is applied as:
E Vits
Bas,
where Titer Bac E TR. D are dataset-specific scaling and bias parameters learned during training, allowing the model to flexibly adapt to dataset-specific distributional differences without disrupting shared representations. The resulting embedding E is then used as the input to the Encoder blocks.
Hierarchical Encoding Blocks. The MuTMoT encoder is multi-layer transformer architecture designed to hierarchically encode motion sequences across multiple temporal scales, capturing both local and global dependencies. Given the motion embedding E, the encoder processes the sequence through series of encoder blocks consisting of transformer encoders interleaved with temporal downsampling operations. After each encoder block, all tokens except for the first (the learnable motion embedding token) are passed through a 1D convolutional downsampling layer with kernel 2, and padding p 1. This reduces the temporal resolution by a factor of 2.
size k- = 4, stride
R is stored, resulting in a stack of B motion embedding tokens across scales. At the end of the final block, these motion embedding tokens fused using learnable softmax-weighter average:
Motion Embedding. At each block b, the motion embedding token E[(
B
motion
Orgy
E
with a = softmax (w) E B
(3)
6m

---

This fused embedding z zmotion E IR" serves as the motion representation in the shared multimodal embedding space.
3.1.2 Motion Decoding
Hierarchical Decoding Blocks. The MuTMoT decoder is symmetric to the MuTMoT encoder, consisting of Decoder blocks that contain transformer encoders followed by temporal upsampling. Starting from a low-resolution latent sequence (either from the encoder or the latent diffusion model discussed in Section B.3). the decoder progressively restores temporal detail each block.
Dateset-Specific Head. After the last block, reature-wise linear modulation is again applied to -pteset-specific styles. Finally, the decoded latent sequence is passed through a datasct-specific head, implemented as lightweight convolutional stack, to map the shared hidden representation back to the original motion format. This modular design ensures generalizability and compatibility with varying motion representations across datasets.
3.2 Multi-Modal Representation Learning
Our goal is to learn a unified multi-modal embedding space that semantically aligns human motion data with textual, visual, and audio modalities via contrastive learning. As shown in Figure Ta build on the frozen multi-modal encoders from LanguageBind and extend the modality set by utilizing MuTMoT for motion encoding. To enable effective alignment between human motion and other modalities, without forcing motion embeddings to conform directly to the frozen LanguageBind space, which may lack the capacity to fully capture motion-specific semantics, adapters on both sides. Specifically, given paired motion sequences {X,) and corresponding non-motion inputs (Y - where each Y; may be text, images, audio or video, we compute motion and non-motion embeddings as:
introduce lightweight
z.motion = fMM (eMM(M)(X)))
Z.onmot
(eB(LLB(Y))))
(4)
where CMM and CLB denote the MuTMoT and LanguageBind encoders, and fMM and fLB lightweight adapters that project embeddings to a common semantic space.
To align the modalities, we use a contrastive InfoNCE loss 330 encouraging embeddings from corre- sponding (positive) pairs to closer the embedding space while pushing apart non-corresponding (negative) pairs. The contrastive loss LCL for given pair (zmotion 2 gnonmott is defined as:
exp(s:/77)
CCL log exp($t/T) + EjEN; Waj exp((>i) - mis)/T)
(5)
where sij and s7 denote the cosine similarities of positive and negative embedding pairs, respectively; T is a temperature hyperparameter, Av denotes the set of negative pairs for sample i, Wij is a motion- length-aware weighting term, and m.j is dynamic semantic margin. The weighting factor Wij accounts for differences in motion duration between samples. For a normalized length difference Aij. set
1 1 1,
if >5
66)
otherwise,
where À and 8 are hyperparameters controlling the strength and threshold of the length-based penalty, respectively. This helps the model better discriminate between motions differing significantly in temporal dynamics, improving the semantic granularity of the embedding space.
In addition temporal penalties, we introduce semantic margin my to modulate the influence of negative pairs based on the similarity between their associated non-motion inputs. Specifically, the margin defined as:
1 cos (CLB((Y)) (Y,)), if cos (CLB(Y,)) (aLe(L(()))) <p 0,
maj
otherwise,
where p [0 is similarity threshold. If two inputs Y, and Y, are semantically dissimilar in the LanguageBind embedding space (i.e., below the threshold p), the corresponding motion-modality pair is penalized more heavily. This encourages the learned motion embeddings to respect the semantic distinctions already encoded in the pre-trained multi-modal space, improving cross- modal alignment quality and retrieval discriminability.
5

---

3.3 Retrieval-Augmented Latent Diffusion Model for Motion Generation
To generate realistic human motion sequences conditioned on multimodal inputs, we introduce Eetrieval-Augmenne Latent Diffusion Model (REALM). Unlike conventional motion diffusion mod- els that directly generate raw motion data [5] 26 27 REALM operates in the compact latent space defined by the MuTMor encoder. Specifically, given conditioning modality (text, video, or audio), REALM synthesizes motion by performing the diffusion process on latent motion representations, effectively leveraging the semantic coherence and expressivity of the MuT'MoT embedding space. An overview of the REALM architecture is shown in Figure 1b In what follows, we describe the overall latent diffusion framework and our relrieval-augmented temporal conditioning module.
3.3.1 Latent Diffusion Framework
Let X E IT and z = EMM (X) E BLXD denote, respectively, raw motion sequence and its corresponding latent embedding obtained from the MuTMoT encoder, where L = T/4 / is the temporally downsampled frame length. We adopt a standard denoising diffusion probabilistic model (DDPM (11 in this latent space. The forward process corrupts the latent Zg over S steps with Gaussian noise:
qZz 22-1)= = =N(zsi V 1,
(8)
5
a
zo) = II g(z, 2)-1); s=1
(9)
where B, is fixed noise schedule. The denoising model learns the reverse process:
(2 Mz, c) (2;-1;;; 11 C, C) Eo()))
(10)
where e the fused conditioning signal derived from a multi-modas condition ye and a retrieved reference motion YR (see Section 3.3.3). and is the current timestep. We use a parameterized
transformer as in 27 to learn via noise prediction objective.
3.3.2 Reference Motion Retrieval
To enhance the generation process, we incorporate a retrieval mechanism that selects semantically relevant motion sequences from a database. Given a condition embedding 2c E IR derived from the multimodal encoder, we first retrieve a of relevant motion sequences from large-scale motion database. Retrieval is performed by measuring cosine similarity between the query embedding and the embeddings of candidate motions previously encoded by MuTMoT and stored offline:
ZR
arg maxsim(Eee (M),
(11)
E.E
where M represents the set of candidate motion embeddings, and दR denotes the selected reference embeddings. These retrieved reference samples provide semantic and structural context for the diffusion model, enhancing realism and consistency during generation.
3.3.3 Temporal Conditioning with Learnable Frame Tokens
main contribution of REALM lies its temporal conditioning module, designed to inject frame- specific and mimestep-adaptive conditioning context into the diffusion process. This module introduces a set learnable latent tokens, one for each frame in the latent motion sequence, that dynamically attend to context throughout the denoising trajectory. Formally, for a latent motion sequence of length L, we define a set of L learnable temporal tokens yı. E L where each token yi, E R corresponds to the i-th frame. These tokens are initialized independently and updated at each denoising step 8 via a series of cross-attention layers applied to the conditioning context. Let the conditioning input be denoted by [ye; y R.. where Yc the embedding of the condition modality (e.g., text, video, or audio), and YR is the latent representation of a retrieved reference motion. At Cross-Attention (TCC)
each timestep s, the frame tokens y", are updated via Time-Condition block, resulting in temporally contextualized conditioning y1. E LRL
Each TCC block consists of a cross-attention module that infuses contextual information into the frame tokens, followed by feedforward network, both modulated by the current timestep embedding

---

((s) E RR. Prior to each transformation, both frame tokens and context tokens are normalized using adaptive layer normalization, which injects the timestep-specifiy modulation via learned affine transformation.
(d, d(s)) = >(() h p()) o(h)
+ B(s), hrss) MLP(
(12)
The update for frame tokens at timestep proceeds as follows:
yz = MHA (AdaLN(y))) 1.) AdaL)) AdaLN(
(13)
Y2 vi. + Dropout(y.).
(14)
8/.0 =YL + Dropout (MIP(AdAINISL
(15)
By stacking Lrec such blocks, the frame tokens evolve encode temporally localized semantic cues, adapting their attention to different aspects of the conditioning modality and the retrieved reference motion as the denoising progresses. This temporal condition mechanism allows REALM to dynamically integrate multi-modal semantics at each denoising step, resulting in motion generations that are both contextually grounded and temporally coherent.
3.3.4 Training and Inference
As mentioned above, we follow a standard denoising diffusion probabilistic model (DDPM (1I) objective in the MuTMoT latent space, predicting the noise added each timestep and minimizing the predicted and actual noise. To improve robustness and support flexible conditioning, we adopt the classifier-fis free guidance technique [32]. During training, we randomly drop 9% of the conditioning embeddings for the input modality and 9% of the retrieved motion reference independently. This encourages the model to learn generate plausible motion both with and without conditioning input or reference motion. The diffusion process during training uses 1,000 timesteps to allow fine- fine-grained and stable learning. However, at inference time, we accelerate sampling by reducing the number of diffusion steps to 50 using linear timestep schedule for efficient generation without compromising motion quality. We apply classifier-free guidance by running two parallel forward passes: one with full conditioning and reference motion, and one with null conditioning. We then interpolate the output using a guidance scale that controls the contribution of each output. This encourages the generation to be loyal to the conditioning input and reference motion and improves the fidelity of the generation motion as result. The final denoised latent representation is decoded by the MuTMoT decoder into full motion sequence. See Appendix A.2.2 for more details.
4 Experiments
Experimental Setup. We evaluate our models on four core tasks: motion reconstruction, cross-
modal retrieval, zero-shot action recognition, and text-to-motion synthesis. Training is conducted using four diverse human motion datasets: (1) AMASS [18] (5), a large-scale motion capture corpus rendered into synthetic video to create motion-vinn video pairs; (2) HumanML3D [16] and (3) KIT-ML [17], both of which provide paired motion and text deseriptions covering a wide range of everyday actions; and (4) AIST++ [19], dance-oriented dataset offering synchronized motion, video, and audio for tri-modal alignment. All four datasets are used to train the MuTMoT encoder-decoder. The REALM generation model is trained specifically on HumanMI.3D and KIT-MI..
Motion Reconstruction. We evaluate reconstruction quality using the Fréchet Inception Distance (FID) and the Mean Per Joint Position Error (MPJPE), which respectively measure the distributional similarity and the per-frame positional accuracy between reconstructed and ground truth motions. As shown in Ta MuTMoT achieves the lowest FID and MPJPE on both KIT-ML and HumanML3D test sets, outperforming prior autoencoding approaches such as M2DM, T2M-GPT, and MoMask, thereby validating the effectiveness of our multi-scale transformer architecture.
Cross-Modal Retrieval. We assess cross-modal retrieval performance on all datasets using Recall at top-k (R @ 1, R R@@ 2, R @3, R R@5) 3)) in both directions: modality -toy--to-mmaa to-motion and motion-to-modan Table shows that MuT outperforms previous models such as TEMOS [33 and TMR [5). achieving higher recall across all ranks. This demonstrates the quality of the shared embedding
7

---

Table 1: Reconstruction results KTT-ML and HumanMLSD test sets. Lower better for FID, MPJPE, and ACCL. indicates a 95% confidence interval. Bold indicates the best, underline indicates the second-best.
KIT-ML MPJPE 4 ACCL 4
HumanML3D
Method
FID 4
FID
MPJPE L
ACCL 1
ACTOR 58
0.341
65.3
7.0
MLD-1 28
0.2474.001
54.4
8.3
T2M-GPT
0.472
0.070#
58.0
MotionGPT 8
0.067#.001
55.8
7.5
M2DM [59]
0.41344099
0.0634.001
MoMask 12
0. 0.112
37.2
0.019
29.5
MuTMoT (Ours)
0.077#0001
23.55
10.8
0.031
25.6
4.7
space in capturing fine-grained semantic correspondence hetween motion and other modalities. For AMASS, which lacks official split, we constructed 70/30 train-test split for evaluation.
Zero-Shot and Few-Shot Action Recognition. We evaluate MuTMoT on BABEL-60 and BABEL- 120 (56) under zero-shot, few-shot, and supervised settings. In the zero-shot setting, action labels are converted into natural language descriptions using GPT-4 [57] and encoded by the text encoder, and motion samples are encoded by MuTMoT and compared with these text embeddings via cosine similarity. In the few-shot setting (N = 10), class prototypes are computed by averaging embeddings of 10 labeled motions per class, and predictions combine few-shot and zero-shot logits scores &-FS + (1 - E) ZS with Ç = 0.7. A supervised linear probe further trains a linear classifier on top of the frozen MuTMoT embeddings. In cases where motion sample has multiple ground-truth labels (as common in BABEL., the prediction is considered correct if any of these labels appears within the Top-1 Top-5 results. As shown in Table 3 MuTMMT (N 0) yields over 50% Top-5 accuracy in zero-shot classification, while the linear probe matches the performance of fully supervised state-of-the-ar method. This demonstrates robust language-motion alignment even for fine-grained or overlapping categories (e.g., walk vs. forward movement).
Text-to-Motion Generation. We evaluate the generative quality of REALM against tatee-of-the-eee art methods using R-Precision, FID, multimodality, and diversity. following HumanMML3D and KIT-ML protocols. As it can be seen in Table 4 MuTMoT achieves competitive performance across all metrics, while additionally supporting conditioning from non-text modalities such as audio and video.
We also present qualitative examples that illustrate the semantic fidelity, diversity, and modality flexibility of our model. Figure? 2 shows results for the text-to-motion synthesis task. The generated motions accurately capture the fine-grained semantics of input prompts, such as "a person climbs up a ladder" and "a man picks up an object using his left hand", demonstrating smooth transitions and coherent articulation consistent with natural human movement.
Due to page limits, we have included additional results, ablation studies, more qualitative results, and architectural and implementation details, in the Appendix.
Altogether, our experiments demonstrate that the proposed methods only perform strongly on standard motion-language benchmarks but also extend effectively to broader multi-modal settings, showing generalization in retrieval, recognition, and generation tasks.
5 Limitations
While our framework demonstrates strong performance across a range of benchmarks and introduces new capabilities for multi-modal motion generation, several limitations remain. First, although REALM supports any-to-motion generation by conditioning on embeddings from arbitrary modalities (e.g., text, image, video, audio), our quantitative evaluation is currently limited to text-conditioned generation. As such, while qualitative results indicate promising generalization, further work is needed to robustly assess generation quality across modalities. Second, our model relies on the assumption that modality-specine encoders from LanguageBind provide sufficiently expressive representations. This assumption may not hold cases where these encoders fail to capture the nuances of the input, potentially limiting the effectiveness of motion retrieval generation. Third, the performance

---

Table 2: Cross-modal retrieval results HumanML3D (Text), KIT-ML (Text), AMASS (Video), and + (Video+AAudio) test sets. Recall @k metrics are shown both inpul-to-motion and motion-to-input retrieval directions. Bold indicates the best, and underline indicates the second-best.
Inpur-to-Motion Retrieval
Motion-to -t-----ottt-tttt Retrieval
Method
R@:1
R@21
R@31
MedR,
R@011
R
R
R@st
MedR]
TEMOS
40.49
53.52
61.14
70.96
2.33
39.96
53.49
61.79
72.40
2.33
HomaamML.D
Guo cl.
16
52.48
71.05
80.65
89.66
1.39
52.00
71.21
81.11
89.87
1.38
TMR [
16
81.32
86.81
91.43
1.04
67.97
81.20
86.35
91.70
1.03
MuTMoT
69.56
85.00
90.45
94.86
1,00
70.65
86.92
91.42
96.26
1.00
TEMOS 13
43.88
58.25
67.00
74.00
2,06
41.88
55.88
65.62
75.25
2.25
Gno et al. 16
42.25
62.52
75.12
87.56
1.88
39.75
62.75
73.62
86.88
1.95
KIT-ML
TMR
49.25
69.75
78.25
87.88
1.50
50.12
62.12
76.88
88, 88
1.53
MuTMoT
51.36
70.44
79.56
88.83
1.43
50,59
69.62
80.25
89.37
1.47
AMASS
MuTMoT
97.16
99.61
99.77
99.84
1.00
96.96
99.27
99.75
99.89
1.00
AIST++
MuTMoT
70.00
87.50
90.00
95.00
1,00
62.50
82.50
90.00
95.00
1.15
Table 3: Top-1 and Top-S classification accuracy on BABEL-60 and BABEL-120 MuTMoT (N 0) denotes
zero-shot classification; MuTMoT (N 10) combines zero-shot alignment with 10-shot support via prototype- based classification. MuTMoT (BABEL) trained using linear classifier frozen embeddings. Bold indicates the best, and underline indicates the second-bes in the supervised setting.
Training
BABEL-60
BABEL-120
Method
Data
Top-1 f
Top-5
Top-1 t
Top-5 f
2s-AGCN 56
60
CE
BABEL
41.14
73.18
38.41
70.49
2s-AGCN 15, 60
Focal
BABEL
33.41
67.83
27.91
57.96
MotionCLIP 16
BABEL
40.90
57.71
MuTMoT (BABEL)
BABEL
42.30
72.40
40.10
70.70
MuTMoT (N = 10)
HumanML33
28.07
56.89
24.10
49.62
MuTMoT (N 0)
HumanML3D
21.24
53.48
20.01
51.10
of REALM is influenced by the quality and coverage of the retrieved reference motions. While retrieval improves realism and control, the absence of semantically relevant references, especially in diitr--dittriution scenarios, may lead to suboptimal or generic outputs. Finally, although we train four diverse motion datasets and evaluate on multiple tasks, broader evaluations diverse movement types and environmental context remains an important area for future exploration. We believe addressing these limitations will further enhance the generality, robustness, and applicability of multi-modal motion models in future works.
6 Conclusion
In this work, we presented MotionBind, unified framework for integrating human motion into multi- modal representation learning and generation. Our approach extends the LanguageBind embedding space incorporate human motion alongside text, images, audio and video. Central to our framework the Multi-Scale Temporal Motion Transformer (Mu'TMoT)). transformer- encoder-decoder architecture that captures fine-grained and long-range motion dynamics while enabling alignment with other modalities through lightweight adapters and contrastive learning. To support motion generation in this shared space, we introduced REALM, a retrieval-aus -ugmeetee latent diffusion model that operates in the latent space of MuTMoT, REALM conditions on multi-modal embeddings and semantically retrieved reference motions, and employs a novel temporal conditioning mechanism with learnalle frame-wise tokens to generate realistic, context-aware motion sequences. Extensive experiments across motion reconstruction, cross-modal retrieval, zero-shot action recognition, and text-to-motion synthesis demonstrated that our approach achieves state-of-the-an or competitive performance, while simultaneously enabling "emergent" capabilities such multi-modal retrieval and uny-to-motion generation.

---

Table 4: Text-to-motion generation evaluation the HumanML.3D and KIT-ML test indicates 95%
confidence interval. Bold indicates best, and underline indicates second-best.
Dataset
Method
R@11
R@21
R@31
FID
MM Dist +
MModality
TM2T 7
0.424 003
0.618 -005
0.729=.000
1,501 #.017
3,467 011
2.424
T2M
16
0,455 003
0.636 0003
0.736 2022
L0874.02:
3.347 ass
2.219#.074
MDM 26
16112-007
0.544
5.566
2.799#.072
MLD
28
0,481=-008
0.673
0.772
0.473
3.196
2.413#-079
HumanML.3
MotionDiffuse
27
0,491=
0.681 0.681
0.782
0.630
3.113
1.553
T2M-GPT 14
0.492 D03
0.679 2002
0,775 #.02
0,14] 4.005
3,121 009
1.831 1.831 #.008
MotionGPT 23
0.492 003
0.681 #000
0.778 #700
2232
3.096 008
2.008#t084
RcMoDiffuse
15
0,510= 905
1.698-.0m 0066
0.795 4. 004
0.103 4. 000
2.974 016
1.795
MoMask 12
0,521 D02
0,713
0.807#0012
0.045
2.958
1.241 1.241
REALM (Ours)
0,511 0,511
0.701
0.801
0.204
2,980
2.483
TM2T 7
0,280= BMG
0.463
0.587#.005
3.5
4.591
3,292++081
T2M [6
0,361
0.559
0,681 0.681#.007
3.022
3,488#.02
2.052#.107
MDM 28
0 0.396
0.497
9.191 022
1.907
MLD
0.390= D08
0.609 D0s
0.734#.007
0.404
3,204 U2T
2.192
KIT-ML
DifuueDiffuse
27
0,417=.004 004
0.621 1201
0,739#.004
1,954
2.958 2.958=.00 DOG
0,730
T2M-GPT 14
0,416 D06
0.627 2000
0.745#t055
0.514 c2s
3.007
1.570#t.000
ReMoDiffuse
15
0,427= 014
0.641
0.765 17654.0mm
0.155
Sgg$L.com
2.814#
1.239
MoMask
12
0,433 DUT
0.656
0.781 #.005
0.204
2.779#.022
1.131
REALM (Ours)
0.435
009
0.661 =.010
0.785 7785=.008
0.264 h.100
2.718 029
0.881 #.079
Figure 2: Text-to-motion generation results from six different prompts. The generated motions closely follow the semantics of the input descriptions.
(a) A person walks in cirele.
(b) A person does cartwheel.
(c) A person climbs up ladder.
(e) A man picks up an object using left hand, cleans it and puts it back.
(f) A person crosses his arms and squats down twice.
(d) Making left turn.
Acknowledgments and Disclosure of Funding
The authors thank Lingjie Liu, Zhiyang (Frank) Dou, Yuyan Ge, and Darshan Thaker for their valuable input and feedback. This research was supported by NSF grant 2124277, NIH grant ROINS135613, and University of Pennsylvania Startup Funds.The views and conclusions expressed in this work are those of the authors and do not necessarily reflect the official policies, either expressed or implied, of NSF, NIH, the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for governmental purposes, notwithstanding any copyright notice herein.
10

---

References
[1] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, Agarwal, Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever, "Learning transferable visual models from natural language supervision." in International Conference on Machine Learning, 2021.
[2] C. Jia, Yang, Y. Xia, Y.-T. Chen, Z. Parekh, H. Pham, Q. Le, Y.-H. Sung, Z. Li, and T. Duerig, "Scaling up visual and vision-language representation learning with noisy text supervision," in International Conference on Machine Learning, 2021.
[3] R. Girdhar, A. El-Nouby, Z. Liu, M. Singh, K. V. Alwala, A. Joulin, and Misra, "ImageBind: One embedding space to bind them all," in Conference on Computer Vision and Pullern Recognition, 2023.
[4] B. Zhu, B. Lin, M. Ning. Y. Yan, Cui, H. Wang. Y. Pang, W. Jiang. J. Zhang, Z. Li, W. Zhang. Z. Li, W. Liu, and L. Yuan, "LanguageBind: Extending video-language pretraining to n- modality by language-based semantic alignment," International Conference on Learning Representations, 2024.
[5] M. Petrovich, M. J. Black, and G. Varol, "TMR: Text-to-motion retrieval using contrastive 3D
human motion synthesis." " in International Conference on Computer Vision, 2023.
[6] G. Tevet, B. Gordon, A. Hertz, A. H. Bermano, and Cohen-Or, "MotionCLIP: Exposing human motion generation to CLIP space," in European Conference on Computer Vision, 2022.
[7] C. Guo, X. Zuo, S. Wang, and L. Cheng, "TM2T: Stochastic and tokenized modeling for the reciprocal generation of 3D human motions and texts," in European Conference on Computer Vision, 2022.
[8] Y. Wang, D. Huang, Y. Zhang, W. Ouyang, J. Jiao, X. Feng, Y. Zhou, P. Wan, S. Tang, and D. Xu, "MotionGPT-2: general-purpose languae model for motion generation and understanding," arXiv preprint arXiv:2410.21744 2024.
[9] Z. Tang, Z. Yang, C. 7hu, M. Zeng, and M. Bansal, "Any-to-any generation via composable diffusion," Advances in Neural Information Processing Systems, vol. 36, pp. 16 083-16 099, 2023.
[10] F. Bao, S. Nie, K. Xue, C. Li, S. Pu, Y. Wang, G. Yue, Y. Cao, H. Su, and J. Zhu, "One transformer fits all distributions in multi-modal diffusion at scale," in International Conference on Machine Learning. PMLR, 2023, pp. 1692-1717.
[11] X. Xu, Z. Wang, G. Zhang, K. Wang, and H. Shi, "Versatile diffusion: Text, images and variations all in diffusion model," International Conference on Computer Vision, 2023, pp. 7754-7765.
[12] C. Guo, Y. Mu, M. G. Javed, S. Wang, and L. Cheng, "MoMask: Generative masked modeling of 3D human motions," in Conference Computer Vision and Pattern Recognition, 2024.
[13] M. Petrovich, M. J. Black, and G. Varol, "TEMOS: Generating diverse human motions from textual descriptions," in European Conference on Computer Vision, 2022.
[14] J. Zhang, Y. Zhang, X. Cun, Y. Zhang. H. Zhao, H. Lu, X. Shen, and Y. Shan, "Generating human motion from textual descriptions with discrete representations," in Conference on Computer Vision and Pattern Recognition, 2023.
[15] M. Zhang, X. Guo, L. Pan, Z. Cai, F. Hong, H. Li, L. Yang, and Z. Liu, "ReMoDiffuse: eetrieval-augmentee motion diffusion model," in International Conference Computer Vision, 2023.
[16] C. Guo, Zou, X. Zuo, S. Wang, W. Ji, X. Li, and L. Cheng, "Generating diverse and natural 3D human motions from text," in Conference on Computer Vision and Pattern Recognition, 2022, pp. 5152-5161.
[17] M. Plappert, Mandery, and T. Asfour, "The KIT motion-language dataset," Big Data, vol. 4, no. 4, pp. 236-252, 2016.

---

[18] N. Mahmood, N. Ghorbani, N. F. Troje, G. Pons-Moll, and M. J. Black, "AMASS: Archive of motion capture as surface shapes," in International Conference on Computer Vision, Oct. 2019, pp. Jp.5442-5451
[19] R. Li, S. Yang, D. A. Ross, and A. Kanazawa, "Al choreographer: Music-conditioned 3D dance generation with AIST++," in International Conference on Computer Vision, 2021, pp. 13 13401-13412,
[20] L. Yuan, D. Chen, Y.-L. Chen, N. Codella, X. Dai, J. Gao, H. Hu, X. Huang, B. Li, Li al., "Florence: A new foundation model for computer vision," arXiv preprint arXiv2211..11432 2021.
[21] B. Xiao, H. Wu, W. Xu, X. Dai, H. Hu, Y. Lu, M. Zeng. C. Liu, and L. Yuan, "Florence-2: Advancing a unified representation for a variety of vision tasks," Conference on Computer Vision and Pattern Recognition, 2024.
[22] J.-B. Alayrac, J. Donahue, P. Luc, A. Miech, I. Barr, Y. Hasson, K. Lene, A. Mensch, K. Millican, M. Reynolds et al., "Flamingo: A visual language model for few-shot learning," " in Advances in Neural Information Processing Systems, 2022.
[23] B. Jiang, X. Chen, W. Liu, J. Yu, G. Yu, and T. Chen, "MotionGPT: Human motion as foreign language," " in Advances Neural Information Procexsing Systems, 2023.
[24] M. Wang, Xing, and Liu, "ActionCLIP: new paradigm for video action recognition," arXiv preprint arXiv:2109 000000 2021.
[25] H. Ahn, T. Ha, Y. Choi, H. Yoo, and S. Oh, "Text2Action: Generative adversarial synthesis from language action," in International Conference on Robotics and Automation. IEEE, 2018, pp. 5915-5920.
[26] G. Tevet, S. Raab, B. Gordon, Y. Shafir, D. Cohen-Or, and A. H. Bermano, "Human motion diffusion model," in International Conference on Learning Representations, 2023.
[27] M. Zhang, Z. Cai, L. Pan, Hong, X. Guo, L. Yang, and Z. Liu, "MotionDiffuse: Text-driven human motion generation with diffusion model," IEEE Transactions on Puttern Analysis and Machine Intelligence, 2024.
[28] X. Chen, B. Jiang, W. Liu, Z. Huang, B. Fu, T. Chen, and G. Yu, "Executing your commands via motion diffusion in latent space," in Conference Computer Vision and Pattern Recognition, 2023.
[29] E. Perez, Strub, H. De Vries, V. Dumoulin, and A. Courville, "FILM: Visual reasoning with a general conditioning layer," in AAAI Conference on Artificial Intelligence, vol. 32, no. 1, 2018.
[30] A. v. d. Oord, Y. Li, and O. Vinyals, "Representation learning with contrastive predictive coding," arXiv preprint arXiv: 8807.03748, 2018
[31] Ho, A. Jain, and P. Abbeel, "Denoising diffusion probabilistic models," arXiv preprint arXiv:2006.1 11339, 2020.
[32] J. Ho and T. Salimans, "Classifier-free diffusion guidance," arXiv preprint arXiv:2207.12599 2022.
[33] Advanced Computing Center for the Arts and Design, "ACCAD MoCap Dataset."
[34] Helm, N. F. Troje, M. Reiser, and J. Munzert, "Bewegungsanalyse getäuschter und nicht- getäuschter 7m-würfe im handball," in 47. Jahrestagung der Arbeitsgermeinschaff für Sportpsy- chologie, Freiburg, Germany, 2015.
[35] S. Ghorbani, K. Mahdaviani, A. Thaler, K. Kording, D. Cook, G. Blohm, and N. F. Troje, "MoVi: A large multipurpose motion and video dataset," arXiv preprint arXiv:2003.01 01888, 2020.
[36] N. F. Troje, "Decomposing biological motion: framework analysis and synthesis of human gait patterns," Journal of Vision, vol. 2, no. 5, pp. 2-2, Sep. 2002.
12

---

[37] Carnegie Mellon University, "CMU MoCap Dataset." T [Online]. Available:
http:
/mocap.cs.ccm.cdd
[38] A. Aristidou, A. Shamir, and Y. Chrysanthou, "Digital dance ethnography: Organizing large dance collections," Comput. Culf. Herit., vol. 12, no. 4, Nov. 2019.
[39] F. Bogo, J. Romero, G. Ponl-Moll, and M. J. Black, "Dynamic FAUST: Registering human bodies in motion," in Conference Computer Vision and Pattern Recognition, Jul. 2017.
[40] E. J. C. Ltd., "Eyes Japan MoCap Dataset." [Online]. Available: http://moccapdata.com
[41] O. Taheri, N. Ghorbani, M. J. Black, and D. Tzionas, "GRAB: A dataset of whole-body human grasping of objects," in European Conference on Computer Vision, 2020. [Online]. Available: https://rrabis.ue.mppa
[42] M. Müller, T. Röder, M. Clausen, Eberhardt, B. Krüger, and A. Weber, "Documentation mocap database hdm05," Universität Bonn, Tech. Rep. CG-2007-2, June 2007.
[43] A. Chatzitofis, L. Saroglou, P. Boutis, P. Drakoulis, N. Zioulis, S. Subramanyam, B. Kevelham, C. Charbonnier, P. Cesar, D. Zarpalas et al., "Human4d: A human-centric multimodal dataset for motions and immersive media," " IEEE Access, vol. pp. 176 241-176 262, 2020.
[44] S. Brahmbhatt, C. Ham, C. C. Kemp, and Hays, "ContacIDB: Analyzing and predicting grasp contact via thermal imaging," in Conference on Computer Vision and Pattern Recognition, 2019.
[45] L. Sigal, A. Balan, and M. J. Black, "HumanEva: Synchronized video and motion capture dataset and baseline algorithm for evaluation of articulated human motion," International Journal of Computer Vision, vol. 87, no. I, pp. 4-27, Mar. 2010.
[46] C. Mandery, O. Terlemez, M. Do, N. Vahrenkamp. and T. Asfour, "The KIT whole- body human motion database," International Conference on Advanced Robotics, 2015, pp. 329-336.
[47] F. Krebs, A. Meixner, I. Patzer, and T. Asfour, "The KIT bimanual manipulation dataset," in JEEE/RAS International Conference on Humanoid Robots (Humanoids). 2021, pp. 499-506.
[48] M. M. Loper, N. Mahmood, and J. Black, "MoSh: Motion and shape capture from sparse markers," ACM Transactions on Graphics, (Proc. SIGGRAPH Asia), vol. 33, no. 6, pp. 220:1- 220:13, Nov. 2014.
[49] S. Tripathi, L. Müller, C.-H. P. Huang, T. Omid, M. J. Black, and D. Tzionas, "3D human pose estimation via intuitive physics," in Conference on Computer Vision and Pattern Recognition, June 2023.
[50] Akhter and M. Black, "Pose-conditioned joint angle limits for human pose reconstruc- tion," in Conference Computer Vision and Pattern Recognition, Jun. 2015.
[51] S. F. University and N. U. of Singapore, "SFU Motion Capture Database." [Online]. Available: http://mocaa.ccssfuu.cu
[52] N. Ghorbani and M. J. Black, "SOMA: Solving optical marker-based mocap automatically." in International Conference on Computer Vision, Oct. 2021, pp. 11 117-11 126.
[53] L. Hoyet, K. Ryall, R. McDonnell, and C. O'Sullivan, "Sleight of hand: Perception of finger motion from reduced marker sets," in ACM SIGGRAPH Symposium on Interactive 3D Graphics and Games, 2012, pp. 79-86.
[54] M. Trumble, A. Gilbert, C. Malleson, A. Hilton, and J. Collomosse, "Total Capture: 3D human pose estimation fusing video and inertial sensors," in 2017 British Machine Vision Conference, 2017.
[55] Y. Li, V. Mollyn, K. Yuan, and P. Carrington, "WheelPoser: Sparse-IMU based body pose estimation for wheelchair users." 26th International ACM SIGACCESS Conference on Computers and Accessibilitty, 2024, pp. 1-17.
13

---

[56] A. R. Punnakkal, A. Chandrasekaran, N. Athanasiou, A. Ramirez, and M. J. Black, "BABEL: Bodies, action and behavior with english labels," in Conference on Computer Vision and Pattern Recognition, 2021, pp. 722-731.
[57] J. Achiam, S. Adler, S. Agarwal, L. Ahmad, 1. Akkaya, F. L. Aleman, D. Almeida, Altenschmidt, S. Altman, S. Anadkat et al., "GPT-4 technical report," arXiv preprint arXiv:2303.08774, 2023.
[58] M. Petrovich, M. J. Black, and G. Varol, "Action-conditioned 3D human motion synthesis with transformer VAE," in International Conference on Computer Vision, 2021, pp. 10 985-10995.
[59] H. Kong, K. Gong, D. Lian, M. B. Mi, and X. Wang. "Priority-centric human motion generation in discrete latent space," in International Conference on Computer Vision, 2023, pp. 14 806- 14 816.
[60] I.. Shi, Y. Zhang. J. Cheng, and H. Lu, "Adaptive spectral graph convolutional networks for skeleton- based action recognition," " arXiv preprint arXiv: 1801 2018.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract and introduction clearly state the main contributions.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The paper includes dedicated limitations section in Section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
15

---

The paper is primarily empirical and does not present formal theoretical results or proofs.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
The paper provides sufficient detail on the model architectures (MuTMoT and REALM), training procedures, datasets used, evaluation metrics, and experimental setup in Figure Section 3, and the Appendix.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
16

---

reproducible, they Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [No]
Justification:
The code is not currently provided, but we intend to release it upon acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https /nips
for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
The paper provides sufficient training and test details to support the reported results. Section 4 outlines the basic experimental setup, while additional details are provided in the appendix.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: The paper reports error bars (+) indicating 95% confidence intervals for key metrics such FID, R-Precision, and MM Dist across multiple benchmarks (see Tables and 3).
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
17

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
The paper provides details on computing resources, including hardware specifications and training configurations, in the appendix.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http://wwwppp.ccooooo....sccctca
Answer: [Yes]
Justification:
The research aligns with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This work is foundational research and therefore has no direct societal impact within the scope of this study.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
18

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The models and datasets used in this work do not pose high risks for misuse.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best. faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The original papers that produced code packages datasets used in this work are cited.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
19

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssit
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
20

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [Yes]
Justification: The paper describes, in the appendix, the use GPT-4 for paraphrasing textual descriptions from HumanML3D and KIT-ML during training, as well as for converting BABEL action categories into natural language descriptions for zero-shot action recognition.
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non non-standard components.
Please refer to our LLM policy for what should or should be described.
21

---

Appendix
A Architectural and Implementation Details
In this section, we provide comprehensive details on the architectural and implementation aspects of our proposed methods, Multi-Scale Temporal Motion Transformer (MuTMoT) and Retrieval- Augmented Latent Diffusion Model (REALM).
A.1 Dataset Preparation and Preprocessing
As outlined the main paper, we use four publicly available human motion datasets our experi- ments: AMASS [18], HumanML3D [16]. KIT-ML [17), and AIST+ [19]. These datasets vary in motion diversity, sequence length, joint representation formats, and available modality pairs:
AMASS is large-scale motion capture dataset aggregated from multiple sources. It is used to generate synthetic motion-video pairs by rendering 3D motion sequences RGB videos using the SMPL mesh.
HumanML3D and KIT-ML provide natural language descriptions paired with 3D human motion sequences, making them well-suited for text-to-motion synthesis and cross-modal retrieval tasks.
+IST+ consists of synchronized 3D motion, video, and audio data, primarily focused on dance sequences, enabling tri tri-modal learning across motion, vision, and music.
To ensure consistency across datasets, all motion data are standardized following the preprocessing pipeline HumanML3D. particular, we convert SMPL pose parameters to unified 22-joint skeleton representation, except for KIT-ML, which uses a 21-joint configuration. The root joint of each sequence centered at the origin to achieve translation invariance, and the motion normalized to maintain consistent orientation and scale. Temporal normalization is also applied: sequences longer than 196 frames (at 20 FPS) are segmented into fixed-length clips, while shorter sequences are zero-padded to ensure consistent input dimensionality across the training pipeline.
To augment the textual diversity and improve robustness in cross-modal training, we generate five paraphrases for each description in HumanML3D and KIT-ML using the GPT-4o model. These paraphrases are generated consistent style with the original annotations and enable improved generalization in text-to-motion synthesis and retrieval. The prompt used to generate paraphrases is as follows:
You are a skilled technical writer trained to generate diverse and natural paraphrases for human action descriptions. Given a sentence describing a human motion, output 5 distinct alternative phrasings that preserve the exact human action description but differ in wording and phrasing. Avoid redundancy or trivial rewordings. Make sure each paraphrase is fluent and consistent with the HumanML3D annotation style.
A.2 Model Architectures
A.2.1 Multi-Scale Temporal Motion Transformer (MuTMoT)
MuTMoT is a hierarchical transformer-based encoder-decoder designed to represent human motion sequences at multiple temporal scales. This section details its four primary modules: (i) Dataset- specific Embedding, (ii) MuTMoT Encoder, (iii) MuTMoT Decoder, and (iv) Dataset-specific Output Head.
Dataset-Specific Embedding. Given the diversity in input dimensionality across datasets, we standardize input motions into a uniform hidden dimension D 512. Specifically, an input motion sequence X E TR TxxxCu undergoes a dataset-specifie 1D convolution with kernel size 3, stride 1, and padding 1. Subsequently, a learnable motion embedding token z E RIxD prepended to the projected sequence, resulting in:
X
1
X
E E
(T+1)xD
where X Conv IXX).
(16)
22

---

To inject temporal order information, we add sinusoidal positional encodings P E ( to the token sequence. The resulting embedding then normalized with LayerNorm (LN) and regularized with dropout rate of 0.1: E (ro)) where x
(17)
To accommodate dataset-specific distributional differences without compromising shared representa- tional learning, Feature-wist Linear Modulation (FiLM) (29 is applied:
E
7de ? + Bds:
(18)
where Yds, Sds R are dataset-specifie scaling and bias parameters learned during training. The resulting embedding E is then used as the input to the MuTMoT encoder.
Motion Encoding. The MuTMoT encoder comprises B = 3 hierarchical transformer blocks, designed capture both fine-grained and long-range temporal dynamics. Each transformer block includes multi-head self-attention (MHSA) with 4 heads and feed-for rred-forward network (FFN) with an intermediate dimension of 1024, each wrapped by residual connections and LayerNorm and dropout regularization (with dropout rate 0. 1): Formally, layer b, the input E(5) E R( is updated as:
E (+++1) = FFN (LN2 (A))
+ A where A (b) = MHSA [NN
( + E (19)
where (b)) [ denotes embeddings at the b-th layer, and T3 represents the sequence length at scale b. After each transformer block, all tokens except for the first (the learnable motion embedding token) are passed through ID convolutional downsampling layer with kernel size 4, stride 2, and padding This reduces the temporal resolution by a factor of 2:
E (b+1) 1:7%
+ Conv ID (b+1)). 1:1h
(20)
The first token, a learnable motion embedding ((x) E R1ID, is preserved across all blocks to accumulate global sequence information. At the final block, motion embedding tokens from each scale are fused via learned softmax- softmmax-weighted aggregation:
B
Z Emotion
Σ b=1
E
where a = softmax RB
(21)
where w E B are learnable weights. The resulting compact embedding Zmotion represents the entire motion sequence within the unified multimodal embedding space.
Motion Decoding. The decoder reconstructs motion sequences from latent embeddings provided by either the encoder the latent diffusion model. Mirroring the encoder architecture, the decoder also contains B transformer blocks, each composed of MHSA and FFN sub-blocks with residual and normalization layers:
Z b++1) = FFN (LN, (A)
A (b) where A (b) = MHSA [LN1 (z())
+ Z
(22)
where ((0) E TRxD The decoder employs temporal upsampling between layers (except the final
layer) using nearest-neighbor interpolation followed by ID convolution (kernel size 3, padding 1), doubling the temporal resolution progressively at each stage until the original sequence length T is recovered:
Z (++1)
Conv ID (Upsample
( (2(tz)))
(23)
Dataset-Specific Head. After the decoder reconstructs the motion embeddings at full temporal resolution, a datasct-specifii FiLM layer and convolutional output head map embed the motion embeddings back to the original dataset-specific dimensions. The output head consists of three convolutional layers, each with kernel size 3, stride and padding 1, formally expressed as:
X Conv (Conv ID (ConvID (xx)) and X. ~ o (D) + 3
(24)
where Yass 8/2 E R are learned sataset- FiLM parameters. The resulting output X recon- structs the motion sequence in the appropriate format for the original dataset.
23

---

Training Objectives. MuTMoT is trained using hybrid objective that combines reconstruction loss and cross-modal contrastive loss to simultaneously enforce imput fidelity and semantic alignment. To ensure that the latent embeddings produced by MuTMM retain sufficient information to reconstruct the original motion sequence, we employ a dataset-specific reconstruction loss at the output of the decoder. Specifically, we use a Smooth L.1 loss (also known as Huber loss ([) between the reconstructed motion sequence X and the ground truth motion X. This loss is computed independently for each dataset d E 'D based on its skeletal structure, with the joint dimensionality set to 22 for Human AMASS, and AIST++ and 21 for KIT-MI.. The loss is applied element-wise across the joint dimensions and timesteps, and then averaged over the sequence. Formally, the reconstruction loss is defined as:
Ca L (X.. Xt,))
recon
(25)
T x
Cde
The overall reconstruction loss is the sum across all datasets:
Crecon Σ
C(d)
(26)
DED
To align motion embeddings with those of other modalities (e.g.text, audio, video), we adopt contrastive learning framework based on the InfoNCE loss. This objective encourages paired samples from different modalities to lie close in the shared embedding space, while unpaired samples are pushed apart. Specifically, for each motion input X,, we compute an embedding z z. R. via the MuTMoT encoder (-M(-), where D 512. This embedding is projected to higher-dimenssonal space 768 via a learned linear projection layer:
2 monion
W 2
motion
(27)
where W proj E 768 We then apply a lightweight residual-style adapter module modulate this embedding while preserving expressivity. The adapter is a two-layer MLP with ReLU non-linearity and dropout, followed by residual connection:
mation
fHM (Z,
motion
(28)
SHM 2
matiion)
i.mottion
We (Dropout(RellUU {z))),,
(29)
with WT, W₂ E R 7688 768 and a dropout probability of 0.1. An identical adapter architecture is used for the frozen LanguageBind modality-speciff embeddings, denoted a. s.moodality
We optimize the following contrastive loss over batch of N motion-modality pairs:
1
exp
LeL
log exp
(30)
+ EjENs Wiy exp
1i-mmm maj
where 5 and 8, denote the cosine similarities of positive and negative embedding pairs, respectively; 7 is temperature hyperparameter, N,, denotes the set of negative pairs for sample i. The weighting factor way and margin may are adaptive, modulated by motion length differences and inter-modality semantic similarity as detailed Section 3.2 of the main paper. our experiments, we set d = 0.5, A = 1.0, and 0.4. The total training objective is:
Liowa Crecon + CCL,
(31)
where ACL controls the relative weighting of the contrastive objective. In our experiments, we set ÀCL 1.0. This dual objective enforces that the latent space preserves input fidelity while semantically aligning the motion with other modalities.
Through these modules, MuTMoT effectively captures motion dynamics at multiple temporal scales, producing semantically expressive embeddings that integrate seamlessly into the unified multimodal embedding space.
24

---

A.2.2 Retrieval-Augmented Latent Diffusion Model (REALM)
REALM is composed of three key components: latent diffusion transformer, a retrieval mecha- nism for semantic grounding, and temporal conditioning module designed to adaptively inject conditioning signals at each denoising timestep.
Latent Diffusion Transformer. Let zo RLXD denote the initial motion latent representation, where is the sequence length in latent space and D 512 is the motion embedding dimension obtained from the MuTMoT encoder. The forward diffusion process is modeled as Markov chain:
(Zz 21-1) N V B,2,-11 8,I),
(32)
(33)
for I...., S, with linear noise schedule Bg [10-4, 0.02] and 1000 steps during training. The denoising model po(2,-1 Zs, v predicts the noise added at each step by minimizing the standard denoising score-matching objective with mean squared error:
Caift = Ez [1100 s, s - ell?]
(34)
where e (O, I), and yL denotes the timestep- attention mechanism. The denoising model is implemented as a 6-layer transformer with latent dimension = 512, feedforward layer width of 1024, 4 attention heads, and sinusoidal timestep embeddings. Dropout with probability of 0.1 is applied at each transformer block.
conditioning computed by the temporal
To enhance the semantic fidelity of generated motions, REALM adopts classifier-fres guidance (CFG) mechanism during inference. This technique enables the generation process to go towards more semantically meaningful outputs by interpolating between conditional and unconditional predictions.
Formally, at each denoising step 8, two parallel forward passes are performed through the denoising network Na: (i) conditional pass that uses the full condition context ý y (ii) unconditional pass with null conditioning (i.e., the conditioning tokens masked out). These are computed as follows:
Ecood = M4g (2, S,
Euncond = (Z,, 0).
(35)
The final noise prediction is obtained via classifier-free free guidance as a linear interpolation between the two estimates:
Eguided =
Euncond + w (Econd Euncond),
(36)
where w = 2.75 is the guidance scale used in our experiments determined via grid search.
Reference Motion Retrieval and Temporal Conditioning. For retrieval, select the top-k 2 most semantically similar motions from a large reference database using cosine similarity between conditioning embeddings (from LanguageBind) and precomputed MuTMoT motion embeddings. These retrieved sequences, along with their global latent representations, are included as reference signals for generation. To integrate conditioning information throughout the denoising process, use temporal conditioning module composed of learnable frame-wise tokens refined by a stack of TTcc = 3 Time- Conditoodd Cross-Attentes (TCC) blocks. Each TCC block employs 4-head attention, a feedforward network of width 1024 with SquaredReLU activation, and adaptive layer normalization modulated by timestep embeddings. Dropout with probability 0.1 is applied after both attention and MLP layers. The output of the TCC stack is used to condition each step of the diffusion model, enabling temporally localized semantic grounding from both the retrieved references and the conditioning modality.
A.3 Computational Resources and Training Configuration
All experiments were conducted using 8 NVIDIA RTX A5000 GPUs (each with 24 GB of memory) distributed across a single node. Model training and inference were implemented in PyTorch and MMCV, leveraging mixed-precision training (via PyTorch AMP) and distributed data parallelism using the NCCL backend for efficiency and scalability. Data loading was parallelized with prefetching, and all models were trained using synchronized batch normalization across GPUs.
We used the AdamW optimizer with cosine annealing and linear warmup for both MuTMoT and REALM. Gradient clipping with maximum norm of 1.0 was applied to stabilize training. For
25

---

MuTMoT, the model was trained for 20 epochs with a batch size of 384 on each GPU, distributed across GPUs with gradient accumulation to ensure stable optimization. REALM was trained for up to 50 epochs using a 1000-step diffusion schedule during training and a reduced 50-step step schedule during inference. Typical training time for MuTMoT is approximately hours, while REALM training takes roughly 5 days to converge. The main training hyperparameters are summarized in Table 5
Table 5: Training hyperparameters used for MuTMoT and REALM models.
Parameter
MuTMoT
REALM
Number of GPUs
8 x RTX A5000
8 x RTX A5000
Batch Size (Each)
384
160
Optimizer
Adam W 10-4 1 x 10-2
AdamW
Learning Rate
2 10-4
Weight Decay
1 x 10-2
LR Schedule
Cosine w/ Warmup 1.0 (global norm) 20
Cosine w/ Warmup 1.0 (global norm)
Gradient Clipping
Epochs
50
Diffusion Steps
1000 (train), 50 (test)
B
Additional Experiments and Ablation Studies
B.1
Evaluation Metries
To assess the effectiveness of our approach across different tasks, we employ a comprehensive set of evaluation metrics that measure semantic alignment, generation quality, motion reconstruction fidelity, and diversity. Below, we define the metrics used in our evaluations.
Fréchet Inception Distance (FID). FID measures the distributional difference between generated and real motion sequences. computes the Fréchet distance between the Gaussian distributions fitted to the features, extracted from the official pretrained motion encoders for HumanML3D and KIT-ML as in [16,, of generated and ground-trun truth motions:
FID
||
1r1122 + Tr E, -(C,E,))))
(37)
where (Mg: Egg) and (Mr; Ex,) denote the means and covariances of the generated and real motion fea- tures, respectively. Lower values indicate closer alignment between generated and real distributions.
Mean Per Joint Position Error (MPJPE). MPJPE evaluates the reconstruction accuracy of the predicted motion by computing the average L2 distance between predicted and ground-truth 3D joint positions over all frames and joints:
T
MPJPE
pred
8t t.j
(38)
T
where 7' is the sequence length and J is the number of joints.
Acceleration Consistency Loss (ACCL). ACCL evaluates temporal smoothness by computing the mean squared difference between predicted and ground-truth accelerations:
ACCL- = T 2
[xx+1
2x, + xx-1) pred - (x1+1 2x, + 1x
(39)
1=2
R- R-Precision (Top-K). R-Precision evaluates text-motion alignment. Given one motion sample and N text candidates (one ground-truth and - negatives), it computes whether the ground-truth text appears in the top-K closest matches:
N
1
R@K =
Σ
I GGT, Top- (rank(ranked:)],
(40)
N
1=1
26

---

where I is the indicator function and N is the number of queries. N = 32 is used in our experiments as in the standard evaluations [[6]. MedR reports the median position of the ground match in the retrieval ranking list. Lower is better, indicating the true match appears closer to the top.
Multimodal Distance (MM-Dist). MM-Dist quantifies the alignment between the embeddings of the conditioning modality (e.g., text) and the generated motion. It is computed as:
MM-Dist
2
text
motion
(41)
N
2
where zt0tt and zmmotion are the embeddings of the text and the generated motion extracted from the pretrained encoders in [16].
Multimodality (MModality). MModality evaluates the intra-conditioning diversity of generated motions, i.c., how much variation exists among motions generated from the same input (e.g.textual description). our text-to-motion setting, we randomly sample C conditioning texts, and for each text, randomly sample two sets of I motion sequences. Let {zmmriionn ..... ,.. motionn] } and {z iz ...... z miptinn } denote the extracted embeddings from the two sets conditioned on the c-th text. MModality computed as:
MModality C' ΣΣ
Ze
metion
Z c,
mootion
(42)
The embeddings are extracted using the pretrained encoders from 16 Higher values indicate greater diversity among outputs conditioned on the same input.
B.2 Ablation Studies
To evaluate the effectiveness of the proposed architectural choices and training objectives, we conduct series of ablation studies on both components of our framework: MuTMoT and REALM.
To better understand the contribution of individual components in the MuTMoT architecture, we conduct comprehensive set of ablation studies. We evaluate five key aspects: (i) the use of lightweight adapter modules, (ii) dataset-specific FiLM layers, (iii) soft-gated multi-scale fusion for motion encoding, (iv) variants of contrastive loss, and (v) the impact of augmenting textual annotations through paraphrasing. All experiments are performed on the HumanML33D dataset using standard cross- modal retrieval evaluation protocols. Performance is reported in terms of Recall @ K and Median Rank (MedR) for both text-to-motion and motion-to to-ten directions.
Impact of Textual Paraphrasing. To improve the diversity and generalization of text-based align- ment, our final model augments each original motion caption with five paraphrased variants generated using GPT-4o. Removing this augmentation leads to a notable performance drop, particularly in Recall @ and Recall@2, 2, indicating that paraphrased descriptions enrich the semantic representation and improve retrieval robustness.
Adapter vs. No Adapter. Removing both the motion and modality- side adapter modules (fIM- fNM) significantly reduces retrieval performance. This suggests that the adapters are essential in aligning heterogeneous embeddings across modalities while preserving modality-speciff semanties.
FiLM Modulation. The dataset-specific FiLM layers play a critical role in adapting the model to the distributional shifts among different datasets. Ablating FiLM results in consistent drop in retrieval scores, validating its importance in generalizing to multi-dataset training setups.
Multi-Scale Fusion Strategy. To capture motion dynamics across varying temporal scales, MuTMoT aggregates the motion embedding tokens across layers via a softmax softmax-weighted sum. When replaced with simple use of the final layer's embedding token, the performance decreases, confirming the benefit of hierarchical temporal fusion.
Contrastive Loss Variants. We also evaluate the effectiveness of our contrastive learning design. While the standard InfoNCE objective provides strong baseline, we observe measurable improve- ments by incorporating length-aware weighting and dynamic semantic margin. The combination of both yields the best results, demonstrating that these enhancements improve fine- fine-grained semantic alignment between motion and other modalities.
27

---

Table Ablation study MuTMoT design variants using text-lo-motion and motion-to-tex. retrieval on HumanML3D lest set. Results are reported Recall @K and Median Rank. All results are computed on the test
split.
Input-to- Retrieval
Motion-tu-Inge Retrieval
Model Variant
R@ 1
R@21
R@
R@51
MedR.
R@ 11
R@21
R@
R@5t
MedR.
Full MuTMoT
69.56
85.00
90.45
94,86
1.00
70.65
86.92
91.42
96.26
1.00
w/o Paraphrasing
62.74
79.82
86.42
92.10
1.04
63.25
79.56
86.57
92.28
1.05
w/o Adapters
62.03
77.67
84,92
91.91
1.08
60,46
76.27
83.50
90.02
1.11
FiLM
67.29
83.34
89.17
92.88
1.01
68.05
84.46
89.25
94.02
1.02
Last Emhedding Only
66.07
81.45
87.95
93.33
1.02
61.75
78.92
86.27
92.96
1.07
Table 7: Ablation study on contrastive loss variants using text-to-motion and motion-to-text retrieval on HumanML3D test set. Results reported Recall @K and Median Rank. All results are computed on the test split.
Text-to-Motion Retrieval
Mation-to- Text Retrieval
Contrastive Loss Type
R@ 11
R@21
R@31
R@5t
McdR.
R@ 11
R@21
R@31
R@51
McdR,
InfoNCE
67.23
82.96
88.59
93.75
1.03
66.18
82.92
88.56
94.18
1.03
length-aware
68. 14
84.29
89.97
94.78
1.01
67.92
84.10
90.64
95.87
1.01
margin only
68.48
83.61
90.08
94.58
1.02
68.14
83.69
90.02
94.99
1.01
Full MuTMoT
69.56
85.00
90.45
94.86
1.00
70.65
86.92
91.42
96.26
1.00
Tables 6 and summmrize the retrieval performance across all ablation settings. The consistent gap between the full MuTMoT model and its ablated variants highlights the complementary value of each architectural and training component.
B.2.1 REALM Ablations
To assess the contribution of key components in the Retrieval-Augmented Latent Diffusion Model (REALM), we perform ablation studies on the KIT-ML dataset using the standard text-to-motion synthesis benchmarks. Specifically, we evaluate the effects of (i) classifier- guidance (CFG) and temporal conditioning. (ii) conditioning granularity (frame-wise vs. global token), and (iii) reference motion conditioning (presence and quality). The results, shown in Tables and 10 are reported using @@ (top-K retrieval precision), Frechet Inception Distance (FID). Multimodal Distance (MM Dist), and Multimodality (MModality).
Conditioning and Guidance. We first analyze conditioning mechanisms and guidance strategies. Removing CFG slightly decreases retrieval performance (e.g.R@) @ drops from 43.5 to 42.4), the more notable degradation is in generation quality, with FID rising from 0.264 to 0.588. This indicates that CFG not only improves alignment with text prompts but also stabilizes generation and enhances realism. Next, compare Friee-Wise Conditioning (FWC) with Global Token Conditioning (GTC). FWC consistently improves all key metrics demonstrating that temporally aligned, fine- grained conditioning better preserves motion dynamics than a single global token.
Retrieval and Textual Conditioning. We then evaluate the contribution of semantic and structural conditioning. The inclusion of a reference motion retrieved from large-scale motion database is designed to provide semantically relevant structure during denoising. Removing this reference signal leads to slight degradation in performance across all retrieval and diversity metrics. R@ 1 drops from 43.5 to 41.9, and MM-Dist slightly increases, confirming the role of retrieval in improving semantic grounding and synthesis consistency.
To evaluate the importance of explicit textual conditioning, we ablate the text encoder and condition generation solely the retrieved reference motion along with its text encoding. This results in substantial performance drop across all metrics: R falls to 39.0, FID increases to 0.478, and MM-Dist rises sharply to 3.065. This highlights that while retrieval helps guide synthesis, semantic alignment to the text is essential for meaningful generation.
Reference Motion Quality. Finally, we evaluate REALM's sensitivity to reference motion quality by varying the retrieval pool size (top-k). Table 1O shows gradual performance degradation as less relevant references are included, yet REALM remains robust even without any reference motion (No
28

---

Ref: R@1 = 0.419, FID = 0.307). This demonstrates the model's robustness and the complementary benefit of retrieval when available.
Together, these results show that frame-wise conditioning and classifier-free guidance jointly improve realism and alignment, while retrieval conditioning contribute complementary roles in improving generation fidelity and semantic alignment, and that textual input remains a critical modality for grounding generated motion as expected.
Table Text-to to-motion generation evaluation on the KIT-MI test set. indicates a 95% confidence interval.
Method
R
R@21 f
R@31
FID
MM Dist
MModality
REALM
0.435
0.66]
0.785 008
0.264#.010
2.718 029
0.88
w/o CFG
0.4244.01 1.11
0.6481.010
0.7731.000 008
0.588-.030
2.7704.044
0.8951.121
Reference Motion
0.419
0.655 015
0.786t.0014
0,307=.011
2.707
0.876*.022
w/o Text Condition
0,390
0.62244.16
0,748 4.122
0,478 4266
3.065 4.00
3.0654.046
1.261 4.19
Table 9: Frame- Wise Conditioning (FWC) Global Token Conditioning (GTC) KIT-MI.
Method
R@1t
R@21
R@31
FID
MM Dist
MModality
REALM (w/ FWC)
0.435 .099
0.661 0.661#.010 4.010
0.7854.008
0.264 1.64=.010 010
2.718
0.881
REALM (w/ GTC)
0.405
0.626#.010
0.744#.005 .003
0.325 025
2.995
0.986#-218
Table 10: Impact of reference motion quality (retrieval pool size, top-k) on KIT-ML.
Top-k
R@
R@2
R@3
FID $
MM Dist 4
MModality 1
2
0.435 009
0.661 010
0.785 008
0.264#.010
2.718 +718 .29
0.881 0.9
20
0.419#.007
0.650).006 000
0.775 008
0.282#.035
2.762#.003
0.965
50
0,411 4.012
0.411
0.645
0.773
0.286
2.762
0.940
100
0.410#t.u12
0.639
0.772
0.291 0.291 4.011
2.771 +.03
1.080
No Ref
0.419=t.009
0.655#.015
0.786+.014
0.307#.011
2.707
0.876+.023
B.3
Qualitative Results
B.3.1
Comparison on HumanML3D.
Figure55preeent qualitative comparisons of the results of text-to-motion generation on HumanML3D for three challenging prompts (as in (23]). Compared to the ofte-of-the-an methods such as Re- MoDiffuse, T2M-GPT, and MotionGPT, REALM generates motions that more accurately follow the describe sequences of actions and maintain spatial and temporal consistency.
B.3.2 Any-to-Motion Generation Results
In Figure we present qualitative examples that highlights the generalization capability of our model by showcasing motion generation from diverse conditioning modalities, including images and videos, despite the model being trained only on text-motion pairs. This supports the effectiveness of our unified multimodal representation.
B.3.3 Failure Case Analysis
While REALM achieves high-quality and semantically aligned motion synthesis, certain artifacts re- main, reflecting broader challenges in generative motion modeling. Figure 4 illustrates representative examples. We categorize the most common failure modes as follows:
1. Floating Feet: feet hover above the ground plane, indicating insufficient ground-contact con- straints.
Abrupt Transitions: sudden pose or velocity changes without proper anticepationn/nteerllaaioo. often at motion boundaries.
29

---

Figure 3: Any-to-motion generation results. Each column shows example. Top input modality (image/video/audio) Bottom corresponding generated motion.
Boxer
Input: Image of person swimming
Input: Frame from dance video
Input: Audio snapshot (boxing)
Generated:
Swimming motion
Generated: The moonwalk dance
Generated: Boxing motion
Figure 4: Failure case examples. (a) Feet hover above the ground: (b) abrupt pose/velocity changes between segments; (c) frequrryyy joint oscillations; (d) unintended foot sliding; (c) rotations beyond biomechanical limits; (I) missing object contact grasp.
(a) Floating Feet
(b) Abrupt Transitions
(c) Velocity Jitter
(d) Foot Skating
(e) Anatomical Violations
(f) Lack of Object Interaction
Velocity Jitter: high-freug feghffreeueccy oscillations in distal joints (e.g., wrists, ankles).
4. Foot Skating: unintended sliding of feet during phases that should remain stationary.
5. Anatomical Violations: rotations beyond biomechanical limits, producing implausible poses.
6. Lack of Object Interaction: motions do not convincingly reflect interactions with implied/visible objects (c.g., grasp/contact).
These artifacts are unique REALM, but common across all motion synthesis models, stemming from the inherent difficulty of jointly modeling physical realism, temporal coherence, and semantic fidelity in unconstrained human motion.
30

---

Figure 5: Comparison of text-to-motion generation results on HumanML.3D. Red phrases highlight difficult
parts of the prompt.
"a person walks forward, then jogs back and forth across the floor before returning to their original position."
"a person walks forward, stops, turns directly to their right, then walks forward again."
"a person walks backwards, then turns around then walks backwards again."
Real
REALM
(Ours)
ReMo- Diffuse
Motion- GPT
RAD
MDM
T2M-GPT
31


Reviews:
[{"review_id": "Yxz5fNF3Jx", "reviewer": "Reviewer_Nskc", "summary": "This paper introduces a unified framework for aligning human motion sequences with multiple modalities (text, video, audio) within a shared embedding space, alongside a novel generative pipeline for motion synthesis conditioned on arbitrary inputs. The key contributions are:\na) MuTMoT: a multi-scale temporal motion Transformer that hierarchically encodes and decodes 3D motion sequences;\nb) REALM: a retrieval-augmented latent diffusion model that utilizes learnable frame tokens and cross-modal conditioning to generate high-quality motion.\nThe model is evaluated across several tasks, including text-to-motion generation, motion retrieval, and zero-shot action recognition.", "questions": "1.Can you provide quantitative or user study results for motion generation from non-text modalities (e.g., video-to-motion, audio-to-motion)?\n2.Could you clarify the impact of frame-wise conditioning versus simpler global token conditioning through ablation?\n3.How are positive and negative samples selected?\n4.How sensitive is the model to the quality or relevance of the retrieved reference motions? And how are the candidate motion embeddings collected?", "limitations": "1. The training process is resource-intensive, requiring 8× RTX A5000 GPUs and ~5 days for REALM to converge, which may limit reproducibility and accessibility.\n2. During contrastive training, all non-corresponding modality-motion pairs appear to be treated as negative samples, without consideration for potential semantic similarity. This could penalize semantically related but unmatched pairs (false negatives), potentially degrading the embedding granularity and generalization ability.", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1.The paper proposes a modular and extensible architecture that combines multi-modal alignment, contrastive learning, and latent diffusion. The use of learnable frame-level tokens and time-aware modulation is a particularly notable design choice.\n2.The model achieves strong quantitative performance on text-to-motion benchmarks (e.g., HumanML3D), outperforming existing baselines in standard metrics.\n3.The supplementary ablation studies are reasonably thorough, covering most core components. \n4.The architecture appears broadly generalizable to other multi-modal backbones.\n\nWeaknesses:\n1.Despite claiming robust multi-modal alignment and generative capabilities, the method relies entirely on frozen, pretrained LanguageBind encoders for all non-motion modalities (text, audio, image, video). As a result, the framework lacks novel contributions toward modality-specific understanding. Moreover, only text-conditioned generation is quantitatively evaluated, while other modalities (audio, video, image) are not assessed in the main paper.\n﻿2.The supplementary generation videos exhibit noticeable artifacts, such as foot sliding and physically implausible transitions (e.g., in stands_up_from_a_laying.mp4, the subject appears to float unnaturally), which undermine the claimed motion quality.\n3.Although the model achieves strong retrieval performance, a significant portion of the improvement appears to stem from GPT-4o-based text paraphrasing augmentation. As shown in Sec. B.3, Table 3, removing this augmentation causes R@1 to drop from 69.56 to 62.74. This raises concerns that the architectural contributions alone may not fully account for the observed gains. Clarifying the role of this augmentation and evaluating performance without it would strengthen the claims.\n4.While training and inference are briefly described in the supplement, the paper lacks a clear, step-by-step explanation or diagram of the overall motion generation pipeline, which affects both clarity and reproducibility.", "was_revised": true, "final_justification": "The rebuttal with detailed analysis and further ablation helps clarify several points of our initial review.\nWhile the quantitative results shows outstanding performance of REALM, their effect during practical implementation to generate motion (qualitative results), is still concerning.", "rebuttal": "We thank Reviewer Nskc for their thoughtful feedback and recognition of the novelty, strength of our quantitative results, and generalizability of our proposed framework. We appreciate the acknowledgment of our modular design choices and thorough ablation studies. The concerns raised primarily stem from misunderstandings or points needing clarification, which we address individually below, along with revisions planned for the final manuscript. We hope that the reviewer will reconsider their rating.\n\n**Response to Major Concerns:**\n\n> The method relies on frozen LanguageBind encoders for all non-motion modalities. The framework lacks novel contributions toward modality-specific understanding.\n   * We respectfully clarify that our method does not aim to advance modality-specific encoding (e.g., of text, video, or audio). **Instead, we address the challenge that existing large-scale, multi-modal pretrained models, such as LanguageBind, contain rich implicit knowledge about human activity due to their training on large-scale datasets with content rich in human activity, but are not directly compatible with human motion representations.**\n   * Currently, no sufficiently large-scale datasets exist with detailed human-motion annotations paired explicitly with other modalities (text, audio, images, video) at the scale required to retrain or substantially enhance encoders like LanguageBind. Given this limitation, retraining these encoders from scratch or fine-tuning them for motion tasks is impractical and data-limited.\n   * Therefore, our proposed method, introducing MuTMoT with lightweight adapters, is strategically designed as the optimal approach to exploit the latent knowledge within pretrained models like LanguageBind. Specifically, MuTMoT encodes detailed human-motion sequences and aligns them effectively to LanguageBind’s latent representations in a shared embedding space via these adapters.\n   * This design choice--recognized as **\"novel and unique contribution to the field\", \"technically sound and well-grounded,\" and \"effective\"** by other reviewers--enables us to explicitly leverage and make the implicit multi-modal understanding already embedded in LanguageBind usable for downstream motion tasks, achieving robust multi-modal alignment and strong downstream performance across a diverse range of motion-related tasks without requiring large-scale paired motion data.\n\n> Only text-conditioned generation is quantitatively evaluated.\n   * As acknowledged in the limitations section, our quantitative evaluation for motion generation focuses on the widely studied text-to-motion generation task, and this decision reflects the current state of the field, where standardized and robust benchmarks exist primarily for text-conditioned motion generation. While we recognize the importance of evaluating other modalities, there is currently a lack of well-established benchmarks for image- or video-conditioned motion generation beyond pose estimation. **That said, our framework is \"an original step,\" as noted by Reviewer cqJT, to support any-to-motion generation at inference time without additional training, thanks to its multi-modal alignment.**\n   * Thus, we argue that our work should not be viewed as limited in scope. **In addition to achieving state-of-the-art or highly competitive results on text-to-motion generation, we demonstrate strong performance on six additional benchmarks across motion reconstruction, cross-modal retrieval, and action recognition.** Our primary goal is to showcase the versatility and generalizability of our multi-modal framework across diverse tasks. We believe the breadth and rigor of our evaluation present a compelling case for the effectiveness of our approach. We agree that expanding quantitative evaluation to additional conditioning modalities is a valuable direction for future work, and we plan to explore this in subsequent research.\n\n> The supplementary generation videos exhibit noticeable artifacts.\n   * We acknowledge that some of the generated motions exhibit artifacts. Such issues are well-known limitations in current motion generation systems and reflect broader challenges in modeling fine-grained physical realism. **These artifacts represent an active area of research and ongoing improvement across the field.** In the final manuscript, we will include a systematic analysis of common failure modes and provide a comparative assessment against existing methods to contextualize these challenges more clearly. \n\n> Although the model achieves strong retrieval performance, removing text paraphrasing causes R@1 to drop from 69.56 to 62.74, as shown in Sec. B.3, Table 3.\n   * We agree that text paraphrasing augmentation contributes to improved retrieval performance by enhancing the model’s robustness to linguistic variation. However, we emphasize that MuTMoT’s architectural design also contributes meaningfully to the overall gains. **Specifically, the removal of core architectural components, such as adapters, leads to even larger performance drops (R@1 drops to 62.03). This indicates that while data augmentation via paraphrasing improves the model’s robustness, our framework’s architectural choices are equally crucial for achieving strong retrieval performance.** We believe that both the data and the model design play complementary roles, and we have included these ablations to transparently present their respective impacts.\n\n> While training and inference are briefly described in the supplement, the paper lacks a step-by-step explanation or diagram of the overall motion generation pipeline.\n   * While Figure 1 provides an overview of the REALM architecture and the motion generation pipeline, we agree that a more detailed, step-by-step depiction would improve both clarity and reproducibility. In the final manuscript, we will revise the figure to explicitly illustrate the full end-to-end motion generation process and include a corresponding textual walkthrough to clearly explain each stage.\n\n> During contrastive training, all non-corresponding modality-motion pairs appear to be treated as negative samples, without consideration for potential semantic similarity. This could penalize semantically related but unmatched pairs (false negatives), potentially degrading the embedding granularity and generalization ability.\n   * We respectfully disagree with the assumption that all non-corresponding modality-motion pairs are treated as negative samples. **As described in LL201-208, we introduce a dynamic semantic margin component into the InfoNCE loss, which modulates the contribution of negative pairs based on the semantic similarity between their associated non-motion inputs.** \nSpecifically, if two non-motion inputs are semantically dissimilar (below a threshold in the LanguageBind space), the corresponding pair is penalized more heavily. This mechanism helps mitigate the impact of false negatives and preserves embedding granularity and generalization as shown in the ablation experiment (Table 4).\n\n**Response to Additional Questions:**\n\n> Could you clarify the impact of frame-wise conditioning versus simpler global token conditioning through ablation?\n   * Thank you for the insightful question. We conducted an ablation to directly assess the impact of Frame-Wise Conditioning (FWC) versus Global Token Conditioning (GTC), and the results are summarized in the table below:\n\nTable 1: Impact of Frame-Wise Conditioning (FWC) vs. Global Token Conditioning (GTC). Evaluation on the KIT-ML test set.\n| Method                   | R@1 ↑        | R@2 ↑        | R@3 ↑        | FID ↓         | MM Dist ↓     | MModality ↑    |\n|--------------------------|--------------|--------------|--------------|----------------|----------------|----------------|\n| REALM (w/ FWC)           | 0.435 ± .009 | 0.661 ± .010 | 0.785 ± .008 | 0.264 ± .010   | 2.718 ± .029   | 0.881 ± .079   |\n| REALM (w/ GTC)           | 0.405 ± .004 | 0.626 ± .010 | 0.744 ± .005 | 0.325 ± .025  | 2.995 ± .030   | 0.986 ± .213   |\n\nThese results demonstrate that FWC provides consistent gains across all key metrics. Notably, FID improves by 18.8%, and R@1 increases by 3.0 points with FWC. This can be attributed to FWC’s ability to provide fine-grained, temporally aligned conditioning signals at every timestep, as opposed to a single global token that may lose important temporal cues.\n\n> How sensitive is the model to the quality of the reference motions?\n\n   * To assess this, we conducted an ablation varying the retrieval set size (top-$k$). We randomly sample two references from the top-$k$ most semantically similar motions in a reference database (training set of the benchmarks). Similarity is computed using cosine distance between text-conditioning embeddings and MuTMoT motion embeddings.\n\nTable 2: Impact of Reference Motion Quality\n| Top-K                   | R@1 ↑        | R@2 ↑        | R@3 ↑        | FID ↓         | MM Dist ↓     | MModality ↑    |\n|--------------------------|--------------|--------------|--------------|----------------|----------------|----------------|\n| 2           | 0.435 ± .009 | 0.661 ± .010 | 0.785 ± .008 | 0.264 ± .010   | 2.718 ± .029   | 0.881 ± .079   |\n| 20          | 0.419 ± .007 | 0.650 ± .006 | 0.775 ± .008 | 0.282 ± .035  | 2.762 ± .038  | 0.965 ± .220   |\n| 50          | 0.411 ± .012 | 0.645 ± .017 | 0.773 ± .006 | 0.286 ± .018  | 2.762 ± .035  | 0.940 ± .286   |\n| 100          | 0.410 ± .012 | 0.639 ± .005 | 0.772 ± .004 | 0.291 ± .011  | 2.771 ± .035  | 1.080 ± .124   |\n| No Ref    | 0.419 ± .009 | 0.655 ± .015 | 0.786 ± .014 | 0.307 ± .011  | 2.707 ± .023  | 0.876 ± .023   |\n\nThese results show that REALM performs consistently well across a range of top-$k$ values, with only gradual performance degradation as less relevant reference motions are included. Notably, even in the absence of reference motion (No Ref), the model still achieves competitive results, confirming the robustness of our generation pipeline."}, {"review_id": "ER99HEDAZh", "reviewer": "Reviewer_S428", "summary": "This paper introduces a multi-modal human motion alignment framework. For motion modality, it introduces the MuTMoT module, a Transformer-based encoder that maps motion sequences into semantically rich embeddings. The paper then extends the LanguageBind embedding space to include human motion. Cross-modal alignment is achieved through contrastive loss applied to motion-text, motion-video, and motion-video-audio pairs. Furthermore, the paper proposes REALM, a retrieval-augmented latent diffusion model for motion synthesis. Experimental results demonstrate the effectiveness of the proposed approach.", "questions": "1. Failure case analysis is currently missing from the paper. Could the authors provide a systematic analysis or categorization of common failure modes?\n\n2. The paper primarily uses the AIST++ dataset, which contains music-driven dance data. How does the model generalize to other audio domains, such as speech? Have the authors tested REALM on inputs from datasets like BEAT or EMAGE, which focus on co-speech gestures? If not, could the authors comment on potential domain gaps, and whether the model would need fine-tuning or architectural adaptation to handle such input?", "limitations": "The authors addressed the limitations and potential negative societal impact of their work.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths\n\nThis work proposes an effective Multi-Modal Human Motion Alignment model that projects data from different modalities into a unified embedding space, achieving SOTA performance on retrieval tasks. The proposed REALM module effectively leverages information from multiple modalities to enhance the performance of motion synthesis.\n\nWeakness\n\n1.\tThe lack of failure case analysis is somewhat concerning. Could the authors provide insights or examples of failure cases?\n\n2.\tHow does REALM respond to input conditions that are unrelated to human motion, for example, an image of a blue sky or other motion-irrelevant modality data? What kind of motion sequences would it generate in such scenarios?\n\n3.\tThe model is trained on the AIST++ dataset, which associates audio with motion. As far as I know, AIST++ provides dance music audio. If the conditioning input is speech audio (such as the BEATv2[1] dataset), would REALM still be able to generate plausible motion sequences?\n\n[1] Liu, Haiyang, et al. \"EMAGE: Towards unified holistic co-speech gesture generation via expressive masked audio gesture modeling.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.", "was_revised": true, "final_justification": "My questions have been solved.", "rebuttal": "We thank Reviewer S428 for the thoughtful and constructive comments. We appreciate that they recognize our proposed method as effective, acknowledge the state-of-the-art performance on multi-modal retrieval, and highlight the benefits of the proposed REALM module for multi-modal motion synthesis.\nBelow, we address each concern and will incorporate these valuable suggestions in the final manuscript.\n\n**Response to Major Concerns:**\n\n> Failure case analysis is currently missing from the paper. Could the authors provide a systematic analysis or categorization of common failure modes?\n   * Thank you for the valuable suggestion. We fully agree that analyzing failure cases is essential for understanding the limitations of generative motion models. Due to format constraints in the rebuttal, we are unable to provide an extensive qualitative breakdown here. However, based on the video examples included in the supplementary material, we have systematically categorized the most common failure modes as follows:\n     1. *Floating Feet*: Instances where the feet hover above the ground plane, breaking physical realism. For example, in `stands_up_from_a_laying.mp4`, the character’s feet appear to hover during the standing motion, indicating insufficient ground-contact constraints, as also observed by Reviewer Nskc.\n     2. *Abrupt Transitions*: Sudden changes in pose or velocity without appropriate anticipation or interpolation. An example, although subtle, can be found in `a_person_steps_forward_while.mp4` (00:00:23 - 00:00:25), where the transition of the arms is unnaturally sharp.\n     3. *Velocity Jitter*: High-frequency trembling or shaking in joint trajectories, particularly at the extremities. This is visible in `a_man_swings_with_his_right_arm.mp4`, where the arm visibly wobbles near the peak of the swing.\n     4. *Foot Skating*: Feet slide or drift across the floor during periods when they should remain stationary. This is observed in `a_person_walks_backwards_then.mp4` (00:01:25 - 00:02:06), where the supporting foot does not stay planted.\n     5. *Anatomical Violations*: Body parts rotate or contort beyond biomechanical limits, resulting in implausible postures. For instance, in `walks_back-and-forth.mp4` (00:03:03 - 00:04:12), the legs and torso exhibit rotations that are not physically plausible for a human.\n     6. *Lack of Object Interaction*: The generated motion fails to convincingly reflect interaction with an implied object. In `a_man_picks_up_object_from_the_ground.mp4`, the hand does not make realistic contact with the imagined object or perform a grasping motion, in part due to limited modeling of fine-grained hand articulation in SMPL.\n\n   * We would like to clarify that these issues are well-known limitations in current motion generation systems and reflect broader challenges in modeling fine-grained physical realism. Such artifacts are not unique to our method but represent active areas of research and ongoing improvement across the field. We are committed to advancing these aspects in future work, and our analysis aims to transparently contextualize both the strengths and current limitations of our approach.\n   * In the final manuscript, we will provide a structured taxonomy of these failure cases, supported by qualitative examples and visual overlays. We will also present a comprehensive analysis to contextualize the model’s behavior, along with side-by-side qualitative comparisons against relevant baselines across diverse scenarios. This will further highlight both the strengths and limitations of our approach.\n\n> How does REALM respond to input conditions that are unrelated to human motion, for example, an image of a blue sky or other motion-irrelevant modality data? What kind of motion sequences would it generate in such scenarios?\n   * Thank you for the interesting question. When presented with entirely motion-irrelevant input conditions, such as an image of a blue sky, REALM tends to produce default or low-activity motions, such as standing still or casual walking. This behavior likely reflects the model’s tendency to fall back on generic motion priors in the absence of semantically meaningful cues. \n   * In contrast, for inputs that are not fully descriptive but still weakly associated with human activity (e.g., an image of a basketball), the model often generates plausible actions such as dribbling or shooting, indicating that it leverages learned semantic associations from pretraining. \n   * We agree that analyzing such cases provides important insight into the model's behavior under ambiguous or irrelevant conditions. We will include these examples and a discussion of their implications in the failure case analysis in the final supplementary material.\n\n> The paper primarily uses the AIST++ dataset, which contains music-driven dance data. How does the model generalize to other audio domains, such as speech? Have the authors tested REALM on inputs from datasets like BEAT or EMAGE, which focus on co-speech gestures? If not, could the authors comment on potential domain gaps, and whether the model would need fine-tuning or architectural adaptation to handle such input?\n   * As shown in the supplementary visualizations, REALM is able to generate plausible motion sequences even when conditioned on audio inputs outside the music domain, such as audio of someone boxing, despite being aligned primarily with the music-driven AIST++ dataset. This generalization is largely enabled by the pre-trained LanguageBind space, which captures broad audio-video and audio-text relationships from other domains as well. However, while this demonstrates some cross-domain robustness, we have not explicitly evaluated REALM on co-speech gesture datasets like BEAT or EMAGE.\n   * While REALM produces reasonable results for speech audio with strong emotional or rhythmic cues (e.g., angry speech), its outputs are currently limited to SMPL body motion, without fine-grained hand or facial keypoints, as in SMPL-X, required by datasets like BEAT v2. Consequently, there would be domain gaps, and further fine-tuning would be necessary to fully handle fine-grained co-speech gestures. We consider this a promising direction for future work.\n\n**Closing Remarks**\n\nWe would like to reiterate that our work introduces an \"effective alignment model\" and a generative model that \"effectively leverages information from multiple modalities,\" as you acknowledged. \nFurthermore, we respectfully highlight that:\n   * We propose \"a broadly generalizable, modular and extensible architecture for multi-modal alignment\" along with \"notable design of learnable frame-level tokens and time-aware modulation\" for motion generation, as noted by Reviewer Nskc. \n   * Our framework is a \"unique contribution to the field\" with \"effective and powerful proposed architectures,\" as noted by Reviewer YpBN.\n   * Our work \"addresses a clear gap by incorporating human motion into unified multi-modal embedding spaces, which until now have largely omitted motion data.\" \"This is a notable innovation beyond prior works that aligned only motion and text,\" as noted by Reviewer cqJT. \n   * \"Moreover, the introduction of REALM is an original step that allows any-to-motion generation -- going beyond conventional text-to-motion synthesis by conditioning on arbitrary modality inputs, which is a novel capability in this domain,\" as noted by Reviewer cqJT. \n\nTaken together, we argue that these reviews affirm that \"the idea of binding human motion into a general multi-modal embedding space and leveraging it for cross-modal retrieval/generation is fresh and significant in scope,\" as summarized by Reviewer cqJT. \n\nFinally, as noted across all reviews, our framework achieves state-of-the-art or highly competitive results across multiple multi-modal motion tasks, including cross-modal retrieval, (zero-shot) action recognition, and motion synthesis, evaluated on several benchmarks (HumanML3D, KIT-ML, AMASS, AIST++, BABEL-60, and BABEL-120).\n\nWe hope these clarifications and acknowledgments help convey the strength and significance of our contributions. We respectfully hope Reviewer S428 will reconsider their rating in light of our clarifications and commitments to address their suggestions in the final manuscript."}, {"review_id": "Oqy1gbQgrS", "reviewer": "Reviewer_cqJT", "summary": "This paper addresses a clear gap by incorporating human motion into unified multi-modal embedding spaces, which until now have largely omitted motion data. The authors extend the LanguageBind framework to include 3D human motion, creating a shared semantic space spanning text, vision, audio, and motion. This is a notable innovation beyond prior works that aligned only motion and text. The proposed Multi-Scale Temporal Motion Transformerand the training on diverse paired datasets (language–motion, video–motion, audio–motion) enable joint semantic reasoning across all four modalities. Moreover, the introduction of REALM, a retrieval-augmented latent motion diffusion model, is an original step that allows “any-to-motion” generation – going beyond conventional text-to-motion synthesis by conditioning on arbitrary modality inputs  which is a novel capability in this domain. Overall, the idea of binding human motion into a general multi-modal embedding space and leveraging it for cross-modal retrieval/generation is fresh and significant in scope.", "questions": "None", "limitations": "None", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "- The paper’s approach is technically sound and well-grounded. The MuTMoT architecture is a transformer-based encoder–decoder that hierarchically encodes motion sequences at multiple temporal scales and decodes them back to full motions. This design captures both fine-grained pose dynamics and high-level action semantics, and the decoder enables accurate reconstruction of motion from the learned latent space.\n- The authors conduct a comprehensive set of experiments covering motion reconstruction, cross-modal retrieval, zero-shot action recognition, and text-driven motion generation, evaluated on four datasets (HumanML3D, KIT-ML, AMASS, AIST++). The results are rigorous and highlight the method’s strengths.\n- The paper is written in a clear and organized manner, making it easy to understand.", "was_revised": true, "final_justification": "I will maintain my recommendation.", "rebuttal": "We sincerely thank Reviewer cqJT for their highly positive feedback and recognition of our paper’s contributions. We greatly appreciate Reviewer cqJT highlighting our work’s novelty and significance in incorporating human motion into unified multi-modal embedding spaces, and the originality and technical soundness of both the proposed MuTMoT and REALM architectures.\n\nWe are also encouraged by the reviewer’s acknowledgment of our comprehensive and rigorous experimental evaluation, covering motion reconstruction, cross-modal retrieval, zero-shot action recognition, and text-driven motion generation, evaluated on several benchmarks (HumanML3D, KIT-ML, AMASS, AIST++, BABEL-60, and BABEL-120). \n\nThis reaffirms our belief in the potential impact our work offers toward general-purpose multi-modal human motion understanding and generation. We respectfully hope that the reviewer will continue to advocate for our paper during the discussion phase."}, {"review_id": "NmMAZSQbfR", "reviewer": "Reviewer_YpBN", "summary": "This paper introduces a novel approach to multi-modal human motion alignment by integrating human motion data with existing modalities like text, vision, and audio within a unified embedding space. The authors build on the LanguageBind framework to extend its capabilities to human motion, and design the Multi-Scale Temporal Motion Transformer (MuTMoT) to encode human motion sequences into semantically meaningful embeddings, and also a retrieval-augmented latent diffusion model (REALM) for motion synthesis.\n\nThe paper demonstrates that the proposed method outperforms existing models in tasks like motion reconstruction, cross-modal retrieval, zero-shot action recognition across multiple benchmark datasets.", "questions": "1. In line 176 it says “This modular design ensures generalizability...” But I think the use of dataset-specific embedding modules and heads seems to limit generalizability instead of improving it. If we want to apply this method to a new, unseen dataset, wouldn’t that require re-training the embeddings and heads? I think that good generalizability means being able to generalize to a new dataset without having to re-train anything. Could the authors clarify more specifically on the generalizability of this method?\n\n2. In the abstract (line 8-9) it says: “...Multi-Scale Temporal Motion Transformer (MuTMoT), a transformer-based encoder that...”, but in the introduction (line 49) it says: “...Multi-Scale Temporal Motion Transformer (MuTMoT), a transformer-based encoder-decoder architecture designed to...” So, is it an encoder or an encoder-decoder? The authors should make the statements consistent.\n\n3. What does the \\mathbf{w} in Eq. (3) represent? How is it obtained?\n\n4. Are the MuTMoT part and the REALM part trained at the same time or separately?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**:\n\n1. The integration of human motion into a multi-modal framework, allowing joint reasoning across text, vision, audio, and motion, is a unique contribution to the field.\n\n2. The use of MuTMoT, a multi-scale transformer architecture, to capture motion dynamics at different temporal scales is a powerful method for encoding complex motion sequences. This, combined with the retrieval-augmented latent diffusion model (REALM), demonstrates the effectiveness of the approach.\n\n3. The paper evaluates the proposed methods across a diverse set of tasks and datasets, demonstrating the robustness of the model. The paper shows competitive performance across various benchmarks.\n\n4. The paper provides detailed descriptions of the model architectures and their components, including the MuTMoT encoder-decoder and the REALM generation model.\n\n**Weaknesses**:\n\n1. On text-to-motion generation on HumanML3D, the method does not deliver the best results in any of the metrics.\n\n2. The qualitative results in the supplementary PDF only present the results of the proposed method. A qualitative comparison with other methods should be provided in order to demonstrate the strength of the proposed method.\n\n3. While the model supports any-to-motion generation, the authors only quantitatively evaluate text-conditioned motion generation. A more comprehensive evaluation across all modalities (such as audio or video) would provide a more holistic assessment of the system's capabilities.\n\n4. The approach assumes that the encoders from LanguageBind are sufficiently expressive for all modalities, including human motion. If LanguageBind's representations do not fully capture the nuances of motion, it might hinder the effectiveness of motion retrieval and generation.\n\n5. The retrieval-augmented component of REALM depends on retrieving relevant motion sequences from a database. This could potentially be a bottleneck if the database is insufficient or out-of-distribution, leading to sub-optimal results. \n\n6. While the paper is generally clear, certain sections, particularly the mathematical formulations and model descriptions, could be simplified for better accessibility.\n\n7. The use of dataset-specific embeddings and heads seem to limit the generalizability instead of “ensuring” it, as the authors claim in line 176.", "was_revised": true, "final_justification": "Thanks to the authors for addressing my concerns, I've decided to raise my score to 5 (accept)", "rebuttal": "We thank Reviewer YpBN for their thoughtful and constructive feedback. We greatly appreciate the recognition of our paper’s novelty and unique contributions in integrating human motion into a unified multimodal framework, the effectiveness and expressive power of the MuTMoT and REALM architectures, and the comprehensive evaluation and robust performance across tasks and datasets.\n\nBelow, we address all the raised concerns and clarify key aspects of our design. We believe the issues noted are either due to missing clarifications or minor omissions and do not outweigh the demonstrated novelty, breadth, and rigor of our proposed framework. We respectfully hope that the reviewer will reconsider their rating, taking into account our clarifications and the demonstrated strengths, novelty, and impact of our unified multi-modal motion framework.\n\n**Response to Major Concerns:**\n\n> On text-to-motion generation on HumanML3D, the method does not deliver the best results in any of the metrics.\n   * Our method consistently delivers highly competitive performance on text-to-motion generation for HumanML3D, frequently ranking second-best, and achieves state-of-the-art results on KIT-ML as well as six additional benchmarks spanning reconstruction, cross-modal retrieval, and zero-shot action recognition. **Our primary objective is to demonstrate the effectiveness and generalizability of our multi-modal framework across a diverse set of tasks, rather than focusing exclusively on optimizing for a single dataset or metric.**\n\n> While the model supports any-to-motion generation, the authors only quantitatively evaluate text-conditioned motion generation.\n   * As acknowledged in the limitations section, our quantitative evaluation for motion generation focuses on the widely studied text-to-motion generation task, and this decision reflects the current state of the field, where standardized and robust benchmarks exist primarily for text-conditioned motion generation. While we recognize the importance of evaluating other modalities, there is currently a lack of well-established benchmarks for image- or video-conditioned motion generation beyond pose estimation. **That said, our framework is \"an original step,\" as noted by Reviewer cqJT, to support any-to-motion generation at inference time without additional training, thanks to its multi-modal alignment.**\n   * Thus, we argue that our work should not be viewed as limited in scope. **In addition to achieving state-of-the-art or highly competitive results on text-to-motion generation, we demonstrate strong performance on six additional benchmarks across motion reconstruction, cross-modal retrieval, and action recognition.** Our primary goal is to showcase the versatility and generalizability of our multi-modal framework across diverse tasks. We believe the breadth and rigor of our evaluation present a compelling case for the effectiveness of our approach. We agree that expanding quantitative evaluation to additional conditioning modalities is a valuable direction for future work, and we plan to explore this in subsequent research.\n\n> The approach assumes encoders from LanguageBind are sufficiently expressive for all modalities, including human motion. If LanguageBind's representations do not fully capture the nuances of motion, it might hinder the effectiveness of motion retrieval and generation.\n   * We would like to clarify that LanguageBind provides a strong foundation for visual, text, and audio modalities. However, **we do not assume LanguageBind is sufficiently expressive for human motion modalities**. In fact, **central components of our method are MuTMoT’s dedicated motion encoder, which is explicitly designed to capture the nuances of human motion from pose sequences, and lightweight adapters that make LanguageBind’s implicit understanding of human movement usable by aligning it with the representations from MuTMoT’s encoder**. Our empirical results, including strong performance across cross-modal retrieval, generation, and zero-shot action recognition, demonstrate the effectiveness of this design.\n\n> The retrieval-augmented component of REALM depends on retrieving relevant motion. This could potentially be a bottleneck if the database is insufficient or out-of-distribution.\n   * While REALM utilizes reference motions to enhance generation fidelity, it continues to perform strongly with less relevant reference motions and even without retrieval, as shown in the ablation below varying the retrieval set size (top-$k$). \n\nTable 1: Impact of Reference Motion Quality. We randomly sample two references from the top-$k$ most semantically similar motions in a reference database.\n| Top-K                   | R@1 ↑        | R@2 ↑        | R@3 ↑        | FID ↓         | MM Dist ↓     | MModality ↑    |\n|--------------------------|--------------|--------------|--------------|----------------|----------------|----------------|\n| 2           | 0.435 ± .009 | 0.661 ± .010 | 0.785 ± .008 | 0.264 ± .010   | 2.718 ± .029   | 0.881 ± .079   |\n| 20          | 0.419 ± .007 | 0.650 ± .006 | 0.775 ± .008 | 0.282 ± .035  | 2.762 ± .038  | 0.965 ± .220   |\n| 50          | 0.411 ± .012 | 0.645 ± .017 | 0.773 ± .006 | 0.286 ± .018  | 2.762 ± .035  | 0.940 ± .286   |\n| 100          | 0.410 ± .012 | 0.639 ± .005 | 0.772 ± .004 | 0.291 ± .011  | 2.771 ± .035  | 1.080 ± .124   |\n| No Ref    | 0.419 ± .009 | 0.655 ± .015 | 0.786 ± .014 | 0.307 ± .011  | 2.707 ± .023  | 0.876 ± .023   |\n\n**These results show that REALM performs consistently, with only gradual performance degradation as less relevant reference motions are included or even when no reference is provided, confirming the robustness of our generation pipeline.**\n\n> A qualitative comparison with other methods should be provided.\n   * We agree that qualitative comparisons are valuable. Accordingly, we have conducted qualitative comparisons against four baseline methods (T2M-GPT, MotionDiffuse, MDM, and ReMoDiffuse), where our approach demonstrates superior or comparable motion fidelity and semantic alignment. Due to format constraints, we are unable to include the visual results here, but we will include a comprehensive version in the supplementary material.\n\n> While the paper is generally clear, certain sections, particularly the mathematical formulations and model descriptions, could be simplified for better accessibility.\n   * We appreciate your suggestion and will revise the mathematical formulations and model descriptions to improve clarity in the final manuscript.\n\n**Response to Additional Questions:**\n\n> I think the use of dataset-specific embedding modules and heads seems to limit generalizability instead of improving it... Could the authors clarify the generalizability of this method?\n   * The dataset-specific embedding modules and heads **do not** limit the model to particular datasets. **Instead, they address a practical and common challenge in motion modeling: different datasets often use different representation formats, such as varying numbers of joints or feature dimensionalities** (e.g., KIT-ML uses 21 joints with 251-dimensional vectors, while HumanML3D uses 22 joints with 263-dimensional vectors). Our dataset-specific embedding modules map these diverse raw motion formats into a unified latent space, and the corresponding heads project the shared representation back to the original format as needed.\n   * This modular design is a key enabler of generalizability and robustness. First, it allows us to train a single model across multiple heterogeneous datasets, exposing it to a wide range of motion patterns, formats, and distributions. Second, by learning from various representations during training, the model is better positioned to handle unseen datasets that share a format with any of the training sets, without requiring any additional steps. This stands in contrast to models trained on a single dataset (format), which typically need additional adaptation steps and/or retraining to generalize to new datasets. **In summary, the dataset-specific modules do not limit the model’s generalization; rather, they are what make it possible to unify and scale across diverse motion data sources within a single framework.** \n\n> Is MuTMoT an encoder or an encoder-decoder?\n   * MuTMoT is an encoder-decoder architecture, and we will update the manuscript to consistently describe it as such. For clarity, only the encoder is used for retrieval and action recognition during inference, whereas both the encoder and decoder are employed for reconstruction and generation tasks. We will make this distinction explicit in the revised manuscript to avoid any confusion.\n\n> What does the \\mathbf{w} in Eq. (3) represent? How is it obtained?\n   * \\mathbf{w} represents learnable contribution weights used to combine motion embeddings from different temporal scales produced by the MuTMoT encoder’s hierarchical blocks.\n   * As described in LL156-162, each block in the MuTMoT encoder captures motion information at a different temporal resolution, producing a global embedding at that scale. The weights \\mathbf{w}, as noted in LL163-165, are learned during training and determine the relative importance of each temporal scale when computing the final fused motion representation. This allows the model to adaptively emphasize the most relevant temporal features.\n\n> Are the MuTMoT part and the REALM part trained at the same time or separately?\n   * We follow a two-stage training process. In the first stage, we train MuTMoT, including both the encoder and decoder, using a combination of motion reconstruction and contrastive alignment objectives, enabling it to learn robust motion representations and align them with other modalities. In the second stage, we train REALM on top of the pretrained MuTMoT encoder and decoder. This two-stage approach ensures that the motion representations are well-formed and semantically aligned before being used for conditional motion generation."}]

Author rebuttal:

Meta-review: ## Summary 

This paper introduces a unified framework for aligning human motion sequences with multiple modalities (text, video, audio) within a shared embedding space, alongside a novel generative pipeline for motion synthesis conditioned on arbitrary inputs. The key contributions are: a) MuTMoT: a multi-scale temporal motion Transformer that hierarchically encodes and decodes 3D motion sequences; b) REALM: a retrieval-augmented latent diffusion model that utilizes learnable frame tokens and cross-modal conditioning to generate high-quality motion. The model is evaluated across several tasks, including text-to-motion generation, motion retrieval, and zero-shot action recognition. 

## Strengths

- The integration of human motion into a multi-modal framework allows joint reasoning across text, vision, audio, and motion, and a modular and extensible architecture combining multi-modal alignment, contrastive learning, and latent diffusion.
- Uses learnable frame-level tokens and time-aware modulation.
- Outperforms existing baselines on four datasets (HumanML3D, KIT-ML, AMASS, AIST++) with a comprehensive supplementary ablation that covered most core components.
- MuTMoT architecture encodes motion sequences at multiple temporal scales and decodes them back to full motions.

## Weaknesses

- The method relies on pre-frozen LanguageBind encoders for all non-motion modalities, limiting its modality-specific understanding.
- The model's retrieval performance is largely attributed to GPT-4o-based text paraphrasing augmentation, which may not fully account for observed gains.
- The authors only quantitatively evaluate text-conditioned motion generation, which could be a bottleneck if the database is insufficient or out-of-distribution. The paper could benefit from a more comprehensive evaluation across all modalities.
- The lack of failure case analysis raises concerns.
- The model's response to input conditions unrelated to human motion is unclear.

## Decision

The paper shows a solid contribution besides some concerns from the reviewers that have been addressed during the rebuttal. Therefore, the recommendation is the acceptance by adding and clarifying the doubts from the reviewers.

## Summarize

The discussion among the reviewers and the authors were fruitful since the authors helped the reviewers to clarify some doubts and misunderstandings during the review. At the end, most of the reviewers are in favour of accepting the paper as a poster besides the most critical reviewer.
Decision: Accept (poster)