Scaling Large Motion Models with Million-Level Human Motions
Authors: Ye Wang
Sipeng Zheng
Bin Cao
Qianshan Wei
Weishuai Zeng
Qin Jin
Zongqing Lu
Venue: ICML 2025 poster (2025)
Keywords: motion generation
Primary area: applications->computer_vision
TL;DR: We introduce a million-scale motion dataset to ease data scarcity. We also explore scaling laws in motion generation.

Abstract: Inspired by the recent success of LLMs, the field of human motion understanding has increasingly shifted toward developing large motion models. Despite some progress, current efforts remain far from achieving truly generalist models, primarily due to the lack of massive high-quality data. To address this gap, we present MotionLib, the first million-level dataset for motion generation, which is at least 15$\times$ larger than existing counterparts and enriched with hierarchical text descriptions. Using MotionLib, we train a large motion model named Being-M0, demonstrating robust performance across a wide range of human activities, including unseen ones.
Through systematic investigation, for the first time, we highlight the importance of scaling both data and model size for advancing motion generation, along with key insights to achieve this goal. To better integrate the motion modality, we propose Motionbook, an innovative motion encoding approach including (1) a compact yet lossless feature to represent motions; (2) a novel 2D lookup-free motion tokenizer that preserves fine-grained motion details while expanding codebook capacity, significantly enhancing the representational power of motion tokens. We believe this work lays the groundwork for developing more versatile and powerful motion generation models in the future. For further details, visit https://beingbeyond.github.io/Being-M0/.

Scaling Large Motion Models with Million-Level Human Motions
Ye Wang" "1
Sipeng Zheng
Bin Cao 23
Qianshan Wei Weishuai Zeng"
Qin Jin
Zongqing Lu 5.66
Abstract
Inspired by the recent success of LLMs, the field of human motion understanding has increasingly shifted toward developing large motion models. Despite some progress, current efforts remain far from achieving truly generalist models, primarily due to the lack of massive high-que high-quality data. To address this gap, we present MotionL.b, the first million-level dataset for motion generation, which is at least 15x larger than existing counterparts and enriched with hierarchical text descriptions. Using MotionI.ib, we train a large motion model named Being-MO, demonstrating robust perfor- mance across wide range of human activities, including unseen ones. Through systematic in- vestigation, for the first time, highlight the importance of scaling both data and model size for advancing motion generation, along with key insights to achieve this goal. To better integrate the motion modality, we propose Motionbook, an innovative motion encoding approach including (I) compact yet lossless feature to represent motions; (2) novel 2D lookup-free motion tok- enizer that preserves fine- tie-ggrined motion details while expanding codebook capacity, significantly enhancing the representational power of motion tokens. We believe this work lays the ground- work for developing more versatile and power- ful motion generation models in the future. For further details, visit https: /bbiingeyynnd. gi thub. i io/Beiinggeiee MO0.
Arua Wil.coedtigg withiiatiing
Seaer
ancardl iight can sings elbadd by that daly withe litt
the rig arn sitedde finundl with the spen
ber thadde
antharr stomacy
figm poining dutsort left hamp by their alde.
sergle lown budly Buck foot co aloane firily or the greass thiing dightt spart pancol woigge appeian
ingls diiliyy fanwe with di leet furco the gioand the right lig luck, wilh luei tightss ralaed, saggectngg
Motioni ib
(6) Motionll
FID
scale datesets
Mistire EL.
MotionLib 2M Distribution
162
Sel
0.368
8B
138
the scale of model parameters
new collected Il Metions [BI HumamMML33
Figure 1: TOP: While existing models perform well on small-scale datasets like Motion-X and HumanML3D, they struggle with out-of-domain concepts on MotionLib, ex- hibiting limited generalization. DOWN: Curves showing the effects of scaling up large motion models. MotionLib is the first large T2M dataset comparable in scale to visual benchmarks like ImageNet.
1. Introduction
Motion generation is an emerging field with wide range of applications video games, filmmaking, and robotics. At the forefront of this area text-to-motion generation (T2M) (Ahn et al., 2018; Ahuja & Morency, 2019), which plays crucial role in translating natural language into hu- motions. State-of-thh--an (SoTA) T2M models typically rely on combination of the motion tokenizer (e.g.. vector quantization VQ (Van Den Oord et al., 2017)), along with a text encoder (e.g., CLIP (Radford et al., 2021)) and decoder (c.g., GPT-2 (Radford et al., 2019)) to generate motion se- quenees from text commands. Despite the availability of
'Equal contribution 'Remmin University of China 'Beijing Academy of Artificial Intelligence 'Instiuue of Automation, Chinese Academy of Sciences "Southeast University 'Peking University BBeingBeyond. Correspondence Zongqing Lu <zongqing. Iu wtr.iiiguuuupuuu..u....
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Scaling Large Motion Models with Million-Level Human Motions
few high-quality datasets (Guo et al., 2022a; Lin al., 2024) curated in recent years, their limited size restricts current methods to narrow range of scenarios, creating performance bottlenecks when addressing diverse or unseen motions, as illustrated in Figure 1 (TOP).
(2) Limited Codebook Size: Existing VQs rely on small codebooks, meaning that all generated motions must be selected from constrained set of predefined options. As result, these ID embeddings fail to capture the full di- versity of human motion. Second, most works adopt the H3D-format feature (Guo et al., 2022a) to represent motion, which omits crucial information (e.g.. original rotation) and requires time-consuming methods (Bogo et al., 2016) to recover. This inefficiency hinders real-world applications (e.g., animation), where real-time generation is essential.
Recently, the rapid advancement of large language mod- els (LLMs) in multimodal learning has been significantly bolstered by the availability of vast data. In contrast, the vol- ume of motion data remains considerably smaller than that of visual-text data, as shown in Figure (DOWN). This dis- parity primarily arises from the high costs associated with motion data collection, which often requires specialized wearable devices and substantial human labor for annota- tion. Consequently, developing a SoTA large motion model based on LLMs presents a significant challenge and remains an unresolved issue. While some recent efforts (Jiang et al., 2023) have explored this direction, the effectiveness of large motion models has yet to be fully demonstrated.
To address these issues, we propose MotionBook, a novel motion encoding approach. Specifically, MotionBook first chooses a more compact yet lossless feature to represent M, which preserves motion information without compromis- ing performance. Additionally, it introduces a new motion quantization method, 2D-LFQ. Our 2D-LFQ reformulates each motion sequence as a 2D image with a single channel, represented as M E RT*Dx1, By expanding the motion se- quence's 's dimensionality from ID to 2D, we enhance the en- coder's capacity, enabling it to capture complex motion pat- terns while retaining more critical details after tokenization. While increasing the codebook size is a straightforward way to enhance expressiveness, it often leads to codebook col- lapse, especially when training samples are limited. To miti- gate this, we introduce a finite scalar quantization method inspired by Mentzer et al. (2023), which enables learning a large motion vocabulary without requiring a lookup for corresponding tokens in the codebook for each entry. As a result, 2D-LFQ expands the motion codebook by at least two orders of magnitude, significantly boosting its represen- tational capacity while maintaining efficiency.
In this paper, we aim to address the question: "Can scaling the large motion model and data benefit motion genera- ?i To tackle this, we develop a sy systematic data collec- tion pipeline to build MotionLib, the first large-scale dataset containing over 1.2M motion sequences - at least 15x larger than current counterparts. This initiative provides a solid foundation for building robust, universally applicable motion models and offers a comprehensive testbed for future research. Using MotionLib, we conduct a comprehensive investigation into the large motion model. For the first time, we show the scaling law of both data and model size in mo- tion generation, which significantly reduces joint prediction errors while improving generalization to novel motions. In addition, this research identifies several key factors driving their advancement and offers valuable insights for future model design (e.g.. LoRA or full-parameter tuning).
We summarize our contributions as: (1) MotionLib The first million-scale motion dataset, comprising over 1.2M motion sequences with hierarchical and detailed text anno- tations. (2) MotionBook - An effective motion encoding approach that combines a lossless and efficient motion fea- ture with lookup-free tokenizer, 2D-LFQ. Our 2D-LFQ preserves essential motion details, expands the motion en- coder's 's capacity without token lookups, and improves the motion tokenizer's s ability to leverage large-scale motion data. (3) Being-MO - A large motion model trained using MotionLib and MotionBook. We provide key insights into scaling both data and model size, identifying critical factors that influence the effectiveness of large motion models.
In addition, we argue that large motion models are con- strained by inadequate motion representations. First, most approaches transform motion into discrete tokens via VQ, which are then processed by autoregressive models to gener- ate motion sequences. While these methods have achieved impressive results, they suffer from two major drawbacks (1) Information Loss: The VQ process inevitably discards critical motion details. Given a motion sequence with D-dimensional features M {m1, (m1,m2,... m2,..., m,),, where m; R.,, VQ compresses it into a sequence of ID embed- dings of size [T/a] d, where a is the temporal down- sampling ratio and d is the codebook dimension. Unlike images, which consist of uniform RGB pixel values, each motion state my encodes diverse features (e.g.. joint po- sition, velocity, foot-ground contact). Representing such complex states with a single ID embedding is insufficient, leading to information loss and restricting the model's abil- ity to generate motion at fine-grained, level resolution.
2. Related Work
Large Language Models and Multi-Modality, Substantial progress has been made in enhancing LLMs (Brown et al., 2020; Raffel et al., 2020; Chowdhery et al., 2022) with the ability to understand and respond human commands, through technique known as instruction tuning (Ouyang et al., 2022). Recent research has extended these capabili-
2

---

Scaling Large Motion Models with Million-Level Human Motions
ties to the multimodal domain (Ye et al., 2023; Zheng et al., 2024; Feng al., 2024; Mei et al., 2024), with notable work by Liu et al. (2023), who pioneered visual instruction tuning to create a highly adaptable assistant. In addition, Li et al. (2023a) integrated multimodal context into instruction data to further enhance performance. Subsequent studies (Zhang et al., 2023c) expanded this research by scaling up instruc- tional data and incorporating image-rich text. Notably, Dai et al. (2023) developed InstructBLIP based on 2LIP-2 (Li et al., 2023b), which features an advanced visual feature extraction mechanism to improve performance across vision- language tasks. Despite these, the application of multimodal models to human motion remains less competitive compared to current SoTA methods, although recent initiatives are be- ginning to explore this domain (Zhang et al., 2024d).
motion. Recently, Guo al. (2024) proposed integrating residual quantization and masked modeling to improve tra- ditional vector quantization (VQ). Lu et al. (2023) designed JluannTomato a hierarchical VQ-VAE to separately en- code body and hand motions. To better align with a motion auto-encoder, MotionCLIP (Tevet et al., 2022) incorporates CLIP (Radford et al., 2021) as the text encoder, bringing in more robust text priors. Additionally, Zhang et al. (2024d) and Jiang et al. (2023) explored the development of unified models based on LLMs that accept multimodal conditions (e.g., vision, text, and pose), enabling the generation of subsequent, preceding, or "in-between" motions. Despite leveraging the power of LLMs, these large motion models remain limited to in-domain text instructions and do not yet perform as competitively as existing SoTA methods.
Motion Vector Quantization. Vector quantization (VQ) has been successful in generating high-quality images (Van Den Oord et al., 2017) and videos (Gupta et al., 2022; Yan et al., 2021). VQ-VAE first converts images into discrete representations and autoregressively models their distribu- tion. Building on this, Lee et al. (2022) introduced residual quantization (RQ), which encodes images into stacked map of discrete codes, efficiently reducing the spatial res- olution of features. You et al. (2022) further developed hierarchical vector quantization (HQ), employing a pyra- mid scheme with two-level codes for image encoding. In motion generation, most existing approaches have adopted VQ or its variants to quantize human motions. However, the small codebook size in traditional VQ methods limits their ability to generalize and accurately represent the diversity of human motions. Although increasing the codebook size can improve representational capacity, it often leads to code- book collapse. Recently, Mentzer et al. (2023) demonstrated that discrete codes can be obtained via scalar quantization, where each scalar entry is quantized to the nearest integer through rounding. Similarly, Yu al. (2023) introduced a lookup-free codebook that maps videos into compact dis- crete tokens, utilizing all codes without auxiliary losses and expanding the codebook size.
In this work, aim to bridge the gap between large lan- guage models and generalizable, reliable large motion mod- els. To achieve this, We introduce MotionLib the first million-level motion dataset designed to support extensive pretraining and comprehensive fair evaluation.
3. MotionLib: A Million-Level Motion Library
Data are the foundation of large motion models. With ad- vancements in fields such as human pose detection, we are now able to extract high-quality motion sequences from vast amounts of online videos. Our Motionl.it Jib dataset con- tains over one million motion clips, totaling approximately 137 million frames. Each clip is annotated with fine- -raiind automatic pseudo-labels. A comparison with existing bench- marks is presented in Table 1. Examples of this dataset are shown in Figure 2. Our data curation pipeline involves the following main procedural steps. For more details, please refer to Appendix B due to limited space.
Million-Level Motion Collection. We begin by collect- ing more than 20 million videos from publicly available datasets (Kay et al., 2017) and online platforms such as YouTube. To ensure motion quality, that do not contain human figures. For each video, we use a pretrained model (Xu et al., 2022) to detect 2D human keypoints and filter out those without visible human activity. The human's 's bounding box is required to occupy a signifi- cant portion of the frame, making human movement clearly visible. Videos with only partially visible humans are re- moved to maintain the quality of the extracted motion. We adopt WHAM (Shin et al., 2024) to extract the SMPL. pa- rameters of collected videos, regressing 3D human motion in the world instead of the camera coordinate system.
filter out videos
Human Motion Generation The task of motion gener- ation involves creating human motion based on various inputs, such as text descriptions (Guo et al., 2022b), ac- tion labels (Cervantes et al., 2022) or motion prefixes (Liu et al., 2022). Among these, texi-to-motion (T2M) gener- ation has received the most attention due to the ease and flexibility of using natural language as input. Early ap- proaches (Fragkiadaki et al., 2015; Ghosh et al., 2017) rely on deterministic motion modeling, which often produces av- craged, blurry results. To overcome this, stochastic methods using models like GANs (Wang et al., 2020) or VAES (Ali- akbarian et al., 2020) have been considered. For instance, T2M-GPT (Zhang et al., 2023a) extends the temporal VAE to capture the probabilistic relationship between text and
Hierarchical Motion Descriptions. Existing benchmarks face inherent limitations in their text descriptions. Previ- ous studies (Guo et al., 2022a) typically use a few short sentences to describe whole-body motions, neglecting finer
3

---

Scaling Large Motion Models with Million-Level Human Motions
Table I: Comparison with existing human motion datasets. More details can be found in Appendix B. In the table, B, H, and F refer to body. hand, and face, respectively. "hier" indicates that the text captions include hierarchical descriptions of motions, while "body" means the descriptions are not as detailed. "multi" and "single" specify whether the dataset contains periti scenarios or only single-pon peggle-prson data. As the largest motion generation dataset and benchmark to date, MotionLib features least 15x more motion and text data than previous datasets, along with additional modalities.
SEQ NUM
TEXT NUM
HOURS
MOTION
TEXT
RGB
DEPTH
BBOX
PERSON
KIT (Plappert et al., 2016)
5.7K
5.7K
11.2
B
body
x
X
X
single
HumanML3D (Guo al., 2022a)
29.2K
89K
28.6
body
X
X
X
single
MotionX (Lin et al., 2024)
81.1K
142K
144.2
B.H.F
body
/
X
X
single
Motion Verse (Zhang ct al., 2024a)
320k
373k
B.H.F
body
v
X
x
single
MotionLib
1.21M
2.48M
1456,4
B.H
hier
V
V
single multi
details of individual body parts, such as the arms or legs. This restricts the model from performing more nuanced body comprehension and more flexible part-level motion control (e.g., raising only the left arm). Moreover, the rich- ness of text labels often varies across different motions. For example, a large portion of the Motion-X X dataset provides only a single sentence, and HumanMI contains numer- ous similar text annotations. contrast, MotionLib offers hierarchical textual annotations for each video inspired by Pi et al. (2023): (1) part-level description, where we care- fully design a prompt format and use Gemini-1.5-pro (Reid et al., 2024) to generate detailed captions for individual body parts (e.g., left arm), assigning dedicated sentence to each; (2) body-level description, which summarizes the whole body movement in a comprehensive paragraph containing 1-3 sentences. These hierarchical motion captions provide far more textual content and levels of detail than previously available. Their richness and structure are crucial for large language models to better understand and align with the motion modality.
Figure 2: Examples from MotionL Lib, which encompasses diverse range of human motions from web videos. It fea- tures various scenes, ranging from outdoor environments to indoor settings, and includes both clean, single-person sce- narios as well as crowded, multi-person scenes. MotionLib provides over 2.4M motion-text pairs in total. The whole illustration with more examples can be seen in Figure 5.
Motion and Description Refinement. It is inevitable that the million-scal motions collected from diverse web sources initially contain noise. According to some previous studies (Holden, 2024), the low-quality motion may damage generation results. To enhance the quality of our collected motion, we first estimate precise 3D keypoints with another pre-trained model (Sárándi et al., 2023) to perform local- global motion optimization, following Lin et al. (2024). In addition, we train an RL-based policy Trefine based on Luo et al. (2023) to elaborate and refine raw motions to obey physical laws (e.g., maintaining balance) and appear more realistic. Trefine takes raw motion sequences as input, treating them target poses, and generates new motion sequences that satisfy physical laws in a simulated environ- ment, thus eliminating issues like jittering and foot-sliding. While effective, Trefine may struggle with drastic movements in target poses, leading to slipping within the simulation. For such cases, we mark them with a smaller weight during pretraining, foreing our model focus more on high-quality motion. To refine the description, since our text labels are
structured into body and part levels, we condition GPT-4o with one level of text while using the other as the target to be refined. GPT-4o then assesses and refines the text content of the target level. enabling the generation of more precise and reliable descriptions.
4. Being-MO: Scaling up Large Motion Model
4.1. Overview
Inspired by the success of Large Language Models (LLMs) in multimodal tasks, we adapt similar approach for mo- tion generation, treating motion as a distinct "language" to be learned (Luo et al., 2020; Zhang et al., 2024c). Simi- lar to previous LLM-based multimodal models, we treat motion as a foreign language. The overall framework is illustrated in Figure 3. Our large motion model, built
4

---

Scaling Large Motion Models with Million-Level Human Motions
2D-LFQ Quantizer
woman
blowing
ballons
while walking
2D-LFQ Encoder
Autoregressive
SMRL 0356
Language Model
Ousput Motion Tokans
2D-IFQ Decoder
2D-LFQ Quantizer
Motion Codebook
0
<modion Jd.
<notion JM
Smotion,
Aimetion,
croooon, Jd. ...
Figure 3: Overview of our large motion model named Being-MO, which can be divided into two stages, In the first stage (left), we pre-train a motion VQ-VAE to quantify motion sequences into tokens. In the second stage (right), we fine-tune an autoregressive language model to predict motion tokens.
on a pre-trained LLM, functions as a generative model that connects motion tokenizer with the LLM backbone O. The motion tokenizer encodes the features M (m,,,,,.... mT) } of raw motion sequence into token em- beddings V = [U1, u,,..., } E R"*c, where n denotes the number of motion tokens and represents the dimen- sionality of each token. To integrate motion tokens into the LLM backbone, we incorporate K discrete codes in the motion codebook additional vocabulary for the LLM. Additionally, introduce two special tokens, <mot>> and </mot>, to signify the start and end of motion sequences within the input/output streams. The LLM backbone θ is built on decoder-only architecture using causal trans- formers. The model generates outputs y {811 V2...., 3m in an auto-regressive manner, where y corresponds to the generated motion sequence based on the provided motion- text input tokens. In this work, each motion-text pair in the MotionLib dataset is framed an cestructioo-followiin instance (XQ,AM }, representing question-an question-answer inter- action between the user and the motion model. The entire instructional dataset adheres to this unified format. To train our model, we optimize the negative log- likelihoood over the predicted tokens which is defined as:
<Caption Placeholder> through movement."). We then select subset of high-quality motions from MotionLib to formulate instructional data. For each motion, randomly choose three templates, inserting the correspond- ing text into the hance naturalness and fluency, we leverage Gemini-Pro to refine these instructions, ensuring they align with human expression patterns. Finally, we construct an instruction fine-tuning dataset containing 900K instructions.
To further en-
4.2. MotionBook: Towards Effective Motion Encoding
MotionBook consists of (1) a lossless feature representation for motion sequences M, and (2) 2D Lookup-Fit Quan- tization (2D-LFQ) motion tokenizer compress M into series of discrete tokens.
Lossless Motion Feature. Conventional HumanML3D fea- ture (H3D-Format) (Guo et al., 2022a) primarily focuses on joint positions and derives rotations using Inverse Kinemat- ies (IK), which discards the original rotational information from SMPL features. Additionally, H3D-Format represents the root's angular velocity using a single scalar, captur- ing only Y-axis rotation, further limiting its expressiveness. Considering this, we adopt feature named SMPL-D135 to represent motion M. Each frame is encoded as m E IR 135 structured as follows: (1) root (9D), 6D rotation Frot E R 2D XZ.-plane velocity r E IR.2 and 1D height rV E (2) body joints (126D), each of 21 key body joints are repre- sented using 6D rotation vectors j" 222 By directly encoding complete joint and root rotation data (j" E 2 and Frot E R²), SMPL-D133 preserves critical motion in- formation lost in H3D-Format while being more compact.
(()) = Σ log Po Payyydesee 11:9-1), 1=1
(1)
where i and y denote the input and target token sequences, respectively. θ represents the model parameters, and L is the length of the target sequence. The input description, desc, can be empty depending on the instruction provided.
Two-Stage Training. Similar to Liu et al. (2023), our training process incorporates motion-text alignment and motion instruction tuning. In the first stage, uti- lize the entire MotionL.b dataset to enable the model to learn fundamental motion-text correlations from diverse data. In the second stage, to better align with real-world human commands, we construct a set of over 250 in- struction templates (e.g., "Show me demonstration of
2D Lookup-Free Motion Quantization. Traditional mo- tion tokenizer typically involves an encoder E, a decoder D, codebook C and quantizer Q. It uses 1D embeddings to represent motion at each timestamp, which inevitably results in the loss of crucial information. Furthermore, this approach limits the tokenizer's 's ability to interpret pevt-
5

---

Scaling Large Motion Models with Million-Level Human Motions
Table 2: Comparisons under different model parameters and data sizes, showing their scaling law for motion generation.
Motion-X-eva
MotvonLib-eval
Decoder
PInst.
Paaram
R@
R@31 1
FID 4
R@1 1
R@31 f
FID L
Real
0.514
0.831
0.046
0.297
0.634
0.004
GPT-2
0.02M
355M
0.213
0.426
47,319
0.058
0.152
30,612
GPT-2
0.08M
355M
0.468
0.792
0.083
0.114
0.281
22.077
GPT-2
0.5M
355M
0,463
0.793
0,121
0.161
0.354
9.157
GPT-2
1.2M
355M
0.472
0.791
0.112
0.166
0.375
6.936
LLaMA-2
0.02M
0.216
0.433
47.538
0.059
0.158
29.643
LLaMA-2
0.08M
0.472
0.798
0,166
0.118
0.294
21.593
LLaMA-2
0.5M
0.468
0.799
0,178
0.164
0.369
9.146
LLaMA-2
1.2M
7B
0.475
0.798
0,156
0.171
0,380
6.632
LLaMA-3
0.02M
8B
0.216
0,435
47.906
0,059
0.162
29.257
LLaMA-3
0
8B
0.483
0.815
0.122
0.120
0.301
21.295
LLaMA-3
0.5M
8B
0,483
0.817
0,113
0.166
0.368
8.973
LLaMA-3
1.2M
0.486
0.820
0.117
0.173
0.386
6.029
LLaMA-2
0.02M
13B
0.223
0.446
47,210
0.061
0.169
29.143
LLaMA-2
0.08M
13B
0,488
0.820
0.156
0.124
0,314
21.001
ILaMA-2
0.5M
13B
0.490
0.819
0.145
0.174
0.374
8.824
LLaMA-2
1.2M
13B
0.491
0.823
0.133
0.185
0.391
6.221
motions. To tackle these, treat the motion sequence M {m1, m2,...,,,y } single-channel image, repre- senting each motion sequence as .M E RTxxXx, Each motion feature m, is divided into P components, capturing distinct features of motion, such as root orientation, joint rotation, and foot contact. Our motion encoder then con- verts M into a feature map E(M) E RLT/a] xxd, where denotes the temporal downsampling ratio. This approach ensures that each body part is tokenized separately, allowing for more granular, part-level motion encoding and decoding.
where Cjj denotes the j-th value of C.. The token index is computed as Index(=) = , 11, > 0}. To train the tokenizer, we employ a standard combination of recon- struction, perceptual, and commitment losses, along with an entropy penalty to promote better codebook utilization (Yu et al., 2023). Importantly, exclude the use of GAN loss, as it is found to negatively impact training stability.
5. Experiments
Due to space limitation, we present more details about the metrics and model implementation in Appendix A, as well as additional experiments in Appendix C.
In addition, previous motion tokenizers are restricted from capturing the full diversity of human motion due to their small codebook sizes. A common approach to address this limitation expand the vocabulary. However, excessively enlarging the codebook can result in codebook collapse, where only a small subset of tokens in the codebook is used, offering minimal improvements even potentially degrading the model's performance. To enable effective learning with larger codebooks suited for million-scale datasets like Mo- tionLib, we adopt a tookup-free quantization strategy. A more effective way is to reduce the dimensionality of code embeddings (Mentzer al., 2023), which limits the rep- resentational capacity of individual tokens and encourages more efficient learning across larger vocabulary. Similar to Yu et al. (2023), we reduce the embedding dimension of the codebook to zero by replacing the codebook C < R xd with an integer set C with [C] = K. Here, C is the Carte- sian product single-dimensional variables C = X d 1 C where C. = {-1,1} and d is equal to log, K. Given feature vector = E Rd. our motion quantizer Q(-) converts each dimension of the quantized representation into:
5.1. Experimental Setup
Datasets. Our investigation is first conducted on Hu- manML3D (Guo al., 2022a) and Motion-X (Lin al., 2024). HumanML3D comprises 14,616 motion sequences from the AMASS dataset (Mahmood et al., 2019), paired with 44,970 textual descriptions. Motion-X, a more recent dataset, includes approximately 81,000 motion sequences. To validate our conclusions on larger-scale data, we also carry out experiments on our MotionLib dataset with two variants: MotionLib-OS. and MootionLibbfull MotionLibh Lib- 0.5 contains 500K clips, while MotionL fublfull encompasses all 1.2M clips of our collected data. Following standard practice, each dataset is split into training, validation, and test sets in proportions of 85%, 5%, and 15%, respectively.
Evaluation. For the motion generation task, we em- play the evaluation metries following (Guo et al., 2022a): Motion-retrieval Precision (R-Precision), Multimodal Dis- tance (MMDist), and Frechet Inception Distance (FID). R-
Q(z;)) = arg ming,
|| = 112 s +0}+1(2 > 0}.
(2) Precision assesses the consistency or matching between
6

---

Scaling Large Motion Models with Million-Level Human Motions
generated motions and input text descriptions. MMDist measures the average distance between generated motion and corresponding text in feature space. FID evaluates the realism and quality of generated motions by comparing the distribution of generated motion features with that of real motion features. In addition, we evaluate our motion to- kenizer on the motion reconstruction task. Besides FID, this task is also measured by MPJPE, which measures the average distance between predicted and ground truth joint positions across all joints, in millimeters.
Table 3: Comparison with existing SoTA methods on Hu- manML3D. Results marked with represent values repro- duced using the official code, while unmarked results are taken from the original papers. 'andd denote different works with the same model name. For fair comparison, ex- periments here conducted using 1AM3D-Format feature.
Decoder
|@11 f R@31 1 FID MMDist 1
Real
0.511
0.797
0.002
2.974
MLD
0.481
0.772
0.473
3.196
MotionDiffuse
0.491
0.782
0.630
3.113
RcMoDiffuse
0.510
0.795
0.103
2.974
5.2. Discussion of Scaling up Motion Generation
F-gT2M++
0.513
0.801
0.089
2.925
LMM
0.525
0.811
0.040
2.943
In this section, we investigate the impact of model size and data scale for motion generation. For evaluating the models, we use the same motion autoencoder architecture (Guo et al., 2022a) retrained separately on Motion-X and MotionLib. This model is used to evaluate the performance of motion generation models on their respective test sets. We catego- rize training data into four scales: 0.02M (HumanML3D only), 0.08M (Motion-X only), 0.5M (MotionLib--0.5), and 1.2M (MotionLib-fulll full).. For fair comparison, employ the same motion tokenizer, maintaining consistency across experiments to validate our conclusions.
T2M-GPT
GPT-2
0.492
0.775
0.141
3.121
DiverseMution
GPT-2
0.510
0.802
0.072
2.941
MoMask
0.521
0.807
0.045
2.958
MotionGPT
T5
0,409
0.667
0.162
3.992
MotionGPT
T5
0.492
0.778
0.232
3.096
MotionGPT
IJaMA2- 13B
0.367
0.654
0.571
3.981
MotionGPT
LLaMA-13B 3B
0.363
0.633
0.592
4.029
MotionGPT2
LLaMA-13B
0.411
0.696
0.542
3.584
Mo
Gemma-2b
0.482
0.770
0.491
3.138
AvatarGPT
LLaMA-13B
0.389
0.623
0.567
MotionGPT-v2
LLLMA3.1-8B
0.496
0.782
0.191
3.080
Being-MO-VO
LLMMA2-13B
0.519
0.803
0.166
2.964
Bcing-MO-LFQ
LLaMA2-13B
0.528
0,820
0.141
2.875
Does increasing model size benefit motion generation? Yes. As shown in Table 2, the experimental results demon- strate that increasing model size leads to a stable perfor- mance enhancement when the training data is kept con- stant. For example, our model achieves best R @ 1 (0.491) on Motion-X when using LLaMA2-13b, which is 0.016 higher than LLaMA2-7b's ' 0.475. On the larger Motion- Lib dataset, the best @@I of LLaMA2- 13b reaches 0. 1815, which is 0.014 higher than LLaMA2-7b's 0.171. This series of results strongly validates the trend of performance gains scaling with model capacity. Consequently, larger models show a greater ability to capture the diverse and intricate patterns and relationships within human motion.
Does our large motion model outperform SoTA mod- els? Yes. We evaluate model Being-MO the widely adopted HumanML33 benchmark. We compare its per- formance against a variety of SoTA approaches. This includes diffusion- methods such as MLD (Chen et al., 2023), MotionDiffuse (Zhang et al., 2022), ReMoDif- fuse (Zhang et al., 2023b), Fg-T2M++ (Wang et al., 2025) and LMM (Zhang et al., 2024a). It also includes autore- gressive models like T2M-GPT (Zhang et al., 2023a), Di- verseMotion (Lou et al., 2023), and MoMask (Guo et al., 2024). We also compare against LLM fine-tuning methods like MotionGPT (Jiang et al., 2023; Zhang et al., 2024d), MotionGPT-v2 (Wang et al., 2024), MotionLLM (Wu et al., 2024), and AvatarGPT (Zhou et al., 2024). As shown in Table 3, our model, which utilizes I.LMMA2133B as the motion decoder, achieves SOTA R-Precision performance. Existing T2M methods can be categorized intow two types: (1) specialist models optimized on specific datasets, and (2) ILLM-based generalist models (aiming for broader instruc- tion and task generalization via LLMs). like AvatarGPT, MotionGPTT-22 and our Puppet. Puppet excels among gener- alist models and remains highly competitive on R@ R@3, and MMDist compared to specialist models. This highlights our model's 's ability to generate motion sequences that are better aligned with text descriptions and of higher quality.
Does increasing the data scale benefit motion generation? Yes. As illustrated in Table 2, when using the same foun- dation model, increasing the scale of training data leads to substantial improvement on MotionLib testing set. aligning with our expected scaling laws. When using I.LaMA2-13h as the LLM backbone for our model, training with 1.2M data achieves an R @@I of O. 1855, which is 0.011 I and 0.061 higher than performance using 0.5M data and 0.08M Motion-X, respectively. This improvement is particularly pronounced in the R-precision metric, emphasizing the critical role of data scale in enhancing semantic alignment between gen- erated motions and text prompts. It's worth to note that the performance of models trained on Motion-X also im- proves, after pretraining on MotionLib. This demonstrates that large-scale pretraining allows the model to have better initialization for motion- languag alignment.
Does large motion model excel in out-of-distribution setup? Yes. We present the results in Table 4. This abla- tion essential for further validating the true generalization
7

---

Scaling Large Motion Models with Million-Level Human Motions
VD 20-UFQ
12
2D
M2yPE
MrPE
22
27
22
212
211
21
27
2.2
211
222
2.5
Codebnol Sone
Corleboni: Size
Figure 4: Comparison with different motion quantization on Motion-X (left) and MotionLib (right). We only show MPJPE (I) results here due to space limitation, with FID results shown in Figure 12.
Table 4: Ablation of out-of-domain evaluation on UNSEEN- 90K dataset, where # N denotes we use N subsets of Mo- tionLib for training.
Table 5: Ablation results of motion instruction tuning.
TRAIN SET
R@ 1
R@31
FID
Real
0.514
0.831
0,046
TRAIN SET
R@I 1
f
R@31 f
FID 4
Pretrain
0.471 0.488
0.788
0,103
Real
0.176
0,379
0.076
Instruction tuning
0.821
0,093
H33
0.034
0.112
82.674
MotionX
0.051
0.141
70.547
Table 6: Ablation results of hicrarchical description vs. single-level description.
MotionLib-411
0.098
0.218
11.930
TRAIN TEXT
R@1 1 f
R@3 f
FID +
capabilities of large motion models, as the improvements observed in Table 2 may result from the inclusion of addi- tional in-domain data similar to testbeds. In this setup, we select 11 subsets from MotionLib, totaling 90K samples (UNSEEN-90K), for evaluation. The remaining subsets are used for training. These 11 subsets include both syn- thetic and real-world data. They cover domains like fitness, person-object interaction, social activities, and diverse envi- ronments such as labs and outdoors. The training set, on the other hand, primarily consists of Motion-X and web-derived data. This ensures that the test set is entirely composed of out-of-domain (OOD) samples. We compare the per- formance of models trained on HumanML33, Motion-X, and Motion-#1 1, all utilizing the GPT-2 architecture, where #N denotes the number of training subsets. The results the OOD test set clearly demonstrate that the model trained on MotionLib significantly outperforms those trained on HumanMI.3D and Motion-X, particularly in terms of R @ and R @ metrics. These findings strongly highlight the superior generalization ability of large motion models when handling unseen OOD data, especially when trained on di- verse, large-scale datasets. This demonstrates the value of utilizing large amount of web-based motion data.
Real
0.297
0.634
0.004
single-level
0.162
0.371
7.018
hierarchical
0.166
0,375
6.936
Table 7: Ablation results different motion features. Here, "FPS" denotes the speed recover original motion informa- tion (e.g., rotation).
Motion Feat
R@
R@31 f
FPS 1
H3D-D263
0,514
0.831
0.41
SMPL-D130
0.517
0.851
>10
SMPL-D135
0.529
0.850
>100
SMPL-D223
0.521
0.843
>100
SMPL.-D268
0.523
0.855
> 1100
and FID increased by 0.010. This indicates a benefit for the model's further language-motion alignment capability. Instruction tuning also makes the model more user-friendly, proving a worthwhile optimization.
Does richer text description help? Yes. To investigate the effectiveness of richer text descriptions, we compared the training results using hicrarchical descriptions, which include both body-level and part-level labels, against those using only whole-body description. As shown in Table 6, training with hierarchical descriptions improves both @@1
Does motion instruction tuning help? Yes. We compare experiments with and without instruction tuning on Motion- X. As shown in Table 5, instruction tuning data effectively improves various metrics, with @@ increased by 0.017
8

---

Scaling Large Motion Models with Million-Leve Human Motions
Table 8: Ablation results of different motion tokenizer trained on HumanML3D on the motion reconstruction task.
M1manM1.3D
Motion-X
Motionl. .ib
Tokenizer
#Num.
#Param,
FID
MPJPE L
FID 4
MPJPE
FID 1
MPIPE
VQ-VAB
512
19.43M
0.078
69.2
0.852
106.4
5.324
123.6
HQQQ
512
62.34
RQ-VAE
512
19.43M
0.052
37.5
0.568
56.9
4.026
78.2
2D-LFQ
16384
108.35M
0.092
45.6
0.295
54.1
2.315
64.1
and R@3 by 0.004 compared single-level descriptions. These results indicate that hierarchical text significantly enhances the model's semantic understanding, leading to improved semantic matching of generated motions.
of 2D-LFQ continues increase with larger codehooks.
6. Conclusion
In this paper, we explore how to advance the field of large motion model. To address the issue of data scarcity in this domain, we introduce MotionLib, the first million-level dataset comprising over 1.2 million motions with hierarchi- cal texts. Building MotionLib, we present key insights into scaling up both data and model size for large-scale motion training. Furthermore, we propose MotionBook, novel motion encoding approach designed to maximize the benefits when trained on extensive motion data. Motion- Book incorporates compact yet lossless features to represent motion, and introduces novel 2D-LFQ motion quantiza- tion method that treats each motion sequence 2D image, constructing finite-scale codebook that eliminates the need for token lookups. Leveraging these advancements, we train Being-MO, a large motion model that achieves SoTTA results compared to current counterparts.
5.3. Effectiveness of Motionbook
Motion Feature Design. We conduct an ablation study to evaluate the impact of different motion feature design, including H3D- Format, SMPL-D130, SMPL-D135, SMPL- D263, and SMPL-D268 (detailed Appendix A.3). As shown in Table 7, retrieval performance remains consistent across all features, with significant differences. Similarly, quantitative evaluations of rendered samples reveal no ticeable variation among different features. This indicates that none of the tested features provide clear performance advantage. However, we observe that -orm-ot discards some original motion information, as its rotations are ob- tained through inverse kinematics. This requires additional time-consuming processing restore missing details- an issue that complicates downstream applications. Consider- ing simplicity, computation cost, and information complete- ness, we finally select SMPL-D135 35 as the motion feature.
Acknowledgements
2D-LFQ Motion Tokenization. We compare our proposed 2D-LFQ against three quantization approaches: vanilla vec- tor quantization (VQ), residual vector quantization (RVQ), and H2VQ (Lu al., 2023)*, across various codebook sizes ranging from 2 to 26 The number of parameters for RVQ/VQ and 2D-LFQ are 19.43M and 108.35M, respec- tively. To evaluate the generalization ability of each alter- native, all results are trained on HumanMI.33 3D dataset. As shown in Table 8, 2D-LFQ outperforms its counterparts on of and larger datasets like Motion-X and MotionLib. In addition, 2D-LFQ continues to enhance per- formance as the codebook size increases as illustrated in Figure 4, while RVQ and VQ experience diminishing re- turns or performance degradation with larger codebooks. Our deeper analysis attributes these gains to better codebook utilization by 2D-LFQ. Figure 11 (RIGHT) illustrates that the utilization rates for VQ and RVQ begin to decline once the codebook size exceeds 20 which corresponds to the peak performance for these methods, whereas the utilization
This work was supported by NSFC under Grant 62450001. The authors would like to thank the anonymous reviewers for their valuable comments and advice.
Impact Statement
This paper introduces MotionLib, the first million-scale motion dataset, and a large motion model named Being-MO trained upon it. This work significantly advances the field of large motion models and lays foundation for future research. It aims to improve a model's understanding and generation of human motion, positively impacting areas like gaming and robotics. In terms of ethical considerations, all motion data comes from open-source and online videos, we will carefully review examples before release to avoid any potential ethical limitations such as copyright issue. We acknowledge the positive implications of such technological ad advancements but also recognize potential risks, like misuse for misleading content. We are committed to responsible development and will continuously monitor the broader societal impact of this technology.
Onnly reported number is shown (MPJPE on Motion-Xi -), since no official code or successful re- re-implementation has been found.

---

Scaling Large Motion Models with Million-Level Human Motions
References
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang. A., Fan, A., et al. The llama 3 herd of models. arXiv preprint arXiv:240.21188,, 2024.
Ahn, H., Ha, T., Choi, Y., Yoo, H., and Oh, S. Text2action: Generative adversarial synthesis from language to action. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pp. 5915-5920. IEEE, 2018.
Feng, Y., Li, Y., Zhang, W., Luo, H., Yue, Z., Zheng, S., and Lu, Z. Videoorion: Tokenizing object dynamics in vidcos. arXiv preprint arXiv:
Ahuja, C. and Morency, L.-P. Langu Natural lan- guage grounded pose forecasting. 2019 International Conference on 3D Vision (3DV), pp. 719-728. IEEE, 2019.
2024.
Fragkiadaki, K., Levine, S., Felsen, P., and Malik, J. Recur- rent network models for human dynamics. In Proceedings of the IEEE international conference on computer vision, pp. 4346-43555 2015.
Aliakbarian, S., Saleh, F. S., Salzmann, M., Petersson, L.., and Gould, S. A stochastic conditioning scheme for diverse human motion prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5223-5232, 2020.
Ghosh, P., Song, J., Aksan, E., and Hilliges, O. Learning human motion models for long-term predictions. In 20/7 International Conference on 3D Vision (3DV), pp. 458- 466. IEEE, 2017.
Bogo, F., Kanazawa, A., Lassner, C., Gehler, P., Romero, J., and Black, M. J. Keep it smpl: Automatic estimation of 3d human pose and shape from single image. In Computer Vision-ECCV 2016: 14th European Confer- ence, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part 14, pp. 561-578. Springer, 2016.
Guo, C., Zou, S., Zuo, X., Wang, S., Ji, W., Li, X., and Cheng, L.. Generating diverse and natural 3d human motions from text. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5152-5161, 2022a.
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in neural information processing systems, 33; 1877-1901, 2020.
Guo, C.,, Zuo, X., Wang, S., and Cheng, Tm2t: Stochastic and tokenized modeling for the reciprocal generation of 3d human motions and texts. In European Conference on Computer Vision, pp. 580-597. Springer, 2022b.
Guo, C., Mu, Y., Javed, M. G., Wang, S., and Cheng, Momask: Generative masked modeling of 3d human motions. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 1900 1910, 2024.
Cervantes, P., Sekikawa, Y., Sato, I., and Shinoda, K. Im- plicit neural representations for variable length human motion generation. In European Conference on Computer Vision, pp. 356 372.332.555 Springer, 2022.
Gupta, A.,
Tian, S., Zhang, Y., Wu, J., Martin- R., and Fei-Fei, L. Maskvit: Masked visual pre-training for video prediction. arXiv preprint arXiv:2206. /1894, 2022.
Chen, X., Jiang, B., Liu, W., Huang, Z., Fu, B., Chen, T., and Yu, G. Executing your commands via motion diffusion in latent space. In Proceedings of the IEEE/VFF Conference on Computer Vision and Puttern Recognition, pp. 18000- 18010, 2023.
Holden, D. http://thooanggduuucc.ooogggagnn quality. In blog. 2024.
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. Palm: Scaling language modeling with pathways. arXiv preprini 2rXiv:2204.023111 2022.
Jiang, B., Chen, X., Liu, W., Yu, J., Yu, G., and Chen, T. Motiongpt: Human motion as a foreign language. Advances in Neural Information Processing Systems, pp. 20067-20079, 2023.
Chung. J., Wuu, C.-h., Yang, H.r.,, Tai, Y.-W., and Tang, C.-K. Haa500: Human-centric atomic action dataset with curated videos. In Proceedings of the IEEE/CVF international conference computer vision, pp. 13465- 14474,2021,
Kay, W., Carreira, J., Simonyan, K., Zhang, B., Hillier, C., Vijayanarasimhan, S., Viola, F., Green, T., Back, T., Natsev, P., et al. The kinetics human action video dataset. arXiv preprint arXiv: 70550695, 2017.
Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A. C., Lo, W.- et al. Segment anything. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4015-4026, 2023.
Dai, w. Li, J. Li, D., Tiong, A. M. H., Zhao, J., Wang, W., Li, B., Fung, P., and Hoi, S. Instructblip: Towards ceneral-purpose vision- laiiun-anguuae models with instruction tuning. arXiv preprint arXiv:2305 065000 2023.
10

---

Scaling Large Motion Models with Million-Level Human Motions
Lee, D., Kim, C., Kim, S., Cho, M., and Han, W.-S. Autore- gressive image generation using residual quantization. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 11523-11523 2022.
Mentzer, F,, Minnen, D., Agustsson, E., and Tschannen, M. Finite scalar quantization: Vq-vae made simple. arXiv preprint arXiv:2309. 15505, 2023.
OpenAT. GPT-4o mini: advancing cost-efficient intelli- gence. http://ppnaiicoomm////////t--/// -iia-m-aiaidddnii------- cost-efficeent-imelliggnneeeeee 2024.
Li, B., Zhang, Y., Chen, L., Wang, J., Pu, F., Yang. J., Li, C., and Liu, Z. Mimic-it: Multi-modal in co-tent instruction tuning. arXiv preprint 254455,300. 2023a.
Ouyang. I.., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P. Zhang, C., Agarwal, S., Slama, K., Ray, A., al. Training language models follow instructions with human feedback. Advances in neural information processing systems, 15:27730-277444 2022.
Li, J., Li, D., Savarese, S., and Hoi, S. Blip-2: Boot- strapping language-inan image pre-training with frozen im- age encoders and large language models. arXiv preprint 0rXiv:2301.12555 2023b.
Lin, J., Zeng, A., Lu, S., Cai, Y., Zhang, R., Wang, H., and Zhang, L. Motion-x: large-scale 3d expressive whole-body human motion dataset. Advances in Neural Information Processing Systems, 36, 2024.
Pavlakos, G., Choutas, V., Ghorbani, N., Bolkart, T., Osman, A., Tzionas, D., and Black, M. J. Expressive body capture: 3d hands, face, and body from a single image. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10975-10985, 2019.
Liu, H., Li, C.,, Wu, Q. and Lee, Y. J. Visual instruction tuning. arXiv preprint arX.v:2304.084444 2023.
Petrovich, M., Black, M. J., and Varol, G. Tmr: Text- to-motion retrieval using contrastive 3d human motion synthesis. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9488 2023,
Liu, Z., Wu, S., Jim, S., Ji, S., Liu, Q., Lu, S., and Cheng. L. Investigating pose representations and motion contexts modeling for 3d motion prediction. IEEE transactions on paltern analysis and machine intelligence, 45(1):681- 697, 2022.
Pi, H., Peng, S., Yang, M., Zhou, X., and Bao, H. Hi- erarchical generation of human-object interactions with diffusion probabilistic models. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 15061-15073, 2023.
Lou, Y., Zhu, L., Wang, Y., Wang, X., and Yang, Y. Di- versemotion: Towards diverse human motion generation via discrete diffusion. arXiv preprint arXiv:2309.01322 2023.
Plappert, M., Mandery, C., and Asfour, T. The kit motion- language dataset. Big data, 4(4):236-22,, 2016.
Lu, S., Chen, L.-H., Zeng, A., Lin, Zhang, R., Zhang, L., and Shum, H.-Y. Humantomato: Text-aligned whole- body motion generation. In Forty-first International Con- ference on Machine Learning, 2023.
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al. Language models are unsupervised multitask learners. OpenAl blog, 18))99
Luo, H., Ji, L., Shi, B., Huang, H., Duan, N., Li, T., Li, Bharti, T., and Zhou, M. Univl: unified video and lan- guage pre-training model for multimodal understanding and generation. arXiv preprint ar XXiv:2002.063533 2020.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763. PMLR, 2021.
Luo. Z., Cao, J., Kitani, K., Xu, W., al. Perpetual hu- manoid control for real-time simulated avatars. In Pro- ceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10895-10904, 2023.
Raffel, C., Shazeer, N. Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. Exploring the limits of transfer learning with unified text-to-text transformer. The Journal of Machine Learning Research, 21()):5485-5511. 2020.
Mahmood, N., Ghorbani, N., Troje, N. F., Pons-Moll, G., and Black, M. J. Amass: Archive of motion capture as surface shapes. In Proceedings of the IEEE/CVF inter- national conference computer vision, pp. 5442-5451, 2019.
Reid, M., Savinov, N., Teplyashin, D., Lepikhin, D., Lilli- crap. T., Alayrac, J.-b., Soricut, R., Lazaridou, A., Firat, O., Schrittwieser, J., et al. Gemini 1.5: Unlocking multi- modal understanding across millions tokens of context. arXiv preprint arXiv:2403.055. 2024.
Mei, Y., Wang, Y., Zheng, S., and Jin, Q. Quadrupedgpt: To- wards a versatile quadruped agent in open open-ended worlds. arXiv preprint arXiv:2406. 16578, 2024.

---

Scaling Large Motion Models with Million-Level Human Motions
Sárándi, I., Hermans, A., and Leibe, B. Learning 3d human pose estimation from dozens of datasets using geometry- aware autoencoder to bridge between skeleton formats. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 2956-2966, 2023.
generators by learning smooth latent transitions. In Pro- ceedings of the AAAI conference on artificial intelligence, volume 34, pp. 12281-12288, 2020.
Wu, Q., Zhao, Y., Wang. Y., Tai, Y.-W., and Tang, C.-K. Motionllm: Multimodal motion- motion-language learning with large language models. arXiv preprint arXiv:2405.17013. 2024.
Shin, S., Kim, J., Halilaj, E., and Black, M. J. Wham: Reconstructing world- gondddgrounde humans with accurate 3d motion. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition. pp. 2070- 2080, 2024.
Xu, Y., Zhang, J., Zhang, Q. and Tao, D. Vitpose: Simple vision transformer baselines for human pose estimation. Advances in Neural Information Processing Systems, 35: 38571-38584, 2022.
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., and Rabinovich, A. Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1-9, 2015.
Yan, w., Zhang, Y., Abbeel, P., and Srinivas, A. Videogpt: Video generation using vq-vae and transformers. arXiv preprint arXiv:2104.1013: 10157, 2021.
Taheri, O., Ghorbani, N., Black, M. J., and Tzionas, D. Grab: A dataset of whole-body human grasping of objects. In Computer Vision-ECCV 2020: 16th European Confer- ence, Glasgow, UK, August 23-28, 2020, Proceedings, Part IV 16, pp. 581 180- Springer, 2020.
Ye, Q., Xu, H., Xu. G., Ye, J., Yan, M., Zhou, Y., Wang. J., Hu, A., Shi, P., Shi, Y., al. mplug-owl: Modulariza- tion empowers large language models with multimodality. arXiv preprint 2rXiv:2304.1417, 2023.
You, T., Kim, S., Kim, C., Lee, D., and Han, B. Locally hi- erarchical auto-regressive modeling for image generation. Advances in Neural Information Processing Systems, 35: 16360-16372, 2022.
Tevet, G., Gordon, B., Hertz, A., Bermano, A. H., and Cohen-Or, D. Motionclip: Exposing human motion gener- ation to clip space. In European Conference on Computer Vision, pp. 358-374. Springer, 2022.
Yu, L.,, Lezama, J., Gundavarapu, N. B., Versari, L.., Sohn, K., Minnen, D., Cheng, Y., Gupta, A., Gu, X., Haupt- mann, A. G., et al. Language model beats diffusion- tokenizer is key to visual generation. arXiv preprint arXiv:2310.05733 2023.
Touvron, H., Martin, L.., Stone, K., Albert, P., Almahairi, A., Babaci, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine- tuned chat models. arXiv preprint arXiv:2307.09288. 2023.
Yuan, Y., Iqbal, U., Molchanov, P., Kitani, K., and Kautz, J. Glamr: Global occlusion-aware human mesh recovery with dynamic cameras. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pp. 11038-11049, 2022.
Van Den Oord, A., Vinyals, O., et al. Neural discrete rep- resentation learning. Advances in neural information processing systems, 30, 2017.
Voas, J., Wang, Y., Huang, Q. and Mooney, R. What is the best automated metric for text to motion generation? In SIGGRAPH Asia 2023 Conference Papers, pp. 1-11, 2023.
Zhang, J., Zhang, Y., Cun, X., Zhang, Y., Zhao H., Lu, H., Shen, X., and Shan, Y. Generating human motion from textual descriptions with discrete representations. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 14730-147400 2023a.
Wang, Y., Huang, D., Zhang, Y., Ouyang, W., Jiao, J., Feng, X., Zhou, Y., Wan, P., Tang, S., and Xu, D. Motiongpt-2: A general-purpose motion-language model for motion generation and understanding. arXiv preprint arXiv:2410.21747, 2024.
Zhang, M., Cai, Z., Pan, L.,, Hong, F,, Guo, X., Yang, I., and Liu, Z. Motiondiffuse: Text-driven human mo- tion generation with diffusion model. arXiv preprint arXiv:2208. htth:22008/0011 15001, 2022.
Wang, Y., Li, M., Liu, J., Leng, Z., Li, F. W., Zhang, Z., and Liang, X. Fg-(2m++: Llms-augmented fie-grained text driven human motion generation. International Journal of Computer Vision, pp. 1-17,201 2025.
Zhang, M., Guo, X., Pan, I.., Cai, Z., Hong, F,, Li, H., Yang, L., and Liu, Z. Remodiffuse: Retrieval-augmenned motion diffusion model. In Proceedings of the IEEE/CVF International Conference on Computer Vision pp. 364 373, 2023b.
Wang, Z., Yu, P., Zhao, Y., Zhang, R., Zhou, Y., Yuan, J., and Chen, C. Learning diverse stochastic human-action
12

---

Scaling Large Motion Models with Million-Leve Human Motions
Zhang, M., Jin, D., Gu, C., Hong, E,, Cai, Z., Huang, J., Zhang, C.,, Guo, X., Yang, L.,, He, Y., et al. Large motion model for unified multi-modal motion generation. In European Conference on Computer Vision, pp. 397-421. Springer, 2024a.
Zhang, S., Bhatnagar, B. I.,, Xu, Y., Winkler, A., Kadlecek, P., Tang, S., and Bogo. F. Rohm: Robust human mo- tion reconstruction via diffusion. In Proceedings of the JEEE/CVE Conference on Computer Vision and Pallern Recognition, pp. 14606-14617, 2024b.
Zhang. W., Xie, Z., Feng, Y., Li, Y., Xing, X., Zheng, S., and Lu, Z. From pixels to tokens: Byte-pair encoding on quan- tized visual modalities. arXiv preprint arXiv:2410.02155, 2024c.
Zhang, Y., Zhang, R., Gu, J.. Zhou, Y., Lipka, N., Yang, D., and Sun, T. Llavar: Enhanced visual instruction tuning for text-rich image understanding, arXiv preprint arXiv:2306. 17107, 2023c.
Zhang, Y., Huang, D., Liu, B., Tang, S., Lu, Y., Chen, L., Bai, L., Chu, Q. Yu, N., and Ouyang, W. Motiongpt: Finetumed Iims are general-purpose motion generators. In Proceedings of the AAA/ Conference on Artificial Inteli- gence, volume 38, pp. 7368-7376 7376, 2024d.
Zheng, S., Zhou, B., Feng. Y., Wang, Y. and Lu, Z. Unicode: Learning unified codebook for multimodal large lan- guage models. arXiv preprins arXiv:2403.09072, 2024.
Zhou, Z.,, Wan, Y., and Wang, B. Avatargpt: All-in-one framework for motion understanding planning generation and beyond. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1357- 1366, 2024.
13

---

Scaling Large Motion Models with Million-Level Human Motions
Appendix
Roadmap. In this appendix, we first provide the additional details of evaluation, model implementation and motion feature design Section A. Then, we introduce additional details our dataset MotionLib and its construction process in Section B. Finally, we carry out thorough experiments and responding analysis in Section C.
Table of Contents
Details of Evaluation and Implementation
15
A.I Evaluation Metrics
15
A.2 Implementation Details
15
A.3 Design of Different Motion Features
15
B Details of Dataset - MotionLib
15
B.1 Motion Data Collection
15
B.2 Motion Data Refinement
16
B.3 Motion Description Generation
17
B.4 Motion Description Quality Assessment
17
B.5 Statistic Analysis of Data and Word Distribution
19
Additional Experimental Analysis
20
C.1
Ablation of Large Motion Model Training
20
C.1.1 LoRA vs. Full Parameter Fine-tuning
20
C.1.2
Learning from Scratch Fine-tuning
20
C.1.3
With vs. Without Description Mask
21
C.1.4
Encoder-decoder vs. Decoder-only.
22
C.1.5
Slow convergence large motion models
23
C.2
Additional Experiments of Motion Quantization
23
C.2.1 Additional FID Results Motion-X
23
C.2.2 Ablation of 2D-LFQ vs. ID-LFQ
24
C,3 Discussion of Static and Synthetic Data.
24
C,4 Discussion of Current Evaluation Metric's Limitation
24
C.5 Additional Qualitative Results
26
14

---

Scaling Large Motion Models with Million-Level Human Motions
A. Details of Evaluation and Implementation
A.1. Evaluation Metrics
We employ the following metrics to assess the quality and alignment of generated motions: (1) Frechet Inception Distance (FID): This metric evaluates the overall motion quality by comparing the distributional differences between high-level features of generated and real motions. (2) Motion-retrieval Precision (R-Precision) and Multimodal Distance (MMDist): These metrics measure the semantic alignment between textual inputs and generated motions. R-Precision evaluates retrieval accuracy at top-1, top-2 and top-3 levels, while MMDist quantifies the distance between matched text and motion pairs. To validate the effectiveness of our motion tokenizer, we also perform experiments on the motion reconstruction task. This task is measured using both Mean Per Joint Position Error (MPJPE) and FID, where MPJPE computes the average distance (in millimeters) between predicted and ground-truth joint positions across all skeleton joints.
A.2. Implementation Details
For a fair comparison with previous works, we implement oue model Being-MO based on two varionts of motion tokenizers: one with a vector quantized (VQ) codebook and another with a 2D lookup-free (2D-LFQ) codebook. By default, our Being-MO is trained using 2D-LFQ. For the motion tokenizer, we implement the VQ codebook C E R0024x512 with an embedding dimensionality of 512. The resulting discrete codes are incorporated as additional vocabulary for the LLM. As comparison, the I.QQ codebook has size of 2 16384. The motion encoder E.. uses temporal downsampling rate of a = 4. We experiment with four large language model (LLM) architectures to construct our large motion model: GPT2-medium (Radford et al., 2019), LLaMA2-7b, LLaMA2- 13b (Touvron et al., 2023), and LLaMA3. 1-8b (Dubey et al., 2024). The motion tokenizer is trained with a learning rate of Ic-4 and a batch size of 256 for 300K iterations. For training the large motion model, full parameter tuning is performed on 8 x A800 GPUs with a batch size of 1024 over 100 epochs. The learning rate is set to 2e-4 for GPT2-medium and 2e-5 for the LLaMA models.
A.3. Design of Different Motion Features
In this section, we provide a detailed introduction of diverse motion feature formats used in our experiments to highlight their key differences:
H3D-Format: This format proposed by Guo al. (2022a) and is widely used by most recent motion generation works. H3D-Format includes relative joint positions (63 dimensions), relative 6D rotations of key joints (126 dimensions), joint velocities (66 dimensions), and foot contact information (4 dimensions). Rotation information is derived from position data using Inverse Kinematics (IK), which may result in the loss of original rotational details. The root node parameters consist of 4 dimensions: 1 for r-rotation (angular velocity), 2 for xz-velocity, and for y-height.
SMPI-DD130 This format uses relative 6D rotations of key joints (126 dimensions) and root node parameters (4 dimensions). The root node parameters include dimension for r-rotation, 2 for xz-velocity, and for y-height.
SMPL-D263: Building on SMPL-D130, this format adds redundant position features (derived from forward kinematics of the SMPL model) and 4 dimensions of foot contact information. The root node parameters remain the same as in SMPL-D130.
SMPL-D135: This format employs relative 6D rotations of key joints (126 dimensions) and 9-dimensional root node parameters. The root node parameters include dimensions for 6D root rotation, 2 for xz-velocity, and for y-height.
SMPL-D268: Extending SMPL-D135, this format incorporates redundant position features (identical to SMPL-DD263 and foot contact information. The root node parameters are consistent with those in SMPL-DI35.
B. Details of Dataset - MotionLib
In this section, we provide a detailed introduction of how MotionLib, the million-level motion generation dataset, is constructed, with more insights into its characteristics and additional data examples shown in Figure 5.
B.1. Motion Data Collection
We begin by elaborating on the process of extracting raw motion data from web videos.
15

---

Scaling Large Motion Models with Million-Level Human Motions
Figure 5: Illustration of examples in MotionL ib, each sample is motion sequnce extracted from an online video.
Short Boundary Detection. We observe that many segments in these videos lack human presence, necessitating the identification and extraction of portions relevant to human motion. To achieve this, we follow a structured step-by-step procedure: For videos shorter than 30 seconds or those with clearly defined temporal boundaries, we either use the entire clip directly or segment based on the provided boundaries. For videos longer than 30 seconds, we first apply scene detection model to divide the video into coarsely segmented parts. Each segment is then refined into shorter clips through the following steps: (1) At the start of segment, we identify the human with the largest bounding box as the anchor and track their trajectory. (2) If the trajectory is interrupted, the start and end times of the interruption define new clip boundary. (3) The process repeats by selecting the next largest visible human in subsequent frames and tracking their trajectory. (4) This continues until no humans remain visible in the segment. (5) Clips without visible humans are discarded. If a resulting clip exceeds 60 seconds, we randomly divide it into sub-elips, ensuring each is shorter than one minute.
Occlusion and Blur Filtering. Occlusion and motion blur are common challenges in human- human-related related videos. To mitigate these issues, We first sample key frames from each clip and apply a pretrained 2D keypoint detector extract skeleton keypoints for each human. If significant portion of the keypoints has confidence scores below predefined threshold, the motion is considered occluded and excluded from further processing. Additionally, we leverage a visual foundation model, such as Segment Anything (Kirillov et al., 2023), to generate segmentation masks for each frame. If a large object is detected obstructing the human, the corresponding motion data is filtered out. To address motion blur, we track the trajectory of each human whose motion data is being extracted. For timestamps where keypoint confidence scores are low, we smooth the trajectory using adjacent detection results, ensuring continuity and accuracy.
B.2. Motion Data Refinement
To ensure physically plausible motion data, we refine the extracted motion using an RL-based policy (Luo et al., 2023). Before this refinement step, we enhance motion quality following the methodology of Lin et al. (2024). We begin by
16

---

Scaling Large Motion Models with Million-Level Human Motions
estimating 2D human keypoints and their confidence scores using the pretrained VitPose model (Xu al., 2022), then infer 3D keypointss via another pretrained 3D keypoint estimation model (Sárándi et al., 2023). To improve stability and consistency, we apply temporal smoothing and enforce 3D bone length constraints during triangulation. As previously mentioned, we fit the SMPL-X body model (Pavlakos et al., 2019) to each frame in MotionLib using the approach from Shin et al. (2024), followed by iterative optimization to align model parameters with the estimated 2D and 3D joint positions. We further refine global motion and camera poses using a global motion optimization technique based on Yuan et al. (2022), ensuring consistency with the original video evidence. For motion sequences affected by noise or occlusion, we employ RoHM (Zhang al., 2024b) to reconstruct complete and physically plausible motions.
B.3. Motion Description Generation
In this paper, we employ Gemini-1.5-PPr (Reid et al., 2024) as a large multimodal model (LMM) to generate text labels for motion data. For each video clip, we first crop and track the human body using the corresponding bounding boxes. The LMMs are then tasked with analyzing the human's physical movements and their spatial positions within the global space to produce detailed motion descriptions. A key distinction from previous datasets is the granularity of these descriptions. Instead of simply generating an overall description of the human's 's movements, we prompt the LMMs to focus on specific body parts by dividing the body into upper-body and lower-body sections, which enables the generation of part-specific descriptions (referred to as "part-level" descriptions). Figure 6 illustrates the corresponding used prompt. Additionally, a comprehensive summary of the whole body's 's movements (referred to as "body-level" descriptions) is also included.
Table 9: Text quality evaluation of different datasets. We use both "text-only" and "visual-text align" to score the text description quality. Here, the score is produced by GPT-4o.
Eval Strategy
HLmanML3D
MotionX
MotionLib
Text-only
1.386
1.703
3,837 (+2.134)
Visual-text align
3.081
2.252
3.823 (+0.742)
B.4. Motion Description Quality Assessment
To ensure the quality of text descriptions, we propose two evaluation strategies. Table 9 compares the results of our MotionL.b with HumanML.3D and Motion-X.
Text-only Evaluation. We first conduct automated evaluation of motion descriptions in MotionLib using GPT-4o- mini (OpenAI, 2024). Unlike the Gemini model used for text generation, we employ a different LMM (GPT-4o) as the evaluator to mitigate model bias. Each motion description is rated on a 1-to-5 scale based on the following criteria:
point (very poor): Vague, irrelevant to the motion, or contains severe grammatical errors.
points (poor): Lacks detail, specificity, or contains clear inaccuracies.
3 points (fair): Broadly reflects the motion but lacks depth and may have minor errors.
points (good): Accurate, detailed, and clearly conveys the motion process.
5 points (excellent); Precise, comprehensive, and fluent, providing in-depth motion analysis.
ollowing the same methodology, we evaluate descriptions from HumanML33D and Motion-X. As shown in Table 9, MotionLib achieves an average score of 3.837, significantly outperforming Motion-X (1.386) and HumanML3D (1.703). These results underscore the superior quality of MotionLib's motion descriptions.
Visual-text Alignment Evaluation. Simply relying on text-based input to generate score may not be ideal, as LLMs are prone to hallucination when they lack visual guidance to contextualize the content. To address this, we pair text descriptions with their corresponding rendered motion videos, and feed them into the GPT-4o for scoring. Here, we extract key frames from the rendered video, concatenating them into a larger image as the input of GPT-4o, considering GPT-4o does not accept videos as input. GPT-4o will evaluate the alignment between motion descriptions and rendered visual content, and output a
17

---

Scaling Large Motion Models with Million-Leve Human Motions
Begin by providing general overview the person's current action (c.g. walking. sitting, interacting) within the BBOX area. Then, proceed focusing exclusively
detailed breakdown.
the physicul movements and positions of the person within the
BBOX. For the upper body, describe the position and movement of the arms, hands, shoulders, and torso. For the lower body, detail the position and movement of the legs, feet, and overall balance. Ensure the description strictly covers physical actions without mentioning facial expressions, clothing, environmental elements outside the BBOX. Example:
The person is standing still, observing something in front of them.
Upper body; Their arms hang relaxed by their sides, with the shoulders slightly back and the chest open. The torso upright, with minimal movement, indicating calm, neutral stance.
Lower body; Both feet are planted firmly on the ground, shoulder-width apart. The knees are slightly bent, and their weight is evenly distributed between both legs.
The person is standing within the designated area, engaging in conversation seemingly directed toward someone positioned off-camera the left.
AI
**Upper Body:**
** **Arms:** Initially held loosely the sides, the arms transition to various positions throughout the interaction. At times, they rise chest level with palms open, suggesting an explanatory gesture. Occasionally, one or both arms extend outwards, indicating direction or emphasis.
**aadds:** ** Hand movements correspond with arm gestures. Palms face upwards and outwards during open-handed motions, then relax to neutral position when the
**Shoulders:** Shoulders remain relatively relaxed throughout, with subtle shifts in position reflecting the arm movements. They don't appear tense or raised, implying a generally comfortable stance.
**Torso: +* The torso largely remains stationary, facing forward, with slight turns coinciding with the shifting weight distribution of the lower body.
*Lowcr Bo
**Less*** Legs maintain comfortable stance, slightly apart, with the weight appearing balanced. There's subtle shift in weight distribution they adjust their stance.
**Fect:** Feet remain planted on ground, primarily shoulder width apart.
The positioning suggests grounded and stable stance.
**Overall Balance:* The individual appears balanced and at case throughout the interaction, with movements suggesting engagement in the conversation rather than discomfort restlessness.
Figure 6: The Prompt template to generate part-level motion description in videos based on powerful large multimodal models (LMMs), such as Bemini--155ppt and GPT-4o-mini. For each sample MotionLib, we provide "body-level" (UP) and "part-level" (DOWN) labels to distinguish between whole-body and partial motion descriptions.
18

---

Scaling Large Motion Models with Million-Level Human Motions
Motion Count per Subset(Sorted)
prox
27
3DPW
dence
163
behave
299
animation
329
perform
474
humans3d
688
humman
744
RICH
EgoBody
960
kungfu
1035
GRAB
1335
eist
1470
1504
Dalaset
Track
arctic
1666
1806
MPI-INF-3OHP
2240
chi3d
music
3552
HAA500
5231
5770
game_ motion
game_motion
10217
12454
fitness
16718
talkshow
human
19101
humanml
29228
BEDLAM
30125
activitynet
70150
BADEL
114596
ntt_rgbd_122 rg0d_120
137514
Kinetics-700
698518
10
10
10
10a
Motion Count (log scale)
Figure 7: The data scale distribution of motion sequences for different subsets in MotionL.ib
score following the same criteria above. As result, our MotionLib achieves average score of 3.823, while Motion-X and HumanML3D score 2.252 and 3.081, respectively, again confirming the quality advantage of MotionLib's 's text descriptions.
B.5. Statistic Analysis of Data and Word Distribution
Data Distribution. MotionLib comprises 1.2 million motion sequences collected from various public datasets and web videos. A significant portion of MotionLib is derived from open-source, human- related datasets, such 698.5K motions from Kinetics-700 (Kay et al., 2017) and 137.5K motions from NTU-RGED- 120 (Lin et al., 2024). Additionally, MotionLib integrates motions from other established datasets, including BEDLAM and GTA--uumn. MotionLib also includes subsets of the Motion-X collection, covering a diverse range of categories such as Animation, Perform, Dance, AIST, Kungfu, GRAB (Taheri al., 2020), Music, Idea400, HAA500 (Chung et al., 2021), Game Motion, and Fitness. It is worth noting that the Motion-X subsets constitute only small portion of the overall MotionLib dataset (around 6.7%). Figure 7 illustrates the scale distribution of motion sequences within the subsets of MotionLib. We also count the average frame number of each subset, shown in Figure 8. Given the high cost of collecting and annotating video data, we also recognize the untapped potential of images for motion understanding. To explore this, we collected approximately 600K images and extracted human poses, which were repeated across 64 frames and treated as motion sequences. While using static data for dynamic motion generation remains a controversial topic, this static data is not included in the 1.2 million motion sequences of Motionl Lib in the main paper and not claimed as part our primary contributions. Nevertheless, we conduct experiments with this static data and hope it will inspire future research into the potential of static data for dynamic motion understanding, can be seen in Apppendix C.3.
Word Distribution. To further explore the annotated motion text, compute the word cloud from the entire text corpus in MitionLib. The word cloud of body-level and part-level descriptions can be seen in Figure 9 and Figure 10, respectively. In Figure 9, we observe that the body-level texts primarily highlight the high-level human activities, such as standing, sitting, and walking. In contrast, Figure 10 illustrates that level descriptions focus more on specific body-par movements, such the torso, shoulders, legs, and arms. We believe that this hierarchical structure of text corpus can enhance the alignment
19

---

Scaling Large Motion Models with Million-Leve Human Motions
Average Frame Count Subset(Sorted)
PoseTrack
16:12
HAAS00
39.07
46.19
Kinelics-700
60.64
on
67.96
BABEL
69.06
nu_gg
69.31
mation
72:08
animation
76.6
humman
93.41
activitynet
104.81
humansc3d
129.91
BEDLAM
134.07
idea400
humanml
140.87
Dataset
fitness
142-15
143.42
dance
146.56
eist
157.89
music
163-74
kungfu
165.04
GRAB
202.09
talkshow
225-31
EgoBocy
297-87
RICH
472.63
SMPLX
482:09
550..99
3DPW
572.92
prox
behave
994.35
400
1000
Average Frame Count
Figure 8: The length distribution across different subsets in MotionLib
between LLM and motion modality, therefore improving the understanding of motion.
C. Additional Experimental Analysis
In this section, present additional experimental analysis that could not be included in the main paper due to space constraints.
C.1. Ablation of Large Motion Model Training
Without further notification, the following experiments carried out MotionLib test set. For simplicity, employ Vector Quantization (VQ) for motion encoding in the following ablation experiments.
C.I.1. LORA VS. FULL PARAMETER FINE-TUNING
We conduct an ablation study comparing LoRA and full parameter fine-tuning, As illustrated in Table 10, LoRA fine-tuning struggles to achieve competitive results. We hypothesize that this limitation stems from the introduction of new motion tokens, which require significant parameter updates for the large language model to effectively learn and interpret these additional tokens. The restricted capacity of LoRA fine-tuning appears inadequate meet these demands, highlighting the challenges of adapting such changes with limited parameter updates.
C.1.2. LEARNING FROM SCRATCH VS. FINE-TUNING
We compare the performance of fine-tuning GPT-2 against training it from scratch with random initialization on Motion-X- eval. As demonstrated in Table 11, finettued models consistently achieve superior results compared to those trained from scratch, particularly when trained on the HumanML33 dataset and evaluated on Motion-X. The significant performance gap underscores the importance of text pre-training, which enhances the model's 's ability to comprehend descriptions
20

---

Scaling Large Motion Models with Million-Level Human Motions
still
likely
still
appearing exercise
Wheelchair
facing forward
person
son
wal
aalkigght
arm
Moboject
playl
g
Lookin
stand
confocting
iin
actively
ly
engaged
speaking
woman
a
appear
holding
ly
seemingli
bending
f racing
mistl
stull
lo
performing
guita
microphone
seated
furns
around
right
0
rightshand
staidting
hand
toar
sur
sur
another
lett
rounding
leg
individual
Fave
forard
eniges
holding
tecting aeother
staps
forward
fft
e
Interact
5
chax
bounding box
geature
parson
appering
sit
interactimg
per
2803
pettrr
seo
son
trea
walking
leaning forward
W1
bounding
within
ing
ball
wall
premmitliion
head
Side
hacigg
something
aund
hand
preparing
another percon
BBOX
S
still
chair
image
table speer
stony
e
Violin
cheld
tylng
presendirg
facing
appent
r
crouching
root
Loans
sen
sulk
camera
seate
back
stald
still
s
lnteeding
ground
phone
teo
Enteraction
people
luineingg
move
seemingly engaged
c
passial
moeeyy to intutt00 2u.
Flzor
facing slightl
engege
Hight
activity
bandini
rightstill
fornard
table
St lanking howar
Caa
possibly
group
leaning
apper
no
slightly
discussion
Lhg
noving
arand
dhnding
terward
Jump
ieright
Left
conversation
arm
Yengaging
individual
S
left
standang
othere
faciig
reschang
right adge
balancing
gesturing
ing
cum
ane hand
ar
m
wa
for
stoter
ward
d
walking
Wa
forward
Still thing
Lielly
singing
observing
g
place
ob serve Ving
standing eppearing
s
ving
some
possibly
ubs
er
vang
soewing.y abservsing
le
stareing
hand
cello
jumplex
cencer
engaged
bos
tranrianiiiionn transition
facing away
son appears netson
watking kid
ares rassed
Figure 9: Word cloud of body-level motion descriptions in MotionLib.
Table 10:
Ablation results of LoRA tuning vs. full-parameter fine-tuning.
TRAIN TYPE
R@I 1 t
R@3 f
FID +
MMDist +
Real
0.297
0.634
0.004
2.068
LoRA
0.157
0.354
9.287
4.832
full-param
0.166
0.375
6.936
4.484
and improves its generalization capabilities across diverse tasks. This observation also validates the effectiveness of our hierarchical text labels. With richer textual content, full-parameter fine-tuning enables better alignment between the motion modality and the language model, further leveraging the strengths of pre-trained motion representations.
Table 11: Ablation results of learning from scratch vs. fine-tuning.
#Inst
From Scratch
R@ t
R@3
FID 4
MMDist $
Real
0.514
0.831
0.046
2.438
0.02M
Yes
0.042
0.116
17.932
8.957
0.02M
No
0.213
0.426
47.319
7.666
0.08M
Yes
0.461
0.784
0.116
2.862
0.08M
No
0.468
0.792
0.083
2.798
C.1.3. WITH vs. WITHOUT DESCRIPTION MASK
We also investigate the impact of different mask strategies of input sequence model performance on Humanm13d: Specifically, we compare two strategies: training with and without masking input motion description. As shown in Table 12, our results indicate that the second strategy yields better performance. This improvement over the first approach can be attributed to the strategy's 's ability to prevent catastrophic forgetting of text understanding, ensuring that the model retains its capacity to interpret textual inputs. Additionally, it helps reduce overfitting to motion patterns in the training data, thereby
21

---

Scaling Large Motion Models with Million-Level Human Motions
should herwidth apart forward indicating
gnn poo lened
slightly turned
a Firmly
l
atraight
tes alanted
extended
sigget bahar
ni apan y eedliin
forward suppersing eiseriotte comus
forward
m
upright indicatin
slightly apart
upright
ing
person
fest
Side
aupear
slightly raised
iesitbomee moel
leaed formard
Lightly
previzion balove nil eilghtty Jeanting
oppears evenly
relere
straight
straig rightt
pes posill Leveet
seriihht su suggesttig
e
ked
Fight
planted
foot bodyy
left
hand
axeo
Band
elaying
oppcery STRRL
herrds
suightlyy
one
Eap
S llggtlly
forwark
rorrd d evenly distributed
ssterele
La
Weight appears
pronding aubility
feciing streadd
face
flet
string
leve
posstionss sixghtiy
lighthlly toaurd
seated
vistble
slighlly
Leg
standing
lower body
sicrophane Leg alaghtiy Loge
arm
hang
awsat
we ght
ject
arm
m
Teen
s
slightly
weight
leaning forward
evenly
bent
Kand
olhon
w.we Setars
image
balarse
halding
arm
Teet e
weight
resting
slughsh ehes
right
hand.
apper body
person
front
nlight
bend
spen
stable
hunched
feet
forward
slightly
Knee
elbow
orso
lean
hand
posstbly
Left
firmly
planted
inticesiin
hand
tright
holding
arm
por ran Via stepping
balanced
Tody
welghe
left leg
mm
stsltn
resting
posture
Tanging
slightly
vis
visible
back serson close together
ground
neutral
stance
sipppert
leaning slightly
stable
stance
Figure 10: Word cloud of part-level motion descriptions in MotionLib.
enhancing the model's generalization capabilities.
Table 12: Ablation results of different mask strategy of input sequence.
Mask Strategy
R@I +
R@3
FID J
MMDist |
Real
0.511
0.797
0.002
2.974
with description mask
0.388
0.650
0.680
3,919
w/o description mask
0.466
0.752
0.101
3.234
1.1.4. ENCODER-DECODER VS. DECODER-ONT ONLY
Existing large motion models typically adopt either an encoder-decoder or decoder-only architecture (Jiang et al., 2023). To evaluate their performance, we carry out abaltions by training an encoder-decoder model, T2M-GPT (Zhang et al., 2023a) on the MotionLib dataset and comparing it with a decoder-only model based on GPT-2 medium. As shown Table 13, despite having comparable parameter counts, T2M-GPT struggles to produce competitive results. This limitation can be attributed to the inherent constraints of text encoding capabilities by using "CL.IPrrandom-iniialiieed decoder", which hinder the model's ability comprehend broader spectrum of motion-related language. In contrast, find that large motion models based decoder-only LLMs, which jointly train text tokens and motion tokens, achieve superior text-motion semantic alignment and exhibit stronger motion generation capabilities.
Table 13: Ablation results of Encoder-Decoder vs. Decoder-only architecture.
Arch
Model Name
#Param.
@@ 1 +
R@3
FID
MMDist 1
Real
0.297
0.634
0.004
2.068
enc-dee
T2M-GPT
380M
0.161
0.364
7.085
4.773
dec-only
GPT-2 Medium
355M
0.166
0.375
6.936
4.484
22

---

Scaling Large Motion Models with Million-Level Human Motions
C.1.5. SLOW CONVERGENCE LARGE MOTION MODELS
To evaluate the convergence speed of large motion models, we train GPT-2, LLaMA2-7B, and LLaMA3-SB for 300 epochs on Motion-X. The training curves, measured by R@) performance, are illustrated in Figure 11 LEFT. We observe that all large motion models nearly converge by 200 epochs, with larger models exhibiting faster convergence rates. Initializing these models with pre-trained weights significantly accelerates convergence, However, compared to large multimodal models like LLaVA (Liu et al., 2023), large motion models require more epochs to capture the intricate representations of motion sequences. We attribute this slower convergence to the limited representation capacity of the motion tokenizer, which currently supports only 1024 motion tokens. This limitation highlights the need to optimize the motion tokenizer and expand its representation space. To address this, we explore the 2D-LFQ quantization method as a promising alternative.
0.50
2
VQ
2D-LFQ
0.45
2
50%
0.40
Codebook Usage
R801
210
2
0.30
0.25
Llama2
Llama3.1
0.20
100
150
200
250
300
24
2
2
2
2
2
2
Epoch
Codebook Size
Figure 11: LEFT: Training curves with Y-axis denoting R@ retrieval accuracy. All these models are trained for 300 epochs at most and are evaluated every 1000 steps; RIGHT: Ablation of codebook usage of different quantization methods
3.0
12
2D-LFQ
2D-UFQ
2.5
10
28
29
210
211
232
213
214
225
236
220
211
212
233
215
236
Codebook Size
Codebook
Figure 12: Comparison with different motion quantization the Motion-X (LEFT) and MotionL.b dataset (RIGHT). The
Y-axis denotes FID ()).
C.2. Additional Experiments of Motion Quantization
1. ADDITIONAL FID RESULTS ON MOTION-X
First, we provide additional FID results on Motion-X in Figure 12. is worth noting that while our motion quantizer performs worse than RQ-VAR VAE on the smaller HumanML.3D dataset, it surpasses both VQ and RQ when evaluated on the
23

---

Scaling Large Motion Models with Million-Level Human Motions
larger Motion-X and MotionLib benchmarks, as can be seen in Table 8. This suggests that our approach offers a greater advantage when applied larger datasets, highlighting its improved generalization compared to previous methods.
C.2.2. ABLATION OF 2D-LFQ vs. ID-LFQ
To validate the effectiveness of our 2D strategy for motion quantization, we compare the 2D-LFQ method with its ID counterpart (which is functionally equivalent to VQ except for the quantization strategy). The results, presented in Table 14, show that 2D quantization in LFQ significantly outperforms the ID version. This underscores the superior ability of 2D quantization enhance the representational capacity of the motion tokenizer.
Table 14: Ablation of 2D motion quantization vs. its 1D version.
HumanML33
Motion-X
MotionL.b
Tokenizer
#Num.
#Param.
FID 4
MPJPE
FID
MPJPE
FID
MPJPE
ID-L.FQ
16384
19,43M
3.85
52.5
2.783
78.9
10.358
80.1
2D-LFQ
16384
108.35M
1.769
45.6
0.295
54.1
7.853
64.1
C.3. Discussion of Static and Synthetic Data.
Although images only capture static poses, exploring their effectiveness for motion generation remains valuable. Additionally, synthetic data may play a significant role, as both image and synthetic data are far more accessible than dynamic vidcos. With this in mind, we collect approximately 600K static human-related images and extract their corresponding human poses. Each pose is repeated 60 times to create a 60-frame sequence. During pretraining, we introduce specific language prompts, such "keep the action still", to explicitly guide the model in distinguishing between static and dynamic actions. Such prompt-based method effectively differentiates between different motion distributions. To validate the effectivenesss of synthetic and static data, we conduct a series of ablation experiments, shown in Table 15. We train GPT-2 medium on three data configurations: MotionLib without synthetic data, the IM-scale MotionLib dataset, and MotionLib augmented with static data. The model is trained for 300 epochs with a learning rate of 2c-4 and evaluated on two benchmarks: a subset of our collected static and synthetic data, and the MotionLib lesting set. Our results indicate that incorporating both static data and synthetic data leads to slight improvements in R- R-Prriision. However, given the marginal gains, further exploration necessary to fully realize the potential of these data types. We emphasize that static data is excluded from Motionl.J Lib in our main paper, and all experiments outside this section are conducted exclusively dynamic motion data.
Table 15: Ablation of the effectiveness of static and synthetic data.
Static-Syno-eval
MotionLib-eval
TRAIN SET
R@I 1
R@31
FID +
MMDist
R@1 1
R@3
FID 4
MMDisL +
Real
0.290
0.563
0.011
3.480
0.
0.474
0.006
1.647
MotionLib with syn data
0.111
0.248
57.719
8.412
0.167
0.396
1.740
2.323
MotionLib
0.120
0.252
55.983
8.175
0.166
0.393
1.780
2.356
MotionLib + static data
0.264
0.542
0.516
4.007
0.168
0.399
1.614
2.300
C.4. Discussion of Current Evaluation Metric's Limitation
During our experiments, we notice the evaluation metrics are not as robust as we expect. Considering this, we carry out experiments for further analysis. Table 16 presents results using different retrieval models as evaluators. Here, we employ the same dual-encoder architecture following Guo et al. (2022a) as the retrieval model, but trained it on two distinct datasets: HumanMI.3D and Motion-X, where HumanML33D is a subset of Motion-X. Unlike robust visual models such as CLIP (Radford et al., 2021), these retrieval models are constrained by their small parameter size and limited training data. As shown in the table, performance using these two evaluators much worse than our results in the main paper using evaluator trained on MotionLib. More importantly, when using the model trained on the larger Motion-X X dataset, performance on HumanML.31 decrease. This suggests that training on the broader Motion-X X dataset negatively impacts
24

---

Scaling Large Motion Models with Million-Leve Human Motions
Text Prompt
Generated Motion Sequences
Person falls to the ground in sitting motion and then pops back up a standing position.
A person squats down then jumps.
A man full-body sideways jumps his left.
A woman is blowing balloon while walking.
A person is building blocks and walking at the same time.
The person performs Lunges Of Crossover Reverse Lunge.
Figure 13: Quantitative examples of motions generated by our large motion model Being-MO.
Table 16: Comparison of evaluations using different retrieval models on the HumanML.3D test set.
Humanml3d-eval
Motion-X-eval
Decoder
#lnst.
#Param.
R@ 1
R@31 f
FID
R@1 f
R@3
FID +
Real
0.511
0.797
0.002
0.496
0.821
0.038
GPT-2
0.02M
355M
0.466
0.752
0.101
0.358
0.651
0.050
GPT-2
0.08M
355M
0.462
0.744
0.208
0.362
0.656
0.754
LLaMA-2
0.02M
7B
0.497
0.778
0.214
0.378
0.671
0.122
LLaMA-2
0.08M
7B
0.474
0.758
0.452
0.376
0.673
0.518
LLaMA-3
0.02M
8B
0.500
0.783
0.173
0.380
0.675
0.094
LLaMA-3
0.08M
8B
0.499
0.786
0.264
0.393
0.696
0.591
LLaMA-2
0.02M
13B
0.519
0.803
0.166
0.395
0.695
0.105
LLaMA-2
0.08M
13B
0.504
0.790
0.393
0.400
0.700
0.637
R-Precision performance on the HumanML3D subset, even though HumanML33 is part of Motion-X. We hypothesize the unexpected results arise from the limited generalization capability of the current evaluator. FID, a standard metric for generation tasks, is typically computed using pretrained evaluator, In image generation, FID relies on robust visual
25

---

Scaling Large Motion Models with Million-Level Human Motions
encoders, such as InceptionNet (Szegedy al., 2015), trained millions of images. In contrast, the evaluator used for motion generation is lightweight motion autoencoder with small parameter scale (Guo al., 2022a), trained only limited data (e.g.. HumanML3D with 20K motions). Such small-scale training data may hinder its ability to generalize effectively, leading to unreliable semantic alignment between text and motion, particularly for unseen motions. In fact, some recent works (Petrovich et al., 2023; Voas et al., 2023) have begun to recognize these limitations. For instance, Petrovich et al. (2023) proposed simple yet effective approach for text-to-3D human motion retrieval, while Voas al. (2023) highlighted that existing metries are sensitive to embedding space quality and often misalign with human perception. These findings emphasize the need for more robust and fair evaluation metries for large motion models. We believe addressing this challenge is of significant value and plan to explore it further in future work.
C.5. Additional Qualitative Results
We present additional examples visualize the human motions generated by our large motion model, Being-M0, trained on the MotionLib iib dataset, shown in Figure 13. The results demonstrate that model can produce motion sequences that closely align with the input texts, highlighting the effectiveness of MotionLib as training resource.
26


Reviews:
[{"review_id": "irRphneCV8", "reviewer": "Reviewer_LYsh", "summary": "The paper investigates scaling motion generation models based on million-level data and LLM-style architecture. The authors first contributes a million-level human motion dataset, named MotionLib. Training models on this data, they highlight the importance of scaling both data and model size for advancing motion generation. To better integrate the motion modality, they propose MotionBook for fine-grained motion features and efficient motion tokenizer. The empirical findings from this work offer valuable insights to researchers in the field.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer,\n\nThank you for your thoughtful review and positive feedback. We have carefully considered your questions and suggestions and provide our responses below. Please let us know if you require further clarification.\n\n---\n## **Response to: Questions about UNSEEN-90K Dataset in OOD Experiments**\n\nIn our OOD experiments (Table 4), the UNSEEN-90K testing set is constructed by excluding 11 subsets (~90K samples) from the full MotionLib dataset. The remaining data (primarily Motion-X and web-derived data) serves as the training set.\n\nThe excluded subsets are intentionally diverse to ensure significant distributional shifts from the training data, enabling a robust evaluation of OOD generalization. These subsets include:\n\n- Synthetic data: gta_human [1], bedlam [2] (distinct from web-derived training data captured in the real world).\n- Domain-specific activities: FIT3d [4] (fitness), RICH [5], ARCTIC [11] (human-object interactions), EgoBody [7] (social interactions).\n- Diverse environments: PoseTrack [3], MPI-INF-3DHP [6] (in-the-wild settings), Human36M [8], CHI3d [9], KIT [10] (daily activities captured in the lab).\n\nThis selection strategy demonstrates that the UNSEEN-90K testing set exhibits substantial distributional shifts compared to our training data, allowing us to rigorously assess the model’s OOD generalization. As shown in Table 4 in our main paper, models trained on MotionLib outperform those trained solely on HumanML3D or Motion-X, highlighting the benefit of large-scale, web-sourced motion data.\n\nRegarding the examples. Thank you for this constructive suggestion. We will incorporate additional visualization examples in the appendix in our revised manuscript.\n\n[1] Playing for 3D Human Recovery. TPAMI 2024\n\n[2] BEDLAM: A Synthetic Dataset of Bodies Exhibiting Detailed Lifelike Animated Motion. CVPR 2023\n\n[3] PoseTrack: A Benchmark for Human Pose Estimation and Tracking. CVPR 2018\n\n[4] AIFit: Automatic 3D Human-Interpretable Feedback Models for Fitness Training. CVPR 2021\n\n[5] Capturing and Inferring Dense Full-Body Human-Scene Contact. CVPR 2022\n\n[6] Monocular 3D Human Pose Estimation In The Wild Using Improved CNN Supervision. 3DV 2017\n\n[7] EgoBody: Human Body Shape and Motion of Interacting People from Head-Mounted Devices. ECCV 2022\n\n[8] Human3.6m: Large scale datasets and predictive methods for 3D human sensing in natural environments. TPAMI 2013\n\n[9] Reconstructing Three-Dimensional Models of Interacting Humans. CVPR 2020\n\n[10] The KIT Motion-Language Dataset. Big data 2016\n\n[11] ARCTIC: A Dataset for Dexterous Bimanual Hand-Object Manipulation. CVPR 2023\n\n---\n## **Response to: The Limitation of MotionBook’s Global Feature Extraction**\n\nThank you for your question. You raised a valid concern regarding MotionBook’s use of 2D convolutions for motion encoding. We clarify our design choices below:\n\n**1. Division of Roles**\n  - **Motion Tokenizer**, which focuses on quantizing continuous, high-dimensional motion features into discrete, low-dimensional token sequences, effectively capturing local spatio-temporal features and achieving a compact representation. 2D convolutions are well-suited for this due to their computational efficiency and inductive bias for local patterns.\n  - **LLM backbones**, which handles global feature modeling and long-range dependencies within the input token sequence via Transformer self-attention, addressing the limitation of CNNs for global context reasoning.\n\nThis hybrid design combines the strengths of CNNs (local feature extraction) and Transformers (global modeling), ensuring efficient and effective motion encoding.\n\n**2. Why CNN Over Alternatives (e.g., RNN or Transformers)?**\n  - **Fair Comparison**: Most prior motion tokenizers (e.g., VQ, RVQ) use CNNs, therefore we maintain consistency for fair benchmarking.\n  - **Empirical Findings**: We experimented with Transformers but observed no performance gain, likely due to the limited data amount and lower resolution compared to vision tasks where ViTs excel.\n\nWe appreciate your insightful feedback and hope our responses address your concerns. Thank you again for your time and constructive comments!", "claims_and_evidence": "yes, all claims are well supported", "methods_and_evaluation_criteria": "yes, they compare their method with other LLM-based methods. The evaluation data comprising both public and self-collected data. They follow the conventional evaluation criteria.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "yes, I check the experiments.\nThe experiments design primarily follows the experiments design of LLM (scaling data/model, architecture design, etc.). Additional, they provide experiments to validate the effectiveness of the proposed motion book.\n\nOne question is about OOD experiment. in Table 4, the authors are suggested to explain the UNSEEN-90K dataset. How the evaluation data differs from training data? (different motion categories? synthetic v.s. realistic? different complexity?) It is suggested to provide some representative examples.", "relation_to_broader_scientific_literature": "N/A", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strengths: The authors provide extensive experiments and this works could be a solid work if the dataset and code are released in future. The intuition of Motionbook makes sense.\nWeakness: The design of Motionbook treats motion sequence as 2D image, and I suppose the authors use convolution layers to extract features? Comparing with RNN/Transformer, convolution network maybe cannot perform well in global feature extraction. How do the authors handle this?", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cVvNrqsEEX", "reviewer": "Reviewer_yHwB", "summary": "This paper proposes a dataset, a VQVAE, and a motion generation model. The dataset MotionLib comprises over 1.2M motion sequences with hierarchical and detailed text annotations. The VQVAE uses a 2D-LFQ for a lookup-free tokenizer. The text-to-motion model is trained on the proposed dataset and VQVAE.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer,\n\nWe appreciate your valuable feedback. Due to space constraints, **we only provide concise responses below but would present more during discussion.** Please let us know if any clarification is needed.\n## W1: WHAM accuracy & dataset validity\nWhile no motion estimation algorithm is perfect, our data pipeline (3D keypoint optimization, physics constraints, RL tuning &mdash; App B.2) refines initial estimates. Despite noises, Table 2 & 4 show large-scale motion pretraining improves generalization by exposing models to diverse motions. MotionLib provides a foundational resource especially high-quality texts. Its quality can be improved via future algorithm updates and filtering. Scale is critical given current data scarcity.\n## W2: Data refinement details (RL, simulation, slipping handling)\nWe refine motion sequences using the PHC policy (Luo et al., 2023) in IsaacGym, which achieves high tracking accuracy (97.1% on AMASS, 95.7% on Human3.6M). The process involves:\n- Inputting video-extracted motion to PHC for physics-compliant tracking (balancing, reducing jitter/sliding).\n- Using PHC to generate physically plausible motions in IsaacGym, with termination conditions (e.g., early termination for balance loss) flagging low-quality sequences.\n- Downweighting flagged sequences during Puppet model training—a common practice for handling noisy data—to prioritize high-quality samples.\n## W3: 2D motion quantization is already used\nWhile ParCo and MoGenTS explored 2D motion quantization, the core innovation of our MotionBook (including 2D-LFQ) lies in its lookup-free (LFQ) mechanism and **its application in the context of large-scale training**. Unlike traditional VQ (limited to small codebooks, e.g., 512/1024 codes), LFQ avoids codebook collapse and supports 16K+ codes, enabling scalable learning on our million-scale MotionLib dataset. This addresses a key bottleneck in prior works, which were not designed for such diversity and scale.\n## W4: T2M model uses a standard AR architecture, lacking novel design\nWhile Puppet uses a standard LLM architecture, our key contribution is the first Scaling Law study for motion generation, systematically analyzing data/model size impacts—similar to foundational VLM work like LLaVA, which also relied on standard architectures.\n\nIn text-to-motion, we argue that constructing large-scale motion data with rich, hierarchical text descriptions (MotionLib) and designing effective motion encodings (MotionBook) to bridge text-motion alignment are themselves significant contributions. Our endeavour in data construction and labeling aims to establish a foundation for future large motion models.\n## W5: Missing comparisons with some recent methods\nThank you for highlighting these recent works. For a more comprehensive comparison, we include their publicly reported results on HumanML3D. Some methods (e.g., LaMP, MoGenTS) were concurrent or very recent (within 4 months) and were not initially compared.\n\nNotably, current T2M methods include specialist models optimized on specific datasets and LLM-based generalist models (aiming for broader instruction/task generalization via LLMs), like our Puppet. Puppet excels among generalist models (Table 3) and remains highly competitive on R@1, R@3, and MMDist compared to specialist models.\n||R@1|R@3| MMDist|FID|\n|-|-|-|-|-|\n|Momask|0.521|0.807|2.958|0.045|\n|ReMoDiffuse|0.51|0.795|2.974|0.103|\n|Puppet|0.528|0.82|2.875|0.141|\n## W6: High FID in OOD experiments; suggestion to evaluate on HumanML3D (HM3D) using MotionLib\nHigher OOD FID Explanation: The UNSEEN-90K test set comprises 11 subsets with substantial distribution shifts from training data, including synthetic data, activity-specific datasets, and varied capture environments. The elevated FID in this challenging OOD setup is expected, similar to observations in other motion tasks (e.g., music-to-motion in LMM). More importantly, Table 4 validates that large-scale training with diverse, large-scale MotionLib data (vs. HM3D or MotionX alone) significantly boosts OOD performance.\n\nHM3D Evaluation: Thanks for the suggestion. We remove all HM3D data from MotionLib and trained models on varying scales of remaining data (0.6M–1.2M samples), then evaluate on the HM3D test set. Results are shown below:\n|Train Data Size|R@1|R@3|MMDist|FID|\n|-|-|-|-|-|\n|0.6M|0.176|0.369|2.980|9.408|\n|1.2M|0.208|0.441|2.964|7.983|\n## W7: Lack of clarity for train/eval settings in Table 2 (\"retrain\", \"pretrain\").\nThese terms are distinct in our context:\n- Retrain: Refers to training the evaluation model (motion autoencoder) separately for each benchmark (MotionX and MotionLib), following HM3D paper’s architecture. This ensures metric fairness (FID, R-Precision) across datasets.\n- Pretrain: Describes Puppet’s training:\n  - Initialize with public LLM weights (GPT-2, LLaMA).\n  - Extend vocabulary with motion tokens.\n  - Continue pretraining on target datasets (HumanML3D/MotionX/MotionLib subsets) via autoregressive loss.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "Yes", "experimental_designs_or_analyses": "Yes", "relation_to_broader_scientific_literature": "Related to human motion generation.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "Strengths:\n\n- The text annotations are detailed on body parts.\n- Using a simulated environment to eliminate some physical issues like jittering and foot-sliding is reasonable.\n\n\nWeaknesses:\n\n- For the dataset, the accuracy of WHAM is limited. I have tried WHAM and other motion estimation methods, but none of them produce decent results which are valid to serve as the ground truth. As shown in Figure 2, the estimated motion is not accurate.\n- Many details about the data refinement are not clear. For example, how is the RL performed, which is the simulated environment, how to find and mark slipping with smaller weight.\n- For the VQVAE, 2D motion quantization is already well-used in motion generation, e.g., ParCo, MogenTS.\n- For the text-to-motion model, it uses a well-used autoregressive model for motion generation, without new designs.\n- In the experiments, the compared methods miss many recent methods with better performance, e.g., MoMask, MoGenTS, LaMP, Diversemotion, ReMoDiffuse, Stablemodiffusion, Fg-T2M++.\n- For the out-of-distribution experiments, the FID is large, which illustrates that the generalization is still weak. Also, I suggest evaluating on HumanML3D while only training on different scales of MotionLib. This should demonstrate the value of MotionLib.\n- For table 2, the training and evaluation setting is not clear. It states it utilizes the autoencoder retrained on motion-x and motionlib, but also states the training sets are HumanML3D only, Motion-X only, MotionLib-0.5, MotionLib-full. Also, it writes \"retrain\", then how is the pretrain performed?", "other_comments_or_suggestions": "No", "questions_for_authors": "No", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Rkbso8rpTF", "reviewer": "Reviewer_CsGh", "summary": "This paper explores various design choices for building large motion models, inspired by the success of LLMs. In the absence of a large-scale motion dataset, it first introduces MotionLib, the first million-level motion dataset, obtained by automatically annotating 3D motion and text descriptions from publicly available videos. It then proposes Pupet, a large motion model trained on the collected MotionLib dataset. Additionally, it introduces Motionbook, a motion encoding method designed to further enhance the model’s representation power. Extensive experiments validate the effectiveness of each design choice and compare the proposed model’s performance with existing motion models.\n\n**Update after rebuttal**\n\nAfter reviewing the rebuttal and the other reviewers' comments, I will maintain my initial recommendation (weak accept). The authors have addressed most of my concerns, as well as those raised by the other reviewers. However, I *strongly* recommend that the authors revise the manuscript to incorporate the contents of the rebuttal.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer,\n\nThank you for your thoughtful review and positive feedback. Below are our responses to your questions and suggestions. Please let us know if you require further clarification.\n## **Response to: the discussion of LMM [1] and Dataset Comparison**\nWe appreciate you highlighting the relevant work of LMM [1]. Below, we clarify the scale and unique contributions of MotionLib in comparison:\n- **Dataset Scale**: The term \"million-scale\" in our paper refers to the number of motion sequences. MotionLib contains 1.2 million (1.2M) motion sequences (as in Table 1), totaling 137 million (137M) frames. In contrast, LMM’s dataset comprises 320K sequences and 100M frames. Thus, MotionLib is currently the largest motion generation dataset in both sequence count and total frames. We will explicitly add the frame count to Table 1 for clarity.\n- **Key Distributions**: \n  - **Task Focus**: MotionLib is optimized for text-to-motion generation, featuring 2.48M fine-grained text-motion pairs (including fine-grained part-level descriptions). LMM, meanwhile, targets multi-modal inputs (text/image/audio).\n  - **Performance**: On HumanML3D, Puppet outperforms LMM in R@1, R@3, and MMDist, showing better text-motion alignment. Notice that our FID performs worse than LMM. We attribute this to the difference of used motion tokenizer. As shown in Table.7, our 2D-LFQ simply performs competitive to vanilla VQ, but showing large improvement as the data become larger and diverse. This means that our lookup-free tokenizer has more advantage when facing large-scale scenarios and thus requires more training data. It's also important to note that Puppet exhibits a strong agent capability to follow the user's instructions empowered by the LLM. Among existing LLM-based generalist models, Puppet excels these models in performance.\n||R@1|R@3|MMDist|FID|\n|-|-|-|-|-|\n|LMM|0.525|0.811|2.943|0.04|\n|Puppet-LFQ|0.528|0.82|2.875|0.141|\n\nWe will expand the discussion of LMM in the revision, emphasizing MotionLib’s scale, annotation richness, and task-specific advantages.\n\n[1] Large Motion Model for Unified Multi-Modal Motion Generation. ECCV 2024\n\n## **Response to: Qualitative Comparisons with MotionGPT & T2M-GPT (Experimental Designs)**\nThank you for this useful feedback! We agree qualitative examples are important. We will add qualitative comparisons of motion results generated by our Puppet versus T2M-GPT and MotionGPT for the same text prompts in the subsequent revision.\n## **Response to: The Foot Sliding Issue in Dataset (Supplementary Material)**\nThank you for raising the issue of foot sliding. Indeed, automatically estimating motion data from large-scale web videos inevitably introduces noises, including physical implausibilities like foot sliding.\n\nThe most important reason we choose to down-weight these low-quality samples rather than completely discarding them is: we adopt two-stage training to address the impact of noisy data:\n  - **Motion pretraining &mdash; Information Preservation:** In a million-scale dataset, even noisy samples might contain valuable motion patterns or contextual information. Discarding them completely could lead to the loss of potentially useful information.\n  - **Motion instruction Tuning &mdash; Future Optimization:** More importantly, after obtaining a good base model through large-scale motion pre-training.  High-quality subsets can later refine the model via instruction tuning, further improving generation quality and physical realism.\n\nSuch approach is motivated by the great success of LLMs. In the pretraining stage of LLM, a large amount of corpus is quite dirty. However LLM can still gain massive knowledge from it and develop good responsive ability through instruction-tuning.\n\nIn addition, we have other considerations to incorporate these part of data:\n\n- **Maximizing the Scale Utilization:** Our goal is to explore the potential of large-scale data training. Retaining all samples (even if down-weighted) maximizes the utilization of diversity brought by the data scale.\n- **Down-weight is a Common Strategy:** When dealing with large noisy datasets, reducing the weight of noisy samples via the loss function is a mature and effective strategy that allows leveraging data scale while mitigating the negative impact of noise.\n\nAs a summary, we believe that pursuing an increase in data scale first, followed by continuous improvement of data quality via further refinement and filtering, to enhance model performance through down-weighting and subsequent fine-tuning, is an effective pathway for developing large motion models at this stage.\n\n## **Response to Other Comments (e.g., adding total frame count and correcting typo)**\nThank you for pointing them out.\n- We will explicitly add the total frame count of MotionLib (approx. 137 million frames) in the revised Table 1 or relevant text.\n- We will correct \"an compact\" to \"a compact\" in line 30 in the revised version.\n\nHoping our response clarifies your concerns.", "claims_and_evidence": "The main claims—the effectiveness of the proposed MotionLib, Pupet, and Motionbook—are mostly well-supported by discussions (in comparison to existing baselines) and empirical validation. However, while MotionLib is claimed to be the first million-scale motion dataset, the below paper (published at ECCV 2024) also introduces a motion dataset with 100 million frames for training a large motion model:\n\n[1] Zhang *et al.*, Large Motion Model for Unified Multi-Modal Motion Generation, ECCV 2024.\n\nIs the \"million-scale\" referred in this paper based on the number of sequences or frames? Clarifying this aspect would help distinguish MotionLib from existing datasets.", "methods_and_evaluation_criteria": "Both the proposed method and evaluation criteria are mostly reasonable. I especially appreciate how this paper considers diverse training datasets, evaluation datasets, and baseline models to demonstrate the effects of core design choices in building a large motion model.\n\nHowever, it seems that one highly relevant work was excluded as a baseline in the experiments:\n\n[1] Zhang *et al.*, Large Motion Model for Unified Multi-Modal Motion Generation, ECCV 2024.", "theoretical_claims": "There is no theoretical claim that requires formal proof in this paper.", "experimental_designs_or_analyses": "I verified the validity of all experimental designs in the main paper and found no issues with the existing designs. However, I notice a lack of **qualitative** comparisons with existing baselines (e.g., MotionGPT, T2M-GPT), despite the paper providing comprehensive quantitative comparisons.", "relation_to_broader_scientific_literature": "This paper presents a comprehensive analysis of the core design choices (e.g., training dataset scale, data encoding method) in building large motion models. Its technical contributions are closely related to those in the literature on other large model training (primarily LLMs), although the problem domain is different, and some contributions (e.g., motion representation) are specific to the motion generation problem.", "essential_references_not_discussed": "As mentioned several times in the above comments, I believe this paper is highly relevant to this work, as it claims similar contributions (e.g., large-scale dataset collection, large motion model training). Discussing this paper would be necessary to better distinguish the unique contributions of this paper.\n\n[1] Zhang *et al.*, Large Motion Model for Unified Multi-Modal Motion Generation, ECCV 2024.", "other_strengths_and_weaknesses": "**Strengths.**\nI appreciate how this paper provides comprehensive experimental results on key design choices for building large motion models, offering valuable insights for future research in this area. Additionally, the collected large-scale motion dataset and the pre-trained large motion model have the potential to be highly beneficial to the research community.\n\n**Weaknesses.**\nComparisons with an important related work [Zhang *et al.*, ECCV 2024] are missing. Without discussing this work, it is difficult to fully assess the unique contributions of this paper. Additionally, while the paper provides extensive quantitative comparisons, including *qualitative* comparisons with existing baselines would be also important.", "other_comments_or_suggestions": "* Please clarify whether \"million-scale\" in this paper refers to the number of sequences or frames. If it does refer to the number of sequences as mentioned in the paper, it would be helpful to also provide the total number of frames in the collected dataset for additional clarity.\n* Typo in line 30: \"an compact\" should be corrected to \"a compact.\"", "questions_for_authors": "Please refer to the above comments. Was there a specific reason why [Zhang *et al.*, ECCV 2024] was not discussed in the paper?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pbsdsEaaei", "reviewer": "Reviewer_q9qB", "summary": "The paper introduces MotionLib, the first million-level dataset for motion generation, which is 15× larger than previous datasets and includes hierarchical text descriptions. Using MotionLib, the authors train Puppet, a large-scale motion model that demonstrates robust generalization across diverse human activities, including unseen scenarios. To improve motion encoding, they propose Motionbook, which includes:\n- A lossless feature representation for motion data.\n- A novel 2D lookup-free motion tokenizer that balances fine-grained details and expanded codebook capacity.\n\nTheir study emphasizes the importance of scaling both data and model size for advancing motion generation and highlights key insights for achieving generalist motion models. Experimental results show existing models struggle with out-of-domain generalization, whereas MotionLib enables better scalability, positioning it as a benchmark comparable to ImageNet for motion data.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer,\n\nThank you for your thoughtful review and positive feedback. We have carefully considered your questions and suggestions and provide our responses below. Please let us know if you require further clarification.\n\n---\n## **Response to: Insufficient visualizations in supplementary material, video blurring, motion quality, and deformation issues.**\n- **Regarding visualization quantity and video blurring**: \n\n    Thank you for your feedback. Due to conference supplementary material size constraints (typically 100MB), we were limited in the number of visual examples we could include and had to compress video resolution. To better demonstrate MotionLib’s richness, we plan to release a dedicated website with additional high-resolution examples upon final publication.\n\n- **Regarding motion quality and deformation**: \n\n    We acknowledge that motion data extracted and refined from web videos may still exhibit imperfections (e.g., deformations), even after RL optimization. However, we also feel more positive since our motion model is significantly enhanced using such motion data, demonstrating the scaling law in this task (Table 2, 4). This suggests the larger potential of our MotionLib dataset. As we keep refining this dataset using additional strategies, the gain of performance and generalization could be more significant. Considering this, we consider the building of MotionLib as a long-term iterative process, and our future work can incorporate stricter filtering or advanced algorithms, in addition to current steps we adopted (e.g., 3D keypoint optimization, physics constraints, and RL strategies; see Appendix B.2).\n\n  In addition, we suggest the issue of data quality and quantity is a trade-off for large-scale pretraining. Motivated by LLM's pretraining, we believe combining large-scale pre-training with motion fine-tuning on high-quality subsets can further mitigate quality issues while preserving scalability benefits.\n\n## **Response to: Questioning the effectiveness of the claimed motion representation improvement (foot sliding, pose jittering).**\n\nIt is important to emphasize that MotionLib’s primary contribution lies in advancing the semantic understanding and generalization capabilities in motion generation, rather than eliminating physical issues (e.g., foot sliding or jittering) entirely. Quantitative improvements in R-Precision, MMDist (Table 3), and OOD generalization (Table 4) demonstrate that our Puppet model achieves superior alignment between text instructions and nuanced human motion, validating the semantic gains we expect. In addition, although MotionLib may include more motion noises compared to datasets like HumanML3D, it excels in the quality and quantity of motion descriptions. After investgation, we notice the texts of HumanML3D, MotionX are hightly duplicate and contain a large amount of noisy texts (e.g.,\"moving the hands speaking some thing\", \"the person's eyeglasses pass\"). Instead, our MotionLib avoids these issues by incorporating more clean and diverse texts.\n\nOur methodology follows a \"first scale up, then refine down\" philosophy.\n\nThat means, while physical issues (e.g., foot sliding and jittering) persist in some generated motions —— an expected byproduct of large-scale automated processing &mdash; they do not negate the foundational progress of motion semantics. Such trade-offs are inherent to data-driven research at scale. Crucially, MotionLib establishes a robust semantic foundation for general-purpose motion models, enabling future work to address physical fidelity through:  (1) further data quality improvements on our MotionLib (e.g., using stricter filtering), (2) post-training refinement (e.g., fine-tuning on high-quality subsets). \n  \nWe posit that prioritizing scalable semantic understanding is pivotal; physical issues can then be iteratively resolved without compromising the model’s generality.\n  \nThank you again for your valuable feedback. We hope our response clarifies your concerns.", "claims_and_evidence": "Overall, the explanations are clear and well-structured. They effectively convey the key ideas and provide a solid understanding of the topic.", "methods_and_evaluation_criteria": "The evaluation criteria sounds reasonable and appropriate for assessing the proposed method.", "theoretical_claims": "No theoretical claims in this paper.", "experimental_designs_or_analyses": "The experimental design appears sound overall.", "relation_to_broader_scientific_literature": "Overall, this work significantly contributes to scaling motion generation and refining motion representation learning. By introducing MotionLib and Motionbook, it bridges the gap between small-scale motion datasets and large-scale generalist models, setting a new benchmark for future research in human motion synthesis and multimodal learning", "essential_references_not_discussed": "None.", "other_strengths_and_weaknesses": "### Strength\n- The dataset is significantly larger compared to previous datasets and provides more fine-grained text descriptions. The authors claim that they optimize motion quality using reinforcement learning (RL).\n- Experiments validate the effectiveness of scaling and propose a lossless motion representation that is better suited for recovery.\n- The 2D encoder indeed enhances performance by preserving more fine-grained information.\n\n### Weakness\n- The supplementary material provides too few visualizations, making it difficult to fully assess the richness and quality of the dataset. Some videos are noticeably blurry, which raises concerns about data quality. I also observed that the motion quality is not very high—even after RL optimization, there are still cases of character deformation. It’s unclear whether this is due to perspective issues from Matplotlib rendering or an inherent flaw in the model itself, but either way, it’s a problem that shouldn’t be ignored.\n- The claimed motion representation improvement is questionable. From the supplementary videos, foot sliding and pose jittering are still quite prevalent, making it hard to see the claimed benefits. If the proposed motion representation is supposed to enhance stability, where is the concrete proof of its effectiveness? The visual evidence so far doesn’t convincingly support this claim.", "other_comments_or_suggestions": "### Post-Comments of Rebuttal\nAfter discussing with other reviewers, I find that the concerns regarding motion quality still persist. Therefore, I will adjust my score accordingly. That said, I am inclined to lean toward acceptance due to the contribution of the enlarged dataset. I hope the authors will continue refining this dataset post-submission, unlike Motion-X, which saw limited adoption due to its subpar quality.", "questions_for_authors": "Please see weakness.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper received mixed but generally positive reviews. Reviewers #CsGh, #LYsh, and #q9qB expressed support, emphasizing the significance of the MotionLib dataset’s unprecedented scale, rich hierarchical text annotations, and the first systematic study of scaling laws in motion generation. Despite concerns about motion quality (e.g., foot sliding, jittering) and generalization gaps raised by reviewers #yHwB and #q9qB, most reviewers acknowledged that the authors' rebuttal adequately addressed the technical and methodological issues. Reviewer #yHwB remained skeptical, primarily due to the reliance on video-estimated poses rather than high-precision motion capture data, and maintained a weak reject. Nonetheless, the AC-facilitated discussion confirmed that most reviewers accepted the trade-off between data scale and accuracy and appreciated the dataset’s broader potential impact. Reviewers also recommended incorporating more qualitative comparisons and clarifying the contributions of the dataset versus the method in the final version. Overall, while some reservations remain, the consensus leans toward acceptance given the dataset’s foundational value for future large motion model research.

After carefully considering the paper, the reviews, and the rebuttal, the AC recommends acceptance. The AC recognizes the potential contribution of the dataset and strongly encourages the authors to incorporate qualitative comparisons and clarify the dataset’s contributions as suggested by the reviewers.
Decision: Accept (poster)