VimoRAG: Video-based Retrieval-augmented 3D Motion Generation for Motion Language Models
Authors: Haidong Xu
Guangwei Xu
Zhedong Zheng
Xiatian Zhu
Wei Ji
Xiangtai Li
Ruijie Guo
Meishan Zhang
Min zhang
Hao Fei
Venue: NeurIPS 2025 poster ()
Keywords: Motion Generation
Retrieval-augmented Generation
Primary area: deep_learning

Abstract: This paper introduces **VimoRAG**, a novel video-based retrieval-augmented motion generation framework for motion large language models (LLMs). 
As motion LLMs face severe out-of-domain/out-of-vocabulary issues due to limited annotated data, **VimoRAG** leverages large-scale in-the-wild video databases to enhance 3D motion generation by retrieving relevant 2D human motion signals. 
While video-based motion RAG is nontrivial, we address two key bottlenecks: 
(1) developing an effective motion-centered video retrieval model that distinguishes human poses and actions, 
and (2) mitigating the issue of error propagation caused by suboptimal retrieval results.
We design the Gemini Motion Video Retriever mechanism and the Motion-centric Dual-alignment DPO Trainer, enabling effective retrieval and generation processes. 
Experimental results show that **VimoRAG** significantly boosts the performance of motion LLMs constrained to text-only input.

VimoRAC. Video-based Retrieval-augmented 3D Motion Generation for Motion Language Models
Haidong Xu', Guangwei Xu, Zhedong Zheng", Xiatian Zhu?
Wei Ji Xiangtai Li,, Ruijic Guo, Meishan Zhang', Min Zhang', Hao Fei
Harbin Institute of Technology (Shenzhen) 2 University of Macau 3 University of Surrey
Nanjing University Nanyang Technological University National University of Singapore 182haidong@gmail. com, haofei74190gmail.com
Abstract
This paper introduces VimoRAG, a novel video-based aetrivaal-augmente motion generation framework for motion large language models (LLMs). As motion LLMs face severe
oo issues due to limited annotated data, VimoRAG leverages large-scale in-the-wild video databases enhance 3D motion generation by retrieving relevant 2D human motion signals. While video-based motion RAG is nontrivial, we address two key bottlenecks: (1) developing effective motion-centers and actions, and (2) mitigating the issue of error propagation caused by suboptimal retrieval results. We design the Gemini Motion Video Retriever mechanism and the Motion-centric Dual-alignment DPO Trainer, enabling effective retrieval and generation processes. Experimental results show that VimoRAG significantly boosts the performance of motion LLMs constrained text-only input. All the resources (https:/ /eaake////akeemiiiii tty. wwt.tiittee.. github. io/ /ooVimmoAG// are available.
video retrieval model that distinguishes human poses
1 Introduction
Generating diverse and realistic human motions from free-form text prompts has significant practical applications, including video gaming, robotic assistance, and virtual reality. Previous advancements, ranging from transformer-basee VAEs [ to recent diffusion- led to increasingly promising generative performance. With the emergence of LLMs, motion- language models (aka. motion LLMs) have been proposed [ 4 These unified architectures not only understand various motions but also support motion generation. To achieve competitive capabilities, motion LLMs require training on extremely large-scale datasets to ensure sufficient capacity. Particularly for motion generation, substantial amount of labeled data (i.e., text-motion pairs) is essential, without which, models face severe out-of-domain (OOD) and out-ol-vocabulary (OOV) issues, making it diffieult generalize to the vast variety of dynamic human motions.
generative models (2), have
However, existing datasets of text-motion pairs are severely limited in scale, comprising only approxi- mately 14k samples (5), while the cost of large-scale annotation is prohibitively high. To address this issue, Zhang et al. introduce ReMoDiffuse [6], a retrieval-augmentte retrieves relevant supplementary supervision signals from 3D motion databases. While this method provides a promising direction to address the scarcity of labeled data, their effectiveness might still remain constrained by the size of existing 3D motion databases, i.c., totaling only 14k samples from datasets such as HumanMI.3D 5
generation (RAG) method that
To address this, this paper explores an innovative RAG paradigm: retrieving information from larger- scale in in-the-wild videos to supplement abundant motion signals that can enhance motion generation. Although videos represent a 2D visual modality, intuitively, the 2D human motions depicted in vidcos
*Corresponding Author: Hao Fei
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

inherently share similar characteristics with 3D human motions [7]. which can be utilized to guide the learning process of motion LLMs. Most importantly, existing video data is highly accessible and virtually unlimited in scale, and -i-the-wild videos capture diverse and unconstrained human motions, offering strong potential to address OOD/V challenges. To this end, we introduce a simple but effective framework, named VimoRAG (cf. Fig. VimoRAG first retrieves video from an unlabeled video database based on the input text, then inputs both the text and the retrieved videos into an LLM to generate motion tokens, which are finally decoded into motion sequence using VQ-VAE g
Yet this framework is nontrivial, facing at least two major challenges within such a new paradigm & Challenge-I: Current video foun- dation models (VFMs) perform poorly re- trieving human-centric videos. Our preliminary experiments indicate that existing VFMs, de- spite performing well on general-purpont video retrieval tasks (e.g., excelling at recognizing ob- jects and attributes), struggle to distinguish hu- man poses, actions, and behaviors in videos (cf. Fig. D Challenge-II: The error propagation caused by inaccurate or low-quality retrieved videos. When the retrieval quality is poor, it significantly affects the quality of the generated content. Unfortunately, this issue is not thor- oughly investigated in ReMoDiffuse [6].
ReMMaDiffae
S
fuomis
"The person appears to be mimicking the action of fiding a bicycle while standing up: alternating raising knees as pedaling.
Srput
Nehon
bandbane
VimoRAG
426k
Video
and puandi lcaers jameet oves Ri Shall
To address the first challenge, we design a Gem- ini Motion Video Retriever (termed Gemini- MVR). Gemini-MVR incorporates dual fine- grained retrieval channels, at the action level and object level respectively, where a keypoints- aware router assigns weights to these two re- trievers, allowing the system simultaneously focus on human pose features and object infor- mation in complex videos, thereby improving the accuracy of human-centric video retrieval.
Port
Figure I: ReMoDiffuse is a RAG-based motion generation method, which is limited by the small scale of motion data and its reliance on annotated captions. We propose VimoRAG, which advances in ! enabling retrieval from large-scale, in-the-wild video databases without text captions. 2 Iden- tifying and overcoming key challenges in human- centric text-to-video retrieval. 3 Ensuring align- ment between retrieved vidcos and generated mo- tions while mitigating error propagation.
To address the second challenge, we propose a Motion-centric Dual-alignment DPO training strategy (named McDPO). McDPO is to guide the LLM on how to utilize the prior information from the retrieval video (i.c., when to use it, when not to use it, and how much to rely on it) by performing self-correction.
We leverage a lightweight LLM Phi3-3.8B g following Maaz et al. 10 to evaluate the performance of this framework. To evaluate its effectiveness, we conduct both cross-domain and in-domain experi- ments. To explore its potential, we conduct scaling experiments with varying retrieval corpus sizes. Specifically, in OOD scenarios, we conduct zero-shot experiments on the challenging IDEA400 (II] test set, where we verify that VimoRAG exhibits strong generalization capabilities. We evaluate in-domain performance on the representative HumanML3D benchmark and observe that VimoRAG consistently improves most of the metrics compared to existing motion LLMs that operate with text-only input, and further exhibit sustained performance gains the retrieval corpus expands. In summary, our contributions are as follows:
To our knowledge, this paper is the first to propose a novel paradigm of video- based 3D motion RAG, which significantly alleviates the motion data scarcity bottleneck in existing methods.
We present the VimoRAG framework, which integrates two plug-and- and-play modules Peee--Gemiiiii retriever and McDPO trainer-to address two key bottlenecks: human-centric video retrieval and error propagation in cross-modal motion RAG pipelines.
Experimental results demonstrate that VimoRAG achieves substantial performance improve- ments in OOD settings and consistently enhances vanilla motion LLMs in in-domain scenarios. Furthermore, it exhibits clear potential for further gains as the retrieval corpus expands.

---

Co
The person per- forming side kick.
Generaled Tokens "345,
Codebook
Decader
datehase
LLM
Cun
VQVAE
Retrievol Model
Gomtn NVR(33.2)
Ground trith Tokens 335,, 258
Codebook
Encoder
Visual
Visual
Encoder
Projector
(53.3)
Figure 2: Overview of the VimoRAG pipeline: (1) text-to-video retrieval via Gemini-MVR, and (2) video-augmented motion generation guided by text and retrieved video. Gemini-MVR (Sec. (.2) is designed to improve cross-modal human-centric video retrieval, while the McDPO training strategy (Sec. 3.3 mitigates error propagation caused by noisy retrievals.
2
Related Work
Motion generation [12 has long been a hot research topic in the related community, aiming to generate human-like 3D motion based on a given prompt, such as text, action, or incomplete motion. Text-to-motion is among the most significant task settings and has attracted substantial research attention [13] 6 [4) For instance, T2M-GPT (I explores a generative framework utilizing VQ-
VAE 8 and Transformer [9] for motion generation. MDM (0 introduces diffusion- [21]
generative model trained across multiple motion tasks. MLD [22 enhances the latent diffusion model to produce motions conditioned on various inputs. These motion generation specialists, following in-house training, deliver high performance in motion generation.
In recent years, the emergence of LLMS 23 has demonstrated unprecedented levels of intelligence, driving the evolution from specialists to generalists. In the motion domain, motion language models central LLM, leading to motion generalists (also named motion LLMs) capable of perceiving various motions. Further, motion generators (e.g., diffusion VQ-VAE models) are integrated to achieve unified motion LLMs for both comprehension and generation 3 To achieve robust motion manipulation capabilities, these motion LLMs require fine-tuning on large annotated datasets. However, compared to comprehension tasks, motion generation is more reliant on data, but motion annotation datasets are often quite limited due to the high cost of annotation.
3 24 25 have been proposed, where motion-aware encoders are connected
In a recent study, Remodiffuse 6 introduces a motion generation method based on the retrieval- augmented generation (RAG) paradigm. performs text-to-text retrieval from labeled 3D motion database to fetch related motion signals and enhance generation quality. However, as previously mentioned, existing motion databases are typically limited in scale. In contrast, large-scale video databases are more accessible and diverse. Motivated by this, we explore a human motion-centric video retrieval framework support 3D motion generation, where notiion-onssisentt 2D features extracted from videos are effectively transferred to guide the 3D motion synthesis. Compared to Remodiffuse, our approach introduces two key innovations. First, we leverage cross-modal text-to- video retrieval to eliminate the reliance on motion databases that require manually written textual descriptions. Second, we are the first to identify the issue of error propagation in motion-RAG frameworks, and propose novel algorithm, McDPO, to address it.
Notably, several motion LLM studies 226 27 24 have also explored human-related video tasks. Inspired by MotionBank, we construct our video corpus from multiple action-centric datasets. Unlike previous works that focus building high-quality video collections, this work instead centers on validating the potential and robustness of VimoRAG framework when retrieving from a wild video corpus, and on addressing the potential inconsistency between video input and the generated motion.
3 Our Approach: VimoRAG
VimoRAG is Video-based Retrieval-augmennte Motion Generation framework. We first introduce the overall architecture and our collected video database in Section 3.1 Then we describe the details of two key components (Gimi-MVR and McDPO) in Section 3.2 and Section 3.3 respectively.

---

3.1 Preliminaries
Overall Architecture. depicted in Figure VimoRAG a pipeline composed of two essential steps. The initial step involves text-to-video retrieval, in which motion description text is used to retrieve semantically relevant videos (the rank-1 video is used in this paper) from an unlabeled wild video database with our Gimi-MVR model. The subsequent step involves video-augmente motion generation, where both the text and retrieved videos are fed into the generation model to produce the motion sequence. Leveraging our novel McDPO trainer, we facilitate the contextually aligned motion generation process.
Human-centric Video Database. In theory, it is possible to retrieve the most semantically relevant videos from all available short videos on the internet to overcome the challenges of open-vocabulary text descriptions in the industry. However, in academia, in order to efficiently verify the feasibility of our method and to advance this research direction, we gather and filter a vast human-centric video database (HeVDr consisting of 425,98 videos, soureed from 28 331
To train better retrieval model on human-centrr videos, we leverage the widely used Qwen2- VL [4 model to synthesize textual descriptions following Zhao et al [35]. We clarify that the synthetic captions are used solely to train the retriever, and are not involved in the motion RAG
pipeline. VimoRAG is ready-to-use framework that requires neither large-scale annotated videos nor text-to-text retrieval. Additionally, we enhance the dataset quality by employing AlphaPose (36) to filter out videos without human detection. More details presented in the Appendix
3.2 Gemini Motion Video Retriever
Figure 3 Illuttrates the overall architecture of our method. The model builds upon the CLIP [37 framework, where the text and video branches are encoded separately, and their similarity is computed for subsequent ranking.
Optimize Forameters
Parameters
Freene
Emooddingg
Similanty
Cooine
+ Add
Action Encoder 0,
Our model consists of two indepen- and action- level-whose outputs are fused by lightweight router to produce the final similarity score. The object-level retriever captures visual entities (objects) and their textual arguments, while the action-level retriever targets motion and predicate-level semanties.
dent setriverrs-obect--eee
Temporal
Keypoints Encoder
Keypeints Detector
Encoder
stage1
performs Statebboand grind pooping the
This design is driven by two main motivations. as discussed in the introduction: First. from our analysis of failure cases in existing VFMs, we observe that when query describes only human actions without clear environmental or object cues, VFMs often struggle to retrieve the correct video. To address this, we introduce dedicated action-level retriever. Second, we observe that many motions involve interaction with the en- vironment, making object-level cues essential for understanding actions. While existing VFMs may struggle with behavior-only queries, they encode rich general knowledge from large-scale pretraining. To leverage this strength, we retain the VFM itself as the object-level retriever (0g and do in Figure 3
boand, and landing the Bail."
stage2
performs skateboard grind. pooping the boand, and anding the Biil.
Figure 3: The architecture of the Gemini-MVR model. Op and fg represent the predicate seman- tic extractor and the argument semantic extractor, respectively. 0 and Op denote the action encoder and object encoder, respectively. We simply in- troduce a lightweight action-level retriever and routing module I, while keeping the architecture of VFMs unchanged. This twin-module design provides strong extensibility.
Specifically, for the action-level retriever, we carefully design an action encoder 0 and pred- icate semantic extractor 0p. In the action en- coder, we first extract the 2D human keypoints from the frames of an input video v. Then

---

encode each frame of keypoints to the feature space separately. After this, we add learnable position embeddings frame by frame and feed them into a temporal encoder. order to avoid information loss in the sequential encoding process, the residual operation is adopted here. To obtain a good
initial weight, we adopt the pretrained AlphaPose (d and the pretrained MotionBERT [8] as the
keypoints detector and encoder accordingly. The temporal encoder is implemented by a transformer block [19] and is initialized randomly. Op is initialized from the text encoder in Intern Video 39
We train the action-level retriever using the contrastive learning loss (Cozp is provided in the Appendix due to space limitations):
40 Luction
Lp20 + La2p
: Σ B
1
B
(xp(s(p;,a:))
Cyea
log
(I)
B Σ exp(s(p:: a,))
where p and a are embedding vectors encoded by Op and 4 for the input value i and u, respectively. B denotes the number of text-vided pairs training batch. s(.) denotes the cosine similarity. For the object-level retriever, we directly adopt Intern Video [39]. one of the most widely used VFMs, owing to its extensive common knowledge. The argument semantic extractor 0g and object encoder lo are initialized using the text encoder and video encoder of InternVideo During the fine-tuning stage, we utilize loss function Lobject similar to Laction. Let g and o denote the embedding vectors encoded by 8g and to for the input value t and v, respectively. We just replace p with g and replace a with o in Luction. and then we can get the symmetric loss function Cobject- is worth emphasizing that the two semantic extractors do not explicitly extract the semantics of predicates and arguments. We hypothesize that, through contrastive learning with different video features, each semantic extractor implicitly captures its respective focus- prediccaas and arguments.
Following the independent training of the two retrievers in stage the subsequent step involves determining effective approach to integrate the two retrievers. A key consideration is that the allocation of weights should adapt to the characteristics of different motion videos. Building on this foundation, we propose an action-aware similarity integrator model, denoted as I. Considering the two-level retrieval models can be significantly large, the optimal I should be sufficiently lightweight to minimize retrieval delay. To achieve this, employ simple linear method. The cosine similarity s(t, u) is calculated as follows:
st,, r) (p)
[1)a)((..)) Ia(a) + (a)
(2)
Io(a) + (a)
where To and In denote two output channels of I. In stage 2, the training loss function is Cinteg = Li2v Loze, where Lize calculated follows:
B log exp(s u:))
Lex
1
Σ
B
Σ exp(s(tr,
Notably, Lu2i and Liz0 exhibit a symmetric structure.
Training and Inference. Since the action-level retriever is trained from scratch, while the object- level retriever has already been pretrained on a large corpus of text-video pairs, we first pretrain the action-level retriever using a subset of the HcVD In Stage 1, we fine-tune both pretrained retrieval models in parallel, allowing for the optimization of all modules. In Stage 2, we freeze the two pretrained models and optimize only the similarity integrator. During the retrieval stage, we compute the similarity between the query text in the benchmark datasets for motion generation and each video in the HD using Equation? 2 More implementation details can be found in the Appendix D
3.3 Motion-centric Dual-alignment DPO Trainer
As illustrated in Figure 4 to fully leverage the descriptive information in the text and the rich 2D visual prior in the retrieved videos, we utilize LLM project all the information from different modalities into the language space in Stage 1. However, as mentioned in the introduction, there is an inherent gap between the visual prior and the target 3D motion, as the motion prior in the retrieved videos represents only sample of the full target motion space. Additionally, the 2D visual priors do not always align semantically with the text. To guide the L.M to learn the appropriate direction for generation when such gaps arise, construct the McDPO training set using dual-alignment reward
5

---

fome Sy frasee apdduross x Sameled Teatet Aeforenco Modiona
Milion
Parameten
Visual
Encoder
Projector
Via VAAE
The person
grodient
Space
fermnng lock
Fusion
LLM
Instruction
Semant's Embedding Space
Visual
Visual P Projecton
forning
The
Pusion
LLM
r(a,,)))) > r(a,,,y))
Stage2 Motion-centric Qual-alignment DPO
Preference Motion Takens Collection
The McDPO training strategy. Given text and retrieved video v, we first perform visual demonstration-ennaaeee based on the motion-centric dual-alignmmnt reward model, we construct preference dataset and apply DPO training. The reward model jointly measures motion similarity in the feature space and semantic consistency with the text, guiding the model learn informative motion priors and maximize preference rewards through self- iefroommroeemet..
Figure 4:
instruction tuning to establish a base reference model Tref- Then,
Table 1: Zero-shot results on IDEA400 test set. All motions are generated by the models trained on HumanML.3D training set. All results reproduced using the officially released models (codes). VimoRAG achieves the best FID score, with other metrics closely matching SoTA.
Model
FID
R-Precision
MM Dist
Diversity
Top
Top 2
Top
Motion Specialists
MoMask 13
5,982 089
0.110 003
0,195 0006
0.266=.008 006
5.625 023
7.558 119
T2M-GPT
5,359 078
0.108= 008
0.186 005
0,255=-000 006
5.773 037
7.648=-100
MDM 20
5.907 107
0.113 004
0.200 004
0.278=000 004
6.013 020
8.131
Motion Diffuse
2
5,485
0.1 0000
0.194
0.266
6.038#.000
6.884
MLD 22
5.410
0.114=.002
0,200
.05
0.270=.00x
6.005 029
7.558 10.g--0000
MotionGPT
6.202 166
0.087= 008
0.151 2. 007
0,209=.000
008
6.640=.02 025
7.684#.111 111
ReMuDiffuse
1
9.6391.069
0.110 0.110=.004
0.188 0. .00
0.256 -005 005
5.465 0:05
7.540
Motion LLMs
MotionGPT 3
5.544#-174
0.096 = 00G
0.171
0.236=
6.300 032
7.509= 009
VimoRAG (Ours)
2.388#-056
0.113
0.193
0.270=.01
5.888
7.688=1190
model. allowing LLM to learn the most informative priors to maximize preference rewards by distinguishing its own struggle cases. We describe the details the following parts.
Visual Demonstration-iie Enhanced Instruction Tuning. Given input text æ {z1, 22; ...T.
of The tokens, a system prompt S of the LLM, an instruction template T. and retrieved {U), 02 ..., Vn. of The frames, we first embed into k segment-wise embeddings E" = (E). Ey E.... } following Maaz et al. [I0] We then concatenate these elements to obtain the final imput embeddings E (emb(P)) T (emb(zz) E")] where emb(-) denotes the embedding layers of the LLM. For the target motion, we leverage the widely used VQ-VAE [8] to encode the con- tinuous motion sequence into discrete motion tokens {3/, y/, ..., Vr,, } following MotionGPT [3] Inspired by them, we fine-tune the LLM using the following instruction instruction-tunnng format in Stage 1:
video
System Prompt P: You are helpful Al assistant.
% <Juser|
Instruction Template T: Generate sequence of motion tokens matching the following human
motion description. You can use the video as a reference. Video information: { Retrieved Video Motion description: (Input Text
v)
% < lassistant]
Answer y:
(Sequence of Motion Tokens)
The loss function Laft in Stage is followed: Laft = -,, log py Ef).
Motion-centric Dual-alignment DPO. empower the motion LLM with the ability to au- tonomously adapt to vidco priors of differing quality during the generation process, we introduce

---

Table 2: Results on HumanML3D test set. "e" denotes results from original papers; others reproduced using official code. VimoRAG achieves the best FID and competitive performance across metrics among existing motion LLMs. This framework significantly improves five metrics (highlighted in red) over MotionGPT 3 (Phi3-3.8B), demonstrating the substantial advantage of incorporating video priors. The complete results with confidence intervals are shown in Table 7
R-Precision
Model
Backbone
FID
MM Dist. Diversityt
Top
Top
Top
Motion Specialists
McMask 13
0.048
0.519
0.715
0.809
2.955
9.632
T2M-GPT
0.112
0.489
0.679
0.774
3.125
9.691
MDM (20
0.454
0.419
0.606
0.712
3.636
9,449
Motion Diffuse
0.672
0.492
0.685
0.784
3.085
MLD
0.425
0.468
0.656
0.759
3.266
9.698
RcMoDiffuse
0.125
0.676
0.775
3.047
9.211
LMM 27
0.040
0.525
0.719
0.811
2.943
9.814
Motionl _ab
16
0.167
0.810
2.912
9.593
MotionI.CM
0.304
0.502
0.698
0.798
3.012
9.607
MotionCLR
48
0.269
0.544
0.732
0.831
2.806
Mo
0.492
0.681
3,096
BiPO [9
0.030
0.523
0.714
0.809
2.880
9,556
StableMoFusionn
50
0.098
0.553
0.748
0.841
9.748
MoGenTS* 51
0.033
0.529
0.719
0.812
2.867
9,570
LAMP 52
0.032
0,557
0.751
0.843
2.759
9.571
Motion LLMs
MotionGPTT22
Llama3-88
0.191
0.496
0.691
0.782
3,080
9,860
Mo
55
GPT4+Gemma-28
0.230
0.515
0.801
2.967
9.908
Wang et al
54
Llama2-138
0.166
0.519
0.803
2,964
ScaMu*
[5S
codesize 512-3B
0.617
0.443
0.627
0.734
3.340
9.217
AvatarGP1
56
Llama-1
0.567
0.389
0.539
0.623
9.489
MotionGPT
3
Llama-13B
0.567
3.775
9.006
MotionGPT
3
Phi3.3.8B
0.501
0.396
0.575
0.673
3.724
9,475
VimoRAG (Ours)
Phi3-3.8B
0.131
0.452 41%
0.655 +14%
0.764
413%
3.146.155
9.424.196
motion-centric dual-aligmment DPO training strategy. As illustrated in Figure after the completion of Stage training, we obtain a base reference model Tref- The videos used in Stage are the retrieval results of the retrieval model, handle the potential gap between text and video during training. However, Tref struggles to learn this aspect. In fact, we discover early in our experimens that Tref 's performance on the training is not stable, particularly when the gap is large. The experimental results shown in Figure support this observation (NMC-R1 worse than MC-R1).
ground truth, meaning that Tref has been learning to
To construct such a training set, we first use Tref to randomly sample x times to generate a motions' candidate set. In contrast recent works [4] 42) which rely human Al-based proxies as reward models, we devise a more efficient dual-alignment reward model. This model outputs a reward score for sampled motion sequence 0ù follows:
E(gh.
d(gs 2)
r(x, u. 9.) = - (we Ejen (is
+ wa Σjen d(ūj,
where E(.) and d(-)) denote the distribution distance and Euclidean distance computed based on the features of the two inputs, respectively. wy and wa are hyper apper-rrrmeterss to respectively control the degree of alignment within the motion modality and between the text-motion modalities. The reward model encourages the preferred motions to be closer to the reference motions in the feature space and more semantically aligned with the paired text in the semantic space. By leveraging the reward model, we collect the chosen motions Yae and the rejected motions from the existing motions' candidate set, thereby constructing DPO dataset Dapo {xx,, u, y", n/'). Finally, we adopt the following DPO training objective 4 44 Here and a denote the policy model and the logistic function, respectively. 7 here denotes the weighting coefficient.
Edpo = - (a, Da log (7 Tn (x)
To To lx,, u) Tref
llog
(5)
Tre (1
as the backbone LLM and utilize the LoRA 455 tuning method in both training stages. In Stage 1, we additionally tune the visual adapter while freezing the remaining modules. In Stage 2, all modules except for the LLM are kept frozen. Due to space constraints, further training configurations detailed in the Appendix
Training Strategy. We employ phi-3-mini
7

---

Text
VimoRAG
ReMoDiffuse
MotonGPT
MoMask
MoticaDifluse
MDM
person oppears minscking action of riding a bicycle while stonding upl olter alternating raising knees as if pedaling
The person is preponing to throw frissee, bringing The arm with the frisbes back for momentun.
He tronsitioning from relaxed stance to a boxing stance, throwing a series of punches, and then returning the relaxed stonce.
sinulating chopping
seated, They rotate their torso,
grobbing and lifting object with their hands and press then in succession
Figure 5: Zero-shot qualitative results on IDEA400 test set. All motions are directly generated by the models trained on HumanML3D training set. The text presented here only includes words related to motion due to space constraints. The full text and more results are available in Figure 12 and [3
4 Experiment
4.1 Datasets, Metrics and Baselines
We conduct extensive experiments two widely used large-scale datasets following the existing works [16). The first is the IDEA400 dataset, a high-quality whole-body motion dataset composed of 12.5K clips and 2.6M frames in MotionX [1I. which is utilized to assess OOD performance. The other dataset, HLmanML3D 5 comprising 14,616 motion clips and 44,970 text descriptions, utilized to evaluate in-domain performance. For evaluation metrics. we adopt several widely recognized measures:
Frechet Inception Distance (FID), R-recall, MulriModal Distance (MM Dist), Diversity. More details can be found in Appendix A due to the space limit.
4.2 Implementation Details
We implement VimoRAG with PyTorch. We use the same Gemini-MVR for all the text-to-motion experiments because the retrieval model is decoupled with the generation phrase in our framework. In Stage 1 of McDPO, we train 2 epochs with a learning rate 2e-4 for LoRA parameters (rank = 128, a 256), with a learning rate 2c-5 for the visual adapter's parameters. In Stage 2 of McDPO, we train 1 epoch with a learning rate 2c-4. Inference is conducted using a single NVIDIA A800 GPU, while training is accelerated using GPUs enhance efficiency. Further details regarding the model configurations, training settings, and pose representation are provided in the Appendix A
4.3 Main Results
Quantitative Results Tables I and 2 present quantitative comparison between Vi and SoTA techniques. For a fair comparison, each experiment is conducted ten times, and we report the results with a 95% confidence interval. As illustrated in Table [] VimoRAG achieves the best FID score, indicating its strong generalization capability in generating high-fidelity motions in OOD scenarios. According
Table 2 VimoRAG outperforms MotionGPT 3 by a large margin across all metrics when using the same backbone. It achieves the best FID score and competitive performance on other metrics among motion LLMs based on the Phi-3 3.8B backbone. As shown in Figure / performance improves steadily with larger retrieval sets, highlighting VimoRAG's potential to enhance motion LLMs.
Qualitative Comparison. test set. Among the results, those from VimoRAG appear to align more closely with the intended meaning of the given text. The text descriptions in IDEA400 are quite intricate and differ significantly from those in the HumDnML3D dataset. In the fourth demonstration, the text entails multiple changes in action, such as "seated, rotate, grabbing. lifting". " Among all the motions showcased, the results from VimoRAG encompass more action types related to the text. Similar phenomena can be observed in other cases as well. More results are shown in Appendix (Section B and our anonymous GitHub.
Siguresddemonstraate the qualitative comparison results on the IDEA400
8

---

Table 3: Ablation study on Hu- manML3D validation set. Gem de- notes Gemini-MVE retriever, Mc denotes McDPO, Ran denotes random retriever, Int denotes Intern Video retriever. The FID drop Gem+Mc relative to each set- ting is highlighted in blue.
Table 4: Text-to-vided retrieval performance on HcVD test sets. Compared to the object- level VFM, Gemini-MVR achieves signifi- cant improvement in the Recall metric.
Retriever
R@11
R@51
R@ 101
MnR
Human-centric Video
R- P-Prrcision?
Intern Vidco
53.6
84.5
92,3
4.2
Settings FID
Top
Top
Top 3
Gemini-MVR 58.3 f8.8%
87.3
93.7
3.6
Gem+Mc 0.148
0.429 0.625 0.756
Single Human-centric Video
Ran+Mc
0.544 172.8%
0.420
0.644 0.750
Intemn Video
52.3
84.0
91.5
4.5
Int+Mc
0.205 127.8%
0.433
0.638 0.736
Gemini-MVR 61.0 f16.6%
89.2
94.1
3.5
Gem
0.260 143.1%
0.403
0.582 0.682
4.4 Ablation Study
Impact Motion Video Retriever. To study
the the influence of the video retriever, we re-
NMc-R1
Mc-R1
Mc-Roo
NMc-Roo
place the GeM with two other retriev-
ers: one being the fine-tuned InternVides [39 model, and the other being random retrieval from the HcVD. The results in Table 3 show that random retrieval leads to a notable increase in the FID score, indicating the importance of accurate video priors in generating high-fidelity results. Moreover, replacing the retriever with internVided also leads to a rise in FID score, further confirming the effectiveness of Gemini- MVR. It is important to note that the perfor- mance of the retrieval model in RAG is influ- enced by the generation module, hence we also conduct text-to-video retrieval experiments. As shown in Table 4 Gemini-MVE achieves an 8.8% increase in R R@I for the human-centric video (pool size 1990), and 16.6% increase in R@ for the single human-centric video These results further validate the effectiveness of Genini-MVR.
0.345
R-Phccuse Sepl.
0.3
1
0.05
0.66
0.5
Figure 6: In-depth exploration of McDPO. Mc stands for McDPO setting, and NMc stands for No McDPO setting. R1 indicates the use of rank- I video during inference, while Roo indicates the use of random video during inference.
Impact of McDPO. Table demonstrates that removing McDPO leads to substantial overall performance drop, as evidenced by the comparison between the Gem+Mc and Gem settings. These findings indicate that the McDPO trainer effectively mitigates the error propagation issue, point we further analyze in detail in the discussion section 4.5
4.5 In-depth Discussion
The Effect of Retrieval Database Size. Figure illustrates the changes in the FID and MM-Dist metrics as the size of the database increases. can be observed that as the database size increases, both metrics show a decreasing trend, demonstrating VimoRAG's scalability potential with larger retrieval corpora - promising property given that wild video datasets can be easily scaled in real-world applications.
The Role of McDPO. To further analyze the role McDPO in VimoRAG, we conduct a crossover experiment involving different video priors. As illustrated in Figure 6 the FID score in the Mc-Roo setting is lower than that in the NMc-Roo setting, indicating that McDPO achieves significantly lower FID score, even when random video priors are utilized. This further suggests that McDPO effectively enables the model to disregard non-informative video priors. Moreover, we observe that McDPO also achieves significantly lower FID score and higher R-Precision score when provided with rank-I video. essence, the model appears to possess the ability to distinguish between informative and non-informative video priors, effectively utilizing relevant information

---

0.2
3,4
0.18
3.3
0.16
0.14
3.2
0.12
0.1
3.1
1.25
2,5
10
20
40
1.25
2.5
5
10
20
40
Size of Vidco Database (x 10*)
Size Vidco Database (x10*)
Figure 7: As the video retrieval database grows, VimoRAG shows steadily improving performance, demonstrating strong potential for real-world applications.
while disregarding noise. We hypothesize that this ability arises from the model's implicit alignment with these two perspectives.
Discussion
Limitations. A limitation of our work is that Vi is designed for LLMs, which results in longer processing times than those of existing smaller models (motion specialists). We evaluate the latency of our framework, with detailed results shown in Table 6 We acknowledge this constraint and intend to explore methods for reducing latency in future works.
Impact Statement. While the capabilities of our framework present significant opportunities for motion generation, they also raise ethical concerns regarding its potential misuse. Malicious users could exploit the system to create content that promotes violence or other harmful behaviors, posing risks socictal well-being. To mitigate these potential impacts, we will implement strict licensing mechanism upon the release of our method. This licensing will govern the academic research and applications of our model, ensuring that its deployment aligned with ethical standards and responsible use.
Conclusion and Future Work
In this paper, we propose VimoRAG, novel framework that integrates large-scale in-the-wild video databases to enhance motion generation for motion LLMs. We tackle two key challenges- challenges-humann humann- centric video retrieval and error coppaggttoo-throuuee the proposed Gemini-MVR model and the McDPO training strategy. Our experiments show that VimoRAG further boosts motion LLMS with substantial performance gains in both OOD and in-domain settings, and its performance steadily improves with larger retrieval corpora, showing strong scalability potential.
In future work, we aim to advance along two directions. First, will explore which types of LLMs are most suitable as the backbone of VimoRAG, and how to define appropriate metrics for automatically selecting them. While numerous LLMs available today, our focus in this paper not on identifying the the RAG system. Second, building upon the success of rideo-based RAG, plan to incorporate video, 3D data, and potentially even image data as priors to develop a unified RAG framework and investigating whether this multimodal integration can lead further performance gains.
LLM, but rather on addressing the core challenges within
Acknowledgments
This work was supported in part by Shenzhen Science (JCYJ20241 1222024120112 (0021225000 We also acknowledge supports from Guangdong Basic and Applied Basic Research Foundation 1525A1515012281, Nanjing Municipal Science and Technology Bureau 202401035 and University of Macau MYRG MYRC
Technology Program
10

---

References
[1] Jianrong Zhang, Yangsong Zhang, Xiaodong Cun, Yong Zhang, Hongwei Zhao, Hongtao Lu, Xi Shen, and Ying Shan. Generating human motion from textual descriptions with discrete representations. In Proceedings of the CVPR, pages 14730-14740, 2023.
[2] Mingyuan Zhang, Zhongang Cai. Liang Pan, Fangzhou Hong. Xinying Guo, Lei Yang. and Ziwei Liu. Motiondiffuse: Text- deiven human motion generation with diftusion model. JEEE Transactions on Puttern Analysis and Machine Intelligence, 46(6):4115 15-4128, 2024.
[3] Yaqi Zhang, Di Huang, Bin Liu, Shixiang Tang. Yan Lu, Lu Chen, Lei Bai, Qi Chu, Nenghai Yu, and Wanli Ouyang. Motiongpt: Finetuned Ilms are general- purpose motion generators. In Proceedings of the AAAI, pages 7368-7376, 2024.
[4] Biao Jiang, Xin Chen, Wen Liu, Jingyi Yu, Gang Yu, and Tao Chen. Motiongpt: Human motion as a forcign language. Proceedings the NeurIPS, pages 20067-20079, 2023.
[5] Chuan Guo, Shihao Zou, Xinxin Zuo, Sen Wang, Wei Ji, Xingyu Li, and Li Cheng. Generating diverse and natural 3d human motions from text. In Proceedings of the CVPR, pages 5152-5161, 2022.
[6] Mingyuan Zhang, Xinying Guo, Liang Pan, Zhongang Cai, Fangzhou Hong, Huirong Li, Lei Yang, and Ziwei Liu. Remodiffuse: 364-373- 2023.
motion diffusion model. In Proceedings of the ICCV, pages
[7] Liangdong Qiu, Chengxing Yu. Yanran Li. Zhao Wang. Haibin Huang, Chongyang Ma. Di Zhang, Pengfei Wan, and Xiaoguang Han. Vimo: Generating motions from casual videos. arXiv preprint 240::iv::40000 14, 2024.
[8] Aaron Van Den Oord, Oriol Vinyals, Neural discrete representation learning. Proceedings of the NewrIPS, 30, 2017.
[9] Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, et al. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404. 142.19994,,,1,, 2024.
[10] Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Khan. Videogpt+: Integrating image and vidco encaders for enhanced video understanding. arXiv preprint arXiv:2406.094116 2024.
[11] Jing Lin, Ailing Zeng, Shunlin Lu, Yuanhao Cai, Ruimao Zhang. Haoqian Wang. and Lei Zhang. Motion-x -x: large-scale 3d expressive whole-body human motion dataset. In Proceedings of the NeurIPS, pages 2023.
[12] Wentao Zhu, Xiaoxuan Ma, Dongwoo Ro, Hai Ci, Jinlu Zhang, Jiaxin Shi, Feng Gao, Qi Tian, and Yizhou Wang. Human motion generation: A survey, IEEE Transactions Pattern Analysis and Machine Intelligence, 46(4):2430 2449, 2024.
[13] Chuan Guo, Yuxuan Mu, Muhammad Gohar Javed, Sen Wang. and Li Cheng. Momask: Generative masked modeling of 3d human motions. In Proceedings of the CVPR, pages 1900-1910, 2024.
[14] Han Liang. Jiacheng Bao, Ruichi Zhang, Sihan Ren, Yuecheng Xu, Sibei Yang, Xin Chen, Jingyi Yu, and Lan Xu. Omg: Towards open- vocabulary motion generation via mixture of controllers. In Proceedings of the CVPR, pages 482-493, 2024.
[15] Guy Tevet, Brian Gordon, Amir Hertz, Amit H Bermano, and Daniel Cohen-Or Motionclip: Exposing human motion generation to clip space. In Proceedings of the ECCV, pages 358-374, 2022.
[16] Jinpeng Liu, Wenxun Dai, Chunyu Wang. Yiji Cheng, Yansong Tang, and Xin Tong. Plan, posture and go: Towards open- vocabulayyy text-to-motion generation. In Proceedings of the ECCV, pages 445-463, 463, 2025.
[17] Ke Fan. Jiangning Zhang, Ran Jingyu Gong, Yabiao Wang, Yating Wang, Xin Tan, Chengjie Wang. and Lizhuang Ma. Textual dccomposition then sub-motion-spartt scattcring for open- open-vocabulary motion generation. arXiv preprint arXiv:241 04479, 2024.
[18] Junfan Lin, Jianlong Chang, Lingbo Liu, Guanbin Li, Liang Lin, Qi Tian, and Chang-wer -enn Chen. Being comes from bot-being: voccbbulayy eoxt-to-motion generation with wordless training. In Proceedings of the CVPR, pages 23222-23231, 2023.
[19] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez. & ukasz Kaiser, and Illia Polosukhin. Attention all you need. In Proceedings the NewrIPS, 2017.
11

---

[20] Guy Tevet. Sigal Raab, Brian Gordon, Yoni Shafir, Daniel Cohen-or, and Amit Haim Bermano. Human motion diffusion model. In Proceedings of the ICLR, 2023.
[21] Jonathan Ho, Ajay Jain, and Pieter Abheel. Denoising diffusion probabilistic models. Proceedings of the NeurIPS, pages 6840-6851, 2020.
[22] Xin Chen, Biao Jiang, Wen Liu, Zilong Huang. Bin Fu, Tao Chen, and Gang Yu. Executing your commands via motion diffusion in latent space. In Proceedings of the CVPR, pages 18000-18010, 2023.
[23] Yingqiang Ge, Wenyue Hua, Kai Mei, jianchao ji, Juntao Tan, Shuyuan Xu, Zelong Li, and Yongfeng Zhang. Openagi: When Ilm meets domain experts. In Proceedings of the NeurlPS, pages 5539-5568, 2023.
[24] Ling-Hao Chen, Shunlin Lu, Ailing Zeng, Hao Zhang, Benyou Wang, Ruimao Zhang, and Lei Zhang. Motionllm: Understanding human behaviors from human motions and videos. arXiv preprint arXiv:2405.200 20340, 2024.
[25] Yuan Wang, Di Huang, Yaqi Zhang, Wanli Ouyang, Jile Jiao, Xuetao Feng, Yan Zhou, Pengfei Wan, Shixiang Tang, and Dan Xu. Motiongpt-2: A general-purpose motion- motion-language model for motion generation and understanding. arXiv preprint arXiv:2410.21777, 2024.
[26] Liang Xu, Shaoyang Hua, Zili Lin, Yifan Liu, Feipeng Ma, Yichao Yan, Xin Jin, Xiaokang Yang. and Wenjun Zeng. Motionbank: A large-scale video motion benchmark with disentangled rule-based annotations. arXiv preprint arXiv:2410.1377 2024.
[27] Mingyuan Zhang. Daisheng Jin, Chenyang Gu, Fangzhou Hong, Zhongang Cai, Jingfang Huang, Chongzhi Zhang, Xinying Guo, Lei Yang, Ying He, et al. Large motion model for unified multi-modal motion generation. In Proceedings of the ECCV, pages 397-421, 2024.
[28] Orit Klopsr-Gross, Tal Hassner, and Lior Wolf. The action similarity labeling challenge. IEEE Transactions on Pattern Analysis and Machine Intelligence, 34(3):615-65 621, 2011.
[29] Hildegard Kuehne, Hueihan Jhuang, Estíbaliz Garrote, Tomaso Poggio, and Thomas Serre. Hmdb: a large video database for human motion recognition. Proceedings of the ICCV, pages 2566,, 2011.
[30] Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. The kinetics human action video dataset. arXiv preprint 17555:17050006950 2017.
[31] Weiyu Zhang, Menglong Zhu, and Derpanis G. From actemes action: A strongly-supervised representa- tion for detailed action understanding. In Proceedings of the ICCV, pages 2248-2255, 2013.
[32] Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. UCF101: A dataset of 101 human actions classes from vidcos the wild. CoRR, abs/1212.04 12204422 2012.
[33] Amir Shahroudy. Jun Liu, Tian-Tsong Ng, and Gang Wang. Ntu rgb+ d: A large scale dataset for 3d human activity analysis. In Proceedings of the CVPR, pages 1010-1019, 2016.
[34] Peng Wang. Shuai Bai, Sinan Tan, Shijie Wang. Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang. Wenbin Ge. et al. Qwen2-vl: Enhancing vision- language model's perception of the world at any resolution. arXiv preprint 2xXx:22409.1219/ 2024.
[35] Jiaxing Zhao, Qize Yang. Yixing Peng, Detao Bai, Shimin Yao, Boyuan Sun, Xiang Chen, Shenghao Fu, Xihan Wei, Liefeng Bo, et al. Humanomni: A large vision-speech language model for human-centric video understanding. arXiv preprint (rXiv:2501.5511 2025.
[36] Hao-Shu Fang, Jiefeng Li, Hongyang Tang, Chao Xu, Haoyi Zhu, Yuliang Xiu, Yong-Lu Li, and Cewu Lu. Alphapose: Whole-body regional multi-person pose estimation and tracking in real-time. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(6)7157-77773 2022.
[37] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In Proceedings of the ICML, pages 8748-8763, 2021.
[38] Wentao Zhu, Xiaoxuan Ma, Libin Liu, Wayne Wu, and Yizhou Wang. Motionhert: A unified perspective on learning human motion representations. In Proceedings of the ICCV, pages 15085-15099, 2023.
[39] Yi Wang. Kunchang Li. Yizhuo Li, Yinan He, Bingkun Huang, Zhiyu Zhao, Hongjie Zhang, Jilan Xu, Yi Liu, Zun Wang, et al. Internvideo: General video foundation models via generative and discriminative learning. arXiv preprint 2rXiv:2212.03111. 2022.
12

---

[40] Huaishao Luo, Lei Ji, Ming Zhong. Wen Lei. Nan Duan, and Tianrui Li. Clip4clip: An empirical study of clip for end to end video clip retrieval and captioning. Neurocomputing. 508:293-304, 2022.
[41] Jenny Sheng, Matthieu Lin, Andrew Zhao, Kevin Pruvost, Yu-Hui Wen, Yangguang Li, Gao Huang, and Yong-Jin Jin Liu. Exploring text-tommotion generation with human preference. In Proceedings of the CVPR. pages 1888-1899, 2024.
[42]
Massimiliano Pappa, Luca Collorone, Indro Spinelli, and Fabio Galasso. Modipo: lext-to-motion alignment via ai-feedback- -drive
driven direct preference optimization. arXiv preprint arXiv:2405. 03803, 2024.
[43] Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D Manning, and Chelsea Finn. Direct preference optimization: your language model secretly a reward model. Proceedings of the NewrlPS, pages 53728-53741, 2023.
[44] Ruohong Zhang, Liangke Gui, Zhiqing Sun, Yihao Feng. Keyang Xu, Yuanhan Zhang, Di Fu, Chunyuan Li, Alexander Hauptmann, Yonatan Bisk. et al. Direct preference optimization of video large multimodal models from language model reward. arXiv preprint arXiv:2404.02233 2024.
[45] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint 1rXiv:210600968 2021.
[46] Ziyan Guo, Zeyu Hu, Na Zhao, and De Wen Soh. Motionlab: Unified human motion generation and editing via the www.--oooniiitiiiiiiii paradigm. arXiv preprint arXiv22502.02358 2025.
[47] Wenxun Dai, Ling-Hao Chen, Jingho Wang, Jinpeng Liu, Bo Dai, and Yansong Tang. Motionlem: Real- time controllable motion generation via latent consistency model. In Proceedings of the ECCV, pages 390-408, 2024.
[48] Ling- Hao Chen. Wenxun Dai, Xuan Ju, Shunlin Lu, and Lei Zhang. Motionclr: Motion generation and training- free editing via understanding attention mechanisms. arXiv e-prints, pages -iiv-2410, 2024.
[49] Seong- Eun Hong, Soobin Lim. Juyeong Hwang, Minwook Chang, and Hyeongyeop Kang. Bipo: Bidirec- tional partial occlusion network for text-to- synthesis. arXiv preprint arXiv:2412.00112, 2024.
[50] Yiheng Huang, Hui Yang, Chuanchen Luo, Yuxi Wang, Shibiao Xu, Zhaoxiang Zhang, Man Zhang, and Junran Peng. Stablemotusion: Towards robust and efficient diffusion based motion generation framework. In Proceedings of the MM, pages 224 232, 2024.
[51] Weihao Yuan, Yisheng He. Weichao Shen, Yuan Dong. Xiaodong Gu, Zilong Dong, Liefeng Bo, and Qixing Huang. Mogents: Motion generation based on spatial-temporal joint modeling. Proceedings of the NewrlPS, pages -30739-130763 2024.
[52] Zhe Li, Weihao Yuan. Yisheng HE, Lingteng Qiu, Shenhao Zhu, Xiaodong Gu. Weichao Shen, Yuan Dong, Zilong Dong. and Laurence Tianruo Yang. LaMP: Language-motion pretraining for motion generation, retrieval, and captioning. In Proceedings of the ICLR, 2025.
[53] Qi Wu, Yubo Zhao, Yifan Wang. Xinhang Liu, Yu-Wing Wing Tai, and Chi- KeiiKKeng Tang. Motion- ageen: A conversational framework for human motion generation with llms. arXiv preprint 2rXiv:2405./7013, 2024.
[54] Ye Wang. Sipeng Zheng, Bin Cao, Qianshan Wei, Qin Jin, and Zongqing Lu. Quo vadis, motion generation? from large language models to large motion models. arXiv preprint Xxii:2410.0331// 2024.
[55] Shunlin Lu, Jingbo Wang, Zeyu Lu, Ling-Har Chen, Wenxun Dai, Junting Dong. Zhiyang Dou, Bo Dai, and Ruimao Zhang. Scamo: Exploring the scaling law in autoregressive motion generation model. arXiv preprint arXiv: 145559... 2024.
[56] Zixiang Zhou, Yu Wan, and Baoyuan Wang. Avatargpt: All-in-one framework for motion understanding planning generation and beyond. In Proceedings of the CVPR, pages 1366, 2024.
[57] Yi Wang. Kunchang Li, Xinhao Li, Jiashuo Yu, Yinan He, Guo Chen, Baoqi Pei, Rongkun Zheng, Zun Wang, Yansong Shi, et al. Internvide02: Scaling foundation models for multimodal video understanding. In Proceedings of the ECCV, pages 396-416, 2025.
[58] Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus), arXiv preprint 160v:.6606.08415 2016.
13

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification:
In the abstract and introduction parts.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: In the Section 5.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
14

---

Justification:
No theoretical results are here.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide full details of the methodology, datasets, and experiments in Section 3 Section 4.2 and the Appendix (Section A Section C and Section D)
Guidelines:
The answer NA means that the paper does include experiments. If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
15

---

Answer:
[Yes]
Justification:
All resources are available at out github page.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https Cooeeeoodeesssseeeeee 7/nips.com /ttc: ssiniioiicyp for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: We provide full details of the methodology, datasets, and experiments in Section Section 4.2 and the Appendix (Section Section C and Section D).
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: In Table Table
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
16

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: In Section 4.2
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics httppp////.arrr....co//////////a
Answer: [Yes]
Justification: We confirm that we have reviewed and adhered to the NeurIPS Code of Ethics throughout the research process.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: the Section 5.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
17

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: In the Appendix Section
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All existing assets used in this work properly cited in the paper.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
18

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: provide full details of the datasets Appendix (Section C github link is also provided for demo access.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Question:
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
19

---

16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core methodology of this research does not involve the use of LLMs as an important, original, or non-standard component.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should be described.
wwwparippsscccoofeeeeeeeee cc/Conferencess//000
20

---

In the appendix, we present more experimental settings and results (Section A), more qualitative results (Section B). details of human-centric video database (Section more implementation details of VimoRAG (Section D).
A More Experimental Settings and Results
A.1 Experimental Settings
In the training configurations for Gemini-MVR, we set the maximum number of video frames to 16. We employ the Adam optimizer with parameters b1=0.9, b2=0.98, epsilon=le-6, and weight decay of 0.2 throughout all training stages. For the action-level retriever, we conduct training for 10 epochs with batch size of 2048 and a learning rate of lc-4. In the case of the object-level retriever, we train for 5 epochs using a batch size of 128, the learning rate is set to 4e-6 for the CLIP-related modules and le-3 for the remaining modules. Regarding the similarity integrator model, we also train for epochs with batch size of 128 and learning rate of 1e-3. Additional details are available in our code.
For the training configurations of McDPO, during Stage 1, we set the batch size to 64, weight decay to 0.0, and the maximum context length to 4096, employing a bf16 precision format. We conduet training for epochs on the HumanML3D training set. In Stage 2, we train for epoch with learning rate 2c-4, weight decay of 0.0, batch size of 8, and y 0.1 as defined in Equation [ We set the temperature of the LLM to 0.9 across all experiments. The maximum number of video frames
16 during the generation stage. Most perameerraeetess selected through grid search techniques applied to the validation sets.
Re-Precssio (TOP1)
0.46
9.9
9.7
0.44
Diversity
0.42
9.3
1.25
2.5
5
10
20
40
1.25
2.5
5
10
20
40
Size Vidco Database (x (x10*)
Size Vidco Database (x (x10*)
Figure 8: Variation of R-Precision (TOPI) and Diversity metrics function of video database size. larger database, exceeding 100,000 entries, enhances R-precision. There appears to be no evident correlation between the diversity metric and the size of the database.
Table Performance of various LoRA parameters in McDPO on the HumanML3D validation set. The best results are highlighted in bold for each setting.
R-Precision
Hank
FID
MultiModal Dist
Diversity
Top-1
Top-2
Top-3
a/rank
0.417
3,216
0.437
0.625
0.745
9.636
32
0.249
3,146
0.426
0.642
0.764
9,936
32
64
0.221
3.109
0.449
0.670
0.759
9.718
64
128
0.192
3.089
0.454
0.664
0.767
9.560
128
256
0.179
3.046
0.447
0.667
0.772
9.567
rank
128
0.235
3.110
0.452
0.656
0.770
128
128
0.156
3.114
0.445
0.637
0.754
9,728
128
256
0.179
3.046
0.447
0.667
0.772
9.567
256
32
256
0.270
3.184
0.437
0.638
0.742
9.294
64
256
0.152
3.168
0.443
0.654
0.761
9.486
2,66
0.179
3,046
0.447
0.667
0.772
9.567
21

---

Table 6: Analysis of average latency per instance during the inference phase (in seconds). Retrieval refers to the process of transforming text into video, Generation indicates the phase of generating motion tokens, and Decoding pertains to the conversion of tokens into features using VQ-VAE. It is evident that the retrieval phase does not represent bottleneck within the entire pipeline, rather, the generation process of LLM constitutes the limiting factor. Notably, the parameter size of the LLM utilized in VimoRAG is 3.8 billion.
Retrieval Time (8)
Generation Time (s)
Decoding Time (s)
Total Time (s)
Tokens/s
MotionGPT-133 VimoRAG
14.89
0.12
15.01
13.44
0.48
7.0
0.12
7.62
27.92
Table 7: Complete results with confidence intervals on the HumanML3D test set.
R-Precision
Model
Backbone
FID +
MM Dist 1 Diversity
Top
Top 2
Top
Motion Specialists
MoMask
13
0.048
0.519 000
0.715
0.809
2.955
2.955
9.632-1.094
T2M-GPT
0.112#-006
0.489#:000
0.679#-006
0.774=
3.125#.015
9.691#-062
MDM 20
0.454#.012
0.419=
0.606=0004
0.712=.004
0014
3.6362.016
9.449=-116
Diflsee
2
0.672#-025
0,492
004
0.685=
003
0.784 .004
3.085#.134
9,499=.184
MLD 22
0.425
0,468
0.656
0.759
3.266+.019
9.698 +.944
ReMoDiffuse
0.125
0.493
0,676*.004
0o0
0.775
3,047#.007
9,211 1219
LMM 2
0.040
0.525
2012
0012
0.719
0.811#
2.943+.012
9.814
Motionl Lah
40
0.1
0.810
2.912
9.593
MotionI.CM CM
47
0.304
0.502
003
0.698
0.798
3.012#.007
9.607
MotionCLR LL
0.269:1001
0.544 001
0.732
0,831=
2.8064.014
MotionGPT
0.232#-008
0.492 003
0.681#-003
0.778=
3.096#.008
9.528=t071
BiPO
40
0.030
0.523 003
0.714#
.012
0,809 002
2.880 009
9.556
StableMoFusion
50
0.098#:003
0.553 003
0.748
0.841
9.748#.092
M S1
0.033
0,529 003
0.719
0.812 .002
2.
9.570
LAMP
S2
0.032 4.00
0.557
0.751
0.843
.001
2.759
9.571
Motion LLMs
MotionGPT-2
25
Llama3-8B
0.191=2004
0.496#-002
0.691
003
0.782 064
3.080
9-860=-026
M
GPT4+ 2B
0.230 0009
4.04
0.801 + .004
2.967+.022
9.908 1002
Wang et al.
Llama2-13B
0.166 +
0,519
0,803
2.964
ScaMo* 55
codesize 512-3B
0.617
0,443
0.627
0.734
3.340
9.217
AvatarGPT
56
IJama-
0.5674
0.389
0.539
0.623
9.489
MotionGPT
Ilama- 13B
0.567
3.775
9.006 2
MotionGPT 3
Phi3-3.8B
0.501
0.396
002
0.575
0.673= 004
3.724
9.475#-110
VimoRAG (Ours)
Phi3-3.8B
0.131
0.452
002
0.655
006
0.764=0005
3.14
9.4
A.2 Metrics
For motion generation evaluation metrics, we adopt several widely recognized measures: Frecher Inception Distance (FID), which quantifies generation fidelity by measuring the distributional distance between the generated motions and reference motions feature space. R-recall, MultiModal Distance (MM Dist), which evaluate the semantic consistency between text and motions. Diversity. which assesses the diversity of the generated motions corresponding to a given textual input. For text-to- video retrieval metrics, adopt the widely used metries retrieval recall (R@) 1, @5, R@ 10 adopted in this paper), Median Rank (MdR) and Mean Rank (MnR). Recall measures the proportion of relevant results returned by the retrieval model within the top-k results. Median Rank represents the middle value of the ranks at which the correct results appear in the retrieval list. Mean Rank calculates the average position of the correct results in the retrieval lists.
A.3 More Details Datasets
IDEA400 represents high-quality subset of Motion-X X [1] consisting of a large-scale whole-body motion dataset composed of 12.5K clips and 2.6M frames. This dataset encompasses diverse array of gestures and detailed pose descriptions. Unlike existing works [16], which meticulously select
22

---

0.000
0,000
03:01
Krepcratacc..
0008
0,000
02000
Kadde Tolect
MuGin Tokes
Muden Tokess
Figure 9: The impact of varying temperature values the distribution of generated motion tokens. Notably, when the temperature set to 0.9, the distribution of the generated motion tokens closely resembles the ground truth (GT) distribution.
10000
06000
0.100
0
00485
0098
30
180
300
700
Socen
Nodze
Figure 10: The influence of different retrieval conditions on the distribution of generated motion tokens depicted.
test set with text descriptions similar to those found H33a we adopt different approach by randomly sampling 10% of the clips without any specific selection criteria. We argue that this methodology more closely resembles an out-of-distribution (OOD) scenario, wherein the test set features distribution distinctly different from that of the HumanML3D training set.
descriptions alongside body-only motions. It includes a total of 14,616 motion clips and 44,970 text descriptions, with 5,37 unique words present across these descriptions. The dataset is partitioned into a training set (80%), validation set (5%), and test set (15%).
HumanML3D [5] constitutes the largest dataset available, providing
adhere the specifications outlined in HumanML.3D (5). is important to note that the number of joints (J) is set at 22 for both HumanML.31 and IDEA400. We process the motion features for IDEA400 using the same settings HumanMI.3D, resulting in a total dimension of 263. These features include root angular velocity, root linear velocities, root height, local joint positions (velocities), and 6D rotations.
Regarding pose representation,
A.4 More Experimental Results
The Effect Video Database Size. To investigate the impact of video database size on perfor- mance, we conduct experiments at six different scales using the validation set from the HumanML3D dataset. Figure illlssraaee the changes in the Frechet Inception Distance (FID) and Multi Modal Distance IMM-Dist) metrics the size of the database increases, discussed in Section 4.3
We also report the changes in R-Precision and diversity metries under the same experimental settings. As illustrated in Figure 8 we observe that a larger database contributes to an improvement in R- Precision, particularly when the database size exceeds 100,000 entries, highlighting the advantages of utilizing large-scale video retrieval repositories. Interestingly, we find that the diversity metric does not increase in tandem with the growth of the video database size. We hypothesize that when the retrieved videos lack informativeness, the noise introduced enhances the diversity of the model's outputs. Consequently, smaller video databases can still yield substantial diversity.
Through this experiment, we demonstrate the significant potential of retrieval-enbanced enhanced methods based on large-scale video databases. However, resource constraints limit us conduct further experiments. We hope to explore the effects of even larger databases on performance in future work.
The Analysis of Lateney. We analyze the latency of the VimoRAG framework, depicted in Table/B] where we report the average time taken per instance during the inference phase. The entire pipeline is divided into three stages: the retrieval phase (conducted by the text-to-video model), the
23

---

hill
skie
speaking
window
bkig
looking
out
walking
other
holding
towards
onto
feet
tree
bends
throwing
down
walking
down
tool
proceed
group
snowy
looking
stands
ineoling
sitting
down
up
at
bork
dee
moving
pushing
adjusting
moveg
cutting
picks
pmppigg
up
Biaelyy
nunning
kicking
pullig
head
Flift
swimmingchair
brushing
walking
back
arm
away
smaing
with
race
people
him
raise
dancing
oftee
putting
game
feeding
sniting
back
holding
riding
drniknn
the
down
from
talking
climbs
white
up
hands
shirt
sody
no
playing
sitting
up
down
cup
sitting
swinging
playing
llgg
with
perees
up
trisbee
ball
with
bending down.
bending
stand
ball
over
chair
turn
standing
holding
plaa
spin
face
sit
holding
rechew
spinning
rope
knife
ground
B
theasr
stick
using
picking
jumpine
walking
place
up
up
rager
down
Tunning
la
max
coln
car
speca
skiing down
wearing
foor
top
bag
hold
hitting
pointing
shon
standing
field
az
watch
walks away
up
brush
wall
bon
begin
looking
tast
hand
bed
at
dog.
rlung
crrd
hair
perform
put
graen
tnughing
string
chair
Figure 11: Words and phrases that frequently appear in text descriptions in HDVD database.
generation phase (performed by the LLM), and the decoding phase (executed by the VQE..E.. The results indicate that the generation phase exhibits the highest latency, accounting for 92% of the total processing time. This is primarily due to the need for the LLM to load a substantial number of parameters (3.8 billion in VimoRAG). In comparison, the retrieval and decoding phases account for only 6% and 2% of the total time, respectively. This finding highlights that, despite the generation of figeliiilitty motion leveraging the world knowledge of the LLM, it remains the latency bottleneck in the overall generation framework. Consequently, this drives us to explore more efficient generative models in our future work. To illustrate the impact of different L.M sizes on latency, we also test the latency of the MotionGPT-13B model [3] the same computational hardware. As shown in Table 6 the total time for the MotionGPT-1333 is nearly twice that of VimoRAG.
Hyper- peeerppaaeeters of LoRA in McDPO. In order to investigate the impact of key hyper- hyper-parametts parameters on the performance of McDPO training, we conduct comparative experiments focusing on the LoRA fine-tuning parameters rank and a. As illustrated in Table 1, varying rank and a significantly influence the outcomes, a phenomenon that aligns with previous findings in the works 3 25 We also observe that when maintaining a constant ratio between rank and a, both the FID and MultiModal Dist metrics gradually decrease as rank and a increase. The increase in rank and a corresponds to a greater number of parameters available for optimization within the model, thereby enhancing its capability to fit the dataset more effectively. However, we note a corresponding decline in diversity, indicating that larger parameter scale adversely affects the model's ability to generate diverse outputs.
The Impact of the Temperature of LLM. As shown in the Figure 9 we visualize the impact of different temperatures on the distribution of generated motion tokens. We examine the effects of three commonly used temperatures (0.7, 0.8, and 0.9) on the results. Firstly, it is evident that the output distributions generated by different temperatures exhibit differences, indicating that the model is sensitive to this hyperparameter. Additionally, we observe that the distribution at a temperature of 0.9 is closest to the ground truth distribution. Generally, as the temperature parameter increases, the output diversity of the model also rises. We believe that a larger temperature in this study aids in enhancing the model's generalization capability.
Analysis of Distribution Resulting f'rom the Retrieval Process. As shown in Figure I0 we visualize the distribution of generated motion tokens under two conditions: one utilizing rank- I videos during inference and the other using random videos. The figure demonstrates that the distributions of motion tokens obtained from random videos and rank-1 videos are generally comparable on macroscopic level. This indicates that even when the retrieval system fails, the distribution of our model's 's generated results remains relatively close to the distributions obtained from rank-1 retrieval and the ground truth distribution from a macroscopic perspective. This demonstrates the model's strong fault tolerance and robustness. Notably, when the motion token index is less than 200, the
24

---

Table 8: Details of the resources utilized in the construction of the HcVD database.
Dataset
Number of Videos
Tasks
UCF101
32
13,320
Action Recognition
NTU RGB+D
114,480
Action Recognition
ASLAN 2X
3,697
Action Similarity Labeling
HMDB51
29
6,849
Human Motion Recognition
Kinetics-400
30
306,245
Human Action Classification
PennAction
31
2,326
Action Classification, Action Detection
MotionX
32,500
Human Mesh Recovery. Human Mesh Generation
generated distribution under the rank-1 condition closely aligns with the ground truth distribution, indicating the effectiveness of this retrieval strategy in capturing motion characteristics.
B More Qualitative Results
To comprehensively evaluate the generation performance VimoRAG in out-of-distribution (OOD) scenarios, we present more visualization results the IDEA400 dataset in Figures [2] and [3 It is noteworthy that the model utilized for generation is trained solely on the HumanML3D training set. We also present the retrieved videos the right side of each case. The cases depicted in Figure 5 are further illustrated in these two figures. In contrast, we provide full-text descriptions in these figures. It is evident that the generated motions align with the text descriptions, aided by the retrieved videos. Taking a complex description, "The person is simulating chopping wood while seated. They rotate their torso, simulate grabbing and lifting an object with their hands, bring it overhead, and then perform a striking molion downward as if impacting a log between their legs. This action is repeated, example, VimoRAG is
emalating the motion of splitling wood with an axe." in Figure 12. capable of generating such uncommon actions and seamless transitions between movements, As supplement, we have showcased some of the original videos in our anonymous repository.
C Details of Human-centrie Video Database
To train retrieval models, annotate text description for each video using the widely utilized LMM, Qwen2- "L--B--Instruc In total, we synthesize 425,988 captions. It is important to note that these text captions exclusively during the training phase of Gemini-MVR and do not employ them in any retrieval processes within VimoRAG. This means that VimoR AG also performs effectively with another large video database during the inference stage. As illustrated in Figure 11 the word cloud presents the diverse types of actions included in the HcVD database. This richness and diversity also elucidate why VimoRAG achieves exceptional performance from a different perspective. We present more details of the resources that are used for the construction of the HcVD in Table B Qwen2-VL supports dynamic frame selection (set FPS as 2.0). The prompt used for data synthesis is "Please describe the person's 's actions in the video using single sentence that contains a series of verbs.".
D More Implementation Details of VimoRAG
To enhance the reproducibility of VimoRAG, this section presents additional model details, which are also available in our code repository.
More Details of the Gemini-MVR. The temporal encoder consists of 4 transformer layers, each featuring 12 attention heads and width of 768. We utilize learnable position embeddings with context length of 77 in the action encoder. For the keypoints encoder, projection layer is placed atop MET 38), with the input and output channels of the projection layer set at 8704 and 768, respectively. To derive the final representation of the keypoints, we implement mean pooling over the encoded frames, in accordance with existing works 44 39]. The similarity integrator is implemented as a linear transformation, with an input channel of 768 and an output channel of 2. The
25

---

Gazp is defined as follows:
1
exp(s(a;: P.))
Lazp
log
(6)
B
2j=1
More Details of the Generation Model. In our framework, we employ the VFM Intern [57
the video encoder and CLIP--lrre 37 the image encoder following Maaz al. [10]
Specifically, we adopt the -eserriiidde2-Ssaae2_---22-22---
variant of Intern Video2 and the
SLLP-ViT-L114-331 model. The visual projector is a two-layer MLP with hidden size of 1024,
where GELU [58] activation function is applied after the first linear layer. For the LoRA parameters, configure the rank 128 and alpha to 256. For the model details and training configurations of VQ-VAE, we adopt the same settings as those used in existing works M1 3 Specifically, the codebook size
512x512. The temporal downsampling rate is
in the encoder of
VQ-VAE.
More Details of the McDPO Dataset. Firstly, we use the Tref model obtained from Stage to sample & times (where k = 3) on random 25% subset of the training set. The reason for extracting this subset to reduce inference costs (which similar approach used by Zhang et al. (44). For each input sample, we obtain & different outputs as candidate samples. To identify positive and negative samples from these multiple outputs, we utilize Equation to calculate the reward scores and select the cases with the highest and lowest scores as the positive and negative samples, respectively. We configure wy 0.9 and wa in Equation 4 in our experiments.
26

---

Text
Retrieved video
Generated motion
The person is performing a side kick, They balance on one leg while the other leg is lifted sideways a controlled motion exccute the lick, then kidknng leg to the starting position.
returned
The person appears to be mimicking action riding a bleycle while standing up; alternating raising lnses pedaling, and swinging arms though holding handlebars.
The person is bending over to put food on the floor for a pet, then straightening up and stepping back to standing position.
The person preparing Uhrow frisbee. Starting with stance where the weight the back foot, they shift the weight forward, bringing the arm with the frisbee back for momentum Then, they step forward with the opposite leg, rotating the torso and extending the arn to release the frishee.
The person in the images appears to he performing series of boodin punches martial arts strikes, rotating from neutral stance through sequence of punching motions with either hand while mainly remaining in place.
The person is simulating chopping wood while seated. They rotate their torso, simulate grabbing and lifting an abject with their hands, bring overlued, and then perform a striking motion downward as if impacting log between their
legs. This action repeated, emulating the motion of splitting wood with
In the sequence images, the person appears to standing upright while performing squeezing motion with their fingers hands The movements concentrated their upper extremities. The individual keeps their elbows dosee torso, with forearms perallel each other they bring their fingertips from both hands together and press them in sucession, producing rhythmic squeezing action.
Figure 12: Additional visualization results on the IDEA400 dataset (Part I).
27

---

Text
Retrieved video
Generated motion
The person standing and making a phone call gesture, They their right hand to ear as holding a phone. Their body remains relatively static while performing the gesture.
The person is squatting down and lifting potted plant while then sitting on the floor with the plant.
The person is standing upright with a rapid sequence of raising both fists from waist level to above the head and then lowering them back down in a cheering motion.
person is performing punching motion while standing stationary. transitioning from relaved throwing punches, then returning the relaved stance.
boxing stance,
The person is running in place lifting alternating knees vertically while opposite touches the ground The arms are moving sync with the legs, reciprocally swinging to maintain belance.
The person performing a stationary haskethall shooting motion. Starting from standing position, they bend knees to generate power, them,
raise the ball with both hands exend amms upwards while jumping slightly, then follow through with hand to release the ball, mimicking a basketbball shot.
The person walking back and forth in room, turning slightly each end, and appears be fanning themeeevvs continuuusly with one hand as they go.
Figure 13: Additional visualization results on the IDEA400 dataset (Part II).
28


Reviews:
[{"review_id": "i14m58RlJG", "reviewer": "Reviewer_BcNS", "summary": "This paper presents VimoRAG, a novel video-based retrieval-augmented generation framework designed to address the motion data scarcity problem in existing approaches. The proposed method introduces two key plug-and-play components: (1) GeminiMVR Retrieval: A human-centric video retrieval module that overcomes limitations of previous text-based retrieval systems. (2) McDPO Trainer: A robust training framework that mitigates error propagation in cross-modal motion generation pipelines. Experimental results show that VimoRAG delivers: (1) Significant performance gains in out-of-distribution (OOD) scenarios. (2) Consistent improvements for vanilla motion LLMs in in-domain settings. By effectively bridging the video-motion modality gap, VimoRAG represents an important advance in data-efficient motion generation.", "questions": "Please refer to the weakness", "limitations": "The limitation of this paper is discussed by the authors. \n\nThe authors provide their solutions.\n\nPlease refer to the weakness", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "### Strengths\n* Clear Presentation: The paper is well-written, logically structured, and easy to follow. The authors effectively address all questions raised in the introduction throughout subsequent sections.\n\n* Novel Contribution: The application of Retrieval-Augmented Generation (RAG) to motion generation presents an interesting approach to addressing data scarcity. While inherent modality gaps between 2D videos and 3D motion exist, the proposed solutions demonstrate meaningful progress.\n\n* Reproducibility: The availability of code and partial data significantly enhances the paper's reproducibility and practical utility.\n\n## Weaknesses & Questions\n* Fundamental Mechanism of 2D-to-3D Transfer\n\nWhile the results demonstrate empirical improvements, the theoretical foundation for how 2D videos (lacking 3D information) enhance motion generation remains unclear. Given the limited category overlap between motion datasets (HumanML3D) and video datasets (Kinetics-400, NTU-RGBD-120, HMDB).\n\nCould the authors provide a more detailed explanation of the underlying transfer mechanism? Or analysis of which specific video features contribute most to motion generation? Or potential limitations when category mismatches are more severe?\n\n*  Scalability Considerations\n\nThe current 400K video dataset, while substantial, shows limitations in category diversity (only 120-400 classes) and relevance to human motion tasks. Have the authors evaluated the noise-performance tradeoff when scaling to larger but noisier datasets? What strategies could maintain quality control with dataset expansion? Or is there a theoretical or empirical threshold for beneficial scale versus noise?", "was_revised": true, "final_justification": "The authors addressed my concerns during the rebuttal process. The paper presents a well-motivated study, and its clear, logical organization enhances its readability. I believe the discussion will provide valuable insights to the motion generation research community.", "rebuttal": "Dear Reviewer BcNS,\n\nWe sincerely thank your comments for recognizing the manuscript’s writing and contributions, and we are happy to address the points raised.\n\n---\n\n**Q1: Clarification on the theoretical foundation of 2D-to-3D transfer, particularly given limited category overlap between video and motion datasets.**\n\n**A:** We are happy to discuss this insightful question. During our early-stage investigation, we also recognized the potential concern about category mismatch. However, our analysis revealed that the HumanML3D dataset primarily consists of compositions of atomic motion units [1] (e.g., walking, running, lifting hands), rather than complex or domain-specific activities. \n\nOur video corpus, though based in part on datasets with labeled categories (e.g., Kinetics-400), also contains a large number of web-sourced videos (e.g., from MotionX) with rich and diverse motion patterns [2]. While these videos are not easily classifiable into discrete categories, they encompass **a wide range of atomic motion units** that appear frequently across different actions.\n\nWe consider the 2D-to-3D mapping to be **achieved via shared atomic action units** present in both video and motion data.\nTo verify this intuition, we conduct interpretability studies (see Appendix, Fig. 12 and Fig. 13). In the second example of Figure 12, for example, although the retrieved video does not exactly match the target motion (it misses the “swing arms” component), it includes “alternating raising knees”, which is highly relevant and beneficial for motion generation. We observe similar meaningful atomic-level alignments in many other examples.\n\n\n[1] Examples in HumanML3D\n```\n1. a person runs and jumps forward.\n2. a person walks in place.\n3. a person jogs in place with their arms swinging.\n4. a person walks forward slowly.\n5. a person lifts both hands toward face and then lowers them to their sides.\n```\n\n[2] Examples in MotionX\n```\n1. The person is simulating chopping wood while seated. They rotate their torso, simulate grabbing and lifting an object with their hands, bring it overhead, and then perform a striking motion downward as if impacting a log between their legs. This action is repeated, emulating the motion of splitting wood with an axe.\n\n2. The person is performing a stationary basketball shooting motion. Starting from a standing position, they bend their knees to generate power, raise the ball with both hands in front of them, extend their arms upwards while jumping slightly, and then follow through with one hand to release the ball, mimicking a basketball shot.\n```\n---\n\n**Q2: On the noise-performance tradeoff when scaling to larger video datasets.**\n\n**A:** We conduct an experiment on the HumanML3D validation set to evaluate the tradeoff between data scale and quality.  we add 4K noisy videos where human poses are often missing.\nThe results are summarized as follows:\n| Dataset  | FID ↓  | Top-1 ↑| Top-2 ↑| Top-3 ↑|\n|----------|-------|-------|-------|-------|\n| Base     | 0.148 | 0.429 | 0.625 | 0.756 |\n| +Noise   | 0.172 | 0.430 | 0.635 | 0.740 |\n\nWe observe a slight performance drop with noisy data, but the degradation remains limited.\nIn our pipeline, we apply basic quality control by using a keypoint detector to remove videos without human presence (as mentioned in Line 130-131), which filters out about 1% of the data. \n\nWhile more sophisticated filtering is possible, our focus lies in building a robust retrieval module and a noise-resistant training framework that can tolerate imperfect data. This design ensures scalability **without heavy reliance** on manual dataset curation.\n\n---\n\nWe'd be happy to discuss further with you if you have any new questions.\n\nBest regards,\n\nAuthors"}, {"review_id": "pEYK2eySsp", "reviewer": "Reviewer_5LZb", "summary": "1. In-the-wild videos capture diverse and unconstrained human motions.  VimoRAG uses large-scale in-the-wild video databases to enhance 3D motion generation by retrieving relevant 2D human motions.\n\n2. VimoRAG consists of  two components: Gemini-MVR retriever and McDPO trainer.  Gemini-MVR is an effective motion-centered video retrieval model , and (2) McDPO trainer mitigates the issue of error propagation caused by suboptimal retrieval results.\n\n3. Gemini-MVR includes two independent retrievers—an object-level retriever and an action-level retriever—to improve retrieval accuracy.", "questions": "See Weaknesses", "limitations": "Yes, the authors discussed the limitations about latency. But I think more other issues should be involved.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths \n1. VimoRAG proposes to use video-based 3D motion RAG, which significantly alleviates the motion data scarcity bottleneck in existing methods.\n2. The research motivation of the paper is reasonable, and the method is relatively easy to understand.\n\nWeaknesses\n1.The paper mentions that 2D visual priors may not align well with the 3D motion space, and i suggests imposing constraints on the Visual Projector to align the 2D visual and 3D motion spaces, to further improve performance. \n\n2.The experimental section lacks a direct comparison of retrieving 3D motion and retrieving videos via the current architecture, so there is no evidence to prove that retrieving videos has a greater advantage than retrieving 3D motion.", "was_revised": true, "final_justification": "All my concerns have been addressed.", "rebuttal": "Dear Reviewer 5LZb,\n\nFirst of all, we sincerely appreciate your recognition of the motivation and other key aspects of our work. We are very glad to address your concerns. \n\n---\n\n**Q1： The reviewer suggests imposing constraints on the Visual Projector to align the 2D visual and 3D motion spaces, to further improve performance.**\n\n**A:** Thank you for the insightful suggestion. We did consider imposing explicit constraints to directly align the 2D visual and 3D motion spaces. However, in our setting, this is technically difficult because the 3D motion is represented as discrete tokens (via VQ-VAE), which are not differentiable. As a result, there is no direct gradient path from the 3D motion tokens back to the 2D visual features, making end-to-end alignment infeasible.\n\nInstead, we map both the 2D visual inputs and the 3D motion tokens into a shared language space and align them implicitly through next-token prediction. This allows us to bypass the non-differentiability issue while still enabling effective cross-modal fusion.\n\n**Even so**, we find the reviewer’s suggestion very promising. In future work, we plan to explore the performance of directly projecting the hidden states of the LLM into the 3D motion space for finer alignment.\n\n---\n\n**Q2: The experimental section lacks a direct comparison of retrieving 3D motion and retrieving videos via the current architecture.**\n\n**A:** We appreciate this point. We do not include results for retrieving 3D motions using our architecture because it is not designed or pre-trained for this task, and a direct comparison might be unfair. Specifically:\n- Our visual projector and LLM are jointly pre-trained on a large-scale 2D visual dataset (CC-595K). To our knowledge, there is no 3D motion-language projector with comparable pretraining scale.\n- The retrieval pipelines for 2D videos and 3D motions differ substantially, and the retrieval quality for text-to-3D motion is still limited compared to 2D video, which would impact the generation results.\n\nIn fact, we compare our method against Remodiffuse, a strong retrieval-augmented 3D motion generation baseline. As shown in Table 2 (in-domain setting) in this paper, Remodiffuse achieves better results than our method. This is likely because it retrieves motions directly from the HumanML3D training set, which shares the same distribution as the test set. In contrast, our retrieval pool comes from a different domain, making the setting more challenging.\nHowever, in out-of-domain scenarios (Table 1 in this paper), our method outperforms Remodiffuse by a clear margin, demonstrating better generalization.\n\nTo further address the reviewer’s concern, we modify our pipeline to support 3D motion retrieval directly. We retrieve motion sequences by performing CLIP-based text-to-text retrieval on their associated textual labels, then encode the retrieved motions into motion tokens using VQ-VAE, and concatenate them with the text prompt. The results are shown below:\n\n### HumanML3D\n| Retrieval Source | FID ↓  | Top-1 ↑| Top-2 ↑| Top-3 ↑|\n|------------------|-------|-------|-------|-------|\n| Motions          | 0.427 | 0.471 | 0.651 | 0.748 |\n| Videos           | 0.131 | 0.452 | 0.655 | 0.764 |\n\n### IDEA400 (zero-shot)\n\n| Retrieval Source | FID ↓  | Top-1 ↑| Top-2 ↑| Top-3 ↑|\n|------------------|-------|-------|-------|-------|\n| Motions          | 4.640 | 0.108 | 0.168 | 0.242 |\n| Videos           | 2.388 | 0.113 | 0.193 | 0.270 |\n\nEven in the in-domain setting, 3D-motion-based retrieval underperforms 2D-video-based retrieval in this \narchitecture. This may be attributed to:\n- Poor retrieval quality from the CLIP text encoder for motion descriptions.\n- The difficulty in aligning motion tokens with the LLM input space, as LLMs are not trained on such token distributions, making both input and output alignment challenging.\n\nWe hope this clarifies our design choice and provides a fair comparison.\n\n---\n\nWe sincerely thank the reviewer once again for the valuable suggestions. We are more than happy to further discuss if there are any additional questions!\n\nBest regards,\n\nAuthors"}, {"review_id": "t529gbuIcc", "reviewer": "Reviewer_SSnd", "summary": "This paper proposes a novel video-based 3D motion RAG framework, named VimoRAG, which comprises Gemini-MVR retriever and McDPO trainer. By constructing a large-scale dataset called HcVD, the authors first trained Gemini-MVR retriever and then combine it with the McDPO training strategy to optimize the overall VimoRAG framework. The dataset preprocessing and organization are handled with notable care, leading to high-quality text–video pairs that strengthen the overall pipeline. The McDPO, as key component of the training strategy in VimoRAG, successfully bridged the gap between 2D visual representations and 3D motion tokens. This paper articulates the framework and its underlying rationale with exemplary clarity, and the extensive experiments results show that VimoRAG surpasses most baselines, thereby validating the effectiveness of the proposed approach. However, despite the well-structured design of the HcVD dataset, the selection and filter mechanism is not fully detailed. Additionally, several methodological details remain ambiguous. Therefore, I recommend an accept score for this paper.", "questions": "1.\tAs illustrated in the paper, HcVD is gathered by several existing video datasets. However, It remains unclear that whether HcVD was constructed by directly including all videos from the source datasets, or by selecting a subset of videos based on certain criteria. If the latter is the case, I would appreciate a clarification on the selection standards applied during this process.\n2.\tAs far as I understand, object-level representations are typically introduced in human–object interaction (HOI) tasks, where modeling object context is essential. However, in VimoRAG, which can be regarded as motion generation task, an object encoder is utilized within the Gemini-MVR retriever. This raises questions about the actual contribution of object encoder to the overall framework. Would you please conduct an additional ablation study accompanied along with an explanatory discussion to better clarify its impact on the overall generation performance?\n3.\tThe paper mentions that the captions in HcVD are not involved in the RAG pipeline. Does it mean the captions are excluded from the McDPO training stage, or that they are not used during model inference and evaluation?\n4.\tIn the experimental section, the HumanML3D dataset is used for evaluation. However, HumanML3D only contains motion and text data, without any video data. Given that the VimoRAG relies on retrieving videos based on input text, how is the model evaluated on HumanML3D dataset? Does this imply that, VimoRAG can operate solely with text input and bypass the video retrieval stage during inference?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Pros：\n1.\tThis paper presents a novel motion generation framework based on video RAG, which leverages a large-scale dataset named HcVD to address the long-standing issue of data scarcity in the motion generation domain.\n2.\tThe Gemini-MVR retriever, which is specifically designed for human-centric video retrieval, is trained using the proposed HcVD dataset to enhance the effectiveness of the RAG pipeline. \n3.\tA novel training strategy, McDPO, is introduced to effectively bridge the gap between 2D visual representations and 3D motion tokens.\n\nCons：\n1.\tAlthough the HcVD is well organized, its selection and filter mechanism involved in its construction process is not fully illustrated.\n2.\tSeveral methodological details remain ambiguous.", "was_revised": true, "final_justification": "The author has addressed my questions. This is a good work, and I will keep my original score. However, I must emphasize that although there are now many motion-related datasets available, a large portion of them rely on annotations generated by models such as GPT-4o, followed by insufficient human review. This has resulted in poor data quality in many datasets, which in turn harms the interests of researchers in the field (especially since reviewers often require comparisons against these flawed datasets). I hope you can carefully screen the data before releasing any public dataset.", "rebuttal": "Dear Reviewer SSnd,\n\nWe sincerely appreciate your recognition of the technical aspects of our work. We are more than happy to address each of the questions you raised in detail.\n\n---\n\n**Q1: HcVD's selection and filter mechanism involved in its construction process is not fully illustrated. Whether HcVD was constructed by directly including all videos from the source datasets, or by selecting a subset of videos？**\n\n**A:** Thank you for pointing this out. We do apply a filtering step when constructing HcVD. Specifically, we remove videos in which no human is detected in the sampled frames. This is briefly mentioned in lines 130–131 of the paper, and we will provide a more detailed explanation in the revised version.\n\nImportantly, our design philosophy is to avoid heavy preprocessing of wild video data. Instead, we aim to handle the inherent noise via a stronger retriever (Gemini-MVR) and a noise-resilient training algorithm (McDPO), rather than by manual curation.\n\n---\n\n**Q2: Several methodological details remain ambiguous.**\n\n**A:** We apologize for any confusion. We will revise the paper to include additional methodological details wherever possible to ensure better clarity.\n\n---\n\n**Q3: The impact of object-encoder on the overall generation performance.**\n\n**A:** The generation performance using only the object-level retriever has already been shown in Table 3 (Setting: _Int+Mc_). To further clarify the impact of object-level and action-level features on generation quality, we provide the generation results using the action-level retriever below:\n| Retriever              | FID ↓  | TOP1 ↑  | TOP2  | TOP3  |\n|------------------------|-------|-------|-------|-------|\n| Gemini-MVR            | 0.148 | 0.429 | 0.625 | 0.756 |\n| InternVideo (object-level) | 0.205 | 0.433 | 0.638 | 0.736 |\n| action-level | 0.342 | 0.426 | 0.630 | 0.731 |\n\nAs shown, using only action-level features yields worse performance than using only object-level features. This is primarily because action-level retrieval is less accurate. We provide retrieval performance below:\n|                | R@1 ↑ | R@5 ↑ | R@10 ↑|\n|----------------|------|------|------|\n| action-level   | 24.0 | 48.8 | 61.4 |\n| object-level   | 52.3 | 84.0 | 91.5 |\n\nThe object-level retriever is not intended to directly guide generation, but to compensate for the current limitations of action-level retrieval. If the latter were sufficiently strong, relying on object-level features might no longer be necessary.\n\nNonetheless, object interactions remain essential in some specific motion generation scenarios, especially in the IDEA-400 test set (see examples in Figures 12 and 13).\n\n---\n\n**Q4: Does it mean the captions are excluded from the McDPO training stage, or that they are not used during model inference and evaluation?**\n\n**A:** Captions in HcVD are only used to train and evaluate the text-to-video retriever. They are not used during McDPO training, model inference, or evaluation. During McDPO training, videos are retrieved from HcVD based on input motion descriptions (from motion datasets), not the video captions. The retrieval stage in VimoRAG uses text-to-video retrieval, not text-to-text retrieval, so HcVD captions are not involved. In other words, if an off-the-shelf text-to-human video retrieval model were available, we wouldn’t need to synthesize captions for HcVD at all.\n\n---\n\n**Q5: Given that the VimoRAG relies on retrieving videos based on input text, how is the model evaluated on HumanML3D dataset? Does this imply that, VimoRAG can operate solely with text input and bypass the video retrieval stage during inference?**\n\n**A:** You're correct that HumanML3D does not contain videos. For both training and evaluation on HumanML3D, we retrieve videos from HcVD using the motion descriptions from HumanML3D, and then use the retrieved videos to condition the motion generation model. Thus, the model still operates in a video-retrieval-augmented manner. Like all existing methods, we only validate the generated motions on the HumanML3D dataset, even though our model retrieves videos during generation.\n\nIt is noted that VimoRAG can be slightly modified to support text-only input. We conduct the following additional experiment to evaluate this setting:\n| Method             | FID ↓  | TOP1 ↑ | TOP2  | TOP3  |\n|--------------------|-------|-------|-------|-------|\n| VimoRAG (text-only) | 0.283 | 0.420 | 0.610 | 0.706 |\n| VimoRAG            | 0.131 | 0.452 | 0.655 | 0.764 |\n\nWhile VimoRAG can operate with text-only input, using retrieved videos yields significantly better performance.\n\n---\n\nPlease let us know if you have any further questions—we would be happy to discuss them!\n\nBest regards,\n\nAuthors"}, {"review_id": "clVUwnCWu4", "reviewer": "Reviewer_waEi", "summary": "This paper explores a new video-based retrieval-augmented motion generation framework for motion large language models.\n\nVimoRAG uses large-scale in-the-wild video databases to enhance 3D motion generation by retrieving relevant 2D human motion signals. \n\nThe authors try to address two key bottlenecks: (1) developing an effective motion-centered video retrieval model that distinguishes human poses and actions, and (2) mitigating the issue of error propagation caused by suboptimal retrieval results. \n\nThey design the Gemini Motion Video Retriever mechanism and the Motion-centric Dual-alignment DPO Trainer, enabling effective retrieval and generation processes. \n\nExperimental results show that VimoRAG significantly boosts the performance of motion LLMs constrained to text-only input.", "questions": "Please refer to weaknesses.", "limitations": "Limitations are provided in appendix section E.", "rating": 2, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 1, "strengths_and_weaknesses": "strengths:\n\n+ This paper uses some nice plots for the pipeline, such as fig 1, 2, 3 and 4.\n\n+ The comparisons show some insights of the proposed framework.\n\n+ Visualisations, while very limited, do show the effectiveness of the model, eg, Fig. 5.\n\nweaknesses:\n\n- The motivations of this work are not strong enough, also the contributions listed do not sound like contributions.\n\n- As a research paper, related work section should present enough closely related works, with differences clearly highlighted between this work and existing works. This paper only presents around 20 lines of related work, this is inappropriate.\n\n- As a main pipeline figure, it is suggested to clearly show the novel designs with enough descriptions to show the insights. For example, fig 2 caption only contains 8 words, and it is unclear how this work differs from existing works.\n\n- It is suggested to have a notation section detailing the maths symbols, operations presented in the paper. \n\n- Some nice visualisations and discussions are presented in the appendix section. The authors should consider move those key insights into main paper, and thoroughly improve the paper to make it better. The current presentations and organisations need significant improvement.\n\n- This paper also reads a bit heavily machine generated.", "was_revised": true, "final_justification": "After carefully re-reading the revised related work section and thoroughly examining the full paper again, I still feel this work lacks sufficient novelty.\n\nWhile the proposed VimoRAG framework introduces some architectural refinements, such as the Gemini-MVR retriever and the McDPO training strategy, and uses large-scale in-the-wild video data, the core idea of retrieval-augmented motion generation is not new.\n\nThe central methodology closely follows the paradigm established by ReMoDiffuse, and the retrieval-augmented generation (RAG) concept has already been explored in that and several other works (e.g., MotionBank, MotionLLM, and the MotionGPT family).\n\nThe claimed novelty of scaling RAG to 2D in-the-wild video data is an incremental step, not a fundamental conceptual advance. The work essentially extends ReMoDiffuse from 3D motion retrieval to 2D video retrieval, with engineering solutions to known challenges (e.g., retrieval quality, modality gap, alignment), but without redefining the problem or offering a genuinely new approach.\n\nGiven the field's rapid progress, meaningful novelty should go beyond scaling existing methods or plugging in improved retrievers or trainers. Despite solid execution and competitive results, this paper falls short of that bar.\n\nI therefore recommend rejection.", "rebuttal": "Dear Reviewer waEi,\n\nWe sincerely thank you for recognizing the value of our experimental results. We apologize for any confusion caused by the presentation of our motivation and greatly appreciate the opportunity to provide further clarification.\n\n---\n\n**Q1: The motivations are not strong enough; contributions do not sound like contributions.**\n\n**A:** We would like to restate our motivation and contributions more clearly. We aim to address a key challenge: existing motion datasets are small and have limited text diversity, making it hard for models to generate realistic motion from complex queries.\nWe observe that human action videos are abundant and diverse. This leads to our main research question: Can 2D human motion signals from videos guide 3D motion generation? If so, what are the challenges, and how do we solve them?\n\nWe restate our specific contributions as follows:\n- (**The  paradigm perspective**) We are **the first** to demonstrate that 2D videos effectively guide 3D motion generation, providing strong generalization ability even beyond the original dataset domain.\n- (**The technical perspective**) We find that existing text-to-video retrieval models underperform on human behavior videos. We analyze the potential reasons in detail and propose Gemini-MVR. We identify the issue of error propagation in cross-modal RAG settings and introduce McDPO training strategy.\n- (**The empirical perspective**) Experiments confirm the effectiveness of the proposed paradigm across multiple benchmarks, while also demonstrating its potential for scalability.\n\nWe believe these contributions are valuable not only for the motion generation domain, but potentially for broader applications:\n- Our text-to-human-video retrieval model has practical utility in applications such as sports clip retrieval and short video search. It outperforms strong basebones like InternVideo in this context, while maintaining a simple and extensible design.\n- Error propagation remains a major limitation in RAG-based systems. McDPO proposes a novel solution by aligning both input and output spaces, providing a generalizable framework that could benefit other LLM applications involving retrieval-based generation.\n\n---\n\n**Q2: Related work section is too brief.**\n\n**A:** We appreciate this suggestion to improve the paper’s quality. We focus on what we believe are the most closely related works, but we are happy to expand this section in the appendix (in revised version) to provide a more comprehensive overview.\n\n---\n\n**Q3: As a main pipeline figure, it is suggested to clearly show the novel designs with enough descriptions to show the insights.**\n\n**A:** We appreciate this helpful suggestion to improve the paper. Due to space constraints, we keep the figure caption concise and provide more detailed explanations in the Introduction. Although we believe the Figure are already quite clear, we agree to revise them again in the revised version. We think all these issues can be easily addressed without significant cost.\n\n---\n\n**Q4: Add a notation section.**\n\n**A:** We appreciate this helpful suggestion to improve the paper. We consider including a notation section in the revised version.\n\n---\n\n**Q5: The authors should consider move those key insights into main paper, and thoroughly improve the paper to make it better.**\n\n**A:** We appreciate this helpful suggestion to improve the paper. According to our point, the current content in the main part is more imortant than appendix. We have tried our best to present the most important content without exceeding the page limit. Of course, we would be happy to revisit this in the revised version.\n\n---\n\n**Q6: This paper also reads a bit heavily machine generated.**\n\n**A:** We apologize for any unpleasant reading experience. However, all the content in this paper is written personally by the author team. We only use LLMs to polish the grammar and word choice of a few individual sentences.\n\n---\n\nWe would be deeply grateful if you could reconsider the contributions of our paper in light of this clarification. If you have any further questions or concerns, we would be more than happy to discuss them in detail. \n\nBest regards,\n\nAuthors"}]

Author rebuttal:

Meta-review: This paper proposes a framework that leverages retrieved videos to augment 3D motion generation. A video retriever is trained on a large-scale dataset constructed by the authors, followed by an overall optimization to bridge 2D video and 3D motion while mitigating error propagation. Extensive empirical evidence demonstrates the effectiveness of the proposed method over baselines. The effort of building the data and framework with detailed analysis provides valuable insights to the community.
Decision: Accept (poster)