HMVLM:Human Motion-Vision-Language Model via MoE LoRA
Authors: Lei Hu
Yongjing Ye
Shihong Xia
Venue: NeurIPS 2025 poster ()
Keywords: Multimodal
Motion synthesis
Pose Estimation
Foundation Model
Primary area: applications
TL;DR: We propose a Human Motion-Vision-Language Model (HMVLM) based on MoE LoRA for diverse human-centric downstream tasks. A novel "zero expert" is introduced to mitigate catastrophic forgetting during the instruction-tuning.

Abstract: The expansion of instruction-tuning data has enabled foundation language models to exhibit improved instruction adherence and superior performance across diverse downstream tasks. Semantically-rich 3D human motion is being progressively integrated with these foundation models to enhance multimodal understanding and cross-modal generation capabilities. However, the modality gap between human motion and text raises unresolved concerns about catastrophic forgetting during this integration. In addition, developing autoregressive-compatible pose representations that preserve generalizability across heterogeneous downstream tasks remains a critical technical barrier. To address these issues, we propose the Human Motion-Vision-Language Model (HMVLM), a unified framework based on the Mixture of Expert Low-Rank Adaption(MoE LoRA) strategy. The framework leverages the gating network to dynamically allocate LoRA expert weights based on the input prompt, enabling  synchronized fine-tuning of multiple tasks. To mitigate catastrophic forgetting during instruction-tuning, we introduce a novel zero expert that preserves the pre-trained parameters for general linguistic tasks. For pose representation, we implement body-part-specific tokenization by partitioning the human body into different joint groups, enhancing the spatial resolution of the representation. Experiments show that our method effectively alleviates knowledge forgetting during instruction-tuning and achieves remarkable performance across diverse human motion downstream tasks.

HMVLM: Human Motion-Vision-Lanuug Model via
MoE LoRA
Lei 22
Yongjing Ye"
Shihong Xia!:21 1:2f
Institute of Computing Technology, Chinese Academy of Sciences University of Chinese Academy of Sciences (hulei19z, yeyongjing, xsh}'ict... cn
Abstract
The expansion of instruction-tuning data has enabled foundation language models to exhibit improved instruction adherence and superior performance across diverse downstream tasks. Semantically-rictt 3D human motion is being progressively integrated with these foundation models to enhance multimodal understanding and cross-modal generation capabilities. However, modality gap between human motion and text raises unresolved concerns about catastrophic forgetting during this integration. addition, developing autoregressive-compaibbe pose representations that preserve generalizability across heterogeneous downstream tasks remains a critical technical barrier. To address these issues, we propose the Human Motion- Vision-Language Model (HMVLM), unified framework based on the Mixture Expert Low-Rank Adaption(MoE LoRA) strategy. The framework leverages the gating network dynamically allocate LoRA expert weights based on the input prompt, enabling synchronized fine-tuning multiple tasks. To mitigate catastrophic forgetting during intructtiontuningg that preserves the pre-trained parameters for general linguistic tasks. For pose representation, we implement -ody-pict-speccffc tokenization by partitioning the human body into different joint groups, enhancing the spatial resolution the representation. Experiments show that method effectively alleviates knowledge forgetting during instruction-tuning and achieves remarkable performance across diverse human motion downstream tasks.
introduce novel zero expert
Introduction
With the capability of encoding semantic information and emotional expression, 3D human motion plays a critical role in virtual reality, embodied intelligence, computer graphics and visions. Recent advances in foundation language models []] 61 58 23 have facilitated multimodal integration. This has stimulated researchers' interest in embedding 3D human motion into these models to address diverse motion-centric tasks, including text-to-motion synthesis 32 76 67), motion video 5 and pose estimation [16]. Notably, M3GPT 44] develops unified vocabulary that integrates text, motion, and music modalities, supporting both text-driven and music-driven motion generation applications.
Although prior work has made progress in motion-centri multimodal modeling, two key issues remain underexplored. First, the effect of incorporating human motion modalities on the foundation
TThese authors contributed equally to this work. Shihong Xia is the corresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Input prospt: penuative infrvented who dislikes public speaking, wolnteer guod spesker al lood event....
your
Input prompt: full tumn faces opposite
Goeward stepe
Output: challenge comfort fert
[Friend's Name).
Output:
hope thi caral linds you well. I was tunking about the upcoming local event.....
Ilere few compelling
why shooldd consider
voluntcering guest speaker:
1. Opportnity shore your exportier: You w...
Input prompt:
What the pense'' pose at the begianing
Input video:
Input proupt: Can you prodict the SMPL pose of the person in Chis image?
Image:
Output::
Output: penso sitting their
fince, danumdrating oblique unmch
Figure I: HMVLM preserves the original knowledge and dialogue capabilities of the foundation model while supporting a wide range human-centric downstream tasks.
model's world knowledge is unclear. Dou al. 112 observed that supervised fine-tuning improves model'ss instruction-followii parameter deviation from pre-trained weights, progressively croding pre-existing knowledge. Al- though approaches such as temporal continual learning [57] can mitigate catastrophic forgetting in unimodal motion tasks, the substantial modality gap between human motion and text necessitates deeper examination of their compatibility within the foundation model. Otherwise, catastrophic forgetting may reduce the model to a task-specific generative system with limited dialogue abilities.
capabilities with expanding training data, yet simultancously induces
Second, how to formulate discrete motion representation compatible with autoregressive architectures foundation models remains an open research question. Prior methods typically apply temporal convolution to extract motion features along the temporal axis and utilize VQ-VAE architecture to obtain discrete tokens. However, this tokenization paradigm overlooks the spatial information of the pose, limiting the expressiveness of single frame representations in tasks like pose estimation. Therefore, developing spatially and semantically grouudegr tokenization methods for both motion sequences and static poses thus becomes imperative.
To address the first challenge, we observe that supervised instruction -uning tends to overly focus on new tokens (e.g., motion-related tokens), causing the model to gradually forget its original world knowledge. Therefore, we introduce the Mixture of Expert LoRA framework (MoE LoRA) A) for multimodal fine-tuning. This framework aims to build robust Human Motion- wwwi-ii-iio-nnaag.. Model (HMVLM) for diverse human-centric downstream tasks (as shown Fig. D. The gating network dynamically routes task instructions to multiple LoRA expert pairs (LoRA CoORR_AALLRR._ A/LLoRABii cnabling task specific adaptation. To avoid knowledge forgetting, we further propose a non-trainable zero expert with gero-initialized parameters. We encourage the gating network to select zero expert for unneeltddd tasks, thus preserving the pretrained weights the foundation model and preventing catastrophic forgetting.
To solve the second issue, we segment the human body into distinct body parts and employ spatial transformers to encode each of them separately. This part-wise encoding, inspired by patch-based tokenization in image processing [T]. enhances the resolution of motion or pose tokens while maintaining computational efficiency.
Experimental results show that the proposed HMVLM, built on the MoE LoRA framework, signif- icantly reduces the model's forgetting rate while achieving strong performance in text-to-motion generation, monocular pose estimation, and motion video understanding. The main contributions of this work are:
1. We propose HMVLM, unified framework that simultaneously supports multiple motion-relevant tasks, including text-to- cext-to-motin motion generation, pose estimation, and motion video understanding.
2. We introduce the MoE LoRA architecture for multimodal and multitask fine-tuning of HMVLM, incorporating the novel concept of a zero expert to mitigate catastrophic forgetting and preserve foundational knowledge.

---

3. We design body-part- part-based tokenizers for pose and motion, improving representation granularity and boosting downstream task performance.
Related Work
2.1 Human Motion Modeling.
Deep learning methods have been extensively employed in conditional human motion generation
and modeling tasks. These include deterministic motion prediction from historical states [17] 80
motion completion [25
54 48]. and motion control [26 72 55, 56 69]. Additionally, deep learning
approaches are widely utilized in human pose estimation tasks from RGB images or videos [3 41 37
63
[9 19
13 34 With growing demands for diverse 3D human motions, probabilistic generation
methods have emerged as a prominent research direction 42 66 24 227.. Moreover, leveraging
large-scale and uniformly formatted datasets to pre-train prior model [5 has been shown to be an effective approach for handling multiple motion-related tasks. Among them, probabilistic text-to- motion (T2M), which involves learning cross modsl mappings between textual descriptions and 3D motions, is most related to our work. Early T2M methods focused on aligning modalities by creating shared representation space for textual and motion features [4 59) MotionCLIP (59) embeds motion features into the CLIP latent space using rendered motion images. These methods faced limited motion diversity due to small datasets. Significant advancements followed the release of large- scale datasets such as KIT Motion- Motion-Language 49 and HumanMLLD [21]. Recent methods including MDM [60). MotionDiffuse 74), ReMoDiffuse 75), and MLD [6] adopted diffusion models for text- guided motion generation, operating in original or compressed motion spaces. Another group of T2M methods employs VA-VAE embed motion into discrete latent embeddings, subsequently generating motion sequences autoregressively through Transformer-base
architectures. Representative models include TM2T (22). T2M-GPT [73, AttT2M [79], and MoMask [20). In addition, several works extend T2M approaches with motion editing capabilities, enabling the generation of motions that satisfy both textual descriptions and user-defined geometric constraints [35] 52 g 2 While these methods have achieved impressive results in specific tasks, they primarily focus on human motion modeling or cross-modal learning rather than building robust multimodal frameworks capable of supporting multiple downstream tasks.
2.2 Foundation Models and Multi-modal.
Recent advances in foundation language models like ChatGPT [1, BERT [10. Llama 61. Gemma 588, and DeepSeek 23 have shown strong performance in language tasks, with excellent under- standing, text generation, and adaptability. These advancements have laid a solid foundation for
multimodal research, where instruction tuning has become central focus, giving rise to frameworks
such as vision-language models [43 64 46
40 audio-vision-languag models (71 68), etc.
In the context of human motion, Jiang et al. motion as a "foreign language" and constructs a unified vocabulary through motion tokenization to support tasks such as text-to-motion and motion prediction. Zhang et al. [76] adopted Llama-2 as the base model and applied LoRA-bases fine-tuning without modifying the word embeddings and prediction head. MotionChain (3 and MotionAgent 667 further enhance motion generation and understanding via multi-round instructions and based coordination, respectively. MotionGPT-22 (65] extends MotionGPT by incorporating hand motion to enable whole-body motion generation. Most recently, M3GPT 4F1 integrates text, motion, and music modalities into a unified framework, supporting diverse cross-modal generation tasks. Although these studies have made progress in integrating human motion into Foundational language models, their effects on the models' pre-trained world knowledge remain unexplored.
32 introduced MotionGPT, which treats 3D human
2.3
MoE LoRA Fine-tuning.
Mixture of Experts (MoE) (30 follows divide-and-coneee strategy by routing inputs to specialized experts, and has been applied across various domains [14
53
72
80
In foundation models,
architectures like Switch Transformers [15] and DeepSeekMoF model capacity without increasing inference cost. Recent work 70
9 leverage sparse routing to expand
38 45
12] integrates MoE
with LoRA, showing it can match full fine-tuning performance while enhancing generalization [22
3

---

Instruction Tuning
Modal Input
Modal Encoder
Projection
Pretrained weights
Modal (ialuce
Madel Emmheedings
AB
Output
Task Instruction
Text Tokenizer
Layer
Bo
#3
Prompt
Word Tokras
Fanbesdiiggs
Zeru Exprr
Export
Giating Network
Encoder
Figure 2: Method overview: task instructions and input prompt processed by gating network to produce a mixture weights. Modality-spedd specific inputs are aligned with word embedding via projection layers, and the final outputs generated through the pre-trained model and the weighted combination of LoRA experts.
Buehler et al. [4] further apply MoE LoRA to bioinformatical tasks such materials analysis and protein design. Building on these insights, we apply MoE LoRA to HMVLM fine-tuning. Unlike prior approaches that place experts in Transformer feed-forward layers, introduce multiple LoRA matrix pairs and employ gating network to route instructions, enabling the model to preserve base model knowledge while adapting diverse motion-related tasks.
Method
The overall framework of the proposed Human Motion-Vision-Lonn Vision-Language Model, based on MoE LoRA, is illustrated in Figure The task instructions and text prompts are encoded using the CLIP text encoder and passed the gating network w, which subsequently produces mixture of expert weights a [au, Qi...., ).). Simultaneously, modality-specif inputs (e.g., image, video, motion sequences) are projected into the foundation model's embedding space. The modal embeddings are then combined with the word embeddings and fed into the foundation model. Guided by the semantics of the task instructions and prompts, the LoRA experts are dynamically combined according to computed weights enabling task-specific modulation.
3.1 LoRA Mixture
We modulate the foundation model's pretrained weights W using multiple LoRA experts:
7 w' W + Σ a,A.B
(1)
Here, represents the total number of experts, while A, and B, are the corresponding LoRA matrices. We introduce a special zero expert, with non-trainable matrices Ap and Bn which are initialized to zero. When the gating network assigns high weight aoli.e., approaching 1). the zero expert helps preserve the pre-trained parameters W, thereby mitigating catastrophic forgetting. Beyond knowledge preservation, the zero expert weight ag indicates the task's 's reliance on the foundation model's knowledge, enabling dynamic knowledge fusion and enhancing the synergy between the model and downstream tasks. This design thus provides flexibility and robustness in multimodal, multitask learning.
as shared, general-purpose expert across tasks. Its
3.2 Multimodal Instruction Format
Given a pre-trained foundation language model fa))), where & denotes the pre-trained parameters, the objective of this work is to construct HMVLM fo(+) leveraging MoE LoRA and instruction

---

(n) Body- based Pose Tokenizer
(b) Instruction tuning for Different Human- centrri Tasks
Codebook Zzo
Ans
-20m Motion
esopp BI 251, Post B2, 3243
Motion 183,
Video of people
Motion B1 415.
bouncing and walkinng
Quantizatin
Seops
Transformee Eacoder
Rody-Par Part
Xorz Hiend
Foundation Model
LoRA &
Word Tokenbzrr
Word Tokombze
Visien Projection
Word Iokeutzze
Vision Projection
Posttiomal Eacodligg
Leamable
Linear Projection
TMM Ieta person
Estimation
Ind
Xv
O
p: parria walks cirele"
I
Premmeters
Figure 3: (a) Pose/motion tokenization scheme, we introduce learnable body-part parameters into the Transformer to facilitate feature pooling and quantization; (b) instruction tuning for diverse human-centric tasks. The discrete tokens are added to the foundation model's 's vocabulary, and then instruction tuning guides the model in generating task-related tokens.
tuning. The resulting model fo should only retain the foundation model's original capabilities and knowledge but also effectively adapt to a diverse set of downstream tasks related to human motion. The general formulation of instruction tuning as follows:
R f (I,
(2)
where I denotes the task instruction, P represents the input prompt, X' is an optional -odality-specific input and R. represents the model responses.
Text-to-motion generation:
For this task, the model input-output formulation R f (Tezm P). where Iezm specifies task-related instruction(e.g.. "an Al assistant generates motion sequence based on user description") and P contains a concrete user prompt (e.g.. "a person walks clockwise in circle."). The response R will then be encouraged contain motion-specific tokens, which can be translated into 31) motion sequences by the motion decoder.
Pose estimation: For this task. the formulation is R fo (Ipos XY), where X2 is image input and Zpos provides the task instructions for pose estimation. The output R contains pose-relevant tokens, which are then used by the pose decoder to infer the human pose from the input image.
Motion video understanding: For this task, the formulation R fy (Ivid: Pv ), where Ivid is the instruction specific to human motion video understanding. P represents the input prompt and Xiv the video input.
3.3 Multimodal Instruction Tuning
To support pose estimation and T2M tasks, we will pre-train a pose and motion tokenizer (detailed in Sec. (.4) discretize the encoding of poses and motions (See Fig. (b)), obtaining the pose vocabulary Vm and the motion vocabulary VM. These vocabularies are merged with the original text
vocabulary Vi to form extended vocabulary V [Vr VM, Vn,), while preserving the original text token order.
As illustrated in Fig. (b), MoE LoRA enables joint fine-tuning across multiple human- tasks. Given instruction data of the form (I, P,P, P, X, Rgt), all tasks share the objective of next-token prediction:
fim = - Ev [log p(R
R55))
where Rt, denotes the ground-trutt token position / the response sequence and X' an optional modality input, described in Sec.
3.2 For image input X;, we use the pre-trained CLIP
ViT-L/14 (50) with the Llava projection layer 43 to align visual features with the model's 's embedding space. For video input X., 8 frames are uniformly sampled and processed similarly.
5

---

To preserve the foundation model's 's world knowledge in untioonddreltted tasks, we supervise the gating network using user prompts from conversation datasets. Specifically, the instruction I and user prompt P are first encoded by the CLIP text encoder, and then input into the gating network w to obtain the expert weights a = [an, 011, O1,,..,, (n)) . To encourage selection of the zero expert (oo) in motion-unrelated tasks, we design the loss:
Lgot Ely log
Pwr (co [I, P)]
Here, 1) is an indicator function such that v = the input (Z, P) is unrelated to human motion, and 1) = O otherwise. This loss helps retain the foundation model's linguistic capabilities and avoid catastrophic forgetting. For human motion-relevant tasks, the gating network dynamically combines experts enhance the performance of downstream tasks. The final instruction tuning loss is given by Ltotal = fm + Lgat.
3.4 Pose and Motion Tokenizer
To obtain the vocabularies V.. and VM. and integrate the pose and motion modalities into the
foundation language model, prior works 73 79 32 76 typically use VQ-VAE discretize motion sequences. Specifically, given a motion sequence MI.F = [m . ..... m J E FxD, a motion
tokenizer & applies 1D convolutions along the temporal axis to produce latent features g1:(F/1) E(MI:F) (1 denotes the temporal compression ratio), followed by quantization:
2 = ((2,) == arg min &|
z
(5)
Here, 2 Z= [=] C CRS is the learned codebook containing K discrete latent vectors, each of dimension S. The full tokenization process is thus expressed as 2 21:(F/1) /t) ((E(M)))) However, this classical tokenization focuses solely on temporal encoding by combining discrete codes across time, which limits its capacity capture spatial granularity. In tasks like pose estimation, where only single-frame imput is involved (i.e. F = ). the accuracy of discrete encoding relies entirely on the fixed codebook. As result, its ability to represent pose variations is severely constrained by the codebook size K, leading to coarse-graines representations.
To address this limitation, we draw inspiration from patch-based image encoding for spatial modeling 11. We exploit the body's 's natural decomposition into limb-based parts. Part-aware modeling has proven effective in motion retargeting (28), motion style transfer [31], and text-to-motion 779 yet
most prior work does not compute discrete codes independently for each part.
In this work, we adopt the spatial Transformer architecture from 28 and build body part-based pose and motion tokenizers, with the architecture illustrated in Fig. 3 each joint feature using a linear projection and go through the positional encoding with the learnable body-part parameters. During the self-attention computation, an attention mask matrix is constructed based on the correspondence between joints and body parts, ensuring that the body part parameters are only associated with joints within that part. Finally, we keep only the outputs corresponding to the body part parameters for pooling the pose embeddings. The spatial modeling process can be formulated as:
(a). For a given pose m/. we process
13/1 -/. 2/x1 =Ex(m²)
(6)
where N denotes the number of body parts. For single-frame frame pose input, separate codebook is constructed for each body part, and the embeddings are quantized independently as 2/20 Q, ( For motion sequences, the spatial embeddings from E, are further compressed along the temporal axis using a temporal convolution module &,, yielding:
(5' 22
& E = E2(2) Bny 27
The tokenization processes for pose and motion are expressed as (I,N(E,(m5) and C1-N (E, (M)), respectively. Following tokenization, separate decoders Dm and Dxs are employed to reconstruct the original input as detokenizers. We adopt the training strategy of T2M- GPT 73 with the loss function Ls Lree + Lemb + Aeom Lron. Specifically, Lree is the reconstruction loss and Lemb used to update the codebooks, while Loom encourages the body part embeddings to remain close to their assigned codebook vectors.

---

4 Experiments
Implementation Details. We use Vicuna-7b-v1.55 7 as the foundation language model with five LoRA experts (including zero expert), each of rank 8. LoRA adapters applied to all linear modules, and the gating network implemented as a two-layer MLP with a hidden dimension 512. It takes the 51 di-dimmenionnal text features output by the CLIP model and predicts the weights for five experts. For detailed implementation, please refer to the Appendix.
Datasets. We train the gating network with the LMSYS SeyYS-Cha--II IM dataset 777. using 80% of the data for training. For the text-to-motion task, we use Human [2T] and KIT-MI. [49 datasets. Notably, the motion tokenizer is trained on the same training splits of HumanML.3D and KIT-MI. for consistency. For pose estimation and pose tokenizer training, we use the Human3.6k (29 and 3DPW [62] datasets. The MoVid dataset [5 is used for instruction tuning in motion video understanding.
4.1 Evaluation on the knowledge preservation
We assess how human motion modalities affect the model's knowledge retention. Specifically, we measure model forgetting by comparing text comprehension performance before and after text-to- motion instruction tuning, using the MT-Bench [78]. MT-Bench evaluates 80 questions across eight topics, each with two-turn dialogues. GPT-4 serves the judge, scoring responses from I (completely incorrect) to 10 (fully correct). We use this to measure performance changes after T2M fine-tuning.
Table I: Evaluation on dialogue abilities of foundation models before and after text-to-motion tuning
Methods
FM
Write
Role
Extract
Reason
Math
Code
Stem
Code
Avg
Llama2
2.70
3.63
2.25
4.00
1.50
1.15
3.50
3.10
2.73
MotionGPT 65
Tuned
1.85
2.75
1.55
2.50
1.45
1.20
3.00
2.58
2.11
Gemma2
8.83
8.65
7.90
7.25
5.80
5,30
8.93
9,70
7.79
MotionAgent
Tuned
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
Gemma2
8.83
8.65
7.90
7.25
5.80
5.30
8.93
9.70
7.79
Ours
Tuned
8.45
8.55
7.85
6.75
5.75
5.20
8.20
9.45
7.53
Vicuna
7.43
7.52
5.21
4,90
3.69
2.68
6.98
5.90
Ours
Tuned
7.75
6.20
5.80
4.50
3.00
2.45
6.45
8.15
5.54
Ours
Vicuna
7,43
7.52
5.21
4,90
3.69
2.68
6.98
9.0
5.90
w/o Lget
Tuned
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
We conduct quantitative comparison between our method and two representative baselines, Mo- tionGPT [32] and MotionAgent (67, with results summarized in Table T Since these methods built on different foundation models, absolute MT-Bench scores not directly comparable. Therefore, we primarily focus on the relative degradation in dialogue performance of each foundation model after instruction-tuningg for the text-to-motion task.
As shown, MotionGPT, which applies LoRA solely the Query and Value matrices without in- troducing new motion tokens, preserves part of the original linguistic capability but still exhibits a noticeable 22.71% performance drop (2.73 > 2.11). In contrast, MotionAgent suffers from drastic 87.16% performance degradation (7.79 - 1.00), with MT-Bench scores across all topics collapsing to (rated as "completely unreasonable" by GPT-4). This severe collapse primarily attributed to overfitting on the newly introduced motion-specific tokens (e.g., Motion_indes resulting in catastrophic forgetting of linguistic knowledge.
In comparison, our approach achieves effective task decoupling and knowledge preservation through the proposed MoE LoRA framework. Using the same foundation model, Gemma-2-2b-it, as Motion- Agent, our method demonstrates only a marginal 3.34% degradation (7.79 - 7.53). This clearly highlights the superior capacity of our framework to retain foundational knowledge while adapting to new modalities and tasks. Moreover, our MoE LoRA framework is model-agnostic and can be seam- lessly integrated into the instruction tuning process of various foundation models. When applied to Vicuna-7b-vl. the primary foundation model used in our study, fine-tuning with MoE LoRA results in only a 10% performance drop (5.90 - 5.54), further demonstrating the broad applicability and robustness of our framework. Additional qualitative results are provided in Appendix A.
7

---

4.2 Evalution on Text-to-Motion Task
For text-to-motion task, we compare our HMVLM with state-of-the art methods on HumanML33D dataset. Following prior work [1 we use four evaluation metrics: R Poriiion(ToplToppa.. and Multi-modal Distance(MM-D) for text-to-motion retrieval accuracy, Frechet Inception Distance (FID) for motion realism, and Diversity (Div.) for motion variation.
Table 2: Quantitative results of text-to-motion on the HumanM1.3D dataset
precision
Methods
FID.
MM-D.4
Div.
Top-1
Top-2
Top-3
GT
0.511 4.003
0.703 4:003
0.797
#.002
0.002 0.000
2.974 1:008
9.503 #:066
TM2T
22
0.424 4.003
0.618
+.0
0.729 + .00
1.501 1.017
3.467
#.011
8.589 1.076
T2M
0.455 4.003
0.636
+:003
0.736 + 002
1.087 021
3.347 +.008
9.175 4.0
MDM
0.320 1.005
0.498
1.004
0.611 1: 1.007
0.544 1.044
5.566
11.077
9.559 110866
MD
74
0.491 1.001
0.681
1.001
0.782 1. 001
0.630 1.001
3.113 1.001
9,410 1.049
MLD
6
0.481 1.003
0,673
1.003
0.772 1. .002
0.473 1.013
3.196 1..10
9.724 1.082
T2M-GPT
73
0,491 +.003
0,680
+00
0.775
0.116 0.04
3.118 #.011
9.761 +.081
ReMoDiffuse
0.510 +.005
0.698 000
0.795
+ 004
0.103 000
2.974 2.016
9,018 0.075
AtTT2M
79
0.499 4.003
0,690
#.002
0.786
000
0.112 006
3.038 007
9.700 9.0
MoMask 20
0.521 4.002
0.713 1.002
0.807
1.002
0.045 002
2.958 1.008
9.620 1.064
MotionGPT 76
0.364
1.005
0.533 1:003
0.629
1: .00
0.805
+.002
3.914 1.013
9.972 1.026
MotionGPT 332
0.492 4.003
0,681 4.003
0.733
# 006
0.232 4.008
3,096 +.008
9.528 #.071
MotionAgent 66
0.482 1.00
0.672 1.003
0.770 + 2.002
0.491 +.019
3.138 1.010
9,838 1.244
MotionGPT-2
0.496 +.002
0.691 4.003
0.782 + 004
0.191 0.004
3.080 +.03
9.860 1.026
Ours (single task)
0.502 + .003
0.692 +: 004
0.785
+ 002
0.123 = 004
3.039
020
9,443 131
Ours
0.463 + 006
0.646 + 004
0.744 = 001
0.156 000
3.328 4.004
9,544 161
Tab. presents quantitative comparisons. Methods the upper section are task-specific T2M models trained from scratch, whereas methods in the lower section are multimodal frameworks based on foundation language models, fine-tuned via instruction tuning. Among these, MoMask 20 achieves tate-of-tthe-ar results in most metrics due to its cascaded mask Transformer design; however, such specialized approaches often lack scalability and generalizability for broader multimodal tasks.
Among multimodal foundation models, we evaluate our proposed HMVLM in two settings (lower section of Tab. 2). Ours (single task) denotes evaluation with only T2M task fine-tuning, using the same MoE LoRA architecture. Our model demonstrates remarkable performance across most metrics, thanks to dynamic expert assignment via MoE and fine-grained body-part tokenization (detailed in Sec. 4.5). Under multi-task fine-tuning (Ours), HMVLM remains competitive across all metrics, although performance decreases compared the single-task setting. This is because, in single-task setting, all LoRA experts focus solely on one downstream task within the same parameter budget.
Input Inage
ChatPose
Ours
Input Image
ChatPose
Inpur Prospt: people doing?
Input Prompt: many steps did person video they move? Inptut Video:
Input Video:
Dutput: peoplein in are playing
Output: The person starts sitting their hands and knees the floor, demonstrating seated oblique crunch evercise.
Figure 4: Qualitative results for human pose estimation and human video understanding.

---

4.3 Evaluation on Human Vision Tasks
Human Pose Estimation. We follow the evaluation setup of ChatPose [6 using the MPIPE (Mean
Per-Joint Position Error) and PA-MPJPE (Procrustes-Alignee MPJPE) as metrics. As shown in Tab. our method surpasses ChatPose comparable
mooee-mbasedd approach, in the single-task fine-tuning scenario. This result highlights the advantage of the MoE LoRA architecture
enabling fine-grained expert assignment. Qualitative comparisons with ChatPose, presented in Fig. demonstrate our method's superior accuracy in capturing limb details using examples from the
4
Human3.6M (left) and MoViD (right) datasets, validating the effectiveness of our body-part-base tokenization strategy.
Motion Video Understanding. Fig- illustrates example of our model's performance in human motion video comprehension and reasoning tasks. Our method successfully identifies motion categories (e.g., baseball motion at the top-left) and exhibits spatio- -empooll reasoning capabilities. For instance, the bottom-left corner shows the model accurately determining the number of steps climbed, while the right side provides concise descriptions of the character's movements and postures.
4.4 Expert Weight Distribution
We analyze the average expert weights from the gating network across different tasks to evaluate the MoË LoRA architecture's -ecoo-pengliing capability. Specifically, we extract textual features by inputting each task's instructions and prompts into CLIP's text encoder, then compute the expert weights using the gating network. As shown Table 3 for the general dialogue task (GD), the loss in Equation 4 encourages the gating network prioritize the zero expert, thereby preserving the pretrained parameters the foundation model. For other tasks, the gating network dynamically adjusts the expert weight based on the instruction and prompt, reflecting the idea of divide and conquer.
Table 3: Average gating weights across different tasks. GD, T2M, HPE, and HVU denote general dialogue, text-to-motion, human pose estimation, and human video understanding tasks, respectively.
Task
Zero Expert
Expert
Expert 2
Expert 3
Expert 4
GD
0.999
2.364x 10-6 6
5.005 x 10
1.357 10 10-6
1.827 10-6
T2M
0.694
0.052
0.067
0.085
0.102
HPE 0.454
0.292
0.084
0.106
0.063
HVU
0.167
0.252
0.150
0.013
0.418
Table 4: Quantitative results of pose estimation on the H3.6M and 3DPW datasets
H3.6M
3DPW
Methods
MPJPEL
PA-MPIPEL
MPJPE4
PA-MPIPEJ
SPIN 36
61.9
42.6
102.9
62.9
HMR 2.0
19
50.0
33.6
91.0
58.4
ChatPose [66
126.0
82.4
163.6
81.9
Ours(single task)
92.8
55.3
105.3
56,24
Ours
114.7
64.8
127.7
63.3
4.5 Ablation Study
HMVLM incorporates the MoE LoRA architecture and body-part- tokenization strategy. Accordingly, conduct ablation studies focusing these two key components and model efficiency.
Effectiveness Body Part-based Tokenization. To evaluate the spatial modeling capability of our proposed hody-part-based tokenization, we conduct ablation studies on both motion reconstruction and T2M performance. As shown in Tab. the baseline tokenizer ("Woo BP") corresponds to the standard whole-body motion tokenizer described in Sec. 3.4 Although enlarging the codebook size

---

Scating Trainin Time Parameters
rence anency and Perfrrance
96
-0 Theiing Time
1.50
Thaing Time
0.59
Trahiing Time (rourslopch)
Todtro Tine (mc/)cke))
4048
-20
120
045
120
041
100
04
Namiae
(a)
(b)
Figure 5: Efficiency analysis of the MoE LoRA model under different numbers of experts. (a)
Training time and parameter scaling. (b) Inference latency and T2M performance.
(i.c., K number of body parts) increases the model capacity, using a single codebook for the entire body fails to enhance spatial expressiveness. In contrast, our wody-part-based tokenizer yields clear improvements in R-precision and reconstruction MSE, demonstrating its superior spatial modeling ability. The slightly higher FID may result from combining multiple part-specific codebooks, which increases pose diversity but introduces minor distribution shift.
Table 5: Abalation study different tokenizers. K represent the codebook size.
Methods
R precision(Top-33)
FID.
MM-D.4
Div.
MSE. 4
Ours (single task)
0.785 0.0
0.123 2.004
3.039 027
9,443 131
0.966
W/o BP (K=512)
0.741 1.001
0.336 2.012
1.291+.003
9.000 2.263
1.377
W/o BP (K=512#5)
0.758
0.110
3.232 016
9.508 2.212
1.34
Effectiveness of Cyat. We investigate the impact of the Cyot on preserving the foundation model's world knowledge. As shown in Tab. removing the Laat results in catastrophic forgetting, attributed to modifications in pre- trained parameters and changes in the prediction head that make the model overly focussed newly introduced motion tokens. These findings support the effectiveness of combining the the MoE LoRA architecture with Lyat for building robust multimodal frameworks while mitigating catastrophic forgetting.
We evaluate the efficiency of the MoE LoRA architecture under different numbers of experts. As shown in Fig. 5Ja,, the number of trainable parameters and training time increase nearly linearly with the number experts, as each LoRA expert shares the same rank. For inference latency, as shown in (b), the LoRA weights must be dynamically combined according to the gating network's 's output, preventing pre-mes vre-megging with the pretrained model and causing a moderate rise in latency. We also observe the T2M Top-1 R-precision improves but gradually saturates as experts increase. Considering the trade-off between efficiency and performance, we adopt five experts.
Efficiency of MoE LoRA.
5 Discussion
In this paper, we present HMVLM. a MoE LoRA-based multimodal framework designed for a range of human-centric tasks, including motion perception, comprehension, and generation. By leveraging zero expert, our approach preserves the foundation
the MoE architecture and the introduction model's world knowledge and generative capabilities during instruction tuning. Furthermore, incorporate a spatial Transformer to independently encode body part features into discrete tokens, enabling precise and fine- fine-grained motion and pose representations.
Limitations and Future Work. While HMVLM advances the joint modeling human motion, language and vision, several limitations remain. The modality connections are still learned in independent pairwise manner, limiting holistic integration. Additionally, domain discrepancies across datasets hinder the model's ability perform seamless any-to-any generation.
10

---

Acknowledgment
This work was supported by National Key R&D Program of China (NO. 2022YFB3303202) and the Innovation Funding of ICT, CAS under Grant NO. E561010
References
[1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidr, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
[2] Nikos Athanasiou, Alpár Cseke, Markos Diomataris, Michael J Black, and Gül Varol. Motionfix: Text-driven 3d human motion editing. In SIGGRAPH Asia 2024 Conference Papers, pages 1-11, 2024.
[3] Federica Bogo, Angjoo Kanazawa, Christoph Lassner, Peter Gehler, Javier Romero, and Michael J. Black. Keep it SMPL: Automatic estimation of 3D human pose and shape from single image. In Computer Vision - ECCV 2016, Lecture Notes in Computer Science. Springer International Publishing. October 2016.
[4] Eric L Buchler and Markus J Buehler. X-lora: Mixture of low-rank adapter experts, flexible framework for large language models with applications protein mechanics and molecular design. APL Machine Leurning, 2(2), 2024.
[5] Ling-Hao Chen, Shunlin Lu, Ailing Zeng, Hao Zhang, Benyou Wang, Ruimao Zhang, and Lei Zhang. Motionllm: Understanding human behaviors from human motions and videos. arXiv preprint
2024.
[6] Xin Chen, Biao Jiang, Wen Liu, Zilong Huang, Bin Fu, Tao Chen, and Gang Yu. Executing your commands via motion diffusion latent space. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 18000-18010, 2023.
[7] Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, lon Stoica, and Eric P. Xing. Vieuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023.
[8] Setareh Cohan, Guy Tevet, Daniele Reda, Xue Bin Peng, and Michiel van de Panne. Flexible motion in-betwenning with diffusion models. In ACM SIGGRAPH 2024 Conference Papers, pages 1-9, 2024.
[9] Damai Dai, Chengqi Deng, Chenggang Zhao Rx Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Y Wu, et al. Deepseekmoe: Towards ultimate expert specialization in mixture-of-experts language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume Long Pupers), pages 1280-1297, 2024.
[10] Jacob Devlin, Ming-Woi Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), pages 4171-4186, 2019.
(11) Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Heigold, S Gelly, et al. image is worth 16x16 words: Transformers for image recognition scale. In International Conference on Learning Representations, 2020.
[12] Shihan Dou, Enyu Zhou, Yan Liu, Songyang Gao, Wei Shen, Limao Xiong, Yuhao Zhou, Xiao Wang, Zhiheng Xi, Xiaoran Fan, et al. Loramoe: Alleviating world knowledge forgetting in large language models via moe-style plugin. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1932-1945, 2024.
[13] Sai Kumar Dwivedi, Yu Sun, Priyanka Patel, Yao Feng, and Michael J. Black. TokenHMR: Ad- vancing human mesh recovery with a tokenized pose representation. In JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), March 2024.

---

[14] David Eigen, Marc' Aurelio Ranzato, and llya Sutskever. Learning factored representations in deep mixture of experts. arXiv preprint arXi4:/312.4314, 2013.
[15] William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(120):1-39, 2022.
[16] Yao Feng, Jing Lin, Sai Kumar Dwivedi, Yu Sun, Priyanka Patel, and Michael J Black. Chatpose: Chatting about human pose. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2093-2103, 2024.
[17] Katerina Fragkiadaki, Sergey Levine, Panna Felsen, and Jitendra Malik. Recurrent network models for human dynamics. In Proceedings of the IEEE international conference on computer vision, pages 4346-4354, 2015.
[18] Yang Gao, Po-Chien Luan, and Alexandre Alahi. Multi-transmotion: Pre-trained model for human motion prediction. arXiv preprint arXiv:2411.026735 2024.
[19] Shubham Goel, Georgios Pavlakos, Jathushan Rajasegaran, Angjoo Kanazawa, and Jitendra Malik. Humans in 4d: Reconstructing and tracking humans with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 14783-14794, 2023.
[20] Chuan Guo, Yuxuan Mu, Muhammad Gohar Javed, Sen Wang, and Li Cheng. Momask: Generative masked modeling of 3d human motions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1900-1911 2024.
[21] Chuan Guo, Shihao Zou, Xinxin Zuo, Sen Wang, Wei Ji, Xingyu Li, and Li Cheng. Generating diverse and natural 3d human motions from text. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5152-5166 51611 2022.
[22] Chuan Guo, Xinxin Zuo, Sen Wang, and Li Cheng- Tm2t: Stochastic and tokenized modeling for the reciprocal generation of 3d human motions and texts. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXV, pages 580-597. Springer, 2022.
[23] Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint arXiv:2501.12944 2025.
[24] Bo Han, Hao Peng, Minjing Dong, Yi Ren, Yixuan Shen, and Chang Xu. Amd: Autoregressive motion diffusion. In Proceedings of the AAAI Conference Artificial Intelligence, volume 38, pages 2022-2030, 2024.
[25] Félix G Harvey, Mike Yurick, Derek Nowrouzezahrai, and Christopher Pal. Robust motion in-betweening. ACM Transactions on Graphics (TOG), 39,,) 2020.
[26] Daniel Holden, Taku Komura, and Jun Saito. Phase-functioned neural networks for character control. ACM Transactions on Graphics (TOG), 36(4):1-13, 2017.
[27] Lei Hu, Zihao Zhang, Yongjing Yc, Yiwen Xu, and Shihong Xia. Diffusion- based human motion style transfer with semantic guidance. In Computer Graphics Forum, volume 43, page c15169. Wiley Online Library, 2024.
[28] Lei Hu, Zihao Zhang, Chongyang Zhong, Boyuan Jiang, and Shihong Xia. Pose-aware attention network for flexible motion retargeting by body part. IEEE Transactions on Visualization and Computer Graphics, 2023.
[29] Catalin lonescu, Dragos Papava, Vlad Olaru, and Cristian Sminchiseseu. Human3.6m Large scale datasets and predictive methods for 3d human sensing in natural environments. IEEE Transactions Pattern Analysis and Machine Intelligence, 36(7): 1325-1339, jul 2014.
[30] Robert A Jacobs, Michael Jordan, Steven J Nowlan, and Geoffrey E. Hinton. Adaptive mixtures of local experts. Neural computation, 3(1):79-87, 1991.
12

---

[31] Deok-Kyeone Jang, Soomin Park, and Sung-Hee Lee. Motion puzzle: Arbitrary motion style transfer by body part. ACM Transactions on Graphics (TOG), 41(3):1-16, 2022.
[32] Biao Jiang, Xin Chen, Wen Liu, Jingyi Yu, Gang Yu, and Tao Chen. Motiongpt: Human motion as foreign language. Advances in Neural Information Processing Systems, 36:20067-20079, 2023.
[33] Biao Jiang, Xin Chen, Chi Zhang, Fukum Yin, Zhuoyuan Li, Gang Yu, and Jiayuan Fan. Mo- tionchain: Conversational motion controllers via multimodal prompts. In European Conference on Computer Vision, pages 54-74. Springer, 2024.
[34] Boyuan Jiang, Lei Hu, and Shihong Xia. Probabilistic triangulation for uncalibrated multi-view 3d human pose estimation. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pages 14850-14860, 2023.
[35] Korrawe Karunratanakul, Konpat Preechakul, Supasorn Suwajanakom, and Siyu Tang. Guided motion diffusion for controllable human motion synthesis. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2151-2162, 2023.
[36] Nikos Kolotouros, Georgios Pavlakos, Michael J Black, and Kostas Daniilidis. Learning to reconstruct 3d human pose and shape via model-fitting the loop. In Proceedings of the IEEE/CVF international conference on computer vision, pages 2252-2261, 2019.
[37] Chen Li and Gim Hee Lee. Generating multiple hypotheses for 3d human pose estimation with mixture density network. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9887-9895, 2019.
[38] Dengchun Li, Yingzi Ma, Naizheng Wang, Zhiyuan Cheng, Lei Duan, Jie Zuo, Cal Yang, and Mingjie Tang. Mixlora: Enhancing large language models fine-tuning with lora based mixture of experts. CoRR, 2024.
[39] Jiefeng Li, Chao Xu, Zhicun Chen, Siyuan Bian, Lixin Yang, and Cewu Lu. Hybrik: A hybrid analyt inverse kinematics solution for 3d human pose and shape estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3383-3393, 2021.
[40] Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual representation by alignment before projection. arXiv preprint arXiv:2311.10122. 2023.
[41] Kevin Lin, Lijuan Wang, and Zicheng Liu. End-to-end human pose and mesh reconstruction with transformers. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 1954-1963, 2021.
[42] Hung Yu Ling, Fabio Zinno, George Cheng, and Michiel Van De Panne. Character controllers using motion vaes. ACM Transactions on Graphics (TOG), 39(4):40-1 2020.
[43] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances neural information processing systems, 01334892-34916 2023.
[44] Mingshuang Luo, Ruibing Hou, Hong Chang, Zimo Liu, Yaowei Wang, and Shiguang Shan. Mgpt: An advanced multimodal, multitask framework for motion comprehension and generation. CoRR. 2024.
[45] Tongxu Luo, Jiahe Lei, Fangyu Lei, Weihao Liu, Shizhu He, Jun Zhao, and Kang Liu. Moelora: Contrastive learning guided mixture of experts on parameter-efficient fine-tuning for large language models. arXiv preprint arXiv:2402. 12857, 2024.
[46] Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. Video-chatgpt: Towards detailed video understanding via large vision and language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL 2024), 2024.
[47] Mathis Petrovich, Michael J Black, and Gül Varol. Temos: Generating diverse human motions from textual descriptions. Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXII, pages 480-497. Springer, 2022.
13

---

[48] Ekkasit Pinyoanuntapong. Pu Wang, Minwoo Lee, and Chen Chen. Mmm Generative masked motion model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1556, 2024.
[49] Matthias Plappert, Christian Mandery, and Tamim Asfour. The kit motion-language dataset. Big data, 444):236- 2016.
[50] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748 8763. PMLR, 2021.
[51] Davis Rempe, Tolga Birdal, Aaron Hertzmann, Jimei Yang. Srinath Sridhar, and Leonidas J Guibas. Humor: 3d human motion model for robust pose estimation. In Proceedings of the IEEE/CVF international conference on computer vision, pages 1 11488-11499, 2021.
[52] Yoni Shafir, Guy Tevet, Roy Kapon, and Amit Haim Bermano. Human motion diffusion as a generative prior. In The Twelfth International Conference on Learning Representations, 2024.
[53] Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gadd -aated mixture-of-experrs layer. arXiv preprint arXiv: 701.06688, 2017.
[54] Paul Starke, Sebastian Starke, Taku Komura, and Frank Steinicke. Motion in-betweening with phase manifolds. Proceedings of the ACM on Computer Gruphics and Inteructive Techniques, (33:11-7,,0233
[55] Sebastian Starke, He Zhang, Taku Komura, and Jun Saito. Neural state machine for character- scene interactions. ACM Trans. Graph., 38(6):209-1, 2019.
[56] Sebastian Starke, Yiwei Zhao, Taku Komura, and Kazi Zaman. Local motion phases for learning multi-contact character movements. ACM Transactions on Graphics (TOG), 39(4):54-1, 2020.
[57] Jianwei Tang, Jiangxin Sun, Xiaotong Lin, Wei-Shi Zheng, Jian-Fang Hu, al. Temporal continual learning with prior compensation for human motion prediction. Advances in Neural Information Processing Systems, 36:65837-65849, 2023.
[58] Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju. Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08295, 2024.
[59] Guy Tevet, Brian Gordon, Amir Hertz, Amit H Bermano, and Daniel Cohen-Or. Motionclip: Exposing human motion generation to clip space. In European Conference Computer Vision pages 358-374. Springer, 2022.
[60] Guy Tevet, Sigal Raab, Brian Gordon, Yoni Shafir, Daniel Cohen-or, and Amit Haim Bermano. Human motion diffusion model. In The Eleventh International Conference Learning Repre- sentations, 2023.
[61] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint 13Xiv 2023.
[62] Timo Von Marcard, Roberto Henschel, Michael J Black, Bodo Rosenhahn, and Gerard Pons- Moll. Recovering accurate 3d human pose in the wild using imus and moving camera. In Proceedings of the European conference on computer vision (ECCV), pages 601-617, 2018.
[63] Luyang Wang, Yan Chen, Zhenhua Guo, Keyuan Qian, Mude Lin, Hongsheng Li, and Jimmy S Ren. Generalizing monocular 3d human pose estimation in the wild. In Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops, pages 0-0, 2019.
[64] Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan Song, Jiazheng Xu, Bin Xu, Juanzi Li, Yuxiao Dong, Ming Ding, and Jie Tang. Cogvlm: Visual expert for pretrained language models, 2023.
14

---

[65] Yuan Wang, Di Huang, Yaqi Zhang, Wanli Ouyang, Jile Jiao, Xuetao Feng, Yan Zhou, Pengfei Wan, Shixiang Tang, and Dan Xu. Motiongpt-2: A general- purpose motion- languuge model for motion generation and understanding. arXiv preprini urXiv:2410.21777, 2024.
[66] Zhiyong Wang, Jinxiang Chai, and Shihong Xia. Combining recurrent neural networks and adversarial training for human motion synthesis and control. IÊEE transactions on visualization and computer graphics, 27(1):14-25 28, 2019.
[67] Qi Wu, Yubo Zhao, Yifan Wang. Xinhang Liu, Yu-Wing Tai, and Chi-Keung Tang. Motion- agent: A conversational framework for human motion generation with llms. arXiv preprint arXiiv2205.. / 30133, 2024.
[68] Shengqiong Wu, Hao Fei, Leigang Qu, Wei Ji, and Tat-Seng Chua. Next-gpt: Any-to-any multimodal 1mm. In Forty-firsi International Conference on Machine Learning, 2024.
[69] Yongjing Ye, Libin Liu, Lei Hu, and Shihong Xia. Neural3points: Learning to generate physically realistic full-body motion for virtual reality users. In Computer Graphics Forum, volume 41, pages 183-194. Wiley Online Library, 2022.
[70] Ted Zadouri, Ahmet Üstün, Arash Ahmadian, Beyza Ermis, Acyr Locatelli, and Sara Hooker. Pushing mixture of experts to the limit: Extremely parameter efficient moe for instruction tuning. CORR, 2023.
[71] Jun Zhan, Junqi Dai, Jiasheng Ye, Yunhua Zhou, Dong Zhang, Zhigeng Liu, Xin Zhang, Ruibin Yuan, Ge Zhang, Linyang Li, et al. Anygpt: Unified multimodal Ilm with discrete sequence modeling. arXiv preprint arXiv:2402. 12226. 2024.
[72] He Zhang, Sebastian Starke, Taku Komura, and Jun Saito. Mode-adaptive neural networks for quadruped motion control. ACM Transactions Graphics (TOG), 37(4):1-11, 2018.
[73] Jianrong Zhang, Yangsong Zhang, Xiaodong Cun, Shaoli Huang, Yong Zhang, Hongwei Zhao, Hongtao Lu, and Xi Shen. T2m-gpt: Generating human motion from textual descriptions with discrete representations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), June 2023.
[74] Mingyuan Zhang, Zhongang Cai, Liang Pan, Fangzhou Hong, Xinying Guo, Lei Yang, and Ziwei Liu. Motiondiffuse: Text-driven human motion generation with diffusion model. arXiv preprint arXiv: 15001, 2022.
[75] Mingyuan Zhang, Xinying Guo, Liang Pan, Zhongang Cai, Fangzhou Hong, Huirong Li, Lei Yang, and Ziwei Liu. Remodiffuse: detrieval-augmenne motion diffusion model. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 364-373, 2023.
[76] Yaqi Zhang, Di Huang, Bin Liu, Shixiang Tang, Yan Lu, Lu Chen, Lei Bai, Qi Chu, Nenghai Yu, and Wanli Ouyang. Motiongpt: Finetuned llms are general-purpor motion generators. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 7368-7376, 2024.
[77] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Tianle Li, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zhuohan Li, Zi Lin, Eric Xing, et al. Lmsys-chat-Im: A large-scale real-world IIm conversation dataset. In The Twelfih International Conference on Leurning Representations, 2024.
[78] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595-46623, 2023.
[79] Chongyang Zhong, Lei Hu, Zihao Zhang, and Shihong Xia. Att22: Text-driven human motion generation with multi- perlpiiperseettiee attention mechanism. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 509-519, 2023.
[80] Chongyang Zhong, Lei Hu, Zihao Zhang, Yongjing Ye, and Shihong Xia. Spatio-temporal gating- aading-adjacency gen for human motion prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6447-643 6456, 2022.
15

---

NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No] or [NA]. .
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justificatin right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No]" ", is perfectly acceptable to answer "[No] "provided proper justification is given (c.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). ). general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only the provided macros for your answers.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction accurately describe the proposed framework (HMVLM, MoE LoRA) and experimental scope (MT-benchmark, R R-precision, MPJPE), which align with the content presented in the paper (Sections 3, 5, 6, and Supplements).
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
16

---

Justification: Section 5 discuss limitations, including challenges in modality connections and any-to-any generation.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proof's
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: Please refer to Section 3.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: The paper describes the experimental datasets Section 4, and evaluation metrics in each task. Appendix D provides hyperparameter and data processing details.
Guidelines:
17

---

The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open- source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome deseribe the particular way they provide for reproducibilityy In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
The code will be released.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
ios.oodeeeeamiiiioooooooooiiie for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
//ipps.cc/publi///uuu
Codeeuumiiii wettsbbmmssiiioi ssionoolicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
18

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: Experimental settings are detailed in Section and the Appendix D. Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Tables 1, 2, 3, 4 and appendix tables report results.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The paper specifies the hardware used (single A800 80G GPU) in Appendix.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
19

---

The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics pt httpppulll/Ehhsssuuuttt
Answer:
[Yes]
The research involves algorithmic development and evaluation on standard benchmarks. It does not involve obviously ethically sensitive applications.
Justification:
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [No]
Justification: The paper focuses on the technical contributions and does not include specific discussion of broader positive or negative socictal impacts.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (c.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
20

---

The paper proposes new human motion reeiian-reevvant multimodal framwork. Neither the model nor the standard benchmark data used appear to pose a high risk for misuse necessitating specific release safeguards beyond standard open-source practices. Guidelines:
Justification:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [No]
Justification: The paper properly cites sources for existing assets like baseline methods and data sources.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, saperswithcode cww/ com/datasete has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to asset's s creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
The primary new asset is the Appendix and supplementary video for the proposed methods.
Justification:
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
21

---

14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The research does not involve crowdsoureing experiments research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: The research does not involve human subjects, therefore IRB approval is not applicable.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
Declaration LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: LLM is used only for writing.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should be described.
22

---

Appendix
This appendix provide qualitative results(SecAA) A) additional experiments(Scc visuiiittis and implementation details(Sec
Video. We also provide the supplementary video to showcase our comparisons with SOTA methods and application examples of our approach, including text-to-motion pose estimation, and human motion video understanding.
A
Qualitative Results
A.1
Qualitative Comparison of Knowledge Retention
Fig. 0 presents a qualitative comparison of the knowledge retention in foundation models across different methods, including MotionAgent [67], MotionGPT (76), and our method. The input prompt sampled from the MT-Bench writing topic.
The left side of Fig 6 shows the foundation model outputs before instruction tuning on the text-to- motion task. All of the models generate well-structured, relevant emails with reasonable suggestions, fully satisfying the prompt requirements. The right side of the figure shows the results after instruction tuning. MotionAgent nearly completely loses the dialog capability of its foundation model, with outputs overwhelmed by repetitive "<Motion <<Motion_ndexx>" iindexx" tokens. This is caused by the introduction of LoRA matrices
new motion vocabulary, modifications the prediction head, and the application across all Transformer modules (Query, Key, Value, and Projection), resulting in severe overfitting to the motion-related task and catastrophic forgetting the original abilities. MotionGPT, which preserves the original prediction head and applies LoRA only the Query and Value matrices, retains a portion the foundation model' knowledge. However, its responses are noticeably more terse, suggesting certain degree of knowledge degradation. Moreover, Tab. 2 in the main text shows that this tin-ttningg strategy cannot achieve superior downstream task performance.
In contrast, our method adopts the MoE LoRA architecture, which enables expert routing through gating network even when LoRA is applied across all linear modules. The zero expert mechanism allows the model to fallback to the original parameter path when handling normal dialogue tasks, effectively preserving the foundational knowledge and achieving a good balance between avoiding model forgetting and enhancing downstream task performance.
A.2 Qualitative Comparison of Text-to-Motion
We compare our method with state-of-the-art foundation model-based T2M approaches on HImanMI33D test set. We use their publicly released pretrained models and LoRA parameters. The results are shown in Fig 7 poses or motions with semantic errors are highlighted with red boxes. In the left example, the prompt requires squat followed by running in place, but neither MotionAgent nor MotionGPT generates the squatting motion. More qualitative results can be found in supplementary video.
A.3 Qualitative Comparison of Pose Estimation
We conducted qualitative comparisons on the 3DPW dataset, following its official training and test split. compare method with ChatPose (16), as shown in Fig. g The results demonstrate that our approach outperforms ChatPose in body pose accuracy and semantic consistency.
A.4 Human Motion Video Understanding
Human motion RGB video understanding. We provide some examples of human motion video understanding. The results are shown in Fig. [2] all the video are extracted from MoVid dataset [5). For more qualitative results, please refer to supplementary vidco.
23

---

Write a persuasive email to convince your introverted friend, who dislikes public speaking, volunteer as guest speaker at local event.....
Subject: Speaking Up for a Cause You Care About Hey [Friend's Name],
Write a persuasive email to convince your introverted--Mtion 55> to make presentation a local event. Use compelling arguments and address potential objections-Motion 458-
Remember how you were talking about [mention their area of expertise/paxxion was thinking, wouldn't it it amazing to share your knowledge and passion with the community? know......
<Motion 358"...... Motton 1o tion 243toMMttoon 358-WMMotion 3-W=Motion MotioM 89><Mofi on Motion 128=-Mottin 247=<<Motion 28tof- 24tooMotion 247==M
247-nMotion 89
Best, (Your Name]
otion 128-of=Motion 247=<Motion 47><Motion 247
Foundation Model (Gemma-2-2b-i3 2-2b-it)
After T2M Instruction Tuning (MotionAgent)
important consider your friend's inferests preferences planning suggestions
Here persuasive email encourage your friend volunteer guest speaker upcoming local
help you craft compelling email: Make the first paragraph your compelling In this opening paragraph, establish your friend's expertise and convey the significunce of the event.
Share your friend's enthusiusm for the event's
email
Communicate how volunteering this event will help your friend develop public speaking skills.
2. Explain why your friend should be guest speaker. Share relevant information from your friend's
Use logic to explain how voluntecring at this event will help your friend grow professionall......
background......
Foundation Model (Llama-2 2-7b)
After T2M Instruction Tuning (MotionGPT)
Subject: Volunteer as a Guest Speaker at Local Event
Let's challenge our comfort zones together!
Subject:
Dear [Friend's Name],
Dear [Friend's Name],
hope email finds you well.
hope email finds you well. was thinking about the upcoming local event......
understand that public speaking not your forte......
Here are compelling reasons why you consider voluntcering guest
compelling arguments
consider:
speaker:
1. Share your expertise: You have wealth knowledge and experience your field. sharing your expertise, you inspire and educate others, making positive impact the community....
your expertise: You have unique perspective and valuable knowledge share. By speaking event. you'll have the chance educate and inspire others......
Opportunity
Foundation Model(vicuma-7--v..55
After T2M Instruction Tuning(Ours)
Figure 6: Qualitative results of question-answering dialogues from foundation language models with different methods, before and after instruction tuning on text-to-motion tasks.
24

---

person doing squats, then rons in place and stops."
"Person moves forward two steps then does full turn and faces opposite direction.
Real
Our
MotionAgent
MotionGPT
Figure 7: Qualitative comparison on text-to-motion task. The provided state-of- the-hhr-rt methods are under the same training and inference setting on HumanML3D [21]. The red box highlights the poses that do not match the prompt semantics.
Input Image
Ours
ChatPose
GT
Input Image
Ours
ChatPose
GT
Figure 8: Qualitative comparison with the ChatPose 16 on the 3DPW test set.
25

---

Input Prompt: What kind of exercise is the woman in the video doing?
The woman in the video is performing the G'an P'ang exercise, Tai Chi form that involves precise movements of the hands and body while standing in specific pose.
Output:
Input Prompt:
Describe the human motion video concisely.
The video shows a man performing a windmill dunk while wearing basketball shoes. He starts with hand toss of the basketball and ends with dunk. The camera zooms in and out during the action.
Output:
Figure 9: Examples of human motion RGB video understanding.
Semantic cycle invariance testing, We combine text-to-motion generation with video understanding tasks evaluate whether the semantic consistency is preserved after the cycle. As shown in the qualitative results in Fig. TO. we first generate a 3D motion based on the input text prompt, then render the 3D motion sequence into 2D character video. The model is then tasked with understanding the motion video (using fixed prompt: "Describe the rendered human motion video concisely"), and finally, we examine the semantic alignment between the input and output texts. In the semantic cycle test, we find that the motion semantics are largely preserved, but errors and hallucinations still occur. Our video understanding strategy involves sampling frames from the video and feeding them into the foundation model. However, if key poses are missed during sampling or represent only small portion of the entire motion sequence, semantic inaccuracies may arise. For example, the left part of Figure I0 kneeling is mistakenly identified as squatting.
B Additional Experiments
The evaluation results on the KIT-ML dataset 49 are shown Table 6 This section adopts the same evaluation metrics used for the HumanML3D dataset. Similarly, Our method with single task instruction tuning outperforms other foundation model-based text-to-motion approaches across most metrics.
C Visualization
As shown in Fig. we visualize the forgetting effects foundation language models before and after the text-to-motion task for our method (Vicuna-7B-vl.5 based), our method (Gemma-2-2B-ir based), MotionGPT, and MotionAgent under the MT Bench [78] benchmark. This corresponds
26

---

Text-to-Motion Task
"A man kneels down then stands back up.
"a man walks around in circle coumterclockvise."
Motion Video Understanding Task
"Describe the rendering human motion video concisely."
"Describe the rendering human motion video concisely.
"The clip shows a person performing a forward turn, a lateral step, and a turn back across square platform."
"The human motion shows a person briefly squatting and standing up.
Figure 10: Semantic loop invariance testing.
Tab. i the main text. MT Bench covers eight topics, including writing, roleplay, extraction, reasoning, and more. In Fig. 11 foundation models. It can be observed that our method (based on Gemma-2-2B-it) and MotionAgent exhibit significant differences in their anti-forgetting ability under the same foundation model, which attributed to our MoE LoRA architecture.
greater overlap of the circles indicates less forgetting in the
D Implementation Details
We conduct experiment on a single NVIDIA A800 80G GPU. the pose and motion tokenizer, the number of body parts is set 5, corresponding to the torso and four limbs with each part having an embedding dimension of S 512. The codebook size for each body part is fixed at K 512 and temporal compression ratio is set to 4 in motion tokenization. During training, the commitment loss coefficient Xeom set to 0.02. We use the Adam W optimizer with hyperparameters 32] [0.9, 0.99) and learning rate of x 10-4
During the instruction tuning stage, simultancous fine-tuning on three human motion-related tasks took total of 120 hours. We used batch size of 32 and a micro batch size of 2. AdamW is also used for optimization in this stage, with an initial learning rate of 3 x 10-3, 3 which is scheduled using Cosine Annealing.
When calculating the MoE LoRA inference latency (as shown in (ggure5((b) we use Vicuna-7b-v1.5 as the foundation model. The inference time is the sum of the gating network computation and multimodal inference time. We conduct latency tests with batch size of and without using caching. Since the model's inference time is related to the number of input tokens, we fix the input token length to 84 when testing latency under different numbers of experts, as this is the average token length in the T2M test set.
27

---

Table 6: Quantitative results of text-to-motion on the KIT-ML dataset
precisiont
Methods
FID.
MM-D.1
Div.
Top-1
Top-2
Top-3
GT
0.424 + 008
0.649 + 006
0.779
0.031
1.004
2.788
010
11.080
097
TM2T 22
0,280 000
0,463
007
0.587
1.005
3.599
1.153
4.591
120
9,473
#.117
T2M
21
0,361 1:006
0.559 + 007
0.681
+.007
3.022
1.107
3,488
1.028
10.720
1.145
MDM
60
0.164 = 004
0.291 = 000
0.396
= .04
0.497
=.021
9.191
-20
10.847
100
MD 74
0.417 = 000
0.621
= .00
0.739
:00
1.954
#.00
2.958
005
11.100
143
MLD 6
0.390 008
0.609
.08
0.734
000
0.404
=2027
3.204
020
10.800
1.1
T2M-GPT 7
0.416 1.006
0.627
1. 1.0006
0.745
1.006
0.514
1.009
3.007 1.023
10.921
1.108
ReMoDiffuse
75
0.427 1.014
0.641
1 .004
0.765
1.055
0.155
1.006
2.814
1.012
10.800
10
AuT22
79
0.413 1.006
0.632
1.006
0.751
1.006
0.870
1.039
3.039
1.021
10.960
1.123
MoMask
20
0.433 + 007
0.656
= 000
0.781
= 000
0.204
10011
2.779
0.02
10.711 0.0
MotionGPT 76
0.340 1.002
0.570
1000
0.660
1.004
0.868
1.032
3.721
1.018
9.972
1:026
Mo
22
0.366 1:005
0.558
1:004
0.680
1.005
0.510
1.016
3.527
1.021
10.350
1.084
Motion gent 67
0.409
0.624
007
0.750
1 1.005
0.781
1.026
2,982
1:022
11.407
103
MotionGPT-2
665
0.427 000
0.627
+ 000
0.764
1000
0.614 1.005
3.164
1.013
11.256
026
Ours (single task)
0.423 100
0.643 + 0003
0.769
000
0.306
-0014
2.848
1.01
11.175
0.9
Ours
0.381 + .005
0.585 + 003
0.680
201
0.567 + 028
3.404
2.019
10.595
1.125
CTM
Emadmn
(a) Ours (Vicuna-7B--ae 7B-v1.B 15 based)
(b) MotionGPTo ((LannPPPPLllaaPPP based)
(0) Ours (Gemma-2-223 based)
(d) Motion-Agent (Gemma--222 2-2B-it based)
Figure 11: Visualization of the forgetting levels before and after T2M fine-tuning across different
methods.
D.1 Evaluation Metrics
This metric evaluates the consistency between the generated motion and the textual description. Specifically, a generated motion paired with its corresponding ground-truft text and 31 randomly selected unrelated text descriptions to form a candidate set. Features of the motion and all text descriptions are extracted using their respective encoders [21]. and pairwise feature distances are computed. These distances are then ranked in ascending order. The probabilities of the ground-truth
R-precision:
text appearing in the 1, Top-2, and Top-3 positions are reported as the evaluation result. Higher values indicate that the generated motion better aligns with the semantic description.
28

---

FID: This metric measures the distributional difference between generated motions and real motions from the dataset. A lower FID score indicates that the generated motions are more similar to real samples in terms of overall feature distribution.
MultiModal Distance (MM-D.): This metric evaluates semantic alignment by computing the Euclidean distance between the text feature and the corresponding generated motion feature. A smaller value indicates better semantic matching.
Diversity (Div):
This metric assesses the diversity of motions generated by the model. Specifically, two subsets, each containing 300 randomly selected generated motions, are sampled. The average Euclidean feature distance between the two subsets is calculated. A larger value indicates greater diversity in the generated results.
MPJPE: Mean Per Joint Position Error measures the average Fuclidean distance between the predicted and ground-truth joint positions of a generated 3D human pose. It is computed by calculating the Euclidean distance for each joint every frame and then averaging over all joints and frames. This metric reflects the spatial accuracy of the generated motion, with lower values indicating closer alignment to the ground-truth poses.
Procrustes Aligned MPJPE calculates the MPJPE after applying rigid alignment (including rotation, scaling, and translation) to the predicted poses to remove global transformation differences. This metric focuses on the structural correctness of the predicted pose regardless of its absolute position and scale. Lower values indicate better structural alignment with the ground truth.
PA-MPJJP::
D.2 Instruction Templates
Tab. presents the templates used for our task instruction tuning, including text- to-motion, pose estimation and video understanding. The [MMtion_Placehoollll <Im- age_ age__lacee Placeholder><<Videe Placeholder>, and <Caption Placeholder> respectively represent the motion sequence, image input, video input and textual description from the trianing datasets.
Table 7: Examples of instruction templates for each task
Tunk
Impul
Output
Generate sequence of motion tokens matching the following homan motson description.
<Motion_ Placeholder>
Toxt-to-Metion
Generate sequence motion tokens matching the following homan motion description given the initial token <Motion Placeholder>. Generate sequence motion tokens matching the following human motion description given the last token <Motion Placeholdrs>.
MMoiion_Pascholddr
<Motion Placeholders
Can predict the SMPL pose of the person in this image <Image_ Placcholder>, There is person in the middle of the image, pleas oulput this person's SMPL pose <Image_ Placeholder>.
Placeoleae
<Pose_ Placebolde>>
Pose Estimation
What the human pose this image? Please respond with SMPL, poxe <Image_ Piaccholders.
Placehlaccooder>
What is the person doing in this image? Please output SMPL pose
<Image Placeholder>.
<Pose Placeholder>
There is person in the middle of the image, use SMPL to deseribe the pose <limage_Plachhueee Placeholder>.
<Pose_ Placchhoddr>>>
Write terse but informative summary of the following buman motion video clip
<Video_Placcholdee Placeholder>.
<Video_
<Caption_Plaaeeettli Placeholders
Describe the following haman motion video conciscly <Vidco Placcholdars. Render a clear and concise summary of the human motion video below <Viide__lluceholde
Caption__lascoollee
Video Understanding
<C CCpption_ PlacehulPa
<Video_ Placeholder>.
Share concise interpretation of the human motion video provided
<Nidco_ Placeholders.
Placcoo_Paschoodder>
Relay brief, clear account of the human motion vidco shown
<Caption_Placeeaei Placeholders
<Video_ Placeholders.
<Cophio__Pasccodéerr Placcholder>
29


Reviews:
[{"review_id": "nPCfmbxEI6", "reviewer": "Reviewer_C5Mg", "summary": "This paper presents HMVLM, a unified multimodal framework that integrates 3D human motion with foundation language models. It focuses on two core challenges: mitigating catastrophic forgetting during multimodal instruction-tuning and designing generalizable, autoregressive-compatible pose representations. To address these, the authors propose a Mixture of Expert Low-Rank Adaptation (MoE LoRA) approach, incorporating a \"zero expert\" to preserve general linguistic capabilities of the foundation model. Additionally, they introduce a body-part-specific tokenization scheme to enhance spatial granularity in motion representation. The framework is evaluated on multiple tasks, including text-to-motion generation, monocular pose estimation, and motion video understanding, showing promising performance and improved resistance to knowledge forgetting.", "questions": "1. Zero Expert Selection: Could the authors clarify how the gating network decides when to activate the zero expert? Is there a learned threshold or attention-based mechanism involved?\n\n2. Efficiency of MoE LoRA: While LoRA is known for being parameter-efficient, how does the proposed MoE variant affect training and inference cost, especially as the number of experts increases?\n\n3. Extending to Other Modalities: Could this framework be adapted to include other modalities such as audio or haptics? If so, what are the expected challenges?\n\n4. Sensitivity to Body Part Definitions: How sensitive is performance to the way the body is partitioned? Is there a principled way the authors chose the granularity of the segmentation?\n\n5. Comparison to Other MoE Architectures: How does this method compare to existing MoE models in terms of scalability, performance, and interpretability?", "limitations": "- Modality interactions are modeled in a pairwise fashion, which may not fully exploit cross-modal synergies.\n\n- Generalization to new domains is limited due to dataset-specific training and domain shifts.\n\n- The paper lacks detailed ablations on expert configuration and gating behavior.\n\n- Qualitative results are underrepresented in the main text, making it harder to fully assess the model’s output quality.", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n\n- Mitigation of Catastrophic Forgetting: The use of a \"zero expert\" within the MoE LoRA framework is a compelling idea. It allows the model to retain general language capabilities while adapting to new motion-related tasks, and empirical results support its effectiveness (e.g., only 6.07% drop vs. larger drops in baselines).\n\n- Unified Framework for Multiple Tasks: HMVLM supports various human-centric tasks simultaneously, marking a step forward compared to existing task-specific pipelines.\n\n- Fine-Grained Pose Representation: The proposed body-part-aware tokenization leads to more granular motion encoding, which appears to benefit downstream performance. The approach draws a useful analogy to patch-based image tokenization and applies it in a novel context.\n\n- Strong Experimental Results: Across several benchmarks, the model achieves competitive or superior performance, particularly in text-to-motion generation (e.g., HumanML3D). These results highlight the potential of the proposed architecture.\n\n**Weaknesses**\n\n- Limited Qualitative Analysis: While quantitative results are solid, the paper could be strengthened by including more qualitative visualizations, especially for complex tasks like motion generation. The supplementary materials are mentioned but not integrated into the main paper, limiting the reader’s ability to assess this aspect.\n\n- Modality Fusion Strategy: The framework currently models modality interactions in a pairwise manner. This may limit the depth of multimodal integration, especially when extending to more complex scenarios requiring holistic reasoning across all modalities.\n\n- Domain Generalization: The model’s performance across diverse datasets is affected by domain discrepancies, which challenges its generalization to unseen settings and limits its flexibility for open-world deployment.\n\n- Ablation on MoE Configuration: There is limited discussion on how the number of experts or their specific configurations affect the model. An ablation study would provide deeper insights into the trade-offs involved.", "was_revised": false, "final_justification": "", "rebuttal": "### We sincerely thank the reviewer for constructive feedback and the time spent evaluating our work. We appreciate the recognition of our key contributions, including zero expert for catastrophic forgetting mitigation, strong results and so on. We provide point-by-point responses with the aim of resolving the reviewer’s concerns.\n\nQ: Limited Qualitative Analysis \n\n- Given the page limit, we will rearrange qualitative results in the main paper, highlighting motion generation, with additional examples in the appendix and supplementary video.\n\n\nQ: Modality Fusion Strategy\n\n- While our current framework adopts supervised pairwise training for several downstream tasks, this approach enables the foundation model to effectively absorb knowledge from other modalities and achieve strong performance across different tasks. Additionally, modality fusion in our system is primarily mediated by text, for example, as demonstrated in the supplementary video, we illustrate a loop from text-driven motion generation to 3D motion understanding. This demonstrates that our model is capable of learning generalizable fusion knowledge from different pairwise datasets to a certain extent. \n\n\nQ: Domian Generalization\n\n- Domain discrepancies are inevitable in multimodal learning. As discussed in the manuscript, the large gap between 3D human motion and text often leads to catastrophic forgetting when introducing and fine-tuning 3D motion data. Our proposed MoE LoRA framework with zero expert effectively mitigates domain discrepancies across datasets. Further improving generalization to unseen open-world scenarios would require scaling up data, which is beyond the scope of our work and contribution. \n\n\nQ: Ablation on MoE Configuration \n\n- Increasing the number of MoE experts allows for more flexible combinations and can improve task performance; however, it also leads to a linear increase in training cost. In our implementation, since our method currently supports three downstream tasks and requires a dedicated \"zero expert\" as well as an additional expert for knowledge sharing, we set the number of MoE LoRA experts to 5. We also have conducted an ablation study on the number of experts. When the number of experts is set to 2 (including the zero expert), the performance on the T2M and Pose Estimation tasks decreased by `5.40 %` and `15.9%`, repectively. Conversely, increasing the number of experts to 8 resulted in performance improvements of `3.54%` and `9.76%` on the T2M and Pose Estimation. We will add a table in main text for illustration. \n\n\nQ: Zero Expert Selection\n\n- As illustrated in the lower-left part of Figure 2, the input text features are encoded and fed into the gating network, which performs a classification-like operation. Owing to the design of loss function (7), we explicitly encourage the gating network to produce a weight distribution that is close to $\\alpha_0 =1$ for pure text-based dialogue tasks, thereby preferentially activating the zero expert.\n\n\nQ: Efficiency of MoE LoRA\n\n- During training, due to the mixture of multiple LoRA experts with the pretrained parameters (see Equation (1)), both the number of trainable parameters and the training time increase linearly with the number of experts. However, during inference, only the computation of the gating network's mixture weights is added. As a result, the increase in inference time is constant and almost negligible.\n\n\nQ: Extending to Other Modalities\n\n- The proposed framework can be extended to other modalities. However, this work primarily focuses on a multimodal framework related to 3D human motion and addresses the challenge of catastrophic forgetting observed in previous works such as MotionGPT and MotionAgent.\n\n\nQ：Sensitivity to Body Part Definitions\n\n- We follow the body partitioning strategy from [26] and [76] in the main paper reference, which has proven effective in motion modeling and text-to-motion generation. While finer segmentation (e.g., hands and fingers) could enhance motion modeling precision, most current 3D human motion datasets describe motions at the limb-segment level, which aligns well with our chosen granularity.\n\n\nQ:  Comparison to Other MoE Architectures\n\n- Existing MoE architectures are primarily applied to the FFN layers of foundation models, focusing on single-modality training or fine-tuning (e.g., text). Our method is the first to introduce a zero expert into the MoE LoRA framework to address catastrophic forgetting in multimodal learning. Therefore, a direct comparison to prior MoE approaches is not straightforward."}, {"review_id": "VIt6AYbkjH", "reviewer": "Reviewer_rpyN", "summary": "This paper introduces HMVLM, a unified Human Motion-Vision-Language Model built on a Mixture of Expert Low-Rank Adaptation (MoE LoRA) framework. It targets multiple human-centric tasks, including text-to-motion generation, pose estimation, and motion video understanding. The core techniques are (1) leveraging a gating network to dynamically allocate LoRA expert weights, including a zero-expert to mitigate catastrophic forgetting due to the modality gap between human motion and text, and (2) designing a body-part-based tokenization strategy for fine-grained pose/motion representation compatible with autoregressive architectures. Experimental results show that HMVLM achieves competitive performance on text-to-motion and pose estimation tasks while preserving foundational world knowledge in language models.", "questions": "1. The multitask performance appears degraded compared to single-task baselines, and it is unclear why unifying these tasks is practically better. It would be better to give more evidence or analysis on why a unified framework is preferable, for example through cross-task transfer benefits, data efficiency, or user-facing advantages.\n2.  In Table 4, the part-based tokenizer does not always outperform a single codebook tokenizer (e.g., worse FID). It would be better to explain why these trade-offs happen, and clarify its benefits on the pose estimation task specifically. Additional analysis or an ablation focused on pose estimation could clarify its usefulness.\n3. There is no analysis of the $\\alpha_i$ weights across tasks. For instance, what is the average or distribution of $\\alpha_0$ on dialogue tasks? Providing such analysis could help validate the effectiveness of the gating mechanism and would strengthen the evidence for the proposed zero expert.\n4. It would be better to discuss why simply forcing all $\\alpha_i$ to zero (effectively disabling experts on dialogue tasks) would not be a sufficient alternative. A comparison or reasoning about this hard-routing baseline could help me better judge whether the zero expert mechanism is truly necessary.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1. This paper uses a technically reasonable framework with a zero expert in MoE to protect pre-trained capabilities.\n2. This paper proposes to use part-level tokenisation to improve the presentation.\n\nWeaknesses:\n1. The paper is somewhat difficult to understand. The writing is dense and overloaded with technical jargon, making it challenging to follow the core ideas, the architecture details, and the implementation steps. This hinders accessibility. For example, only after careful and repeated reading does it become apparent that the authors intend to describe the model structure first, followed by the training of the tokenizer and then instruction tuning. However, Sec 3.1 and 3.2 actually are parts of 3.4.\n2. The authors claim to present a unified framework; however, the experimental results show performance degradation when training on multiple tasks compared to specialized single-task baselines. The paper does not sufficiently justify why combining these tasks in one framework is beneficial in practical or real-world settings.\n3. As shown in Table 4, the part-based tokenizer does not consistently outperform a single codebook tokenizer across all metrics (for example, FID is 0.123 vs. 0.110). Some explanation or analysis is missing to justify these trade-offs. Additionally, the impact of the part-based tokenizer on the pose estimation task is not clearly discussed.\n4. The paper lacks analysis of the learned gating weights $\\alpha_i$ for each task. For instance, it would be helpful to report the average value of $\\alpha_0$ for the dialogue task, to better understand how often/much the zero expert is actually selected.\n5.  It is unclear whether explicitly setting all $\\alpha_i$ values to zero (i.e., bypassing all experts) whenever a task instruction indicates a dialogue task could be a simpler alternative to preserve pre-trained knowledge. A discussion comparing this hard routing strategy to the proposed zero expert would strengthen the justification for its necessity.", "was_revised": true, "final_justification": "Thank you for the authors' feedback. The authors have addressed my concerns. I raise my rating from 3 to 4 (Borderline accept).", "rebuttal": "### We sincerely thank the reviewer for the constructive suggestions and the time spent our work. We appreciate the recognition of our contributions, including the zero expert in MoE to preserve pre-trained capabilities, and the introduction of part-level tokenization to enhance motion representation. We provide point-by-point responses with the aim of addressing the reviewer’s concerns.\n\n\nQ： The paper is somewhat difficult to understand.\n\n- Our writing aimed to emphasize the core concept of MoE LoRA, starting from the generalized framework (Sec. 3.1) to the details of downstream task fine-tuning (Sec. 3.3, 3.4). We will refine the presentation in the methods section to make the narrative clearer and the core ideas easier to follow.\n\n\nQ: Performance degradation when training on multiple tasks. Why combining different tasks in one framework is beneficial in practical or real-world settings.\n\n- The higher performance of single-task training arises because all LoRA expert can focus solely on one downstream task under the same parameter budget. The MoE LoRA framework integrates multiple tasks into a unified model, enabling automatic selection of the optimal combination of LoRA experts based on the input semantics, without requiring manual task-specific parameter selection.\n\n\nQ: The part-based tokenizer does not outperform the single-codebook tokenizer across all metrics, and its impact on pose estimation is unclear.\n\n- The slightly higher FID of the part-based tokenizer may stem from the combination of multiple part-specific codebooks, which enriches generated pose diversity but introduces a minor distribution shift from the training set. Nevertheless, our method shows clear improvements over the single-codebook tokenizer in key metrics such as R-precision and MSE. We will add the impact of the part-based tokenizer on the pose estimation task. \n\n\nQ: The paper lacks analysis of gating weights, particularly how often the zero expert is selected for each task.\n\n- For the dialogue task, the average gating weights are `[0.999973, 2.3638 * 10^{-6},  5.0053 * 10^{-6}, 1.3566 * 10^{-6}, 1.8274 * 10^{-6}]`, indicating that the zero expert is predominantly activated to preserve the foundation model’s language capabilities. In contrast, for text-to-motion generation and pose estimation tasks, the average weights are `[0.6938, 0.0522, 0.0671, 0.0850, 0.1018]` and `[0.4543,  0.2919, 0.0844, 0.1062, 0.0632]`, respectively, showing that the model effectively leverages the newly learned LoRA experts for these tasks.\n\n\nQ: Could setting all expert weights to zero for dialogue tasks be a simpler alternative to the zero expert?\n\n- The gating network enables automatic routing among experts, allowing the model to switch between downstream tasks without manual selection of LoRA experts. Setting all expert weights to zero can indeed replicate the effect of $\\alpha_0 = 1$, but it still requires a classifier-like mechanism to identify dialogue-only inputs. Thus, this routing strategy is not simpler than the proposed zero expert approach, while also lacking its flexibility for mixed or ambiguous instructions."}, {"review_id": "MMFdc3cXs1", "reviewer": "Reviewer_sc8H", "summary": "This paper proposes HMVLM, a unified framework to integrate 3D human motion with vision and language modalities into a foundation language model. The work tackles two critical challenges: the catastrophic forgetting of the model's general knowledge when fine-tuned on specialized motion data, and the difficulty of creating motion representations that are both spatially detailed and compatible with autoregressive models. The authors introduce a MoE LoRA strategy, which notably includes a non-trainable zero expert designed to preserve the foundation model's original weights for non-motion tasks. A gating network dynamically routes inputs to different LoRA experts, guided by a specific loss function to prevent knowledge degradation. They propose a body-part-based tokenizer that partitions the human skeleton into semantic parts and encodes them separately to enhance the spatial resolution of the pose representation. The authors conduct experiments on tasks including text-to-motion generation, pose estimation, and motion video understanding, to show HMVLM's knowledge preservation ability and performance on downstream tasks.", "questions": "1. The gating network and the zero expert are central to your contribution. Could you provide more details about the structure of gating network and analysis of the expert weights assigned by the gating network? e.g., showing how the weight for the zero expert changes between linguistic prompts, simple motion tasks, and complex motion tasks would provide excellent insight into whether the framework is behaving as intended.\n\n2. The results in table 2 and 3 show a performance gap between the single task and multi-task models. Could you elaborate on this trade-off? Is this degradation an inherent cost of building a generalized multi-task model with this framework, or potentially mitigated through different training strategies, a different number of experts, or adjustments to the loss functions?\n\n3. The paper mentions partitioning the body into N parts, and Figure 3 seems to suggest 5 parts. How was this specific partition chosen, and how sensitive is the model's performance in both reconstruction and downstream tasks to this design choice, e.g., using more granular parts like hands/feet, or fewer parts.\n\n4. Section 4.1 mentions gating network is trained on LMSYS-Chat-1M. For clarity, could you explicitly state in the main text which datasets were used for the pre-training of the motion and pose tokenizers, respectively? While it can be inferred from Section 3.3 and 4.1, a direct statement would improve readability.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. The paper addresses a crucial problem in multimodal AI that how to incorporate specialized modality like human motion in foundation model without it catastrophic forgetting its pre-existing world knowledge. \n\n2. The proposed MoE LoRA framework and zero expert is a novel and simple solution to catastrophic forgetting. The idea of having a non-trainable expert that acts as identity path for the original model combined with a gating mechanism that learns to route general queries to it, is highly intuitive. The experimental results in Table 1, showing only a ~6% performance drop on MT-Bench compared to competitors' >22% or even >87% drops, provide strong evidence for this claim. Crucial ablation studies are provided for the gating loss.\n\n3. Proposed body-part-based tokenizer inspired by patch-based methods in vision is a well-motivated,  which improves the spatial granularity of pose representations. \n\nWeaknesses:\n\n1. The gating network is central to the MoE LoRA framework. However, the paper only provides an ablation study on its training loss, but does not offer any qualitative analysis of the expert weights it learns. It would be insightful to see how these weights are distributed for different types of prompts The architecture of the gating network is not specified as well. \n\n2. In section 4.3, the paper could benefit from a deeper analysis of the performance trade-off observed when moving from single-task to multi-task training. The results in Tables 2 and 3 show a noticeable performance drop in T2M and pose estimation metrics in the multi-task setting compared to the single-task one. \n\n3. The comparison of knowledge preservation in Table 1 is weakened by the fact that the baseline models MotionAgent and MotionGPT use different foundation models than the proposed HMVLM. The authors fairly acknowledge this and focus on the relative performance degradation, but a more direct comparison against a baseline fine-tuned on the same base model would have been more powerful.", "was_revised": true, "final_justification": "The rebuttal has addressed most of my concerns and strengthened the paper, and I will maintain my score of 4.", "rebuttal": "### We sincerely thank the reviewer for the constructive feedback and the time spent evaluating our work. We are glad to see the recognition of our contributions and method novelty. In this response, we address the questions point-by-point and aim to resolve the reviewer’s concerns.\n\n\nQ： The paper lacks qualitative analysis and architectural details of the gating network, including how expert weights are distributed for different prompts.\n\n- For the dialogue task, the average gating weights are `[0.999973, 2.3638 * 10^{-6}, 5.0053 * 10^{-6}, 1.3566 * 10^{-6}, 1.8274 * 10^{-6}]`, indicating that the zero expert is predominantly activated to preserve the foundation model’s language capabilities. In contrast, for text-to-motion generation and pose estimation tasks, the average weights are `[0.6938, 0.0522, 0.0671, 0.0850, 0.1018]` and `[0.4543, 0.2919, 0.0844, 0.1062, 0.0632]`, respectively, showing that the model effectively leverages the newly learned LoRA experts for these tasks. The gating network architecture is specified in Appendix D. \n\n\nQ: What causes the performance gap between single-task and multi-task models, and can it be mitigated through training strategies, expert numbers, or loss adjustments?\n\n- The higher performance of single-task training arises because all LoRA expert can focus solely on one downstream task under the same parameter budget.  Multi-task performance can be improved by increasing the number of experts—for example, using 8 experts improves Top-1 R precision in the T2M task by `3.54%`. However, this comes at the cost of higher training complexity due to the expanded MoE configuration. We will add a table in main text for illustration. \n\n\nQ: A fairer comparison would use baselines fine-tuned on the same foundation model as HMVLM.\n\n- We will replace the base model in our ablation to match the one used in the baselines, ensuring a more direct and fair comparison. And it is worth noting that the capability of our MoE LoRA framework to preserve the foundation model’s knowledge is not dependent on the specific choice of foundation model.\n\n\nQ:   How was the 5-part body partition chosen, and how sensitive is performance to using more or fewer parts?\n\n- We follow the body partitioning strategy from [26] and [76] in the main paper reference, which has proven effective in motion modeling and text-to-motion generation. While finer segmentation (e.g., feet and fingers) could enhance motion modeling precision and reconstruction, most current 3D human motion datasets describe motions at the limb-segment level, which aligns well with our chosen granularity.\n\n\nQ: Clarify in the main text which datasets were used to pre-train the motion and pose tokenizers.\n\n- We will explicitly add this clarification in the main text. The motion tokenizer is pre-trained on the HumanML3D training set, while the pose tokenizer is pre-trained on the Human3.6M training set."}, {"review_id": "EVJKXVrtlZ", "reviewer": "Reviewer_t2iJ", "summary": "This paper proposes the Human Motion-Vision-Language Model (HMVLM), a unified framework that integrates 3D human motion with foundation language models by leveraging the MoE LoRA strategy. HMVLM introduces a ``zero expert’’ in MoE LoRA to preserve pre-trained linguistic knowledge and adopts body-part-specific tokenization to enhance the spatial resolution in motion/pose encoding. The experimental results demonstrate the effectiveness of the proposed framework.", "questions": "1.Would further subdividing body parts (e.g., into smaller joint groups) improve representation accuracy?\n\n2.Are the other methods in Table 2 (such as MotionGPT) also fine-tuned for single task?\n\n3.Does the MoE strategy incur greater memory and time overhead? It would be better to add a table for illustration.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**:\n1. The MoE LoRA framework with a novel zero expert effectively alleviates catastrophic forgetting problem, which is verified by the maintenance of performance in language tasks.\n2. Body-part tokenization improves spatial detail in pose encoding, boosting downstream task performance.\n3. HMVLM demonstrates competitive results across diverse tasks under multi-task fine-tuning, highlighting its generalizability.\n\n**Weaknesses**:\n1. Some of the symbolic expressions in the paper are difficult for readers to understand. It would be better to simplify them appropriately.\n2. Some relevant works are missing. For example, [1] also proposed a pre-trained model for human motion; [2] also addressed the problem of catastrophic forgetting in human motion. These could be briefly discussed in related work.\n\n**References**:\n\n[1] Gao Y, Luan P C, Alahi A. Multi-Transmotion: Pre-trained Model for Human Motion Prediction[J]. arXiv preprint arXiv:2411.02673, 2024.\n\n[2] Tang J, Sun J, Lin X, et al. Temporal continual learning with prior compensation for human motion prediction[J]. Advances in Neural Information Processing Systems, 2023, 36: 65837-65849.", "was_revised": true, "final_justification": "I appreciate the authors' efforts to address my concerns. I will maintain my current rating and hope the authors can make the modifications as promised in future versions.", "rebuttal": "### We sincerely thank the reviewer for the constructive feedback and the time spent evaluating our work. We also appreciate the recognition of the novelty of our method. Point-by-point responses were provided with the aim of addressing the reviewer’s concerns.\n\nQ：Some of the symbolic expressions in the paper are difficult for readers to understand. It would be better to simplify them appropriately.\n\n-  We will simplify and reorganize the symbols in the main text.\n\n\nQ: Some relevant works are missing.\n\n-  We will add the suggested citations in the appropriate sections of the related work and briefly discuss their connections to our approach.\n\n\nQ：Would further subdividing body parts (e.g., into smaller joint groups) improve representation accuracy?\n\n- Finer segmentation (e.g., into smaller joint groups ) could enhance motion modeling precision and decrease the MSE in motion reconstruction. However most current 3D human motion datasets describe motions at the limb-segment level, which aligns well with our chosen granularity.\n\n\nQ: Are the other methods in Table 2 (such as MotionGPT) also fine-tuned for single task?\n\n- Yes. These works apply foundation language models to a single downstream task, i.e., text-to-motion and are therefore fine-tuned for single-task settings. \n\n\nQ: Does the MoE strategy incur greater memory and time overhead? It would be better to add a table for illustration.\n\n- During training, due to the mixture of multiple LoRA experts with the pretrained parameters (see Equation (1)), both the number of trainable parameters and the training time increase linearly with the number of experts. However, during inference, only the computation of the gating network's mixture weights is added. As a result, the increase in inference time is constant and almost negligible. We will add a table for illustration."}]

Author rebuttal:

Meta-review: Despite some presentation density and trade-offs in multi-task performance, the paper offers a useful and novel mechanism, which is zero expert in MoE-LoRA, that reduces catastrophic forgetting in a challenging multimodal setting, plus a thoughtful pose tokenization that improves spatial fidelity. The added analyses during discussion resolved core concerns about how the gating behaves and why the zero expert is preferable to simpler routing. 

For the next revision, please include a small table or plot showing the gating network's expert weight distributions for representative prompts, along with a brief description of the gating architecture. It would also help to add a compact cost table for number of experts vs trainable parameters, training time, inference latency. Please deepen the multi-task discussion and explain why part-based tokenization can increase FID yet improve R-precision/MSE. Finally, include matched-backbone baselines and add the cited works on motion pretraining and continual learning.
Decision: Accept (poster)